http://www.java.net/forum/topic/glassfish/glassfish/sslerrorrxmalformedcertrequest-two-way-ssl-authentication

In Glassfish 3.1.1 I have two-way ssl authentication and cacerts.jks has 498 certificates now. When I have 516 entries (size 487KB) the server is starting but when I try to connect is rising:

    "Secure Connection Failed SSL received a malformed Certificate

I think the user is also seeing in OpenSSL:

    ssl_error_rx_malformed_cert_request

I tried to increase the allocated memory" the same result. The only way to make it to run is to delete one certificate.

This is probably because there is an hardcoded limit in the protocol: The CertificateRequest message must specify the DNs of accepted CA. This DNs, all together, can occupy at the most 2^16-1= 65535 bytes, so if there are too many CAs, this limit can be encountered.

For reference, this is paragraph 7.4.4 of RFC 2246. It describes TLS 1.0, but there is little significant difference between TLS and SSL for what interest us.