A DESCRIPTION OF THE PROBLEM :
When using a PKCS11 keystore (using the SunPKCS11 Provider) for client authentication in TLSv1.2, the client chooses a signature algorithm which does not support the P11PrivateKey.

The server sends a list of supported SignatureSchemes for use in client authentication. The method sun.security.ssl.SignatureScheme.getPreferableAlgorithm(List<SignatureScheme>, PrivateKey, ProtocolVersion) chooses which SignatureScheme to use in client authentication. It can choose any algorithm for which a provider exists, even if the algorithm's java.security.Provider$Service.supportsParameter(privateKey) == false. In this case it chooses RSASSA-PSS from the provider sun.security.rsa.SunRsaSign.

Because of this, signing with P11PrivateKey fails (with a misleading message stating no providers support the key, when it's a question of the <algorithm name> implementations not supporting the key):

Caused by: javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:590)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:740)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1356)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1331)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:241)
<snip>
Caused by: java.security.InvalidKeyException: No installed provider supports this key: sun.security.pkcs11.P11Key$P11PrivateKey
at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1163)
at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1204)
at java.base/java.security.Signature.initSign(Signature.java:546)
at java.base/sun.security.ssl.SignatureScheme.getSignature(SignatureScheme.java:473)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:580)
... 26 more

The problem cannot be worked around by adding RSASSA-PSS to the java.security configuration jdk.tls.disabledAlgorithms property.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Regular Java TLS client using PKCS11 KeyStore from the SunPKCS11 provider for client authentication.

TLSv1.2 server with client authentication, with supported_signature_algorithms preferring RSASSA-PSS (not supported by PKCS11 provider) over RSA_PKCS1_SHA256 (supported by PKCS11 provider).

Open connection from client to server.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The client chooses the RSA_PKCS1_SHA256 signature algorithm. The TLS connection is established with PKCS11 for client authentication.
ACTUAL -
The client chooses the RSASSA-PSS signature algorithm. The client cannot complete TLS handshake because it cannot use the P11PrivateKey in the signature algorithm.

CUSTOMER SUBMITTED WORKAROUND :
Configure server so that supported_signature_algorithms prefers signature algorithms supported by the SunPKCS11 provider (RSA_PKCS1_SHA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA_512, RSA_SHA224, RSA_PKCS1_SHA1).

FREQUENCY : always