ADDITIONAL SYSTEM INFORMATION :
OS: Linux and Mac
Arch: x86-64, Darwin Arm64
~~~
openjdk 19.0.2 2023-01-17
OpenJDK Runtime Environment Temurin-19.0.2+7 (build 19.0.2+7)
OpenJDK 64-Bit Server VM Temurin-19.0.2+7 (build 19.0.2+7, mixed mode)
~~~


A DESCRIPTION OF THE PROBLEM :
The SunPKCS11 module introduced a [change][1] where it attempts to verify that the digest signature is available as mechanism from the PKCS11 provider. In case of NSS, it doesn't return `CKM_SHA256` as an available mechanism, hence the certificate verification fails in Java 19 (Java 16 and above). The same code works in Java 11. This particular line of code is executed during ssl handshake on server side.

Note that only PKCS11+nss with RSA key fails. Using SoftHSM works as it does return CKM_SHA256 as a available mechanism.

[1]: https://github.com/openjdk/jdk17u/blob/2fe42855c48c49b515b97312ce64a5a8ef3af407/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11PSSSignature.java#L425-L428

REGRESSION : Last worked in version 11.0.18

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The complete steps and instructions to reproduce are documented at https://github.com/usmansaleem/pkcs11_tls_demo

In summary,
1. Create nss database, load RSA private key and trusted CA certificates
2. Use SunPKCS11 security provider and load the Keystore from the nss database
3. Create SSLServerSocket with above keystore (and truststore) as keymanager and trustmanager.
4. Set `sslServerSocket.setNeedClientAuth(true);`
5. Call `sslSocket.startHandshake();`
6. Establish a connection from the client (Use Client.java or curl command)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The SSL handshake is successful and connection is established with the client.
ACTUAL -
In Java16 and above, an exception is raised during handshake.
~~~
javax.net.ssl|ALL|10|main|2023-01-31 14:32:42.328 AEST|SignatureScheme.java:625|Ignore unsupported signature algorithm (rsa_pss_rsae_sha256) (
"throwable" : {
  java.security.InvalidAlgorithmParameterException: Unsupported digest algorithm: SHA-256
   at jdk.crypto.cryptoki/sun.security.pkcs11.P11PSSSignature.setSigParams(P11PSSSignature.java:434)
   at jdk.crypto.cryptoki/sun.security.pkcs11.P11PSSSignature.engineSetParameter(P11PSSSignature.java:745)
   at java.base/java.security.SignatureSpi.engineInitSign(SignatureSpi.java:161)
   at java.base/java.security.Signature$Delegate.tryOperation(Signature.java:1320)
   at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1272)
   at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1384)
   at java.base/java.security.Signature.initSign(Signature.java:683)
   at java.base/java.security.Signature$1.initSign(Signature.java:147)
   at java.base/sun.security.util.SignatureUtil.initSignWithParam(SignatureUtil.java:189)
   at java.base/sun.security.ssl.SignatureScheme.getSigner(SignatureScheme.java:617)
   at java.base/sun.security.ssl.SignatureScheme.getSignerOfPreferableAlgorithm(SignatureScheme.java:537)
   at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:903)
   at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1111)
   at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1104)
   at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
   at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1246)
   at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1182)
   at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:840)
   at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:801)
   at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
   at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
   at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
   at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
   at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
   at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
   at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
   at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
   at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
   at Server.main(Server.java:54)
   at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
   at java.base/java.lang.reflect.Method.invoke(Method.java:578)
   at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:434)
   at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:205)
   at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:132)}

)
~~~

~~~
Exception in thread "main" javax.net.ssl.SSLException: No supported CertificateVerify signature algorithm for RSA key
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:132)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:305)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:911)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1111)
at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1104)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1246)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1182)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:840)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:801)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at Server.main(Server.java:54)
~~~

---------- BEGIN SOURCE ----------
The complete source code is also available at https://github.com/usmansaleem/pkcs11_tls_demo/. The code assumes that nss database and client keystore are already setup. Invoke `java -Djavax.net.debug=ssl:handshake:verbose Server.java` and `java -Djavax.net.debug=ssl:handshake:verbose Client.java` respectively from different terminal windows.

Server.java
~~~
import javax.net.ssl.*;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.security.*;
import java.security.cert.X509Certificate;

public class Server {
    public static void main(String[] args) throws Exception {
        System.out.println("Initializing SunPKCS11 Provider...");
        final String keystorePassword = "test123";
        final Path nssConfigPath = Path.of("./server/pkcs11.cfg");
        //final Path nssConfigPath = Path.of("./server/pkcs11_softhsm.cfg");
        final Provider uninit_Provider = Security.getProvider("SunPKCS11");
        final Provider provider = uninit_Provider.configure(nssConfigPath.toString());
        Security.addProvider(provider);

        final KeyStore keystore = KeyStore.getInstance("PKCS11", provider);
        keystore.load(null, keystorePassword.toCharArray());


        final KeyManagerFactory kmf = KeyManagerFactory.getInstance("PKIX");
        kmf.init(keystore, keystorePassword.toCharArray());
        final TrustManagerFactory tmf = TrustManagerFactory.getInstance("PKIX");
        tmf.init(keystore);

        final SSLContext sslContext = SSLContext.getInstance("TLSv1.3");
        sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);

        // Use the SSL context to create an SSLServerSocketFactory
        SSLServerSocketFactory sslServerSocketFactory = sslContext.getServerSocketFactory();

        // Create an SSLServerSocket that listens on port 8443
        SSLServerSocket sslServerSocket = (SSLServerSocket) sslServerSocketFactory.createServerSocket(8443);

        // Configure the SSLServerSocket to require client authentication
        sslServerSocket.setNeedClientAuth(true);

        // Start listening for incoming connections
        while (true) {
            System.out.println("Waiting for client...");
            SSLSocket sslSocket = (SSLSocket) sslServerSocket.accept();
            System.out.println("Starting handshake ...");
            sslSocket.startHandshake();
            System.out.println("Handshare successful");

            // Get the client's certificate chain
            X509Certificate[] clientCertificates = (X509Certificate[]) sslSocket.getSession().getPeerCertificates();
            X509Certificate clientCertificate = clientCertificates[0];

            // Verify the client's certificate
            clientCertificate.checkValidity();
            // trustStore.getCertificateAlias(clientCertificate) != null
            // additional check on clientCertificate

            InputStream in = sslSocket.getInputStream();
            OutputStream out = sslSocket.getOutputStream();

            // read the incoming request
            BufferedReader reader = new BufferedReader(new InputStreamReader(in));
            String request = reader.readLine();

            // check GET
            if (request != null && request.startsWith("GET")) {
                // Write "HTTP/1.1 200 OK\n" to the client
                out.write("HTTP/1.1 200 OK\n".getBytes());
                out.flush();
                out.write("Content-Type: text/plain\n".getBytes());
                out.flush();
                out.write("Content-Length: 2\n".getBytes());
                out.flush();
                out.write("\n".getBytes());
                out.flush();
                out.write("OK".getBytes());
                out.flush();
            }

            // Close the socket
            sslSocket.close();
        }

    }
}
~~~

pkcs11.cfg
~~~
name = NSScrypto-compa}-server
nssSecmodDirectory = ./server/nssdb
nssDbMode = readOnly
nssModule = keystore
showInfo = true
~~~

Client.java
~~~
import java.net.*;
import java.io.*;
import javax.net.ssl.*;
import javax.security.cert.X509Certificate;
import java.security.KeyStore;


public class Client {

    public static void main(String[] args) throws Exception {
        String host = "localhost";
        int port = 8443;

        SSLSocketFactory factory = null;
        SSLContext ctx;
        KeyManagerFactory kmf;
        TrustManagerFactory tmf;
        KeyStore ks;
        KeyStore ts;
        char[] passphrase = "test123".toCharArray();

        ctx = SSLContext.getInstance("TLSv1.3");
        kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        ks = KeyStore.getInstance("PKCS12");
        ts = KeyStore.getInstance("PKCS12");

        //load keystore
        ks.load(new FileInputStream("./client/client.p12"), passphrase);
        //load truststore
        ts.load(new FileInputStream("./ca_certs/truststore.p12"), passphrase);

        kmf.init(ks, passphrase);
        tmf.init(ts);
        
        ctx.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);

        factory = ctx.getSocketFactory();

        // connect to host
        SSLSocket socket = (SSLSocket)factory.createSocket(host, port);
        socket.startHandshake();

        PrintWriter out = new PrintWriter(
                                  new BufferedWriter(
                                  new OutputStreamWriter(
                                  socket.getOutputStream())));
                                
        out.println("GET / HTTP/1.0");
        out.println();
        out.flush();

        /*
         * Make sure there were no surprises
         */
         if (out.checkError()) {
           System.out.println( "Client: java.io.PrintWriter error");
         }

         /* read response */
         BufferedReader in = new BufferedReader(
                                    new InputStreamReader(
                                    socket.getInputStream()));

         String inputLine;

         while ((inputLine = in.readLine()) != null) {
             System.out.println(inputLine);
         }

         in.close();
         out.close();
         socket.close();
        
    }
    
}
~~~
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
RSA based private keys always fails when used with PKCS11+nss. EC based keys works though.

FREQUENCY : always