Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8271199

Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key

XMLWordPrintable

        There is a use case with a custom PKCS11 provider (IAIK) which fails during the mutual TLS handshake.
        TLS handshake fails to create a signature for the client certificate using sensitive PKCS11 RSA key.

        EXPECTED BEHAVIOUR:
        JDK selects custom RSASSA-PSS Signature implementation from the IAIK provider, signs client certificate, and completes TLS handshake

        ACTUAL BEHAVIOUR:
        JDK selects RSASSA-PSS Signature implementation from the SunRSASign provider and fails signing client certificate with the following stack trace:
          java.lang.UnsupportedOperationException: Prime P value is sensitive.
          at iaik.pkcs.pkcs11.provider.keys.IAIKPKCS11RsaPrivateKey.getPrimeP(Unknown Source)
          at java.base/sun.security.rsa.RSACore.crtCrypt(RSACore.java:176)
          at java.base/sun.security.rsa.RSACore.rsa(RSACore.java:130)
          at java.base/sun.security.rsa.RSAPSSSignature.engineSign(RSAPSSSignature.java:385)
          at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1404)
          at java.base/java.security.Signature.sign(Signature.java:712)
          at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:612)
          at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:764)
          at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
          at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
          at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
          at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
          at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
          at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
          at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
          at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)

        The implementation uses a private PKCS11 key from the IAIK provider :
        http://javadoc.iaik.tugraz.at/pkcs11_provider/current/iaik/pkcs/pkcs11/provider/keys/IAIKPKCS11RsaPrivateKey.html

        This issue happens because of SignatureScheme.getSigner() selects RSAPSSSignature signer from the SunRSASign provider and RSAPSSSignature.engineInitSign() successfully initialises signature object.

        Stack trace to RSAPSSSignature.engineInitSign():
        java.base/sun.security.rsa.RSAPSSSignature.engineInitSign(RSAPSSSignature.java:148)
           at java.base/java.security.SignatureSpi.engineInitSign(SignatureSpi.java:167)
           at java.base/java.security.Signature$Delegate.tryOperation(Signature.java:1318)
           at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1270)
           at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1382)
           at java.base/java.security.Signature.initSign(Signature.java:683)
           at java.base/java.security.Signature$1.initSign(Signature.java:147)
           at java.base/sun.security.util.SignatureUtil.initSignWithParam(SignatureUtil.java:194)
           at java.base/sun.security.ssl.SignatureScheme.getSigner(SignatureScheme.java:595)
           at java.base/sun.security.ssl.SignatureScheme.getSignerOfPreferableAlgorithm(SignatureScheme.java:542)
           at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:593)
           at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:764)

        As result, JDK does not try other signature providers. Later, RSAPSSSignature fails to sign the client certificate with the PKCS11 sensitive key.

        This issue is possibly related to JDK-8226374 and JDK-8222937

              abakhtin Alexey Bakhtin
              abakhtin Alexey Bakhtin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: