As part of the TLS 1.3 handshake, keys are derived using the HKDF class [1]. In order to derive keys, an HMAC algorithm is used [2]. The underlying MAC service (obtained here [3]) can be the one provided by the SunPKCS11 security provider [4]. A MAC service needs to be initialized with a key. In the TLS 1.3 SunJSSE implementation, the MAC key is a salt and comes from a plain array of zeros [5] [6] [7].

To use a plain-array-of-zeros key inside the SunPKCS11, the key needs to be translated into a P11 Key [8]. However, OpenJDK does not set the attribute CKA_SIGN = TRUE (CK_ATTRIBUTE.SIGN_TRUE) on the newly created key. As part of the P11 MAC initialization sequence, a PKCS#11 C_SignInit API is called [9].

When the SunPKCS11 backend provider is the NSS Software Token, the C_SignInit call fails because there is a consistency check that the key must have the CKA_SIGN = TRUE attribute [10]. Note: in [10], 'operation' is 'CKA_SIGN' and 'key->objclass' is 'CKO_SECRET_KEY'.

Note: while I have not tried other PKCS#11 backends -and it's possible that they don't check against consistency-, looks to me that a key used for C_SignInit with a CKA_SIGN = TRUE attribute should cause no harm if we make it the default.

Failure stack-trace example:

javax.net.ssl.SSLHandshakeException: Could not generate secret
   at java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(KAKeyDerivation.java:128)
   at java.base/sun.security.ssl.KAKeyDerivation.deriveKey(KAKeyDerivation.java:63)
   at java.base/sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(ServerHello.java:1291)
   at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(ServerHello.java:1006)
   at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(ServerHello.java:894)
   at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
   at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
   at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
   at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:200)
   at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
   at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500)
   at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415)
   at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450)
   at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421)
   at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
   at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
   at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142)
   at Main.main(Main.java:55)
  Caused by: java.security.InvalidKeyException: init() failed
   at jdk.crypto.cryptoki/sun.security.pkcs11.P11Mac.engineInit(P11Mac.java:214)
   at java.base/javax.crypto.Mac.init(Mac.java:432)
   at java.base/sun.security.ssl.HKDF.extract(HKDF.java:90)
   at java.base/sun.security.ssl.HKDF.extract(HKDF.java:118)
   at java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(KAKeyDerivation.java:118)
   ... 17 more
  Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_KEY_TYPE_INCONSISTENT
   at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_SignInit(Native Method)
   at jdk.crypto.cryptoki/sun.security.pkcs11.P11Mac.initialize(P11Mac.java:183)
   at jdk.crypto.cryptoki/sun.security.pkcs11.P11Mac.engineInit(P11Mac.java:212)
   ... 21 more}

--
[1] - https://github.com/openjdk/jdk/blob/jdk-19+1/src/java.base/share/classes/sun/security/ssl/HKDF.java#L37
[2] - https://github.com/openjdk/jdk/blob/jdk-19+1/src/java.base/share/classes/sun/security/ssl/HKDF.java#L63
[3] - https://github.com/openjdk/jdk/blob/jdk-19+1/src/java.base/share/classes/sun/security/ssl/HKDF.java#L64
[4] - https://github.com/openjdk/jdk/blob/jdk-19%2B1/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Mac.java#L56
[5] - https://github.com/openjdk/jdk/blob/jdk-19+1/src/java.base/share/classes/sun/security/ssl/HKDF.java#L90
[6] - https://github.com/openjdk/jdk/blob/jdk-19+1/src/java.base/share/classes/sun/security/ssl/HKDF.java#L118
[6] - https://github.com/openjdk/jdk/blob/jdk-19%2B1/src/java.base/share/classes/sun/security/ssl/KAKeyDerivation.java#L118
[7] - https://github.com/openjdk/jdk/blob/jdk-19%2B1/src/java.base/share/classes/sun/security/ssl/KAKeyDerivation.java#L114
[8] - https://github.com/openjdk/jdk/blob/jdk-19%2B1/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Mac.java#L215
[9] - https://github.com/openjdk/jdk/blob/jdk-19%2B1/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Mac.java#L188
[10] - https://github.com/nss-dev/nss/blob/NSS_3_67_RTM/lib/softoken/pkcs11c.c#L439