SunPKCS11 tries to extract keys from the native to the Java heap when the underlying token is NSS in order to avoid memory leak issues. Extracted keys are re-built in the token when needed. The PKCS #11 API C_UnwrapKey is used as part of this process. However, when the key class is CKO_SECRET_KEY and the key's length is greater than 256 bytes (defined as MAX_KEY_LEN in pkcs11i.h), NSS' NSC_UnwrapKey returns a CKR_TEMPLATE_INCONSISTENT error [1].

To avoid the described problem we will not extract large CKO_SECRET_KEY keys out of the NSS Software Token. The destruction of these keys will be handled by the SessionKeyRef mechanism and the GC.

This bug is hard to reproduce under normal circumstances. It requires a FIPS-configured NSS Software Token so C_UnwrapKey is called from Java_sun_security_pkcs11_wrapper_PKCS11_createNativeKey. In addition, creating a large secret key is not common because there are checks depending on the key type. Methods such as C_GenerateKey have checks as well, and refuse to create secret keys larger than 256 bytes. I attached to this ticket the TestLargeKeys.java reproducer that will generate a large key and trigger the check in NSS. Please notice that this reproducer does not reflect the case that we are fixing here: we will only fix the C_UnwrapKey call from Java_sun_security_pkcs11_wrapper_PKCS11_createNativeKey by making the large native key non-extractable.

--
[1] - https://github.com/nss-dev/nss/blob/NSS_3_90_RTM/lib/softoken/pkcs11c.c#L6492