Summary

A memory growth issue is observed when using the SunPKCS11 provider with the NSS PKCS#11 library. This is due to the clean up of native handles which is managed by java GC and which may not kick in before the native heap is exhausted. To address the memory growth issue, this fix extracts native cryptographic keys into Java byte arrays when not in use and re-creates them when needed.

Problem

The SunPKCS11 provider is functionally a java wrapper for the native PKCS#11 library. The cryptographic keys (sun.security.pkcs11.P11Key objects) of the SunPKCS11 provider encapsulate native key handles created by the native PKCS#11 library. When the keys are no longer needed, their memory is freed through the phantom reference mechanism which is managed by java GC. However, the native heap may be exhausted when java GC doesn't kick in or is frequent enough.

This has been a long standing issue, manifested in 32 bit processors.

Solution

The proposed solution is to extract native cryptographic keys when they are no longer in use and hold them in a Java byte array in each P11Key object. Key extraction is achieved by retrieving all PKCS#11 attributes (I.e.: CKA_CLASS, CKA_VALUE, CKA_ID, etc.) and saving the result into a Java byte array. With this approach, the native key handle is only kept for a short duration and for most of the time it will not take up space in the native heap.

The feature has been implemented to minimize a performance penalty through different levels of optimizations. As an example, P11Key keys that have been just created keep their native key handle for its first use. JNI methods were implemented to query all native key attributes at once, and minimize context switching between Java and C code.

Since this issue was reported when using the NSS PKCS#11 library, this fix applies key extraction only when the NSS PKCS#11 library is used. However, if the same issue is observed with another PKCS#11 library, it is possible to update this fix to cover other PKCS#11 libraries.

Review discussion: http://mail.openjdk.java.net/pipermail/security-dev/2017-October/016400.html

Specification

As part of this change, a property to completely disable this change has been introduced: sun.security.pkcs11.disableKeyExtraction (defaults to false). See the "Compatibility Risk Description" above.