A DESCRIPTION OF THE PROBLEM :
This was discovered in a large java project which recently switched to the PKCS11 security provider (sun.security.pkcs11) and started crashing due to native memory leaks. After running with jemalloc we discovered the leaking memory was malloced in Java_sun_security_pkcs11_wrapper_PKCS11_C_1SignInit. I initially thought it might be the same issue as JDK-8306294 but we were still having the issue on 11.0.20.

After taking many thread dumps from the application, we found that there were only 2 places we could see calls to C_SignInit: P11PSSSignature.initialize() and P11TlsPrfGenerator.engineGenerateKey(). Of the two, P11TlsPrfGenerator.engineGenerateKey() seems like the place most likely to be leaking since it does not seem to ever call freeHandle() on the CK_MECHANISM instances it creates.

In P11PSSSignature.java mechanism is a CK_MECHANISM instance which is cleaned up in reset() via mechanism.freeHandle(). I haven't gone through all of the code to be 100% certain that all the handling there is correct, but it does seem to have some cleanup handling to free the handles allocated by for the CK_MECHANISM.

In P11TlsPrfGenerator though, the mechanism instance variable is just the long value, and the CK_MECHANISM instance is created and passed to C_SignInit directly, and there is no subsequent calls to freeHandle that I can see for it.

From a read of CK_MECHANISM, it seems that the calling method is required to call freeHandle() to free the native memory if the params method is passed to the CK_MECHANISM constructor, which it is in P11TlsPrfGenerator.engineGenerateKey().

So this seems like the cause of our leak, it would be nice to get a fix for this in a future JDK11 release. For now, we have rolled back to the sun.security.provider.Sun provider as it doesn't seem to leak any native memory that we can see.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Perform TLS 1.2 handshakes against a Netty XNIO server using sun.security.pkcs11.SunPKCS11 security provider while profiling native memory with jemalloc (I don't know if the cipher matters, but we are mostly using CHACHA20-POLY1305)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No native memory leak should be observed in the jeprof output
ACTUAL -
We see a growing accumulation of native memory allocations in Java_sun_security_pkcs11_wrapper_PKCS11_C_1SignInit(). My best guess as to the cause is in the description.

CUSTOMER SUBMITTED WORKAROUND :
Use a different security provider (like sun.security.provider.Sun)

FREQUENCY : always