FULL PRODUCT VERSION :
java version "1.6.0_38"
Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)

java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
not relevant

EXTRA RELEVANT SYSTEM CONFIGURATION :
not relevant

A DESCRIPTION OF THE PROBLEM :
Java decompiler shows following static Cache definitions in sun.security.provider.X509Factory (same initializer at JRE of Java 5 through 8), which is caching parsed copies of X509Certificate and X509CRL objects.


private static final Cache<Object, X509CertImpl> certCache =
    Cache.newSoftMemoryCache(750);
private static final Cache<Object, X509CRLImpl> crlCache =
    Cache.newSoftMemoryCache(750);


The problem here is that Certificates have sizes of few kB, and CRLs have sizes close to 1 MB at our customer's case. Those certificates are re-issued every two hours, and the crlCache just keeps growing week after week.

In this case all those crlCache entries except two are obsolete data, and the code handling their loading has already thrown away obsolete original byte[] and parsed X509CRL objects.

There is no public documented way to manage this cache.
Indeed there is no document of cache's existence in related public APIs.

Also there is internal sanity limit of 4 MB for CRL sizes. That may not be enough in all cases.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Construct a tool code generating a CRL DER form of around 1 MB in size. Important thing is to have the serial numbers distinct for every one -- for example System.currentTimeMillis() should do it.

Feed the result to java.security.cert.CertificateFactory's generateCRL().
Throw both original byte[] and parsed result away. Keep repeating.

                InputStream is = new ByteArrayInputStream(crldata);
                Object o = this.cf.generateCRL(is);
                is.close();

Observe how the cache grows, and the JVM heap size grows with obsolete data.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Smaller cache.
ACTUAL -
System slowing down due to full cycle GC running more frequently as free heap size becomes insufficient, and ultimately JVM/application crashing as the free heap runs out.


REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
So far we are using undocumented internal API to purge this crlCache every hour, but it really calls for something nicer, like being able to configure the cache size.

Another approach is to set the heap size limit to somewhere around 10 GB, while 2 GB should be enough for our application.

..
import sun.security.provider.X509Factory;
...

        try {

            // Purge JRE internal parsed CRL cache as a side effect
            // of calling the engineGenerateCRL() with null parameter.
            // This is documented in JRE source as a way to debug it.
            // That thing is NOT documented in any JavaDoc!

            X509Factory fact = new X509Factory();
            fact.engineGenerateCRL(null);

        } catch (Throwable t) {
            // .. that does always throw an exception, catch and ignore.
        }