FULL PRODUCT VERSION :
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)


FULL OS VERSION :
Linux 4.6.2-1-ARCH #1 SMP PREEMPT Wed Jun 8 08:40:59 CEST 2016 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
This is about Java agent, or Java Instrumentation. I created a ClassFileTransformer which does basically nothing. It looks like

public class LoggingTransformer implements ClassFileTransformer {
  static int transformCount = 0;

  public LoggingTransformer() {
  }

  public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined,
    ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {

    transformCount++;
    if (transformCount % 100 == 0) System.out.println("transformCount:" + transformCount);

   return null;
  }
}

Then I created this transformer and repeatedly invoke inst.retransformClasses(), as depicted below:

public class LoggingAgent {
   public static void premain(String agentArgs, Instrumentation inst) throws Exception {
       LoggingTransformer t = new LoggingTransformer();
       inst.addTransformer(t, true);
       Class demoClass = Class.forName("Tester");

       while (true) {
           inst.retransformClasses(demoClass);
      }
  }
}
The command I used to run the application with my Java agent is :
java -XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:MetaspaceSize=12m -XX:MaxMetaspaceSize=12m -javaagent:../lib/loggingagent.jar MyDemoMain

Note I set metaspace to 12m just to make it fail in a shorter time.

The result is that, after about 5600 times of invocation of inst.retransformClasses() which triggers LoggingTransformer.transform(), the JVM crashes with below error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.OutOfMemoryError
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at LoggingAgent.premain(LoggingAgent.java:14)
... 6 more

Since I enabled classloading and class unloading logs, I do see messages 'Loaded Tester from __VM_RedefineClasses__]' appearing in the log output, but I can't find any unloading messages related.


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Write a couple of Agent classes as mentioned above;
2. Write a manifest file as below:
Premain-Class: LoggingAgent
Agent-Class: LoggingAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true
3. Make sure you call inst.retransformClasses() releatedly from within the agent's premain method
4. package your agent classes with manifest file and produces a jar file
5. Write a simple class MyDemoMain which has a main() method which just sleeps forever.
6. Once the number of invocation of inst.retransformClasses() is high enough, you will see JVM complains.

EXPECTED VERSUS ACTUAL BEHAVIOR :
1. JVM should garbage collect the old class definitions when a class is transformed;
2. Such that JVM should not run out of memory if the same class is being redfined again and again;
3. We shall see class unloading logs from enabling -XX:+TraceClassUnloading option when running the program.
REPRODUCIBILITY :
This bug can be reproduced always.