Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8164921

Memory leaked when instrumentation.retransformClasses() is called repeatedly

XMLWordPrintable

    • b143
    • x86_64
    • linux

      FULL PRODUCT VERSION :
      java version "1.8.0_102"
      Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
      Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)


      FULL OS VERSION :
      Linux 4.6.2-1-ARCH #1 SMP PREEMPT Wed Jun 8 08:40:59 CEST 2016 x86_64 GNU/Linux

      A DESCRIPTION OF THE PROBLEM :
      This is about Java agent, or Java Instrumentation. I created a ClassFileTransformer which does basically nothing. It looks like

      public class LoggingTransformer implements ClassFileTransformer {
        static int transformCount = 0;

        public LoggingTransformer() {
        }

        public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined,
          ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {

          transformCount++;
          if (transformCount % 100 == 0) System.out.println("transformCount:" + transformCount);

         return null;
        }
      }

      Then I created this transformer and repeatedly invoke inst.retransformClasses(), as depicted below:

      public class LoggingAgent {
         public static void premain(String agentArgs, Instrumentation inst) throws Exception {
             LoggingTransformer t = new LoggingTransformer();
             inst.addTransformer(t, true);
             Class demoClass = Class.forName("Tester");

             while (true) {
                 inst.retransformClasses(demoClass);
            }
        }
      }
      The command I used to run the application with my Java agent is :
      java -XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:MetaspaceSize=12m -XX:MaxMetaspaceSize=12m -javaagent:../lib/loggingagent.jar MyDemoMain

      Note I set metaspace to 12m just to make it fail in a shorter time.

      The result is that, after about 5600 times of invocation of inst.retransformClasses() which triggers LoggingTransformer.transform(), the JVM crashes with below error:

      Exception in thread "main" java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
      at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
      Caused by: java.lang.OutOfMemoryError
      at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
      at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
      at LoggingAgent.premain(LoggingAgent.java:14)
      ... 6 more

      Since I enabled classloading and class unloading logs, I do see messages 'Loaded Tester from __VM_RedefineClasses__]' appearing in the log output, but I can't find any unloading messages related.


      THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

      THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      1. Write a couple of Agent classes as mentioned above;
      2. Write a manifest file as below:
      Premain-Class: LoggingAgent
      Agent-Class: LoggingAgent
      Can-Redefine-Classes: true
      Can-Retransform-Classes: true
      3. Make sure you call inst.retransformClasses() releatedly from within the agent's premain method
      4. package your agent classes with manifest file and produces a jar file
      5. Write a simple class MyDemoMain which has a main() method which just sleeps forever.
      6. Once the number of invocation of inst.retransformClasses() is high enough, you will see JVM complains.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      1. JVM should garbage collect the old class definitions when a class is transformed;
      2. Such that JVM should not run out of memory if the same class is being redfined again and again;
      3. We shall see class unloading logs from enabling -XX:+TraceClassUnloading option when running the program.
      REPRODUCIBILITY :
      This bug can be reproduced always.

            coleenp Coleen Phillimore
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: