Analysis of the random ClassNotFoundException/NoClassDefFoundError when running jtreg tests in parallel. Overview: ======== These errors are caused by a long standing bug in jtreg compilation: When tests are executed in a certain order, jtreg does not correctly generate all class files needed from a test library. This bug has caused many issues in the past where HotSpot nightly tests would fail with mysterious, and unreproducible, ClassNotFoundException/NoClassDefFoundError. So far the associated bugs have been dealt with by rearranging the test source code, etc, without fully understadning the root cause. The following provides a reliably reproduction of the bug to explain the root cause. A patch is also provided to fix this bug. Trace: ===== Run the following test using [0] Get reproduce_7901986.sh from the attachment of https://bugs.openjdk.java.net/browse/CODETOOLS-7901986 [1] Get the JDK9 tree from http://hg.openjdk.java.net/jdk9/dev; update all repos to label "jdk-9+169" [2] Run the tests with the official JDK 9 ea 169 build $ cd hotspot/test $ bash ~/reproduce.sh /java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ + rm -rf ./reproduce_7901986_tmp ======================================================================[step 1] + /java/re/jtreg/4.2/promoted/fcs/b07/binaries/jtreg/bin/jtreg -J-Djavatest.maxOutputSize=1000000 -conc:1 -testjdk:/java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ -compilejdk:/java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ -verbose:2 -timeout:4.0 -noreport -w ./reproduce_7901986_tmp/work -r ./reproduce_7901986_tmp/report runtime/modules/AccessCheckRead.java runtime/RedefineTests/ModifyAnonymous.java Directory "./reproduce_7901986_tmp/work" not found: creating runner starting test: runtime/modules/AccessCheckRead.java runner finished test: runtime/modules/AccessCheckRead.java Passed. Execution successful runner starting test: runtime/RedefineTests/ModifyAnonymous.java runner finished test: runtime/RedefineTests/ModifyAnonymous.java Passed. Execution successful Test results: passed: 2 Results written to /jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work -------------------------------------- ./reproduce_7901986_tmp/work/classes/runtime/RedefineTests/jdk/test/lib/InMemoryJavaCompiler.class ======================================================================[step 2] + /java/re/jtreg/4.2/promoted/fcs/b07/binaries/jtreg/bin/jtreg -J-Djavatest.maxOutputSize=1000000 -conc:1 -testjdk:/java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ -compilejdk:/java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ -verbose:2 -timeout:4.0 -noreport -w ./reproduce_7901986_tmp/work -r ./reproduce_7901986_tmp/report runtime/RedefineTests/RedefineRunningMethodsWithResolutionErrors.java runner starting test: runtime/RedefineTests/RedefineRunningMethodsWithResolutionErrors.java runner finished test: runtime/RedefineTests/RedefineRunningMethodsWithResolutionErrors.java Passed. Execution successful Test results: passed: 1 Results written to /jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work -------------------------------------- ./reproduce_7901986_tmp/work/scratch/RedefineClassHelper.class ./reproduce_7901986_tmp/work/classes/test/lib/RedefineClassHelper.class ./reproduce_7901986_tmp/work/classes/runtime/RedefineTests/jdk/test/lib/InMemoryJavaCompiler.class ======================================================================[step 3] + /java/re/jtreg/4.2/promoted/fcs/b07/binaries/jtreg/bin/jtreg -J-Djavatest.maxOutputSize=1000000 -conc:1 -testjdk:/java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ -compilejdk:/java/re/jdk/9/promoted/ea/169/binaries/linux-x64/ -verbose:2 -timeout:4.0 -noreport -w ./reproduce_7901986_tmp/work -r ./reproduce_7901986_tmp/report testlibrary_tests/RedefineClassTest.java runner starting test: testlibrary_tests/RedefineClassTest.java runner finished test: testlibrary_tests/RedefineClassTest.java Failed. Execution failed: `main' threw exception: java.lang.NoClassDefFoundError: jdk/test/lib/InMemoryJavaCompiler Test results: failed: 1 Results written to /jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work Error: Some tests failed or other problems occurred. -------------------------------------- ./reproduce_7901986_tmp/work/scratch/RedefineClassHelper.class ./reproduce_7901986_tmp/work/classes/test/lib/RedefineClassHelper.class ./reproduce_7901986_tmp/work/classes/runtime/RedefineTests/jdk/test/lib/InMemoryJavaCompiler.class $ Explanation: =========== JTREG runs the tests alphabetically. The test script splits into 3 steps just to illustrate the order in which the .class files are compiled: runtime/modules/AccessCheckRead.java runtime/RedefineTests/ModifyAnonymous.java runtime/RedefineTests/RedefineRunningMethodsWithResolutionErrors.java testlibrary_tests/RedefineClassTest.java [step 1] During this step, only InMemoryJavaCompiler.class is generated, due to the execution of ModifyAnonymous.java. Because ModifyAnonymous.java does not explicitly build InMemoryJavaCompiler (which could be done with "@build jdk/test/lib/InMemoryJavaCompiler), InMemoryJavaCompiler.class is compiled as a side effect of reproduce_7901986_tmp/work/runtime/RedefineTests/ModifyAnonymous.d/compile.0.jta: -d /jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/runtime/RedefineTests -sourcepath /jdk/jdk9dev/hotspot/test/runtime/RedefineTests:/jdk/jdk9dev/test/lib Due to the -d switch, InMemoryJavaCompiler.class is written into work/classes/runtime/RedefineTests/ [step 2] During this step, RedefineClassHelper.class is compiled as the result of the line "@run main RedefineClassHelper" in RedefineRunningMethodsWithResolutionErrors.java. See -d /jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/test/lib -sourcepath /jdk/jdk9dev/test/lib -classpath /jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/runtime/RedefineTests:/jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/test/lib /jdk/jdk9dev/test/lib/RedefineClassHelper.java Due to the -d switch, RedefineClassHelper.class is written into work/classes/test/lib Note that InMemoryJavaCompiler is not recompiled, even though RedefineClassHelper.java explicitly references it. That's because work/classes/runtime/RedefineTests/ is in the classpath, where javac discovers InMemoryJavaCompiler.class and finds no need to regenerate it. As a result, the .class files of test/lib are split into two locations: ./reproduce_7901986_tmp/work/classes/test/lib/RedefineClassHelper.class ./reproduce_7901986_tmp/work/classes/runtime/RedefineTests/jdk/test/lib/InMemoryJavaCompiler.class Nevertheless, RedefineRunningMethodsWithResolutionErrors.java succeeded because it's executed with both locations on its classpath. See ./reproduce_7901986_tmp/work/runtime/RedefineTests/RedefineRunningMethodsWithResolutionErrors.jtr: CLASSPATH=/jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/runtime/RedefineTests:/jdk/jdk9dev/hotspot/test/runtime/RedefineTests:/jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/test/lib:.... [step 3] When running RedefineClassTest.java, jtreg notices that RedefineClassHelper.class is needed, but there's no need to recompile it because the class file already exists and is accessible to this test: work/classes/test/lib/RedefineClassHelper.class However, when executing "@run main/othervm -javaagent:redefineagent.jar RedefineClassTest", the specified classpath does not contain InMemoryJavaCompiler.class. See ./reproduce_7901986_tmp/work/testlibrary_tests/RedefineClassTest.jtr CLASSPATH=/jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/testlibrary_tests:/jdk/jdk9dev/hotspot/test/testlibrary_tests:/jdk/jdk9dev/hotspot/test/reproduce_7901986_tmp/work/classes/test/lib:/jdk/jdk9dev/test/lib:/net/scanas416.us.oracle.com/export/java_re2/misc/promoted/jtreg/4.2/fcs/b07/binaries/jtreg/lib/javatest.jar:/net/scanas416.us.oracle.com/export/java_re2/misc/promoted/jtreg/4.2/fcs/b07/binaries/jtreg/lib/jtreg.jar \\ ... Fix: === The bug is caused by the "split" of test/lib during [step 2], which improperly use -classpath to make classes that are local to a test case visible to test/lib, which should be shared by all test cases. The fix is to avoid adding locations.absTestClsDir() to -classpath when jtreg is building a library. The patch is here (on top of label "jtreg4.2-b07") http://cr.openjdk.java.net/~iklam/jtreg/7901986_split_library/