See https://github.com/iklam/jdk/commit/1fea40ae592d123be1a88af2570021f7308eaaf0 for a simple reproducer.

Detailed explanations:
https://github.com/iklam/jdk/blob/1fea40ae592d123be1a88af2570021f7308eaaf0/test/hotspot/jtreg/demo/README.txt

The problem can be shown in Test1.jtr:

Note that /jdk2/tmp/jtreg/work/classes/demo/Test1.d (the classes directory of Test1) is given in the -classpath for javac. This is incorrect. It makes classes in the @library depend on classes in the test case.

cd /jdk2/tmp/jtreg/work/scratch && \\
DISPLAY=:2 \\
HOME=/home/iklam \\
JTREG_COMPILEJDK=/home/iklam/jdk/bld/ken/images/jdk \\
JTREG_RETAIN=-retain:all \\
LANG=en_US.UTF-8 \\
PATH=/bin:/usr/bin:/usr/sbin \\
    /home/iklam/jdk/bld/ken/images/jdk/bin/javac \\
        -J-XX:MaxRAM=8g \\
        -J-Djava.library.path=/home/iklam/jdk/bld/ken/images/jdk/../../images/test/hotspot/jtreg/native \\
        -J-Dtest.vm.opts=-XX:MaxRAM=8g \\
        -J-Dtest.tool.vm.opts=-J-XX:MaxRAM=8g \\
        -J-Dtest.compiler.opts= \\
        -J-Dtest.java.opts= \\
        -J-Dtest.jdk=/home/iklam/jdk/bld/ken/images/jdk \\
        -J-Dcompile.jdk=/home/iklam/jdk/bld/ken/images/jdk \\
        -J-Dtest.timeout.factor=4.0 \\
        -J-Dtest.nativepath=/home/iklam/jdk/bld/ken/images/jdk/../../images/test/hotspot/jtreg/native \\
        -J-Dtest.root=/jdk2/ken/open/test/hotspot/jtreg \\
        -J-Dtest.name=demo/Test1.java \\
        -J-Dtest.file=/jdk2/ken/open/test/hotspot/jtreg/demo/Test1.java \\
        -J-Dtest.src=/jdk2/ken/open/test/hotspot/jtreg/demo \\
        -J-Dtest.src.path=/jdk2/ken/open/test/hotspot/jtreg/demo:/jdk2/ken/open/test/hotspot/jtreg/demo/lib \\
        -J-Dtest.classes=/jdk2/tmp/jtreg/work/classes/demo/Test1.d \\
        -J-Dtest.class.path=/jdk2/tmp/jtreg/work/classes/demo/Test1.d:/jdk2/tmp/jtreg/work/classes/test/hotspot/jtreg/demo/lib \\
        -J-Dtest.class.path.prefix=/jdk2/tmp/jtreg/work/classes/demo/Test1.d:/jdk2/ken/open/test/hotspot/jtreg/demo:/jdk2/tmp/jtreg/work/classes/test/hotspot/jtreg/demo/lib \\
        -d /jdk2/tmp/jtreg/work/classes/test/hotspot/jtreg/demo/lib \\
        -sourcepath /jdk2/ken/open/test/hotspot/jtreg/demo/lib \\
        -classpath /jdk2/tmp/jtreg/work/classes/demo/Test1.d:/jdk2/tmp/jtreg/work/classes/test/hotspot/jtreg/demo/lib /jdk2/ken/open/test/hotspot/jtreg/demo/lib/LibB.java