This happens if your -testjdk is a debug build but your -compilejdk is a product build. E.g.,

cd open/test/hotspot/jtreg/runtime/cds/appcds
env JAVA_HOME=/home/iklam/jdk/official/jdk15 /home/iklam/jdk/tools/jtreg/5.1-b01/bin/jtreg \
    -testjdk:/home/iklam/jdk/bld/fastdebug/images/jdk \
    -compilejdk:/home/iklam/jdk/bld/product/images/jdk \
    -agentvm \
    -w /home/iklam/tmp/jtreg/work -r /home/iklam/tmp/jtreg/report/ \
    \
    -vmoption:-XX:+VerifyStringTableAtExit HelloTest.java

jtreg would invoke javac wth "-J-XX:+VerifyStringTableAtExit", which would fail with something like:

$ javac -J-XX:+VerifyStringTableAtExit HelloTest.java
Error: VM option 'VerifyStringTableAtExit' is diagnostic and must be enabled via -XX:+UnlockDiagnosticVMOptions.
Error: The unlock option must precede 'VerifyStringTableAtExit'.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

However, it seems like jtreg doesn't check for this condition, and keeps waiting for HelloTest.class to be compiled, so it will get stuck for a long time. Eventually the test would timeout and the jtr file would indicate that the "rerun" command is javac, but doesn't say why javac has failed. The only clue is in ./work/jtData/agent.trace which records the above javac error message.

I think it's better for jtreg to check for the javac failure, terminate the test immediately, and report the error more clearly to the user.