I was looking at ProblematicFrameTest.java and noticed that it should be generating an hs_err file:

            Utils.getUnsafe().getInt(0);

This is intended to cause a crash. However, there is no hs_err file and no indication in the .jtr file that anything went wrong. If you add the following to the test:

         OutputAnalyzer output = new OutputAnalyzer(pb.start());
+ System.out.println("[STDERR]\n" + output.getStderr());

This is the output:
 
[STDERR]
Exception in thread "main" java.lang.IllegalAccessError: class jdk.test.lib.Utils (in unnamed module @0x157f54e) cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @0x157f54e
at jdk.test.lib.Utils.getUnsafe(Utils.java:360)
at ProblematicFrameTest$Crasher.main(ProblematicFrameTest.java:45)

So it appears this problem was introduced by jigsaw and the Unsafe access limitations. Besides resolving this issue, the test should also be checking to make sure that no exception was thrown by the subprocess so problems like this are detected by the test.