I've seen at least at least a few tests in vmTestbase/nsk/jdi that pass even if the debuggee fails to launch. I intentionally passed an illegal argument to the debuggee, and log for stepreq002 contained:

--> debugger: test cancelled
TEST FAILED
#>
#> SUMMARY: Following errors occured
#> during test execution:
#>
# ERROR: ##> debugger: ERROR: Exception : nsk.share.Failure: Caught exception while starting debugee VM:
# ERROR: com.sun.jdi.connect.VMStartException: VM initialization failed for: /scratch/cplummer/ws/jdk/jdk.clean/build/linux-x64-debug/images/jdk/bin/java -XX:MaxRAMPercentage=1.78571 -Dtest.boot.jdk=/scratch/cplummer/jib/install/jdk/19/36/bundles/linux-x64/jdk-19_linux-x64_bin.tar.gz/jdk-19 -Djava.io.tmpdir=/scratch/cplummer/ws/jdk/jdk.clean/build/linux-x64-debug/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jdi_EventRequestManager_stepRequests_stepreq002/tmp -Dmain.wrapper=Virtual --enabl-preview -Djdk.defaultScheduler.parallelism=10 -Xdebug -Xrunjdwp:transport=dt_socket,address=localhost:38905,suspend=y,includevirtualthreads=y nsk.share.MainWrapper Virtual nsk.jdi.EventRequestManager.stepRequests.stepreq002a -vbs -verbose -arch=linux-x64 -waittime=5 -debugee.vmkind=java -transport.address=dynamic -debugee.vmkeys="-XX:MaxRAMPercentage=1.78571 -Dtest.boot.jdk=/scratch/cplummer/jib/install/jdk/19/36/bundles/linux-x64/jdk-19_linux-x64_bin.tar.gz/jdk-19 -Djava.io.tmpdir=/scratch/cplummer/ws/jdk/jdk.clean/build/linux-x64-debug/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jdi_EventRequestManager_stepRequests_stepreq002/tmp -Dmain.wrapper=Virtual " -pipe.port=41351
# ERROR: stdout:
# ERROR: stderr: Unrecognized option: --enabl-preview
# ERROR: Error: Could not create the Java Virtual Machine.
# ERROR: Error: A fatal exception has occurred. Program will exit.

However, the test passed. This is the run() method for stepreq002

    public static int run (String argv[], PrintStream out) {
        int exitCode = new stepreq002().runThis(argv, out);
        if (exitCode != PASSED) {
            System.out.println("TEST FAILED");
        }
        return testExitCode;
     }

In this failing test run the code returning to run() is:

        } catch ( Exception e ) {
            log3("ERROR: Exception : " + e);
            log2(" test cancelled");
            return FAILED;
        }

The problem is testExitCode never got set. I'm not so sure it should be in this case. I think we should just be returning exitCode from run(). Usually we return testExitCode to run(), so exitCode and testExitCode are the same. The problem is when we directly return FAILED instead of setting testExitCode to FAILED and then returning testExitCode.

This issue seems pervasive in the jdi tests. It could possibly be hiding a lot of test failures. This is especially true if you are experimenting with arguments passed to the debuggee like I was. I knew I was passing an invalid arg, yet the test was still passing.