The test is failing with:

 > java.lang.RuntimeException: Wrong number of class unload events: expected 10 got 0

Note this is different than JDK-8291456, which fails with:

 > java.lang.RuntimeException: Wrong number of class unload events: expected 10 got 4

com/sun/jdi/ClassUnloadEventTest.java is a newer tests that is testing the generating of JDI ClassUnloadEvents. There were some rare cases of it failing to properly generate the events. JDK-8291456 was filed, and it was determined that the failure was due to the debuggee exiting before jvmti could flush out all the queued up events. This was worked around in the test with a short delay before the debuggee exits. However, the test is still failing, although it is much rarer now. I did try extending the delay before the debuggee exits to 60 seconds, and it still fails, so clearly it's not the same issue as JDK-8291456.

Some unique attributes of this failure when compared to JDK-8291456 include:

1. Always on windows-x64-debug. Not on any other platforms (unlike JDK-8291456).
2. Only with -Xcomp plus some other compiler related options that may or may not also have an impact.
3. The test complains that it got 0 of 10 expected events, whereas for JDK-8291456 is was always 4 of 10
4. Added delay in the debuggee does not fix the problem.

I made some changes to the test to print the output of the debuggee and also turned on -Xlog:class+unload=info, and indeed the test classes are not being unloaded, so this does not appear to be a JVMTI or debug agent issue.

The test uses ClassUnloadCommon.triggerUnloading(), just like many other tests do. I assume this is reliable, but maybe it is not for some reason with this particular test case.

The other possible cause I thought of that is in rare cases the debuggee is compiled in a way that results in a reference to the ClassLoader or a Class being retained.