JDK-8305083 removed the use of finalizers in nsk/jdi tests and replaced with using Cleaners instead:

https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/Cleaner.html

However, the new implementation has a bug that causes the Cleaner to never be executed and the associated Cleaner thread is never freed. When running the nsk/jdi tests, this results in 100's of Cleaner threads accumulating in the AgentServer process. You see them when a test times out and the jtreg failure handler dumps all the threads in the AgentServer process. Since the tests are run in driver mode, the AgentServer process is re-used and accumulates more Cleaner threads each time a test is run.

Note up until a little over a month ago, you would only end up with at most a few lingering Cleaner threads. This is because JDK-8327704 changed all the tests to use driver mode instead of othervm, so the tests are no longer getting a clean VM to run in.

The problematic code is in test/hotspot/jtreg/vmTestbase/nsk/share/Finalizable.java:

    default public void registerCleanup() {
       Finalizer finalizer = new Finalizer(this);
       finalizer.activate();

       Cleaner.create().register(this, () -> cleanup());
    }

The use of a lambda as the cleanup action results in the lambda capturing a reference to the Cleaner object, meaning the Cleaner object will always be reachable, and therefore will never trigger a cleanup action. This issue is described in the above javadoc link for Cleaner.

I ran all the nsk/jdi tests while forcing one to timeout (to produce the thread dump), and verified that there are no longer any Cleaner threads if the about registerCleanup() code is turned into a no-op.

The solution is to use a static method instead of a lamda. I haven't looked at this code closely enough to determine if that is easily accomplished, I'm still trying to wrap my head around how this is suppose to work and also looking at the example in the javadoc. [UPDATE: there is not way to do this with a static method as you need a unique cleaner method for each object you want to clean. See comments from Stuart and David below]

Another option would be to just remove this finalization support and not longer rely on it. The tests seem to run fine with it disabled. However, I think there could be some ugly side affects we just aren't noticing, but could be problematic at times. The uses of this support are varied, and it's unclear how important each is. It looks like it is used for shutting down socket listeners, closing a socket, flushing log files, and terminating the debuggee process.