The test vmTestbase/gc/gctests/WeakReferenceGC.java has some heuristics and complexity that are really not necessary, and appear to arise from a misunderstanding about how references are notified.

In a long comment it discusses the "non-deterministic" enqueuing of references, how this test is structured to work around that nondeterminism, and how it might fail anyway. And it may do multiple GCs in an attempt to overcome that nondeterminism.

But there's no nondeterminism involved at all. Instead, the issue is that there are races.

(1) The GC clears references and adds them to the internal pending list. The reference processing Java thread takes references off the pending list and adds them to their associated queue, if any. The test was using Reference.isEnqueued to check whether a reference had been enqueued, so there was a race between the reference processing thread and the test.

This race was removed by JDK-8257876, which eliminated the use of the now deprecated Reference.isEnqueued. But that was done as a fairly minimal change to just eliminate the use of that deprecated function.

(2) If a concurrent GC is in progress at certain points in the test, one test-invoked GC might not be sufficient to cause the references to be cleared. It might also depend on specifics of how System.gc() is implemented by the particular collector; there's been some variation in whether it might complete an in-progress collection or always trigger another. So instead of just one, it might take two GCs to get the references notified.

But the test is using allocation until OOME to try to get the references cleared. This may involve a lot more GCs, and a lot more time, than is actually needed.

The test should be rewritten to do the setup, trigger a couple of GCs, and verify the expected references were enqueued in a reasonably timely manner. This would make it easier to understand and require fewer resources (both time and actually used memory).