In ReferenceQueue.enqueue0, if the reference being enqueued is a FinalReference then VM.addFinalRefCount(1) is performed. Similarly, in ReferenceQueue.poll0 there is a check for the found reference being a FinalReference. If it is, then VM.addFinalRefCount(-1) is called.

This is the wrong place for these operations. FinalReferences only appear in the Finalizer.queue. The proper place for this is in the handling of that queue, whenever a reference is added to or removed from it.

The difficulty is that VM.addFinalRefCount is currently unsynchronized. It requires the caller to hold a lock that synchronizes the update. And the updates can come from multiple threads, due to runFinalization. To solve that the counters could be AtomicInteger instead of volatile int. That might not even have any noticeable performance impact, because of the counters being volatile.