While testing finalization for other changes I encountered a problem where when pushing the heap to the limits of available space finalization would appear to stall until new finalizable objects were created and reclaimed. I tracked it down to GenCollectedHeap::do_collection's initialization of notify_ref_lock to false. GenCollectorPolicy::satisfy_failed_allocation can call do_collection twice and if the first collection finds refs but the second doesn't the call to do_collection will reset notify_ref_lock to false. notify_ref_lock initialized to false by the vm operation and I don't think anyone else should be clearing it. Actually I think the whole logic of passing around a pointer to notify_ref_lock is overly complicated. instanceRefKlass::release_and_notify_pending_list_lock should always perform a notify if the pending list is non-null. Either the ReferenceHandler is asleep waiting for work so it needs the notify or it knows there is work in which case it's not waiting so it won't even see the notify. Either way it seems more straightforward than the current logic.

I've attached a test case foom.java.

allocated 94245 finalizable objects
94080 unfinalized objects
pending = java.lang.ref.Finalizer@f9f9d8
queueLength = 78959
14439 unfinalized objects
pending = java.lang.ref.Finalizer@f9f9d8
queueLength = 0
14439 unfinalized objects
pending = java.lang.ref.Finalizer@f9f9d8
queueLength = 0
14439 unfinalized objects
pending = java.lang.ref.Finalizer@f9f9d8
queueLength = 0
queueLength = 0
kicking the queue
14439 unfinalized objects
pending = java.lang.ref.Finalizer@a981ca
queueLength = 398
pending = null
queueLength = 0
drained

It quickly reaches a state where the are no FinalReferences on the queue but Reference.pending is non-null indicating that it has work to do. repeated System.gc, System.runFinalization and sleep calls don't allow it to make any progress. Kicking the pending list causes it to complete.