The function CMSCollector::acquire_control_and_collect() [in vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp] calls the function ReferenceProcessor::clean_up_discovered_references() before transitioning to the compacting collector. That cleanup function removes entries from the discovered set that have null referents or are no longer active, because they would violate assumptions made by the mark-sweep-compact collector.

Note that this is the only call to clean_up_discovered_references. G1, in a similar situation, instead abandons any previously collected references, using ReferenceProcessor::abandon_partial_discovery().

It's not at all clear that there's any real benefit to retaining entries in the discovered list when transitioning from the CMS collector to a full GC using another collector. Abandonment involves just walking down the list and unlinking each element; cleanup requires additional work. More importantly, this is only occurring when a concurrent collection is being abandoned in order to do a full gc, so any performance difference between these approaches is going to be completely swamped by other factors.

And the attempt to retain some entries in the discovered list introduces significant complications.

Specifically, a reference that was added to the discovered set during concurrent marking can become unreachable (except via the discovered set) between that addition and the switch to the compacting collector. The (cleaned up) discovered set is not part of the root set for the compacting collector. It almost seems like this could result in an unmarked (and so presumably reclaimable) reference ultimately making its way from the discovered set to the pending list, and that would be very bad. Fortunately, during the iteration over the final discovered reference lists by ReferenceProcessor::processor_phase3(), the processor's keep_alive function is applied to each of the retained references (using DiscoveredListIterator::update_discovered()).

Normally, references are only added to the discovered list as they are found during liveness analysis, so all references in the list will be alive already, making update_discovered() unnecessary. It is only because of this attempt to retain some discovered references when transitioning from CMS to a full GC that update_discovered() will ever be more than an expensive nop.

The solution to all this is to replace the call to clean_up_discovered_references with:

  ref_processor()->disable_discovery();
  ref_processor()->abandon_partial_discovery();
  ref_processor()->verify_no_references_recorded();

and remove clean_up_discovered_references and associated support, including update_discovered.

In addition, the function ReferenceProcessor::enable_discovery() has two "verification" arguments, which are always true except in the one case where clean_up_discovered_references is being used. Those arguments should be eliminated and the associated checks made unconditional.