There is a potential infinite loop in G1's concurrent mark thread's service routine. The main concurrent marking loop continues iterating if cm->restart_for_overflow() is true. That test is set to false in the cm->mark_from_roots() call at the head of the loop. But that call is conditional on !cm->has_aborted(). If both cm->has_aborted() and cm->restart_for_overflow() are true, the loop will never terminate.
restart_for_overflow is only set true by checkpointRootsFinal(), which is called during the remark pause. That assignment is protected by an earlier has_aborted test; if has_aborted, the assignment of restart_for_overflow to true won't be reached. So a full GC with associated abort request before the remark pause won't trigger this problem.
However, there is a narrow window of concurrent execution between the end of the remark pause and the next iteration's conditional call to mark_from_roots. If a full GC occurred during that window, the true value for has_aborted would prevent restart_from_overflow from being cleared by the next iteration, and we loop forever.
restart_for_overflow is only set true by checkpointRootsFinal(), which is called during the remark pause. That assignment is protected by an earlier has_aborted test; if has_aborted, the assignment of restart_for_overflow to true won't be reached. So a full GC with associated abort request before the remark pause won't trigger this problem.
However, there is a narrow window of concurrent execution between the end of the remark pause and the next iteration's conditional call to mark_from_roots. If a full GC occurred during that window, the true value for has_aborted would prevent restart_from_overflow from being cleared by the next iteration, and we loop forever.