This problem was discovered during the work for SoftMaxHeapSize (JDK-8236073) (and our internal ProposedHeapSize). In G1CollectedHeap::expand_and_allocate(), it does:

if (expand(expand_bytes, _workers)) {
    ...
    return attempt_allocation_at_safepoint(word_size, false);
}

We found that when expand() returns true, attempt_allocation_at_safepoint() could immediately return nullptr. This is counter-intuitive, because allocation immediately after a heap expansion should normally succeed. One possible cause is another thread immediately uses up the newly created region, because expand() and attempt_allocation_at_safepoint() are not atomic.

This mostly affects use cases for SoftMaxHeapSize, where setting a small SoftMaxHeapSize could cause OutOfMemoryError. To work around it, we internally revived the removed attempt_allocation_force() method (deleted by JDK-8331562), and use it when this problem happens.