Shenandoah Generational: Mutator threads may fail to evacuate

We've observed an assertion error that results when a mutator thread tries to cancel GC due to an allocation failure for an evacuation. The following stack trace demonstrates the issue:
```
Thread 2 hit Breakpoint 2, report_vm_error (file=0x0, line=0, error_msg=0x0, detail_fmt=0x0) at /home/ubuntu/Devel/kdnilsen/gitfarm\
/borrow-from-old-gen/src/hotspot/share/utilities/debug.cpp:269
269 {
Current language: auto
The current source language is "auto; currently c++".
(rr) up

#1 0x000012b0211ce129 in ThreadStateTransition::transition_from_vm (thread=Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitf\
arm/borrow-from-old-gen/src/hotspot/share/runtime/thread.cpp...done.
0xbd97809a310, to=_thread_blocked, check_asyncs=true) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/\
runtime/interfaceSupport.inline.hpp:106
106 assert(thread->thread_state() == _thread_in_vm, "coming from wrong thread state");
(rr) p thread->thread_state()
Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/memory/allocation.cpp...done.
Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shared/threadLocalAllocBuffer.c\
pp...done.
Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shared/gcUtil.cpp...done.
Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/runtime/mutex.cpp...done.
Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/os/posix/os_posix.cpp...done.
$1 = _thread_in_Java

;; The assertion requires thread state to be _thread_in_vm

 (rr) where
 #0 report_vm_error (file=0x0, line=0, error_msg=0x0, detail_fmt=0x0) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/s\
rc/hotspot/share/utilities/debug.cpp:269
 #1 0x000012b0211ce129 in ThreadStateTransition::transition_from_vm (thread=0xbd97809a310, to=_thread_blocked, check_asyncs=true) \
at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/runtime/interfaceSupport.inline.hpp:106

 #2 0x000012b0211ce869 in ThreadBlockInVMPreprocess<void (JavaThread*)>::ThreadBlockInVMPreprocess(JavaThread*, void (&)(JavaThrea\
d*), bool) (Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shenandoah/shenando\
ahHeap.cpp...done.
 this=0x7ff1b80f0580, thread=0xbd97809a310, pr=@0x12b0211ce7ef: {void (JavaThread *)} 0x12b0211ce7ef <ThreadBlockInVMPreprocess<voi\
d (JavaThread*)>::emptyOp(JavaThread*)>, allow_suspend=false) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspo\
t/share/runtime/interfaceSupport.inline.hpp:227

 #3 0x000012b021dda7bd in ShenandoahHeap::try_cancel_gc (this=0xbd9780b4b20) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-ol\
d-gen/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp:2170

 #4 0x000012b021dda86b in ShenandoahHeap::cancel_gc (Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-ge\
n/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp...done.
 this=0xbd9780b4b20, cause=GCCause::_shenandoah_allocation_failure_evac) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen\
/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp:2185

 #5 0x000012b021dba276 in ShenandoahControlThread::handle_alloc_failure_evac (Reading in symbols for /home/ubuntu/Devel/kdnilsen/g\
itfarm/borrow-from-old-gen/src/hotspot/cpu/x86/abstractInterpreter_x86.cpp...done.
 this=0xbd9781249d0, words=2) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shenandoah/shenandoah\
ControlThread.cpp:823

 #6 0x000012b020f42b4c in ShenandoahHeap::try_evacuate_object (this=0xbd9780b4b20, p=0x60336f6c8, thread=0xbd97809a310, from_regio\
n=0x764871e92600, target_gen=YOUNG_GENERATION) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shen\
andoah/shenandoahHeap.inline.hpp:445

 #7 0x000012b020f42843 in ShenandoahHeap::evacuate_object (Reading in symbols for /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-\
old-gen/src/hotspot/share/gc/shenandoah/shenandoahRuntime.cpp...done.
 this=0xbd9780b4b20, p=0x60336f6c8, thread=0xbd97809a310) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/sh\
are/gc/shenandoah/shenandoahHeap.inline.hpp:365

 #8 0x000012b021e61237 in ShenandoahBarrierSet::load_reference_barrier_mutator<narrowOop> (this=Reading in symbols for /home/ubunt\
u/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shenandoah/shenandoahBarrierSet.cpp...done.
 0xbd9780dfd20, obj=0x60336f6c8, load_addr=0x7fe840fbc) at /home/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/shar\
e/gc/shenandoah/shenandoahBarrierSet.inline.hpp:71

 #9 0x000012b021e60457 in ShenandoahRuntime::load_reference_barrier_strong_narrow (src=0x60336f6c8, load_addr=0x7fe840fbc) at /hom\
e/ubuntu/Devel/kdnilsen/gitfarm/borrow-from-old-gen/src/hotspot/share/gc/shenandoah/shenandoahRuntime.cpp:59
 #10 0x0000647e7aaf997f in ?? ()
 #11 0x0000084f00000001 in ?? ()
 #12 0x0000000000000000 in ?? ()
```
There has been discussion with engineers responsible for the design of the Shenandoah OOM-during-evacuation protocol, and the current assessment is that some redesign of this protocol is required.