There are a couple of bugs in ShenandoahEvacOOMHandler related code.

1) ShenandoahEvacOOMHandler::register_thread()
It does not do OOM check before blindly cmpxchg in threads_in_evac. The cmpxchg can success even OOM is in progress, that results the thread can continue evacuate objects.

2) When a nested OOM scope encounters OOM, it does not decrease threads_in_evac count before entering wait_for_no_evac_threads() loop, it deadlocks itself.