It seems, there a GC issue that is provoked by the stress test
com/sun/jdi/OomDebugTest.java added as new unit test with the fix of:
   https://bugs.openjdk.java.net/browse/JDK-8153711

Below is a copy of the comment from the JDK-8153711 bug report:

Note that I'm still seeing fairly rare cases (1 out of 1000) where OomDebugTest would timeout. The debugger waits for the reply from the debuggee which is in process of fulfilling a newInstance() request of a primitive array. However that reply never seems to come, because for some reason the call to jni_NewByteArray never completes? This looks like an issue in GC to me and not related to this bug. Here is the jstack snippet of the hung OomDebugTestTarg JVM process:

[...]
----------------- 28829 -----------------
0x00007efeb8e9cb10 __pthread_cond_wait + 0xc0
0x00007efeb7f3419f Monitor::IWait(Thread*, long) + 0xef
0x00007efeb7f3541e Monitor::wait(bool, long, bool) + 0x22e
0x00007efeb7fff7aa ReferencePendingListLockerThread::receive_and_handle_messages() + 0x4a
0x00007efeb7fff879 ????????
0x00007efeb80dd227 JavaThread::thread_main_inner() + 0x1e7
0x00007efeb7f6f430 java_start(Thread*) + 0xf0
Locked ownable synchronizers:
    - None
[...]
----------------- 28837 -----------------
0x00007efeb8e9ceb9 __pthread_cond_timedwait + 0x129
0x00007efeb7f56bf3 ObjectMonitor::EnterI(Thread*) + 0x3c3
0x00007efeb7f572c0 ObjectMonitor::enter(Thread*) + 0x320
0x00007efeb7fff3ea ReferencePendingListLocker::lock() + 0xca
0x00007efeb813af3f VM_GC_Operation::doit_prologue() + 0x6f
0x00007efeb8149be1 VM_G1IncCollectionPause::doit_prologue() + 0x11
0x00007efeb814818e VMThread::execute(VM_Operation*) + 0x22e
0x00007efeb7bd4fb5 G1CollectedHeap::collect(GCCause::Cause) + 0x95
0x00007efeb7bdb28a G1CollectedHeap::attempt_allocation_humongous(unsigned long, unsigned int*, unsigned int*) + 0x2ea
0x00007efeb7bdb519 G1CollectedHeap::mem_allocate(unsigned long, bool*) + 0x219
0x00007efeb80f6501 TypeArrayKlass::allocate_common(int, bool, Thread*) + 0xa1
0x00007efeb7ce5713 jni_NewByteArray + 0xa3
0x00007efeb641ab56 newInstance + 0x496
0x00007efeb6428fd8 debugLoop_run + 0x268
0x00007efeb643b075 attachThread + 0x25
0x00007efeb7dbaa00 JvmtiAgentThread::start_function_wrapper(JavaThread*, Thread*) + 0xb0
0x00007efeb80dd227 JavaThread::thread_main_inner() + 0x1e7
0x00007efeb7f6f430 java_start(Thread*) + 0xf0
Locked ownable synchronizers:
    - None

Interestingly enough it always seems to get stuck in test2() which instantiates larg-ish primitive byte arrays in the debuggee.


I also, observed a similar deadlock with the OomDebugTest test:

Below is a jstack output fragment:

----------------- 9131 -----------------
0x00007f82c9063d84 __pthread_cond_wait + 0xc4
0x00007f82c84e83c7 Monitor::IWait(Thread*, long) + 0x127
0x00007f82c84ea3ed Monitor::wait(bool, long, bool) + 0x1fd
0x00007f82c877b114 _ZN10JavaThread17java_suspend_selfEv.part.195 + 0xb4
0x00007f82c82cd919 JvmtiRawMonitor::raw_wait(long, bool, Thread*) + 0x229
0x00007f82c82a2a82 JvmtiEnv::RawMonitorWait(JvmtiRawMonitor*, long) + 0xa2
0x00007f82c69d9b49 debugMonitorWait + 0x29
0x00007f82c69cad6e enqueueCommand + 0x11e
0x00007f82c69c6a22 reportEvents.part.3 + 0xa2
0x00007f82c69c7046 event_callback + 0x446
0x00007f82c69c9d55 cbBreakpoint + 0xc5
0x00007f82c82bc78f JvmtiExport::post_raw_breakpoint(JavaThread*, Method*, unsigned char*) + 0x1ef
0x00007f82c80bf662 InterpreterRuntime::_breakpoint(JavaThread*, Method*, unsigned char*) + 0xa2
0x00007f82a89af712 Locked ownable synchronizers:
    - None
. . . . . . . . . . . . . . . . . .

----------------- 9169 -----------------
0x00007f82c9063d84 __pthread_cond_wait + 0xc4
0x00007f82c84e83c7 Monitor::IWait(Thread*, long) + 0x127
0x00007f82c84ea98f Monitor::wait(bool, long, bool) + 0x79f
0x00007f82c861d380 ReferencePendingListLockerThread::receive_and_handle_messages() + 0xb0
0x00007f82c861d8a9 ????????
0x00007f82c878c991 JavaThread::thread_main_inner() + 0x1d1
0x00007f82c878cb3a JavaThread::run() + 0x15a
0x00007f82c854a3b2 java_start(Thread*) + 0x142
Locked ownable synchronizers:
    - None
. . . . . . . . . . . . . . . . . .

----------------- 9202 -----------------
0x00007f82c90640fe __pthread_cond_timedwait + 0x13e
0x00007f82c85228f1 ObjectMonitor::EnterI(Thread*) + 0x581
0x00007f82c8523147 ObjectMonitor::enter(Thread*) + 0x307
0x00007f82c872fb11 ObjectSynchronizer::slow_enter(Handle, BasicLock*, Thread*) + 0xa1
0x00007f82c872fdcb ObjectSynchronizer::fast_enter(Handle, BasicLock*, bool, Thread*) + 0x8b
0x00007f82c861d9ef ReferencePendingListLocker::lock() + 0x13f
0x00007f82c87ffdef VM_GC_Operation::doit_prologue() + 0xbf
0x00007f82c8832221 VM_G1IncCollectionPause::doit_prologue() + 0x11
0x00007f82c882d49e VMThread::execute(VM_Operation*) + 0x2ee
0x00007f82c7f4a5ee G1CollectedHeap::collect(GCCause::Cause) + 0x1ce
0x00007f82c7f56c41 G1CollectedHeap::attempt_allocation_humongous(unsigned long, unsigned int*, unsigned int*) + 0x3e1
0x00007f82c7f56f1d G1CollectedHeap::mem_allocate(unsigned long, bool*) + 0x26d
0x00007f82c7abe0d7 CollectedHeap::array_allocate(KlassHandle, int, int, Thread*) + 0x2d7
0x00007f82c87b0864 TypeArrayKlass::allocate_common(int, bool, Thread*) + 0x1d4
0x00007f82c8179c3d jni_NewByteArray + 0x16d
0x00007f82c69b5d20 newInstance + 0x430
0x00007f82c69c3fd8 debugLoop_run + 0x298
0x00007f82c69d661e attachThread + 0x2e
0x00007f82c82c6093 JvmtiAgentThread::call_start_function() + 0x153
0x00007f82c878c991 JavaThread::thread_main_inner() + 0x1d1
0x00007f82c878cb3a JavaThread::run() + 0x15a
0x00007f82c854a3b2 java_start(Thread*) + 0x142
Locked ownable synchronizers:
    - None