Debug agent can deadlock on callbackLock when using StackFrame.PopFrames

There is a deadlock with the callbackLock that currently happens with the work I've done for JDK-8328866 (ranked monitor support), but it could potentially happen without it. The only thing that saves the current implementation from the deadlock is the fact that RawMonitorExit releases the monitor before doing a self suspend, and there is nothing else done while holding the callbackLock that might allow for a self suspend.

Before explaining the deadlock, it would help to understand all the threads involved:

 • Thread 1: An app thread that gets an event that does a suspend all, and the debugger does a StackFrame.PopFrames on.
 • Thread 2: A 2nd app thread that gets an event. Suspend policy doesn't matter.
 • JDWP Command Reader thread (Reader thread): Executes commands sent by the debugger (StackFrame.PopFrames in this case).
 • JDWP Event Helper Thread (Helper thread): JVMTI events come in on the app threads (Thread 1 and Thread 2), but are queued up for later processing by the Helper thread. Part of this processing includes doing any suspension the event requires (eg. suspend all)

The deadlock is with the callbackLock. The debug agent grabs the callbackLock on entry and exit of the JVMTI event handlers. It only holds it for a very short time so it can atomically update activeCallbacks. This is part of what is done by the BEGIN_CALLBACK and END_CALLBACK macros, which are used by every JVMTI callback.

The deadlock happens because there are events coming in on two different threads (Thread 1 and Thread 2) at nearly the same time, but it also requires StackFrame.PopFrames to be used. Thread 1 gets past the part where callbackLock is acquired and released by BEGIN_CALLBACK, and has queued up the event for processing by the Helper thread. It has not gotten to the END_CALLBACK yet.

Before the Helper thread gets to the point of doing the suspend all, a 2nd event comes in on Thread 2. It manages to acquire the callbackLock in BEGIN_CALLBACK, but before it can release it, the Helper thread suspends Thread 2 as part of doing the suspend all. So now Thread 2 is suspended and holding the callbackLock. Normally this would quickly become unstuck when the debugger is done with the event on Thread 1 and does a resume all. However, in this test case the debugger is instead issuing a StackFrame.PopFrames of Thread 1. The implementation of StackFrame.PopFrames (executed on the Reader thread) resumes Thread 1 after doing the JVMTI PopFrames on Thread 1. This allows Thread 1 to finally get to the exit the JVMTI callback, and as part of this it executes END_CALLBACK. But END_CALLBACK tries to acquire callbackLock, and blocks on the Thread 2, which is still suspended. So we are deadlocked.

I'm not actually seeing this deadlock without my ranked monitor support. I've concluded the reason that the deadlock does not happen without my ranked monitor support is because once callbackLock is entered, if there is a self suspend request it will not happen until after callbackLock is exited. RawMonitorExit self suspends after releasing callbackLock, never before, so after entering callbackLock, we can be sure that it will always be exited before any suspension is done, thus no deadlock. I noticed that if while holding the callbackLock I do anything that might allow the thread to self suspend (a call to JNI IsSameObject() is sufficient), then the deadlock will happen. This is true without any ranked monitor support in place.

The reason it deadlocks with my ranked monitor support in place is because before doing the RawMonitorExit, first a RawMonitorEnter of dbgRawMonitor is done, and this adds an opportunity for self suspension.

I think the solution is to have getLocks() also grab callbackLock. It may need to grab callbackBlock also. I still need to experiment with this.