###@###.### 2003-11-12

The recent fix for the following bug:

4936227 2/4 JDWP: JVMTI_ERROR_WRONG_PHASE(112) on getting class status

introduced a pair of RawMonitors for coordinating JVM/TI callback
functions during VM shutdown. There is a deadlock in that fix.


The deadlock reproduces quite nicely with the following NSK test:

    nsk/jdi/ObjectReference/getValue/getvalue001

One thread is trying to shutdown the VM and is posting a VM_DEATH
event. It is in the cbVMDeath() handler function:

    debugMonitorEnter(callbackLock); {
        vm_death_callback_active = JNI_TRUE;
        threadControl_resumeAll();
        while (active_callbacks > 0) {
HERE=> debugMonitorWait(callbackLock); /* Some CALLBACK is still active */
        }
    } debugMonitorExit(callbackLock);

From the debugger:
    active_callbacks = 0

Here is a snippet of the thread's call stack:

  [8] jvmti_RawMonitorWait(env = 0x30394, monitor = 0x164678, millis = -1LL), line 3602 in "jvmtiEnter.cpp"
  [9] debugMonitorWait(monitor = 0x164678), line 999 in "util.c"
  [10] cbVMDeath(jvmti_env = 0x30394, env = 0x47584), line 1050 in "eventHandler.c"
  [11] JvmtiExport::post_vm_death(), line 749 in "jvmtiExport.cpp"
  [12] before_exit(thread = 0x474b0), line 388 in "java.cpp"
  [13] JVM_Halt(code = 95), line 306 in "jvm.cpp"
  [14] Java_java_lang_Shutdown_halt0(0x47584, 0xffbedeb8, 0x5f, 0x0, 0x0, 0x0), at 0xfd7c06cc
  [15] 0xf8812564(0xf2d580f8, 0xffbedf34, 0xffbedeb8, 0xffffff80, 0xb8, 0x0), at 0xf8812563


The other thread is just finishing up the posting of a BREAKPOINT event.
It is in the cbBreakpoint() handler function:

    BEGIN_CALLBACK() {
        (void)memset(&info,0,sizeof(info));
        info.ei = EI_BREAKPOINT;
        info.thread = thread;
        info.clazz = getMethodClass(jvmti_env, method);
        info.method = method;
        info.location = location;
        event_callback(env, &info);
    } END_CALLBACK(); <= HERE

The END_CALLBACK macro looks like (backslashes removed):

#define END_CALLBACK() /* Part of bypass if body */
        debugMonitorEnter(callbackLock); {
            active_callbacks--;
            if (active_callbacks < 0) {
                EXIT_ERROR(0, "Problems tracking active callbacks");
            }
            if (vm_death_callback_active) {
                if (active_callbacks == 0) {
                    debugMonitorNotifyAll(callbackLock);
                }
                /* Now block because VM is about to die */
                debugMonitorEnter(callbackBlock);
HERE=> debugMonitorWait(callbackBlock);
                debugMonitorExit(callbackBlock);
            }
        } debugMonitorExit(callbackLock);
    }
} /* END OF CALLBACK */

Here is a snippet of the thread's call stack:

  [8] jvmti_RawMonitorWait(env = 0x30394, monitor = 0x164728, millis = -1LL), line 3602 in "jvmtiEnter.cpp"
  [9] debugMonitorWait(monitor = 0x164728), line 999 in "util.c"
  [10] cbBreakpoint(jvmti_env = 0x30394, env = 0x114f74, thread = 0x1c6ca4, method = 0x115b9c, location = 10LL), line 731 in "eventHandler.c"
  [11] JvmtiExport::post_raw_breakpoint(thread = 0x114ea0, method = 0xf2e11750, location = 0xf2e11722 "\xca\xbb"), line 427 in "jvmtiExport.cpp"
  [12] InterpreterRuntime::_breakpoint(thread = 0x114ea0, method = 0xf2e11750, bcp = 0xf2e11722 "\xca\xbb"), line 513 in "interpreterRuntime.cpp"
  [13] 0xf882e184(0xeec6dec8, 0xb6, 0x66, 0xf882e030, 0xeec6dec0, 0xee881550), at 0xf882e183

The complete thread dump is attached as threads.log.1

This is a classic deadlock scenario. The BREAKPOINT posting thread is
waiting on callbackBlock while still holding callbackLock; it did the
notify on callbackLock, but has not released it. The shutdown thread is
waiting to reaquire on callbackLock; it has seen the notification on
callbackLock, but it cannot get the monitor back yet. After it regains
callbackLock it is supposed to notify callbackBlock.


###@###.### 2003-11-13

This deadlock has also been observed with the following NSK test:

    nsk/jdi/ThreadReference/suspend/suspend001