Test nsk/jvmti/unit/timers/JvmtiTest experienced a timeout on 2/5/14's nightly test run.

Test run URL: http://aurora.ru.oracle.com/functional/faces/RunDetails.xhtml?names=381301.JAVASE.NIGHTLY.VM.RT_Baseline.2014-02-05-251&show-limit=0&filter=
Host: jtg-v240-4, Sun Sparcv9 1280 MHz, 2 cores, 4G, Solaris / Solaris 10, sun4u
Options: -d64 -server -Xcomp -XX:MaxRAMFraction=8 -XX:+CreateMinidumpOnCrash -XX:ReservedCodeCacheSize=256M


Dan Daughtery's initial analysis:

For this failure, the JvmtiTest_copy_1/26317.jstack.fml
shows several threads in JVM_MonitorWait() which gets us
into ObjectMonitor::wait() which is the sweet spot of
my hang in 8028073...

For this failure, we have info in JvmtiTest_copy_1/26317.jstack
so we have more 'thread dump' like info:

"JvmtiTest-4" #11 prio=5 os_prio=64 tid=0x000000010110f000 nid=0x10 in Object.wait() [0xffffffff048fe000]
   java.lang.Thread.State: WAITING (on object monitor)
   JavaThread state: _thread_blocked
Thread: 0x000000010110f000 [0x10] State: _at_safepoint _has_called_back 0 _at_poll_safepoint 0
   JavaThread state: _thread_blocked
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at nsk.jvmti.unit.timers.JvmtiTest.completeThread(JvmtiTest.java:86)
    - locked <0xffffffff4980af18> (a java.lang.Object)
    at nsk.jvmti.unit.timers.JvmtiTest$TestThread.run(JvmtiTest.java:112)

This thread previously locked the monitor and called monitor.wait()
on this line of code:

    static void completeThread() {
<snip>
                finalLock.wait();

There are a total of four threads in the same finalLock.wait() call
and one thread still in the TestThread.run() method...

Info from 26317.jstack does not say much:

"JvmtiTest-5" #12 prio=5 os_prio=64 tid=0x0000000101111000 nid=0x11 runnable [0xffffffff046fe000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_blocked
Thread: 0x0000000101111000 [0x11] State: _call_back _has_called_back 1 _at_poll_safepoint 0
   JavaThread state: _thread_blocked
    at nsk.jvmti.unit.timers.JvmtiTest$TestThread.run(JvmtiTest.java:107)

Info from 26317.jstack.fml:

----------------- t@17 -----------------
0xffffffff7eedaa38 __systemcall6 + 0x24
0xffffffff7eec9da0 call_user_handler + 0x33c
0xffffffff7eeca03c sigacthandler + 0x54
0xffffffff6aa2fda0 <Unknown compiled code>
0xffffffff6a4003b8 <StubRoutines>
0xffffffff7d627b4c void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x6f4
0xffffffff7d626528 void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*) + 0x1c8
0xffffffff7d626648 void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*) + 0xd0
0xffffffff7d78c42c void thread_entry(JavaThread*,Thread*) + 0xf4
0xffffffff7ddacd98 void JavaThread::thread_main_inner() + 0x260
0xffffffff7ddacab4 void JavaThread::run() + 0x40c
0xffffffff7db306a8 java_start + 0x250
0xffffffff7eed61b4 _lwp_start
Locked ownable synchronizers:
    - None

Based on this:

"main" #1 prio=5 os_prio=64 tid=0x0000000100112800 nid=0x2 in Object.wait() [0xffffffff7b0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
   JavaThread state: _thread_blocked
Thread: 0x0000000100112800 [0x 2] State: _at_safepoint _has_called_back 0 _at_poll_safepoint 0
   JavaThread state: _thread_blocked
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at nsk.jvmti.unit.timers.JvmtiTest.run(JvmtiTest.java:55)
    - locked <0xffffffff4980af28> (a java.lang.Object)
    at nsk.jvmti.unit.timers.JvmtiTest.main(JvmtiTest.java:41)

it looks like the main thread is stuck here:

                while (counter < THREADS_LIMIT) {
                    counterLock.wait();
                }

because JvmtiTest-5 has not finished. If there were signs in the
thread stack for JvmtiTest-5 that it had made it to the
completeThread() call, then we could have a 8028073 hang here...

Time to check the JVM/TI agent... this JVM/TI agent only
enables the JVMTI_EVENT_VM_INIT and JVMTI_EVENT_VM_DEATH events
and does not use JVM/TI RawMonitors...

Without the right kind of JVM/TI agent, this can't be 8028073...