intermittent failures of assert(orig_key != 0L,"jni weak reference cleared!!")

The test is
     nsk/jdi/stress/serial/heapwalking002

See for example nightly result:

/java/sqe/results/vm/gtee/JDK7/NIGHTLY/VM/2008-07-21/GC_Baseline-XXParallel/vm/solaris-amd64/server/mixed/vm-solaris-amd64_server_mixed_nsk.quick-jdi.testlist2008-07-21-22-51-58/ResultDir/heapwalking001/heapwalking001.log

Also, test
      nsk/jdi/VMOutOfMemoryException/VMOutOfMemoryException001
has been seen to fail with the same error.


The error is
[2008-07-20T12:42:19.37] debugee.stderr> Debuggee nsk.share.jdi.AbstractJDIDebuggee : sending the command: ready
[2008-07-20T13:08:20.46] debugee.stdout> # To suppress the following error report, specify this argument
[2008-07-20T13:08:20.46] debugee.stdout> # after -XX: or in .hotspotrc: SuppressErrorAt=/jvmtiTagMap.cpp:280
[2008-07-20T13:08:20.46] debugee.stdout> #
[2008-07-20T13:08:20.46] debugee.stdout> # A fatal error has been detected by the Java Runtime Environment:
[2008-07-20T13:08:20.46] debugee.stdout> #
[2008-07-20T13:08:20.46] debugee.stdout> # Internal Error (/tmp/jprt-jprtadm/P1/B/172814.iv159533/source/src/share/vm/prims/jvmtiTagMap.cpp:280), pid=15473, tid=16
[2008-07-20T13:08:20.46] debugee.stdout> # Error: assert(orig_key != 0L,"jni weak reference cleared!!")
[2008-07-20T13:08:20.46] debugee.stdout> #
[2008-07-20T13:08:20.46] debugee.stdout> # Java VM: OpenJDK 64-Bit Server VM (14.0-b01-2008-07-17-172814.iv159533.numa-solaris-config-fastdebug mixed mode solaris-sparc )
[2008-07-20T13:08:20.46] debugee.stdout> # An error report file with more information is saved as:
[2008-07-20T13:08:20.46] debugee.stdout> # /export/local/6653.JDK7.NIGHTLY.VM+solaris-sparcv9_server_mixed_nsk.quick-jdi.testlist/results/ResultDir/VMOutOfMemoryException001/hs_err_pid15473.log
[2008-07-20T13:08:20.46] debugee.stdout> #
[2008-07-20T13:08:20.46] debugee.stdout> # If you would like to submit a bug report, please visit:
[2008-07-20T13:10:50.49] debugee.stdout> # http://java.sun.com/webapps/bugreport/crash.jsp
[2008-07-20T13:10:50.49] debugee.stdout> #
[2008-07-20T13:10:50.49] debugee.stdout> VM option '-PrintVMOptions'
[2008-07-20T13:10:50.49] debugee.stdout> VM option '+UseParallelGC'
[2008-07-20T13:10:50.49] debugee.stdout> VM option '+UseParallelOldGC'

At least so far, the failure has only been seen wihen the debuggee is run with -XX:+UseParallelOldGC.
The failure is intermittent.

Update: This original sighting occurred in the 2008.07.21 nightly
and used HSX-14-B01 bits. That is after the fix for 6539517 was
available so there is still some issue with the Parallel collectors
at this point in time.
Test
nsk/jdi/ObjectReference/referringObjects/referringObjects003
failed with the same assertion during PIT for hs14b08.

Link to the log:
http://sqeweb.sfbay.sun.com/nfs/comp/vm/misc/results/vm/gtee/HSX/PIT/VM/14/b08/jdk7b41/fastdebug/vm/windows-i586/client/comp/windows-i586_client_comp_nsk.jdi.testlist/ResultDir/referringObjects003/referringObjects003.log

Since test was run on windows with flag '-client' SerialGC should be used, so looks like this failure isn't specific for -XX:+UseParallelOldGC.

Update: This sighting occurred in the HSX-14-B08 PIT so the
bug introduced by 6711316 is present in the GC framework
collectors. It is very likely that this sighting is fixed by
the work on 6862295.
Another sighting in nightly testing for another test:

New nsk.quick-jdi failures (from 2008.12.17)
* nsk/jdi/stress/serial/mixed001
        This test failed the following assertion on Win32 Server VM
        (machine cheesepuff):

            Internal Error (src/share/vm/prims/jvmtiTagMap.cpp:280)
            Error: assert(orig_key != 0,"jni weak reference cleared!!")

        This failure mode is covered by the following bug:

            6729714 4/4 nsk/jdi/stress/serial/heapwalking002 fails

        I will copy this entry to 6729714.

Update: This sighting occurred in the 2008.12.17 nightly so the
bug introduced by 6711316 is present in the GC framework
collectors. It is very likely that this sighting is fixed by
the work on 6862295.
nsk/jdi/stress/serial/heapwalking002 continues to fail in
Parallel GC, but with different failure modes:

http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-20/GC_Baseline-XXParallel/vm/solaris-sparcv9/server/mixed/solaris-sparcv9_server_mixed_nsk.quick-jdi.testlist/ResultDir/heapwalking002/heapwalking002.log


[2009-07-21T14:19:44.93] debugee.stderr> # ERROR: Unexpected exception in debuggee: nsk.share.TestBug: Test execution error: WeakReference was collected
[2009-07-21T14:19:44.93] # ERROR: Unexpected exception during test execution(debugger: nsk.jdi.ReferenceType.instances.instances003.instances003@8f0c85e): java.lang.NullPointerException
[2009-07-21T14:19:44.93] java.lang.NullPointerException
[2009-07-21T14:19:44.93] at nsk.share.jdi.TestDebuggerType2.isDebuggeeReady(TestDebuggerType2.java:218)
[2009-07-21T14:19:44.93] at nsk.jdi.ReferenceType.instances.instances003.instances003.testClass(instances003.java:82)
[2009-07-21T14:19:44.93] at nsk.jdi.ReferenceType.instances.instances003.instances003.doTest(instances003.java:105)
[2009-07-21T14:19:44.93] at nsk.share.jdi.SerialExecutionDebugger.executeTests(SerialExecutionDebugger.java:271)
[2009-07-21T14:19:44.93] at nsk.share.jdi.SerialExecutionDebugger.doTest(SerialExecutionDebugger.java:210)
[2009-07-21T14:19:45.07] at nsk.share.jdi.TestDebuggerType2.runIt(TestDebuggerType2.java:192)
[2009-07-21T14:19:45.07] at nsk.share.jdi.SerialExecutionDebugger.main(SerialExecutionDebugger.java:42)
[2009-07-21T14:19:45.07] # ERROR: Unexpected exception: java.lang.NullPointerException
[2009-07-21T14:19:45.07] java.lang.NullPointerException
[2009-07-21T14:19:45.07] at nsk.share.jdi.TestDebuggerType2.isDebuggeeReady(TestDebuggerType2.java:218)
[2009-07-21T14:19:45.07] at nsk.share.jdi.SerialExecutionDebugger.executeTests(SerialExecutionDebugger.java:292)
[2009-07-21T14:19:45.07] at nsk.share.jdi.SerialExecutionDebugger.doTest(SerialExecutionDebugger.java:210)
[2009-07-21T14:19:45.07] at nsk.share.jdi.TestDebuggerType2.runIt(TestDebuggerType2.java:192)
[2009-07-21T14:19:45.07] at nsk.share.jdi.SerialExecutionDebugger.main(SerialExecutionDebugger.java:42)
[2009-07-21T14:19:45.07] # ERROR: TEST FAILED: debuggee's process finished with status: 97
[2009-07-21T14:19:45.07] TEST FAILED


http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-06-06/GC_Baseline-XXParallel/vm/solaris-sparcv9/server/mixed/solaris-sparcv9_server_mixed_nsk.quick-jdi.testlist/ResultDir/heapwalking002/heapwalking002.log

[2009-06-07T14:02:29.35] # ERROR: Wrong value was returned by VirtualMachine.instanceCounts(java.lang.String): 1183, expected: >= 1222
[2009-06-07T14:02:29.35] # ERROR: Unexpected size of referenceType.instances(java.lang.String): 1183, expected: >= 1222
[2009-06-07T14:02:29.35] # ERROR: Unexpected exception during test execution(debugger: nsk.jdi.ReferenceType.instances.instances003.instances003@528acf6e): java.lang.IndexOutOfBoundsException: Index: 11, Size: 11
[2009-06-07T14:02:29.35] java.lang.IndexOutOfBoundsException: Index: 11, Size: 11
[2009-06-07T14:02:29.35] at java.util.ArrayList.rangeCheck(ArrayList.java:570)
[2009-06-07T14:02:29.35] at java.util.ArrayList.get(ArrayList.java:348)
[2009-06-07T14:02:29.35] at nsk.jdi.ReferenceType.instances.instances003.instances003.testClass(instances003.java:77)
[2009-06-07T14:02:29.37] at nsk.jdi.ReferenceType.instances.instances003.instances003.doTest(instances003.java:105)
[2009-06-07T14:02:29.37] at nsk.share.jdi.SerialExecutionDebugger.executeTests(SerialExecutionDebugger.java:271)
[2009-06-07T14:02:29.37] at nsk.share.jdi.SerialExecutionDebugger.doTest(SerialExecutionDebugger.java:210)
[2009-06-07T14:02:29.37] at nsk.share.jdi.TestDebuggerType2.runIt(TestDebuggerType2.java:192)
[2009-06-07T14:02:29.37] at nsk.share.jdi.SerialExecutionDebugger.main(SerialExecutionDebugger.java:42)
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee: received the command: quit
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee: exiting
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee nsk.share.jdi.HeapwalkingDebuggee@15af33d6 finished successfully
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee nsk.share.jdi.SerialExecutionDebuggee : sending the command: ready
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee: received the command: COMMAND_CLEAR_DEBUGGEE
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee nsk.share.jdi.SerialExecutionDebuggee : sending the command: ready
[2009-06-07T14:02:29.37] WARNING: stop test execution because of timeout (max execution time for single test: 119101, time left: 0)
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee: received the command: quit
[2009-06-07T14:02:29.37] debugee.stderr> Debuggee: exiting
[2009-06-07T14:02:29.77] Debuggee's process finished with status: 95
[2009-06-07T14:02:29.77] TEST FAILED


Update: This entry is now being tracked by 6867612.
nsk/jdi/VMOutOfMemoryException/VMOutOfMemoryException001 continues
to fail in Parallel GC, but with a different failure mode:

http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-30/GC_Baseline-XXParallel/vm/solaris-i586/client/mixed/solaris-i586_client_mixed_nsk.quick-jdi.testlist/ResultDir/VMOutOfMemoryException001/VMOutOfMemoryException001.log

[2009-07-31T07:16:46.56] binder> Launching debugee
[2009-07-31T07:16:46.56] binder> Waiting for VM initialized
[2009-07-31T07:16:50.75] Initial VMStartEvent received: VMStartEvent in thread main
[2009-07-31T07:16:50.75] debugee.stderr> Debuggee nsk.share.jdi.AbstractJDIDebuggee : sending the command: ready
[2009-07-31T07:16:50.75] Got expected exception: com.sun.jdi.VMOutOfMemoryException
[2009-07-31T07:16:50.75] # ERROR: TEST FAILED: debuggee's process finished with status: 1
[2009-07-31T07:16:50.75] TEST FAILED


The exit status of 1 indicates an ungraceful termination due
to an out of memory condition. My search showed failures
like this on Solaris X86, Solaris AMD64, and Solaris SPARC


Update: This entry is now being tracked by 6867617.
Nightly search of Parallel GC did not find any failures for this test:

    nsk/jdi/ObjectReference/referringObjects/referringObjects003

Looks like nightly data goes back to 2008.08.01 so this failure
didn't show up outside of HSX-14-B08 PIT.

Nightly search of Parallel GC found no valid failures for this test:

    nsk/jdi/stress/serial/mixed001

but but it did find one general timeout.

nsk/jdi/stress/serial/mixed002 continues to fail in
Parallel GC, but with different failure modes:

http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-03-09/GC_Baseline-XXParallel/vm/solaris-amd64/server/mixed/solaris-amd64_server_mixed_nsk.quick-jdi.testlist/ResultDir/mixed002/mixed002.log

[2009-03-10T09:41:42.43] debugee.stderr> Debuggee nsk.share.jdi.HeapwalkingDebuggee : sending the command: ready
[2009-03-10T09:41:42.43] # ERROR: Wrong value was returned by VirtualMachine.instanceCounts(nsk.share.jdi.TestClass1): 7, expected: 50
[2009-03-10T09:41:42.43] # ERROR: Unexpected size of referenceType.instances(nsk.share.jdi.TestClass1): 7, expected: 50
[2009-03-10T09:41:42.43] # ERROR: Unexpected exception during test execution(debugger: nsk.jdi.ReferenceType.instances.instances003.instances003@27b15692): java.lang.IndexOutOfBoundsException: Index: 7, Size: 7
[2009-03-10T09:41:42.43] java.lang.IndexOutOfBoundsException: Index: 7, Size: 7
[2009-03-10T09:41:42.43] at java.util.ArrayList.rangeCheck(ArrayList.java:571)
[2009-03-10T09:41:42.43] at java.util.ArrayList.get(ArrayList.java:349)
[2009-03-10T09:41:42.43] at nsk.jdi.ReferenceType.instances.instances003.instances003.testClass(instances003.java:77)
[2009-03-10T09:41:42.43] at nsk.jdi.ReferenceType.instances.instances003.instances003.doTest(instances003.java:105)
[2009-03-10T09:41:42.43] at nsk.share.jdi.SerialExecutionDebugger.executeTests(SerialExecutionDebugger.java:271)
[2009-03-10T09:41:42.43] at nsk.share.jdi.SerialExecutionDebugger.doTest(SerialExecutionDebugger.java:210)
[2009-03-10T09:41:42.43] at nsk.share.jdi.TestDebuggerType2.runIt(TestDebuggerType2.java:192)
[2009-03-10T09:41:42.43] at nsk.share.jdi.SerialExecutionDebugger.main(SerialExecutionDebugger.java:42)
[2009-03-10T09:41:42.43] debugee.stderr> Debuggee: received the command: quit
[2009-03-10T09:41:42.43] debugee.stderr> Debuggee: exiting


http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-02-11/GC_Baseline-XXParallel/vm/solaris-i586/server/mixed/solaris-i586_server_mixed_nsk.quick-jdi.testlist/ResultDir/mixed002/mixed002.log

[2009-02-12T10:43:51.84] debugee.stderr> Debuggee: received the command: deleteInstances:java.lang.String:50
[2009-02-12T10:43:51.84] debugee.stderr> # ERROR: Unexpected exception in debuggee: java.util.NoSuchElementException
[2009-02-12T10:43:51.84] debugee.stderr> java.util.NoSuchElementException
[2009-02-12T10:43:51.84] debugee.stderr> at java.util.ArrayList$Itr.next(ArrayList.java:757)
[2009-02-12T10:43:51.84] debugee.stderr> at nsk.share.ObjectInstancesManager.deleteAllReferrers(ObjectInstancesManager.java:102)
[2009-02-12T10:43:51.96] debugee.stderr> at nsk.share.jdi.HeapwalkingDebuggee.parseCommand(HeapwalkingDebuggee.java:115)
[2009-02-12T10:43:51.96] debugee.stderr> at nsk.share.jpda.AbstractDebuggeeTest.doTest(AbstractDebuggeeTest.java:284)
[2009-02-12T10:43:51.96] debugee.stderr> at nsk.share.jdi.SerialExecutionDebuggee.parseCommand(SerialExecutionDebuggee.java:92)
[2009-02-12T10:43:51.96] debugee.stderr> at nsk.share.jpda.AbstractDebuggeeTest.doTest(AbstractDebuggeeTest.java:284)
[2009-02-12T10:43:51.96] debugee.stderr> at nsk.share.jpda.AbstractDebuggeeTest.doTest(AbstractDebuggeeTest.java:269)
[2009-02-12T10:43:51.96] debugee.stderr> at nsk.share.jdi.SerialExecutionDebuggee.main(SerialExecutionDebuggee.java:23)
[2009-02-12T10:43:51.96] debugee.stderr>


Update: The mixed002 failure in this entry is now being tracked by 6867612.
Nightly search of Parallel GC did not find any valid failures
for this test after the G1 push:

    nsk/jdi/stress/serial/heapwalking001

The one failure was an infrastructure timeout.


Nightly search of Parallel GC did not find any valid failures
for this test:

    nsk/jdi/ReferenceType/instances/instances005


Nightly search of Parallel GC did not find any failures
for these tests:

    closed/compiler/6507107/HeapwalkingTest.java
    nsk/jdi/VirtualMachine/instanceCounts/instancecounts004
The following VM/NSK test is also
failing due to this assertion:

    nsk/hprof/regression/HeapThrasher

http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-30/RT_Baseline/vm/linux-i586/client/mixed/linux-i586_client_mixed_nsk.hprof.testlist/analysis.html
http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-30/RT_Baseline/vm/solaris-i586/client/mixed/solaris-i586_client_mixed_nsk.hprof.testlist/analysis.html
http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-30/RT_Baseline/vm/solaris-sparc/client/mixed/solaris-sparc_client_mixed_nsk.hprof.testlist/analysis.html
http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-30/RT_Baseline/vm/solaris-sparcv9/server/mixed/solaris-sparcv9_server_mixed_nsk.hprof.testlist/analysis.html
http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2009-07-30/RT_Baseline/vm/windows-i586/client/mixed/windows-i586_client_mixed_nsk.hprof.testlist/ResultDir/HeapThrasher/HeapThrasher.log