This assertion failure was observed during some ad-hoc testing of G1. The test directory can be found at
 http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK6/ADHOC/VM/2010-04-13_07/vm/solaris-i586/server/mixed/solaris-i586_server_mixed_vm.gc.testlist/ResultDir/JumbleGC002/

The machine on which the test ran is vm-b1600x-2, a single cpu intel x86 machine:

vm-b1600x-2{jc234399}:263> uname -a
SunOS vm-b1600x-2 5.10 Generic_139556-08 i86pc i386 i86pc
vm-b1600x-2{jc234399}:264> psrinfo -v
Status of virtual processor 0 as of: 04/21/2010 11:43:59
  on-line since 11/15/2009 14:52:04.
  The i386 processor operates at 1533 MHz,
        and has an i387 compatible floating point processor.

The stack trace (from the hs_err file) is:

Stack:
[error occurred during error reporting (printing stack bounds), id 0xe0000000]

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x176faaa];; __1cHVMErrorGreport6MpnMoutputStream__v_+0x5be
V [libjvm.so+0x1770c32];; __1cHVMErrorOreport_and_die6M_v_+0x586
V [libjvm.so+0x7f24b9];; __1cYreport_assertion_failure6Fpkci1_v_+0x61
V [libjvm.so+0x1770586];; __1cHVMErrorGreport6MpnMoutputStream__v_+0x109a
V [libjvm.so+0x1770c32];; __1cHVMErrorOreport_and_die6M_v_+0x586
V [libjvm.so+0x7f24b9];; __1cYreport_assertion_failure6Fpkci1_v_+0x61
V [libjvm.so+0x8cb9b0];; __1cFframebDinterpreter_frame_monitor_end6kM_pnPBasicObjectLock__+0x70
V [libjvm.so+0x8c5a32];; __1cFframeToops_interpreted_do6MpnKOopClosure_pknLRegisterMap_b_v_+0x85e
V [libjvm.so+0x8c92d0];; __1cFframeQoops_do_internal6MpnKOopClosure_pnPCodeBlobClosure_pnLRegisterMap_b_v_+0xb8
V [libjvm.so+0x1683244];; __1cKJavaThreadHoops_do6MpnKOopClosure_pnPCodeBlobClosure__v_+0x290
V [libjvm.so+0x168b182];; __1cHThreadsZpossibly_parallel_oops_do6FpnKOopClosure_pnPCodeBlobClosure__v_+0x96
V [libjvm.so+0x14b2546];; __1cKSharedHeapUprocess_strong_roots6Mbbn0AOScanningOption_pnKOopClosure_pnPCodeBlobClosure_pnQOopsInGenClosure__v_+0xc2
V [libjvm.so+0x8e2f2a];; __1cPG1CollectedHeapXg1_process_strong_roots6MbnKSharedHeapOScanningOption_pnKOopClosure_pnXOopsInHeapRegionClosure_6pnQOopsInGenClosure_i_v_+0x1b6
V [libjvm.so+0x8e96ff];; __1cJG1ParTaskEwork6Mi_v_+0x4d7
V [libjvm.so+0x17b58b9];; __1cKGangWorkerEloop6M_v_+0x409
V [libjvm.so+0x17b5350];; __1cKGangWorkerDrun6M_v_+0x2c
V [libjvm.so+0x1341e2f];; java_start+0x117
C [libc.so.1+0xa7055];; _thr_setup+0x4e
C [libc.so.1+0xa7340];; _lwp_start+0x0

i.e. one of the GC worker threads is scanning the stack frames of a JavaThread as part of the root-set processing during an evacuation pause. The thread that is being processed by the GC has the following stack trace:

JavaThread 0x09380000 (nid = 105) was being processed
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+0
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98

   .... ~3300 calls deep

j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j nsk.share.gc.NonbranchyTree.<init>(IFI)V+155
j gc.gctests.JumbleGC002.JumbleGC002$AuxiliaryThread.fillVector()V+191
j gc.gctests.JumbleGC002.JumbleGC002$AuxiliaryThread.run()V+55
v ~StubRoutines::call_stub