G1: crash during evacuation failure ostensibly due to prematurely collected reference

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x876ac1];; oopDesc*G1ParCopyHelper::copy_to_survivor_space(oopDesc*)+0x131
V [libjvm.so+0x882dd7];; void G1ParCopyClosure<false,(G1Barrier)2,false,false>::do_oop(unsigned*)+0xd7
V [libjvm.so+0x90fdaf];; int instanceKlass::oop_oop_iterate_backwards_v(oopDesc*,OopClosure*)+0x25f
V [libjvm.so+0x8757ca];; void G1CollectedHeap::handle_evacuation_failure_common(oopDesc*,markOopDesc*)+0x8aa
V [libjvm.so+0x874640];; oopDesc*G1CollectedHeap::handle_evacuation_failure_par(OopsInHeapRegionClosure*,oopDesc*)+0xf0
V [libjvm.so+0x877502];; oopDesc*G1ParCopyHelper::copy_to_survivor_space(oopDesc*)+0xb72
V [libjvm.so+0x887d97];; void G1ParCopyClosure<false,(G1Barrier)2,false,true>::do_oop_work<unsigned>(__type_4*)+0x97
V [libjvm.so+0x87ebd2];; void G1ParTask::work(int)+0x1842
V [libjvm.so+0xd862eb];; void GangWorker::loop()+0xab
V [libjvm.so+0xbef813];; java_start+0x4c3
C [libc.so.1+0xd6e3b] _thr_slot_offset+0x31b;; _thr_setup+0x5b
C [libc.so.1+0xd7070] _thr_slot_offset+0x550;; _lwp_start+0x0


Easy to reproduce with:

$JAVA -d64 -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseCompressedOops -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xms32m -Xmx196m -XX:G1SteadyStateUsed=50 -XX:G1SteadyStateUsedDelta=40 dacapo-2006-10.jar -s default xalan

This also reproduces with the same command-line as above even when you drop
-XX:+UseCompressedOops and/or -d64. It occurs on Intel as well as Sparc.

In a typical crash we found that we were iterating over the oops of a ByteBuffer.
The Byte[] payload pointer from the ByteBuffer was a humongous byte array that
had been reclaimed. -XX:+Verify{Before,After,During}GC would be useful fo
debugging.

If you run with a fastdebug JVM, you run into a different bug first (6826736,
which is believed to be caused by compiler2 bug 6851282), so this might be a
bit difficult to debug efficiently until those have been gotten out of the way.

I am logging this bug as a placeholder and will fill in further information
shortly when I get to debugging it in earnest (shortly).