segv in JvmtiEnvBase::check_for_periodic_clean_up()

Compiler nightly tests saw a failure running regression test HeapwalkingTest.java with the option -XX:CompileThreshold=100.

I was able to reproduce this on the nightly testing machine em64t-002.SFBay (a 4cpu xeon) although it was fairly difficult. Only 1 out of 20-30 runs would fail. It appears to fail after the test runs for quite a bit (this is significant given the failure). After finally getting a core file here is what I observed. The vmthread crashes with this stack trace:

#0 0x00000037b922e37d in raise () from /lib64/tls/libc.so.6
#1 0x00000037b922faae in abort () from /lib64/tls/libc.so.6
#2 0x0000002a95e5fd5e in os::abort () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#3 0x0000002a9603b675 in VMError::report_and_die () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#4 0x0000002a95e66e00 in JVM_handle_linux_signal () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#5 0x0000002a95e6238a in signalHandler () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#6 <signal handler called>
#7 0x0000002a95d0a670 in JvmtiEnvBase::check_for_periodic_clean_up()::ThreadInsideIterationClosure::do_thread ()
#9 0x0000002a95d054fc in JvmtiEnvBase::check_for_periodic_clean_up () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#10 0x0000002a95d23912 in JvmtiGCMarker::JvmtiGCMarker$base () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#11 0x0000002a9603cc82 in VM_ParallelGCFailedAllocation::doit () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#12 0x0000002a96075e57 in VM_Operation::evaluate () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#13 0x0000002a96074dc6 in VMThread::evaluate_operation () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#14 0x0000002a96075238 in VMThread::loop () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#15 0x0000002a96074a23 in VMThread::run () from /tmp/jdk/jre/lib/amd64/server/libjvm.so
#16 0x0000002a95e655dd in java_start () from /tmp/jdk/jre/lib/amd64/server/libjvm.so

The reason for the SEGV is that the thread that has been passed to JvmtiEnvBase::check_for_periodic_clean_up()::ThreadInsideIterationClosure::do_thread
is NULL. The reason the value is NULL is because the value of WatcherThread::_watcher_thread is NULL. I don't know what happened to the watcher thread.

I've attached a snapshot of the stacks for the threads that are active, there is no watcher thread. I'm not sure if this is a runtime problem or a jvmti problem. I flipped a coin and picked runtime. Until they happen to reboot the machine I reproduced this on the core file will be available for further examination. Let me (SteveG) know if you want to find it.