Heapdump test
heapdump/JMapPerm
intermittently fails in SPB nightly with solaris-amd64 JDK. This test fills perm gen and uses jmap to create heap dump, this test fails because of segmentation fault in observed VM. I can reproduce this failure since b79.

To reproduce:
ssh vmsqe-amd-06.russia.sun.com
cd /net/vmsqe.russia/export1/vmsqe/misc/sb196040/JMapPerm
bash run_test.sh
(test sources also in /net/vmsqe.russia/export1/vmsqe/misc/sb196040/JMapPerm)

Output example:
bash run_test.sh
Starting eating perm gen.
Executing /net/vmsqe.russia/export3/vmsqe/jdk-builds/JDK6b79/solaris-amd64//bin/jmap -d64 -dump:format=b,file=heap.dump 24321
Dumping heap to /set/vmsqe/misc/sb196040/JMapPerm/heap.dump ...
Heap dump file created
run_test.sh: line 15: 24321 Segmentation Fault (core dumped) ${JAVA} ${JAVA_OPTS} heapdump.share.EatMemory -permGen -exec "${JMAP} -dump:format=b,file=${DUMPFILE} %p"
JVM exit code: 139
Test
heapdump/OnOOMToPath
failed with fastdebug build because of segmentation fault. This test verifies -XX:+HeapDumpOnOutOfMemory flag, it runs java with -XX:+HeapDumpOnOutOfMemory flag, fills heap until OutOfMemoryError and checks heap dump.

I tried run java without '-XX:+HeapDumpOnOutOfMemory' but it still fails with segmentation fault(I can reproduce this failure only with fastdebug build), so this failure doesn't relate to heapdump. I am not sure is the reason of segmentation fault the same as in 'heapdump/JMapPerm', possible new CR should be opened or this CR should be moved to the another category?

To reproduce:
ssh vmsqe-amd-06.russia.sun.com
cd /net/vmsqe.russia/export4/results/nightly/2006-10-28_1/vm/64BITSOLARIS-AMD64/server/mixed/vm-vm_testbase_SPB_1.6.0-NIGHTLY_server_mixed_64BITSOLARIS-AMD642006-10-28-04-06-35/ResultDir/OnOOMToPath
sh rerun.sh
Move this CR to the 'runtime' category, as was suggetsed by Sundararajan.

Additional information:
VM doesn't create hs_err files, core file from first case is in '/net/vmsqe.russia/export1/vmsqe/misc/sb196040/JMapPerm' and core file for second case is in '/net/vmsqe.russia/export4/results/nightly/2006-10-28_1/vm/64BITSOLARIS-AMD64/server/mixed/vm-vm_testbase_SPB_1.6.0-NIGHTLY_server_mixed_64BITSOLARIS-AMD642006-10-28-04-06-35/ResultDir/OnOOMToPath'

'jstack -m' output for core file from first case:
Attaching to core core from executable /set/vmsqe/jdk-builds/latest/solaris-amd64/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0-rc-b103
Deadlock Detection:

No deadlocks found.

----------------- t@1 -----------------
0xfffffd7fff30ae1a umount2 + 0xa
0xfffffd7fff29c489 fdetach + 0x9
0xfffffd7ffeb4fb4a listener_cleanup + 0x5a
0xfffffd7fff29847b _exithandle + 0x3b
0xfffffd7fff289cf1 exit + 0x11
java.lang.NullPointerException
        at sun.jvm.hotspot.tools.PStack.printUnknown(PStack.java:207)
        at sun.jvm.hotspot.tools.PStack.run(PStack.java:130)
        at sun.jvm.hotspot.tools.PStack.run(PStack.java:36)
        at sun.jvm.hotspot.tools.PStack.run(PStack.java:31)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:43)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
        at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- t@3 -----------------
0xfffffd7fff30817b fix_reg + 0x4b
0xfffffd7fff309415 complete_context + 0x65
0xfffffd7fff3094bd down_one + 0x3d
0xfffffd7fff3096dc _Unwind_ForcedUnwind_Body + 0x3c
0xfffffd7fff309802 _SUNW_Unwind_ForcedUnwind + 0x52
0xfffffd7fff1798ff _ex_unwind + 0x6f
0xfffffd7fff3092cc _thrp_unwind + 0x3c
0xfffffd7fff3047cc _thr_exit_common + 0x9c
0xfffffd7fff30480e pthread_exit + 0xe
0xfffffd7ffee5182a java_start + 0x21a
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@4 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffee562cd void os::PlatformEvent::park() + 0x9d
0xfffffd7ffeeef81d void ObjectMonitor::wait(long long,bool,Thread*) + 0x3cd
0xfffffd7ffe9c6cee void ObjectSynchronizer::wait(Handle,long long,Thread*) + 0xde
0xfffffd7ffe9c7bdb JVM_MonitorWait + 0x21b
0xfffffd7ffb00f903 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xfffffd7ffb0030a2 * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xfffffd7ffb0030a2 * java.lang.ref.Reference$ReferenceHandler.run() bci:46 line:116 (Interpreted frame)
0xfffffd7ffb00032e <StubRoutines>
0xfffffd7ffe90499a void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x22a
0xfffffd7ffe904688 void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x18
0xfffffd7ffe9d3bac void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,symbolHandle,symbolHandle,Thread*) + 0x1bc
0xfffffd7ffe9edabb void thread_entry(JavaThread*,Thread*) + 0x10b
0xfffffd7ffe9e82ed void JavaThread::thread_main_inner() + 0x4d
0xfffffd7ffe9e7c12 void JavaThread::run() + 0x132
0xfffffd7ffee517dd java_start + 0x1cd
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@5 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffee562cd void os::PlatformEvent::park() + 0x9d
0xfffffd7ffeeef81d void ObjectMonitor::wait(long long,bool,Thread*) + 0x3cd
0xfffffd7ffe9c6cee void ObjectSynchronizer::wait(Handle,long long,Thread*) + 0xde
0xfffffd7ffe9c7bdb JVM_MonitorWait + 0x21b
0xfffffd7ffb00f903 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xfffffd7ffb0030a2 * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116 (Interpreted frame)
0xfffffd7ffb002f49 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132 (Interpreted frame)
0xfffffd7ffb002f49 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Interpreted frame)
0xfffffd7ffb00032e <StubRoutines>
0xfffffd7ffe90499a void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x22a
0xfffffd7ffe904688 void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x18
0xfffffd7ffe9d3bac void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,symbolHandle,symbolHandle,Thread*) + 0x1bc
0xfffffd7ffe9edabb void thread_entry(JavaThread*,Thread*) + 0x10b
0xfffffd7ffe9e82ed void JavaThread::thread_main_inner() + 0x4d
0xfffffd7ffe9e7c12 void JavaThread::run() + 0x132
0xfffffd7ffee517dd java_start + 0x1cd
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@7 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffee41b98 bool Monitor::wait(bool,long) + 0x2d8
0xfffffd7ffea46e1e void CompileBroker::compiler_thread_loop() + 0x2de
0xfffffd7ffea41d69 void compiler_thread_entry(JavaThread*,Thread*) + 0x9
0xfffffd7ffe9e82ed void JavaThread::thread_main_inner() + 0x4d
0xfffffd7ffe9e7c12 void JavaThread::run() + 0x132
0xfffffd7ffee517dd java_start + 0x1cd
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@8 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffee41b98 bool Monitor::wait(bool,long) + 0x2d8
0xfffffd7ffea46e1e void CompileBroker::compiler_thread_loop() + 0x2de
0xfffffd7ffea41d69 void compiler_thread_entry(JavaThread*,Thread*) + 0x9
0xfffffd7ffe9e82ed void JavaThread::thread_main_inner() + 0x4d
0xfffffd7ffe9e7c12 void JavaThread::run() + 0x132
0xfffffd7ffee517dd java_start + 0x1cd
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@9 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffee41d48 bool Monitor::wait(bool,long) + 0x488
0xfffffd7ffeaca0d1 void LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*,Thread*) + 0x1a1
0xfffffd7ffe9e82ed void JavaThread::thread_main_inner() + 0x4d
0xfffffd7ffe9e7c12 void JavaThread::run() + 0x132
0xfffffd7ffee517dd java_start + 0x1cd
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@12 -----------------
0xfffffd7fff307df7 __lwp_park + 0x17
0xfffffd7fff2fcc7b sema_wait + 0xb
0xfffffd7ffeb5042c AttachOperation*AttachListener::dequeue() + 0x12c
0xfffffd7ffeb4f2dd void attach_listener_thread_entry(JavaThread*,Thread*) + 0x1dd
0xfffffd7ffe9e82ed void JavaThread::thread_main_inner() + 0x4d
0xfffffd7ffe9e7c12 void JavaThread::run() + 0x132
0xfffffd7ffee517dd java_start + 0x1cd
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@13 -----------------
0xfffffd7fff30b188 _door_return + 0x18
0xfffffd7ffeb4fac7 enqueue_proc + 0x127
0xfffffd7fff30b215 _door_return + 0xa5
----------------- t@14 -----------------
0xfffffd7fff30b188 _door_return + 0x18
0xfffffd7fff14162d door_create_func + 0x1d
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start



'jstack -m' output for core file from second case:
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0-rc-fastdebug-b103-debug
Deadlock Detection:

No deadlocks found.

----------------- t@1 -----------------
0xfffffd7fff30abda __lwp_wait + 0xa
0xfffffd7fff30484e _thrp_join + 0x3e
0xfffffd7fff3049fe thr_join + 0xe
0x0000000000407745 ContinueInNewThread + 0x45
0x00000000004027fb main + 0x30b
java.lang.NullPointerException
        at sun.jvm.hotspot.tools.PStack.printUnknown(PStack.java:207)
        at sun.jvm.hotspot.tools.PStack.run(PStack.java:130)
        at sun.jvm.hotspot.tools.PStack.run(PStack.java:36)
        at sun.jvm.hotspot.tools.PStack.run(PStack.java:31)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:43)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
        at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- t@2 -----------------
0xfffffd7fff30a05a _close + 0xa
0xfffffd7ffe254603 __SLIP.DELETER__A + 0x23
0xfffffd7ffe25454a __SLIP.DELETER__D + 0xba
0xfffffd7ffe253d3c void ostream_exit() + 0x9c
0xfffffd7ffdb76444 void exit_globals() + 0x24
0xfffffd7ffe4e7b53 bool Threads::destroy_vm() + 0x183
0xfffffd7ffdd391af jni_DestroyJavaVM + 0x38f
0x0000000000402d37 JavaMain + 0x507
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@3 -----------------
0xfffffd7fff308196 fix_reg + 0x66
0xfffffd7fff309415 complete_context + 0x65
0xfffffd7fff3094bd down_one + 0x3d
0xfffffd7fff3096dc _Unwind_ForcedUnwind_Body + 0x3c
0xfffffd7fff309802 _SUNW_Unwind_ForcedUnwind + 0x52
0xfffffd7fff1798ff _ex_unwind + 0x6f
0xfffffd7fff3092cc _thrp_unwind + 0x3c
0xfffffd7fff3047cc _thr_exit_common + 0x9c
0xfffffd7fff30480e pthread_exit + 0xe
0xfffffd7ffe243748 java_start + 0x288
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@4 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffe24f9fb void os::PlatformEvent::park() + 0xcb
0xfffffd7ffe4535a0 void ObjectMonitor::wait(long long,bool,Thread*) + 0x570
0xfffffd7ffe44a89d void ObjectSynchronizer::wait(Handle,long long,Thread*) + 0x30d
0xfffffd7ffdd9b32b JVM_MonitorWait + 0x8db
0xfffffd7ffa0210f6 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xfffffd7ffa0030a2 * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xfffffd7ffa0030a2 * java.lang.ref.Reference$ReferenceHandler.run() bci:46 line:116 (Interpreted frame)
0xfffffd7ffa0004af <StubRoutines>
0xfffffd7ffdc4f18d void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x7ad
0xfffffd7ffe24ba98 void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x18
0xfffffd7ffdc4d3b2 void JavaCalls::call_virtual(JavaValue*,KlassHandle,symbolHandle,symbolHandle,JavaCallArguments*,Thread*) + 0x322
0xfffffd7ffdc4d614 void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,symbolHandle,symbolHandle,Thread*) + 0x104
0xfffffd7ffddffb33 void thread_entry(JavaThread*,Thread*) + 0x273
0xfffffd7ffe4e06a9 void JavaThread::thread_main_inner() + 0xe9
0xfffffd7ffe4e052b void JavaThread::run() + 0x20b
0xfffffd7ffe2436ee java_start + 0x22e
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@5 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffe24f9fb void os::PlatformEvent::park() + 0xcb
0xfffffd7ffe4535a0 void ObjectMonitor::wait(long long,bool,Thread*) + 0x570
0xfffffd7ffe44a89d void ObjectSynchronizer::wait(Handle,long long,Thread*) + 0x30d
0xfffffd7ffdd9b32b JVM_MonitorWait + 0x8db
0xfffffd7ffa0210f6 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xfffffd7ffa0030a2 * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116 (Interpreted frame)
0xfffffd7ffa002f49 * java.lang.ref.ReferenceQueue.remove() bci:2 line:132 (Interpreted frame)
0xfffffd7ffa002f49 * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Interpreted frame)
0xfffffd7ffa0004af <StubRoutines>
0xfffffd7ffdc4f18d void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x7ad
0xfffffd7ffe24ba98 void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x18
0xfffffd7ffdc4d3b2 void JavaCalls::call_virtual(JavaValue*,KlassHandle,symbolHandle,symbolHandle,JavaCallArguments*,Thread*) + 0x322
0xfffffd7ffdc4d614 void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,symbolHandle,symbolHandle,Thread*) + 0x104
0xfffffd7ffddffb33 void thread_entry(JavaThread*,Thread*) + 0x273
0xfffffd7ffe4e06a9 void JavaThread::thread_main_inner() + 0xe9
0xfffffd7ffe4e052b void JavaThread::run() + 0x20b
0xfffffd7ffe2436ee java_start + 0x22e
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@7 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffe1eccc9 bool Monitor::wait(bool,long) + 0x549
0xfffffd7ffd9bd0d2 void CompileBroker::compiler_thread_loop() + 0x7f2
0xfffffd7ffe4e5ea6 void compiler_thread_entry(JavaThread*,Thread*) + 0x16
0xfffffd7ffe4e06a9 void JavaThread::thread_main_inner() + 0xe9
0xfffffd7ffe4e052b void JavaThread::run() + 0x20b
0xfffffd7ffe2436ee java_start + 0x22e
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@8 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffe1eccc9 bool Monitor::wait(bool,long) + 0x549
0xfffffd7ffd9bd0d2 void CompileBroker::compiler_thread_loop() + 0x7f2
0xfffffd7ffe4e5ea6 void compiler_thread_entry(JavaThread*,Thread*) + 0x16
0xfffffd7ffe4e06a9 void JavaThread::thread_main_inner() + 0xe9
0xfffffd7ffe4e052b void JavaThread::run() + 0x20b
0xfffffd7ffe2436ee java_start + 0x22e
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
----------------- t@9 -----------------
0xfffffd7fff30ac7a ___lwp_cond_wait + 0xa
0xfffffd7ffe1ecf0f bool Monitor::wait(bool,long) + 0x78f
0xfffffd7ffe134977 void LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*,Thread*) + 0x387
0xfffffd7ffe4e06a9 void JavaThread::thread_main_inner() + 0xe9
0xfffffd7ffe4e052b void JavaThread::run() + 0x20b
0xfffffd7ffe2436ee java_start + 0x22e
0xfffffd7fff307b4b _thr_setup + 0x5b
0xfffffd7fff307d80 _lwp_start
Another test fails in similar way with fastdebug VM on the same machine(vmsqe-amd-06.russia.sun.com).
Test name:
runtime/jbe/strengthreduc/sreduc01

To reproduce:
ssh vmsqe-amd-06.russia.sun.com
cd /net/vmsqe.russia/export4/results/nightly/2006-11-04_1/vm/64BITSOLARIS-AMD64/server/int/vm-vm_testbase_SPB_1.6.0-NIGHTLY_server_int_64BITSOLARIS-AMD642006-11-04-04-04-09
sh sreduc01.tlog

Output example:
-bash-3.00$ sh sreduc01.tlog
Unoptimized method results
piece 0 at 1
piece 14 at 354
piece 12 at 330
piece 4 at 291
piece 24 at 278
piece 22 at 276
piece 8 at 275
piece 0 at 267
piece 0 at 219
piece 4 at 203
piece 0 at 202
piece 0 at 154
piece 16 at 138
piece 2 at 110
piece 2 at 108
piece 0 at 106
piece 4 at 90
Hand-optimized method results
piece 0 at 1
piece 14 at 354
piece 12 at 330
piece 4 at 291
piece 24 at 278
piece 22 at 276
piece 8 at 275
piece 0 at 267
piece 0 at 219
piece 4 at 203
piece 0 at 202
piece 0 at 154
piece 16 at 138
piece 2 at 110
piece 2 at 108
piece 0 at 106
piece 4 at 90
Segmentation Fault - core dumped