-
Bug
-
Resolution: Fixed
-
P4
-
21
-
b16
I came across an intermittent issue where the JVM is crashing when dumping dynamic archive.
Analyzing the logs and core dump indicate there is a race condition between the Signal handler thread and the DestroyJavaVM thread for global data structures in ClassPrelinker.
Thread stack of the DestroyJavaVM thread at the time of crash:
Thread 1 (Thread 0x7f5ca7f68700 (LWP 1878717)):
<signal handling frames are removed>
#7 <signal handler called>
#8 ResourceHashtableBase<FixedResourceHashtableStorage<15889u, InstanceKlass*, bool>, InstanceKlass*, bool, (AnyObj::allocation_type)2, (MEMFLAGS)13, &(unsigned int primitive_hash<InstanceKlass*>(InstanceKlass* const&)), &(bool primitive_equals<InstanceKlass*>(InstanceKlass* const&, InstanceKlass* const&))>::lookup_node (key=<synthetic pointer>: <optimized out>, hash=18926782, this=0x0)
at src/hotspot/share/utilities/resourceHash.hpp:83
#9 ResourceHashtableBase<FixedResourceHashtableStorage<15889u, InstanceKlass*, bool>, InstanceKlass*, bool, (AnyObj::allocation_type)2, (MEMFLAGS)13, &(unsigned int primitive_hash<InstanceKlass*>(InstanceKlass* const&)), &(bool primitive_equals<InstanceKlass*>(InstanceKlass* const&, InstanceKlass* const&))>::put_if_absent (p_created=<synthetic pointer>, key=<synthetic pointer>: <optimized out>, this=0x0)
at src/hotspot/share/utilities/resourceHash.hpp:168
#10 ClassPrelinker::dumptime_resolve_constants (ik=ik@entry=0x80100d670, __the_thread__=__the_thread__@entry=0x7f5ca0029f10) at src/hotspot/share/cds/classPrelinker.cpp:125
#11 0x00007f5ca6708b14 in MetaspaceShared::link_class_for_cds (__the_thread__=0x7f5ca0029f10, ik=0x80100d670) at src/hotspot/share/cds/metaspaceShared.cpp:621
#12 MetaspaceShared::link_shared_classes (jcmd_request=jcmd_request@entry=false, __the_thread__=__the_thread__@entry=0x7f5ca0029f10) at src/hotspot/share/cds/metaspaceShared.cpp:651
#13 0x00007f5ca625f49a in DynamicArchive::prepare_for_dump_at_exit () at src/hotspot/share/cds/dynamicArchive.cpp:383
#14 0x00007f5ca63fd6cd in JavaThread::invoke_shutdown_hooks (this=this@entry=0x7f5ca0029f10) at src/hotspot/share/runtime/javaThread.cpp:2038
#15 0x00007f5ca6a2778d in Threads::destroy_vm () at src/hotspot/share/runtime/threads.cpp:1081
#16 0x00007f5ca647af3b in jni_DestroyJavaVM_inner (vm=<optimized out>) at src/hotspot/share/prims/jni.cpp:3735
#17 jni_DestroyJavaVM (vm=<optimized out>) at src/hotspot/share/prims/jni.cpp:3747
#18 0x00007f5ca792eb44 in JavaMain (_args=<optimized out>) at src/java.base/share/native/libjli/java.c:555
#19 0x00007f5ca7932af9 in ThreadJavaMain (args=<optimized out>) at src/java.base/unix/native/libjli/java_md.c:650
#20 0x00007f5ca77131cf in start_thread () from /lib64/libpthread.so.0
#21 0x00007f5ca717bdd3 in clone () from /lib64/libc.so.6
Thread stack of the signal handler thread at the time of crash:
Thread 2 (Thread 0x7f5c4e420700 (LWP 1879009)):
#0 0x00007f5ca771ca07 in write () from /lib64/libpthread.so.0
#1 0x00007f5ca6799a38 in os::write (fd=1, buf=0x7f5ca017af90, nBytes=2) at src/hotspot/os/posix/os_posix.cpp:776
#2 0x00007f5ca679f16e in defaultStream::write (this=0x7f5ca0000b70, s=0x7f5ca017af90 "of", len=2) at src/hotspot/share/utilities/ostream.cpp:895
#3 0x00007f5ca679ca02 in outputStream::do_vsnprintf_and_write_with_automatic_buffer (this=this@entry=0x7f5ca0000b70, format=<optimized out>, ap=ap@entry=0x7f5c4e41f2e8, add_cr=add_cr@entry=false)
at src/hotspot/share/utilities/ostream.cpp:129
#4 0x00007f5ca679cfe9 in outputStream::do_vsnprintf_and_write (add_cr=false, ap=0x7f5c4e41f2e8, format=0x7f5ca6b7e5e6 "%s", this=0x7f5ca0000b70) at src/hotspot/share/utilities/ostream.cpp:142
#5 outputStream::print (this=this@entry=0x7f5ca0000b70, format=format@entry=0x7f5ca6b7e5e6 "%s") at src/hotspot/share/utilities/ostream.cpp:149
#6 0x00007f5ca69c26e7 in Symbol::print_symbol_on (this=0x800493cd0, st=0x7f5ca0000b70) at src/hotspot/share/oops/symbol.cpp:191
#7 0x00007f5ca671251e in Method::print_invocation_count (this=this@entry=0x7f5bfc315e80) at src/hotspot/share/oops/constantPool.hpp:457
#8 0x00007f5ca63e2d15 in print_method_profiling_data () at src/hotspot/share/runtime/java.cpp:132
#9 print_method_profiling_data () at src/hotspot/share/runtime/java.cpp:117
#10 0x00007f5ca63e301d in print_statistics () at src/hotspot/share/runtime/java.cpp:342
#11 0x00007f5ca63e4d39 in before_exit (thread=thread@entry=0x7f5c40000fe0, halt=halt@entry=true) at src/hotspot/share/runtime/java.cpp:532
#12 0x00007f5ca64b008a in JVM_Halt (code=143) at src/hotspot/share/prims/jvm.cpp:444
#13 0x00007f5c901a1ca0 in ?? ()
#14 0x00007f5c901a193f in ?? ()
#15 0x00007f5c4e41f728 in ?? ()
#16 0x000000080067db80 in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x000000080003d600 in ?? ()
#19 0x0000000000000000 in ?? ()
Race condition:
There are two exit paths in the JVM:
1. When the JVM falls off the end of main() method, the main thread (in java launcher) attaches itself as
DestroyJavaVM thread and initiates the shutdown sequence by calling destroy_vm(). DestroyJavaVM waits for all
non-daemon threads to stop before starting the shutdown sequence.
2. When the application is sent a signal like SIGTERM, the signal handler thread starts the shutdown sequence
by invoking Shutdown.exit() which in turns calls JVM_BeforeHalt(), runs shutdown hooks and calls JVM_Halt().
It is possible that the application has registered a shutdown hook that results in normal termination of main method().
In such case path 2 can trigger path 1 as well.
In both shutdown sequences JVM attempts to dump dynamic archive by calling DynamicArchive::prepare_for_dump_at_exit() followed by DynamicArchive::dump().
DynamicArchive::prepare_for_dump_at_exit() initializes ClassPrelinker::_processed_classes and DynamicArchive::dump() frees the field and sets it to NULL.
If the two sequences run together, as it happened in this case, it is possible that after DestroyJavaVM thread has initialized ClassPrelinker::_processed_classes (as part of DynamicArchive::prepare_for_dump_at_exit()), the signal handler thread frees it in DynamicArchive::dump() which which will cause DestroyJavaVM to access invalid memory when it subsequently
uses ClassPrelinker::_processed_classes.
One of the ways to fix it is to prohibit DestroyJavaVM from executing DynamicArchive::prepare_for_dump_at_exit() if the signal handler thread has completed DynamicArchive::dump().
Analyzing the logs and core dump indicate there is a race condition between the Signal handler thread and the DestroyJavaVM thread for global data structures in ClassPrelinker.
Thread stack of the DestroyJavaVM thread at the time of crash:
Thread 1 (Thread 0x7f5ca7f68700 (LWP 1878717)):
<signal handling frames are removed>
#7 <signal handler called>
#8 ResourceHashtableBase<FixedResourceHashtableStorage<15889u, InstanceKlass*, bool>, InstanceKlass*, bool, (AnyObj::allocation_type)2, (MEMFLAGS)13, &(unsigned int primitive_hash<InstanceKlass*>(InstanceKlass* const&)), &(bool primitive_equals<InstanceKlass*>(InstanceKlass* const&, InstanceKlass* const&))>::lookup_node (key=<synthetic pointer>: <optimized out>, hash=18926782, this=0x0)
at src/hotspot/share/utilities/resourceHash.hpp:83
#9 ResourceHashtableBase<FixedResourceHashtableStorage<15889u, InstanceKlass*, bool>, InstanceKlass*, bool, (AnyObj::allocation_type)2, (MEMFLAGS)13, &(unsigned int primitive_hash<InstanceKlass*>(InstanceKlass* const&)), &(bool primitive_equals<InstanceKlass*>(InstanceKlass* const&, InstanceKlass* const&))>::put_if_absent (p_created=<synthetic pointer>, key=<synthetic pointer>: <optimized out>, this=0x0)
at src/hotspot/share/utilities/resourceHash.hpp:168
#10 ClassPrelinker::dumptime_resolve_constants (ik=ik@entry=0x80100d670, __the_thread__=__the_thread__@entry=0x7f5ca0029f10) at src/hotspot/share/cds/classPrelinker.cpp:125
#11 0x00007f5ca6708b14 in MetaspaceShared::link_class_for_cds (__the_thread__=0x7f5ca0029f10, ik=0x80100d670) at src/hotspot/share/cds/metaspaceShared.cpp:621
#12 MetaspaceShared::link_shared_classes (jcmd_request=jcmd_request@entry=false, __the_thread__=__the_thread__@entry=0x7f5ca0029f10) at src/hotspot/share/cds/metaspaceShared.cpp:651
#13 0x00007f5ca625f49a in DynamicArchive::prepare_for_dump_at_exit () at src/hotspot/share/cds/dynamicArchive.cpp:383
#14 0x00007f5ca63fd6cd in JavaThread::invoke_shutdown_hooks (this=this@entry=0x7f5ca0029f10) at src/hotspot/share/runtime/javaThread.cpp:2038
#15 0x00007f5ca6a2778d in Threads::destroy_vm () at src/hotspot/share/runtime/threads.cpp:1081
#16 0x00007f5ca647af3b in jni_DestroyJavaVM_inner (vm=<optimized out>) at src/hotspot/share/prims/jni.cpp:3735
#17 jni_DestroyJavaVM (vm=<optimized out>) at src/hotspot/share/prims/jni.cpp:3747
#18 0x00007f5ca792eb44 in JavaMain (_args=<optimized out>) at src/java.base/share/native/libjli/java.c:555
#19 0x00007f5ca7932af9 in ThreadJavaMain (args=<optimized out>) at src/java.base/unix/native/libjli/java_md.c:650
#20 0x00007f5ca77131cf in start_thread () from /lib64/libpthread.so.0
#21 0x00007f5ca717bdd3 in clone () from /lib64/libc.so.6
Thread stack of the signal handler thread at the time of crash:
Thread 2 (Thread 0x7f5c4e420700 (LWP 1879009)):
#0 0x00007f5ca771ca07 in write () from /lib64/libpthread.so.0
#1 0x00007f5ca6799a38 in os::write (fd=1, buf=0x7f5ca017af90, nBytes=2) at src/hotspot/os/posix/os_posix.cpp:776
#2 0x00007f5ca679f16e in defaultStream::write (this=0x7f5ca0000b70, s=0x7f5ca017af90 "of", len=2) at src/hotspot/share/utilities/ostream.cpp:895
#3 0x00007f5ca679ca02 in outputStream::do_vsnprintf_and_write_with_automatic_buffer (this=this@entry=0x7f5ca0000b70, format=<optimized out>, ap=ap@entry=0x7f5c4e41f2e8, add_cr=add_cr@entry=false)
at src/hotspot/share/utilities/ostream.cpp:129
#4 0x00007f5ca679cfe9 in outputStream::do_vsnprintf_and_write (add_cr=false, ap=0x7f5c4e41f2e8, format=0x7f5ca6b7e5e6 "%s", this=0x7f5ca0000b70) at src/hotspot/share/utilities/ostream.cpp:142
#5 outputStream::print (this=this@entry=0x7f5ca0000b70, format=format@entry=0x7f5ca6b7e5e6 "%s") at src/hotspot/share/utilities/ostream.cpp:149
#6 0x00007f5ca69c26e7 in Symbol::print_symbol_on (this=0x800493cd0, st=0x7f5ca0000b70) at src/hotspot/share/oops/symbol.cpp:191
#7 0x00007f5ca671251e in Method::print_invocation_count (this=this@entry=0x7f5bfc315e80) at src/hotspot/share/oops/constantPool.hpp:457
#8 0x00007f5ca63e2d15 in print_method_profiling_data () at src/hotspot/share/runtime/java.cpp:132
#9 print_method_profiling_data () at src/hotspot/share/runtime/java.cpp:117
#10 0x00007f5ca63e301d in print_statistics () at src/hotspot/share/runtime/java.cpp:342
#11 0x00007f5ca63e4d39 in before_exit (thread=thread@entry=0x7f5c40000fe0, halt=halt@entry=true) at src/hotspot/share/runtime/java.cpp:532
#12 0x00007f5ca64b008a in JVM_Halt (code=143) at src/hotspot/share/prims/jvm.cpp:444
#13 0x00007f5c901a1ca0 in ?? ()
#14 0x00007f5c901a193f in ?? ()
#15 0x00007f5c4e41f728 in ?? ()
#16 0x000000080067db80 in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x000000080003d600 in ?? ()
#19 0x0000000000000000 in ?? ()
Race condition:
There are two exit paths in the JVM:
1. When the JVM falls off the end of main() method, the main thread (in java launcher) attaches itself as
DestroyJavaVM thread and initiates the shutdown sequence by calling destroy_vm(). DestroyJavaVM waits for all
non-daemon threads to stop before starting the shutdown sequence.
2. When the application is sent a signal like SIGTERM, the signal handler thread starts the shutdown sequence
by invoking Shutdown.exit() which in turns calls JVM_BeforeHalt(), runs shutdown hooks and calls JVM_Halt().
It is possible that the application has registered a shutdown hook that results in normal termination of main method().
In such case path 2 can trigger path 1 as well.
In both shutdown sequences JVM attempts to dump dynamic archive by calling DynamicArchive::prepare_for_dump_at_exit() followed by DynamicArchive::dump().
DynamicArchive::prepare_for_dump_at_exit() initializes ClassPrelinker::_processed_classes and DynamicArchive::dump() frees the field and sets it to NULL.
If the two sequences run together, as it happened in this case, it is possible that after DestroyJavaVM thread has initialized ClassPrelinker::_processed_classes (as part of DynamicArchive::prepare_for_dump_at_exit()), the signal handler thread frees it in DynamicArchive::dump() which which will cause DestroyJavaVM to access invalid memory when it subsequently
uses ClassPrelinker::_processed_classes.
One of the ways to fix it is to prohibit DestroyJavaVM from executing DynamicArchive::prepare_for_dump_at_exit() if the signal handler thread has completed DynamicArchive::dump().
- relates to
-
JDK-8304996 Add missing HandleMarks
- Resolved
-
JDK-8304381 Investigate the concurrent execution of DynamicArchive dumping
- Closed
-
JDK-8303422 Use common functions to exit the VM for -Xshare:dump and CDS errors
- Resolved