Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8304147

JVM crash during shutdown when dumping dynamic archive

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: P4 P4
    • 21
    • 21
    • hotspot
    • b16

      I came across an intermittent issue where the JVM is crashing when dumping dynamic archive.
      Analyzing the logs and core dump indicate there is a race condition between the Signal handler thread and the DestroyJavaVM thread for global data structures in ClassPrelinker.

      Thread stack of the DestroyJavaVM thread at the time of crash:

      Thread 1 (Thread 0x7f5ca7f68700 (LWP 1878717)):

      <signal handling frames are removed>
      #7 <signal handler called>
      #8 ResourceHashtableBase<FixedResourceHashtableStorage<15889u, InstanceKlass*, bool>, InstanceKlass*, bool, (AnyObj::allocation_type)2, (MEMFLAGS)13, &(unsigned int primitive_hash<InstanceKlass*>(InstanceKlass* const&)), &(bool primitive_equals<InstanceKlass*>(InstanceKlass* const&, InstanceKlass* const&))>::lookup_node (key=<synthetic pointer>: <optimized out>, hash=18926782, this=0x0)
          at src/hotspot/share/utilities/resourceHash.hpp:83
      #9 ResourceHashtableBase<FixedResourceHashtableStorage<15889u, InstanceKlass*, bool>, InstanceKlass*, bool, (AnyObj::allocation_type)2, (MEMFLAGS)13, &(unsigned int primitive_hash<InstanceKlass*>(InstanceKlass* const&)), &(bool primitive_equals<InstanceKlass*>(InstanceKlass* const&, InstanceKlass* const&))>::put_if_absent (p_created=<synthetic pointer>, key=<synthetic pointer>: <optimized out>, this=0x0)
          at src/hotspot/share/utilities/resourceHash.hpp:168
      #10 ClassPrelinker::dumptime_resolve_constants (ik=ik@entry=0x80100d670, __the_thread__=__the_thread__@entry=0x7f5ca0029f10) at src/hotspot/share/cds/classPrelinker.cpp:125
      #11 0x00007f5ca6708b14 in MetaspaceShared::link_class_for_cds (__the_thread__=0x7f5ca0029f10, ik=0x80100d670) at src/hotspot/share/cds/metaspaceShared.cpp:621
      #12 MetaspaceShared::link_shared_classes (jcmd_request=jcmd_request@entry=false, __the_thread__=__the_thread__@entry=0x7f5ca0029f10) at src/hotspot/share/cds/metaspaceShared.cpp:651
      #13 0x00007f5ca625f49a in DynamicArchive::prepare_for_dump_at_exit () at src/hotspot/share/cds/dynamicArchive.cpp:383
      #14 0x00007f5ca63fd6cd in JavaThread::invoke_shutdown_hooks (this=this@entry=0x7f5ca0029f10) at src/hotspot/share/runtime/javaThread.cpp:2038
      #15 0x00007f5ca6a2778d in Threads::destroy_vm () at src/hotspot/share/runtime/threads.cpp:1081
      #16 0x00007f5ca647af3b in jni_DestroyJavaVM_inner (vm=<optimized out>) at src/hotspot/share/prims/jni.cpp:3735
      #17 jni_DestroyJavaVM (vm=<optimized out>) at src/hotspot/share/prims/jni.cpp:3747
      #18 0x00007f5ca792eb44 in JavaMain (_args=<optimized out>) at src/java.base/share/native/libjli/java.c:555
      #19 0x00007f5ca7932af9 in ThreadJavaMain (args=<optimized out>) at src/java.base/unix/native/libjli/java_md.c:650
      #20 0x00007f5ca77131cf in start_thread () from /lib64/libpthread.so.0
      #21 0x00007f5ca717bdd3 in clone () from /lib64/libc.so.6

      Thread stack of the signal handler thread at the time of crash:

      Thread 2 (Thread 0x7f5c4e420700 (LWP 1879009)):
      #0 0x00007f5ca771ca07 in write () from /lib64/libpthread.so.0
      #1 0x00007f5ca6799a38 in os::write (fd=1, buf=0x7f5ca017af90, nBytes=2) at src/hotspot/os/posix/os_posix.cpp:776
      #2 0x00007f5ca679f16e in defaultStream::write (this=0x7f5ca0000b70, s=0x7f5ca017af90 "of", len=2) at src/hotspot/share/utilities/ostream.cpp:895
      #3 0x00007f5ca679ca02 in outputStream::do_vsnprintf_and_write_with_automatic_buffer (this=this@entry=0x7f5ca0000b70, format=<optimized out>, ap=ap@entry=0x7f5c4e41f2e8, add_cr=add_cr@entry=false)
          at src/hotspot/share/utilities/ostream.cpp:129
      #4 0x00007f5ca679cfe9 in outputStream::do_vsnprintf_and_write (add_cr=false, ap=0x7f5c4e41f2e8, format=0x7f5ca6b7e5e6 "%s", this=0x7f5ca0000b70) at src/hotspot/share/utilities/ostream.cpp:142
      #5 outputStream::print (this=this@entry=0x7f5ca0000b70, format=format@entry=0x7f5ca6b7e5e6 "%s") at src/hotspot/share/utilities/ostream.cpp:149
      #6 0x00007f5ca69c26e7 in Symbol::print_symbol_on (this=0x800493cd0, st=0x7f5ca0000b70) at src/hotspot/share/oops/symbol.cpp:191
      #7 0x00007f5ca671251e in Method::print_invocation_count (this=this@entry=0x7f5bfc315e80) at src/hotspot/share/oops/constantPool.hpp:457
      #8 0x00007f5ca63e2d15 in print_method_profiling_data () at src/hotspot/share/runtime/java.cpp:132
      #9 print_method_profiling_data () at src/hotspot/share/runtime/java.cpp:117
      #10 0x00007f5ca63e301d in print_statistics () at src/hotspot/share/runtime/java.cpp:342
      #11 0x00007f5ca63e4d39 in before_exit (thread=thread@entry=0x7f5c40000fe0, halt=halt@entry=true) at src/hotspot/share/runtime/java.cpp:532
      #12 0x00007f5ca64b008a in JVM_Halt (code=143) at src/hotspot/share/prims/jvm.cpp:444
      #13 0x00007f5c901a1ca0 in ?? ()
      #14 0x00007f5c901a193f in ?? ()
      #15 0x00007f5c4e41f728 in ?? ()
      #16 0x000000080067db80 in ?? ()
      #17 0x0000000000000004 in ?? ()
      #18 0x000000080003d600 in ?? ()
      #19 0x0000000000000000 in ?? ()

      Race condition:

      There are two exit paths in the JVM:
      1. When the JVM falls off the end of main() method, the main thread (in java launcher) attaches itself as
      DestroyJavaVM thread and initiates the shutdown sequence by calling destroy_vm(). DestroyJavaVM waits for all
      non-daemon threads to stop before starting the shutdown sequence.
      2. When the application is sent a signal like SIGTERM, the signal handler thread starts the shutdown sequence
      by invoking Shutdown.exit() which in turns calls JVM_BeforeHalt(), runs shutdown hooks and calls JVM_Halt().

      It is possible that the application has registered a shutdown hook that results in normal termination of main method().
      In such case path 2 can trigger path 1 as well.

      In both shutdown sequences JVM attempts to dump dynamic archive by calling DynamicArchive::prepare_for_dump_at_exit() followed by DynamicArchive::dump().
      DynamicArchive::prepare_for_dump_at_exit() initializes ClassPrelinker::_processed_classes and DynamicArchive::dump() frees the field and sets it to NULL.

      If the two sequences run together, as it happened in this case, it is possible that after DestroyJavaVM thread has initialized ClassPrelinker::_processed_classes (as part of DynamicArchive::prepare_for_dump_at_exit()), the signal handler thread frees it in DynamicArchive::dump() which which will cause DestroyJavaVM to access invalid memory when it subsequently
      uses ClassPrelinker::_processed_classes.

      One of the ways to fix it is to prohibit DestroyJavaVM from executing DynamicArchive::prepare_for_dump_at_exit() if the signal handler thread has completed DynamicArchive::dump().

            dholmes David Holmes
            asmehra Ashutosh Mehra
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: