Genshen: crash at ShenandoahScanRemembered::process_clusters after JDK-8371667

XMLWordPrintable

    • gc
    • 26
    • b27

      Noticed the crash is happening very often when I run
      specjbb2015 bennchmark(5 out of 6 test runs) after merging the latest the change from tip, I have reproduced the issue on TIP. After checking and testing the changes introduced to Shenandoah recently and the line of code where the crashes were from, I found that it is caused by JDK-8371667.

      ```
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x0000ffffac294540, pid=40921, tid=40923
      #
      # JRE version: OpenJDK Runtime Environment (26.0) (build 26-internal-adhoc.ec2user.jdk)
      # Java VM: OpenJDK 64-Bit Server VM (26-internal-adhoc.ec2user.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-aarch64)
      # Problematic frame:
      # V [libjvm.so+0xe94540] void ShenandoahScanRemembered::process_clusters<ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2> >(unsigned long, unsigned long, HeapWordImpl**, ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2>*, bool, unsigned int)+0x4a0
      #
      # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/ec2-user/tools/specjbb2015/core.40921)
      #
      # If you would like to submit a bug report, please visit:
      # https://bugreport.java.com/bugreport/crash.jsp
      #

      --------------- S U M M A R Y ------------

      Command Line: -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+AlwaysPreTouch -Xms31g -Xmx31g -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:async -Xlog:gc*:file=./logs/gc/1/specjbb2015-genshen-master-gc-31g.log ./specjbb2015.jar -m composite -ikv -p ./config/specjbb2015.props -raw ./config/template-C.raw

      Host: AArch64, 16 cores, 123G, Amazon Linux release 2023.9.20251117 (Amazon Linux)
      Time: Wed Nov 26 06:30:04 2025 UTC elapsed time: 2026.242452 seconds (0d 0h 33m 46s)

      --------------- T H R E A D ---------------

      Current thread (0x0000ffffa411ec00): WorkerThread "Shenandoah GC Threads#0" [id=40923, stack(0x0000ffffa9008000,0x0000ffffa9206000) (2040K)]

      Stack: [0x0000ffffa9008000,0x0000ffffa9206000], sp=0x0000ffffa9204510, free space=2033k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0xe94540] void ShenandoahScanRemembered::process_clusters<ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2> >(unsigned long, unsigned long, HeapWordImpl**, ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2>*, bool, unsigned int)+0x4a0 (shenandoahScanRemembered.inline.hpp:200)
      V [libjvm.so+0xe90d30] ShenandoahScanRememberedTask::do_work(unsigned int)+0x4e8 (shenandoahScanRemembered.inline.hpp:377)
      V [libjvm.so+0xe90d98] ShenandoahScanRememberedTask::work(unsigned int)+0x58 (shenandoahScanRemembered.cpp:774)
      V [libjvm.so+0x106bc18] WorkerThread::run()+0x98 (workerThread.cpp:69)
      V [libjvm.so+0xfa9e14] Thread::call_run()+0xa8 (thread.cpp:242)
      V [libjvm.so+0xd13ae4] thread_native_entry(Thread*)+0xd8 (os_linux.cpp:862)
      C [libc.so.6+0x8bb78] start_thread+0x2d8

      ```

       crash log from fastdebug build:
      ```
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # Internal Error (/home/ec2-user/repos/jdk/src/hotspot/share/runtime/mutexLocker.cpp:177), pid=3534433, tid=3534468
      # fatal error: must own lock ClassLoaderDataGraph_lock
      #
      # JRE version: OpenJDK Runtime Environment (26.0) (fastdebug build 26-internal-adhoc.ec2user.jdk)
      # Java VM: OpenJDK 64-Bit Server VM (fastdebug 26-internal-adhoc.ec2user.jdk, mixed mode, sharing, tiered, compressed class ptrs, shenandoah gc, linux-aarch64)
      # Problematic frame:
      # V [libjvm.so+0x15094c8] assert_locked_or_safepoint(Mutex const*)+0xa8
      #
      # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /data/tools/specjbb2015/core.3534433)
      #
      # If you would like to submit a bug report, please visit:
      # https://bugreport.java.com/bugreport/crash.jsp
      #

      --------------- S U M M A R Y ------------

      Command Line: -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+AlwaysPreTouch -Xms512g -Xmx512g -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:async -Xlog:gc*:file=./logs/gc/test/specjbb2015-genshen-cas-alloc-refactor-gc-crash-512g.log ./specjbb2015.jar -m composite -ikv -p ./config/specjbb2015.props -raw ./config/template-C.raw

      Host: ip-172-31-42-91.ec2.internal, AArch64, 96 cores, 753G, Amazon Linux release 2023.9.20251027 (Amazon Linux)
      Time: Wed Nov 26 08:08:21 2025 UTC elapsed time: 2307.589793 seconds (0d 0h 38m 27s)

      --------------- T H R E A D ---------------

      Current thread (0x0000ffffac1755d0): WorkerThread "Shenandoah GC Threads#33" [id=3534468, stack(0x0000ff7e45f30000,0x0000ff7e4612e000) (2040K)]

      Stack: [0x0000ff7e45f30000,0x0000ff7e4612e000], sp=0x0000ff7e4612c2f0, free space=2032k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x15094c8] assert_locked_or_safepoint(Mutex const*)+0xa8 (mutexLocker.cpp:177)
      V [libjvm.so+0x9cf92c] ClassLoaderDataGraph::is_valid(ClassLoaderData*)+0x20 (classLoaderDataGraph.cpp:394)
      V [libjvm.so+0x18cb5e4] ShenandoahCardCluster::first_object_start(unsigned long, ShenandoahMarkingContext const*, HeapWordImpl**, HeapWordImpl**) const+0xa44 (shenandoahScanRemembered.cpp:389)
      V [libjvm.so+0x180f624] void ShenandoahScanRemembered::process_clusters<ShenandoahConcUpdateRefsClosure>(unsigned long, unsigned long, HeapWordImpl**, ShenandoahConcUpdateRefsClosure*, bool, unsigned int)+0x504 (shenandoahScanRemembered.inline.hpp:181)
      V [libjvm.so+0x1810dd4] void ShenandoahGenerationalUpdateHeapRefsTask<true>::update_references_in_remembered_set<ShenandoahConcUpdateRefsClosure>(unsigned int, ShenandoahConcUpdateRefsClosure&, ShenandoahMarkingContext const*, bool)+0x6d4 (shenandoahScanRemembered.inline.hpp:377)
      V [libjvm.so+0x1811440] void ShenandoahGenerationalUpdateHeapRefsTask<true>::do_work<ShenandoahConcUpdateRefsClosure>(unsigned int)+0x380 (shenandoahGenerationalHeap.cpp:846)
      V [libjvm.so+0x18115b4] ShenandoahGenerationalUpdateHeapRefsTask<true>::work(unsigned int)+0x74 (shenandoahGenerationalHeap.cpp:771)
      V [libjvm.so+0x1bea15c] WorkerThread::run()+0x98 (workerThread.cpp:69)
      V [libjvm.so+0x1a93710] Thread::call_run()+0xb0 (thread.cpp:242)
      V [libjvm.so+0x15b5e20] thread_native_entry(Thread*)+0x134 (os_linux.cpp:862)
      C [libc.so.6+0x8bb78] start_thread+0x2d8

      ```

            Assignee:
            Xiaolong Peng
            Reporter:
            Xiaolong Peng
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: