Noticed the crash is happening very often when I run
specjbb2015 bennchmark(5 out of 6 test runs) after merging the latest the change from tip, I have reproduced the issue on TIP. After checking and testing the changes introduced to Shenandoah recently and the line of code where the crashes were from, I found that it is caused byJDK-8371667.
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000ffffac294540, pid=40921, tid=40923
#
# JRE version: OpenJDK Runtime Environment (26.0) (build 26-internal-adhoc.ec2user.jdk)
# Java VM: OpenJDK 64-Bit Server VM (26-internal-adhoc.ec2user.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0xe94540] void ShenandoahScanRemembered::process_clusters<ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2> >(unsigned long, unsigned long, HeapWordImpl**, ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2>*, bool, unsigned int)+0x4a0
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/ec2-user/tools/specjbb2015/core.40921)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
--------------- S U M M A R Y ------------
Command Line: -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+AlwaysPreTouch -Xms31g -Xmx31g -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:async -Xlog:gc*:file=./logs/gc/1/specjbb2015-genshen-master-gc-31g.log ./specjbb2015.jar -m composite -ikv -p ./config/specjbb2015.props -raw ./config/template-C.raw
Host: AArch64, 16 cores, 123G, Amazon Linux release 2023.9.20251117 (Amazon Linux)
Time: Wed Nov 26 06:30:04 2025 UTC elapsed time: 2026.242452 seconds (0d 0h 33m 46s)
--------------- T H R E A D ---------------
Current thread (0x0000ffffa411ec00): WorkerThread "Shenandoah GC Threads#0" [id=40923, stack(0x0000ffffa9008000,0x0000ffffa9206000) (2040K)]
Stack: [0x0000ffffa9008000,0x0000ffffa9206000], sp=0x0000ffffa9204510, free space=2033k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xe94540] void ShenandoahScanRemembered::process_clusters<ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2> >(unsigned long, unsigned long, HeapWordImpl**, ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2>*, bool, unsigned int)+0x4a0 (shenandoahScanRemembered.inline.hpp:200)
V [libjvm.so+0xe90d30] ShenandoahScanRememberedTask::do_work(unsigned int)+0x4e8 (shenandoahScanRemembered.inline.hpp:377)
V [libjvm.so+0xe90d98] ShenandoahScanRememberedTask::work(unsigned int)+0x58 (shenandoahScanRemembered.cpp:774)
V [libjvm.so+0x106bc18] WorkerThread::run()+0x98 (workerThread.cpp:69)
V [libjvm.so+0xfa9e14] Thread::call_run()+0xa8 (thread.cpp:242)
V [libjvm.so+0xd13ae4] thread_native_entry(Thread*)+0xd8 (os_linux.cpp:862)
C [libc.so.6+0x8bb78] start_thread+0x2d8
```
crash log from fastdebug build:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/home/ec2-user/repos/jdk/src/hotspot/share/runtime/mutexLocker.cpp:177), pid=3534433, tid=3534468
# fatal error: must own lock ClassLoaderDataGraph_lock
#
# JRE version: OpenJDK Runtime Environment (26.0) (fastdebug build 26-internal-adhoc.ec2user.jdk)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 26-internal-adhoc.ec2user.jdk, mixed mode, sharing, tiered, compressed class ptrs, shenandoah gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0x15094c8] assert_locked_or_safepoint(Mutex const*)+0xa8
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /data/tools/specjbb2015/core.3534433)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
--------------- S U M M A R Y ------------
Command Line: -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+AlwaysPreTouch -Xms512g -Xmx512g -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:async -Xlog:gc*:file=./logs/gc/test/specjbb2015-genshen-cas-alloc-refactor-gc-crash-512g.log ./specjbb2015.jar -m composite -ikv -p ./config/specjbb2015.props -raw ./config/template-C.raw
Host: ip-172-31-42-91.ec2.internal, AArch64, 96 cores, 753G, Amazon Linux release 2023.9.20251027 (Amazon Linux)
Time: Wed Nov 26 08:08:21 2025 UTC elapsed time: 2307.589793 seconds (0d 0h 38m 27s)
--------------- T H R E A D ---------------
Current thread (0x0000ffffac1755d0): WorkerThread "Shenandoah GC Threads#33" [id=3534468, stack(0x0000ff7e45f30000,0x0000ff7e4612e000) (2040K)]
Stack: [0x0000ff7e45f30000,0x0000ff7e4612e000], sp=0x0000ff7e4612c2f0, free space=2032k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x15094c8] assert_locked_or_safepoint(Mutex const*)+0xa8 (mutexLocker.cpp:177)
V [libjvm.so+0x9cf92c] ClassLoaderDataGraph::is_valid(ClassLoaderData*)+0x20 (classLoaderDataGraph.cpp:394)
V [libjvm.so+0x18cb5e4] ShenandoahCardCluster::first_object_start(unsigned long, ShenandoahMarkingContext const*, HeapWordImpl**, HeapWordImpl**) const+0xa44 (shenandoahScanRemembered.cpp:389)
V [libjvm.so+0x180f624] void ShenandoahScanRemembered::process_clusters<ShenandoahConcUpdateRefsClosure>(unsigned long, unsigned long, HeapWordImpl**, ShenandoahConcUpdateRefsClosure*, bool, unsigned int)+0x504 (shenandoahScanRemembered.inline.hpp:181)
V [libjvm.so+0x1810dd4] void ShenandoahGenerationalUpdateHeapRefsTask<true>::update_references_in_remembered_set<ShenandoahConcUpdateRefsClosure>(unsigned int, ShenandoahConcUpdateRefsClosure&, ShenandoahMarkingContext const*, bool)+0x6d4 (shenandoahScanRemembered.inline.hpp:377)
V [libjvm.so+0x1811440] void ShenandoahGenerationalUpdateHeapRefsTask<true>::do_work<ShenandoahConcUpdateRefsClosure>(unsigned int)+0x380 (shenandoahGenerationalHeap.cpp:846)
V [libjvm.so+0x18115b4] ShenandoahGenerationalUpdateHeapRefsTask<true>::work(unsigned int)+0x74 (shenandoahGenerationalHeap.cpp:771)
V [libjvm.so+0x1bea15c] WorkerThread::run()+0x98 (workerThread.cpp:69)
V [libjvm.so+0x1a93710] Thread::call_run()+0xb0 (thread.cpp:242)
V [libjvm.so+0x15b5e20] thread_native_entry(Thread*)+0x134 (os_linux.cpp:862)
C [libc.so.6+0x8bb78] start_thread+0x2d8
```
specjbb2015 bennchmark(5 out of 6 test runs) after merging the latest the change from tip, I have reproduced the issue on TIP. After checking and testing the changes introduced to Shenandoah recently and the line of code where the crashes were from, I found that it is caused by
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000ffffac294540, pid=40921, tid=40923
#
# JRE version: OpenJDK Runtime Environment (26.0) (build 26-internal-adhoc.ec2user.jdk)
# Java VM: OpenJDK 64-Bit Server VM (26-internal-adhoc.ec2user.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0xe94540] void ShenandoahScanRemembered::process_clusters<ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2> >(unsigned long, unsigned long, HeapWordImpl**, ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2>*, bool, unsigned int)+0x4a0
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/ec2-user/tools/specjbb2015/core.40921)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
--------------- S U M M A R Y ------------
Command Line: -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+AlwaysPreTouch -Xms31g -Xmx31g -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:async -Xlog:gc*:file=./logs/gc/1/specjbb2015-genshen-master-gc-31g.log ./specjbb2015.jar -m composite -ikv -p ./config/specjbb2015.props -raw ./config/template-C.raw
Host: AArch64, 16 cores, 123G, Amazon Linux release 2023.9.20251117 (Amazon Linux)
Time: Wed Nov 26 06:30:04 2025 UTC elapsed time: 2026.242452 seconds (0d 0h 33m 46s)
--------------- T H R E A D ---------------
Current thread (0x0000ffffa411ec00): WorkerThread "Shenandoah GC Threads#0" [id=40923, stack(0x0000ffffa9008000,0x0000ffffa9206000) (2040K)]
Stack: [0x0000ffffa9008000,0x0000ffffa9206000], sp=0x0000ffffa9204510, free space=2033k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xe94540] void ShenandoahScanRemembered::process_clusters<ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2> >(unsigned long, unsigned long, HeapWordImpl**, ShenandoahMarkRefsClosure<(ShenandoahGenerationType)2>*, bool, unsigned int)+0x4a0 (shenandoahScanRemembered.inline.hpp:200)
V [libjvm.so+0xe90d30] ShenandoahScanRememberedTask::do_work(unsigned int)+0x4e8 (shenandoahScanRemembered.inline.hpp:377)
V [libjvm.so+0xe90d98] ShenandoahScanRememberedTask::work(unsigned int)+0x58 (shenandoahScanRemembered.cpp:774)
V [libjvm.so+0x106bc18] WorkerThread::run()+0x98 (workerThread.cpp:69)
V [libjvm.so+0xfa9e14] Thread::call_run()+0xa8 (thread.cpp:242)
V [libjvm.so+0xd13ae4] thread_native_entry(Thread*)+0xd8 (os_linux.cpp:862)
C [libc.so.6+0x8bb78] start_thread+0x2d8
```
crash log from fastdebug build:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/home/ec2-user/repos/jdk/src/hotspot/share/runtime/mutexLocker.cpp:177), pid=3534433, tid=3534468
# fatal error: must own lock ClassLoaderDataGraph_lock
#
# JRE version: OpenJDK Runtime Environment (26.0) (fastdebug build 26-internal-adhoc.ec2user.jdk)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 26-internal-adhoc.ec2user.jdk, mixed mode, sharing, tiered, compressed class ptrs, shenandoah gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0x15094c8] assert_locked_or_safepoint(Mutex const*)+0xa8
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /data/tools/specjbb2015/core.3534433)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
--------------- S U M M A R Y ------------
Command Line: -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+AlwaysPreTouch -Xms512g -Xmx512g -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -Xlog:async -Xlog:gc*:file=./logs/gc/test/specjbb2015-genshen-cas-alloc-refactor-gc-crash-512g.log ./specjbb2015.jar -m composite -ikv -p ./config/specjbb2015.props -raw ./config/template-C.raw
Host: ip-172-31-42-91.ec2.internal, AArch64, 96 cores, 753G, Amazon Linux release 2023.9.20251027 (Amazon Linux)
Time: Wed Nov 26 08:08:21 2025 UTC elapsed time: 2307.589793 seconds (0d 0h 38m 27s)
--------------- T H R E A D ---------------
Current thread (0x0000ffffac1755d0): WorkerThread "Shenandoah GC Threads#33" [id=3534468, stack(0x0000ff7e45f30000,0x0000ff7e4612e000) (2040K)]
Stack: [0x0000ff7e45f30000,0x0000ff7e4612e000], sp=0x0000ff7e4612c2f0, free space=2032k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x15094c8] assert_locked_or_safepoint(Mutex const*)+0xa8 (mutexLocker.cpp:177)
V [libjvm.so+0x9cf92c] ClassLoaderDataGraph::is_valid(ClassLoaderData*)+0x20 (classLoaderDataGraph.cpp:394)
V [libjvm.so+0x18cb5e4] ShenandoahCardCluster::first_object_start(unsigned long, ShenandoahMarkingContext const*, HeapWordImpl**, HeapWordImpl**) const+0xa44 (shenandoahScanRemembered.cpp:389)
V [libjvm.so+0x180f624] void ShenandoahScanRemembered::process_clusters<ShenandoahConcUpdateRefsClosure>(unsigned long, unsigned long, HeapWordImpl**, ShenandoahConcUpdateRefsClosure*, bool, unsigned int)+0x504 (shenandoahScanRemembered.inline.hpp:181)
V [libjvm.so+0x1810dd4] void ShenandoahGenerationalUpdateHeapRefsTask<true>::update_references_in_remembered_set<ShenandoahConcUpdateRefsClosure>(unsigned int, ShenandoahConcUpdateRefsClosure&, ShenandoahMarkingContext const*, bool)+0x6d4 (shenandoahScanRemembered.inline.hpp:377)
V [libjvm.so+0x1811440] void ShenandoahGenerationalUpdateHeapRefsTask<true>::do_work<ShenandoahConcUpdateRefsClosure>(unsigned int)+0x380 (shenandoahGenerationalHeap.cpp:846)
V [libjvm.so+0x18115b4] ShenandoahGenerationalUpdateHeapRefsTask<true>::work(unsigned int)+0x74 (shenandoahGenerationalHeap.cpp:771)
V [libjvm.so+0x1bea15c] WorkerThread::run()+0x98 (workerThread.cpp:69)
V [libjvm.so+0x1a93710] Thread::call_run()+0xb0 (thread.cpp:242)
V [libjvm.so+0x15b5e20] thread_native_entry(Thread*)+0x134 (os_linux.cpp:862)
C [libc.so.6+0x8bb78] start_thread+0x2d8
```
- caused by
-
JDK-8371667 Shenandoah: Re-design alloc request type enum for better efficiency and cleaner code
-
- Resolved
-
- duplicates
-
JDK-8372542 [genshen] Error: Remembered set violation at init-update-references; object not properly registered
-
- Resolved
-
- links to
-
Commit(master)
openjdk/jdk/79e99bb0
-
Review(master)
openjdk/jdk/28521