The attached hs_err shows that ZGC is in the relocate phase (Event: 66.375 Executing VM operation: ZRelocateStartYoung (Allocation Rate) done). After this phase, there may exist oops pointing into regions of memory that have been deallocated in favor of moving/compacting objects in another location. Dereferencing an oop might therefore be unsafe if the oop is still pointing into the region of memory that has been deallocated and has not been updated to point to the new location.

From the stack trace listed below we can see that the crash occurs in lockStack.inline.hpp:219, inside LockStack::do_oops(), which is a call to LockStack::verify(), which verifies the consistency of the lock stack. LockStack::do_oops() calls LockStack::verify() before and after doing its actual work, exactly like is done for many other operations on the lock stack. In the first call to verify, before the actual work, LockStack::verify() gets an oop to the current entry "oop o = _base[i]", which will cause ZGC to check the oop from oop::check_oop() and do a dereference test, which will crash when the oop has not been updated. See crash details below.

java/lang/Thread/virtual/stress/Skynet.java run with "-XX:+UseZGC -Xcomp -XX:-TieredCompilation" crashed with

# SIGSEGV (0xb) at pc=0x0000ffff916b6d08, pid=1061311, tid=1061314
#
# JRE version: Java(TM) SE Runtime Environment (24.0+1) (fastdebug build 24-jep483+1-9)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 24-jep483+1-9, compiled mode, sharing, compressed class ptrs, z gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0x17c8d08] check_is_valid_zaddress(oopDesc*)+0x158


Current thread (0x0000ffff880ccef0): JavaThread "main" [_thread_in_vm, id=1061314, stack(0x0000ffff8fc0e000,0x0000ffff8fe0c000) (2040K)]

Stack: [0x0000ffff8fc0e000,0x0000ffff8fe0c000], sp=0x0000ffff8fe073c0, free space=2020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x17c8d08] check_is_valid_zaddress(oopDesc*)+0x158 (atomic.hpp:553)
V [libjvm.so+0xd87280] JavaThread::oops_do_no_frames(OopClosure*, NMethodClosure*)+0x1d0 (lockStack.inline.hpp:219)
V [libjvm.so+0x1869778] ZStackWatermark::process_head(void*)+0x88 (zStackWatermark.cpp:169)
V [libjvm.so+0x186a344] ZStackWatermark::start_processing_impl(void*)+0x24 (zStackWatermark.cpp:181)
V [libjvm.so+0x150a2ac] StackWatermark::on_safepoint()+0x7c (stackWatermark.cpp:325)
V [libjvm.so+0x148dccc] SafepointMechanism::process(JavaThread*, bool, bool)+0x5c (safepointMechanism.cpp:157)
V [libjvm.so+0x12a5f70] Monitor::wait(unsigned long)+0x420 (safepointMechanism.inline.hpp:83)
V [libjvm.so+0x901974] CompileBroker::wait_for_completion(CompileTask*)+0xe0 (mutexLocker.hpp:283)
V [libjvm.so+0x9069b4] CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, DirectiveSet*, JavaThread*)+0x340 (compileBroker.cpp:1475)
V [libjvm.so+0x906fe8] CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+0xc4 (compileBroker.cpp:1348)
V [libjvm.so+0x8d4f6c] CompilationPolicy::compile_if_required(methodHandle const&, JavaThread*)+0x16c (compilationPolicy.cpp:108)
V [libjvm.so+0x10f6aa0] LinkResolver::runtime_resolve_special_method(CallInfo&, LinkInfo const&, methodHandle const&, Handle, JavaThread*)+0x150 (linkResolver.cpp:77)
V [libjvm.so+0x10f70dc] LinkResolver::resolve_special_call(CallInfo&, Handle, LinkInfo const&, JavaThread*)+0x98 (linkResolver.cpp:1156)
V [libjvm.so+0x10f7260] LinkResolver::resolve_invokespecial(CallInfo&, Handle, constantPoolHandle const&, int, JavaThread*)+0x8c (linkResolver.cpp:1753)
V [libjvm.so+0x10f96d8] LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle const&, int, Bytecodes::Code, JavaThread*)+0xf8 (linkResolver.cpp:1706)
V [libjvm.so+0x14c2290] SharedRuntime::find_callee_info_helper(vframeStream&, Bytecodes::Code&, CallInfo&, JavaThread*)+0x470 (sharedRuntime.cpp:1269)
V [libjvm.so+0x14ca3c4] SharedRuntime::resolve_helper(bool, bool, JavaThread*)+0x104 (sharedRuntime.cpp:1155)
V [libjvm.so+0x14cafa8] SharedRuntime::resolve_opt_virtual_call_C(JavaThread*)+0x78 (sharedRuntime.cpp:1578)
v ~RuntimeStub::Shared Runtime resolve_opt_virtual_call_blob 0x0000ffff8cbfff10
J 4106 c2 java.util.jar.JarFile.getInputStream(Ljava/util/zip/ZipEntry;)Ljava/io/InputStream; java.base@24-jep483 (88 bytes) @ 0x0000ffff8cde0b8c [0x0000ffff8cde08c0+0x00000000000002cc]
J 4105 c2 jdk.internal.loader.URLClassPath$JarLoader$2.getInputStream()Ljava/io/InputStream; java.base@24-jep483 (15 bytes) @ 0x0000ffff8cdb7de8 [0x0000ffff8cdb7d40+0x00000000000000a8]
J 4089 c2 jdk.internal.loader.BuiltinClassLoader.defineClass(Ljava/lang/String;Ljdk/internal/loader/Resource;)Ljava/lang/Class; java.base@24-jep483 (121 bytes) @ 0x0000ffff8ce91a24 [0x0000ffff8ce91300+0x0000000000000724]
J 4053 c2 jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Ljava/lang/String;)Ljava/lang/Class; java.base@24-jep483 (64 bytes) @ 0x0000ffff8cc94ec8 [0x0000ffff8cc94dc0+0x0000000000000108]
J 2177 c2 jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Ljava/lang/String;Z)Ljava/lang/Class; java.base@24-jep483 (143 bytes) @ 0x0000ffff8ceb6250 [0x0000ffff8ceb5e40+0x0000000000000410]
J 1410 c2 jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; java.base@24-jep483 (40 bytes) @ 0x0000ffff8cde6540 [0x0000ffff8cde64c0+0x0000000000000080]
J 1409 c2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; java.base@24-jep483 (7 bytes) @ 0x0000ffff8cde3a4c [0x0000ffff8cde39c0+0x000000000000008c]
v ~StubRoutines::call_stub 0x0000ffff8cb2e180
V [libjvm.so+0xd490a4] JavaCalls::call_helper(JavaVal