The following test failed in the JDK18 CI:

vmTestbase/nsk/sysdict/vm/stress/btree/btree006/btree006.java

Here's a snippet from the log file:

#section:main
----------messages:(4/331)----------
command: main -XX:-UseGCOverheadLimit nsk.sysdict.share.BTreeTest -jarpath btree.jar:fats.jar -useFatClass
reason: User specified action: run main/othervm -XX:-UseGCOverheadLimit nsk.sysdict.share.BTreeTest -jarpath btree.jar${path.separator}fats.jar -useFatClass
Mode: othervm [/othervm specified]
elapsed time (seconds): 13.209
----------configuration:(0/0)----------
----------System.out:(57/4169)----------
The level = 3 the height = 4
Stress time: 120 seconds
Stress iterations factor: 1
Stress threads factor: 1
Stress runs factor: 1
Max memory: 3120562176
Sleep time: 500
Iterations: 0
Number of threads: 16
Run GC thread: false
Run mem diag thread: false
Run forever: false
For random generator using seed: 4242882341725657613
To re-run test with same seed value please add "-Djdk.test.lib.random.seed=4242882341725657613" to command line.
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@15ee8277,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@b8188f5,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@6a1d4ec8,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@36ffcba8,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@2e533164,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@2992c27b,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@2979b28f,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@61139e83,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@56fc4bd4,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@6942ca6a,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@d4bf007,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@70d190ab,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@4447d78e,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@3ba954f6,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@6930404c,5,MainThreadGroup]
Starting Thread[nsk.sysdict.share.SysDictTest$Worker@1598c109,5,MainThreadGroup]
 Locks owned:
Mutex: [0x0000fffc540264a0] MultiArray_lock - owner: 0x0000fffc54317010 safepoint_check_always
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=/mutex.cpp:418
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/opt/mach5/mesos/work_dir/slaves/ff806ead-2cac-495d-9cbc-62116f99bf14-S30351/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/05471bf8-5f3a-4bd5-9073-1b709c806616/runs/77cd69ac-438f-48dc-82b6-c1adf3484e80/workspace/open/src/hotspot/share/runtime/mutex.cpp:418), pid=3796057, tid=3796091
# assert(false) failed: Attempting to acquire lock JNICritical_lock/41 out of order with lock MultiArray_lock/41 -- possible deadlock
#
# JRE version: Java(TM) SE Runtime Environment (18.0+19) (fastdebug build 18-ea+19-1101)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 18-ea+19-1101, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0x13e6a6c][thread 3796124 also had an error]
  Mutex::check_rank(Thread*)+0x10c
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13291/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/ee5067a9-ec34-41ad-acb0-52fabfebbd75/runs/556f6905-c928-4d38-80e8-acf92eb12676/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_sysdict/scratch/2/core.3796057)
#
# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13291/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/ee5067a9-ec34-41ad-acb0-52fabfebbd75/runs/556f6905-c928-4d38-80e8-acf92eb12676/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_sysdict/scratch/2/hs_err_pid3796057.log
#
# Compiler replay data is saved as:
# /opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13291/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/ee5067a9-ec34-41ad-acb0-52fabfebbd75/runs/556f6905-c928-4d38-80e8-acf92eb12676/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_sysdict/scratch/2/replay_pid3796057.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
----------System.err:(0/0)----------
----------rerun:(33/7574)*----------


Here's the crashing thread's stack:

--------------- T H R E A D ---------------

Current thread (0x0000fffc54317010): JavaThread "C1 CompilerThread0" daemon [_thread_in_vm, id=3796091, stack(0x0000fffc24e00000,0x0000fffc25000000)]


Current CompileTask:
C1: 6863 833 3 java.security.AccessControlContext::optimize (319 bytes)

Stack: [0x0000fffc24e00000,0x0000fffc25000000], sp=0x0000fffc24ffc920, free space=2034k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x13e6a6c] Mutex::check_rank(Thread*)+0x10c
V [libjvm.so+0x13e84bc] Mutex::lock(Thread*)+0x38
V [libjvm.so+0xcd929c] GCLocker::stall_until_clear()+0x9c
V [libjvm.so+0xbfc444] G1CollectedHeap::attempt_allocation_slow(unsigned long)+0x834
V [libjvm.so+0xc0140c] G1CollectedHeap::attempt_allocation(unsigned long, unsigned long, unsigned long*)+0x3ec
V [libjvm.so+0x132ed94] MemAllocator::allocate_inside_tlab_slow(MemAllocator::Allocation&) const+0x300
V [libjvm.so+0x132fd84] MemAllocator::allocate() const+0x154
V [libjvm.so+0xe0a788] InstanceMirrorKlass::allocate_instance(Klass*, JavaThread*)+0x58
V [libjvm.so+0xe5f5a8] java_lang_Class::create_mirror(Klass*, Handle, Handle, Handle, Handle, JavaThread*)+0x148
V [libjvm.so+0x6574f4] ArrayKlass::complete_create_array_klass(ArrayKlass*, Klass*, ModuleEntry*, JavaThread*)+0x214
V [libjvm.so+0x1426cf0] ObjArrayKlass::allocate_objArray_klass(ClassLoaderData*, int, Klass*, JavaThread*)+0x50c
V [libjvm.so+0xdeb704] InstanceKlass::array_klass(int, JavaThread*)+0xd4
V [libjvm.so+0x142697c] ObjArrayKlass::allocate_objArray_klass(ClassLoaderData*, int, Klass*, JavaThread*)+0x198
V [libjvm.so+0xdeb704] InstanceKlass::array_klass(int, JavaThread*)+0xd4
V [libjvm.so+0x8e0484] ciObjArrayKlass::make_impl(ciKlass*)+0x80
V [libjvm.so+0x88c9bc] ciEnv::get_klass_by_name_impl(ciKlass*, constantPoolHandle const&, ciSymbol*, bool)+0x428
V [libjvm.so+0x88ce8c] ciEnv::get_klass_by_index_impl(constantPoolHandle const&, int, bool&, ciInstanceKlass*)+0x14c
V [libjvm.so+0x89d5e0] ciField::ciField(ciInstanceKlass*, int)+0x790
V [libjvm.so+0x88ba78] ciEnv::get_field_by_index_impl(ciInstanceKlass*, int)+0x78
V [libjvm.so+0x88c560] ciEnv::get_field_by_index(ciInstanceKlass*, int)+0x90
V [libjvm.so+0x8ffc64] ciBytecodeStream::get_field(bool&)+0x84
V [libjvm.so+0x71fac4] GraphBuilder::access_field(Bytecodes::Code)+0x30
V [libjvm.so+0x724d20] GraphBuilder::iterate_bytecodes_for_block(int)+0x910
V [libjvm.so+0x726798] GraphBuilder::iterate_all_blocks(bool)+0x94
V [libjvm.so+0x727a2c] GraphBuilder::GraphBuilder(Compilation*, IRScope*)+0x568
V [libjvm.so+0x736860] IRScope::IRScope(Compilation*, IRScope*, int, ciMethod*, int, bool)+0x3c0
V [libjvm.so+0x736a74] IR::IR(Compilation*, ciMethod*, int)+0xd0
V [libjvm.so+0x6fe77c] Compilation::build_hir() [clone .part.0]+0x22c
V [libjvm.so+0x702ef4] Compilation::compile_java_method()+0x1b0
V [libjvm.so+0x703d48] Compilation::compile_method()+0x1d8
V [libjvm.so+0x7046d8] Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*, bool, DirectiveSet*)+0x2d8
V [libjvm.so+0x705bf8] Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x158
V [libjvm.so+0x9e4ec8] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x8b8
V [libjvm.so+0x9e593c] CompileBroker::compiler_thread_loop()+0x2dc
V [libjvm.so+0x1756074] JavaThread::thread_main_inner()+0x284
V [libjvm.so+0x175c648] Thread::call_run()+0xf8
V [libjvm.so+0x1487594] thread_native_entry(Thread*)+0x104
C [libpthread.so.0+0x7738] start_thread+0x198

Starting this one in hotspot/runtime since @coleenp has been
changing the internal mutex code.