While downporting JDK-8261027 to jdk11 we ran into the crashes like the following:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x0000ffff8c88576c, pid=49567, tid=49573
#
# JRE version: (11.0.12) (slowdebug build )
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 11.0.12-internal+0-adhoc.ubuntu.corretto-11, mixed mode, tiered, compressed oops, g1 gc, linux-aarch64)
# Problematic frame:
# v ~StubRoutines::atomic entry points
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /corretto-11/build/linux-aarch64-normal-server-slowdebug/test-support/jtreg_test_hotspot_jtreg_runtime_CompressedOops_CompressedClassPointers_java/scratch/0/core.49567)
#
#

--------------- S U M M A R Y ------------

Command Line: -XX:+UnlockDiagnosticVMOptions -XX:SharedBaseAddress=8g -Xmx128m -Xlog:gc+metaspace=trace -Xshare:off -Xlog:cds=trace -XX:+VerifyBeforeGC

Host: ip-xxx-xxx-xxx-xxx, AArch64, 64 cores, 246G, Ubuntu 18.04.5 LTS
Time: Fri Mar 12 02:30:27 2021 UTC elapsed time: 0.176642 seconds (0d 0h 0m 0s)

--------------- T H R E A D ---------------

Current thread (0x0000ffffa418c800): ConcurrentGCThread "G1 Refine#0" [stack: 0x0000ffff7b5d8000,0x0000ffff7b7d8000] [id=49573]

Stack: [0x0000ffff7b5d8000,0x0000ffff7b7d8000], sp=0x0000ffff7b7d6470, free space=2041k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
v ~StubRoutines::atomic entry points
V [libjvm.so+0x287650] int atomic_fastcall<unsigned long (*)(void volatile*, unsigned long, unsigned long), int, int, int>(unsigned long (*)(void volatile*, unsigned long, unsigned long), int volatile*, int, int)+0x34
V [libjvm.so+0x287608] int Atomic::PlatformCmpxchg<4ul>::operator()<int>(int, int volatile*, int, atomic_memory_order) const+0x70
V [libjvm.so+0x287590] Atomic::CmpxchgImpl<int, int, int, void>::operator()(int, int volatile*, int, atomic_memory_order) const+0x34
V [libjvm.so+0x287554] int Atomic::cmpxchg<int, int, int>(int, int volatile*, int, atomic_memory_order)+0x30
V [libjvm.so+0xf01280] os::PlatformEvent::park(long)+0xb8
V [libjvm.so+0xe64f34] ParkCommon(ParkEvent*, long)+0x38
V [libjvm.so+0xe65ed0] Monitor::IWait(Thread*, long)+0x124
V [libjvm.so+0xe67660] Monitor::wait(bool, long, bool)+0x4d0
V [libjvm.so+0x71f220] ConcurrentGCThread::wait_for_universe_init()+0x88
V [libjvm.so+0x71f320] ConcurrentGCThread::run()+0x1c
V [libjvm.so+0x118bd70] Thread::call_run()+0xbc
V [libjvm.so+0xee5254] thread_native_entry(Thread*)+0x190
C [libpthread.so.0+0x7088] start_thread+0xb0

However, the assembly at the offending address looks fine:

   0xffff8c885764: cas w3, w2, [x0]
   0xffff8c885768: cmp w3, w1
=> 0xffff8c88576c: mov w0, w3
   0xffff8c885770: ret

JDK-8261027 actually introduced two version of atomics. There's a static version (e.g. aarch64_atomic_cmpxchg_4_default_impl) which is generated at build time and there's a stub generated at runtime (in generate_atomic_entry_points()). Dispatching is done in "atomic_fastcall(F stub, ...)" which takes the function address of the actual implementation and calls it.

Initially, before the stubs are generated, the function addresses are initialized to point to the static implementations (e.g. "aarch64_atomic_cmpxchg_4_impl = aarch64_atomic_cmpxchg_4_default_impl). Once the dynamic stubs are generated, the function addresses will be re-assigned to point to the generated stubs. This happens as follows (in generate_atomic_entry_points()):

    aarch64_atomic_cmpxchg_4_impl = (aarch64_atomic_stub_t)__ pc();
    {
      Register prev = r3, ptr = c_rarg0, compare_val = c_rarg1,
        exchange_val = c_rarg2;
      __ cmpxchg(ptr, compare_val, exchange_val

As you can see, "aarch64_atomic_cmpxchg_4_impl" is reassigned to point into the code cache BEFORE the code is actually generated. This introduces a small time window where we can jump to the dynamically generated stun address without actually having any code there. And I think this is exactly what happens in the above crash. It also explains why the code looks just fine in the hs_err file, because at the time the hs_err file dumps the content at the offending address, the complete stub has already been successfully generated.

The fix is easy. Only re-assign the function pointers after all the dynamic stubs have been successfully created and flushed.