Following crash is observed in the production step of running the springboot-petclinic with load in 5-step workflow

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fea2d91a811, pid=162478, tid=162513
#
# JRE version: OpenJDK Runtime Environment (23.0) (build 23-internal-adhoc.asmehra.leyden)
# Java VM: OpenJDK 64-Bit Server VM (23-internal-adhoc.asmehra.leyden, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 15587 c1 org.unbescape.html.HtmlEscapeSymbols.<init>(Lorg/unbescape/html/HtmlEscapeSymbols$References;[B)V (637 bytes) @ 0x00007fea2d91a811 [0x00007fea2d919980+0x0000000000000e91]
#
# Core dump will be written. Default location: /home/asmehra/data/ashu-mehra/leyden/test/hotspot/jtreg/premain/spring-petclinic/core.162478
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

Stack: [0x00007f43b31fb000,0x00007f43b32fb000], sp=0x00007f43b32f6be0, free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 15612 c1 org.unbescape.html.HtmlEscapeSymbols.<init>(Lorg/unbescape/html/HtmlEscapeSymbols$References;[B)V (637 bytes) @ 0x00007f43bd96ba11 [0x00007f43bd96ab80+0x0000000000000e91]
j org.unbescape.html.Html5EscapeSymbolsInitializer.initializeHtml5()Lorg/unbescape/html/HtmlEscapeSymbols;+22399
j org.unbescape.html.HtmlEscapeSymbols.<clinit>()V+6
v ~StubRoutines::call_stub 0x00007f43c3f37cc6

Relevant registers:

RDX=0x00000000b9382570 is an oop: [C
R13=0x00000008000024a8 is a pointer to class
R14=0x000000001402014b is an unknown value

Compiled code around the crash:

  0x00007f43bd96b9db: mov r11d,DWORD PTR [rdx+0x8] ; implicit exception: dispatches to 0x00007f43bd96c67c
  0x00007f43bd96b9df: movabs r10,0x800000000
  0x00007f43bd96b9e9: add r11,r10 // r11 = Klass* for the array object; should be of type ObjArrayKlass
  0x00007f43bd96b9ec: mov r13d,DWORD PTR [r9+0x8]
  0x00007f43bd96b9f0: movabs r10,0x800000000
  0x00007f43bd96b9fa: add r13,r10 // r13 = Klass* for the value object
  0x00007f43bd96b9fd: mov r11,QWORD PTR [r11+0xd0] // r11 = ObjArrayKlass::_element_klass
  0x00007f43bd96ba04: cmp r13,r11
  0x00007f43bd96ba07: je 0x00007f43bd96ba3c
  0x00007f43bd96ba0d: mov r14d,DWORD PTR [r11+0x14]
  0x00007f43bd96ba11: cmp r11,QWORD PTR [r13+r14*1+0x0] << crash point
  
This code is for checking that the type of value stored in the array is of expected type. It is emitted by [0].
Here rdx is the array object and r9 is the value object to be stored in the array.
At 0x00007f43bd96b9e9 r11 should be the klass for the array object and is expected to be of type ObjArrayKlass. Instruction at 0x00007f43bd96b9fd fetches the klass of the elements of this array (using ObjArrayKlass::_element_klass which is at offset 0xd0 in the ObjArrayKlass object).
However, the klass of the array object in this case turns out to be of type TypeArrayKlass for "[C" (array of char), which results in r11 having garbage value after 0x00007f43bd96b9fd, ultimately leading to the crash some instructions later.

The Java code corresponding to this assembly code is [1]:

final char[] ncr = ncrsOrdered.get(i);
SORTED_NCRS[i] = ncr;
            
SORTED_NCRS is declared as:

final char[][] SORTED_NCRS;

The array object in rdx is SORTED_NCRS, and its type should be ObjArrayKlass, but, as mentioned earlier, its type is TypeArrayKlass for "[C".

Tracing back to the point where SORTED_NCRS is created in the compiled code:

  0x00007f43bd96b6a7: movabs rdx,0x8000024a8 ; {metadata({type array char})}
  0x00007f43bd96b6b1: mov r8,rax
  0x00007f43bd96b6b4: movsxd rbx,ebx
  0x00007f43bd96b6b7: mov rdi,rbx
  0x00007f43bd96b6ba: cmp rbx,0xffffff
  0x00007f43bd96b6c1: ja 0x00007f43bd96c574
  0x00007f43bd96b6c7: mov esi,0x17
  0x00007f43bd96b6cc: lea rsi,[rsi+rbx*4]
  0x00007f43bd96b6d0: and rsi,0xfffffffffffffff8
  0x00007f43bd96b6d4: mov rax,QWORD PTR [r15+0x1b8]
  0x00007f43bd96b6db: lea rsi,[rax+rsi*1]
  0x00007f43bd96b6df: cmp rsi,QWORD PTR [r15+0x1c8]
  0x00007f43bd96b6e6: ja 0x00007f43bd96c574
  0x00007f43bd96b6ec: mov QWORD PTR [r15+0x1b8],rsi
  0x00007f43bd96b6f3: sub rsi,rax
  0x00007f43bd96b6f6: mov QWORD PTR [rax],0x1
  0x00007f43bd96b6fd: mov rcx,rdx
  0x00007f43bd96b700: movabs r10,0x800000000
  0x00007f43bd96b70a: sub rcx,r10

The first instruction incorrectly uses a TypeArrayKlass instead of an ObjArrayKlass as the klass for the array object referenced by SORTED_NCRS. This seems to be the source of the problem.

This compiled code body is loaded from the code archive, which means the method is actually compiled in the previous step when the code archive was created.
The assembly code for this method during the code archive creation shows it correctly used the ObjArrayKlass when creating the SORTED_NCRS object:

  0x00007f909ca01027: movabs rdx,0x807478df0 ; {metadata({type array char}[])}
  0x00007f909ca01031: mov r8,rax
  0x00007f909ca01034: movsxd rbx,ebx
  0x00007f909ca01037: mov rdi,rbx
  0x00007f909ca0103a: cmp rbx,0xffffff
  0x00007f909ca01041: ja 0x00007f909ca01ef4
  0x00007f909ca01047: mov esi,0x17
  0x00007f909ca0104c: lea rsi,[rsi+rbx*4]
  0x00007f909ca01050: and rsi,0xfffffffffffffff8
  0x00007f909ca01054: mov rax,QWORD PTR [r15+0x1b8]
  0x00007f909ca0105b: lea rsi,[rax+rsi*1]
  0x00007f909ca0105f: cmp rsi,QWORD PTR [r15+0x1c8]
  0x00007f909ca01066: ja 0x00007f909ca01ef4
  0x00007f909ca0106c: mov QWORD PTR [r15+0x1b8],rsi
  0x00007f909ca01073: sub rsi,rax
  0x00007f909ca01076: mov QWORD PTR [rax],0x1
  0x00007f909ca0107d: mov rcx,rdx
  0x00007f909ca01080: movabs r10,0x800000000
  0x00007f909ca0108a: sub rcx,r10

So the code was generated correct, which indicates the code was not patched properly when it was loaded again in the production run.
When the code is genearted SCCache::write_klass() writes the ObjArrayKlass::_bottom_klass and array dimensions for the ObjArrayKlass when writing the metadata.
In this case ObjArrayKlass::_bottom_klass is TypeArrayKlass and dimensions is 1.
But It turns out the SCCReader::read_klass() has a bug due to which when the klass is TypeArrayKlass and dimensions is > 1, instead of returning ObjArrayKlass, it incorrectly returns TypeArrayKlass.

[0] https://github.com/openjdk/leyden/blob/83a963565d7a050938f81f17c96cd1f6324d6d3b/src/hotspot/cpu/x86/c1_LIRAssembler_x86.cpp#L1876-L1882
[1] https://github.com/unbescape/unbescape/blob/ec5435fb3508c2eed25d8165dc27ded2602cae13/src/main/java/org/unbescape/html/HtmlEscapeSymbols.java#L261