ADDITIONAL SYSTEM INFORMATION :
Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 8 cores, 30G, CentOS Linux release 7.9.2009 (Core)
JRE version: OpenJDK Runtime Environment Zulu11.58+15-CA (11.0.16+8) (build 11.0.16+8-LTS)
Java VM: OpenJDK 64-Bit Server VM Zulu11.58+15-CA (11.0.16+8-LTS, mixed mode, tiered, compressed oops, parallel gc, linux-amd64)

A DESCRIPTION OF THE PROBLEM :
After upgrading systems from 11.0.15 to 11.0.16 (July release) linux x86_64, services have begun crashing frequently upon startup with the following. This has been reproducible across many servers using 11.0.16, which all fail within C2 compilation of sun.security.ssl.SSLEngineInputRecord::decodeInputRecord, and only after the jdk upgrade.

In particular, this segment from the native-memory-tracking section showing compiler memory is particularly concerning:

- Compiler (reserved=26335956KB, committed=26335956KB)
                            (malloc=1218KB #3043)
                            (arena=26334738KB #28)

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2147483664 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# JVM is running with Zero Based Compressed Oops mode in which the Java heap is
# placed in the first 32GB address space. The Java Heap base address is the
# maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
# to set the Java Heap base and to place the Java Heap above 32GB virtual address.
# This output file may be truncated or incomplete.
#
# Out of Memory Error (arena.cpp:197), pid=30772, tid=31459
#
# JRE version: OpenJDK Runtime Environment Zulu11.58+15-CA (11.0.16+8) (build 11.0.16+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu11.58+15-CA (11.0.16+8-LTS, mixed mode, tiered, compressed oops, parallel gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

--------------- S U M M A R Y ------------

Command Line: -XX:+CrashOnOutOfMemoryError -Djava.io.tmpdir=var/data/tmp -XX:ErrorFile=var/log/hs_err_pid%p.log -XX:HeapDumpPath=var/log -Dsun.net.inetaddr.ttl=20 -XX:NativeMemoryTracking=summary -XX:FlightRecorderOptions=stackdepth=256 -XX:-UseBiasedLocking -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParallelOldGC -Xms4g -Xmx4g com.MyServer

Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 8 cores, 30G, CentOS Linux release 7.9.2009 (Core)
Time: Tue Jul 26 18:35:44 2022 UTC elapsed time: 68.889468 seconds (0d 0h 1m 8s)

--------------- T H R E A D ---------------

Current thread (0x00007f4b3015aa30): JavaThread "C2 CompilerThread2" daemon [_thread_in_native, id=31459, stack(0x00007f4afba9c000,0x00007f4afbb9d000)]


Current CompileTask:
C2: 68890 17228 ! 4 sun.security.ssl.SSLEngineInputRecord::decodeInputRecord (812 bytes)

Stack: [0x00007f4afba9c000,0x00007f4afbb9d000], sp=0x00007f4afbb974c0, free space=1005k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xe7fd76] VMError::report_and_die(int, char const, char const, __va_list_tag, Thread, unsigned char, void, void, char const, int, unsigned long)+0x1c6
V [libjvm.so+0xe80ceb] VMError::report_and_die(Thread, char const, int, unsigned long, VMErrorType, char const, __va_list_tag)+0x2b
V [libjvm.so+0x69f165] report_vm_out_of_memory(char const, int, unsigned long, VMErrorType, char const, ...)+0xc5
V [libjvm.so+0x4436f6] Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x156
V [libjvm.so+0x443902] Arena::Arealloc(void, unsigned long, unsigned long, AllocFailStrategy::AllocFailEnum)+0x142
V [libjvm.so+0xba7479] Node_Array::grow(unsigned int)+0x59
V [libjvm.so+0xc31f97] PhaseIterGVN::register_new_node_with_optimizer(Node, Node)+0x107
V [libjvm.so+0xd71afc] PhaseIdealLoop::register_new_node(Node, Node)+0x2c
V [libjvm.so+0xd7252f] PhaseIdealLoop::split_up(Node, Node, Node) [clone .part.46]+0x92f
V [libjvm.so+0xd74c65] PhaseIdealLoop::do_split_if(Node)+0x895
V [libjvm.so+0xa99daf] PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&, bool)+0x1af
V [libjvm.so+0xa936bb] PhaseIdealLoop::build_and_optimize()+0xcab
V [libjvm.so+0x63bd2c] Compile::optimize_loops(int&, PhaseIterGVN&, LoopOptsMode) [clone .part.371]+0x23c
V [libjvm.so+0x63ec26] Compile::Optimize()+0x566
V [libjvm.so+0x640271] Compile::Compile(ciEnv, C2Compiler, ciMethod, int, bool, bool, bool, bool, DirectiveSet)+0xd41
V [libjvm.so+0x5694b4] C2Compiler::compile_method(ciEnv, ciMethod, int, DirectiveSet)+0xd4
V [libjvm.so+0x649869] CompileBroker::invoke_compiler_on_method(CompileTask)+0x539
V [libjvm.so+0x64ac08] CompileBroker::compiler_thread_loop()+0x438
V [libjvm.so+0xe1edf9] JavaThread::thread_main_inner()+0x1e9
V [libjvm.so+0xe1ae99] Thread::call_run()+0x149
V [libjvm.so+0xbdca36] thread_native_entry(Thread*)+0xe6

< cutting out some cruft >

Native Memory Tracking:

Total: reserved=32716409KB, committed=31356453KB
- Java Heap (reserved=4194304KB, committed=4194304KB)
                            (mmap: reserved=4194304KB, committed=4194304KB)
 
- Class (reserved=1296409KB, committed=282193KB)
                            (classes #46659)
                            ( instance classes #44335, array classes #2324)
                            (malloc=10265KB #145784)
                            (mmap: reserved=1286144KB, committed=271928KB)
                            ( Metadata: )
                            ( reserved=237568KB, committed=236760KB)
                            ( used=229499KB)
                            ( free=7261KB)
                            ( waste=0KB =0.00%)
                            ( Class space:)
                            ( reserved=1048576KB, committed=35168KB)
                            ( used=31052KB)
                            ( free=4116KB)
                            ( waste=0KB =0.00%)
 
- Thread (reserved=200000KB, committed=21772KB)
                            (thread #194)
                            (stack: reserved=199392KB, committed=21164KB)
                            (malloc=319KB #1166)
                            (arena=288KB #385)
 
- Code (reserved=255488KB, committed=87976KB)
                            (malloc=7800KB #24416)
                            (mmap: reserved=247688KB, committed=80176KB)
 
- GC (reserved=183400KB, committed=183400KB)
                            (malloc=30156KB #7999)
                            (mmap: reserved=153244KB, committed=153244KB)
 
- Compiler (reserved=26335956KB, committed=26335956KB)
                            (malloc=1218KB #3043)
                            (arena=26334738KB #28)
 
- Internal (reserved=10147KB, committed=10147KB)
                            (malloc=10115KB #77992)
                            (mmap: reserved=32KB, committed=32KB)
 
- Other (reserved=128033KB, committed=128033KB)
                            (malloc=128033KB #219)
 
- Symbol (reserved=65939KB, committed=65939KB)
                            (malloc=62319KB #699226)
                            (arena=3620KB #1)
 
- Native Memory Tracking (reserved=22008KB, committed=22008KB)
                            (malloc=50KB #633)
                            (tracking overhead=21959KB)
 
- Arena Chunk (reserved=8018KB, committed=8018KB)
                            (malloc=8018KB)
 
- Tracing (reserved=14457KB, committed=14457KB)
                            (malloc=14457KB #120)
 
- Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #193)
 
- Arguments (reserved=21KB, committed=21KB)
                            (malloc=21KB #537)
 
- Module (reserved=1121KB, committed=1121KB)
                            (malloc=1121KB #6233)
 
- Synchronizer (reserved=1095KB, committed=1095KB)
                            (malloc=1095KB #9266)
 
- Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Unfortunately I do not have steps to reproduce prepared, and have not been successful in reproducing in my development environment.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Ideally the jvm does not crash.
ACTUAL -
jvm crash with the provided hs_err_pid crash file

CUSTOMER SUBMITTED WORKAROUND :
Roll back to the previous openjdk release

FREQUENCY : often