We hit the debug memory limit in PhaseChaitin::Split by removing the temporary flag -XX:CompileCommand=MemLimit,*.*,0 from VectorReplicateLongSpecialImmTest.java and running the below command on aarch64.

  make test TEST=compiler/vectorapi/VectorReplicateLongSpecialImmTest.java JTREG='JAVA_OPTIONS=-XX:-UseTLAB'

The issue triggers after the fix for JDK-8324345, which results in skipping escape analysis in more cases than before. Potentially, skipping escape analysis and leaving a larger set of nodes could explain the present issue.

After fixing the (potential) issue, please remove the flag -XX:CompileCommand=MemLimit,*.*,0 from VectorReplicateLongSpecialImmTest.java.


#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/scratch/dlunden/jdk/open/src/hotspot/share/compiler/compilationMemoryStatistic.cpp:568), pid=677343, tid=677364
# fatal error: c2 compiler/vectorapi/VectorReplicateLongSpecialImmTest::testReplicateL_Imm(()V): Hit MemLimit (limit: 1073741824 now: 1073748984)
#
# JRE version: Java(TM) SE Runtime Environment (24.0) (slowdebug build 24-internal-2024-08-05-1215129.dlunden...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 24-internal-2024-08-05-1215129.dlunden..., mixed mode, sharing, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Core dump will be written. Default location: /scratch/dlunden/jdk/build/linux-aarch64-slowdebug/test-support/jtreg_open_test_hotspot_jtreg_compiler_vectorapi_VectorReplicateLongSpecialImmTest_java/scratch/0/core.677343
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x0000fffbfc1af270): JavaThread "C2 CompilerThread3" daemon [_thread_in_native, id=677364, stack(0x0000fffc19710000,0x0000fffc19910000) (2048K)]


Current CompileTask:
C2:100677 1365 % compiler.vectorapi.VectorReplicateLongSpecialImmTest::testReplicateL_Imm @ 2 (3851 bytes)

Stack: [0x0000fffc19710000,0x0000fffc19910000], sp=0x0000fffc1990a540, free space=2025k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x15b0e54] VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x92c (compilationMemoryStatistic.cpp:568)
V [libjvm.so+0x9d6f78] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0x0 (debug.cpp:214)
V [libjvm.so+0x9226f8] CompilationMemoryStatistic::on_arena_change(long, Arena const*)+0x368 (compilationMemoryStatistic.cpp:568)
V [libjvm.so+0x64866c] Arena::set_size_in_bytes(unsigned long)+0xf8 (arena.cpp:266)
V [libjvm.so+0x648828] Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x124 (arena.cpp:308)
V [libjvm.so+0x48d6a4] Arena::internal_amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)+0x11c (arena.hpp:134)
V [libjvm.so+0x61b4c0] Arena::Amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)+0xc4 (arena.hpp:155)
V [libjvm.so+0x138bd84] ResourceArea::allocate_bytes(unsigned long, AllocFailStrategy::AllocFailEnum)+0x2c (resourceArea.inline.hpp:35)
V [libjvm.so+0x1388088] PhaseChaitin::Split(unsigned int, ResourceArea*)+0x374 (reg_split.cpp:555)
V [libjvm.so+0x835afc] PhaseChaitin::Register_Allocate()+0x3f0 (chaitin.cpp:481)
V [libjvm.so+0x9372c4] Compile::Code_Gen()+0x248 (compile.cpp:2966)
V [libjvm.so+0x92d740] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x11f8 (compile.cpp:885)
V [libjvm.so+0x7e5f78] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1b4 (c2compiler.cpp:142)
V [libjvm.so+0x951a98] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x8dc (compileBroker.cpp:2303)
V [libjvm.so+0x950688] CompileBroker::compiler_thread_loop()+0x3a4 (compileBroker.cpp:1961)
V [libjvm.so+0x972360] CompilerThread::thread_entry(JavaThread*, JavaThread*)+0xb0 (compilerThread.cpp:65)
V [libjvm.so+0xdb0200] JavaThread::thread_main_inner()+0x188 (javaThread.cpp:757)
V [libjvm.so+0xdb006c] JavaThread::run()+0x208 (javaThread.cpp:742)
V [libjvm.so+0x14fa214] Thread::call_run()+0x1d8 (thread.cpp:225)
V [libjvm.so+0x12bf274] thread_native_entry(Thread*)+0x1d0 (os_linux.cpp:858)
C [libpthread.so.0+0x7950] start_thread+0x190
Lock stack of current Java thread (top to bottom):