In low memory scenarios, libgraal can prematurely cause the VM to exit. For example, the following occurred in the java/util/concurrent/locks/Lock/OOMEInAQS.java test:

command: main -XX:-UseGCOverheadLimit -Xmx24M -XX:-UseTLAB OOMEInAQS
reason: User specified action: run main/othervm -XX:-UseGCOverheadLimit -Xmx24M -XX:-UseTLAB OOMEInAQS
started: Wed Apr 26 12:16:33 PDT 2023
Mode: othervm [/othervm specified]
finished: Wed Apr 26 12:16:38 PDT 2023
elapsed time (seconds): 5.073
configuration:
STDOUT:
Exception during JVMCI compiler initialization
Exception in thread "main": java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: OutOfMemoryError occurred while encoding an exception to translate it from the JVM heap
at java.base@20.0.1/jdk.internal.vm.VMSupport.decodeAndThrowThrowable(VMSupport.java:138)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.CompilerToVM.lookupType(Native Method)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.CompilerToVM.lookupType(CompilerToVM.java:245)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.createClass(HotSpotJVMCIRuntime.java:627)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime$1.computeValue(HotSpotJVMCIRuntime.java:642)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime$1.computeValue(HotSpotJVMCIRuntime.java:639)
at java.base@20.0.1/java.lang.ClassValue.get(JavaLangSubstitutions.java:667)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.fromClass0(HotSpotJVMCIRuntime.java:649)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.fromClass(HotSpotJVMCIRuntime.java:670)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotMetaAccessProvider.lookupJavaType(HotSpotMetaAccessProvider.java:63)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotSnippetMetaAccessProvider.lookupJavaType(HotSpotSnippetMetaAccessProvider.java:58)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.replacements.HotSpotReplacementsUtil.referenceType(HotSpotReplacementsUtil.java:951)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.replacements.HotSpotReplacementsUtil.referentField(HotSpotReplacementsUtil.java:946)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotBackendFactory.createBarrierSet(HotSpotBackendFactory.java:244)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotBackendFactory.createBackend(HotSpotBackendFactory.java:159)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalRuntime.<init>(HotSpotGraalRuntime.java:157)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompilerFactory.createCompiler(HotSpotGraalCompilerFactory.java:212)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompilerFactory.createCompiler(HotSpotGraalCompilerFactory.java:190)
at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompilerFactory.createCompiler(HotSpotGraalCompilerFactory.java:53)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.getCompiler(HotSpotJVMCIRuntime.java:806)
at jdk.internal.vm.ci@20.0.1/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.compileMethod(HotSpotJVMCIRuntime.java:920)
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=\jvmciRuntime.cpp:1637
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (c:\users\buildbot\worker\e\main\jdk_tlda\open\src\hotspot\share\jvmci\jvmciRuntime.cpp:1637), pid=20928, tid=29700
# fatal error: Fatal exception in JVMCI: Exception during JVMCI compiler initialization
#
# JRE version: Java(TM) SE Runtime Environment Oracle GraalVM 20.0.1-dev+9.1 (20.0.1+9) (fastdebug build 20.0.1+9-jvmci-23.0-b10)
# Java VM: Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 20.0.1-dev+9.1 (fastdebug 20.0.1+9-jvmci-23.0-b10, mixed mode, tiered, jvmci, jvmci compiler, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
# Core dump will be written. Default location: C:\Users\Buildbot\worker\e\main\labsjdk\open\build\run-test-prebuilt\test-support\jtreg_open_test_jdk_jdk_loom\scratch\2\hs_err_pid20928.mdmp
#
# An error report file with more information is saved as:
# C:\Users\Buildbot\worker\e\main\labsjdk\open\build\run-test-prebuilt\test-support\jtreg_open_test_jdk_jdk_loom\scratch\2\hs_err_pid20928.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

More defensive code is needed to treat this as a compilation failure instead of a fatal error.