ExitOnOutOfMemoryError doesn't cover OOMEs from Java libraries

Not only ExitOnOutOfMemoryError, other flags such as OnOutOfMemoryError, CrashOnOutOfMemoryError and HeapDumpOnOutOfMemoryError all lie with the internal function debug.cpp::report_java_out_of_memory of JVM.

report_java_out_of_memory() only covers OOMEs that raise from the internal JVM(hotspot). Some OOMEs are thrown in Java libraries. In short words, OOMEs are not reported to hotspot in those cases. here is non-exhaustive lists. OOME("Direct buffer memory") is one of them.

$ grep "new OutOfMemoryError" -R java.base/
java.base/share/classes/jdk/internal/util/ArraysSupport.java: throw new OutOfMemoryError("Required array length too large");
java.base/share/classes/jdk/internal/misc/Unsafe.java: throw new OutOfMemoryError("Unable to allocate " + bytes + " bytes");
java.base/share/classes/jdk/internal/misc/Unsafe.java: throw new OutOfMemoryError("Unable to allocate " + bytes + " bytes");
java.base/share/classes/jdk/internal/icu/text/BidiBase.java: throw new OutOfMemoryError("Failed to allocate memory for "
java.base/share/classes/jdk/internal/icu/text/BidiBase.java: throw new OutOfMemoryError("Failed to allocate memory for "
java.base/share/classes/jdk/internal/icu/text/BidiBase.java: throw new OutOfMemoryError("Failed to allocate memory for paras");
java.base/share/classes/jdk/internal/icu/text/BidiBase.java: throw new OutOfMemoryError("Failed to allocate memory for openings");
java.base/share/classes/java/util/Base64.java: throw new OutOfMemoryError("Encoded size is too large");
java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java: throw new OutOfMemoryError(OOME_MSG);
java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java: throw new OutOfMemoryError(OOME_MSG);
java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java: throw new OutOfMemoryError(OOME_MSG);
java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java: throw new OutOfMemoryError(OOME_MSG);
java.base/share/classes/java/util/jar/JarFile.java: throw new OutOfMemoryError("Required array size too large");
java.base/share/classes/java/util/StringJoiner.java: throw new OutOfMemoryError("Requested array size exceeds VM limit");
java.base/share/classes/java/util/regex/Pattern.java: throw new OutOfMemoryError("Required pattern length too large");
java.base/share/classes/java/nio/Bits.java: throw new OutOfMemoryError
java.base/share/classes/java/nio/file/Files.java: throw new OutOfMemoryError("Required array size too large");
java.base/share/classes/java/io/InputStream.java: throw new OutOfMemoryError("Required array size too large");
java.base/share/classes/java/lang/String.java: throw new OutOfMemoryError("Required length exceeds implementation limit");
java.base/share/classes/java/lang/String.java: throw new OutOfMemoryError("Required length exceeds implementation limit");
java.base/share/classes/java/lang/AbstractStringBuilder.java: throw new OutOfMemoryError("Required length exceeds implementation limit");
java.base/share/classes/java/lang/StringUTF16.java: throw new OutOfMemoryError("UTF16 String size is " + len +
java.base/share/classes/java/lang/StringUTF16.java: throw new OutOfMemoryError("Required length exceeds implementation limit");
java.base/share/classes/java/lang/StringLatin1.java: throw new OutOfMemoryError("Required length exceeds implementation limit");
java.base/share/classes/java/lang/StringConcatHelper.java: throw new OutOfMemoryError("Overflow: String length out of range");

Back to ExitOnOutOfMemoryError, which is introduced by JDK-8138745, it intercepts OOME and abruptly exits JVM with exitcode 3. If users don't catch and suppress it, this error will keep unwinding the stack and eventually exits JVM. I think it's a good manner and that's why ExitOnOutOfMemoryError is false by default.

Hotspot itself doesn't have any effective way to relief OOMEs, so the termination is doomed, but that may not be true in Java libraries. The higher level of abstraction, more wiggle room there is. Exiting JVM at first OOME seems to be arbitrary. I intend to believe this behavior is by design. On that other side, it's misleading indeed to have a product flag ExitOnOutOfMemoryError -- "JVM exits on the first occurrence of an out-of-memory error".