Context: we are debugging an OutOfMemoryError from filling up NIO direct memory (-XX:MaxDirectMemorySize). We tried to use -XX:+HeapDumpOnOutOfMemoryError to collect heap dumps, and found out the flag does not work for direct memory OutOfMemoryError.

There are a few JVM flags related to handling OutOfMemoryError:
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError
-XX:+CrashOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError

The work for handling these flags is in `report_java_out_of_memory(const char*)` in debug.cpp. However, `report_java_out_of_memory()` is only called from HotSpot's C++ code, not when a JDK Java class throws OutOfMemoryError directly.

The case relevant to our direct memory investigation is in java/nio/Bits.java line 175 in `reserveMemory()`. Currently there are 24 occurrences of "throw new OutOfMemoryError" in JDK classes. None of them respects the 4 OutOfMemoryError-handling JVM flags.

I propose we add a JNI function that calls `report_java_out_of_memory()`, possibly in jdk/internal/misc/VM.java. Then add a call to this function before the 24 places of throwing OutOfMemoryError.

Update#1: Upon discussion, we determined that those JVM flags are intended for OOMEs coming from JVM, not from Java code. Certain cases of OOMEs from Java classes could benefit from respecting those JVM flags. We have changed this RFE to only for OOME from NIO direct memory.