If we use 'jcmd %p Thread.print' or 'jcmd %p GC.heap_dump <file>' in
OnError=, (%p means the java process itself), the main java thread which
is waiting for os::fork_and_exec(cmd) will prevent HotSpot from reaching the
safepoint. It's deadlock because no safepoint means fork_and_exec() will never complete.

Technically speaking, the main java thread which is in VM should transit itself to Native before os::fork_and_exec(). It waits for the child process is equivalent to trap into native code.

Similar option OnOutOfMemoryError clearly rejects such operation.
It's because it is implemented by a VM operation
`VM_ReportJavaOutOfMemory`. When it executes, it has been at safepoint already. As a result, Attach Listener thread can't response.

com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
        at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
        at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
        at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:147)
        at sun.tools.jcmd.JCmd.main(JCmd.java:131)
2021-09-10 16:59:03

The following is an example we try to use jcmd %p to get a thread dump in OnError. -XX:+SafepointTimeout gives us information of which thread blocks us. 107553 is the thread who is executing os::fork_and_exec().

$java -Xmx50m -XX:AbortVMOnException=java.lang.OutOfMemoryError
-XX:OnError='jcmd %p Thread.print' -XX:+SafepointTimeout OomDumpExample
direct
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=/exceptions.cpp:541
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error
(/home/xxinliu/Devel/jdk/src/hotspot/share/utilities/exceptions.cpp:541), pid=107552,
tid=107553
# fatal error: Saw java.lang.OutOfMemoryError, aborting
#
# JRE version: OpenJDK Runtime Environment (18.0) (slowdebug build
18-internal+0-adhoc.xxinliu.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug
18-internal+0-adhoc.xxinliu.jdk, mixed mode, tiered, compressed oops,
compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x924e8c] Exceptions::debug_check_abort(char const*,
char const*)+0x8a
#
# No core dump will be written. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /local/home/xxinliu/JDK-2085/hs_err_pid107552.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
#
# -XX:OnError="jcmd %p Thread.print"
# Executing /bin/sh -c "jcmd 107552 Thread.print" ...
107552:
[13.045s][warning][safepoint]
[13.045s][warning][safepoint] # SafepointSynchronize::begin: Timeout
detected:
[13.045s][warning][safepoint] # SafepointSynchronize::begin: Timed out
while spinning to reach a safepoint.
[13.045s][warning][safepoint] # SafepointSynchronize::begin: Threads
which did not reach the safepoint:
[13.045s][warning][safepoint] # "main" #1 prio=5 os_prio=0 cpu=1552.12ms
elapsed=13.04s tid=0x00007f43600278e0 nid=107553 runnable
[0x00007f4369d9f000]
[13.045s][warning][safepoint] java.lang.Thread.State: RUNNABLE
[13.045s][warning][safepoint] Thread: 0x00007f43600278e0 [0x1a421]
State: _running _at_poll_safepoint 0
[13.045s][warning][safepoint] JavaThread state: _thread_in_vm
[13.045s][warning][safepoint]
[13.045s][warning][safepoint] # SafepointSynchronize::begin: (End of list)