Background:

In complicated environments, when mixing Java and native code, it is possible for the process to be force killed by the Linux kernel with signal 11. This occurs when the stack is very close to the yellow zone, and the JVM calls a native function with a large stack frame. It may jump past the yellow zone entirely into the red zone or past the end of the stack. If past the end of the stack, since no alternative signal handler stack is present, the kernel force kills the process with almost no information (no stack trace, hs-err log, or core dump). If in the red zone, the signal handler will fault again, it will be unprotected, and resume from the red zone. Potentially the native code will go past the end of the stack. In these extreme cases almost no information is retrievable, making the root cause un-debuggable. This type of un-debuggable stack overflow crash could also happen from JVM internal threads, e.g. JDK-8366118.

Proposal:

Add diagnostic options to use sigaltstack on Linux for Hotspot created threads through two new flags. "-XX:+UseSigAltStack" and "-XX:SigAltStackSize=<size>". The former is used to enable it, and the latter would have a size starting at SIGSTKSZ. Additionally when UseSigAltStack is enabled, we should pass SA_ONSTACK when setting up signal handlers.

Due to complicated interaction with JVM's signal handling and stack-overflow detection mechanisms, as well as issues such as specifying a sufficient SigAltStackSize and potential memory overhead, UseSigAltStack and SigAltStackSize will remain as diagnostic options and will not be turned on by default. This feature is only supposed to help debug those un-debuggable crashes mentioned above.