Summary

Add an experimental flag -XX:+DisablePrimordialThreadGuardPages (default: false)

Problem

Non-Java language runtimes sometimes interoperate with the JVM via JNI, starting the JVM via JNI_CreateJavaVM. R is one such language runtime (with its rJava package). This leads to issues when both language runtimes attempt to do stack overflow checking on the same (initial) thread. It would be nice to have an option to disable stack overflow checking (not insert guard pages) to the stack of the initial thread of the JVM

Solution

Add an experimental flag -XX:+DisablePrimordialThreadGuardPages (default: false) to skip setting Java guard pages on the primordial thread. This only affects applications that load the JVM through the JNI invocation API on the primordial thread. None of the JDK launchers do this so the flag has no affect on them (whether enabled or disabled).

As discussed in:

http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html

a more general solution would be to enhance the JNI invocation API so that the creating thread, and any attaching thread could control the placement of stack guard pages - including omitting them altogether. That would require a much more extensive effort and is not presently in scope for consideration. The current proposal solves a very specific problem in a very simple and direct way.

Specification

+ experimental(bool DisablePrimordialThreadGuardPages, false,                    \
+         "Disable the use of stack guard pages if the JVM is loaded "       \
+         "on the primordial process thread")                                \