Delayed starting of debugging via jcmd

Summary

Allow the JDWP debugging backend to be started delayed via a diagnostic command.

Problem

The JDK already provides a mechanism to start debugging delayed in case specific exceptions are thrown or not catched via the onthrow and onuncaught JDWP options.

But a JVM might enter a 'bad' state without throwing an exception (e.g. by being in an endless loop). Some of these problematic states cannot be analyzed by the usual tools like stack- or heap-dumps, but only by debugging via a Java debugger. So it would be nice to be able to enable debugging when such a bad state is actually encountered.

Solution

Analogously to the onthrow/onuncaught options we add a boolean onjcmd option to the JDWP agentlib. If it is enabled, we don't start debugging directly, but wait for it to be requested via a diagnostic command.

The option is named onjcmd, because this is how the user will usually start the debugging, even if technically it could be started otherwise, e.g. programmatically via the DiagnosticCommandMBean or via jconsole. The latter methods are not very well known in the Java community.

Like the onthrow/onuncaught options, the onjcmd option is currently only supported when the server=y JDWP option is in effect.

In contrast to the onthrow/onuncaught options, the launch option is currently not supported by the onjcmd option.

The JDWP library exports a JNI call to start the debugging delayed when onjcmd=y was specified in the agentlib command line. If called, it initializes the debugging system on the first call. On subsequent calls it currently just returns.

This JNI call is then used in a diagnostic command, which is exported to jcmd and to the DiagnosticCommandMBean under the name VM.start_java_debugging. It currently supports no additional options.

Specification

A boolean JDWP option named onjcmd is added.

It cannot currently be combined with the server=n configuration or the launch option.

A diagnostic command named VM.start_java_debugging is added.

If the onjcmd option is enabled (onjcmd=y), the first call via the VM.start_java_debugging command tries to start the debugging backend. If this can not be completed, the feature will be disabled, but the VM is not exited. Otherwise the debugging backend is started in server mode.

Further calls to VM.start_java_debugging have currently no effect.

The permission required to start debugging via VM.start_java_debugging will be "java.lang.management.ManagementPermission" "monitor".