Default configuration should disallow loading agents

As part of making the platform more secure by default, and improving the integrity of the platform, then we need to re-examine the ability to load arbitrary code (both native and java) into a running VM with the attach mechanism.

This issue tracks changing the VM side of the attach mechanism to disallow the "load" command by default. The "load" command is what the VirtualMachine.loadAgentXXX methods use to load java and JVM TI agents into the target VM. A non-manageable command line (-XX:+EnableDynamicAgentLoading or better name) will be needed to allow opt-in and allow agents to be loaded. Note that the VM already has -XX:+DisableAttachMechanism to completely disable the attach mechanism but that disables it completely and prevents the use of the cooperative troubleshooting tools.

A few notes on the proposal:

1. It should have no impact on command-line/troubleshooting tools.

2. It should have no impact on tools that start the JMX agent with the attach mechanism.

3. No impact on the JVM TI or java.lang.instrument specifications as Java SE does not specify the mechanism, it just allows for the possibility of agents being loaded in a running VM.

4 The changes to implement this are likely to be small and low-risk. The main thing is to make sure that the error on the attach API side is useful. A small number of existing tests will need to be updated to run with the new XX option.

5. The change should only impact a small number of tools but it will need to be documented in the JDK 9 release notes.


Also a few notes on some of the alternatives that have explored:

1. Do not resolve the jdk.attach module by default. Unfortunately it's not possible to do this in a reliable way. One reason is that several tools with APIs require jdk.attach. Service binding also complicates this. In short, it's not feasible without completely changing the policy for root modules in JEP 261. Even then, it runs into the same issues as #3 below.

2. Degrading the redefineXXXX and retransformClasses functions/methods so that it can't be used to redefine modules or classes defined to the boot or platform loaders. This helps preserve the integrity of the platform modules but is not completely robust without disallowing core modules to export internal packages to tool modules. Even if these issues are solved then it would still require a command line option to allow the platform classes and modules to be transformed by late binding agents.

3. Disable "self attach". The attach API was never intended to be used this way. It was suggested during the original development to disallow this and has also been looked at several times since then. To date then efforts here have been futile (sneaky libraries will just start a new VM that attaches back to the parent).