CDS should be disabled if JvmtiExport::should_post_class_file_load_hook() is true

If JvmtiExport::should_post_class_file_load_hook() is true, a Java agent may modify the classfile bytes of any class before the class is parsed by ClassFileParser::parseClassFile. If CDS is enabled in this situation, the JVM might load classes from the shared archive in an way that's unexpected by the JVMTI agent.

In JDK8 and before, we disable CDS altogether under these conditions:

bool FileMapInfo::initialize() {
  ...
  if (JvmtiExport::can_modify_any_class() || JvmtiExport::can_walk_any_space()) {
    fail_continue("Tool agent requires sharing to be disabled.");
    return false;
  }

But this check has been removed in the fix for JDK-8054386, replaced by code later on to keep using CDS, but just change the RO space to be mapped RW.

To keep CDS safe w.r.t JvmtiExport::should_post_class_file_load_hook(), we should restore the hook in FileMapInfo::initialize() to be something like:

  if (JvmtiExport::should_post_class_file_load_hook()) {
    fail_continue("Tool agent requires sharing to be disabled.");
    return false;
  }

Adding this checking should still keep the Java debugging working. I've traced the code in a C debugger. At least jdb does not set the _should_post_class_file_load_hook flag. I would guess other debuggers such as eclipse normally would not set this flag.

$ gdb --args java -classpath /home/iklam/tmp -Xdebug -Xrunjdwp:transport=dt_socket,address=127.0.0.1:12345,suspend=y -Xshare:on Foo

(gdb) b FileMapInfo::initialize
Breakpoint 2 at 0x7ffff65477d1: file /home/iklam/jdk/appcdsv2/hotspot/src/share/vm/memory/filemap.cpp, line 835.
(gdb) r

Breakpoint 2, FileMapInfo::initialize (this=0x7ffff0331a50) at /home/iklam/jdk/appcdsv2/hotspot/src/share/vm/memory/filemap.cpp:835
835 bool FileMapInfo::initialize() {
(gdb) p JvmtiExport::_should_post_class_file_load_hook
$2 = false