Run jlink to create a run-time image that does not contain java.instrument, then try this:

$ myimage/bin/java -javaagent:agent.jar -version
Error occurred during initialization of VM
Could not find agent library instrument on the library path, with error: dlopen(libinstrument.dylib, 1): image not found

The "Could not find agent library" message is confusing.

A similar scenario is starting the JMX agent on the command line, for this case then a usable error is printed:

$ myimage/bin/java -Dcom.sun.management.jmxremote.port=5000 -version
Error occurred during initialization of boot layer
java.lang.module.FindException: Module jdk.management.agent not found

We need to see if we can get the -javaagent case to print a useful message like the JMX agent case.

This issue was reported on the jigsaw-dev mailing list here:
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-May/012671.html