Following tests fail because there are no required native libraries (demo/jvmti) in the modular image.

demo/jvmti/gctest/Gctest.java
demo/jvmti/heapTracker/HeapTrackerTest.java
demo/jvmti/heapViewer/HeapViewerTest.java
demo/jvmti/minst/MinstTest.java
demo/jvmti/mtrace/TraceJFrame.java
demo/jvmti/versionCheck/FailsWhenJvmtiVersionDiffers.java
demo/jvmti/waiters/WaitersTest.java

Example log:
----------messages:(3/141)----------
command: main CompiledMethodLoadTest Hello
reason: User specified action: run main CompiledMethodLoadTest Hello
elapsed time (seconds): 0.4
----------System.out:(12/1266)----------
Starting: /home/ykantser/java/9-jigsaw-m2/jdk1.9.0/bin/java -cp /home/ykantser/projects/jdk9/hs-rt/jdk/test/demo/jvmti/JTwork/classes/2/demo/jvmti/compiledMethodLoad -Dtest.classes=/home/ykantser/projects/jdk9/hs-rt/jdk/test/demo/jvmti/JTwork/classes/2/demo/jvmti/compiledMethodLoad -Xcheck:jni -Xverify:all -agentpath:/home/ykantser/java/9-jigsaw-m2/jdk1.9.0/demo/jvmti/compiledMethodLoad/lib/libcompiledMethodLoad.so Hello
Error occurred during initialization of VM
Could not find agent library /home/ykantser/java/9-jigsaw-m2/jdk1.9.0/demo/jvmti/compiledMethodLoad/lib/libcompiledMethodLoad.so in absolute path, with error: /home/ykantser/java/9-jigsaw-m2/jdk1.9.0/demo/jvmti/compiledMethodLoad/lib/libcompiledMethodLoad.so: cannot open shared object file: No such file or directory
Exit code is 1
<beginning of Error Stream buffer>

<end of buffer>
<beginning of Input Stream buffer>
Error occurred during initialization of VM
Could not find agent library /home/ykantser/java/9-jigsaw-m2/jdk1.9.0/demo/jvmti/compiledMethodLoad/lib/libcompiledMethodLoad.so in absolute path, with error: /home/ykantser/java/9-jigsaw-m2/jdk1.9.0/demo/jvmti/compiledMethodLoad/lib/libcompiledMethodLoad.so: cannot open shared object file: No such file or directory

<end of buffer>
----------System.err:(15/913)----------
java.lang.RuntimeException: Test failed - exit return code non-zero (exitStatus==1)
at DemoRun.runit(DemoRun.java:207)
at DemoRun.runit(DemoRun.java:109)
at CompiledMethodLoadTest.main(CompiledMethodLoadTest.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.javatest.regtest.MainWrapper$MainThread.run(MainWrapper.java:94)
at java.lang.Thread.run(Thread.java:745)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Test failed - exit return code non-zero (exitStatus==1)
JavaTest Message: shutting down test