To reproduce this, run the following in non-jigsaw "legacy" mode.

$ unset JIGSAW_HOME
$ gradle --info -PFULL_TEST=true :systemTests:test

The following tests fail to run:

MainLauncherTest
SandboxAppTest
ShutdownHookTest

The error is the same for all of them:

Error: Could not create the Java Virtual Machine.
-Djava.ext.dirs=/localhome/kcr/javafx/9-kcr/jfx/rt/build/sdk/lib is not supported. Use -classpath instead.

This issue was fixed in 9-dev with JDK-8150307, so either the bug was reintroduced or the fix was lost when merging FX 9 into the 9-jake sandbox.