Using the JDK build support for running a test. The JTREG... adds the obvious -javaoption to the jtreg invocation.

make test-only TEST=<my test> JTREG="JAVA_OPTIONS=-XX:+UnknownOption"

Produces output like this:

Running test '<my test>'
Unrecognized VM option 'UnknownOption'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: failed to get JDK properties for /path/to/images/jdk/bin/java -XX:MaxRAMPercentage=2 -XX:+UnknownOption -Djava.library.path=/path/to/images/test/hotspot/jtreg/native; exit code 1

...

==============================
Test summary
==============================
   TEST TOTAL PASS FAIL ERROR
   jtreg:<my test>
                                                         1 1 0 0
==============================
TEST SUCCESS