When testing wildcard Java flags like -Xlog:aot*=debug, via VM_OPTIONS or JAVA_OPTIONS, jtreg crashes with the following:
Error: Unexpected exception occurred! java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
at java.base/java.util.Properties.loadConvert(Properties.java:664)
at java.base/java.util.Properties.load0(Properties.java:458)
at java.base/java.util.Properties.load(Properties.java:384)
at com.sun.javatest.regtest.config.JDK.loadProperties(JDK.java:688)
at com.sun.javatest.regtest.config.JDK.execGetProperties(JDK.java:665)
at com.sun.javatest.regtest.config.JDK.getProperties(JDK.java:436)
at com.sun.javatest.regtest.config.RegressionContext.<init>(RegressionContext.java:78)
at com.sun.javatest.regtest.config.RegressionParameters.initExprContext(RegressionParameters.java:206)
at com.sun.javatest.regtest.tool.Tool.createParameters(Tool.java:1872)
at com.sun.javatest.regtest.tool.Tool.run(Tool.java:1348)
at com.sun.javatest.regtest.tool.Tool.run(Tool.java:1120)
at com.sun.javatest.regtest.tool.Tool.main(Tool.java:153)
at com.sun.javatest.regtest.Main.main(Main.java:46)

This is most likely due to the logs interleaving with the actual properties as both are printed via System.out. Thus, when loading the properties, the syntax does not correspond to the expected properties format and it manifests in this failure.

This is observed with both jtreg 7 and jtreg 8. One way to work around is to hardcode the flags into the test files, which is not viable when one wants to experimentally run a batch of tests with specific flags. Another is to log to a file instead, though this adds some overhead. The most viable workaround is to explicitly redirect the logs to stderr.