----------messages:(5/315)----------
JDK under test: (/Users/yqi/ws/jdks/recent)
java version "9-internal"
Java(TM) SE Runtime Environment (build 9-internal+0-2016-01-08-215809.amurillo.jdk9-hs-2016-01-05-8146521-control)
Java HotSpot(TM) 64-Bit Server VM (build 9-internal+0-2016-01-08-215809.amurillo.jdk9-hs-2016-01-05-8146521-control, mixed mode)

...

----------System.out:(3/184)----------
PASSED: got expected error message with stack size of 16k
PASSED: got expected error message with stack size of 32k
FAILED: VM failed to launch with minimum allowed stack size of 100k
----------System.err:(14/774)----------
java.lang.RuntimeException: test fails
        at TooSmallStackSize.checkMinStackAllowed(TooSmallStackSize.java:130)
        at TooSmallStackSize.main(TooSmallStackSize.java:156)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:520)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:92)
        at java.lang.Thread.run(Thread.java:804)

JavaTest Message: Test threw exception: java.lang.RuntimeException: test fails
JavaTest Message: shutting down test

Looks the MinStackAllowed changed from 160K to 100K (earlier version use 160K):

java -Xss100K -version

The stack size specified is too small, Specify at least 100k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.