When setting the System property javax.net.ssl.keyStoreType or the Security property keystore.type to 'PKCS11', an exception is thrown:

Caused by: java.security.KeyManagementException: if keyStoreType is PKCS11, then keyStore must be NONE
at java.base/sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(SSLContextImpl.java:942)
at java.base/sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:1112)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at java.base/java.security.Provider.newInstanceUtil(Provider.java:155)
at java.base/java.security.Provider$Service.newInstance(Provider.java:1820)
... 10 more

The reason is that when any of the previous properties is set to the 'PKCS11', javax.net.ssl.keyStore is expected to be 'NONE' and an empty string or the property not set does not count. See here: http://hg.openjdk.java.net/jdk/jdk/file/085463e75652/src/java.base/share/classes/sun/security/ssl/SSLContextImpl.java#l1000

In the following line, the check condition is properly handled: http://hg.openjdk.java.net/jdk/jdk/file/085463e75652/src/java.base/share/classes/sun/security/ssl/SSLContextImpl.java#l1010 The reason is that 'NONE' is expected only if the property was set to something.

A test will be submitted as part of the fix proposal.