Unable to use custom SSLEngine with default TrustManagerFactory after updating to JDK 11 b21

ADDITIONAL SYSTEM INFORMATION :
# java -version
java version "11-ea" 2018-09-25
Java(TM) SE Runtime Environment 18.9 (build 11-ea+21)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11-ea+21, mixed mode)


A DESCRIPTION OF THE PROBLEM :
Hi all,

I just tried to run netty[1] testsuite with the latest jdk11 EA release (21) and saw some class-cast-exception with our custom SSLEngine implementation:


Caused by: java.lang.ClassCastException: class io.netty.handler.ssl.OpenSslEngine cannot be cast to class sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is in module java.base of loader 'bootstrap')
at java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
... 27 more


This change seems to be related to:
http://hg.openjdk.java.net/jdk/jdk11/rev/68fa3d4026ea

I think you miss an instanceof check here in SSLAlgorithmConstraints before try to cast to SSLEngineImpl, as otherwise it will be impossible to use custom implementations of SSLEngine (which we have in netty) with the default TrustManagerFactory.

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create custom SSLEngine implementation and invoke sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(...) with it as argument.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Should not throw ClassCastException, X509TrustManagerImpl should not depend on the fact that SSLEngineImpl is used.
ACTUAL -
Throws an ClassCastException like this:

Caused by: java.lang.ClassCastException: class io.netty.handler.ssl.OpenSslEngine cannot be cast to class sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is in module java.base of loader 'bootstrap')
at java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
... 27 more


CUSTOMER SUBMITTED WORKAROUND :
There is no workaround, like stated it worked in ea19 last. That said the fix should be as simple as adding an instanceof SSLEngineImpl check before trying to cast.

FREQUENCY : always