ADDITIONAL SYSTEM INFORMATION :
17.0.5+0-suse-150400.3.6.1.25358.2.PTF.1205916-x8664

A DESCRIPTION OF THE PROBLEM :
My code works on java 8 and java 11, but java 17 raises an exception during the TLS handshake because of a "Malformed SNI name".
It seems that the SNIMatcher I set in the socket's SSLParameters is ignored.

Just to test I set the SNIMatcher in my TLS Server in this way:
SNIMatcher myMatcher = new SNIMatcher(0) {
@Override
public boolean matches(javax.net.ssl.SNIServerName sniServerName) {
return true;
}
};

But when a client connects using SNI like "This is a custom SNI name", I get the exception below:
javax.net.ssl.SSLProtocolException: Illegal server name, type=host_name(0), name=This is a custom SNI name, value={....}

It seems it is caused by the underlying exception:
Caused by: java.lang.IllegalArgumentException: Contains non-LDH ASCII characters

It can be tested just running a TLS Server and you can try the client connection using openssl command line like this:
openssl s_client -host xxx.xxx.xxx.xxx -port 9095 -servername 'This is a custom SNI name'

REGRESSION : Last worked in version 11.0.18

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Run a TLS Server with Java 17 and override the SNIMatcher in order to accept non standard SNI names.
2) run a TLS client like this:
openssl s_client -host 10.63.133.233 -port 9095 -servername 'This is a custom SNI name'

NOTE: in my case the server is listening on the port 9095

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Like Java 11, I expect that I can override the SNI Matcher in the SSL Socket Parameters in order to accept custom SNI names.
ACTUAL -
TThreadPoolServer.java::logException:302@Thrift Error occurred during processing of message.@ org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLProtocolException: Illegal server name, type=host_name(0), name=This is a custom SNI name, value={4572696373736F6E204C5320526F6F744341204131}
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:173) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:100) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:457) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:359) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:254) ~[libthrift-0.17.0.jar:0.17.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: javax.net.ssl.SSLProtocolException: Illegal server name, type=host_name(0), name=This is a custom SNI name, value={4572696373736F6E204C5320526F6F744341204131}
at sun.security.ssl.ServerNameExtension$CHServerNamesSpec.<init>(ServerNameExtension.java:138) ~[?:?]
at sun.security.ssl.ServerNameExtension$CHServerNameConsumer.consume(ServerNameExtension.java:299) ~[?:?]
at sun.security.ssl.SSLExtension.consumeOnLoad(SSLExtension.java:609) ~[?:?]
at sun.security.ssl.SSLExtensions.consumeOnLoad(SSLExtensions.java:201) ~[?:?]
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1176) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:840) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:801) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:920) ~[?:?]
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1011) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:244) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:343) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:169) ~[libthrift-0.17.0.jar:0.17.0]
... 9 more
Caused by: java.lang.IllegalArgumentException: The encoded server name value is invalid
at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:179) ~[?:?]
at sun.security.ssl.ServerNameExtension$CHServerNamesSpec.<init>(ServerNameExtension.java:131) ~[?:?]
at sun.security.ssl.ServerNameExtension$CHServerNameConsumer.consume(ServerNameExtension.java:299) ~[?:?]
at sun.security.ssl.SSLExtension.consumeOnLoad(SSLExtension.java:609) ~[?:?]
at sun.security.ssl.SSLExtensions.consumeOnLoad(SSLExtensions.java:201) ~[?:?]
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1176) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:840) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:801) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:920) ~[?:?]
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1011) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:244) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:343) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:169) ~[libthrift-0.17.0.jar:0.17.0]
    ... 9 more
Caused by: java.lang.IllegalArgumentException: Contains non-LDH ASCII characters
at java.net.IDN.toASCIIInternal(IDN.java:297) ~[?:?]
at java.net.IDN.toASCII(IDN.java:123) ~[?:?]
at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:175) ~[?:?]
at sun.security.ssl.ServerNameExtension$CHServerNamesSpec.<init>(ServerNameExtension.java:131) ~[?:?]
at sun.security.ssl.ServerNameExtension$CHServerNameConsumer.consume(ServerNameExtension.java:299) ~[?:?]
at sun.security.ssl.SSLExtension.consumeOnLoad(SSLExtension.java:609) ~[?:?]
at sun.security.ssl.SSLExtensions.consumeOnLoad(SSLExtensions.java:201) ~[?:?]
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1176) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:840) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:801) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:920) ~[?:?]
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1011) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:244) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:343) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:169) ~[libthrift-0.17.0.jar:0.17.0]
    ... 9 more


FREQUENCY : always