ADDITIONAL SYSTEM INFORMATION :
Tested on Fedora 29 and Debian Jessie
With Java 11.0.1 (OpenJDK + Oracle)

A DESCRIPTION OF THE PROBLEM :
We came across a major Java 11 bug within the SSL session resumption/Server name indication code
causing StackOverflowErrors when using the built-in HTTPClient (or HttpURLConnection)
together with HTTPS and TLS version 1.2.

We could observe the problem on production systems which are making lots of
HTTPS service calls to clustered endpoints. Depending on the amount of requests
and the thread stack size, the StackOverflowErrors are showing up after a few days
uptime and we have to restart the JVMs.

I could track down the problem to the class sun.security.ssl.SSLSessionImpl,
where a list of requestedServerNames from the HandshakeContext is put into an unmodifiable list
again and again, when the same session is resumed, thus ending up with a nested
list exceeding the thread stack size, when accessed in
sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce().

I attached some test code, which can reproduce the problem using raw sockets.
The bug seems to be new in Java 11 and only applying to TLS 1.2.

REGRESSION : Last worked in version 10

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Make thousands of HTTPS requests using HTTPClient/HttpURLConnections to a clustered server using TLSv1.2 (probably it's important that no session tickets are used and the endpoint is clustered, so that you get session id cache misses on the server and the client creates new ones).


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Should just work.
ACTUAL -
Exception in thread "Thread-1" java.lang.StackOverflowError
at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
...
at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
at java.base/sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce(ServerNameExtension.java:228)
at java.base/sun.security.ssl.SSLExtension.produce(SSLExtension.java:532)
at java.base/sun.security.ssl.SSLExtensions.produce(SSLExtensions.java:228)
at java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:648)
at java.base/sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:515)
at java.base/sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:104)
at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:228)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:970)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:942)

---------- BEGIN SOURCE ----------
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.SocketTimeoutException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

import javax.net.ssl.SNIHostName;
import javax.net.ssl.SNIMatcher;
import javax.net.ssl.SNIServerName;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLServerSocket;
import javax.net.ssl.SSLServerSocketFactory;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;

/**
 * Test showing StackOverflowError within
 * sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce().
 *
 * The test should be started with minimal thread stack size and increased
 * stacktrace depth:
 *
 * <pre>
 * java -Xss140k -XX:MaxJavaStackTraceDepth=2000 SNIBugTest
 * </pre>
 *
 * Using these settings the test should fail after about 778 iterations.
 *
 * By default the test uses the hostname localhost and localhost.localdomain as
 * SNI name. By default it creates a keystore with a self-signed certificate.
 * You can change the host names and provide custom javax.net.ssl.XXX settings
 * if you want to test with an alternative domain or another
 * keystore/truststore.
 */
public class SNIBugTest {

    static String hostName = "localhost";
    static String sniHostName = "localhost.localdomain";

    static int maxRequests = 1000;

    static volatile int serverPort;
    static AtomicInteger requestCount = new AtomicInteger();

    public static void main(String[] args) throws Exception {

        if (System.getProperty("javax.net.ssl.keystore") == null) {
            createKeystore();
            System.setProperty("javax.net.ssl.keyStore", "testkeystore");
            System.setProperty("javax.net.ssl.keyStorePassword", "passphrase");
            System.setProperty("javax.net.ssl.trustStore", "testkeystore");
            System.setProperty("javax.net.ssl.trustStorePassword",
                    "passphrase");
        }

        ServerThread server = new ServerThread();
        server.start();

        while (serverPort == 0) {
            Thread.sleep(100);
        }

        ClientThread client = new ClientThread();
        client.start();
        client.join();

        server.interrupt();
    }

    static void createKeystore() throws Exception {

        ProcessBuilder builder = new ProcessBuilder("keytool", "-genkey",
                "-alias", "dummy", "-keyalg", "RSA", "-keysize", "2048",
                "-sigalg", "SHA256withRSA", "-validity", "365", "-keypass",
                "passphrase", "-keystore", "testkeystore", "-storepass",
                "passphrase", "-dname", "CN=localhost.localdomain, OU=Dummy,"
                + " O=Dummy, L=Cupertino, ST=CA, C=US");
        builder.redirectErrorStream(true);
        Process process = builder.start();
        int exitCode = process.waitFor();
        if (exitCode != 0) {
            BufferedReader reader = new BufferedReader(
                    new InputStreamReader(process.getInputStream()));
            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }

    static class ClientThread extends Thread {

        @Override
        public void run() {
            SSLSocketFactory factory = (SSLSocketFactory)
                    SSLSocketFactory.getDefault();
            for (int i = 0; i < maxRequests; i++) {
                try (SSLSocket sslSocket = (SSLSocket) factory
                        .createSocket(hostName, serverPort)) {
                    SNIHostName serverName = new SNIHostName(sniHostName);
                    List<SNIServerName> serverNames = new ArrayList<>(1);
                    serverNames.add(serverName);
                    SSLParameters params = sslSocket.getSSLParameters();
                    params.setServerNames(serverNames);
                    sslSocket.setSSLParameters(params);
                    OutputStream out = sslSocket.getOutputStream();
                    InputStream in = sslSocket.getInputStream();
                    out.write(0);
                    in.read();
                } catch (IOException x) {
                    x.printStackTrace();
                }
            }
        }
    }

    static class ServerThread extends Thread {

        @Override
        public void run() {

            SSLServerSocketFactory factory = (SSLServerSocketFactory)
                    SSLServerSocketFactory.getDefault();
            try (SSLServerSocket serverSocket = (SSLServerSocket) factory
                    .createServerSocket(0)) {
                serverPort = serverSocket.getLocalPort();
                serverSocket.setSoTimeout(1000);
                // force TLS version 1.2
                serverSocket.setEnabledProtocols(new String[] { "TLSv1.2" });
                while (!isInterrupted()) {
                    try (SSLSocket socket =
                            (SSLSocket) serverSocket.accept()) {
                        SNIMatcher matcher = SNIHostName
                                .createSNIMatcher(sniHostName);
                        Collection<SNIMatcher> matchers = new ArrayList<>(1);
                        matchers.add(matcher);
                        SSLParameters params = serverSocket.getSSLParameters();
                        params.setSNIMatchers(matchers);
                        System.out.println(requestCount.incrementAndGet());
                        OutputStream out = socket.getOutputStream();
                        InputStream in = socket.getInputStream();
                        int data = in.read();
                        out.write(data);
                        // simulate failed session lookup on clustered system
                        socket.getSession().invalidate();
                    } catch (SocketTimeoutException x) {
                        continue;
                    } catch (IOException x) {
                        x.printStackTrace();
                    }
                }
            } catch (IOException x) {
                x.printStackTrace();
            }
        }
    }

}
---------- END SOURCE ----------

FREQUENCY : always