Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8214129

SSL session resumption/SNI with TLS1.2 causes StackOverflowError

    XMLWordPrintable

Details

    • b24
    • x86_64
    • linux
    • Verified

    Backports

      Description

        ADDITIONAL SYSTEM INFORMATION :
        Tested on Fedora 29 and Debian Jessie
        With Java 11.0.1 (OpenJDK + Oracle)

        A DESCRIPTION OF THE PROBLEM :
        We came across a major Java 11 bug within the SSL session resumption/Server name indication code
        causing StackOverflowErrors when using the built-in HTTPClient (or HttpURLConnection)
        together with HTTPS and TLS version 1.2.

        We could observe the problem on production systems which are making lots of
        HTTPS service calls to clustered endpoints. Depending on the amount of requests
        and the thread stack size, the StackOverflowErrors are showing up after a few days
        uptime and we have to restart the JVMs.

        I could track down the problem to the class sun.security.ssl.SSLSessionImpl,
        where a list of requestedServerNames from the HandshakeContext is put into an unmodifiable list
        again and again, when the same session is resumed, thus ending up with a nested
        list exceeding the thread stack size, when accessed in
        sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce().

        I attached some test code, which can reproduce the problem using raw sockets.
        The bug seems to be new in Java 11 and only applying to TLS 1.2.

        REGRESSION : Last worked in version 10

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Make thousands of HTTPS requests using HTTPClient/HttpURLConnections to a clustered server using TLSv1.2 (probably it's important that no session tickets are used and the endpoint is clustered, so that you get session id cache misses on the server and the client creates new ones).


        EXPECTED VERSUS ACTUAL BEHAVIOR :
        EXPECTED -
        Should just work.
        ACTUAL -
        Exception in thread "Thread-1" java.lang.StackOverflowError
        at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
        at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
        ...
        at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
        at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
        at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
        at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
        at java.base/sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce(ServerNameExtension.java:228)
        at java.base/sun.security.ssl.SSLExtension.produce(SSLExtension.java:532)
        at java.base/sun.security.ssl.SSLExtensions.produce(SSLExtensions.java:228)
        at java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:648)
        at java.base/sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:515)
        at java.base/sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:104)
        at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:228)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
        at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:970)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:942)

        ---------- BEGIN SOURCE ----------
        import java.io.BufferedReader;
        import java.io.IOException;
        import java.io.InputStream;
        import java.io.InputStreamReader;
        import java.io.OutputStream;
        import java.net.SocketTimeoutException;
        import java.util.ArrayList;
        import java.util.Collection;
        import java.util.List;
        import java.util.concurrent.atomic.AtomicInteger;

        import javax.net.ssl.SNIHostName;
        import javax.net.ssl.SNIMatcher;
        import javax.net.ssl.SNIServerName;
        import javax.net.ssl.SSLParameters;
        import javax.net.ssl.SSLServerSocket;
        import javax.net.ssl.SSLServerSocketFactory;
        import javax.net.ssl.SSLSocket;
        import javax.net.ssl.SSLSocketFactory;

        /**
         * Test showing StackOverflowError within
         * sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce().
         *
         * The test should be started with minimal thread stack size and increased
         * stacktrace depth:
         *
         * <pre>
         * java -Xss140k -XX:MaxJavaStackTraceDepth=2000 SNIBugTest
         * </pre>
         *
         * Using these settings the test should fail after about 778 iterations.
         *
         * By default the test uses the hostname localhost and localhost.localdomain as
         * SNI name. By default it creates a keystore with a self-signed certificate.
         * You can change the host names and provide custom javax.net.ssl.XXX settings
         * if you want to test with an alternative domain or another
         * keystore/truststore.
         */
        public class SNIBugTest {

            static String hostName = "localhost";
            static String sniHostName = "localhost.localdomain";

            static int maxRequests = 1000;

            static volatile int serverPort;
            static AtomicInteger requestCount = new AtomicInteger();

            public static void main(String[] args) throws Exception {

                if (System.getProperty("javax.net.ssl.keystore") == null) {
                    createKeystore();
                    System.setProperty("javax.net.ssl.keyStore", "testkeystore");
                    System.setProperty("javax.net.ssl.keyStorePassword", "passphrase");
                    System.setProperty("javax.net.ssl.trustStore", "testkeystore");
                    System.setProperty("javax.net.ssl.trustStorePassword",
                            "passphrase");
                }

                ServerThread server = new ServerThread();
                server.start();

                while (serverPort == 0) {
                    Thread.sleep(100);
                }

                ClientThread client = new ClientThread();
                client.start();
                client.join();

                server.interrupt();
            }

            static void createKeystore() throws Exception {

                ProcessBuilder builder = new ProcessBuilder("keytool", "-genkey",
                        "-alias", "dummy", "-keyalg", "RSA", "-keysize", "2048",
                        "-sigalg", "SHA256withRSA", "-validity", "365", "-keypass",
                        "passphrase", "-keystore", "testkeystore", "-storepass",
                        "passphrase", "-dname", "CN=localhost.localdomain, OU=Dummy,"
                        + " O=Dummy, L=Cupertino, ST=CA, C=US");
                builder.redirectErrorStream(true);
                Process process = builder.start();
                int exitCode = process.waitFor();
                if (exitCode != 0) {
                    BufferedReader reader = new BufferedReader(
                            new InputStreamReader(process.getInputStream()));
                    String line;
                    while ((line = reader.readLine()) != null) {
                        System.out.println(line);
                    }
                }
            }

            static class ClientThread extends Thread {

                @Override
                public void run() {
                    SSLSocketFactory factory = (SSLSocketFactory)
                            SSLSocketFactory.getDefault();
                    for (int i = 0; i < maxRequests; i++) {
                        try (SSLSocket sslSocket = (SSLSocket) factory
                                .createSocket(hostName, serverPort)) {
                            SNIHostName serverName = new SNIHostName(sniHostName);
                            List<SNIServerName> serverNames = new ArrayList<>(1);
                            serverNames.add(serverName);
                            SSLParameters params = sslSocket.getSSLParameters();
                            params.setServerNames(serverNames);
                            sslSocket.setSSLParameters(params);
                            OutputStream out = sslSocket.getOutputStream();
                            InputStream in = sslSocket.getInputStream();
                            out.write(0);
                            in.read();
                        } catch (IOException x) {
                            x.printStackTrace();
                        }
                    }
                }
            }

            static class ServerThread extends Thread {

                @Override
                public void run() {

                    SSLServerSocketFactory factory = (SSLServerSocketFactory)
                            SSLServerSocketFactory.getDefault();
                    try (SSLServerSocket serverSocket = (SSLServerSocket) factory
                            .createServerSocket(0)) {
                        serverPort = serverSocket.getLocalPort();
                        serverSocket.setSoTimeout(1000);
                        // force TLS version 1.2
                        serverSocket.setEnabledProtocols(new String[] { "TLSv1.2" });
                        while (!isInterrupted()) {
                            try (SSLSocket socket =
                                    (SSLSocket) serverSocket.accept()) {
                                SNIMatcher matcher = SNIHostName
                                        .createSNIMatcher(sniHostName);
                                Collection<SNIMatcher> matchers = new ArrayList<>(1);
                                matchers.add(matcher);
                                SSLParameters params = serverSocket.getSSLParameters();
                                params.setSNIMatchers(matchers);
                                System.out.println(requestCount.incrementAndGet());
                                OutputStream out = socket.getOutputStream();
                                InputStream in = socket.getInputStream();
                                int data = in.read();
                                out.write(data);
                                // simulate failed session lookup on clustered system
                                socket.getSession().invalidate();
                            } catch (SocketTimeoutException x) {
                                continue;
                            } catch (IOException x) {
                                x.printStackTrace();
                            }
                        }
                    } catch (IOException x) {
                        x.printStackTrace();
                    }
                }
            }

        }
        ---------- END SOURCE ----------

        FREQUENCY : always


        Attachments

          1. out
            55 kB
          2. SNIBugTest.java
            5 kB

          Issue Links

            Activity

              People

                jnimeh Jamil Nimeh
                webbuggrp Webbug Group
                Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved: