Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8223061

Missing Read Timeout for CRL load

XMLWordPrintable

      ADDITIONAL SYSTEM INFORMATION :
      Sample stuck thread (from thread dump):

      Thread 73765: (state = IN_NATIVE)
       - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
       - java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame)
       - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame)
       - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame)
       - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)
       - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=286 (Compiled frame)
       - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
       - sun.net.www.http.HttpClient.parseHTTPHeader(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) @bci=51, line=704 (Interpreted frame)
       - sun.net.www.http.HttpClient.parseHTTP(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) @bci=56, line=647 (Interpreted frame)
       - sun.net.www.protocol.http.HttpURLConnection.getInputStream0() @bci=327, line=1569 (Interpreted frame)
       - sun.net.www.protocol.http.HttpURLConnection.getInputStream() @bci=52, line=1474 (Interpreted frame)
       - sun.security.provider.certpath.URICertStore.engineGetCRLs(java.security.cert.CRLSelector) @bci=151, line=396 (Interpreted frame)
       - java.security.cert.CertStore.getCRLs(java.security.cert.CRLSelector) @bci=5, line=181 (Interpreted frame)
       - sun.security.provider.certpath.DistributionPointFetcher.getCRL(sun.security.x509.URIName) @bci=92, line=245 (Interpreted frame)
       - sun.security.provider.certpath.DistributionPointFetcher.getCRLs(java.security.cert.X509CRLSelector, sun.security.x509.X509CertImpl, sun.security.x509.DistributionPoint, boolean[], boolean, java.security.PublicKey, java.security.cert.X509Certificate, java.lang.String, java.util.List, java.util.Set, java.util.Date) @bci=199, line=189 (Interpreted frame)
       - sun.security.provider.certpath.DistributionPointFetcher.getCRLs(java.security.cert.X509CRLSelector, boolean, java.security.PublicKey, java.security.cert.X509Certificate, java.lang.String, java.util.List, boolean[], java.util.Set, java.util.Date) @bci=165, line=121 (Interpreted frame)
       - sun.security.provider.certpath.RevocationChecker.checkCRLs(java.security.cert.X509Certificate, java.security.PublicKey, java.security.cert.X509Certificate, boolean, boolean, java.util.Set, java.util.Set) @bci=449, line=552 (Interpreted frame)
       - sun.security.provider.certpath.RevocationChecker.checkCRLs(java.security.cert.X509Certificate, java.util.Collection, java.util.Set, java.security.PublicKey, boolean) @bci=16, line=465 (Interpreted frame)
       - sun.security.provider.certpath.RevocationChecker.check(java.security.cert.X509Certificate, java.util.Collection, java.security.PublicKey, boolean) @bci=152, line=367 (Interpreted frame)
       - sun.security.provider.certpath.RevocationChecker.check(java.security.cert.Certificate, java.util.Collection) @bci=14, line=337 (Interpreted frame)
       - sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(java.security.cert.CertPath, java.util.List, java.util.List) @bci=316, line=125 (Interpreted frame)
       - sun.security.provider.certpath.PKIXCertPathValidator.validate(java.security.cert.TrustAnchor, sun.security.provider.certpath.PKIX$ValidatorParams) @bci=333, line=219 (Interpreted frame)
       - sun.security.provider.certpath.PKIXCertPathValidator.validate(sun.security.provider.certpath.PKIX$ValidatorParams) @bci=217, line=140 (Interpreted frame)
       - sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(java.security.cert.CertPath, java.security.cert.CertPathParameters) @bci=7, line=79 (Interpreted frame)
       - java.security.cert.CertPathValidator.validate(java.security.cert.CertPath, java.security.cert.CertPathParameters) @bci=6, line=292 (Interpreted frame)
       - sun.security.validator.PKIXValidator.doValidate(java.security.cert.X509Certificate[], java.security.cert.PKIXBuilderParameters) @bci=34, line=347 (Interpreted frame)
       - sun.security.validator.PKIXValidator.engineValidate(java.security.cert.X509Certificate[], java.util.Collection, java.security.AlgorithmConstraints, java.lang.Object) @bci=278, line=260 (Interpreted frame)
       - sun.security.validator.Validator.validate(java.security.cert.X509Certificate[], java.util.Collection, java.security.AlgorithmConstraints, java.lang.Object) @bci=6, line=260 (Interpreted frame)
       - sun.security.ssl.X509TrustManagerImpl.validate(sun.security.validator.Validator, java.security.cert.X509Certificate[], java.security.AlgorithmConstraints, java.lang.String) @bci=10, line=324 (Interpreted frame)
       - sun.security.ssl.X509TrustManagerImpl.checkTrusted(java.security.cert.X509Certificate[], java.lang.String, java.net.Socket, boolean) @bci=218, line=229 (Interpreted frame)
       - sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(java.security.cert.X509Certificate[], java.lang.String, java.net.Socket) @bci=5, line=124 (Interpreted frame)
       - sun.security.ssl.ClientHandshaker.serverCertificate(sun.security.ssl.HandshakeMessage$CertificateMsg) @bci=163, line=1491 (Interpreted frame)
       - sun.security.ssl.ClientHandshaker.processMessage(byte, int) @bci=237, line=216 (Interpreted frame)
       - sun.security.ssl.Handshaker.processLoop() @bci=96, line=979 (Compiled frame)
       - sun.security.ssl.Handshaker.process_record(sun.security.ssl.InputRecord, boolean) @bci=24, line=914 (Compiled frame)
       - sun.security.ssl.SSLSocketImpl.readRecord(sun.security.ssl.InputRecord, boolean) @bci=357, line=1062 (Compiled frame)
       - sun.security.ssl.SSLSocketImpl.performInitialHandshake() @bci=84, line=1375 (Interpreted frame)
       - sun.security.ssl.SSLSocketImpl.startHandshake(boolean) @bci=13, line=1403 (Interpreted frame)
       - sun.security.ssl.SSLSocketImpl.startHandshake() @bci=2, line=1387 (Interpreted frame)
       - org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(java.net.Socket, java.lang.String, int, org.apache.http.protocol.HttpContext) @bci=258, line=396 (Interpreted frame)
       - org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(int, java.net.Socket, org.apache.http.HttpHost, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.protocol.HttpContext) @bci=195, line=355 (Interpreted frame)



      A DESCRIPTION OF THE PROBLEM :
      When JVM is run with the flags:
      -Dcom.sun.security.enableCRLDP=true -Dcom.sun.net.ssl.checkRevocation=true

      To enable CRL verification, CRL request is being issued without any read-timeout. Implication of this is that given some network condition / bad backend behaviour, network connections might be blocked indifinitely.

      We've seen that happen in production systems with CRL request being blocked indefinitely against http://crl3.digicert.com/DigiCertGlobalRootCA.crl

      Offending code can be found in URICertStore::engineGetCRLs:
                  URLConnection connection = uri.toURL().openConnection();
                  if (lastModified != 0) {
                      connection.setIfModifiedSince(lastModified);
                  }
                  long oldLastModified = lastModified;
                  connection.setConnectTimeout(CRL_CONNECT_TIMEOUT);
      >> missing setReadTimeout() <<
                  try (InputStream in = connection.getInputStream()) {

      Would probably make sense to use the same connect timeout also as read timeout to ensure read requests always return at some point.

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Given production system doing ALOT of secure connections requiring CRL verifications, under certain network / CRL-backend system condition one would see such threads get blocked.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      Such network request fails on timeout after some max amount of time (can probably use the same CONNECT timeout)
      ACTUAL -
      Thread/System might get stuck indefinitely.

      CUSTOMER SUBMITTED WORKAROUND :
      None.

      FREQUENCY : rarely


            Unassigned Unassigned
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: