Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8179389

X509Certificate generateCRLs is extremely slow using a PEM crl list

XMLWordPrintable

    • b08
    • x86_64
    • linux

      FULL PRODUCT VERSION :
      java -version
      openjdk version "1.8.0_121"
      OpenJDK Runtime Environment (build 1.8.0_121-b14)
      OpenJDK 64-Bit Server VM (build 25.121-b14, mixed mode)


      ADDITIONAL OS VERSION INFORMATION :
      Linux 4.10.9-200.fc25.x86_64 #1 SMP Mon Apr 10 14:48:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

      A DESCRIPTION OF THE PROBLEM :
      Loading a PEM CRL list with 110 CRLs and 200MB in size is 25 minutes in my laptop. Using a DER list is 12 seconds.

      The main reason is the X509Factory.java class uses a buffer of 2048 to store the base64 data and if it needs more it adds chucks by 1024 each (using Arrays.copyOf). That means to create a 10MB pem it calls to Arrays.copyOf 10000 times (allocate and copy). That's the reason to be painfully slow.

      See here:
      http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/security/provider/X509Factory.java#l482



      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Just download a big CRL, convert to PEM and try the generateCRLs method. See the difference between PEM and DER.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      Load the PEM crl in a reasonable time. My patch below does the load of 200MB in 83s and a 33MB CRL in 14s.
      ACTUAL -
      A 33MB CRL is 6 minutes in time to be loaded. A 200MB file with several CRLs is 25 minutes.

      REPRODUCIBILITY :
      This bug can be reproduced always.

      ---------- BEGIN SOURCE ----------
      import java.io.File;
      import java.io.FileInputStream;
      import java.util.Collection;
      import java.security.cert.Certificate;
      import java.security.cert.CertificateFactory;

      public class LoadCerts {

          public static void main(String[] args) throws Exception {
              if (args.length != 1) {
                  throw new IllegalArgumentException("The first argument should be the PEM file.");
              }
              File f = new File(args[0]);
              if (!f.exists() || !f.canRead()) {
                  throw new IllegalArgumentException(String.format("Invalid file %s", args[0]));
              }
              try (FileInputStream is = new FileInputStream(f)) {
                  CertificateFactory cf = CertificateFactory.getInstance("X.509");
                  long start = System.currentTimeMillis();
                  Collection crls = cf.generateCRLs(is);
                  long end = System.currentTimeMillis();
                  System.out.println(String.format("Loaded %s certificates from %s in %d seconds.", crls.size(), args[0], (end - start) / 1000L));
              }
          }
      }

      ---------- END SOURCE ----------

      CUSTOMER SUBMITTED WORKAROUND :
      Consider this patch, it uses the new java8 Base64.Decoder to just use an InputStream over the Base64 (avoiding intermediary buffers). It loads a 33MB crl in 14s instead of 6 minutes.

      --- /home/rmartinc/jdk8/jdk/src/share/classes/sun/security/provider/X509Factory.java 2017-04-26 09:23:14.807876940 +0200
      +++ X509Factory.java 2017-04-26 13:28:51.687191779 +0200
      @@ -546,22 +546,17 @@
                   }
       
                   // Step 3: Read the data
      - while (true) {
      - int next = is.read();
      - if (next == -1) {
      - throw new IOException("Incomplete data");
      - }
      - if (next != '-') {
      - data[pos++] = (char)next;
      - if (pos >= data.length) {
      - data = Arrays.copyOf(data, data.length+1024);
      - }
      - } else {
      - break;
      - }
      - }
      + InputStream b64is = Base64.getMimeDecoder().wrap(is);
      + ByteArrayOutputStream bout = new ByteArrayOutputStream(2048);
      + c = b64is.read();
      + bout.write(c);
      + readBERInternal(b64is, bout, c);
       
                   // Step 4: Consume the footer
      + c = is.read();
      + while (c != '-' && c != -1) {
      + c = is.read();
      + }
                   StringBuffer footer = new StringBuffer("-");
                   while (true) {
                       int next = is.read();
      @@ -575,7 +570,7 @@
       
                   checkHeaderFooter(header.toString(), footer.toString());
       
      - return Base64.getMimeDecoder().decode(new String(data, 0, pos));
      + return bout.toByteArray();
               }
           }


            weijun Weijun Wang
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: