-
Bug
-
Resolution: Fixed
-
P4
-
8, 9
-
b08
-
x86_64
-
linux
FULL PRODUCT VERSION :
java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b14)
OpenJDK 64-Bit Server VM (build 25.121-b14, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Linux 4.10.9-200.fc25.x86_64 #1 SMP Mon Apr 10 14:48:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
A DESCRIPTION OF THE PROBLEM :
Loading a PEM CRL list with 110 CRLs and 200MB in size is 25 minutes in my laptop. Using a DER list is 12 seconds.
The main reason is the X509Factory.java class uses a buffer of 2048 to store the base64 data and if it needs more it adds chucks by 1024 each (using Arrays.copyOf). That means to create a 10MB pem it calls to Arrays.copyOf 10000 times (allocate and copy). That's the reason to be painfully slow.
See here:
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/security/provider/X509Factory.java#l482
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Just download a big CRL, convert to PEM and try the generateCRLs method. See the difference between PEM and DER.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Load the PEM crl in a reasonable time. My patch below does the load of 200MB in 83s and a 33MB CRL in 14s.
ACTUAL -
A 33MB CRL is 6 minutes in time to be loaded. A 200MB file with several CRLs is 25 minutes.
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.io.File;
import java.io.FileInputStream;
import java.util.Collection;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
public class LoadCerts {
public static void main(String[] args) throws Exception {
if (args.length != 1) {
throw new IllegalArgumentException("The first argument should be the PEM file.");
}
File f = new File(args[0]);
if (!f.exists() || !f.canRead()) {
throw new IllegalArgumentException(String.format("Invalid file %s", args[0]));
}
try (FileInputStream is = new FileInputStream(f)) {
CertificateFactory cf = CertificateFactory.getInstance("X.509");
long start = System.currentTimeMillis();
Collection crls = cf.generateCRLs(is);
long end = System.currentTimeMillis();
System.out.println(String.format("Loaded %s certificates from %s in %d seconds.", crls.size(), args[0], (end - start) / 1000L));
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Consider this patch, it uses the new java8 Base64.Decoder to just use an InputStream over the Base64 (avoiding intermediary buffers). It loads a 33MB crl in 14s instead of 6 minutes.
--- /home/rmartinc/jdk8/jdk/src/share/classes/sun/security/provider/X509Factory.java 2017-04-26 09:23:14.807876940 +0200
+++ X509Factory.java 2017-04-26 13:28:51.687191779 +0200
@@ -546,22 +546,17 @@
}
// Step 3: Read the data
- while (true) {
- int next = is.read();
- if (next == -1) {
- throw new IOException("Incomplete data");
- }
- if (next != '-') {
- data[pos++] = (char)next;
- if (pos >= data.length) {
- data = Arrays.copyOf(data, data.length+1024);
- }
- } else {
- break;
- }
- }
+ InputStream b64is = Base64.getMimeDecoder().wrap(is);
+ ByteArrayOutputStream bout = new ByteArrayOutputStream(2048);
+ c = b64is.read();
+ bout.write(c);
+ readBERInternal(b64is, bout, c);
// Step 4: Consume the footer
+ c = is.read();
+ while (c != '-' && c != -1) {
+ c = is.read();
+ }
StringBuffer footer = new StringBuffer("-");
while (true) {
int next = is.read();
@@ -575,7 +570,7 @@
checkHeaderFooter(header.toString(), footer.toString());
- return Base64.getMimeDecoder().decode(new String(data, 0, pos));
+ return bout.toByteArray();
}
}
java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b14)
OpenJDK 64-Bit Server VM (build 25.121-b14, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Linux 4.10.9-200.fc25.x86_64 #1 SMP Mon Apr 10 14:48:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
A DESCRIPTION OF THE PROBLEM :
Loading a PEM CRL list with 110 CRLs and 200MB in size is 25 minutes in my laptop. Using a DER list is 12 seconds.
The main reason is the X509Factory.java class uses a buffer of 2048 to store the base64 data and if it needs more it adds chucks by 1024 each (using Arrays.copyOf). That means to create a 10MB pem it calls to Arrays.copyOf 10000 times (allocate and copy). That's the reason to be painfully slow.
See here:
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/security/provider/X509Factory.java#l482
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Just download a big CRL, convert to PEM and try the generateCRLs method. See the difference between PEM and DER.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Load the PEM crl in a reasonable time. My patch below does the load of 200MB in 83s and a 33MB CRL in 14s.
ACTUAL -
A 33MB CRL is 6 minutes in time to be loaded. A 200MB file with several CRLs is 25 minutes.
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.io.File;
import java.io.FileInputStream;
import java.util.Collection;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
public class LoadCerts {
public static void main(String[] args) throws Exception {
if (args.length != 1) {
throw new IllegalArgumentException("The first argument should be the PEM file.");
}
File f = new File(args[0]);
if (!f.exists() || !f.canRead()) {
throw new IllegalArgumentException(String.format("Invalid file %s", args[0]));
}
try (FileInputStream is = new FileInputStream(f)) {
CertificateFactory cf = CertificateFactory.getInstance("X.509");
long start = System.currentTimeMillis();
Collection crls = cf.generateCRLs(is);
long end = System.currentTimeMillis();
System.out.println(String.format("Loaded %s certificates from %s in %d seconds.", crls.size(), args[0], (end - start) / 1000L));
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Consider this patch, it uses the new java8 Base64.Decoder to just use an InputStream over the Base64 (avoiding intermediary buffers). It loads a 33MB crl in 14s instead of 6 minutes.
--- /home/rmartinc/jdk8/jdk/src/share/classes/sun/security/provider/X509Factory.java 2017-04-26 09:23:14.807876940 +0200
+++ X509Factory.java 2017-04-26 13:28:51.687191779 +0200
@@ -546,22 +546,17 @@
}
// Step 3: Read the data
- while (true) {
- int next = is.read();
- if (next == -1) {
- throw new IOException("Incomplete data");
- }
- if (next != '-') {
- data[pos++] = (char)next;
- if (pos >= data.length) {
- data = Arrays.copyOf(data, data.length+1024);
- }
- } else {
- break;
- }
- }
+ InputStream b64is = Base64.getMimeDecoder().wrap(is);
+ ByteArrayOutputStream bout = new ByteArrayOutputStream(2048);
+ c = b64is.read();
+ bout.write(c);
+ readBERInternal(b64is, bout, c);
// Step 4: Consume the footer
+ c = is.read();
+ while (c != '-' && c != -1) {
+ c = is.read();
+ }
StringBuffer footer = new StringBuffer("-");
while (true) {
int next = is.read();
@@ -575,7 +570,7 @@
checkHeaderFooter(header.toString(), footer.toString());
- return Base64.getMimeDecoder().decode(new String(data, 0, pos));
+ return bout.toByteArray();
}
}