-
Bug
-
Resolution: Duplicate
-
P3
-
7u21
FULL PRODUCT VERSION :
java version " 1.7.0_21 "
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Linux 300427-app01.netbiscuits.de 2.6.18-348.4.1.el5 #1 SMP Fri Mar 22 05:41:51 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
A DESCRIPTION OF THE PROBLEM :
We have a Web Application running on a JBoss AS 7.0.2.Final server with 6.5 GB heap memory. One of the tasks this application performs is downsizing arbitrary images (the only limit is 16 MPixel or 10 MB filesize). After updating to jdk1.7.0_21 on our production environment a whole cluster was failing because it ran into
java.lang.OutOfMemoryError: Java heap space
problems.
After analyzing a heap dump we saw that there reside a lot of JPEGImageReader objects in the heap that hold references to byte arrays (inside ByteInterleavedRaster) which in sum took up nearly 60% of the heap (another 29% were reserved by ImageIO's MemoryCache).
Especially interesting is that the GC of all remaining JPEGImageReader instances is the Native Stack.
The GC log also showed that after a short period it starts to only perform Full GC runs in a row (usually not more than 2-3s between to runs and one run taking 1s+).
I assume that the changes introduced for CVE-2013-2430 in native code now keep a reference to the instance of JPEGImageReader in the native code which is not freed up when calling the dispose() method of the JPEGImageReader and therefore prevent or at least remarkably slow down garbage collection of these unused objects.
After switching back to jdk1.7.0_17 it worked again as usual.
REGRESSION. Last worked in version 7u17
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The problem is rather hard to reproduce outside our server application which is performing also other tasks concurrently.
The source code below can sometimes reproduce the behavior although not reliably. It still can be always reproduced in our server environment.
Required: A large JPEG file (I used an 8 MPixel image with 4.36 MB and an 5 MPixel image with 5.49 MB).
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The GC should get busy but should still handle this OK.
ACTUAL -
The GC is totally overloaded, running Full GC permanently at some point in time no longer being able to free more than 1 MB per run as it cannot collect the JPEGImageReader objects.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.lang.OutOfMemoryError: Java heap space
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
// Run this with -verbose:gc -Xmx1G
public static void main(String[] args) throws IOException {
ImageIO.setUseCache(false);
int threadCount = 20; // This varies depending on the systems performance
ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++) {
executorService.submit(new ImageLoader());
}
}
static class ImageLoader implements Runnable {
@Override
public void run() {
Path path = FileSystems.getDefault().getPath( " your/path/to " , " large.jpg " );
byte[] source = null;
try {
source = Files.readAllBytes(path);
} catch (IOException e) {
}
while (true) {
try {
ImageReader reader = ImageIO.getImageReadersByFormatName( " jpeg " ).next();
ImageInputStream iis = ImageIO.createImageInputStream(new ByteArrayInputStream(source));
reader.setInput(iis);
BufferedImage bi = reader.read(0);
IIOImage container = new IIOImage(bi, null, null);
((ImageInputStream) reader.getInput()).reset();
reader.dispose();
} catch (IOException ignore) { }
}
}
}
---------- END SOURCE ----------
java version " 1.7.0_21 "
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Linux 300427-app01.netbiscuits.de 2.6.18-348.4.1.el5 #1 SMP Fri Mar 22 05:41:51 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
A DESCRIPTION OF THE PROBLEM :
We have a Web Application running on a JBoss AS 7.0.2.Final server with 6.5 GB heap memory. One of the tasks this application performs is downsizing arbitrary images (the only limit is 16 MPixel or 10 MB filesize). After updating to jdk1.7.0_21 on our production environment a whole cluster was failing because it ran into
java.lang.OutOfMemoryError: Java heap space
problems.
After analyzing a heap dump we saw that there reside a lot of JPEGImageReader objects in the heap that hold references to byte arrays (inside ByteInterleavedRaster) which in sum took up nearly 60% of the heap (another 29% were reserved by ImageIO's MemoryCache).
Especially interesting is that the GC of all remaining JPEGImageReader instances is the Native Stack.
The GC log also showed that after a short period it starts to only perform Full GC runs in a row (usually not more than 2-3s between to runs and one run taking 1s+).
I assume that the changes introduced for CVE-2013-2430 in native code now keep a reference to the instance of JPEGImageReader in the native code which is not freed up when calling the dispose() method of the JPEGImageReader and therefore prevent or at least remarkably slow down garbage collection of these unused objects.
After switching back to jdk1.7.0_17 it worked again as usual.
REGRESSION. Last worked in version 7u17
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The problem is rather hard to reproduce outside our server application which is performing also other tasks concurrently.
The source code below can sometimes reproduce the behavior although not reliably. It still can be always reproduced in our server environment.
Required: A large JPEG file (I used an 8 MPixel image with 4.36 MB and an 5 MPixel image with 5.49 MB).
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The GC should get busy but should still handle this OK.
ACTUAL -
The GC is totally overloaded, running Full GC permanently at some point in time no longer being able to free more than 1 MB per run as it cannot collect the JPEGImageReader objects.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.lang.OutOfMemoryError: Java heap space
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
// Run this with -verbose:gc -Xmx1G
public static void main(String[] args) throws IOException {
ImageIO.setUseCache(false);
int threadCount = 20; // This varies depending on the systems performance
ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++) {
executorService.submit(new ImageLoader());
}
}
static class ImageLoader implements Runnable {
@Override
public void run() {
Path path = FileSystems.getDefault().getPath( " your/path/to " , " large.jpg " );
byte[] source = null;
try {
source = Files.readAllBytes(path);
} catch (IOException e) {
}
while (true) {
try {
ImageReader reader = ImageIO.getImageReadersByFormatName( " jpeg " ).next();
ImageInputStream iis = ImageIO.createImageInputStream(new ByteArrayInputStream(source));
reader.setInput(iis);
BufferedImage bi = reader.read(0);
IIOImage container = new IIOImage(bi, null, null);
((ImageInputStream) reader.getInput()).reset();
reader.dispose();
} catch (IOException ignore) { }
}
}
}
---------- END SOURCE ----------
- duplicates
-
JDK-8020983 OutOfMemoryError caused by non garbage collected JPEGImageWriter Instances
- Closed