P3
FULL PRODUCT VERSION :
1.6.0_24
ADDITIONAL OS VERSION INFORMATION :
Linux 64 bits (tried on CentOS 6.3, 6.4, Redhat 6.4, all 64 bits)
EXTRA RELEVANT SYSTEM CONFIGURATION :
Running on Linux VM on AWS.
A DESCRIPTION OF THE PROBLEM :
I write a piece of java code to create 500K small files (each is 40KB) on Linux(CentOS, Redhat). The original code is like this:
package MyTest;
import java.io.*;
public class SimpleWriter {
public static void main(String[] args) {
String dir = args[0];
int fileCount = Integer.parseInt(args[1]);
String content= " @#$% SDBSDGSDF ASGSDFFSAGDHFSDSAWE^@$^HNFSGQW%#@&$%^J#%@#^$#UHRGSDSDNDFE$T#@$UERDFASGWQR!@%!@^$#@YEGEQW%!@%!!GSDHWET!^ " ;
StringBuilder sb = new StringBuilder();
int count = 40 * 1024 / content.length();
int remainder = (40 * 1024) % content.length();
for (int i=0; i < count; i++)
{
sb.append(content);
}
if (remainder > 0)
{
sb.append(content.substring(0, remainder));
}
byte[] buf = sb.toString().getBytes();
for (int j=0; j < fileCount; j++)
{
String path = String.format( " %s%sTestFile_%d.txt " , dir, File.separator, j);
try{
BufferedOutputStream fs = new BufferedOutputStream(new FileOutputStream(path));
fs.write(buf);
fs.close();
}
catch(FileNotFoundException fe)
{
System.out.printf( " Hit filenot found exception %s " , fe.getMessage());
}
catch(IOException ie)
{
System.out.printf( " Hit IO exception %s " , ie.getMessage());
}
}
}
}
I run this by issue following command:
java -jar SimpleWriter.jar my_test_dir 500000
which will create 500000 small 40KB files under my_test_dir. I thought this is a simple code, but then I realize that this code is using up to 14G of memory. I know that because when I use free -m to check the memory. I compile this against JDK 1.6 and then JDK1.7. The result is the same.
The reason I suspect about write() methods is that, if I comment out fs.write(), just open and close the stream, the memory stabilized at certain point. Once I put fs.write() back, the program will keep eating up memory. It seems the closing of stream has no effects, and something in the underline system is " saving " all the data it wrote.
Some people in online community suggest that JVM only clean memory when it feel that it is needed, so it is possible JVM will continue to allocate. They suggest that use -Xmx to limit the allocation. I tried:
java -Xmx2048m -jar SimpleWriter.jar my_test_dir 500000
This take no effect, as the program will keep using up all the memory, regardless what limit I set.
Some people in the online community tested my code under Windows. Strangely, the memory issue does not appear in Windows. It only appear in Linux system.
Also, some people in the online community suggested me not to use BufferedOutputStream. I changed it to FileOutputStream and the result is the same.
I don't believe Java should behave this way. Since that:
a) the buffer I used to write data is allocated just once, and it is only 40K
b) the stream for each file has been closed diligently, there should not be any resource leak related to the writing data to files
c) underline system which allocate memory has no regard to -Xmx limit.
So, it appears to me this is a like a memory leak on OutputStream.write operation, even though it is hard to believe such thing could happen. But reality it seems to be happening.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Please just run the sample program I posted in Description.
java -jar SimpleWriter.jar test_dir 500000
test_dir has to be pre-exist. Program will create 500K files under it.
the disk need to have at least 20G space to hold these files as each file is 40KB in size
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
When the program is running, system available memory may decrease but quickly stabilized. Actually, I expect the program only use < 10MB memory
ACTUAL -
When the program is running, you can use free -m to check the available memory. You will see all memory will be dried up until only around 80M left. e.g. on the VM I tested it, the VM has 15G memory, and nearly all of them were used up.
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
package MyTest;
import java.io.*;
public class SimpleWriter {
public static void main(String[] args) {
String dir = args[0];
int fileCount = Integer.parseInt(args[1]);
String content= " @#$% SDBSDGSDF ASGSDFFSAGDHFSDSAWE^@$^HNFSGQW%#@&$%^J#%@#^$#UHRGSDSDNDFE$T#@$UERDFASGWQR!@%!@^$#@YEGEQW%!@%!!GSDHWET!^ " ;
StringBuilder sb = new StringBuilder();
int count = 40 * 1024 / content.length();
int remainder = (40 * 1024) % content.length();
for (int i=0; i < count; i++)
{
sb.append(content);
}
if (remainder > 0)
{
sb.append(content.substring(0, remainder));
}
byte[] buf = sb.toString().getBytes();
for (int j=0; j < fileCount; j++)
{
String path = String.format( " %s%sTestFile_%d.txt " , dir, File.separator, j);
try{
BufferedOutputStream fs = new BufferedOutputStream(new FileOutputStream(path));
fs.write(buf);
fs.close();
}
catch(FileNotFoundException fe)
{
System.out.printf( " Hit filenot found exception %s " , fe.getMessage());
}
catch(IOException ie)
{
System.out.printf( " Hit IO exception %s " , ie.getMessage());
}
}
}
}
---------- END SOURCE ----------
1.6.0_24
ADDITIONAL OS VERSION INFORMATION :
Linux 64 bits (tried on CentOS 6.3, 6.4, Redhat 6.4, all 64 bits)
EXTRA RELEVANT SYSTEM CONFIGURATION :
Running on Linux VM on AWS.
A DESCRIPTION OF THE PROBLEM :
I write a piece of java code to create 500K small files (each is 40KB) on Linux(CentOS, Redhat). The original code is like this:
package MyTest;
import java.io.*;
public class SimpleWriter {
public static void main(String[] args) {
String dir = args[0];
int fileCount = Integer.parseInt(args[1]);
String content= " @#$% SDBSDGSDF ASGSDFFSAGDHFSDSAWE^@$^HNFSGQW%#@&$%^J#%@#^$#UHRGSDSDNDFE$T#@$UERDFASGWQR!@%!@^$#@YEGEQW%!@%!!GSDHWET!^ " ;
StringBuilder sb = new StringBuilder();
int count = 40 * 1024 / content.length();
int remainder = (40 * 1024) % content.length();
for (int i=0; i < count; i++)
{
sb.append(content);
}
if (remainder > 0)
{
sb.append(content.substring(0, remainder));
}
byte[] buf = sb.toString().getBytes();
for (int j=0; j < fileCount; j++)
{
String path = String.format( " %s%sTestFile_%d.txt " , dir, File.separator, j);
try{
BufferedOutputStream fs = new BufferedOutputStream(new FileOutputStream(path));
fs.write(buf);
fs.close();
}
catch(FileNotFoundException fe)
{
System.out.printf( " Hit filenot found exception %s " , fe.getMessage());
}
catch(IOException ie)
{
System.out.printf( " Hit IO exception %s " , ie.getMessage());
}
}
}
}
I run this by issue following command:
java -jar SimpleWriter.jar my_test_dir 500000
which will create 500000 small 40KB files under my_test_dir. I thought this is a simple code, but then I realize that this code is using up to 14G of memory. I know that because when I use free -m to check the memory. I compile this against JDK 1.6 and then JDK1.7. The result is the same.
The reason I suspect about write() methods is that, if I comment out fs.write(), just open and close the stream, the memory stabilized at certain point. Once I put fs.write() back, the program will keep eating up memory. It seems the closing of stream has no effects, and something in the underline system is " saving " all the data it wrote.
Some people in online community suggest that JVM only clean memory when it feel that it is needed, so it is possible JVM will continue to allocate. They suggest that use -Xmx to limit the allocation. I tried:
java -Xmx2048m -jar SimpleWriter.jar my_test_dir 500000
This take no effect, as the program will keep using up all the memory, regardless what limit I set.
Some people in the online community tested my code under Windows. Strangely, the memory issue does not appear in Windows. It only appear in Linux system.
Also, some people in the online community suggested me not to use BufferedOutputStream. I changed it to FileOutputStream and the result is the same.
I don't believe Java should behave this way. Since that:
a) the buffer I used to write data is allocated just once, and it is only 40K
b) the stream for each file has been closed diligently, there should not be any resource leak related to the writing data to files
c) underline system which allocate memory has no regard to -Xmx limit.
So, it appears to me this is a like a memory leak on OutputStream.write operation, even though it is hard to believe such thing could happen. But reality it seems to be happening.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Please just run the sample program I posted in Description.
java -jar SimpleWriter.jar test_dir 500000
test_dir has to be pre-exist. Program will create 500K files under it.
the disk need to have at least 20G space to hold these files as each file is 40KB in size
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
When the program is running, system available memory may decrease but quickly stabilized. Actually, I expect the program only use < 10MB memory
ACTUAL -
When the program is running, you can use free -m to check the available memory. You will see all memory will be dried up until only around 80M left. e.g. on the VM I tested it, the VM has 15G memory, and nearly all of them were used up.
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
package MyTest;
import java.io.*;
public class SimpleWriter {
public static void main(String[] args) {
String dir = args[0];
int fileCount = Integer.parseInt(args[1]);
String content= " @#$% SDBSDGSDF ASGSDFFSAGDHFSDSAWE^@$^HNFSGQW%#@&$%^J#%@#^$#UHRGSDSDNDFE$T#@$UERDFASGWQR!@%!@^$#@YEGEQW%!@%!!GSDHWET!^ " ;
StringBuilder sb = new StringBuilder();
int count = 40 * 1024 / content.length();
int remainder = (40 * 1024) % content.length();
for (int i=0; i < count; i++)
{
sb.append(content);
}
if (remainder > 0)
{
sb.append(content.substring(0, remainder));
}
byte[] buf = sb.toString().getBytes();
for (int j=0; j < fileCount; j++)
{
String path = String.format( " %s%sTestFile_%d.txt " , dir, File.separator, j);
try{
BufferedOutputStream fs = new BufferedOutputStream(new FileOutputStream(path));
fs.write(buf);
fs.close();
}
catch(FileNotFoundException fe)
{
System.out.printf( " Hit filenot found exception %s " , fe.getMessage());
}
catch(IOException ie)
{
System.out.printf( " Hit IO exception %s " , ie.getMessage());
}
}
}
}
---------- END SOURCE ----------