Name: nt126004 Date: 11/20/2002


FULL PRODUCT VERSION :
java version "1.4.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-b92)
Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode)

and

java version "1.4.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1-b21)
Java HotSpot(TM) Client VM (build 1.4.1-b21, mixed mode)


FULL OPERATING SYSTEM VERSION :
Linux 2.4.18-5smp
Red Hat Linux 7.3 2.96-110
glibc-2.2.5-36

EXTRA RELEVANT SYSTEM CONFIGURATION :
total: used: free: shared: buffers: cached:
Mem: 393924608 386863104 7061504 0 33955840 204648448
Swap: 797843456 0 797843456


A DESCRIPTION OF THE PROBLEM :
I am doing nio performance prototyping, and attempted to
transpose a 3d, 153x701x1501 byte data volume from i,j,k to
k,i,j order, using ScatterByteChannel (see attached test case).

Using vmstat and/or top while the program is running shows
RSS and SIZE increasing monotonically. Eventually the
system starts to swap, then thrash, and sometime after that,
the kernel kills the java processes with an "Out of
Memory..." error.

I have run this on Windows 2000/JDK1.4 without any problem
(see Expected and Actual Results). I am guessing there is a
leak in the native IO code on Linux.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1.java -Xms16m -Xmx200m ijk2kij <infile> "/dev/null" ni nj nk
where ni, nj, nk are the dimensions of the data volume.
2. Using -verbose:gc shows garbage collection occuring
several times during each islice read.
3. Using the -server option makes no apparent difference.
4. Modifying or eliminating the -Xms, -Xmx options makes no
apparent difference.

EXPECTED VERSUS ACTUAL BEHAVIOR :
On Win2K, the program takes ~17-40 sec. to read/write each
islice, about an hour to complete. Performance monitor
shows available memory decreasing to a few MB within the
first 50 islices or so, and is stable thereafter, so free'd
memory is being returned to the system.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Sep 12 17:02:09 mango kernel: Out of Memory: Killed process 32564 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32565 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32566 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32567 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32568 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32569 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32570 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32571 (java).
Sep 12 17:02:11 mango kernel: Out of Memory: Killed process 32572 (java).


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.*;
import java.nio.*;
import java.nio.channels.*;

/**
 * Describe class <code>ijk2kij</code> here.
 *
 * @author <a href="mailto:###@###.###">Kai Peter Ford</a>
 * @version 1.0
 */
public class ijk2kij
{
      /**
       * Transpose file from i,j,k to k,i,j
       *
       * @param argv a <code>String[]</code> value
       */
      public static void main(String[] argv)
      {
if ( argv.length != 5 ) {
System.err.println( "Usage:" );
System.err.println( "java ijk2kij in_file out_file ni nj nk" );
System.exit(0);
}
try {
System.err.println("ijk2kij: reading dataset "+argv[0]+ " ...");
long start = System.currentTimeMillis();

FileChannel source = new FileInputStream(argv[0]).getChannel();
FileChannel dest = new FileOutputStream(argv[1]).getChannel();
int ni = Integer.parseInt(argv[2]);
int nj = Integer.parseInt(argv[3]);
int nk = Integer.parseInt(argv[4]);

long nr; // # bytes read (scattered)
long slot=0; // Current line position in output file
long skip = nj*ni; // Size of one kslice
int i,j,k,n;
ByteBuffer wbuf; // = wbufs[k]
ByteBuffer wbufs[] = new ByteBuffer[nk];
// Create nk ByteBuffers of capacity nj for storing an
// nk x nj islice:
for ( k=0; k<nk; k++ ) {
wbufs[k] = ByteBuffer.allocateDirect(nj);
// Set up to scatter the first line to one sample
// (byte) per wbuf:
wbufs[k].limit(1);
}

long s2,s1 = System.currentTimeMillis(); // debug
Runtime rt = Runtime.getRuntime(); // debug
for ( i=0; i<ni; i++ ) {
// Get the i'th islice into wbufs:
for (j=0; j<(nj-1); j++ ) {
nr = 0;
// Read nk samples and scatter them, 1/wbuf.
while ( nr < nk )nr += source.read(wbufs);
System.err.println("ijk2kij: islice "+i+", jslice= "+j+" read; free = "
+rt.freeMemory()+"; total = "
+rt.totalMemory()); // debug
// Increment each buffer limit by one (#
// bytes/sample); this is why one can't "scatter"
// more than a line at a time:
n = j+2;
for(k=0; k<nk; k++ ) wbufs[k].limit(n);
}
// Do the last j:
nr = 0;
while ( nr < nk )nr += source.read(wbufs);
// The i'th islice is now stored in the wbufs as
// constant k lines of length nj.... Write
// them out.
slot = i*nj;
for(k=0; k<nk; k++ ) {
wbuf = wbufs[k];
wbuf.position(0).limit(nj); // Prepare to write k'th line
while (wbuf.hasRemaining())
dest.write(wbuf, slot); // Write line for i'th islice, k'th kslice, to file
slot+=skip; // Increment file position of next line
wbuf.position(0).limit(1); // Set up for next islice scatter.
}
s2 = System.currentTimeMillis(); // debug
System.err.println("ijk2kij: Time to read/write islice "+i+
" = "+((float)(0.001*(s2-s1)))+" seconds."); // debug
s1 = s2; // debug
}
source.close();
dest.close();

long stop = System.currentTimeMillis();
System.err.println("ijk2kij: Time to write file =
"+((float)0.001*(stop-start))+" seconds.");


} catch(Exception ex) {
System.err.println(ex.getMessage());
ex.printStackTrace(System.err);
} catch(Error er) {
System.err.println(er.getMessage());
er.printStackTrace(System.err);
}
      }
}



---------- END SOURCE ----------

CUSTOMER WORKAROUND :
Don't use ScatterByteChannel for disk io, read entire islice
and transpose in code.
Need to do this for performance reasons anyway.
(Review ID: 164467)
======================================================================