Name: bsC130419 Date: 06/27/2001


java version "1.3.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1-b24)
Java HotSpot(TM) Client VM (build 1.3.1-b24, mixed mode)


GZipOutputStream leaks memory to varying degrees depending on configuration
of the HotspotVM. The memory is leaked into a native area (not the
Java heap) which we detected using the pmap command:

$ pmap 12389 | grep heap
00026000 421256K read/write/exec [ heap ]

The code at the end can be used to reproduce the problem. The appearance
of the problem seems to be affected by VM memory and GC configuration
options as follows:

Case1:
java -server -Xmx700m -Xms700m -verbose:gc -Xss128k test
- native heap grows fast, eventually exhausts all available memory
- minor GC's occur every 5 seconds
- full GCs practically never

Case2:
java -server -Xincgc -Xmx700m -Xms700m -verbose:gc -Xss128k test
- native heap grows really fast, exhausts all available memory quickly
- incGC's occur 10-15 every 5 seconds
- full GC's never

Case3:
java -server -verbose:gc -Xss128k test
- native heap grows slowly, if at all, so no problem
- minor GC's occur every second or so
- full GC's occur every 3 secs

It looks like memory that is allocated inside the ZLIB library is leaked
when lots of minor and incgc's occur and there are no full GCs happening.

The code below can use DataOuputStreams or ObjectOutputStreams, I wanted
to make sure it wasnt the serialization. Case 3 isnt a practical workaround
because we naturally dont want full GC's and we need the big heaps.

import java.util.*;
import java.util.zip.*;
import java.io.*;
import java.net.*;

public class test {

    public static void main(String args[]) {

        Runnable r = new Runnable() {
            public void run() {
                test t = new test();
            }
        };

        Thread t = new Thread(r);
        t.start();
        t = new Thread(r);
        t.start();
        t = new Thread(r);
        t.start();
        t = new Thread(r);
        t.start();


    }

    public test() {
        try {
            while (true) {
                List l = new ArrayList();
                for (int i=0;i<1000;i++) {
                    String str = String.valueOf(Math.random());
                    l.add(str);
                }
                byte[] payload = compressData(l);
                System.out.print("c[" + payload.length + "]\n");
            }

        }
        catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    private byte[] compressData(Object o) throws Exception {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        GZIPOutputStream gos = new GZIPOutputStream(baos);
        DataOutputStream dos = new DataOutputStream(gos);
        //ObjectOutputStream oos = new ObjectOutputStream(gos);
        //oos.writeObject(o);
        List l = (List) o;
        for (int i=0;i<l.size();i++) {
            dos.writeUTF((String) l.get(i));
        }
        //oos.flush();
        gos.finish();
        gos.close();
        baos.close();
        return baos.toByteArray();
    }
}
(Review ID: 126664)
======================================================================



A different test case has been attached. To run:

java -server -Xcomp -ms384m -mx512m GZIPBug ./storage_dir IM66 10

(Be sure to run purge.sh or you'll run out of disk space fairly quickly!)

Watch the heap growth of the process using pmap:
pmap 1439 | grep heap
00026000 4976K read/write/exec [ heap ]
           ^^^^^
          watch this number

With the suggested fix, the heap area remains constant.

jane.loizeaux@East 2001-07-13