Name: rmT116609 Date: 05/13/2004


FULL PRODUCT VERSION :
java version "1.4.2_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_01-b06)
Java HotSpot(TM) Client VM (build 1.4.2_01-b06, mixed mode)

FULL OS VERSION :
Microsoft Windows 2000 [Version 5.00.2195]

A DESCRIPTION OF THE PROBLEM :
I was attempting to track resources using a ReferenceQueue of SoftRefs.
No entries were showing up in queue. Used a profiler (OptimizeIt) to confirm that all strong refs were correctly cleared. They where. Wrote a small test program to confirm I was correctly using the Reference API. This program demonstrated that not all Refs where enqueued. Adding a finalizer to the Refs demonstrated that they were GC'ed but not enqueued.

Reviewing the Bug database turned up two related issues: 4268317 and 4243978.
However, neither points out the actual problem.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the provided program.
It runs two threads: one producing WeakRefs and another consuming them off a ReferenceQueue. This program will eventually consume 2000 "lost" references and exit.

This is a sample of the output

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
This is a sample of the output:
    nothing in queue
    dequeued: 35
    refset: size=86 first=0 last=86
    dequeued: 86 85
    refset: size=84 first=0 last=84
    dequeued: 137 188 187 186 185 136 135
    refset: size=245 first=0 last=254
    nothing in queue
    sleeping...wakeup call
    dequeued: 290
    refset: size=280 first=0 last=289

refset is Set tracking "resources" allocation but not released via the ReferenceQueue. It keeps growing.
There is also some code to throttle the produce as the size of the set grows. This only slows down the eventual failure.

If you uncomment the finalizer on MyWeakRef the program will run indefinitely.
ACTUAL -
Eventual "resource" exhaustion

ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.lang.IllegalStateException: exceeded arbitrary resource limit
at ReferenceLeak$MyWeakReference.<init>(ReferenceLeak.java:129)
at ReferenceLeak$MyWeakReference.<init>(ReferenceLeak.java:114)
at ReferenceLeak$1.produce(ReferenceLeak.java:48)
at ReferenceLeak$1.run(ReferenceLeak.java:37)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.lang.ref.*;
import java.util.*;

/**
 * Class to demostrate how WeakRefs may be leaked by the GC.
 * This class enqueues a medium number of WeakRefs tracking creations.
 * It then tracks what it dequeues. Some Refs never show up in queue.
 * They may even be finalized but still never show up in queue.
 *
 * BugIds 4268317 and 4243978 are probably related to this but don't stress the
 * the problem.
 *
 * If you can't count on SoftRefs being properly enqueued you can't really
 * count on them at all.
 *
 * @author rmp
 */
public class ReferenceLeak {

    // RefQueue<MyWeakRefs>
    private static ReferenceQueue refQueue = new ReferenceQueue();

    // SortedSet<Integer> -- tracks MyWeakRefs that have been created.
    // Entries are removed when dequeued
    // Also removed if the finalize() code below is uncommented.
    // Should hit a steady state -- 0 in a perfect world.
    // However, unless we also cleanup in finalize() Set keeps growing. I.e
    // some WeakRefs are never enqueued.
    static SortedSet refset = Collections.synchronizedSortedSet(new TreeSet());

    public static void main(String[] args)
    {
        // this Thread is a producer of WeakRefs
        new Thread() {
            public void run() {
                try {
                    produce();
                } catch (Exception e) {
                    e.printStackTrace();
                    System.exit(1);
                }
            }

            public void produce() {
                for (int i = 0; ; i++)
                {
                    // create some junk -- a lot of junk
                    Object[] junk = new Object[] {
                        new MyWeakReference(new byte[10000], refQueue),
                        new MyWeakReference(new byte[10000], refQueue),
                        new MyWeakReference(new byte[10000], refQueue),
                        new MyWeakReference(new byte[10000], refQueue),
                        new MyWeakReference(new byte[10000], refQueue),
                    };

                    if (i % 50 == 0)
                    {
                        System.err.print("sleeping...");
                        try {
                            Thread.sleep(1000);
                            System.err.println("wakeup call");
                        } catch (InterruptedException e) {
                            System.err.println("interrupted");
                        }
                    }
                }
            }
        }.start();

        // main thread is the consumer of WeakRefs
        while (true) {
            MyWeakReference ref;

            // block for the first dequeue
            try {
                ref = (MyWeakReference) refQueue.remove(500);
            } catch (InterruptedException e) {
                System.out.println("interrupted...");
                return;
            }

            if (ref == null)
            {
                System.out.println("nothing in queue");

                // GC or not doesn't seem to matter
// System.gc();
// System.runFinalization();
                continue;
            }

            // output what we dequeue
            System.out.print("dequeued:");
            do { // poll for rest
                int k = ref.getK();
                boolean missing = !refset.remove(new Integer(k)); // already removed
                if (missing)
                    System.err.print("(missing " + k + ")");
                System.out.print(" " + k);

            } while (null != (ref = (MyWeakReference) refQueue.poll()));

            // output current state of refset -- produced but not consumed yet
            System.out.println();
            System.out.print("refset: size="+refset.size());
            if (refset.size() > 0)
               System.out.print(" first=" + refset.first() + " last=" + refset.last());
            System.out.println();
        }
    }

    // Simple WeakRef has no other strong refs just an ivar
    // In any event, instances are lost -- not properly enqueued
    private static class MyWeakReference extends WeakReference
    {
        private static int __counter;
        private int _k;

        private MyWeakReference(Object ref, ReferenceQueue queue) {
            super(ref, queue);
            _k = __counter++;
            refset.add(new Integer(_k)); // track ref

            //
            // add some arbitrary code to throttle down any producer
            //

            if (refset.size() > 2000) {
                throw new IllegalStateException("exceeded arbitrary resource limit");
            }

            if (refset.size() > 1900) {
                System.err.println("gc()");
                System.gc();
            }

            if (refset.size() > 1800) {
                System.err.println("runFinalization()");
                System.runFinalization();
            }

            // If all refs are actually enqueued this lets consumer catch up
            if (refset.size() > 1500)
            {
                // backoff 10-60ms (more as we approach limit)
                int backoff = ((refset.size() - 2000 + 500) / 10) + 10;
                System.err.println("backing off " + backoff);
                try { Thread.sleep(backoff); } catch (InterruptedException e) { }
            }

        }

        public int getK() {
            return _k;
        }

// enabling the finalizer code here will stop resource exhaustion issues
// as we are no longer relying on WeakRef alone :(

/**
        public void finalize() throws Throwable {
            try {
                super.finalize();
            } finally {
                boolean removed = refset.remove(new Integer(_k));

                // VERY verbose if enabled
                // if (removed)
                // System.err.println("finalizer removed: " + _k);
            }
        }
/**/
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use a combination of WeakRef and finalization.

(Incident Review ID: 208488)
======================================================================