FULL PRODUCT VERSION :
Java 1.7.0_60 VM: Java HotSpot(TM) Client VM 24.60-b09

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]


A DESCRIPTION OF THE PROBLEM :
Method DeployCacheHandler.get(final URI uri, String rqstMethod, Map requestHeaders)
keeps adding request URIs to member hash 'isProgress', but (because of code bug) NEVER removes those entries.

As a result, application making high number of requests ends up caching EVERY distinct HTTP request it has ever made. In our case, application processing over 2000 documents (in 12+ hours) ends up having over 1 million cached URIs (amounting to over 400 MB of memory).

Looking at an (older version) of source, the URI should be released by the following code:
...
            } finally {
                inCacheHandler.set(null);
                //remove object we used for synchronization
                synchronized (inProgress) {
                    inProgress.remove(key);
                }
            }
 

The problem is that the 'key' in the above call is NOT the hash table key, it was allocated earlier in the method using:

        Object key;
        //make sure there is lock object for given URI
        synchronized (inProgress) {
            if (!inProgress.containsKey(uri)) {
                inProgress.put(uri, new Object());
            }
            key = inProgress.get(uri);


Obviously, the key is the cached VALUE, not the value KEY.

On a minor note, using URI as a hash key may be problematic: hash value of URI may change as a result of URI method calls...






ADDITIONAL REGRESSION INFORMATION:
Java 1.7.0_60 VM: Java HotSpot(TM) Client VM 24.60-b09
but as far as I can tell, this bug dates back to Java 6.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Use an application making frequent and _varying_ HTTP GET requests, and execute this application using Java Web Start.

Either using memory leak detector (such as Eclipse memory Analyzer) or using an explicit code (attached below) monitor how the size of hash table
DeployCacheHandler.inProgress
keeps growing.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Size of
DeployCacheHandler.inProgress
should remain at zero or very low count - number of threads making the same URI request at the same time.
ACTUAL -
Log from attached tool:
15:45:28,867 [aemon 4300] INFO Main$ResponseCacheWrap:1259 - DeployCacheHandler LEAKS inProgress=5
15:45:59,131 [aemon 4300] INFO Main$ResponseCacheWrap:1259 - DeployCacheHandler LEAKS inProgress=465
15:46:29,582 [aemon 4300] INFO Main$ResponseCacheWrap:1259 - DeployCacheHandler LEAKS inProgress=588
15:46:59,643 [aemon 4300] INFO Main$ResponseCacheWrap:1259 - DeployCacheHandler LEAKS inProgress=1000
15:47:29,658 [aemon 4300] INFO Main$ResponseCacheWrap:1259 - DeployCacheHandler LEAKS inProgress=1362
...
16:01:55,803 [aemon 4300] INFO Main$ResponseCacheWrap:1259 - DeployCacheHandler LEAKS inProgress=12902


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Since most (heap)n memory analyzers produce too much (and confusing) data,
I intercepted all calls to DeployCacheHandler by a 'wrapper' class, inserted at the start of my application as follows:

ResponseCache cache = ResponseCache.getDefault();
if (cache != null) {
log.info("WRAPPING ResponseCache " + cache.getClass().getName());
ResponseCache.setDefault(new ResponseCacheWrap(cache));
} else {
log.info("WRAPPING found no ResponseCache to WRAP ");
}

private static class ResponseCacheWrap extends ResponseCache {
private ResponseCache delegate;

public ResponseCacheWrap(ResponseCache del) {
this.delegate = del;
}

@Override
public CacheResponse get(URI uri, String rqstMethod, Map<String, List<String>> rqstHeaders) throws IOException {
try {
// HACK preventing handling of our Knowledge Database server requests
// if (uri.getPath().indexOf("knowledge") >= 0) { return null; }
CacheResponse resp = delegate.get(uri, rqstMethod, rqstHeaders);
if (resp != null) {
log.info("DeployCacheHandler get(" + uri + ") != null");
}
logSizes();
return resp;
} catch (Throwable t) {
String msg = "DeployCacheHandler CacheResponse.get(" + uri + ") failed " + t;
log.error(msg);
throw new IOException(msg);
}
}

@Override
public CacheRequest put(URI uri, URLConnection conn) throws IOException {
try {
// HACK preventing handling of our Knowledge Database server requests
// if (uri.getPath().indexOf("knowledge") >= 0) { return null; }
CacheRequest req = delegate.put(uri, conn);
if (req != null) {
log.info("DeployCacheHandler put(" + uri + ") != null");
}
return req;
} catch (Throwable t) {
String msg = "DeployCacheHandler CacheResponse.put(" + uri + ") failed " + t;
log.error(msg);
throw new IOException(msg);
}
}

private long lastLog = 0;

private synchronized void logSizes() {
if (System.currentTimeMillis() > lastLog + 30*1000) {
int inProgress = getSizeFromField("inProgress");
log.info("DeployCacheHandler LEAKS inProgress=" + inProgress);
lastLog = System.currentTimeMillis();
}
}

private int getSizeFromField(String collName) {
try {
Class cls = delegate.getClass();
Field fld = cls.getDeclaredField(collName);
fld.setAccessible(true);
Object coll = fld.get(delegate);
synchronized(coll) {
if (coll instanceof HashMap) {
return ((HashMap)coll).size();
} else if (coll instanceof Collection) {
return ((Collection)coll).size();
} else {
return -1;
}
}
} catch (Throwable t) {
log.warn("Failed getting collection size for " + collName + "', " + t);
return 0;
}
}
}


---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
As shown (as comment) in my 'wrapper' class, I can make most of 'my' (frequent) requests bypass the DeployCacheHandler. That REDUCES (but does not eliminate) the problem.