FULL PRODUCT VERSION :
J2SE 1.6.0_20

ADDITIONAL OS VERSION INFORMATION :
Any OS and any version.

A DESCRIPTION OF THE PROBLEM :
Memory leak in com.sun.net.httpserver.
Instances of some classes are not released when close HttpConnection such as BufferedInputStream, SelectionKey, HttpConnection, FileDescriptor and etc.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Start httpserver(simple httpserver which only processes Get request and returns a string).
2. Use JProfiler to monitor this application (filter: java.io, you will see the number of instances such as BufferedInputStream, FileDescriptor).
3. Telnet this server, the instances of BufferedInputStream will increased by 1. Httpserver sends "Bad request" back.
4. Stop telnet process.
5. Do step 3 and step 4 for 10 times.
6. Wait for 1 hour (HttpServer will release idle connection in 300 seconds, so we give it enough time to do it)
7. Call System.gc();

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The number of instances of BufferedInputStream should be decreased to the previouse value before we did this test.
ACTUAL -
These instances of BufferedInputStream increased to 10 and not be released any more.

When we used jmap in JDK to see the number of class instance, we found more than 10000 BufferedInputStream in out server deployed no more than on week. But there are no more than 100 clients connected to our server current time.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
I checked the sourse code of com.sun.net.httpserver.
I found this bug is caused by sun.net.httpserver.HttpConnection.close().
I can't understand why don't release the "selectionKey" in HttpConnection; the problem is HttpConnection contains this "selectionKey", and the "selectionKey" attached this HttpConnection. So these two Objects can't be release by jvm.
What we need to do is just call
 "selectionKey.cancel();
selectionKey.attach(null);
selectionKey = null;" in the "close" method of HttpConnection.

Sorry for my bad English, hope the below workaround could help you to solve this problem.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The following steps are what we did to solve this problem.
Get sourse code of com.sun.httpserver.
Change this method: sun.net.httpserver.HttpConnection.close()
--------------------------------------------------------------------------------------------
Original code:
synchronized void close() {
if (closed) {
return;
}
closed = true;
if (logger != null && chan != null) {
logger.finest("Closing connection: " + chan.toString());
}

if (!chan.isOpen()) {
ServerImpl.dprint("Channel already closed");
return;
}
try {
/* need to ensure temporary selectors are closed */
if (raw != null) {
raw.close();
}
} catch (IOException e) {
ServerImpl.dprint(e);
}
try {
if (rawout != null) {
rawout.close();
}
} catch (IOException e) {
ServerImpl.dprint(e);
}
try {
if (sslStreams != null) {
sslStreams.close();
}
} catch (IOException e) {
ServerImpl.dprint(e);
}
try {
chan.close();
} catch (IOException e) {
ServerImpl.dprint(e);
}
}
--------------------------------------------------------------------------------------------------
I changed it to:
synchronized void close() {
if (closed) {
return;
}
closed = true;
if (logger != null && chan != null) {
logger.finest("Closing connection: " + chan.toString());
}

if (!chan.isOpen()) {
ServerImpl.dprint("Channel already closed");
return;
}
try {
/* need to ensure temporary selectors are closed */
if (raw != null) {
raw.close();
raw = null; // Changed
}
} catch (IOException e) {
ServerImpl.dprint(e);
}
try {
if (rawout != null) {
rawout.close();
rawout = null; // Changed

}
} catch (IOException e) {
ServerImpl.dprint(e);
}
try {
if (sslStreams != null) {
sslStreams.close();
}
} catch (IOException e) {
ServerImpl.dprint(e);
}
try {
chan.close();
chan = null; // Changed
selectionKey.cancel(); // Changed
selectionKey.attach(null); // Changed
selectionKey = null; // Changed
} catch (IOException e) {
ServerImpl.dprint(e);
}
}