FULL PRODUCT VERSION :
java version " 1.7.0_07 "
Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
Java HotSpot(TM) Client VM (build 23.3-b01, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Windows 7 (64bit)

A DESCRIPTION OF THE PROBLEM :
Repeatedly creating java.nio.channel.SocketChannels without closing them leads to increasing memory consumption on a windows 7 machine. With the test code listed it is even possible to exceed the complete (physical) main memory on a machine. This is strange as one would expect to get at least an OutOfMemoryException not until the vm's memory limits are exceeded. In our case we did't specify any special vm settings (heap size, etc.)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Close unrelated programs, open Windows task manager (memory view) and run the test program.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Expexted behaviour is OutOfMemoryException, as it should not be possible to exceed the vm's memory limits.
ACTUAL -
Running the test program you will notice a constant increase of memory consumption until the complete ram is filled. In our case we have a 8 gb machine with 2 gb reserved at launch time. Complete memory fill up took about 1-2 minutes. Finally the program exits with a SocketException:

Exception in thread " main " java.net.SocketException: No buffer space available (maximum connections reached?): socket
        at sun.nio.ch.Net.socket0(Native Method)
        at sun.nio.ch.Net.socket(Net.java:97)
        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:86)
        at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:37)
        at java.nio.channels.SocketChannel.open(SocketChannel.java:105)
        at SocketChannelMemTest.main(SocketChannelMemTest.java:9)

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread " main " java.net.SocketException: No buffer space available (maximum connections reached?): socket
        at sun.nio.ch.Net.socket0(Native Method)
        at sun.nio.ch.Net.socket(Net.java:97)
        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:86)
        at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:37)
        at java.nio.channels.SocketChannel.open(SocketChannel.java:105)
        at SocketChannelMemTest.main(SocketChannelMemTest.java:9)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.nio.channels.SocketChannel;

public class SocketChannelMemTest {

public static void main(String[] args) throws Exception {
System.out.println( " starting... " );
final int numRounds = 100000000;
for (int i = 0; i < numRounds; i++) {
SocketChannel sc = SocketChannel.open();
System.out.println(i + " of " + numRounds);
}
System.out.println( " done.... " );
}
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
One should close the SocketChannel properly. Nevertheless the above mentioned symptoms seem to open some security issues.