FULL PRODUCT VERSION :
1.6.0_12-b04, 1.6.0_14-ea-b03 & 1.7.0-ea-b44

A DESCRIPTION OF THE PROBLEM :
Before throwing an OutOfMemory error for direct memory (allocated using ByteBuffer.allocateDirect), I've noticed that JVM does a full GC to reclaim the direct memory. However, if I have multiple threads trying to allocate direct memory, I get a "java.lang.OutOfMemoryError: Direct buffer memory," even though I know that the garbage collection should have freed up enough direct memory.

In the test case I've attached I have 2 threads which loop, creating 1MB direct byte buffers and then immediately dereferencing them. So I should need a maximum of 2MB of direct memory. However with a 32 MB direct memory size I still get an OutOfMemory error.

When I run the same test case with only 1 thread it passes.

I suspect that maybe the direct memory is being freed up in a java thread, rather than during the garbage collection. I noticed that the JVM creates a DirectByteBuffer$Deallocator object for each of these buffers, which is run by a sun.misc.Cleaner, which I think is run in the reference handler thread.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run "java -Xmx32m -XX:MaxDirectMemorySize=32m test.DirectBufferTest"

ACTUAL -
Thread Test thread 0 got an OOM on iteration 223
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at test.DirectBufferTest$1.run(DirectBufferTest.java:24)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
package test;

import java.nio.ByteBuffer;

public class DirectBufferTest {
  private static int NUM_THREADS = 2;
  
  public static void main(String[] args) throws InterruptedException {
    for(int i = 0; i < NUM_THREADS; i++) {
      Thread thread = new Thread("Test thread " + i) {
        public void run() {
          int i = 0;
          try {
            while(true) {
              ByteBuffer bb = ByteBuffer.allocateDirect(1024 * 1024);
              i++;
            }
          } catch(Throwable t) {
            System.err.println("Thread " + Thread.currentThread().getName() + " got an OOM on iteration " + i);
            t.printStackTrace();
            System.exit(1);
          }
        }
      };
      thread.start();
    }
    
    Thread.sleep(60 * 1000);
    System.out.println("No errors after 60 seconds.");
    System.exit(0);
  }
}
---------- END SOURCE ----------