FULL PRODUCT VERSION :
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
SunOS tc-perf-026 5.11 11.2 i86pc i386 i86pc

A DESCRIPTION OF THE PROBLEM :
When I start the jvm with the following flags:

-XX:MaxDirectMemorySize=100g -Xmx2048m

I should be able to call java.nio.ByteBuffer.allocateDirect(32 * 1024 * 1024) 512 times to allocate a total of 8GB - but I get an OutOfMemoryError a bit before 2GB have been allocated.

This works fine on OS X, Windows and Linux but not on Solaris. But If I do start the solaris JVM with the following flags:

-XX:MaxDirectMemorySize=100g -Xmx2049m

then I'm able to allocate the 8GB of direct byte buffers.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
I've included a simple test class (Buffers.java) that reproduces the problem. You just have to make sure:
 - that you're running it on a machine with enough free ram
 - that you add -XX:MaxDirectMemorySize=100g and -Xmx2g on the command line


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The above Buffers class should report that it successfully allocated the 8GB.
ACTUAL -
The above Buffers class catches OutOfMemoryError when a little below 2GB have been allocated.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.nio.*;
import java.util.*;

public class Buffers {
  // 32MB * 256 == 8GB
  private static final int BUFFER_SIZE = 32 * 1024 * 1024;
  private static final int BUFFER_COUNT = 256;

  public static void main(String[] args) throws Exception {
    long counter = 0L;
    try {
        List<ByteBuffer> buffers = new ArrayList<ByteBuffer>();
        for (int i=0 ; i<BUFFER_COUNT ; i++) {
                ByteBuffer b = ByteBuffer.allocateDirect(BUFFER_SIZE);
                buffers.add(b);
                counter += BUFFER_SIZE;
        }
    } catch(Throwable t) {
        t.printStackTrace();
    }
    System.out.println("allocated : " + (counter / 1024 / 1024 + " MB"));
  }

}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
I found two:

1) tell the JVM that its max heap size is over 2GB (ie: -Xmx2049m)
2) use libumem's mmap backend to start the JVM:
LD_PRELOAD=libumem.so UMEM_OPTIONS=backend=mmap java ...