A DESCRIPTION OF THE PROBLEM :
When starting JVM without -Xmx parameter ByteBuffer.allocateDirect() successfully allocates off-heap memory. However, when one limits JVM heap on startup with -Xmx parameter (like -Xmx1g) the amount of memory that ByteBuffer.allocateDirect() can allocate is limited by the size of JVM heap. Off-heap memory allocation should not depend on JVM on-heap configuration options.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the following source code and pass -Xmx1g option to JVM:
import java.nio.ByteBuffer;

class Scratch_1 {
    public static void main(String[] args) {
        ByteBuffer bb = ByteBuffer.allocateDirect(Integer.MAX_VALUE);
    }
}

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The program runs and allocates Integer.MAX_VALUE bytes of off-heap memory.
ACTUAL -
The exception is thrown:

Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:187)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:310)
at Scratch_1.main(Scratch_1.java:5)

---------- BEGIN SOURCE ----------
import java.nio.ByteBuffer;

class Scratch_1 {
    public static void main(String[] args) {
        ByteBuffer bb = ByteBuffer.allocateDirect(Integer.MAX_VALUE);
    }
}
---------- END SOURCE ----------

FREQUENCY : always