-XX:MallocMaxTestWords is used to limit amount of native memory available to VM.

Yes, indeed. If I run vm with -XX:+UnlockDiagnosticVMOptions -XX:MallocMaxTestWords=16m

Unsafe.allocateMemory(1G) will cause OOM, as expected.

But, reallocate will not...

address = Unsafe.allocateMemory(100); // ok
address = Unsafe.reallocateMemory(address, 1G); // no exception!!!
address = Unsafe.allocateMemory(1G); // OOM - ok

attached ReallocatedTest - demonstrates the problem:

# java -XX:+UnlockDiagnosticVMOptions -XX:MallocMaxTestWords=16m ReallocateTest
Allocating: 100
  successful: 9415472
Re-allocating: 1000000000
  successful: 9440064
Memory reallocated... OOM is expected here
Allocating: 1000000000
OOM: java.lang.OutOfMemoryError
Test failed