Java memory usage on Oracle Linux:

1. Unset any limits for virtual memory size
ulimit -Sv unlimited
./8u40b16/bin/java -version
java version "1.8.0_40-ea"
Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b20, mixed mode)

./9b40/bin/java -version
java version "1.9.0-ea"
Java(TM) SE Runtime Environment (build 1.9.0-ea-b40)
Java HotSpot(TM) 64-Bit Server VM (build 1.9.0-ea-b40, mixed mode)

As we can see - all works.

Set virtual memory limit to 2700000k (ulimit parses this values as 1k blocks)
ulimit -Sv 2700000
./8u40b16/bin/java -version
[thread 140677693236992 also had an error][thread 140677694289664 also had an error][thread 140677703763712 also had an error]

[thread 140677698500352 also had an error][thread 140677699553024 also had an error][thread 140677709027072 also had an error]

[thread 140678034056960 also had an error]
[thread 140677701658368 also had an error]
#

# There is insufficient memory for the Java Runtime Environment to continue.

# pthread_getattr_np
# An error report file with more information is saved as:
# /export/home/gtee/jdk/hs_err_pid10846.log
Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes
pure virtual method called
cannot allocate memory for thread-local data: ABORT

hs_err contains:
*** START of hs_log*
#
# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux_x86.cpp:728), pid=10846, tid=140677706921728
#
# JRE version: (8.0_40-b16) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b20 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /export/home/gtee/jdk/core or core.10846
#

--------------- T H R E A D ---------------

Current thread (0x00007ff22402c800): GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=10859
]

Stack: [0x0000000000000000,0x0000000000000000], sp=0x00007ff214bfa540, free space=137380573161k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xaaa87a]
V [libjvm.so+0x4f29db]
V [libjvm.so+0x9133bf]
V [libjvm.so+0x913405]
V [libjvm.so+0xa51f20]
V [libjvm.so+0x5c970c]
V [libjvm.so+0x90f768]


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007ff22402c800 (exited) GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=10859]

VM state:not at safepoint
*** END of hs_log*

JDK 9 build 40

./9b40/bin/java -version
[thread 140435840255744 also had an error][thread 140435846571776 also had an error][thread 140435841308416 also had an error][thread 140435834992384 also had an error]



[thread 140435845519104 also had an error]#
[thread 140435850782464 also had an error]
# There is insufficient memory for the Java Runtime Environment to continue.

# pthread_getattr_np
[thread 140435836045056 also had an error][thread 140435844466432 also had an error]

Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes# An error report file with more information is saved as:
# /export/home/gtee/jdk/hs_err_pid28845.log

pure virtual method called
cannot allocate memory for thread-local data: ABORT

*** hs_err:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
*** end of hs_err

Issue is reproduced on Oracle Linux machines with 32 cores.