JVM initialization fails with parallel GC on large memory systems with ulimits

FULL PRODUCT VERSION :
$ java -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)


FULL OS VERSION :
$ uname -a
Linux d-ubuntu12x64-02 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
Initialization of the JVM fails with insufficient memory errors when the (default) parallel garbage collector is used on large memory systems when virtual memory ulimits are set, even when the maximum heap size is capped well below the virtual memory ulimit:

# On host 1....

# 64GB of physical memory...
$ free
             total used free shared buffers cached
Mem: 65950952 7360552 58590400 0 490820 5380160
-/+ buffers/cache: 1489572 64461380
Swap: 124999676 0 124999676

# ulimit unlimited...
$ ulimit -v
unlimited

# java -version succeeds...
$ java -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)

# Reduce virtual memory soft ulimit to 2GB
$ ulimit -S -v 2097152

# java -version fails...
$ java -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

# Reduce maximum heap for the JVM, java -version still fails, but with a different error message...
$ java -Xmx256m -version
#
# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
Aborted (core dumped)

# java -version succeeds wth the serial garbage collector...
$ java -Xmx256m -XX:+UseSerialGC -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)

# On host 2...

# 32GB of physical memory...
$ free
             total used free shared buffers cached
Mem: 32962112 31729260 1232852 0 864788 22910492
-/+ buffers/cache: 7953980 25008132
Swap: 5111800 144 5111656

$ ulimit -v
unlimited

# java -version succeeds...
$ java -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)

# Reduce virtual memory soft ulimit to 2GB
$ ulimit -S -v 2097152

# java -version fails...
$ java -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

# Reduce maximum heap for the JVM, java -version now succeeds using the default parallel garbage collector...
$ java -Xmx256m -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)

The fact that the program fails on host 1, but not on host 2, with the same virtual memory ulimits set because host 1 has *more* physical memory than host 2 is unexpected. Having more physical memory available is generally not a predictor of program failure. It appears that some initialization code associated with the parallel garbage collector bases memory allocations on physical memory without regard to the specified maximum heap size or current virtual memory ulimit settings. This is problematic since the net result is that adding physical memory to a system may cause Java programs that were running successfully previously to fail after memory is added.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected results: The JVM should not attempt to allocate memory beyond that required to accommodate the specified maximum heap size when using the parallel garbage collector. At a minimum, virtual memory ulimits should be consulted in addition to the physical memory on the system.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
The crash file created using Java 1.7.0_05 is always empty when reproducing this problem. However, this issue also exists with Java 1.6.0_24. The following was produced using Java 1.6.0_24:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux_x86.cpp:715), pid=8991, tid=140291153549056
#
# JRE version: 6.0_24-b24
# Java VM: OpenJDK 64-Bit Server VM (20.0-b12 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.11.1
# Distribution: Ubuntu 12.04 LTS, package 6b24-1.11.1-4ubuntu3

--------------- T H R E A D ---------------

Current thread (0x00007f9818093800): JavaThread "Finalizer" daemon [_thread_new, id=9013, stack(0x0000000000000000,0x0000000000000000)]

Stack: [0x0000000000000000,0x0000000000000000], sp=0x00007f98145f4960, free space=137003079634k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)


REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Use the serial garbage collector by running java with -XX:+UseSerialGC