Test scenario:

1) test environment
 - Linux machine
 - jdk8 x64
 - memory 5g or ulimit 5g
virtual memory (kbytes, -v) 5360352
 - java options ( -Xmx2100m -XX:MetaspaceSize=128M )
2) Test passes

Maximum heap size outside 32-bit address range
/export/jdk/jdk1.8.0b76/fastdebug//bin/java -cp . -Xmx2100m -XX:MetaspaceSize=128M gc.huge.quicklook.largeheap.MemOptions.MemStat
Max memory : 1957363712
Total memory : 123863040
Free memory : 123216840

3) setting up ulimit -v 2586944
> virtual memory (kbytes, -v) 2586944

4) got crash.
Maximum heap size outside 32-bit address range
/export/jdk/jdk1.8.0b76/fastdebug//bin/java -cp . -Xmx2100m -XX:MetaspaceSize=128M gc.huge.quicklook.largeheap.MemOptions.MemStat
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 152 bytes for AllocateHeap
# An error report file with more information is saved as:
# /home/m2n/Projects/INTJDK-7600693/results/ResultDir/MemOptions/hs_err_pid24081.log
Exit code: 1

Think that this is a bug because there are enough memory for heap and metaspace.

Test and hs_err file are attached.

Test is reproducible
1) unix\linux
2) java 7\8
3) crash is only specific for PermGen + ulimit + jdk7 or Metaspace + ulimit + jdk8
4) not specific for x64\x86


#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 152 bytes for AllocateHeap
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (/HUDSON/workspace/2-build-linux-amd64/jdk8/3240/hotspot/src/share/vm/memory/allocation.inline.hpp:60), pid=24081, tid=139798013282048
#
# JRE version: (8.0-b76) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b17-fastdebug mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

--------------- T H R E A D ---------------

Current thread (0x0000000001ba4800): JavaThread "main" [_thread_in_vm, id=24082, stack(0x00007f2542dc0000,0x00007f2542ec1000)]

Stack: [0x00007f2542dc0000,0x00007f2542ec1000], sp=0x00007f2542ebda80, free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xd90c62] VMError::report_and_die()+0x302
V [libjvm.so+0x6793eb] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x392851] CHeapObj<(unsigned short)1792>::operator new(unsigned long, unsigned char*)+0x61
V [libjvm.so+0x66bdc7] ConstantPool::ConstantPool(Array<unsigned char>*)+0x77
V [libjvm.so+0x66bf9b] ConstantPool::allocate(ClassLoaderData*, int, Thread*)+0x12b
V [libjvm.so+0x593951] ClassFileParser::parse_constant_pool(ClassLoaderData*, Thread*)+0xe1
V [libjvm.so+0x59a983] ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>*, TempNewSymbol&, bool, Thread*)+0x563
V [libjvm.so+0x5a2974] ClassLoader::load_classfile(Symbol*, Thread*)+0x214
V [libjvm.so+0xce4490] SystemDictionary::load_instance_class(Symbol*, Handle, Thread*)+0x3c0
V [libjvm.so+0xce4d31] SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*)+0x801
V [libjvm.so+0xce5248] SystemDictionary::resolve_or_null(Symbol*, Handle, Handle, Thread*)+0xd8
V [libjvm.so+0xce6163] SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*)+0x33
V [libjvm.so+0x66f891] ConstantPool::klass_at_impl(constantPoolHandle, int, Thread*)+0x211
V [libjvm.so+0x85e8b3] InterpreterRuntime::_new(JavaThread*, ConstantPool*, int)+0xe3
j java.lang.ref.Finalizer.<clinit>()V+0
v ~StubRoutines::call_stub
V [libjvm.so+0x870f0e] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x17fe
V [libjvm.so+0x81b141] InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0x131
V [libjvm.so+0x81d382] InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x332
V [libjvm.so+0x81d702] InstanceKlass::initialize(Thread*)+0x92
V [libjvm.so+0xd271ac] Threads::create_vm(JavaVMInitArgs*, bool*)+0x7ac
V [libjvm.so+0x891fa3] JNI_CreateJavaVM+0xa3
C [libjli.so+0x767e] JavaMain+0x9e

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.lang.ref.Finalizer.<clinit>()V+0
v ~StubRoutines::call_stub