G1: VM crashed with OOME on solaris with large heap

Tests

nsk/stress/except/except001
nsk/stress/except/except004
nsk/stress/except/except006
nsk/stress/except/except007
nsk/stress/except/except008
nsk/stress/except/except009

crashed with options -d64 -server -Xcomp -XX:+UseG1GC

;; Using jvm: "/bpool/local/aurora/sandbox/sca/vmsqe/jdk/pit/2013-07-13-001338.amurillo.hs25-b41-jdk8-b99-control/fastdebug/solaris-amd64/jre/lib/amd64/server/libjvm.so"
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 4088 bytes for AllocateHeap
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (/opt/jprt/T/P1/001338.amurillo/s/hotspot/src/share/vm/memory/allocation.inline.hpp:62), pid=5464, tid=24
#
# JRE version: Java(TM) SE Runtime Environment (8.0) (build 1.8.0-internal-fastdebug-jprtadm_2013_07_12_17_28-b00)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b41-fastdebug compiled mode solaris-amd64 compressed oops)
# Core dump written. Default location: /bpool/local/aurora/sandbox/results/ResultDir/except001/core or core.5464
#

--------------- T H R E A D ---------------

Current thread (0x0000000000481000): GCTaskThread [stack: 0xfffffd7ffcbfe000,0xfffffd7ffccfe000] [id=24]

Stack: [0xfffffd7ffcbfe000,0xfffffd7ffccfe000], sp=0xfffffd7ffccf5920, free space=990k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x2774b48] void VMError::report(outputStream*)+0x92c;; __1cHVMErrorGreport6MpnMoutputStream__v_+0x92c
V [libjvm.so+0x2775fb0] void VMError::report_and_die()+0x56c;; __1cHVMErrorOreport_and_die6M_v_+0x56c
V [libjvm.so+0xf6a5ab] void report_vm_out_of_memory(const char*,int,unsigned long,VMErrorType,const char*)+0x497;; __1cXreport_vm_out_of_memory6FpkciLnLVMErrorType_1_v_+0x497
V [libjvm.so+0x11a01a3] StarTask*Stack<StarTask,(unsigned short)1280>::alloc(unsigned long)+0x83;; __1cFStack4nIStarTask_vbxG_Falloc6ML_pn0A__+0x83
V [libjvm.so+0x1180ff8] void G1ParPushHeapRSClosure::do_oop(unsigned*)+0x6fc;; __1cWG1ParPushHeapRSClosureGdo_oop6MpI_v_+0x6fc
V [libjvm.so+0x11df94d] void FilterIntoCSClosure::do_oop(unsigned*)+0x20d;; __1cTFilterIntoCSClosureGdo_oop6MpI_v_+0x20d
V [libjvm.so+0x20de0c4] int ObjArrayKlass::oop_oop_iterate_v_m(oop,ExtendedOopClosure*,MemRegion)+0x4ac;; __1cNObjArrayKlassToop_oop_iterate_v_m6MnDoop_pnSExtendedOopClosure_nJMemRegion__i_+0x4ac
V [libjvm.so+0x13158da] void HeapRegionDCTOC::walk_mem_region_with_cl(MemRegion,HeapWord*,HeapWord*,ExtendedOopClosure*)+0x856;; __1cPHeapRegionDCTOCXwalk_mem_region_with_cl6MnJMemRegion_pnIHeapWord_3pnSExtendedOopClosure__v_+0x856
V [libjvm.so+0x246b778] void Filtering_DCTOC::walk_mem_region(MemRegion,HeapWord*,HeapWord*)+0xb0;; __1cPFiltering_DCTOCPwalk_mem_region6MnJMemRegion_pnIHeapWord_3_v_+0xb0
V [libjvm.so+0x1323943] void HeapRegionDCTOC::walk_mem_region(MemRegion,HeapWord*,HeapWord*)+0x23;; __1cPHeapRegionDCTOCPwalk_mem_region6MnJMemRegion_pnIHeapWord_3_v_+0x23
V [libjvm.so+0x246b1ed] void DirtyCardToOopClosure::do_MemRegion(MemRegion)+0x155;; __1cVDirtyCardToOopClosureMdo_MemRegion6MnJMemRegion__v_+0x155
V [libjvm.so+0x11de501] bool ScanRSClosure::doHeapRegion(HeapRegion*)+0x531;; __1cNScanRSClosureMdoHeapRegion6MpnKHeapRegion__b_+0x531
V [libjvm.so+0x11475fa] void G1CollectedHeap::collection_set_iterate_from(HeapRegion*,HeapRegionClosure*)+0xae;; __1cPG1CollectedHeapbBcollection_set_iterate_from6MpnKHeapRegion_pnRHeapRegionClosure__v_+0xae
V [libjvm.so+0x11dbc7b] void G1RemSet::oops_into_collection_set_do(OopsInHeapRegionClosure*,int)+0x53f;; __1cIG1RemSetbBoops_into_collection_set_do6MpnXOopsInHeapRegionClosure_i_v_+0x53f
V [libjvm.so+0x1155eab] void G1CollectedHeap::g1_process_strong_roots(bool,SharedHeap::ScanningOption,OopClosure*,OopsInHeapRegionClosure*,G1KlassScanClosure*,int)+0x3db;; __1cPG1CollectedHeapXg1_process_strong_roots6MbnKSharedHeapOScanningOption_pnKOopClosure_pnXOopsInHeapRegionClosure_pnSG1KlassScanClosure_i_v_+0x3db
V [libjvm.so+0x1166ee3] void G1ParTask::work(unsigned)+0x747;; __1cJG1ParTaskEwork6MI_v_+0x747
V [libjvm.so+0x27d2c4b] void GangWorker::loop()+0x533;; __1cKGangWorkerEloop6M_v_+0x533
V [libjvm.so+0x2183182] java_start+0x1ce;; java_start+0x1ce
C [libc.so.1+0x12257d] _thrp_setup+0xa5;; _thrp_setup+0xa5
C [libc.so.1+0x122820] _lwp_start+0x0;; _lwp_start+0x0


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
  0x0000000002632000 JavaThread "Service Thread" daemon [_thread_blocked, id=64, stack(0xfffffd7f9c52a000,0xfffffd7f9c62a000)]
  0x00000000023fb000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=63, stack(0xfffffd7f9c62b000,0xfffffd7f9c72b000)]
  0x00000000023f8000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=62, stack(0xfffffd7f9c72c000,0xfffffd7f9c82c000)]
  0x00000000023f5800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=61, stack(0xfffffd7f9c82d000,0xfffffd7f9c92d000)]
  0x00000000023f3000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=60, stack(0xfffffd7f9c92e000,0xfffffd7f9ca2e000)]
  0x00000000023a4800 JavaThread "Finalizer" daemon [_thread_blocked, id=59, stack(0xfffffd7f9ca2f000,0xfffffd7f9cb2f000)]
  0x0000000002398800 JavaThread "Reference Handler" daemon [_thread_blocked, id=58, stack(0xfffffd7f9cb30000,0xfffffd7f9cc30000)]
  0x0000000000441800 JavaThread "main" [_thread_blocked, id=2, stack(0xfffffd7ffed3f000,0xfffffd7ffee3f000)]

Other Threads:
  0x0000000002394000 VMThread [stack: 0xfffffd7f9d5fe000,0xfffffd7f9d6fe000] [id=57]
  0x00000000026a7000 WatcherThread [stack: 0xfffffd7f9c429000,0xfffffd7f9c529000] [id=65]

=>0x0000000000481000 (exited) GCTaskThread [stack: 0xfffffd7ffcbfe000,0xfffffd7ffccfe000] [id=24]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000000000043f7d8] Threads_lock - owner thread: 0x0000000002394000
[0x000000000043fff8] Heap_lock - owner thread: 0x0000000000441800

Heap:
 garbage-first heap total 20119552K, used 7196160K [0x0000000046400000, 0x0000000512400000, 0x0000000800000000)
  region size 1024K, 982 young (1005568K), 104 survivors (106496K)
 Metaspace total 4494K, used 3636K, reserved 110592K
  data space 4108K, used 3319K, reserved 8192K
  class space 386K, used 316K, reserved 102400K