C2 tries to allocate a 2GB chunk after PhaseIdealLoop::do_peeling() and PhaseIdealLoop::clone_loop()
and runs out of memory:

% more hs_err_pid10640.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2147483664 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:328), pid=10640, tid=139787454797568
#
# JRE version: 7.0_17-b02
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

--------------- T H R E A D ---------------

Current thread (0x00007f22d40f5800): JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=10655, stack(0x00007f22cd865000,0x00007f22cd966000)]

Stack: [0x00007f22cd865000,0x00007f22cd966000], sp=0x00007f22cd960ad0, free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x8a540a] VMError::report_and_die()+0x2ea
V [libjvm.so+0x40abfb] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x22b063] Chunk::operator new(unsigned long, unsigned long)+0x133
V [libjvm.so+0x22b0af] Arena::grow(unsigned long)+0x3f
V [libjvm.so+0x783267] NodeHash::grow()+0x197
V [libjvm.so+0x783493] NodeHash::hash_find_insert(Node*)+0x173
V [libjvm.so+0x69564f] PhaseIdealLoop::clone_loop(IdealLoopTree*, Node_List&, int, Node*)+0x165f
V [libjvm.so+0x678afa] PhaseIdealLoop::do_peeling(IdealLoopTree*, Node_List&)+0x6a
V [libjvm.so+0x67f2ce] IdealLoopTree::iteration_split_impl(PhaseIdealLoop*, Node_List&)+0x2de
V [libjvm.so+0x67f648] IdealLoopTree::iteration_split(PhaseIdealLoop*, Node_List&)+0x148
V [libjvm.so+0x67f534] IdealLoopTree::iteration_split(PhaseIdealLoop*, Node_List&)+0x34
V [libjvm.so+0x68cf62] PhaseIdealLoop::build_and_optimize(bool, bool)+0x932
V [libjvm.so+0x3bb55c] Compile::Optimize()+0x21c
V [libjvm.so+0x3bce7c] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool)+0xdbc
V [libjvm.so+0x32ebd2] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x142
V [libjvm.so+0x3c1df3] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x323
V [libjvm.so+0x3c2f5d] CompileBroker::compiler_thread_loop()+0x43d
V [libjvm.so+0x8651b8] JavaThread::thread_main_inner()+0xc8
V [libjvm.so+0x865308] JavaThread::run()+0x138
V [libjvm.so+0x746cf0] java_start(Thread*)+0x100
[...]
--------------- S Y S T E M ---------------

OS:CentOS release 6.3 (Final)

uname:Linux 2.6.32-279.22.1.el6.x86_64 #1 SMP Wed Feb 6 03:10:46 UTC 2013 x86_64
[...]
vm_info: Java HotSpot(TM) 64-Bit Server VM (23.7-b01) for linux-amd64 JRE (1.7.0_17-b02), built on Mar 1 2013 02:59:06 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Mon Jan 20 16:56:56 2014
elapsed time: 4786 seconds
%