I'm not sure that the failure happens due to memory exhaustion. Many JCK tests failed with:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for AllocateHeap
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (.../hotspot/src/share/vm/memory/allocation.inline.hpp:59), pid=5523, tid=2
#
# JRE version: (7.0)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b25-fastdebug compiled mode solaris-amd64 compressed oops)
# Core dump written. Default location: ...
#

--------------- T H R E A D ---------------

Current thread (0x0000000000423800): JavaThread "Unknown thread" [_thread_in_vm, id=2, stack(0xfffffd7ffeeef000,0xfffffd7ffefef000)]

Stack: [0xfffffd7ffeeef000,0xfffffd7ffefef000], sp=0xfffffd7ffefecef0, free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x2b0e1fa] void VMError::report(outputStream*)+0x8de;; __1cHVMErrorGreport6MpnMoutputStream__v_+0x8de
V [libjvm.so+0x2b0f3ff] void VMError::report_and_die()+0x507;; __1cHVMErrorOreport_and_die6M_v_+0x507
V [libjvm.so+0x109c29d] void report_vm_out_of_memory(const char*,int,unsigned long,const char*)+0x91;; __1cXreport_vm_out_of_memory6FpkciL1_v_+0x91
V [libjvm.so+0x26081ee] void ParCompactionManager::initialize(ParMarkBitMap*)+0xdde;; __1cUParCompactionManagerKinitialize6FpnNParMarkBitMap__v_+0xdde
V [libjvm.so+0x2637e61] void PSParallelCompact::post_initialize()+0x22d;; __1cRPSParallelCompactPpost_initialize6F_v_+0x22d
V [libjvm.so+0x254826d] void ParallelScavengeHeap::post_initialize()+0x21;; __1cUParallelScavengeHeapPpost_initialize6M_v_+0x21
V [libjvm.so+0x2a3092a] bool universe_post_init()+0x3c02;; __1cSuniverse_post_init6F_b_+0x3c02
V [libjvm.so+0x149166a] int init_globals()+0x106;; __1cMinit_globals6F_i_+0x106
V [libjvm.so+0x2995f54] int Threads::create_vm(JavaVMInitArgs*,bool*)+0x1bc;; __1cHThreadsJcreate_vm6FpnOJavaVMInitArgs_pb_i_+0x1bc
V [libjvm.so+0x19d15b8] JNI_CreateJavaVM+0x78;; JNI_CreateJavaVM+0x78
C [libjli.so+0x7b29] InitializeJVM+0x109;; InitializeJVM+0x109
C [libjli.so+0x59fc] JavaMain+0x5c;; JavaMain+0x5c
C [libc.so.1+0x1222ad] _thrp_setup+0xa5;; _thrp_setup+0xa5
C [libc.so.1+0x122550] _lwp_start+0x0;; _lwp_start+0x0

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x0000000000423800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2, stack(0xfffffd7ffeeef000,0xfffffd7ffefef000)]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen total 57344K, used 983K [0x00000000c0000000, 0x00000000c4000000, 0x0000000100000000)
  eden space 49152K, 2% used [0x00000000c0000000,0x00000000c00f5c98,0x00000000c3000000)
  from space 8192K, 0% used [0x00000000c3800000,0x00000000c3800000,0x00000000c4000000)
  to space 8192K, 0% used [0x00000000c3000000,0x00000000c3000000,0x00000000c3800000)
 ParOldGen total 129024K, used 0K [0x0000000040200000, 0x0000000048000000, 0x00000000c0000000)
  object space 129024K, 0% used [0x0000000040200000,0x0000000040200000,0x0000000048000000)
 PSPermGen total 22528K, used 813K [0x000000003b000000, 0x000000003c600000, 0x0000000040200000)
  object space 22528K, 3% used [0x000000003b000000,0x000000003b0cb550,0x000000003c600000)

Card table byte_map: [0xfffffd7ff9a00000,0xfffffd7ffa029000] byte_map_base: 0xfffffd7ff9828000

Polling page: 0xfffffd7ffedb0000

Code Cache [0xfffffd7ffac00000, 0xfffffd7ffb000000, 0xfffffd7ffdc00000)
 total_blobs=36 nmethods=0 adapters=20 free_code_cache=48284Kb largest_free_block=49442752

Compilation events (0 events):
No events

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Internal exceptions (0 events):
No events

Events (10 events):
Event: 1.174 loading class 0x00000000004b2088
Event: 1.175 loading class 0x00000000004b2088 done
Event: 1.175 loading class 0x00000000004b1a68
Event: 1.175 loading class 0x00000000004b1a68 done
Event: 1.175 loading class 0x0000000000602e48
Event: 1.175 loading class 0x0000000000602e48 done
Event: 1.175 loading class 0x00000000004b1a18
Event: 1.175 loading class 0x00000000004b1a18 done
Event: 1.175 loading class 0x00000000004b24c8
Event: 1.175 loading class 0x00000000004b24c8 done


Dynamic libraries:
0x0000000000400000 .../bin/amd64/java
0xfffffd7ffdd1c000 /lib/64/libthread.so.1
0xfffffd7d61250000 .../jre/lib/amd64/jli/libjli.so
0xfffffd7ffdd1b000 /lib/64/libdl.so.1
0xfffffd7fff140000 /lib/64/libc.so.1
0xfffffd7cec350000 .../jre/lib/amd64/server/libjvm.so
0xfffffd7ffed30000 /lib/64/libsocket.so.1
0xfffffd7ffe63f000 /usr/lib/64/libsched.so.1
0xfffffd7ffa2a0000 /lib/64/libm.so.1
0xfffffd7ffa270000 /usr/lib/64/libCrun.so.1
0xfffffd7ffea6f000 /lib/64/libdoor.so.1
0xfffffd7ffa230000 /usr/lib/64/libdemangle.so.1
0xfffffd7ffee40000 /lib/64/libm.so.2
0xfffffd7ffec90000 /lib/64/libnsl.so.1
0xfffffd7ffa210000 /lib/64/libmd.so.1
0xfffffd7ffa1f0000 /lib/64/libmp.so.2
0xfffffd7d61220000 .../jre/lib/amd64/libverify.so
0xfffffd7d611e0000 .../jre/lib/amd64/libjava.so
0xfffffd7ffa130000 /lib/64/libscf.so.1
0xfffffd7ffa110000 /lib/64/libuutil.so.1
0xfffffd7ffa0f0000 /lib/64/libgen.so.1
0xfffffd7ffdeb0000 /lib/64/libnvpair.so.1
0xfffffd7ffa0d0000 /usr/lib/64/libsmbios.so.1
0xfffffd7d611b0000 .../jre/lib/amd64/libzip.so

VM Arguments:
jvm_args: -Xcomp -Djava.awt.headless=true -Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=100:27000:3600:5 -Djava.ext.dirs=.../jre/lib/ext -Xverify:all -Djava.security.policy=.../jck/7/lib/jck.policy
java_command: javasoft.sqe.tests.api.java.lang.ArithmeticException.serial.ConstructorTests
Launcher Type: SUN_STANDARD


--------------- S Y S T E M ---------------

OS: Oracle Solaris 11 11/11 X86
  Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
                            Assembled 18 October 2011

uname:SunOS 5.11 11.0 i86pc
  (T2 libthread)
rlimit: STACK 10240k, CORE infinity, NOFILE 65536, AS infinity
load average:0.87 0.70 1.50

CPU:total 24 (6 cores per cpu, 2 threads per core) family 6 model 44 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht, tsc, tscinvbit, tscinv

Memory: 4k page, physical 12571012k(2727408k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (24.0-b25-fastdebug) for solaris-amd64 JRE (1.7.0-internal-fastdebug-...), built on Nov 9 2012 08:02:25 by "..." with Sun Studio 12u1

time: Mon Nov 12 22:17:44 2012
elapsed time: 5 seconds