RULE bigapps/Weblogic+medrec/stability Crash Out of Memory Error

hs_err:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 195376 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (C:\ws32\jdk7u-jfr\2187\hotspot\src\share\vm\memory\allocation.cpp:323), pid=16216, tid=27632
#
# JRE version: Java(TM) SE Runtime Environment (7.0_12)
# Java VM: Java HotSpot(TM) Server VM (24.0-b25-fastdebug mixed mode windows-x86 )
# Failed to write core dump.
#

--------------- T H R E A D ---------------

Current thread (0x60bd8000): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=27632, stack(0x62260000,0x62360000)]

Stack: [0x62260000,0x62360000], sp=0x6235d570, free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x29aec1]Warning: Warning: no map file found for C:/local/common/jdk/baseline/windows-i586/jre/bin/server/jvm.dll!


V [jvm.dll+0x2b39c1]
V [jvm.dll+0x2b67e4]
V [jvm.dll+0x2b8066]
C [msvcr100.dll+0x5c6de]
C [msvcr100.dll+0x5c788]
C [kernel32.dll+0x13677]
C [ntdll.dll+0x39f42]
C [ntdll.dll+0x39f15]


Current CompileTask:
C2:26828958 9377 ! com.sun.el.lang.ExpressionBuilder::createNodeInternal (248 bytes)


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
  0x6a19c800 JavaThread "[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=22140, stack(0x6ca10000,0x6ca60000)]
  0x6a19c000 JavaThread "[ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_native, id=19516, stack(0x6caa0000,0x6caf0000)]
  0x6a19bc00 JavaThread "[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=20980, stack(0x6cb50000,0x6cba0000)]
  0x6a19ac00 JavaThread "[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_vm, id=7456, stack(0x6c860000,0x6c8b0000)]
  0x67dd9400 JavaThread "[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=9932, stack(0x6c990000,0x6c9e0000)]
  0x67ddd000 JavaThread "[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=25368, stack(0x6c780000,0x6c7d0000)]
  0x67dd7400 JavaThread "[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=15468, stack(0x6c800000,0x6c850000)]
  0x67dd6c00 JavaThread "[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=13100, stack(0x6c6b0000,0x6c700000)]
  0x67dda400 JavaThread "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=19420, stack(0x6c580000,0x6c5d0000)]
  0x67dd7c00 JavaThread "[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=2740, stack(0x6c5d0000,0x6c620000)]
  0x67ddd800 JavaThread "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=21172, stack(0x6bc10000,0x6bc60000)]
  0x67ddac00 JavaThread "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=7432, stack(0x6c1d0000,0x6c220000)]
  0x67ddc800 JavaThread "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_native, id=25228, stack(0x6c3d0000,0x6c420000)]
  0x6a19b400 JavaThread "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_native, id=10384, stack(0x6f740000,0x6f790000)]
  0x67ddb000 JavaThread "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=5560, stack(0x6c340000,0x6c390000)]
  0x67dd9c00 JavaThread "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=8016, stack(0x6bdf0000,0x6be40000)]
  0x67dd8400 JavaThread "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=6924, stack(0x6c0b0000,0x6c100000)]
  0x67dd8c00 JavaThread "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=20716, stack(0x6bea0000,0x6bef0000)]
  0x67ddb800 JavaThread "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=20704, stack(0x6bef0000,0x6bf40000)]
  0x67ddc000 JavaThread "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=4092, stack(0x6bc60000,0x6bcb0000)]
  0x63ab5c00 JavaThread "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=27392, stack(0x6cda0000,0x6cdf0000)]
  0x67dd6800 JavaThread "[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=14848, stack(0x6bcc0000,0x6bd10000)]
  0x63308000 JavaThread "DynamicListenThread[Default[9]]" daemon [_thread_in_native, id=21268, stack(0x6bab0000,0x6bb00000)]
  0x63307800 JavaThread "DynamicSSLListenThread[DefaultSecure[7]]" daemon [_thread_in_native, id=8572, stack(0x6b9d0000,0x6ba20000)]
  0x63307000 JavaThread "DynamicListenThread[Default[2]]" daemon [_thread_in_native, id=22576, stack(0x6ba20000,0x6ba70000)]
  0x63306800 JavaThread "DynamicSSLListenThread[DefaultSecure[2]]" daemon [_thread_in_native, id=26496, stack(0x6b980000,0x6b9d0000)]
  0x63306000 JavaThread "DynamicSSLListenThread[DefaultSecure[5]]" daemon [_thread_in_native, id=21956, stack(0x6b830000,0x6b880000)]
  0x63305c00 JavaThread "DynamicSSLListenThread[DefaultSecure[3]]" daemon [_thread_in_native, id=11176, stack(0x6b7b0000,0x6b800000)]
  0x63305400 JavaThread "DynamicSSLListenThread[DefaultSecure[9]]" daemon [_thread_in_native, id=17184, stack(0x6b650000,0x6b6a0000)]
  0x63304c00 JavaThread "DynamicSSLListenThread[DefaultSecure[8]]" daemon [_thread_in_native, id=18496, stack(0x6b4f0000,0x6b540000)]
  0x63304400 JavaThread "DynamicListenThread[Default[4]]" daemon [_thread_in_native, id=18860, stack(0x6b0a0000,0x6b0f0000)]
  0x63303c00 JavaThread "DynamicSSLListenThread[DefaultSecure[6]]" daemon [_thread_in_native, id=23788, stack(0x6b280000,0x6b2d0000)]
  0x63303400 JavaThread "DynamicListenThread[Default[3]]" daemon [_thread_in_native, id=20904, stack(0x6b1d0000,0x6b220000)]
  0x63302c00 JavaThread "DynamicListenThread[Default[7]]" daemon [_thread_in_native, id=5240, stack(0x6b3e0000,0x6b430000)]
  0x63302400 JavaThread "DynamicListenThread[Default[1]]" daemon [_thread_in_native, id=23096, stack(0x6b350000,0x6b3a0000)]
  0x63302000 JavaThread "DynamicListenThread[Default[8]]" daemon [_thread_in_native, id=8028, stack(0x6b2d0000,0x6b320000)]
  0x63301800 JavaThread "DynamicListenThread[Default[5]]" daemon [_thread_in_native, id=27164, stack(0x6ad70000,0x6adc0000)]
  0x6386e000 JavaThread "DynamicListenThread[Default[6]]" daemon [_thread_in_native, id=9816, stack(0x6b150000,0x6b1a0000)]
  0x6386d800 JavaThread "DynamicListenThread[Default]" daemon [_thread_in_native, id=12804, stack(0x6b040000,0x6b090000)]
  0x6386d000 JavaThread "DynamicSSLListenThread[DefaultSecure[4]]" daemon [_thread_in_native, id=3744, stack(0x6afd0000,0x6b020000)]
  0x6386b800 JavaThread "DynamicSSLListenThread[DefaultSecure[1]]" daemon [_thread_in_native, id=8280, stack(0x6aee0000,0x6af30000)]
  0x6386c800 JavaThread "DynamicSSLListenThread[DefaultSecure]" daemon [_thread_in_native, id=16128, stack(0x69a10000,0x69a60000)]
  0x6386ac00 JavaThread "weblogic.GCMonitor" daemon [_thread_blocked, id=13800, stack(0x6ae40000,0x6ae90000)]
  0x6386c000 JavaThread "Thread-14" [_thread_blocked, id=12080, stack(0x6abe0000,0x6ac30000)]
  0x6386a400 JavaThread "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=284, stack(0x69870000,0x698c0000)]
  0x63869c00 JavaThread "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=6612, stack(0x69980000,0x699d0000)]
  0x63869400 JavaThread "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=26848, stack(0x686c0000,0x68710000)]
  0x63868c00 JavaThread "derby.rawStoreDaemon" daemon [_thread_blocked, id=21388, stack(0x6aa40000,0x6aa90000)]
  0x63868400 JavaThread "Timer-2" daemon [_thread_blocked, id=7508, stack(0x698c0000,0x69910000)]
  0x63867c00 JavaThread "derby.antiGC" daemon [_thread_blocked, id=12312, stack(0x67600000,0x67650000)]
  0x63867000 JavaThread "Thread-12" [_thread_blocked, id=10088, stack(0x67450000,0x674a0000)]
  0x62eb4c00 JavaThread "DoSManager" daemon [_thread_blocked, id=20344, stack(0x67690000,0x676e0000)]
  0x635a8400 JavaThread "VDE Transaction Processor Thread" daemon [_thread_blocked, id=2852, stack(0x644b0000,0x64500000)]
  0x63728400 JavaThread "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_native, id=8380, stack(0x67330000,0x67380000)]
  0x638ed800 JavaThread "weblogic.time.TimeEventGenerator" daemon [_thread_blocked, id=12992, stack(0x621e0000,0x62230000)]
  0x63d0e400 JavaThread "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=5136, stack(0x646d0000,0x64720000)]
  0x61194000 JavaThread "Timer-0" daemon [_thread_blocked, id=3904, stack(0x64510000,0x64560000)]
  0x60c51c00 JavaThread "Service Thread" daemon [_thread_blocked, id=25512, stack(0x61dd0000,0x61e20000)]
  0x60bd8800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=24060, stack(0x62410000,0x62510000)]
=>0x60bd8000 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=27632, stack(0x62260000,0x62360000)]
  0x60b07c00 JavaThread "VM JFR Buffer Thread" daemon [_thread_blocked, id=10348, stack(0x5f200000,0x5f250000)]
  0x60233000 JavaThread "JFR request timer" daemon [_thread_blocked, id=23708, stack(0x61e90000,0x61ee0000)]
  0x601fa800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9500, stack(0x61c90000,0x61ce0000)]
  0x603c6400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=8692, stack(0x61c40000,0x61c90000)]
  0x5f88dc00 JavaThread "Finalizer" daemon [_thread_blocked, id=15624, stack(0x5b8d0000,0x5b920000)]
  0x5f889c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=20856, stack(0x5fa20000,0x5fa70000)]
  0x00c38000 JavaThread "main" [_thread_blocked, id=22912, stack(0x00be0000,0x00c30000)]

test:
bigapps/Weblogic+medrec/stability

environment: SLCAF436, Intel Xeon 3059 MHz, 24 cores, 144G, Win32 / Microsoft Windows 2008 R2
jdk: 1.7.0_12 b0 (1.7.0_12-ea-fastdebug-jfr-nightly-h2187-20121207-01-b00)
arch: 32
run time: 7+h
reproducable: not yet

investigation is in progress

according to gc.log ParOldGen reached it's maximum, numerous FullGC only events, then OOME