FULL PRODUCT VERSION :
Java 1.6 Update 7

ADDITIONAL OS VERSION INFORMATION :
Windows 2008

EXTRA RELEVANT SYSTEM CONFIGURATION :
Virtual Machine

A DESCRIPTION OF THE PROBLEM :
I have an application that uses Java to run jobs. The jobs are configured for a maximum heap size of 12 GBs per agent/thread. Occasionally, a the job will hang on a particular agent and throws the hs_err_pid log. When this happens, the server still has plenty of memory left.

Details of the log:


#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 397936 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:218), pid=54876, tid=244824
# Error: Chunk::new
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode windows-amd64)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x0000000005dc7c00): JavaThread "CompilerThread0" daemon [_thread_in_native, id=244824, stack(0x0000000006500000,0x0000000006600000)]

Stack: [0x0000000006500000,0x0000000006600000]

Current CompileTask:
C2:587 ! citynet.common.Util.Replace(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Lcitynet/common/Messages;)Ljava/lang/String; (423 bytes)


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
  0x0000000006dfcc00 JavaThread "KeepAlive" daemon [_thread_blocked, id=56988, stack(0x0000000007d60000,0x0000000007e60000)]
  0x0000000006dfd000 JavaThread "Agent" daemon [_thread_in_native, id=246740, stack(0x0000000008fb0000,0x00000000090b0000)]
  0x0000000006dfe000 JavaThread "RMI RenewClean-[10.197.223.246:56647]" daemon [_thread_blocked, id=79992, stack(0x00000000088b0000,0x00000000089b0000)]
  0x0000000000483000 JavaThread "DestroyJavaVM" [_thread_blocked, id=246124, stack(0x00000000010b0000,0x00000000011b0000)]
  0x0000000006ba0800 JavaThread "RMI Reaper" [_thread_blocked, id=59428, stack(0x0000000008cb0000,0x0000000008db0000)]
  0x0000000006ba0000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=242668, stack(0x0000000008bb0000,0x0000000008cb0000)]
  0x00000000004c9000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=242796, stack(0x00000000089b0000,0x0000000008ab0000)]
  0x00000000004c8800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=36120, stack(0x00000000087b0000,0x00000000088b0000)]
  0x0000000006af2c00 JavaThread "GC Daemon" daemon [_thread_blocked, id=246448, stack(0x00000000086b0000,0x00000000087b0000)]
  0x00000000071c0400 JavaThread "RMI RenewClean-[10.197.223.246:8088]" daemon [_thread_blocked, id=20696, stack(0x00000000085b0000,0x00000000086b0000)]
  0x0000000006c6c000 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=9608, stack(0x0000000007c60000,0x0000000007d60000)]
  0x0000000005dd2000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=65036, stack(0x0000000006700000,0x0000000006800000)]
  0x0000000005dd0c00 JavaThread "CompilerThread1" daemon [_thread_blocked, id=208296, stack(0x0000000006600000,0x0000000006700000)]
=>0x0000000005dc7c00 JavaThread "CompilerThread0" daemon [_thread_in_native, id=244824, stack(0x0000000006500000,0x0000000006600000)]
  0x0000000005dc3400 JavaThread "Attach Listener" daemon [_thread_blocked, id=232380, stack(0x0000000006400000,0x0000000006500000)]
  0x0000000005d9ec00 JavaThread "Finalizer" daemon [_thread_blocked, id=65212, stack(0x00000000060a0000,0x00000000061a0000)]
  0x0000000005d99000 JavaThread "Reference Handler" daemon [_thread_blocked, id=244908, stack(0x0000000005fa0000,0x00000000060a0000)]

Other Threads:
  0x0000000005d8e400 VMThread [stack: 0x0000000005ea0000,0x0000000005fa0000] [id=240224]
  0x0000000005dd3c00 WatcherThread [stack: 0x0000000006800000,0x0000000006900000] [id=24040]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen total 215424K, used 45881K [0x00000001853f0000, 0x0000000197930000, 0x00000002053f0000)
  eden space 178304K, 25% used [0x00000001853f0000,0x00000001880be560,0x0000000190210000)
  from space 37120K, 0% used [0x0000000190210000,0x0000000190210000,0x0000000192650000)
  to space 36032K, 0% used [0x0000000195600000,0x0000000195600000,0x0000000197930000)
 PSOldGen total 44352K, used 44352K [0x00000000853f0000, 0x0000000087f40000, 0x00000001853f0000)
  object space 44352K, 100% used [0x00000000853f0000,0x0000000087f40000,0x0000000087f40000)
 PSPermGen total 23872K, used 16636K [0x000000007fff0000, 0x0000000081740000, 0x00000000853f0000)
  object space 23872K, 69% used [0x000000007fff0000,0x000000008102f1d8,0x0000000081740000)

Dynamic libraries:
0x0000000000400000 - 0x000000000042d000 D:\RECS7Engines\Java\jre1.6.0_07\bin\java.exe
0x0000000077780000 - 0x0000000077929000 C:\Windows\SYSTEM32\ntdll.dll
0x0000000077400000 - 0x000000007751f000 C:\Windows\system32\kernel32.dll
0x000007fefdf40000 - 0x000007fefdfab000 C:\Windows\system32\KERNELBASE.dll
0x000007feff980000 - 0x000007feffa5b000 C:\Windows\system32\ADVAPI32.dll
0x000007fefe0d0000 - 0x000007fefe16f000 C:\Windows\system32\msvcrt.dll
0x000007fefe3d0000 - 0x000007fefe3ef000 C:\Windows\SYSTEM32\sechost.dll
0x000007feff850000 - 0x000007feff97d000 C:\Windows\system32\RPCRT4.dll
0x0000000008000000 - 0x00000000085aa000 D:\RECS7Engines\Java\jre1.6.0_07\bin\server\jvm.dll
0x0000000077520000 - 0x000000007761a000 C:\Windows\system32\USER32.dll
0x000007feff600000 - 0x000007feff667000 C:\Windows\system32\GDI32.dll
0x000007feff360000 - 0x000007feff36e000 C:\Windows\system32\LPK.dll
0x000007fefe300000 - 0x000007fefe3c9000 C:\Windows\system32\USP10.dll
0x000007fef4690000 - 0x000007fef46cb000 C:\Windows\system32\WINMM.dll
0x000007feffa60000 - 0x000007feffa8e000 C:\Windows\system32\IMM32.DLL
0x000007fefe190000 - 0x000007fefe299000 C:\Windows\system32\MSCTF.dll
0x0000000010000000 - 0x000000001000a000 D:\RECS7Engines\Java\jre1.6.0_07\bin\hpi.dll
0x0000000077940000 - 0x0000000077947000 C:\Windows\system32\PSAPI.DLL
0x0000000000250000 - 0x000000000025e000 D:\RECS7Engines\Java\jre1.6.0_07\bin\verify.dll
0x0000000000260000 - 0x0000000000287000 D:\RECS7Engines\Java\jre1.6.0_07\bin\java.dll
0x0000000000390000 - 0x00000000003a2000 D:\RECS7Engines\Java\jre1.6.0_07\bin\zip.dll
0x000007fefe5d0000 - 0x000007feff358000 C:\Windows\system32\SHELL32.DLL
0x000007feff370000 - 0x000007feff3e1000 C:\Windows\system32\SHLWAPI.dll
0x000007feff3f0000 - 0x000007feff5f3000 C:\Windows\system32\ole32.dll
0x000007fefdad0000 - 0x000007fefdadf000 C:\Windows\system32\CRYPTBASE.dll
0x000007fefc590000 - 0x000007fefc784000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_fa396087175ac9ac\comctl32.dll
0x000007fefe490000 - 0x000007fefe567000 C:\Windows\system32\OLEAUT32.dll
0x000007fefdfb0000 - 0x000007fefe049000 C:\Windows\system32\CLBCatQ.DLL
0x000007feff670000 - 0x000007feff847000 C:\Windows\system32\SETUPAPI.dll
0x000007fefdd90000 - 0x000007fefddc6000 C:\Windows\system32\CFGMGR32.dll
0x000007fefdd30000 - 0x000007fefdd4a000 C:\Windows\system32\DEVOBJ.dll
0x000007fefbec0000 - 0x000007fefbfec000 C:\Windows\system32\propsys.dll
0x00000000061a0000 - 0x00000000061a6000 D:\Recs7Engines\Java\jre1.6.0_07\bin\rmi.dll
0x00000000075c0000 - 0x00000000075d7000 D:\Recs7Engines\Java\jre1.6.0_07\bin\net.dll
0x000007fefe580000 - 0x000007fefe5cd000 C:\Windows\system32\WS2_32.dll
0x000007fefe570000 - 0x000007fefe578000 C:\Windows\system32\NSI.dll
0x000007fefd410000 - 0x000007fefd465000 C:\Windows\system32\mswsock.dll
0x000007fefd400000 - 0x000007fefd407000 C:\Windows\System32\wship6.dll
0x000007fefc570000 - 0x000007fefc585000 C:\Windows\system32\NLAapi.dll
0x000007fefafb0000 - 0x000007fefafc5000 C:\Windows\system32\napinsp.dll
0x000007fefd290000 - 0x000007fefd2eb000 C:\Windows\system32\DNSAPI.dll
0x000007fefafe0000 - 0x000007fefafeb000 C:\Windows\System32\winrnr.dll
0x000007fefcdf0000 - 0x000007fefcdf7000 C:\Windows\System32\wshtcpip.dll
0x000007fefbbe0000 - 0x000007fefbc07000 C:\Windows\system32\IPHLPAPI.DLL
0x000007fefbbd0000 - 0x000007fefbbdb000 C:\Windows\system32\WINNSI.DLL
0x000007fefb2c0000 - 0x000007fefb2c8000 C:\Windows\system32\rasadhlp.dll
0x000007fefb610000 - 0x000007fefb663000 C:\Windows\System32\fwpuclnt.dll
0x000007fefd470000 - 0x000007fefd487000 C:\Windows\system32\CRYPTSP.dll
0x000007fefd170000 - 0x000007fefd1b7000 C:\Windows\system32\rsaenh.dll
0x000007fefcf10000 - 0x000007fefcf2e000 C:\Windows\system32\USERENV.dll
0x000007fefdbe0000 - 0x000007fefdbef000 C:\Windows\system32\profapi.dll
0x00000000075e0000 - 0x00000000075ea000 D:\Recs7Engines\Java\jre1.6.0_07\bin\management.dll
0x0000000070310000 - 0x0000000070329000 D:\Recs7Engines\ApplicationEngines\Engines\Run\sqljdbc_auth.dll
0x000007fefd9f0000 - 0x000007fefd9fb000 C:\Windows\system32\secur32.dll
0x000007fefda40000 - 0x000007fefda65000 C:\Windows\system32\SSPICLI.DLL
0x000007fefd070000 - 0x000007fefd07a000 C:\Windows\system32\credssp.dll
0x000007fefd490000 - 0x000007fefd544000 C:\Windows\system32\kerberos.DLL
0x000007fefd6f0000 - 0x000007fefd704000 C:\Windows\system32\cryptdll.dll
0x000007fefdc80000 - 0x000007fefdc8f000 C:\Windows\system32\MSASN1.dll
0x000007fefd3a0000 - 0x000007fefd3f1000 C:\Windows\system32\msv1_0.DLL
0x000007fefb270000 - 0x000007fefb297000 C:\Windows\system32\ntdsapi.dll

VM Arguments:
jvm_args: -Djava.security.policy=D:\RECS7Engines\ApplicationEngines\Engines\Run\server.policy -Djava.rmi.server.codebase=file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/AAADateRoll.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/activation.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/Amalgamation.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/apache-mime4j-0.6.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/Archival.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/asm-3.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/axis.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/bcel-5.1-jnbridge.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/cglib-2.2.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecs.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecsTest.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetSchedulerPlugIn.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudio.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudioTest.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mq.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqbind.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqjms.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-codec-1.3.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-discovery-0.2.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-io-2.4.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-lang3-3.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-logging-1.1.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/connector.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/fscontext.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpclient-4.0.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpcore-4.0.1.jar file:///D:\RECS7Engines\ApplicationEng-Xmx6144M -Dclasspath=D:\RECS7Engines\ApplicationEngines\Engines\Jars/AAADateRoll.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/activation.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/Amalgamation.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/apache-mime4j-0.6.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/Archival.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/asm-3.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/axis.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/bcel-5.1-jnbridge.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/cglib-2.2.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecs.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecsTest.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetSchedulerPlugIn.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudio.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudioTest.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mq.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqbind.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqjms.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-codec-1.3.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-discovery-0.2.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-io-2.4.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-lang3-3.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-logging-1.1.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/connector.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/fscontext.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpclient-4.0.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpcore-4.0.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpmime-4.0.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/ifxjdbc.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/jaxb-api.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/jaxen-1.1.1.jar;D:\RECS7E-Xrs
java_command: sun.rmi.server.ActivationGroupInit
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Symantec\Critical System Protection\Agent\IPS\bin;C:\IBM\ITM\bin;C:\IBM\ITM\TMAITM6;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia AUX;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia AUX\Support binaries;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia Broker;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia Client;C:\IBM\ITM\InstallITM
USERNAME=precseng
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 47 Stepping 2, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows NT 6.1 Build 7601 Service Pack 1

CPU:total 6 (1 cores per cpu, 1 threads per core) family 6 model 15 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 33553912k(17588460k free), swap 42255028k(5492k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (10.0-b23) for windows-amd64 JRE (1.6.0_07-b06), built on Jun 10 2008 01:06:11 by "java_re" with MS VC++ 8.0

time: Tue Aug 05 10:37:16 2014
elapsed time: 226 seconds



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Can't constantly be reproduce but usually these are the events prior to the issue happening:

1. about 1000+ jobs runs on a specific agent (varies on using different amounts of memory)
2. Happens over the course of 3 days - 2 weeks
3. The problem happens and job stuck on the tread

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
- Job should not Hang

ACTUAL -
- Job will Hang
- Hs_err_pid log will be created.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 397936 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:218), pid=54876, tid=244824
# Error: Chunk::new
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode windows-amd64)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x0000000005dc7c00): JavaThread "CompilerThread0" daemon [_thread_in_native, id=244824, stack(0x0000000006500000,0x0000000006600000)]

Stack: [0x0000000006500000,0x0000000006600000]

Current CompileTask:
C2:587 ! citynet.common.Util.Replace(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Lcitynet/common/Messages;)Ljava/lang/String; (423 bytes)


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
  0x0000000006dfcc00 JavaThread "KeepAlive" daemon [_thread_blocked, id=56988, stack(0x0000000007d60000,0x0000000007e60000)]
  0x0000000006dfd000 JavaThread "Agent" daemon [_thread_in_native, id=246740, stack(0x0000000008fb0000,0x00000000090b0000)]
  0x0000000006dfe000 JavaThread "RMI RenewClean-[10.197.223.246:56647]" daemon [_thread_blocked, id=79992, stack(0x00000000088b0000,0x00000000089b0000)]
  0x0000000000483000 JavaThread "DestroyJavaVM" [_thread_blocked, id=246124, stack(0x00000000010b0000,0x00000000011b0000)]
  0x0000000006ba0800 JavaThread "RMI Reaper" [_thread_blocked, id=59428, stack(0x0000000008cb0000,0x0000000008db0000)]
  0x0000000006ba0000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=242668, stack(0x0000000008bb0000,0x0000000008cb0000)]
  0x00000000004c9000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=242796, stack(0x00000000089b0000,0x0000000008ab0000)]
  0x00000000004c8800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=36120, stack(0x00000000087b0000,0x00000000088b0000)]
  0x0000000006af2c00 JavaThread "GC Daemon" daemon [_thread_blocked, id=246448, stack(0x00000000086b0000,0x00000000087b0000)]
  0x00000000071c0400 JavaThread "RMI RenewClean-[10.197.223.246:8088]" daemon [_thread_blocked, id=20696, stack(0x00000000085b0000,0x00000000086b0000)]
  0x0000000006c6c000 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=9608, stack(0x0000000007c60000,0x0000000007d60000)]
  0x0000000005dd2000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=65036, stack(0x0000000006700000,0x0000000006800000)]
  0x0000000005dd0c00 JavaThread "CompilerThread1" daemon [_thread_blocked, id=208296, stack(0x0000000006600000,0x0000000006700000)]
=>0x0000000005dc7c00 JavaThread "CompilerThread0" daemon [_thread_in_native, id=244824, stack(0x0000000006500000,0x0000000006600000)]
  0x0000000005dc3400 JavaThread "Attach Listener" daemon [_thread_blocked, id=232380, stack(0x0000000006400000,0x0000000006500000)]
  0x0000000005d9ec00 JavaThread "Finalizer" daemon [_thread_blocked, id=65212, stack(0x00000000060a0000,0x00000000061a0000)]
  0x0000000005d99000 JavaThread "Reference Handler" daemon [_thread_blocked, id=244908, stack(0x0000000005fa0000,0x00000000060a0000)]

Other Threads:
  0x0000000005d8e400 VMThread [stack: 0x0000000005ea0000,0x0000000005fa0000] [id=240224]
  0x0000000005dd3c00 WatcherThread [stack: 0x0000000006800000,0x0000000006900000] [id=24040]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen total 215424K, used 45881K [0x00000001853f0000, 0x0000000197930000, 0x00000002053f0000)
  eden space 178304K, 25% used [0x00000001853f0000,0x00000001880be560,0x0000000190210000)
  from space 37120K, 0% used [0x0000000190210000,0x0000000190210000,0x0000000192650000)
  to space 36032K, 0% used [0x0000000195600000,0x0000000195600000,0x0000000197930000)
 PSOldGen total 44352K, used 44352K [0x00000000853f0000, 0x0000000087f40000, 0x00000001853f0000)
  object space 44352K, 100% used [0x00000000853f0000,0x0000000087f40000,0x0000000087f40000)
 PSPermGen total 23872K, used 16636K [0x000000007fff0000, 0x0000000081740000, 0x00000000853f0000)
  object space 23872K, 69% used [0x000000007fff0000,0x000000008102f1d8,0x0000000081740000)

Dynamic libraries:
0x0000000000400000 - 0x000000000042d000 D:\RECS7Engines\Java\jre1.6.0_07\bin\java.exe
0x0000000077780000 - 0x0000000077929000 C:\Windows\SYSTEM32\ntdll.dll
0x0000000077400000 - 0x000000007751f000 C:\Windows\system32\kernel32.dll
0x000007fefdf40000 - 0x000007fefdfab000 C:\Windows\system32\KERNELBASE.dll
0x000007feff980000 - 0x000007feffa5b000 C:\Windows\system32\ADVAPI32.dll
0x000007fefe0d0000 - 0x000007fefe16f000 C:\Windows\system32\msvcrt.dll
0x000007fefe3d0000 - 0x000007fefe3ef000 C:\Windows\SYSTEM32\sechost.dll
0x000007feff850000 - 0x000007feff97d000 C:\Windows\system32\RPCRT4.dll
0x0000000008000000 - 0x00000000085aa000 D:\RECS7Engines\Java\jre1.6.0_07\bin\server\jvm.dll
0x0000000077520000 - 0x000000007761a000 C:\Windows\system32\USER32.dll
0x000007feff600000 - 0x000007feff667000 C:\Windows\system32\GDI32.dll
0x000007feff360000 - 0x000007feff36e000 C:\Windows\system32\LPK.dll
0x000007fefe300000 - 0x000007fefe3c9000 C:\Windows\system32\USP10.dll
0x000007fef4690000 - 0x000007fef46cb000 C:\Windows\system32\WINMM.dll
0x000007feffa60000 - 0x000007feffa8e000 C:\Windows\system32\IMM32.DLL
0x000007fefe190000 - 0x000007fefe299000 C:\Windows\system32\MSCTF.dll
0x0000000010000000 - 0x000000001000a000 D:\RECS7Engines\Java\jre1.6.0_07\bin\hpi.dll
0x0000000077940000 - 0x0000000077947000 C:\Windows\system32\PSAPI.DLL
0x0000000000250000 - 0x000000000025e000 D:\RECS7Engines\Java\jre1.6.0_07\bin\verify.dll
0x0000000000260000 - 0x0000000000287000 D:\RECS7Engines\Java\jre1.6.0_07\bin\java.dll
0x0000000000390000 - 0x00000000003a2000 D:\RECS7Engines\Java\jre1.6.0_07\bin\zip.dll
0x000007fefe5d0000 - 0x000007feff358000 C:\Windows\system32\SHELL32.DLL
0x000007feff370000 - 0x000007feff3e1000 C:\Windows\system32\SHLWAPI.dll
0x000007feff3f0000 - 0x000007feff5f3000 C:\Windows\system32\ole32.dll
0x000007fefdad0000 - 0x000007fefdadf000 C:\Windows\system32\CRYPTBASE.dll
0x000007fefc590000 - 0x000007fefc784000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_fa396087175ac9ac\comctl32.dll
0x000007fefe490000 - 0x000007fefe567000 C:\Windows\system32\OLEAUT32.dll
0x000007fefdfb0000 - 0x000007fefe049000 C:\Windows\system32\CLBCatQ.DLL
0x000007feff670000 - 0x000007feff847000 C:\Windows\system32\SETUPAPI.dll
0x000007fefdd90000 - 0x000007fefddc6000 C:\Windows\system32\CFGMGR32.dll
0x000007fefdd30000 - 0x000007fefdd4a000 C:\Windows\system32\DEVOBJ.dll
0x000007fefbec0000 - 0x000007fefbfec000 C:\Windows\system32\propsys.dll
0x00000000061a0000 - 0x00000000061a6000 D:\Recs7Engines\Java\jre1.6.0_07\bin\rmi.dll
0x00000000075c0000 - 0x00000000075d7000 D:\Recs7Engines\Java\jre1.6.0_07\bin\net.dll
0x000007fefe580000 - 0x000007fefe5cd000 C:\Windows\system32\WS2_32.dll
0x000007fefe570000 - 0x000007fefe578000 C:\Windows\system32\NSI.dll
0x000007fefd410000 - 0x000007fefd465000 C:\Windows\system32\mswsock.dll
0x000007fefd400000 - 0x000007fefd407000 C:\Windows\System32\wship6.dll
0x000007fefc570000 - 0x000007fefc585000 C:\Windows\system32\NLAapi.dll
0x000007fefafb0000 - 0x000007fefafc5000 C:\Windows\system32\napinsp.dll
0x000007fefd290000 - 0x000007fefd2eb000 C:\Windows\system32\DNSAPI.dll
0x000007fefafe0000 - 0x000007fefafeb000 C:\Windows\System32\winrnr.dll
0x000007fefcdf0000 - 0x000007fefcdf7000 C:\Windows\System32\wshtcpip.dll
0x000007fefbbe0000 - 0x000007fefbc07000 C:\Windows\system32\IPHLPAPI.DLL
0x000007fefbbd0000 - 0x000007fefbbdb000 C:\Windows\system32\WINNSI.DLL
0x000007fefb2c0000 - 0x000007fefb2c8000 C:\Windows\system32\rasadhlp.dll
0x000007fefb610000 - 0x000007fefb663000 C:\Windows\System32\fwpuclnt.dll
0x000007fefd470000 - 0x000007fefd487000 C:\Windows\system32\CRYPTSP.dll
0x000007fefd170000 - 0x000007fefd1b7000 C:\Windows\system32\rsaenh.dll
0x000007fefcf10000 - 0x000007fefcf2e000 C:\Windows\system32\USERENV.dll
0x000007fefdbe0000 - 0x000007fefdbef000 C:\Windows\system32\profapi.dll
0x00000000075e0000 - 0x00000000075ea000 D:\Recs7Engines\Java\jre1.6.0_07\bin\management.dll
0x0000000070310000 - 0x0000000070329000 D:\Recs7Engines\ApplicationEngines\Engines\Run\sqljdbc_auth.dll
0x000007fefd9f0000 - 0x000007fefd9fb000 C:\Windows\system32\secur32.dll
0x000007fefda40000 - 0x000007fefda65000 C:\Windows\system32\SSPICLI.DLL
0x000007fefd070000 - 0x000007fefd07a000 C:\Windows\system32\credssp.dll
0x000007fefd490000 - 0x000007fefd544000 C:\Windows\system32\kerberos.DLL
0x000007fefd6f0000 - 0x000007fefd704000 C:\Windows\system32\cryptdll.dll
0x000007fefdc80000 - 0x000007fefdc8f000 C:\Windows\system32\MSASN1.dll
0x000007fefd3a0000 - 0x000007fefd3f1000 C:\Windows\system32\msv1_0.DLL
0x000007fefb270000 - 0x000007fefb297000 C:\Windows\system32\ntdsapi.dll

VM Arguments:
jvm_args: -Djava.security.policy=D:\RECS7Engines\ApplicationEngines\Engines\Run\server.policy -Djava.rmi.server.codebase=file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/AAADateRoll.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/activation.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/Amalgamation.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/apache-mime4j-0.6.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/Archival.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/asm-3.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/axis.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/bcel-5.1-jnbridge.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/cglib-2.2.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecs.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecsTest.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetSchedulerPlugIn.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudio.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudioTest.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mq.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqbind.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqjms.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-codec-1.3.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-discovery-0.2.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-io-2.4.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-lang3-3.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-logging-1.1.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/connector.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/fscontext.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpclient-4.0.1.jar file:///D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpcore-4.0.1.jar file:///D:\RECS7Engines\ApplicationEng-Xmx6144M -Dclasspath=D:\RECS7Engines\ApplicationEngines\Engines\Jars/AAADateRoll.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/activation.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/Amalgamation.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/apache-mime4j-0.6.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/Archival.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/asm-3.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/axis.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/bcel-5.1-jnbridge.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/cglib-2.2.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecs.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetRecsTest.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetSchedulerPlugIn.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudio.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/CityNetStudioTest.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mq.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqbind.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/com.ibm.mqjms.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-codec-1.3.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-discovery-0.2.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-io-2.4.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-lang3-3.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/commons-logging-1.1.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/connector.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/fscontext.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpclient-4.0.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpcore-4.0.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/httpmime-4.0.1.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/ifxjdbc.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/jaxb-api.jar;D:\RECS7Engines\ApplicationEngines\Engines\Jars/jaxen-1.1.1.jar;D:\RECS7E-Xrs
java_command: sun.rmi.server.ActivationGroupInit
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Symantec\Critical System Protection\Agent\IPS\bin;C:\IBM\ITM\bin;C:\IBM\ITM\TMAITM6;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia AUX;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia AUX\Support binaries;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia Broker;C:\Program Files (x86)\SSH Communications Security\SSH Tectia\SSH Tectia Client;C:\IBM\ITM\InstallITM
USERNAME=precseng
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 47 Stepping 2, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows NT 6.1 Build 7601 Service Pack 1

CPU:total 6 (1 cores per cpu, 1 threads per core) family 6 model 15 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 33553912k(17588460k free), swap 42255028k(5492k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (10.0-b23) for windows-amd64 JRE (1.6.0_07-b06), built on Jun 10 2008 01:06:11 by "java_re" with MS VC++ 8.0

time: Tue Aug 05 10:37:16 2014
elapsed time: 226 seconds


REPRODUCIBILITY :
This bug can be reproduced often.

CUSTOMER SUBMITTED WORKAROUND :
abort the job and re run it.

SUPPORT :
YES