FULL PRODUCT VERSION :
JRE (1.6.0_07-b06)

FULL OS VERSION :
Windows Server 2003 family Build 3790 Service Pack 2


EXTRA RELEVANT SYSTEM CONFIGURATION :
CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 13 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k

A DESCRIPTION OF THE PROBLEM :
JVM crash is causing my production system go down frequently.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# An unexpected error has been detected by Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d855ce5, pid=1456, tid=5584
#
# Java VM: Java HotSpot(TM) Client VM (10.0-b23 mixed mode windows-x86)
# Problematic frame:
# V [jvm.dll+0x95ce5]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x0a780800): VMThread [stack: 0x0a810000,0x0a860000] [id=5584]

siginfo: ExceptionCode=0xc0000005, reading address 0x81efdd27

Registers:
EAX=0x6d99b860, EBX=0x00299050, ECX=0x0a85fc14, EDX=0x6d99b860
ESP=0x0a85fa74, EBP=0x00138000, ESI=0x81efdd27, EDI=0x0a85fc14
EIP=0x6d855ce5, EFLAGS=0x00010286

Top of Stack: (sp=0x0a85fa74)
0x0a85fa74: 81efdd27 6d91d75f 81efdd27 0b043400
0x0a85fa84: 0a85fc14 6d964ea0 0a85fc14 0a85fc14
0x0a85fa94: 0b043400 00138000 00299050 0d62fc98
0x0a85faa4: 00610247 00610188 00000000 0d62fcbc
0x0a85fab4: 0d62fca0 0b5fe6d4 0b5fe6d0 0b5fe6cc
0x0a85fac4: 0b5fe6c8 6d9b3174 0d62fc58 0b5fe6bc
0x0a85fad4: 0b5fe6b8 0a85fc40 0a85fc40 0a77f338
0x0a85fae4: 6d86cf2f 6d86cf3f 0029aea8 0029b658

Instructions: (pc=0x6d855ce5)
0x6d855cd5: c2 04 00 cc cc cc cc cc cc cc cc 56 8b 74 24 08
0x6d855ce5: 8b 06 85 c0 57 8b f9 74 55 3b 47 1c 73 50 8b 08


Stack: [0x0a810000,0x0a860000], sp=0x0a85fa74, free space=318k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x95ce5]

[error occurred during error reporting (printing native stack), id 0xc0000005]

VM_Operation (0x0b5fe4f0): GenCollectForAllocation, mode: safepoint, requested by thread 0x0b039400


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
  0x0bbddc00 JavaThread "RMI TCP Connection(130712)-10.53.44.244" daemon [_thread_in_native, id=3108, stack(0x0d680000,0x0d6d0000)]
  0x0bbac400 JavaThread "RMI TCP Connection(130708)-10.53.44.243" daemon [_thread_in_native, id=3552, stack(0x0d630000,0x0d680000)]
  0x0bc0e400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=6124, stack(0x0c3d0000,0x0c420000)]
  0x0b039400 JavaThread "RMI TCP Connection(130710)-10.53.44.242" daemon [_thread_blocked, id=4004, stack(0x0b5b0000,0x0b600000)]
  0x0bbff400 JavaThread "RMI TCP Connection(130711)-10.53.44.243" daemon [_thread_in_native, id=5488, stack(0x0d5e0000,0x0d630000)]
  0x0b043400 JavaThread "RMI TCP Connection(130705)-10.53.44.242" daemon [_thread_in_native, id=4684, stack(0x0d4c0000,0x0d510000)]
  0x0bbf6c00 JavaThread "AD Thread Pool-Global1091" daemon [_thread_blocked, id=6100, stack(0x0b2f0000,0x0b340000)]
  0x0bc1a400 JavaThread "RMI TCP Connection(130707)-10.53.44.244" daemon [_thread_in_native, id=5480, stack(0x0b110000,0x0b160000)]
  0x0bb98800 JavaThread "RMI TCP Connection(130709)-10.53.44.243" daemon [_thread_in_native, id=5656, stack(0x0c690000,0x0c6e0000)]
  0x0bbc5400 JavaThread "AD Thread Pool-Global1088" daemon [_thread_blocked, id=5160, stack(0x0b160000,0x0b1b0000)]
  0x0a72b400 JavaThread "AD Thread Pool-CGG0" daemon [_thread_blocked, id=2596, stack(0x0b600000,0x0b650000)]
  0x0aa3a400 JavaThread "AD Thread Pool-Coalescable_CS_Scheduler0" daemon [_thread_blocked, id=5152, stack(0x0b420000,0x0b470000)]
  0x00296000 JavaThread "DestroyJavaVM" [_thread_blocked, id=3956, stack(0x005c0000,0x00610000)]
  0x0afe3800 JavaThread "RMI Reaper" [_thread_blocked, id=5296, stack(0x0b560000,0x0b5b0000)]
  0x0ae4c800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=4636, stack(0x0b510000,0x0b560000)]
  0x0bab8000 JavaThread "AD Thread-Hotspot Processor0" daemon [_thread_blocked, id=3896, stack(0x12100000,0x12150000)]
  0x0a9e8800 JavaThread "GC Daemon" daemon [_thread_blocked, id=5988, stack(0x10db0000,0x10e00000)]
  0x0a80bc00 JavaThread "RMI RenewClean-[10.53.20.212:1943]" daemon [_thread_blocked, id=4304, stack(0x10d60000,0x10db0000)]
  0x0a9f4c00 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=2892, stack(0x10d10000,0x10d60000)]
  0x0ba64800 JavaThread "AD Thread-Top Backends Processor0" daemon [_thread_blocked, id=628, stack(0x0b6a0000,0x0b6f0000)]
  0x0aadcc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=5304, stack(0x0b060000,0x0b0b0000)]
  0x0ae2a400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=5912, stack(0x0b470000,0x0b4c0000)]
  0x0ad57400 JavaThread "AD Thread-Metric Reporter0" daemon [_thread_blocked, id=5592, stack(0x0b390000,0x0b3e0000)]
  0x0ad92800 JavaThread "AD Thread-Config Poller" daemon [_thread_blocked, id=4168, stack(0x0b340000,0x0b390000)]
  0x0ad26800 JavaThread "Thread-1" daemon [_thread_blocked, id=5612, stack(0x0b0c0000,0x0b110000)]
  0x0a798000 JavaThread "Attach Listener" daemon [_thread_blocked, id=4028, stack(0x0a950000,0x0a9a0000)]
  0x0a797000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3384, stack(0x0a900000,0x0a950000)]
  0x0a789400 JavaThread "Finalizer" daemon [_thread_blocked, id=1400, stack(0x0a8b0000,0x0a900000)]
  0x0a785000 JavaThread "Reference Handler" daemon [_thread_blocked, id=4320, stack(0x0a860000,0x0a8b0000)]

Other Threads:
=>0x0a780800 VMThread [stack: 0x0a810000,0x0a860000] [id=5584]
  0x0ae27000 WatcherThread [stack: 0x0b4c0000,0x0b510000] [id=4088]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x002956a0/0x000006f4] Threads_lock - owner thread: 0x0a780800
[0x00295840/0x000006b4] Heap_lock - owner thread: 0x0b039400

Heap
 def new generation total 2496K, used 2306K [0x02690000, 0x02940000, 0x02b70000)
  eden space 2240K, 99% used [0x02690000, 0x028bfff8, 0x028c0000)
  from space 256K, 26% used [0x028c0000, 0x028d0ba8, 0x02900000)
  to space 256K, 0% used [0x02900000, 0x029006d8, 0x02940000)
 tenured generation total 32632K, used 15417K [0x02b70000, 0x04b4e000, 0x06690000)
   the space 32632K, 47% used [0x02b70000, 0x03a7e5f8, 0x03a7e600, 0x04b4e000)
 compacting perm gen total 28672K, used 28472K [0x06690000, 0x08290000, 0x0a690000)
   the space 28672K, 99% used [0x06690000, 0x0825e3c0, 0x0825e400, 0x08290000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00423000 C:\Program Files\Java\jre1.6.0_07\bin\java.exe
0x7c800000 - 0x7c8c4000 C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f44000 C:\WINDOWS\system32\kernel32.dll
0x7d1e0000 - 0x7d27c000 C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cf1000 C:\WINDOWS\system32\RPCRT4.dll
0x76f50000 - 0x76f63000 C:\WINDOWS\system32\Secur32.dll
0x7c340000 - 0x7c396000 C:\Program Files\Java\jre1.6.0_07\bin\msvcr71.dll
0x6d7c0000 - 0x6da10000 C:\Program Files\Java\jre1.6.0_07\bin\client\jvm.dll
0x77380000 - 0x77411000 C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c49000 C:\WINDOWS\system32\GDI32.dll
0x76aa0000 - 0x76acd000 C:\WINDOWS\system32\WINMM.dll
0x6d270000 - 0x6d278000 C:\Program Files\Java\jre1.6.0_07\bin\hpi.dll
0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL
0x6d2b0000 - 0x6d2ca000 C:\Program Files\Java\jre1.6.0_07\bin\instrument.dll
0x6d770000 - 0x6d77c000 C:\Program Files\Java\jre1.6.0_07\bin\verify.dll
0x6d310000 - 0x6d32f000 C:\Program Files\Java\jre1.6.0_07\bin\java.dll
0x6d7b0000 - 0x6d7bf000 C:\Program Files\Java\jre1.6.0_07\bin\zip.dll
0x68000000 - 0x68035000 C:\WINDOWS\system32\rsaenh.dll
0x77ba0000 - 0x77bfa000 C:\WINDOWS\system32\msvcrt.dll
0x76920000 - 0x769e2000 C:\WINDOWS\system32\USERENV.dll
0x71c40000 - 0x71c97000 C:\WINDOWS\system32\netapi32.dll
0x6d570000 - 0x6d583000 C:\Program Files\Java\jre1.6.0_07\bin\net.dll
0x71c00000 - 0x71c17000 C:\WINDOWS\system32\WS2_32.dll
0x71bf0000 - 0x71bf8000 C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b61000 C:\WINDOWS\System32\mswsock.dll
0x76ed0000 - 0x76efa000 C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000 C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000 C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000 C:\WINDOWS\system32\rasadhlp.dll
0x6d4c0000 - 0x6d4c9000 C:\Program Files\Java\jre1.6.0_07\bin\management.dll
0x76cf0000 - 0x76d0a000 C:\WINDOWS\system32\iphlpapi.dll
0x76cd0000 - 0x76ce9000 C:\WINDOWS\system32\MPRAPI.dll
0x76df0000 - 0x76e24000 C:\WINDOWS\system32\ACTIVEDS.dll
0x76dc0000 - 0x76de8000 C:\WINDOWS\system32\adsldpc.dll
0x76b80000 - 0x76bae000 C:\WINDOWS\system32\credui.dll
0x7c8d0000 - 0x7d0cf000 C:\WINDOWS\system32\SHELL32.dll
0x7d180000 - 0x7d1d2000 C:\WINDOWS\system32\SHLWAPI.dll
0x76a80000 - 0x76a92000 C:\WINDOWS\system32\ATL.DLL
0x77670000 - 0x777aa000 C:\WINDOWS\system32\ole32.dll
0x7d0e0000 - 0x7d16c000 C:\WINDOWS\system32\OLEAUT32.dll
0x76e30000 - 0x76e3c000 C:\WINDOWS\system32\rtutils.dll
0x7e020000 - 0x7e02f000 C:\WINDOWS\system32\SAMLIB.dll
0x770e0000 - 0x771e8000 C:\WINDOWS\system32\SETUPAPI.dll
0x77420000 - 0x77523000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.5190_x-ww_319264BE\comctl32.dll
0x5f270000 - 0x5f2ca000 C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000 C:\WINDOWS\System32\wshtcpip.dll
0x10000000 - 0x10050000 D:\tril7v0\bin\trilRouterLibrary.dll
0x0b6f0000 - 0x0b8d3000 D:\tril7v0\bin\trilJMatcherLibrary.dll
0x0b8e0000 - 0x0b93f000 D:\tril7v0\bin\trilConverterLibrary.dll
0x0b940000 - 0x0b9be000 D:\tril7v0\bin\trilParserLibrary.dll
0x0c2d0000 - 0x0c30d000 D:\tril7v0\bin\trilWindowKeyLibrary.dll
0x0ffa0000 - 0x0ffe3000 D:\tril7v0\bin\trilGeocoderLibrary.dll
0x104b0000 - 0x104da000 D:\tril7v0\bin\trilCAGeocoderLibrary.dll
0x6d710000 - 0x6d716000 C:\Program Files\Java\jre1.6.0_07\bin\rmi.dll
0x104e0000 - 0x10507000 D:\tril7v0\bin\trilTGGeocoderLibrary.dll

VM Arguments:
jvm_args: -javaagent:D:\AppDynamics-4.2\AppAgent\javaagent.jar -Dappdynamics.agent.tierName=TrlWrap -Dappdynamics.agent.nodeName=TrlWrap -Dappdynamics.agent.applicationName=Trillium -Dappdynamics.agent.uniqueHostId=USFLTRA0 -Djava.security.policy=policy.txt
java_command: tw.app.lead.trillium.engine.TrilEngine usfltra0
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\Java\jre1.6.0_07
PATH=D:\tril7v0\bin;D:\Program Files\TrilliumWrapper\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\tril7v0\bin;D:\Program Files\NetIQ\Common\Bin;D:\Program Files\NetIQ\AppManager\bin;C:\Program Files\Windows Imaging\
USERNAME=svc-trillium
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 45 Stepping 7, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 13 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Client VM (10.0-b23) for windows-x86 JRE (1.6.0_07-b06), built on Jun 10 2008 01:14:11 by "java_re" with MS VC++ 7.1

time: Wed Apr 20 01:02:50 2016
elapsed time: 1570388 seconds


REPRODUCIBILITY :
This bug can be reproduced often.