-
Bug
-
Resolution: Incomplete
-
P4
-
None
-
6
-
sparc_64
-
solaris_10
-
Verified
FULL PRODUCT VERSION :
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Solaris 10 8/07 s10s_u4wos_12b SPARC
EXTRA RELEVANT SYSTEM CONFIGURATION :
System Configuration: Sun Microsystems sun4v Sun Fire(TM) T1000
Memory size: 16256 Megabytes
JVM OPTS
-d64
-XX:PermSize=384m
-XX:MaxPermSize=384m
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xmx5120m
A DESCRIPTION OF THE PROBLEM :
We have got web applications hosted on Tomcat 6 which runs on jdk 1.6.0_06
We are using Oracle UCP connection pool for DB connections along with ojdbc6
The application were running all good all of a sudden we got reports when we checked found the java process got crashed and killed so had to start up again
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
We have not got exact replication steps for the crash, the applications were being used all good until the point we do not see any out of memory error but from the crash dump able to see we have got threads blocked hence not sure if this was due to GC thread which caused stop the world or is some thing else that trigerred it.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Java process should not be crashed / killed
ACTUAL -
Java process was crashed
ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0xffffffff7e3b088c, pid=28352, tid=30
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode solaris-sparc)
# Problematic frame:
# V [libjvm.so+0x3b088c]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0000000100616800): GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=30]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN), si_addr=0x000000000000001d
Registers:
O0=0x0000000000000003 O1=0x0000000000002a54 O2=0x0000000000000002 O3=0x0000000000000002
O4=0x0000000000002a52 O5=0x000000002a5253c7 O6=0xffffffff6defef01 O7=0x00000000000152a0
G1=0x0000000000002a54 G2=0x0000000000042400 G3=0x0000000000000018 G4=0xffffffffffffc018
G5=0x000000000000000d G6=0x0000000000000000 G7=0xffffffff7f40ea00 Y=0x0000000000000000
PC=0xffffffff7e3b088c nPC=0xffffffff7e3b0890
Top of Stack: (sp=0xffffffff6deff700)
0xffffffff6deff700: ffffffffffffffff ffffffff6deff818
0xffffffff6deff710: 0000000000003ffe 0000000000003fff
0xffffffff6deff720: 0000000100134ce0 0000000000002a52
0xffffffff6deff730: 0000000000002a54 0000000000042440
0xffffffff6deff740: ffffffff6deffa10 00000001006185f0
0xffffffff6deff750: 0000000000002a55 fffffffecdacb950
0xffffffff6deff760: 000000002a5253c7 ffffffff7e99e000
0xffffffff6deff770: ffffffff6deff091 ffffffff7e3350e8
0xffffffff6deff780: 0000000000000000 0000000000000000
0xffffffff6deff790: 0000000000000000 0000000000000000
0xffffffff6deff7a0: 0000000000000000 0000000000000000
0xffffffff6deff7b0: 0000000100611f50 ffffffff01604690
0xffffffff6deff7c0: ffffffff6daff768 ffffffff6deffa10
0xffffffff6deff7d0: 2a5253c600000000 0000000000000000
0xffffffff6deff7e0: ffffffff7e3b0548 ffffffff2a5253c6
0xffffffff6deff7f0: 0000000100611d50 000000002743403f
Instructions: (pc=0xffffffff7e3b088c)
0xffffffff7e3b087c: 80 90 00 08 22 48 00 22 ea 5e 20 48 ca 5e e0 08
0xffffffff7e3b088c: c8 59 60 10 92 10 00 1b 94 10 00 11 c6 59 22 60
Stack: [0x0000000000000000,0x0000000000000000], sp=0xffffffff6deff700, free space=-2393091k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x3b088c]
V [libjvm.so+0x3350f0]
V [libjvm.so+0x3a639c]
V [libjvm.so+0x3a5de0]
V [libjvm.so+0x73d968]
V [libjvm.so+0x634324]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000104b45000 JavaThread "Thread-4518" daemon [_thread_blocked, id=8514, stack(0xfffffffde7700000,0xfffffffde7800000)]
0x0000000104724c00 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=8513, stack(0xfffffffde7100000,0xfffffffde7200000)]
0x0000000104b4c800 JavaThread "Thread-4365" daemon [_thread_in_native, id=8291, stack(0xfffffffde7300000,0xfffffffde7400000)]
0x0000000105041c00 JavaThread "http-8080-172" daemon [_thread_in_native, id=8202, stack(0xfffffffde7500000,0xfffffffde7600000)]
0x0000000102da3800 JavaThread "http-8080-171" daemon [_thread_in_native, id=8201, stack(0xfffffffde7b00000,0xfffffffde7c00000)]
0x0000000103376800 JavaThread "http-8080-170" daemon [_thread_in_native, id=8200, stack(0xfffffffde7f00000,0xfffffffde8000000)]
0x00000001017bb800 JavaThread "http-8080-169" daemon [_thread_in_native, id=8199, stack(0xfffffffde7d00000,0xfffffffde7e00000)]
0x0000000104af6800 JavaThread "http-8080-168" daemon [_thread_in_native, id=8198, stack(0xfffffffdea300000,0xfffffffdea400000)]
0x00000001011a0400 JavaThread "http-8080-167" daemon [_thread_in_native, id=8197, stack(0xfffffffde8100000,0xfffffffde8200000)]
0x0000000104958c00 JavaThread "http-8080-166" daemon [_thread_in_native, id=8196, stack(0xfffffffde8300000,0xfffffffde8400000)]
0x0000000102f87c00 JavaThread "http-8443-3" daemon [_thread_blocked, id=3556, stack(0xfffffffde8f00000,0xfffffffde9000000)]
0x0000000104fa4000 JavaThread "http-8443-2" daemon [_thread_blocked, id=3546, stack(0xfffffffde8500000,0xfffffffde8600000)]
0x0000000105a51800 JavaThread "http-8443-1" daemon [_thread_blocked, id=3545, stack(0xfffffffde9300000,0xfffffffde9400000)]
0x000000010587bc00 JavaThread "http-8080-165" daemon [_thread_in_native, id=3532, stack(0xfffffffde8b00000,0xfffffffde8c00000)]
0x0000000104b04c00 JavaThread "http-8080-164" daemon [_thread_in_native, id=3531, stack(0xfffffffde8900000,0xfffffffde8a00000)]
0x0000000104dc8400 JavaThread "http-8080-163" daemon [_thread_in_native, id=2664, stack(0xfffffffde8700000,0xfffffffde8800000)]
0x0000000105a56000 JavaThread "http-8080-162" daemon [_thread_in_native, id=2555, stack(0xfffffffde8d00000,0xfffffffde8e00000)]
0x0000000105a5a000 JavaThread "http-8080-161" daemon [_thread_in_native, id=2554, stack(0xfffffffde9100000,0xfffffffde9200000)]
0x00000001025c6c00 JavaThread "http-8080-160" daemon [_thread_in_native, id=2056, stack(0xfffffffde9500000,0xfffffffde9600000)]
0x00000001025c6400 JavaThread "http-8080-159" daemon [_thread_in_native, id=2055, stack(0xfffffffde9700000,0xfffffffde9800000)]
0x00000001042f7400 JavaThread "http-8080-158" daemon [_thread_in_native, id=2054, stack(0xfffffffde9900000,0xfffffffde9a00000)]
0x00000001042f7000 JavaThread "http-8080-157" daemon [_thread_in_native, id=2053, stack(0xfffffffde9b00000,0xfffffffde9c00000)]
0x0000000103bf1000 JavaThread "http-8080-156" daemon [_thread_in_native, id=2033, stack(0xfffffffde9d00000,0xfffffffde9e00000)]
0x0000000102b95c00 JavaThread "http-8080-155" daemon [_thread_blocked, id=2032, stack(0xfffffffdeea00000,0xfffffffdeeb00000)]
0x00000001046a4800 JavaThread "http-8080-154" daemon [_thread_in_native, id=2002, stack(0xfffffffde9f00000,0xfffffffdea000000)]
0x0000000102b31800 JavaThread "http-8080-153" daemon [_thread_in_native, id=1906, stack(0xfffffffdea100000,0xfffffffdea200000)]
0x0000000102325400 JavaThread "http-8080-152" daemon [_thread_in_native, id=1864, stack(0xfffffffdea500000,0xfffffffdea600000)]
0x0000000102324c00 JavaThread "http-8080-151" daemon [_thread_in_native, id=1863, stack(0xfffffffdea700000,0xfffffffdea800000)]
0x0000000102a5e400 JavaThread "http-8080-150" daemon [_thread_in_native, id=1862, stack(0xfffffffdeab00000,0xfffffffdeac00000)]
0x0000000102a5dc00 JavaThread "http-8080-149" daemon [_thread_in_native, id=1861, stack(0xfffffffdea900000,0xfffffffdeaa00000)]
0x0000000101694c00 JavaThread "http-8080-148" daemon [_thread_in_native, id=1860, stack(0xfffffffdead00000,0xfffffffdeae00000)]
0x0000000103b2d000 JavaThread "http-8080-147" daemon [_thread_in_native, id=1859, stack(0xfffffffdeaf00000,0xfffffffdeb000000)]
0x000000010392b800 JavaThread "http-8080-146" daemon [_thread_in_native, id=1792, stack(0xfffffffdeb100000,0xfffffffdeb200000)]
0x0000000103303c00 JavaThread "http-8080-145" daemon [_thread_in_native, id=1791, stack(0xfffffffdeb300000,0xfffffffdeb400000)]
0x0000000102461000 JavaThread "http-8080-144" daemon [_thread_blocked, id=1790, stack(0xfffffffdeb500000,0xfffffffdeb600000)]
0x0000000102d08000 JavaThread "http-8080-143" daemon [_thread_in_native, id=1789, stack(0xfffffffdeb700000,0xfffffffdeb800000)]
0x0000000102eaa800 JavaThread "http-8080-142" daemon [_thread_in_native, id=1788, stack(0xfffffffdecd00000,0xfffffffdece00000)]
0x0000000102f51400 JavaThread "http-8080-141" daemon [_thread_in_Java, id=1584, stack(0xfffffffdeb900000,0xfffffffdeba00000)]
0x0000000102f51000 JavaThread "http-8080-140" daemon [_thread_blocked, id=1583, stack(0xfffffffdebb00000,0xfffffffdebc00000)]
0x00000001049c1800 JavaThread "http-8080-139" daemon [_thread_in_native, id=1582, stack(0xfffffffdebd00000,0xfffffffdebe00000)]
0x0000000102ea8400 JavaThread "http-8080-138" daemon [_thread_in_native, id=1581, stack(0xfffffffdebf00000,0xfffffffdec000000)]
0x00000001025fa400 JavaThread "http-8080-137" daemon [_thread_in_native, id=1520, stack(0xfffffffdec100000,0xfffffffdec200000)]
0x00000001025fc400 JavaThread "http-8080-136" daemon [_thread_in_native, id=1519, stack(0xfffffffdec300000,0xfffffffdec400000)]
0x00000001025ff800 JavaThread "http-8080-135" daemon [_thread_blocked, id=1518, stack(0xfffffffdec500000,0xfffffffdec600000)]
0x00000001025f9400 JavaThread "http-8080-134" daemon [_thread_blocked, id=1517, stack(0xfffffffdec700000,0xfffffffdec800000)]
0x00000001025a1400 JavaThread "http-8080-133" daemon [_thread_in_native, id=1516, stack(0xfffffffdec900000,0xfffffffdeca00000)]
0x0000000104b81000 JavaThread "http-8080-132" daemon [_thread_in_native, id=1515, stack(0xfffffffdecb00000,0xfffffffdecc00000)]
0x0000000103373c00 JavaThread "http-8080-131" daemon [_thread_in_native, id=1441, stack(0xfffffffdecf00000,0xfffffffded000000)]
0x0000000102186c00 JavaThread "http-8080-130" daemon [_thread_in_native, id=1440, stack(0xfffffffded100000,0xfffffffded200000)]
0x0000000101a3b400 JavaThread "http-8080-129" daemon [_thread_in_native, id=1438, stack(0xfffffffded300000,0xfffffffded400000)]
0x0000000103d9cc00 JavaThread "http-8080-128" daemon [_thread_in_native, id=1437, stack(0xfffffffded900000,0xfffffffdeda00000)]
0x0000000102185000 JavaThread "http-8080-127" daemon [_thread_in_native, id=1435, stack(0xfffffffded500000,0xfffffffded600000)]
0x0000000103373000 JavaThread "http-8080-126" daemon [_thread_in_native, id=1434, stack(0xfffffffded700000,0xfffffffded800000)]
0x0000000103c58400 JavaThread "http-8080-125" daemon [_thread_in_native, id=1432, stack(0xfffffffdedb00000,0xfffffffdedc00000)]
0x00000001011b4c00 JavaThread "http-8080-124" daemon [_thread_in_native, id=1431, stack(0xfffffffdedd00000,0xfffffffdede00000)]
0x0000000102298800 JavaThread "http-8080-123" daemon [_thread_in_native, id=1430, stack(0xfffffffdee600000,0xfffffffdee700000)]
0x00000001024a7400 JavaThread "http-8080-122" daemon [_thread_in_native, id=1415, stack(0xfffffffdee400000,0xfffffffdee500000)]
0x000000010229c000 JavaThread "http-8080-121" daemon [_thread_in_native, id=1414, stack(0xfffffffdee800000,0xfffffffdee900000)]
0x0000000102f21000 JavaThread "http-8080-120" daemon [_thread_in_native, id=1251, stack(0xfffffffdee200000,0xfffffffdee300000)]
0x0000000103aabc00 JavaThread "http-8080-119" daemon [_thread_in_native, id=934, stack(0xfffffffdf6800000,0xfffffffdf6900000)]
0x0000000101723400 JavaThread "http-8080-118" daemon [_thread_in_Java, id=933, stack(0xfffffffdeec00000,0xfffffffdeed00000)]
0x0000000101e3f800 JavaThread "http-8080-117" daemon [_thread_in_native, id=932, stack(0xfffffffdeee00000,0xfffffffdeef00000)]
0x0000000101498c00 JavaThread "http-8080-116" daemon [_thread_in_native, id=931, stack(0xfffffffdef200000,0xfffffffdef300000)]
0x00000001037a3000 JavaThread "http-8080-115" daemon [_thread_in_native, id=930, stack(0xfffffffdef000000,0xfffffffdef100000)]
0x00000001023c7000 JavaThread "http-8080-114" daemon [_thread_in_native, id=928, stack(0xfffffffdef400000,0xfffffffdef500000)]
0x0000000103a72000 JavaThread "http-8080-113" daemon [_thread_in_native, id=927, stack(0xfffffffdef600000,0xfffffffdef700000)]
0x0000000103a71400 JavaThread "http-8080-112" daemon [_thread_in_native, id=926, stack(0xfffffffdef800000,0xfffffffdef900000)]
0x0000000103a70800 JavaThread "http-8080-111" daemon [_thread_in_native, id=925, stack(0xfffffffdefa00000,0xfffffffdefb00000)]
0x0000000103a6fc00 JavaThread "http-8080-110" daemon [_thread_in_native, id=924, stack(0xfffffffdefc00000,0xfffffffdefd00000)]
0x000000010213fc00 JavaThread "http-8080-109" daemon [_thread_in_native, id=923, stack(0xfffffffdefe00000,0xfffffffdeff00000)]
0x000000010213f800 JavaThread "http-8080-108" daemon [_thread_in_native, id=922, stack(0xfffffffdf0000000,0xfffffffdf0100000)]
0x000000010379f400 JavaThread "http-8080-107" daemon [_thread_in_native, id=921, stack(0xfffffffdf0200000,0xfffffffdf0300000)]
0x000000010379ec00 JavaThread "http-8080-106" daemon [_thread_in_native, id=920, stack(0xfffffffdf0400000,0xfffffffdf0500000)]
0x000000010175e800 JavaThread "http-8080-105" daemon [_thread_in_native, id=919, stack(0xfffffffdf0600000,0xfffffffdf0700000)]
0x000000010175dc00 JavaThread "http-8080-104" daemon [_thread_in_native, id=918, stack(0xfffffffdf0800000,0xfffffffdf0900000)]
0x00000001013d9800 JavaThread "http-8080-103" daemon [_thread_in_native, id=917, stack(0xfffffffdf0a00000,0xfffffffdf0b00000)]
0x00000001013d8c00 JavaThread "http-8080-102" daemon [_thread_in_native, id=916, stack(0xfffffffdf0c00000,0xfffffffdf0d00000)]
0x00000001013d7c00 JavaThread "http-8080-101" daemon [_thread_in_native, id=915, stack(0xfffffffdf0e00000,0xfffffffdf0f00000)]
0x00000001013d6c00 JavaThread "http-8080-100" daemon [_thread_in_native, id=914, stack(0xfffffffdf1000000,0xfffffffdf1100000)]
0x00000001013d6000 JavaThread "http-8080-99" daemon [_thread_in_native, id=913, stack(0xfffffffdf1200000,0xfffffffdf1300000)]
0x0000000101903400 JavaThread "http-8080-98" daemon [_thread_in_native, id=912, stack(0xfffffffdf1400000,0xfffffffdf1500000)]
0x0000000101902400 JavaThread "http-8080-97" daemon [_thread_in_native, id=911, stack(0xfffffffdf1600000,0xfffffffdf1700000)]
0x0000000101901400 JavaThread "http-8080-96" daemon [_thread_blocked, id=910, stack(0xfffffffdf1800000,0xfffffffdf1900000)]
0x0000000101900c00 JavaThread "http-8080-95" daemon [_thread_in_native, id=909, stack(0xfffffffdf1a00000,0xfffffffdf1b00000)]
0x00000001037a8800 JavaThread "http-8080-94" daemon [_thread_blocked, id=908, stack(0xfffffffdf1c00000,0xfffffffdf1d00000)]
0x00000001037a7800 JavaThread "http-8080-93" daemon [_thread_in_native, id=907, stack(0xfffffffdf1e00000,0xfffffffdf1f00000)]
0x0000000103147400 JavaThread "http-8080-92" daemon [_thread_in_native, id=906, stack(0xfffffffdf2000000,0xfffffffdf2100000)]
0x0000000103146c00 JavaThread "http-8080-91" daemon [_thread_blocked, id=905, stack(0xfffffffdf2600000,0xfffffffdf2700000)]
0x0000000103146400 JavaThread "http-8080-90" daemon [_thread_in_native, id=904, stack(0xfffffffdf2800000,0xfffffffdf2900000)]
0x00000001037a6c00 JavaThread "http-8080-89" daemon [_thread_in_native, id=903, stack(0xfffffffdf2400000,0xfffffffdf2500000)]
0x000000010135c800 JavaThread "http-8080-88" daemon [_thread_in_native, id=617, stack(0xfffffffdf2a00000,0xfffffffdf2b00000)]
0x0000000102850000 JavaThread "http-8080-87" daemon [_thread_in_native, id=616, stack(0xfffffffdf2c00000,0xfffffffdf2d00000)]
0x0000000101d42000 JavaThread "http-8080-86" daemon [_thread_blocked, id=615, stack(0xfffffffdf2e00000,0xfffffffdf2f00000)]
0x0000000102be4000 JavaThread "http-8080-85" daemon [_thread_in_native, id=614, stack(0xfffffffdf3000000,0xfffffffdf3100000)]
0x0000000102d5a000 JavaThread "http-8080-84" daemon [_thread_in_native, id=613, stack(0xfffffffdf3200000,0xfffffffdf3300000)]
0x000000010247cc00 JavaThread "http-8080-83" daemon [_thread_in_native, id=612, stack(0xfffffffdf3400000,0xfffffffdf3500000)]
0x00000001028dfc00 JavaThread "http-8080-82" daemon [_thread_in_native, id=611, stack(0xfffffffdf3600000,0xfffffffdf3700000)]
0x000000010247e800 JavaThread "http-8080-81" daemon [_thread_in_native, id=610, stack(0xfffffffdf3800000,0xfffffffdf3900000)]
0x0000000102be3800 JavaThread "http-8080-80" daemon [_thread_in_native, id=609, stack(0xfffffffdf3a00000,0xfffffffdf3b00000)]
0x00000001017f9400 JavaThread "http-8080-79" daemon [_thread_in_native, id=608, stack(0xfffffffdf3c00000,0xfffffffdf3d00000)]
0x00000001014b6400 JavaThread "http-8080-78" daemon [_thread_in_native, id=607, stack(0xfffffffdf3e00000,0xfffffffdf3f00000)]
0x0000000102e10c00 JavaThread "http-8080-77" daemon [_thread_in_native, id=606, stack(0xfffffffdf4000000,0xfffffffdf4100000)]
0x0000000101bc0800 JavaThread "http-8080-76" daemon [_thread_in_native, id=605, stack(0xfffffffdf4200000,0xfffffffdf4300000)]
0x000000010267dc00 JavaThread "http-8080-75" daemon [_thread_in_native, id=602, stack(0xfffffffdf7c00000,0xfffffffdf7d00000)]
0x0000000103d7b000 JavaThread "http-8080-74" daemon [_thread_in_native, id=600, stack(0xfffffffdf4400000,0xfffffffdf4500000)]
0x0000000102808c00 JavaThread "http-8080-73" daemon [_thread_blocked, id=599, stack(0xfffffffdf4600000,0xfffffffdf4700000)]
0x0000000102ed0000 JavaThread "http-8080-72" daemon [_thread_in_native, id=598, stack(0xfffffffdf4800000,0xfffffffdf4900000)]
0x0000000101448000 JavaThread "http-8080-71" daemon [_thread_in_native, id=597, stack(0xfffffffdf5600000,0xfffffffdf5700000)]
0x0000000101e62c00 JavaThread "http-8080-70" daemon [_thread_in_Java, id=596, stack(0xfffffffdf4a00000,0xfffffffdf4b00000)]
0x0000000101aee000 JavaThread "http-8080-69" daemon [_thread_in_native, id=595, stack(0xfffffffdf4c00000,0xfffffffdf4d00000)]
0x000000010311a400 JavaThread "http-8080-68" daemon [_thread_in_native, id=584, stack(0xfffffffdf4e00000,0xfffffffdf4f00000)]
0x0000000103119000 JavaThread "http-8080-67" daemon [_thread_blocked, id=583, stack(0xfffffffdf5000000,0xfffffffdf5100000)]
0x0000000103117800 JavaThread "http-8080-66" daemon [_thread_blocked, id=582, stack(0xfffffffdf5200000,0xfffffffdf5300000)]
0x0000000102bffc00 JavaThread "http-8080-65" daemon [_thread_in_native, id=581, stack(0xfffffffdf5400000,0xfffffffdf5500000)]
0x000000010177b400 JavaThread "http-8080-64" daemon [_thread_in_native, id=578, stack(0xfffffffdf5800000,0xfffffffdf5900000)]
0x0000000103d6a800 JavaThread "http-8080-63" daemon [_thread_blocked, id=577, stack(0xfffffffdf5a00000,0xfffffffdf5b00000)]
0x0000000103d6a000 JavaThread "http-8080-62" daemon [_thread_in_native, id=576, stack(0xfffffffdf6600000,0xfffffffdf6700000)]
0x0000000101b3fc00 JavaThread "http-8080-61" daemon [_thread_in_native, id=574, stack(0xfffffffdf5c00000,0xfffffffdf5d00000)]
0x000000010270e400 JavaThread "http-8080-60" daemon [_thread_in_native, id=573, stack(0xfffffffdf5e00000,0xfffffffdf5f00000)]
0x00000001028dcc00 JavaThread "http-8080-59" daemon [_thread_in_native, id=571, stack(0xfffffffdf6000000,0xfffffffdf6100000)]
0x00000001028dc400 JavaThread "http-8080-58" daemon [_thread_in_Java, id=570, stack(0xfffffffdf6200000,0xfffffffdf6300000)]
0x00000001028db000 JavaThread "http-8080-57" daemon [_thread_in_native, id=569, stack(0xfffffffdf6400000,0xfffffffdf6500000)]
0x00000001014a4800 JavaThread "http-8080-56" daemon [_thread_in_native, id=566, stack(0xfffffffdf6a00000,0xfffffffdf6b00000)]
0x0000000102481c00 JavaThread "http-8080-55" daemon [_thread_blocked, id=549, stack(0xfffffffdf6c00000,0xfffffffdf6d00000)]
0x0000000100fc2000 JavaThread "http-8080-54" daemon [_thread_in_native, id=548, stack(0xfffffffdf6e00000,0xfffffffdf6f00000)]
0x0000000101877000 JavaThread "http-8080-53" daemon [_thread_in_native, id=547, stack(0xfffffffdf7000000,0xfffffffdf7100000)]
0x0000000103acac00 JavaThread "http-8080-52" daemon [_thread_blocked, id=546, stack(0xfffffffdf7200000,0xfffffffdf7300000)]
0x0000000103aca000 JavaThread "http-8080-51" daemon [_thread_in_native, id=545, stack(0xfffffffdf7400000,0xfffffffdf7500000)]
0x0000000101bc4400 JavaThread "http-8080-50" daemon [_thread_blocked, id=544, stack(0xfffffffdf7600000,0xfffffffdf7700000)]
0x0000000100fbfc00 JavaThread "http-8080-49" daemon [_thread_blocked, id=543, stack(0xfffffffdf7800000,0xfffffffdf7900000)]
0x0000000101d9c000 JavaThread "http-8080-48" daemon [_thread_in_Java, id=541, stack(0xfffffffdf7a00000,0xfffffffdf7b00000)]
0x00000001032b3000 JavaThread "http-8080-47" daemon [_thread_in_native, id=539, stack(0xfffffffdf7e00000,0xfffffffdf7f00000)]
0x00000001032b1c00 JavaThread "http-8080-46" daemon [_thread_in_native, id=538, stack(0xfffffffdf8000000,0xfffffffdf8100000)]
0x00000001032b5000 JavaThread "http-8080-45" daemon [_thread_blocked, id=537, stack(0xfffffffdf8200000,0xfffffffdf8300000)]
0x00000001032b3c00 JavaThread "http-8080-44" daemon [_thread_blocked, id=536, stack(0xfffffffdf8400000,0xfffffffdf8500000)]
0x0000000103a77c00 JavaThread "http-8080-43" daemon [_thread_in_native, id=535, stack(0xfffffffdf8600000,0xfffffffdf8700000)]
0x00000001028acc00 JavaThread "http-8080-42" daemon [_thread_in_native, id=534, stack(0xfffffffdfba00000,0xfffffffdfbb00000)]
0x00000001032b1000 JavaThread "http-8080-41" daemon [_thread_in_native, id=533, stack(0xfffffffdf8800000,0xfffffffdf8900000)]
0x0000000102e15c00 JavaThread "http-8080-40" daemon [_thread_in_native, id=532, stack(0xfffffffdf8a00000,0xfffffffdf8b00000)]
0x0000000102e15400 JavaThread "http-8080-39" daemon [_thread_in_native, id=531, stack(0xfffffffdf8e00000,0xfffffffdf8f00000)]
0x0000000101238800 JavaThread "http-8080-38" daemon [_thread_in_native, id=530, stack(0xfffffffdf8c00000,0xfffffffdf8d00000)]
0x0000000101ce1400 JavaThread "http-8080-37" daemon [_thread_in_native, id=528, stack(0xfffffffdf9000000,0xfffffffdf9100000)]
0x0000000101651400 JavaThread "http-8080-36" daemon [_thread_in_native, id=527, stack(0xfffffffdf9200000,0xfffffffdf9300000)]
0x0000000101650800 JavaThread "http-8080-35" daemon [_thread_in_native, id=526, stack(0xfffffffdf9400000,0xfffffffdf9500000)]
0x00000001023ddc00 JavaThread "http-8080-34" daemon [_thread_in_native, id=525, stack(0xfffffffdf9600000,0xfffffffdf9700000)]
0x00000001023dd000 JavaThread "http-8080-33" daemon [_thread_in_native, id=524, stack(0xfffffffdf9800000,0xfffffffdf9900000)]
0x00000001017fb800 JavaThread "http-8080-32" daemon [_thread_in_native, id=523, stack(0xfffffffdf9a00000,0xfffffffdf9b00000)]
0x00000001023df000 JavaThread "http-8080-31" daemon [_thread_in_native, id=522, stack(0xfffffffdf9c00000,0xfffffffdf9d00000)]
0x0000000101ce6c00 JavaThread "http-8080-30" daemon [_thread_in_native, id=521, stack(0xfffffffdf9e00000,0xfffffffdf9f00000)]
0x0000000101c00400 JavaThread "http-8080-29" daemon [_thread_blocked, id=520, stack(0xfffffffdfa000000,0xfffffffdfa100000)]
0x0000000102807800 JavaThread "http-8080-28" daemon [_thread_in_native, id=519, stack(0xfffffffdfa200000,0xfffffffdfa300000)]
0x0000000102807000 JavaThread "http-8080-27" daemon [_thread_blocked, id=518, stack(0xfffffffdfa400000,0xfffffffdfa500000)]
0x0000000101035c00 JavaThread "http-8080-26" daemon [_thread_in_native, id=492, stack(0xfffffffdfae00000,0xfffffffdfaf00000)]
0x0000000103d15c00 JavaThread "http-8080-25" daemon [_thread_in_native, id=366, stack(0xfffffffdfa600000,0xfffffffdfa700000)]
0x000000010271f000 JavaThread "http-8080-24" daemon [_thread_in_native, id=365, stack(0xfffffffdfa800000,0xfffffffdfa900000)]
0x0000000102973400 JavaThread "http-8080-23" daemon [_thread_in_native, id=364, stack(0xfffffffdfaa00000,0xfffffffdfab00000)]
0x0000000102972800 JavaThread "http-8080-22" daemon [_thread_in_native, id=363, stack(0xfffffffdfac00000,0xfffffffdfad00000)]
0x0000000102974400 JavaThread "http-8080-21" daemon [_thread_in_native, id=362, stack(0xfffffffdfb000000,0xfffffffdfb100000)]
0x0000000103249c00 JavaThread "http-8080-20" daemon [_thread_in_native, id=316, stack(0xfffffffdfb200000,0xfffffffdfb300000)]
0x0000000103249400 JavaThread "http-8080-19" daemon [_thread_in_native, id=315, stack(0xfffffffdfb400000,0xfffffffdfb500000)]
0x00000001016ac000 JavaThread "http-8080-18" daemon [_thread_in_native, id=314, stack(0xfffffffe00500000,0xfffffffe00600000)]
0x0000000101678000 JavaThread "http-8080-17" daemon [_thread_in_native, id=306, stack(0xfffffffdfb600000,0xfffffffdfb700000)]
0x0000000101677000 JavaThread "http-8080-16" daemon [_thread_in_native, id=305, stack(0xfffffffdfb800000,0xfffffffdfb900000)]
0x0000000102161400 JavaThread "http-8080-15" daemon [_thread_in_native, id=304, stack(0xfffffffdfe900000,0xfffffffdfea00000)]
0x0000000101165c00 JavaThread "http-8080-14" daemon [_thread_in_native, id=303, stack(0xfffffffe00300000,0xfffffffe00400000)]
0x0000000102575000 JavaThread "http-8080-13" daemon [_thread_in_native, id=235, stack(0xfffffffe00a00000,0xfffffffe00b00000)]
0x0000000103aa2c00 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=224, stack(0xfffffffdfc200000,0xfffffffdfc300000)]
0x0000000101293400 JavaThread "UCP-worker-thread-24" daemon [_thread_blocked, id=223, stack(0xfffffffdfc700000,0xfffffffdfc800000)]
0x0000000101293c00 JavaThread "UCP-worker-thread-23" daemon [_thread_blocked, id=222, stack(0xfffffffdfc900000,0xfffffffdfca00000)]
0x0000000102ab2000 JavaThread "UCP-worker-thread-22" daemon [_thread_blocked, id=221, stack(0xfffffffdfcb00000,0xfffffffdfcc00000)]
0x00000001019be400 JavaThread "UCP-worker-thread-21" daemon [_thread_blocked, id=220, stack(0xfffffffdfcd00000,0xfffffffdfce00000)]
0x0000000103aa4400 JavaThread "UCP-worker-thread-20" daemon [_thread_blocked, id=219, stack(0xfffffffe03e00000,0xfffffffe03f00000)]
0x0000000103d88800 JavaThread "UCP-worker-thread-19" daemon [_thread_blocked, id=218, stack(0xfffffffe04400000,0xfffffffe04500000)]
0x0000000100f57800 JavaThread "UCP-worker-thread-18" daemon [_thread_blocked, id=216, stack(0xfffffffdfcf00000,0xfffffffdfd000000)]
0x0000000101e21c00 JavaThread "UCP-worker-thread-17" daemon [_thread_blocked, id=215, stack(0xfffffffdfd100000,0xfffffffdfd200000)]
0x00000001038ac400 JavaThread "pool-1-thread-10" [_thread_blocked, id=214, stack(0xfffffffdfd300000,0xfffffffdfd400000)]
0x00000001020c9800 JavaThread "pool-1-thread-9" [_thread_blocked, id=213, stack(0xfffffffdfd500000,0xfffffffdfd600000)]
0x0000000103c34000 JavaThread "pool-1-thread-8" [_thread_blocked, id=212, stack(0xfffffffdfd700000,0xfffffffdfd800000)]
0x00000001015b5800 JavaThread "pool-1-thread-7" [_thread_blocked, id=211, stack(0xfffffffdfd900000,0xfffffffdfda00000)]
0x00000001040e0800 JavaThread "pool-1-thread-6" [_thread_blocked, id=210, stack(0xfffffffdfdb00000,0xfffffffdfdc00000)]
0x00000001020cf800 JavaThread "UCP-worker-thread-16" daemon [_thread_blocked, id=209, stack(0xfffffffdfdd00000,0xfffffffdfde00000)]
0x0000000101e23000 JavaThread "GC Daemon" daemon [_thread_blocked, id=208, stack(0xfffffffdfdf00000,0xfffffffdfe000000)]
0x0000000102677400 JavaThread "RMI RenewClean-[10.72.109.90:51744]" daemon [_thread_blocked, id=207, stack(0xfffffffdfe100000,0xfffffffdfe200000)]
0x0000000103d1c800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=206, stack(0xfffffffdfe300000,0xfffffffdfe400000)]
0x0000000103b35c00 JavaThread "UCP-worker-thread-15" daemon [_thread_blocked, id=205, stack(0xfffffffdfeb00000,0xfffffffdfec00000)]
0x0000000103d1dc00 JavaThread "UCP-worker-thread-14" daemon [_thread_blocked, id=204, stack(0xfffffffdfed00000,0xfffffffdfee00000)]
0x000000010250e800 JavaThread "UCP-worker-thread-13" daemon [_thread_blocked, id=203, stack(0xfffffffdfef00000,0xfffffffdff000000)]
0x0000000103c05400 JavaThread "UCP-worker-thread-12" daemon [_thread_blocked, id=202, stack(0xfffffffe00700000,0xfffffffe00800000)]
0x0000000101813000 JavaThread "UCP-worker-thread-24" daemon [_thread_blocked, id=201, stack(0xfffffffdff100000,0xfffffffdff200000)]
0x00000001037b7400 JavaThread "UCP-worker-thread-11" daemon [_thread_blocked, id=200, stack(0xfffffffdff300000,0xfffffffdff400000)]
0x0000000101679000 JavaThread "UCP-worker-thread-23" daemon [_thread_blocked, id=199, stack(0xfffffffe09300000,0xfffffffe09400000)]
0x00000001024a9400 JavaThread "UCP-worker-thread-22" daemon [_thread_blocked, id=198, stack(0xfffffffdff600000,0xfffffffdff700000)]
0x0000000103e53800 JavaThread "UCP-worker-thread-21" daemon [_thread_blocked, id=197, stack(0xfffffffdff800000,0xfffffffdff900000)]
0x00000001017ca000 JavaThread "UCP-worker-thread-20" daemon [_thread_blocked, id=196, stack(0xfffffffe09700000,0xfffffffe09800000)]
0x000000010376c400 JavaThread "UCP-worker-thread-19" daemon [_thread_blocked, id=195, stack(0xfffffffe0fe00000,0xfffffffe0ff00000)]
0x0000000103a81800 JavaThread "UCP-worker-thread-18" daemon [_thread_blocked, id=194, stack(0xfffffffe0c500000,0xfffffffe0c600000)]
0x000000010112e400 JavaThread "http-8080-12" daemon [_thread_in_native, id=193, stack(0xfffffffdffa00000,0xfffffffdffb00000)]
0x00000001021ad800 JavaThread "UCP-worker-thread-17" daemon [_thread_blocked, id=192, stack(0xfffffffdffc00000,0xfffffffdffd00000)]
0x0000000103062c00 JavaThread "UCP-worker-thread-16" daemon [_thread_blocked, id=191, stack(0xfffffffdffe00000,0xfffffffdfff00000)]
0x00000001039e4400 JavaThread "UCP-worker-thread-10" daemon [_thread_blocked, id=190, stack(0xfffffffe00100000,0xfffffffe00200000)]
0x0000000103107800 JavaThread "TP-Monitor" daemon [_thread_blocked, id=188, stack(0xfffffffe00c00000,0xfffffffe00d00000)]
0x00000001028e2400 JavaThread "TP-Processor4" daemon [_thread_in_native, id=187, stack(0xfffffffe00e00000,0xfffffffe00f00000)]
0x0000000100f1b000 JavaThread "TP-Processor3" daemon [_thread_blocked, id=186, stack(0xfffffffe01000000,0xfffffffe01100000)]
0x00000001041fa400 JavaThread "TP-Processor2" daemon [_thread_blocked, id=185, stack(0xfffffffe01200000,0xfffffffe01300000)]
0x00000001028e3000 JavaThread "TP-Processor1" daemon [_thread_blocked, id=184, stack(0xfffffffe01400000,0xfffffffe01500000)]
0x0000000101f84c00 JavaThread "http-8080-11" daemon [_thread_blocked, id=183, stack(0xfffffffe01600000,0xfffffffe01700000)]
0x0000000101ab8c00 JavaThread "http-8080-10" daemon [_thread_in_native, id=182, stack(0xfffffffe01800000,0xfffffffe01900000)]
0x0000000101ab8400 JavaThread "http-8080-9" daemon [_thread_blocked, id=181, stack(0xfffffffe01a00000,0xfffffffe01b00000)]
0x0000000101f85400 JavaThread "http-8080-8" daemon [_thread_in_native, id=180, stack(0xfffffffe01c00000,0xfffffffe01d00000)]
0x0000000101f84000 JavaThread "http-8080-7" daemon [_thread_in_native, id=179, stack(0xfffffffe01e00000,0xfffffffe01f00000)]
0x0000000102b28c00 JavaThread "http-8080-6" daemon [_thread_in_native, id=178, stack(0xfffffffe02000000,0xfffffffe02100000)]
0x0000000102b27c00 JavaThread "http-8080-5" daemon [_thread_in_native, id=177, stack(0xfffffffe02200000,0xfffffffe02300000)]
0x00000001012f5800 JavaThread "http-8080-4" daemon [_thread_in_native, id=176, stack(0xfffffffe02400000,0xfffffffe02500000)]
0x00000001037f8800 JavaThread "http-8080-3" daemon [_thread_in_native, id=175, stack(0xfffffffe02600000,0xfffffffe02700000)]
0x0000000103804c00 JavaThread "http-8080-2" daemon [_thread_in_native, id=174, stack(0xfffffffe02800000,0xfffffffe02900000)]
0x00000001037f7800 JavaThread "http-8080-1" daemon [_thread_in_native, id=173, stack(0xfffffffe02a00000,0xfffffffe02b00000)]
0x000000010103ac00 JavaThread "http-8080-Acceptor-0" daemon [_thread_in_native, id=172, stack(0xfffffffe02c00000,0xfffffffe02d00000)]
0x00000001012f6400 JavaThread "http-8443-Acceptor-0" daemon [_thread_in_native, id=171, stack(0xfffffffe02e00000,0xfffffffe02f00000)]
0x00000001037f7400 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=170, stack(0xfffffffe03000000,0xfffffffe03100000)]
0x0000000102147c00 JavaThread "UCP-worker-thread-9" daemon [_thread_blocked, id=169, stack(0xfffffffe03200000,0xfffffffe03300000)]
0x0000000101be6400 JavaThread "pool-1-thread-5" [_thread_blocked, id=168, stack(0xfffffffe03400000,0xfffffffe03500000)]
0x0000000101be5c00 JavaThread "pool-1-thread-4" [_thread_blocked, id=167, stack(0xfffffffe03600000,0xfffffffe03700000)]
0x0000000102108400 JavaThread "pool-1-thread-3" [_thread_blocked, id=166, stack(0xfffffffe03800000,0xfffffffe03900000)]
0x0000000101aafc00 JavaThread "pool-1-thread-2" [_thread_blocked, id=165, stack(0xfffffffe03a00000,0xfffffffe03b00000)]
0x00000001021b6400 JavaThread "pool-1-thread-1" [_thread_blocked, id=164, stack(0xfffffffe03c00000,0xfffffffe03d00000)]
0x0000000102c46400 JavaThread "collector" daemon [_thread_blocked, id=162, stack(0xfffffffe04000000,0xfffffffe04100000)]
0x0000000101ac9c00 JavaThread "UCP-worker-thread-15" daemon [_thread_blocked, id=161, stack(0xfffffffe04200000,0xfffffffe04300000)]
0x0000000103aa3400 JavaThread "UCP-worker-thread-14" daemon [_thread_blocked, id=159, stack(0xfffffffe04700000,0xfffffffe04800000)]
0x0000000102f1c400 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=158, stack(0xfffffffe04900000,0xfffffffe04a00000)]
0x0000000103c07400 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=157, stack(0xfffffffe04b00000,0xfffffffe04c00000)]
0x0000000103064000 JavaThread "javamelody mobile" daemon [_thread_blocked, id=156, stack(0xfffffffe04d00000,0xfffffffe04e00000)]
0x0000000103063800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=154, stack(0xfffffffe04f00000,0xfffffffe05000000)]
0x000000010195fc00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=153, stack(0xfffffffe05100000,0xfffffffe05200000)]
0x0000000103d59c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=152, stack(0xfffffffe05300000,0xfffffffe05400000)]
0x0000000103015000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=151, stack(0xfffffffe05500000,0xfffffffe05600000)]
0x0000000103014c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=150, stack(0xfffffffe05700000,0xfffffffe05800000)]
0x0000000102775800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=149, stack(0xfffffffe05900000,0xfffffffe05a00000)]
0x0000000102774400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=148, stack(0xfffffffe05b00000,0xfffffffe05c00000)]
0x0000000102773800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=147, stack(0xfffffffe05d00000,0xfffffffe05e00000)]
0x000000010232a400 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=146, stack(0xfffffffe05f00000,0xfffffffe06000000)]
0x00000001019de000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" [_thread_blocked, id=145, stack(0xfffffffe06100000,0xfffffffe06200000)]
0x00000001019dd400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10" [_thread_blocked, id=144, stack(0xfffffffe06300000,0xfffffffe06400000)]
0x0000000102ef2c00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9" [_thread_blocked, id=143, stack(0xfffffffe06500000,0xfffffffe06600000)]
0x0000000102ef2000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8" [_thread_blocked, id=142, stack(0xfffffffe06700000,0xfffffffe06800000)]
0x0000000102e63000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-7" [_thread_blocked, id=141, stack(0xfffffffe06900000,0xfffffffe06a00000)]
0x0000000102e62000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6" [_thread_blocked, id=140, stack(0xfffffffe06b00000,0xfffffffe06c00000)]
0x0000000102bf3400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-5" [_thread_blocked, id=139, stack(0xfffffffe06d00000,0xfffffffe06e00000)]
0x0000000102bf2c00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4" [_thread_blocked, id=138, stack(0xfffffffe06f00000,0xfffffffe07000000)]
0x0000000102bf2400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3" [_thread_blocked, id=137, stack(0xfffffffe07100000,0xfffffffe07200000)]
0x0000000103d05800 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2" [_thread_blocked, id=136, stack(0xfffffffe07300000,0xfffffffe07400000)]
0x0000000102d3b000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1" [_thread_blocked, id=135, stack(0xfffffffe07500000,0xfffffffe07600000)]
0x00000001029c2000 JavaThread "UCP-worker-thread-8" daemon [_thread_blocked, id=134, stack(0xfffffffe07700000,0xfffffffe07800000)]
0x000000010180b400 JavaThread "UCP-worker-thread-13" daemon [_thread_blocked, id=133, stack(0xfffffffe07900000,0xfffffffe07a00000)]
0x00000001026c8c00 JavaThread "UCP-worker-thread-7" daemon [_thread_blocked, id=132, stack(0xfffffffe07c00000,0xfffffffe07d00000)]
0x0000000101ecb400 JavaThread "UCP-worker-thread-12" daemon [_thread_blocked, id=131, stack(0xfffffffe0bf00000,0xfffffffe0c000000)]
0x00000001039f1800 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=130, stack(0xfffffffe07e00000,0xfffffffe07f00000)]
0x00000001039f1000 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=129, stack(0xfffffffe08000000,0xfffffffe08100000)]
0x0000000101da7400 JavaThread "UCP-worker-thread-6" daemon [_thread_blocked, id=128, stack(0xfffffffe08200000,0xfffffffe08300000)]
0x0000000101da5000 JavaThread "UCP-worker-thread-11" daemon [_thread_blocked, id=127, stack(0xfffffffe08400000,0xfffffffe08500000)]
0x000000010248f400 JavaThread "UCP-worker-thread-5" daemon [_thread_blocked, id=126, stack(0xfffffffe08600000,0xfffffffe08700000)]
0x00000001029e2000 JavaThread "UCP-worker-thread-10" daemon [_thread_blocked, id=125, stack(0xfffffffe0ef00000,0xfffffffe0f000000)]
0x0000000102d67800 JavaThread "UCP-worker-thread-4" daemon [_thread_blocked, id=124, stack(0xfffffffe08800000,0xfffffffe08900000)]
0x0000000101ab1800 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=123, stack(0xfffffffe08a00000,0xfffffffe08b00000)]
0x0000000101ab2000 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=122, stack(0xfffffffe08c00000,0xfffffffe08d00000)]
0x00000001025b5000 JavaThread "UCP-worker-thread-9" daemon [_thread_blocked, id=121, stack(0xfffffffe08f00000,0xfffffffe09000000)]
0x0000000101014800 JavaThread "UCP-worker-thread-3" daemon [_thread_blocked, id=120, stack(0xfffffffe09100000,0xfffffffe09200000)]
0x000000010182e400 JavaThread "UCP-worker-thread-8" daemon [_thread_blocked, id=118, stack(0xfffffffe09500000,0xfffffffe09600000)]
0x0000000102e9a800 JavaThread "UCP-worker-thread-2" daemon [_thread_blocked, id=116, stack(0xfffffffe09900000,0xfffffffe09a00000)]
0x0000000103d7cc00 JavaThread "Timer-3" daemon [_thread_blocked, id=115, stack(0xfffffffe09b00000,0xfffffffe09c00000)]
0x0000000102e99000 JavaThread "UCP-worker-thread-7" daemon [_thread_blocked, id=114, stack(0xfffffffe09d00000,0xfffffffe09e00000)]
0x0000000102479c00 JavaThread "UCP-worker-thread-1" daemon [_thread_blocked, id=113, stack(0xfffffffe09f00000,0xfffffffe0a000000)]
0x0000000103b61800 JavaThread "Timer-2" daemon [_thread_blocked, id=112, stack(0xfffffffe0a100000,0xfffffffe0a200000)]
0x0000000101060400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=110, stack(0xfffffffe0a300000,0xfffffffe0a400000)]
0x0000000103784c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=109, stack(0xfffffffe0a500000,0xfffffffe0a600000)]
0x0000000102c4b400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=108, stack(0xfffffffe0a700000,0xfffffffe0a800000)]
0x0000000103d4bc00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=107, stack(0xfffffffe0a900000,0xfffffffe0aa00000)]
0x0000000102b75c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=106, stack(0xfffffffe0ab00000,0xfffffffe0ac00000)]
0x0000000102d21c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=105, stack(0xfffffffe0ad00000,0xfffffffe0ae00000)]
0x0000000102b73c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=104, stack(0xfffffffe0af00000,0xfffffffe0b000000)]
0x0000000102b73000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=103, stack(0xfffffffe0b100000,0xfffffffe0b200000)]
0x0000000102b71000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=102, stack(0xfffffffe0b300000,0xfffffffe0b400000)]
0x0000000102b70c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=101, stack(0xfffffffe0b500000,0xfffffffe0b600000)]
0x0000000102b70400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=100, stack(0xfffffffe0b700000,0xfffffffe0b800000)]
0x0000000101f88400 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=99, stack(0xfffffffe0ba00000,0xfffffffe0bb00000)]
0x0000000102b2a000 JavaThread "UCP-worker-thread-6" daemon [_thread_blocked, id=98, stack(0xfffffffe0bc00000,0xfffffffe0bd00000)]
0x0000000101dee400 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=96, stack(0xfffffffe0c100000,0xfffffffe0c200000)]
0x0000000101dedc00 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=95, stack(0xfffffffe0c300000,0xfffffffe0c400000)]
0x0000000101def800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=92, stack(0xfffffffe0c700000,0xfffffffe0c800000)]
0x0000000102687c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=91, stack(0xfffffffe0c900000,0xfffffffe0ca00000)]
0x0000000103288c00 JavaThread "UCP-worker-thread-5" daemon [_thread_blocked, id=90, stack(0xfffffffe0cb00000,0xfffffffe0cc00000)]
0x0000000101ded400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=89, stack(0xfffffffe0cd00000,0xfffffffe0ce00000)]
0x0000000101f98000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=88, stack(0xfffffffe0cf00000,0xfffffffe0d000000)]
0x0000000103581400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=87, stack(0xfffffffe0d100000,0xfffffffe0d200000)]
0x000000010390f800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=86, stack(0xfffffffe0d300000,0xfffffffe0d400000)]
0x000000010390e400 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=85, stack(0xfffffffe0d500000,0xfffffffe0d600000)]
0x0000000103a39400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" [_thread_blocked, id=84, stack(0xfffffffe0d700000,0xfffffffe0d800000)]
0x00000001030cb000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10" [_thread_blocked, id=83, stack(0xfffffffe0d900000,0xfffffffe0da00000)]
0x00000001040e9400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9" [_thread_blocked, id=82, stack(0xfffffffe0db00000,0xfffffffe0dc00000)]
0x00000001040e8400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8" [_thread_blocked, id=81, stack(0xfffffffe0dd00000,0xfffffffe0de00000)]
0x00000001033bf400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-7" [_thread_blocked, id=80, stack(0xfffffffe0df00000,0xfffffffe0e000000)]
0x00000001033bec00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6" [_thread_blocked, id=79, stack(0xfffffffe0e100000,0xfffffffe0e200000)]
0x00000001033be400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-5" [_thread_blocked, id=78, stack(0xfffffffe0e300000,0xfffffffe0e400000)]
0x000000010225cc00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4" [_thread_blocked, id=77, stack(0xfffffffe0e500000,0xfffffffe0e600000)]
0x000000010225c800 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3" [_thread_blocked, id=76, stack(0xfffffffe0e700000,0xfffffffe0e800000)]
0x000000010225c000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2" [_thread_blocked, id=75, stack(0xfffffffe0e900000,0xfffffffe0ea00000)]
0x0000000102fdd800 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1" [_thread_blocked, id=74, stack(0xfffffffe0eb00000,0xfffffffe0ec00000)]
0x0000000103195800 JavaThread "UCP-worker-thread-4" daemon [_thread_blocked, id=73, stack(0xfffffffe0ed00000,0xfffffffe0ee00000)]
0x0000000102dfa800 JavaThread "UCP-worker-thread-3" daemon [_thread_blocked, id=71, stack(0xfffffffe0fc00000,0xfffffffe0fd00000)]
0x0000000101907800 JavaThread "UCP-worker-thread-2" daemon [_thread_blocked, id=69, stack(0xffffffff68200000,0xffffffff68300000)]
0x0000000101547c00 JavaThread "Timer-1" daemon [_thread_blocked, id=68, stack(0xffffffff68400000,0xffffffff68500000)]
0x0000000102115c00 JavaThread "UCP-worker-thread-1" daemon [_thread_blocked, id=67, stack(0xffffffff68600000,0xffffffff68700000)]
0x00000001017f4000 JavaThread "Timer-0" daemon [_thread_blocked, id=66, stack(0xffffffff68800000,0xffffffff68900000)]
0x0000000101ee4400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=64, stack(0xffffffff68a00000,0xffffffff68b00000)]
0x000000010170b400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=63, stack(0xffffffff68c00000,0xffffffff68d00000)]
0x00000001028c6800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=62, stack(0xffffffff68e00000,0xffffffff68f00000)]
0x00000001028c5c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=61, stack(0xffffffff69000000,0xffffffff69100000)]
0x0000000101a4e800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=60, stack(0xffffffff69200000,0xffffffff69300000)]
0x0000000101a4d400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=59, stack(0xffffffff69400000,0xffffffff69500000)]
0x0000000101de9800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=58, stack(0xffffffff69600000,0xffffffff69700000)]
0x0000000102624400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=57, stack(0xffffffff69800000,0xffffffff69900000)]
0x00000001011cd400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=56, stack(0xffffffff69a00000,0xffffffff69b00000)]
0x0000000102fad800 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=55, stack(0xffffffff69c00000,0xffffffff69d00000)]
0x0000000102203800 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=54, stack(0xffffffff69e00000,0xffffffff69f00000)]
0x0000000101f38c00 JavaThread "DefaultQuartzScheduler_Worker-10" [_thread_blocked, id=53, stack(0xffffffff6a000000,0xffffffff6a100000)]
0x00000001010d0c00 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=52, stack(0xffffffff6a200000,0xffffffff6a300000)]
0x00000001014d8000 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=51, stack(0xffffffff6a400000,0xffffffff6a500000)]
0x00000001019fcc00 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=50, stack(0xffffffff6a600000,0xffffffff6a700000)]
0x00000001019fc000 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=49, stack(0xffffffff6a800000,0xffffffff6a900000)]
0x0000000102a13c00 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=48, stack(0xffffffff6aa00000,0xffffffff6ab00000)]
0x00000001018c0c00 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=47, stack(0xffffffff6ac00000,0xffffffff6ad00000)]
0x00000001010d7400 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=46, stack(0xffffffff6ae00000,0xffffffff6af00000)]
0x0000000101e91800 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=45, stack(0xffffffff6b000000,0xffffffff6b100000)]
0x0000000101ff3c00 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=44, stack(0xffffffff6b200000,0xffffffff6b300000)]
0x00000001013fb800 JavaThread "RMI TCP Accept-8999" daemon [_thread_in_native, id=42, stack(0xffffffff6b600000,0xffffffff6b700000)]
0x00000001013fb000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=41, stack(0xffffffff6b800000,0xffffffff6b900000)]
0x0000000100ec6c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=40, stack(0xffffffff6c800000,0xffffffff6c900000)]
0x0000000100ec2c00 JavaThread "CompilerThread1" daemon [_thread_blocked, id=39, stack(0xffffffff6ca00000,0xffffffff6cb00000)]
0x0000000100ec1000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=38, stack(0xffffffff6cc00000,0xffffffff6cd00000)]
0x0000000100ebfc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=37, stack(0xffffffff6ce00000,0xffffffff6cf00000)]
0x0000000100ebec00 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=36, stack(0xffffffff6d000000,0xffffffff6d100000)]
0x0000000100e96000 JavaThread "Finalizer" daemon [_thread_blocked, id=35, stack(0xffffffff6d200000,0xffffffff6d300000)]
0x0000000100e8fc00 JavaThread "Reference Handler" daemon [_thread_blocked, id=34, stack(0xffffffff6d400000,0xffffffff6d500000)]
0x0000000100117000 JavaThread "main" [_thread_in_native, id=2, stack(0xffffffff7c600000,0xffffffff7c700000)]
Other Threads:
0x0000000100e8b000 VMThread [stack: 0xffffffff6d600000,0xffffffff6d700000] [id=33]
0x0000000101148400 WatcherThread [stack: 0xffffffff6b400000,0xffffffff6b500000] [id=43]
=>0x0000000100616800 (exited) GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=30]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
par new generation total 440896K, used 287771K [0xfffffffe10000000, 0xfffffffe2de60000, 0xfffffffe2de60000)
eden space 391936K, 60% used [0xfffffffe10000000, 0xfffffffe1ea00c18, 0xfffffffe27ec0000)
from space 48960K, 100% used [0xfffffffe27ec0000, 0xfffffffe2ae90000, 0xfffffffe2ae90000)
to space 48960K, 0% used [0xfffffffe2ae90000, 0xfffffffe2ae90000, 0xfffffffe2de60000)
concurrent mark-sweep generation total 4753024K, used 4396503K [0xfffffffe2de60000, 0xffffffff50000000, 0xffffffff50000000)
concurrent-mark-sweep perm gen total 393216K, used 195407K [0xffffffff50000000, 0xffffffff68000000, 0xffffffff68000000)
Dynamic libraries:
0x0000000100000000 /usr/jdk/instances/jdk1.6.0_06/bin/sparcv9/java
0xffffffff7f700000 /lib/64/libthread.so.1
0xffffffff7f200000 /usr/jdk/instances/jdk1.6.0_06/bin/sparcv9/../../jre/lib/sparcv9/jli/libjli.so
0xffffffff7f000000 /lib/64/libdl.so.1
0xffffffff7ed00000 /lib/64/libc.so.1
0xffffffff7ec00000 /platform/SUNW,Sun-Fire-T1000/lib/sparcv9/libc_psr.so.1
0xffffffff7e000000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/server/libjvm.so
0xffffffff7de00000 /lib/64/libsocket.so.1
0xffffffff7eb00000 /usr/lib/64/libsched.so.1
0xffffffff7dc00000 /lib/64/libm.so.1
0xffffffff7da00000 /usr/lib/64/libCrun.so.1
0xffffffff7d800000 /lib/64/libdoor.so.1
0xffffffff7d600000 /lib/64/libnsl.so.1
0xffffffff7d400000 /lib/64/libm.so.2
0xffffffff7d200000 /lib/64/libscf.so.1
0xffffffff7d000000 /lib/64/libuutil.so.1
0xffffffff7ce00000 /lib/64/libgen.so.1
0xffffffff7cc00000 /lib/64/libmd.so.1
0xffffffff7ca00000 /platform/SUNW,Sun-Fire-T1000/lib/sparcv9/libmd_psr.so.1
0xffffffff7c800000 /lib/64/libmp.so.2
0xffffffff7c400000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/native_threads/libhpi.so
0xffffffff7c000000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libverify.so
0xffffffff7be00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libjava.so
0xffffffff7bc00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libzip.so
0xffffffff6c600000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libmanagement.so
0xffffffff6c400000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libnet.so
0xffffffff6c200000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libj2pkcs11.so
0xffffffff6c000000 /usr/lib/sparcv9/libpkcs11.so
0xffffffff6be00000 /usr/lib/64/libcryptoutil.so.1
0xffffffff6bc00000 /usr/lib/security/64/pkcs11_kernel.so
0xffffffff6ba00000 /usr/lib/security/64/pkcs11_softtoken_extra.so
0xfffffffe0f900000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libawt.so
0xfffffffe0f700000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/headless/libmawt.so
0xfffffffe0f500000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libnio.so
0xfffffffe0f300000 /lib/64/librt.so.1
0xfffffffe0f100000 /lib/64/libaio.so.1
0xfffffffdfe600000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/librmi.so
0xfffffffdfc400000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libfontmanager.so
0xfffffffdfc000000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libdcpr.so
0xfffffffdfbe00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libcmm.so
0xfffffffdfbc00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libjpeg.so
VM Arguments:
jvm_args: -Dcom.sun.management.jmxremote=8081 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djavax.net.ssl.keyStore=/rtiapp/apache-tomcat-6.0.16/conf/ASD-TREE-CA.keystore -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=/rtiapp/apache-tomcat-6.0.16/conf/ASD-TREE-CA.keystore -Djavax.net.ssl.trustStorePassword=changeit -Djava.util.logging.config.file=/rtiapp/apache-tomcat-6.0.16/conf/logging.properties -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xmx5120m -Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.endorsed.dirs=/rtiapp/apache-tomcat-6.0.16/endorsed -Dcatalina.base=/rtiapp/apache-tomcat-6.0.16 -Dcatalina.home=/rtiapp/apache-tomcat-6.0.16 -Djava.io.tmpdir=/rtiapp/apache-tomcat-6.0.16/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/bin
LD_LIBRARY_PATH=/usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/server:/usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9:/usr/jdk/instances/jdk1.6.0_06/jre/../lib/sparcv9
SHELL=/bin/sh
Signal Handlers:
SIGSEGV: [libjvm.so+0x733d70], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0x733d70], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x636530], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: [libjvm.so+0x636530], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x636530], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x639610], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
--------------- S Y S T E M ---------------
OS: Solaris 10 8/07 s10s_u4wos_12b SPARC
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007
uname:SunOS 5.10 Generic_142900-02 sun4v (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 32000, AS infinity
load average:4.01 3.93 3.46
CPU:total 32 has_v8, has_v9, has_vis1, is_sun4v, is_niagara1
Memory: 8k page, physical 16646144k(8588672k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (10.0-b22) for solaris-sparc JRE (1.6.0_06-b02), built on Mar 25 2008 00:31:18 by "" with Workshop 5.8
time: Wed Mar 18 10:53:46 2015
elapsed time: 106039 seconds
REPRODUCIBILITY :
This bug can be reproduced occasionally.
CUSTOMER SUBMITTED WORKAROUND :
Have to restart the server.
SUPPORT :
YES
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Solaris 10 8/07 s10s_u4wos_12b SPARC
EXTRA RELEVANT SYSTEM CONFIGURATION :
System Configuration: Sun Microsystems sun4v Sun Fire(TM) T1000
Memory size: 16256 Megabytes
JVM OPTS
-d64
-XX:PermSize=384m
-XX:MaxPermSize=384m
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xmx5120m
A DESCRIPTION OF THE PROBLEM :
We have got web applications hosted on Tomcat 6 which runs on jdk 1.6.0_06
We are using Oracle UCP connection pool for DB connections along with ojdbc6
The application were running all good all of a sudden we got reports when we checked found the java process got crashed and killed so had to start up again
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
We have not got exact replication steps for the crash, the applications were being used all good until the point we do not see any out of memory error but from the crash dump able to see we have got threads blocked hence not sure if this was due to GC thread which caused stop the world or is some thing else that trigerred it.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Java process should not be crashed / killed
ACTUAL -
Java process was crashed
ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0xffffffff7e3b088c, pid=28352, tid=30
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode solaris-sparc)
# Problematic frame:
# V [libjvm.so+0x3b088c]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0000000100616800): GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=30]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN), si_addr=0x000000000000001d
Registers:
O0=0x0000000000000003 O1=0x0000000000002a54 O2=0x0000000000000002 O3=0x0000000000000002
O4=0x0000000000002a52 O5=0x000000002a5253c7 O6=0xffffffff6defef01 O7=0x00000000000152a0
G1=0x0000000000002a54 G2=0x0000000000042400 G3=0x0000000000000018 G4=0xffffffffffffc018
G5=0x000000000000000d G6=0x0000000000000000 G7=0xffffffff7f40ea00 Y=0x0000000000000000
PC=0xffffffff7e3b088c nPC=0xffffffff7e3b0890
Top of Stack: (sp=0xffffffff6deff700)
0xffffffff6deff700: ffffffffffffffff ffffffff6deff818
0xffffffff6deff710: 0000000000003ffe 0000000000003fff
0xffffffff6deff720: 0000000100134ce0 0000000000002a52
0xffffffff6deff730: 0000000000002a54 0000000000042440
0xffffffff6deff740: ffffffff6deffa10 00000001006185f0
0xffffffff6deff750: 0000000000002a55 fffffffecdacb950
0xffffffff6deff760: 000000002a5253c7 ffffffff7e99e000
0xffffffff6deff770: ffffffff6deff091 ffffffff7e3350e8
0xffffffff6deff780: 0000000000000000 0000000000000000
0xffffffff6deff790: 0000000000000000 0000000000000000
0xffffffff6deff7a0: 0000000000000000 0000000000000000
0xffffffff6deff7b0: 0000000100611f50 ffffffff01604690
0xffffffff6deff7c0: ffffffff6daff768 ffffffff6deffa10
0xffffffff6deff7d0: 2a5253c600000000 0000000000000000
0xffffffff6deff7e0: ffffffff7e3b0548 ffffffff2a5253c6
0xffffffff6deff7f0: 0000000100611d50 000000002743403f
Instructions: (pc=0xffffffff7e3b088c)
0xffffffff7e3b087c: 80 90 00 08 22 48 00 22 ea 5e 20 48 ca 5e e0 08
0xffffffff7e3b088c: c8 59 60 10 92 10 00 1b 94 10 00 11 c6 59 22 60
Stack: [0x0000000000000000,0x0000000000000000], sp=0xffffffff6deff700, free space=-2393091k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x3b088c]
V [libjvm.so+0x3350f0]
V [libjvm.so+0x3a639c]
V [libjvm.so+0x3a5de0]
V [libjvm.so+0x73d968]
V [libjvm.so+0x634324]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000104b45000 JavaThread "Thread-4518" daemon [_thread_blocked, id=8514, stack(0xfffffffde7700000,0xfffffffde7800000)]
0x0000000104724c00 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=8513, stack(0xfffffffde7100000,0xfffffffde7200000)]
0x0000000104b4c800 JavaThread "Thread-4365" daemon [_thread_in_native, id=8291, stack(0xfffffffde7300000,0xfffffffde7400000)]
0x0000000105041c00 JavaThread "http-8080-172" daemon [_thread_in_native, id=8202, stack(0xfffffffde7500000,0xfffffffde7600000)]
0x0000000102da3800 JavaThread "http-8080-171" daemon [_thread_in_native, id=8201, stack(0xfffffffde7b00000,0xfffffffde7c00000)]
0x0000000103376800 JavaThread "http-8080-170" daemon [_thread_in_native, id=8200, stack(0xfffffffde7f00000,0xfffffffde8000000)]
0x00000001017bb800 JavaThread "http-8080-169" daemon [_thread_in_native, id=8199, stack(0xfffffffde7d00000,0xfffffffde7e00000)]
0x0000000104af6800 JavaThread "http-8080-168" daemon [_thread_in_native, id=8198, stack(0xfffffffdea300000,0xfffffffdea400000)]
0x00000001011a0400 JavaThread "http-8080-167" daemon [_thread_in_native, id=8197, stack(0xfffffffde8100000,0xfffffffde8200000)]
0x0000000104958c00 JavaThread "http-8080-166" daemon [_thread_in_native, id=8196, stack(0xfffffffde8300000,0xfffffffde8400000)]
0x0000000102f87c00 JavaThread "http-8443-3" daemon [_thread_blocked, id=3556, stack(0xfffffffde8f00000,0xfffffffde9000000)]
0x0000000104fa4000 JavaThread "http-8443-2" daemon [_thread_blocked, id=3546, stack(0xfffffffde8500000,0xfffffffde8600000)]
0x0000000105a51800 JavaThread "http-8443-1" daemon [_thread_blocked, id=3545, stack(0xfffffffde9300000,0xfffffffde9400000)]
0x000000010587bc00 JavaThread "http-8080-165" daemon [_thread_in_native, id=3532, stack(0xfffffffde8b00000,0xfffffffde8c00000)]
0x0000000104b04c00 JavaThread "http-8080-164" daemon [_thread_in_native, id=3531, stack(0xfffffffde8900000,0xfffffffde8a00000)]
0x0000000104dc8400 JavaThread "http-8080-163" daemon [_thread_in_native, id=2664, stack(0xfffffffde8700000,0xfffffffde8800000)]
0x0000000105a56000 JavaThread "http-8080-162" daemon [_thread_in_native, id=2555, stack(0xfffffffde8d00000,0xfffffffde8e00000)]
0x0000000105a5a000 JavaThread "http-8080-161" daemon [_thread_in_native, id=2554, stack(0xfffffffde9100000,0xfffffffde9200000)]
0x00000001025c6c00 JavaThread "http-8080-160" daemon [_thread_in_native, id=2056, stack(0xfffffffde9500000,0xfffffffde9600000)]
0x00000001025c6400 JavaThread "http-8080-159" daemon [_thread_in_native, id=2055, stack(0xfffffffde9700000,0xfffffffde9800000)]
0x00000001042f7400 JavaThread "http-8080-158" daemon [_thread_in_native, id=2054, stack(0xfffffffde9900000,0xfffffffde9a00000)]
0x00000001042f7000 JavaThread "http-8080-157" daemon [_thread_in_native, id=2053, stack(0xfffffffde9b00000,0xfffffffde9c00000)]
0x0000000103bf1000 JavaThread "http-8080-156" daemon [_thread_in_native, id=2033, stack(0xfffffffde9d00000,0xfffffffde9e00000)]
0x0000000102b95c00 JavaThread "http-8080-155" daemon [_thread_blocked, id=2032, stack(0xfffffffdeea00000,0xfffffffdeeb00000)]
0x00000001046a4800 JavaThread "http-8080-154" daemon [_thread_in_native, id=2002, stack(0xfffffffde9f00000,0xfffffffdea000000)]
0x0000000102b31800 JavaThread "http-8080-153" daemon [_thread_in_native, id=1906, stack(0xfffffffdea100000,0xfffffffdea200000)]
0x0000000102325400 JavaThread "http-8080-152" daemon [_thread_in_native, id=1864, stack(0xfffffffdea500000,0xfffffffdea600000)]
0x0000000102324c00 JavaThread "http-8080-151" daemon [_thread_in_native, id=1863, stack(0xfffffffdea700000,0xfffffffdea800000)]
0x0000000102a5e400 JavaThread "http-8080-150" daemon [_thread_in_native, id=1862, stack(0xfffffffdeab00000,0xfffffffdeac00000)]
0x0000000102a5dc00 JavaThread "http-8080-149" daemon [_thread_in_native, id=1861, stack(0xfffffffdea900000,0xfffffffdeaa00000)]
0x0000000101694c00 JavaThread "http-8080-148" daemon [_thread_in_native, id=1860, stack(0xfffffffdead00000,0xfffffffdeae00000)]
0x0000000103b2d000 JavaThread "http-8080-147" daemon [_thread_in_native, id=1859, stack(0xfffffffdeaf00000,0xfffffffdeb000000)]
0x000000010392b800 JavaThread "http-8080-146" daemon [_thread_in_native, id=1792, stack(0xfffffffdeb100000,0xfffffffdeb200000)]
0x0000000103303c00 JavaThread "http-8080-145" daemon [_thread_in_native, id=1791, stack(0xfffffffdeb300000,0xfffffffdeb400000)]
0x0000000102461000 JavaThread "http-8080-144" daemon [_thread_blocked, id=1790, stack(0xfffffffdeb500000,0xfffffffdeb600000)]
0x0000000102d08000 JavaThread "http-8080-143" daemon [_thread_in_native, id=1789, stack(0xfffffffdeb700000,0xfffffffdeb800000)]
0x0000000102eaa800 JavaThread "http-8080-142" daemon [_thread_in_native, id=1788, stack(0xfffffffdecd00000,0xfffffffdece00000)]
0x0000000102f51400 JavaThread "http-8080-141" daemon [_thread_in_Java, id=1584, stack(0xfffffffdeb900000,0xfffffffdeba00000)]
0x0000000102f51000 JavaThread "http-8080-140" daemon [_thread_blocked, id=1583, stack(0xfffffffdebb00000,0xfffffffdebc00000)]
0x00000001049c1800 JavaThread "http-8080-139" daemon [_thread_in_native, id=1582, stack(0xfffffffdebd00000,0xfffffffdebe00000)]
0x0000000102ea8400 JavaThread "http-8080-138" daemon [_thread_in_native, id=1581, stack(0xfffffffdebf00000,0xfffffffdec000000)]
0x00000001025fa400 JavaThread "http-8080-137" daemon [_thread_in_native, id=1520, stack(0xfffffffdec100000,0xfffffffdec200000)]
0x00000001025fc400 JavaThread "http-8080-136" daemon [_thread_in_native, id=1519, stack(0xfffffffdec300000,0xfffffffdec400000)]
0x00000001025ff800 JavaThread "http-8080-135" daemon [_thread_blocked, id=1518, stack(0xfffffffdec500000,0xfffffffdec600000)]
0x00000001025f9400 JavaThread "http-8080-134" daemon [_thread_blocked, id=1517, stack(0xfffffffdec700000,0xfffffffdec800000)]
0x00000001025a1400 JavaThread "http-8080-133" daemon [_thread_in_native, id=1516, stack(0xfffffffdec900000,0xfffffffdeca00000)]
0x0000000104b81000 JavaThread "http-8080-132" daemon [_thread_in_native, id=1515, stack(0xfffffffdecb00000,0xfffffffdecc00000)]
0x0000000103373c00 JavaThread "http-8080-131" daemon [_thread_in_native, id=1441, stack(0xfffffffdecf00000,0xfffffffded000000)]
0x0000000102186c00 JavaThread "http-8080-130" daemon [_thread_in_native, id=1440, stack(0xfffffffded100000,0xfffffffded200000)]
0x0000000101a3b400 JavaThread "http-8080-129" daemon [_thread_in_native, id=1438, stack(0xfffffffded300000,0xfffffffded400000)]
0x0000000103d9cc00 JavaThread "http-8080-128" daemon [_thread_in_native, id=1437, stack(0xfffffffded900000,0xfffffffdeda00000)]
0x0000000102185000 JavaThread "http-8080-127" daemon [_thread_in_native, id=1435, stack(0xfffffffded500000,0xfffffffded600000)]
0x0000000103373000 JavaThread "http-8080-126" daemon [_thread_in_native, id=1434, stack(0xfffffffded700000,0xfffffffded800000)]
0x0000000103c58400 JavaThread "http-8080-125" daemon [_thread_in_native, id=1432, stack(0xfffffffdedb00000,0xfffffffdedc00000)]
0x00000001011b4c00 JavaThread "http-8080-124" daemon [_thread_in_native, id=1431, stack(0xfffffffdedd00000,0xfffffffdede00000)]
0x0000000102298800 JavaThread "http-8080-123" daemon [_thread_in_native, id=1430, stack(0xfffffffdee600000,0xfffffffdee700000)]
0x00000001024a7400 JavaThread "http-8080-122" daemon [_thread_in_native, id=1415, stack(0xfffffffdee400000,0xfffffffdee500000)]
0x000000010229c000 JavaThread "http-8080-121" daemon [_thread_in_native, id=1414, stack(0xfffffffdee800000,0xfffffffdee900000)]
0x0000000102f21000 JavaThread "http-8080-120" daemon [_thread_in_native, id=1251, stack(0xfffffffdee200000,0xfffffffdee300000)]
0x0000000103aabc00 JavaThread "http-8080-119" daemon [_thread_in_native, id=934, stack(0xfffffffdf6800000,0xfffffffdf6900000)]
0x0000000101723400 JavaThread "http-8080-118" daemon [_thread_in_Java, id=933, stack(0xfffffffdeec00000,0xfffffffdeed00000)]
0x0000000101e3f800 JavaThread "http-8080-117" daemon [_thread_in_native, id=932, stack(0xfffffffdeee00000,0xfffffffdeef00000)]
0x0000000101498c00 JavaThread "http-8080-116" daemon [_thread_in_native, id=931, stack(0xfffffffdef200000,0xfffffffdef300000)]
0x00000001037a3000 JavaThread "http-8080-115" daemon [_thread_in_native, id=930, stack(0xfffffffdef000000,0xfffffffdef100000)]
0x00000001023c7000 JavaThread "http-8080-114" daemon [_thread_in_native, id=928, stack(0xfffffffdef400000,0xfffffffdef500000)]
0x0000000103a72000 JavaThread "http-8080-113" daemon [_thread_in_native, id=927, stack(0xfffffffdef600000,0xfffffffdef700000)]
0x0000000103a71400 JavaThread "http-8080-112" daemon [_thread_in_native, id=926, stack(0xfffffffdef800000,0xfffffffdef900000)]
0x0000000103a70800 JavaThread "http-8080-111" daemon [_thread_in_native, id=925, stack(0xfffffffdefa00000,0xfffffffdefb00000)]
0x0000000103a6fc00 JavaThread "http-8080-110" daemon [_thread_in_native, id=924, stack(0xfffffffdefc00000,0xfffffffdefd00000)]
0x000000010213fc00 JavaThread "http-8080-109" daemon [_thread_in_native, id=923, stack(0xfffffffdefe00000,0xfffffffdeff00000)]
0x000000010213f800 JavaThread "http-8080-108" daemon [_thread_in_native, id=922, stack(0xfffffffdf0000000,0xfffffffdf0100000)]
0x000000010379f400 JavaThread "http-8080-107" daemon [_thread_in_native, id=921, stack(0xfffffffdf0200000,0xfffffffdf0300000)]
0x000000010379ec00 JavaThread "http-8080-106" daemon [_thread_in_native, id=920, stack(0xfffffffdf0400000,0xfffffffdf0500000)]
0x000000010175e800 JavaThread "http-8080-105" daemon [_thread_in_native, id=919, stack(0xfffffffdf0600000,0xfffffffdf0700000)]
0x000000010175dc00 JavaThread "http-8080-104" daemon [_thread_in_native, id=918, stack(0xfffffffdf0800000,0xfffffffdf0900000)]
0x00000001013d9800 JavaThread "http-8080-103" daemon [_thread_in_native, id=917, stack(0xfffffffdf0a00000,0xfffffffdf0b00000)]
0x00000001013d8c00 JavaThread "http-8080-102" daemon [_thread_in_native, id=916, stack(0xfffffffdf0c00000,0xfffffffdf0d00000)]
0x00000001013d7c00 JavaThread "http-8080-101" daemon [_thread_in_native, id=915, stack(0xfffffffdf0e00000,0xfffffffdf0f00000)]
0x00000001013d6c00 JavaThread "http-8080-100" daemon [_thread_in_native, id=914, stack(0xfffffffdf1000000,0xfffffffdf1100000)]
0x00000001013d6000 JavaThread "http-8080-99" daemon [_thread_in_native, id=913, stack(0xfffffffdf1200000,0xfffffffdf1300000)]
0x0000000101903400 JavaThread "http-8080-98" daemon [_thread_in_native, id=912, stack(0xfffffffdf1400000,0xfffffffdf1500000)]
0x0000000101902400 JavaThread "http-8080-97" daemon [_thread_in_native, id=911, stack(0xfffffffdf1600000,0xfffffffdf1700000)]
0x0000000101901400 JavaThread "http-8080-96" daemon [_thread_blocked, id=910, stack(0xfffffffdf1800000,0xfffffffdf1900000)]
0x0000000101900c00 JavaThread "http-8080-95" daemon [_thread_in_native, id=909, stack(0xfffffffdf1a00000,0xfffffffdf1b00000)]
0x00000001037a8800 JavaThread "http-8080-94" daemon [_thread_blocked, id=908, stack(0xfffffffdf1c00000,0xfffffffdf1d00000)]
0x00000001037a7800 JavaThread "http-8080-93" daemon [_thread_in_native, id=907, stack(0xfffffffdf1e00000,0xfffffffdf1f00000)]
0x0000000103147400 JavaThread "http-8080-92" daemon [_thread_in_native, id=906, stack(0xfffffffdf2000000,0xfffffffdf2100000)]
0x0000000103146c00 JavaThread "http-8080-91" daemon [_thread_blocked, id=905, stack(0xfffffffdf2600000,0xfffffffdf2700000)]
0x0000000103146400 JavaThread "http-8080-90" daemon [_thread_in_native, id=904, stack(0xfffffffdf2800000,0xfffffffdf2900000)]
0x00000001037a6c00 JavaThread "http-8080-89" daemon [_thread_in_native, id=903, stack(0xfffffffdf2400000,0xfffffffdf2500000)]
0x000000010135c800 JavaThread "http-8080-88" daemon [_thread_in_native, id=617, stack(0xfffffffdf2a00000,0xfffffffdf2b00000)]
0x0000000102850000 JavaThread "http-8080-87" daemon [_thread_in_native, id=616, stack(0xfffffffdf2c00000,0xfffffffdf2d00000)]
0x0000000101d42000 JavaThread "http-8080-86" daemon [_thread_blocked, id=615, stack(0xfffffffdf2e00000,0xfffffffdf2f00000)]
0x0000000102be4000 JavaThread "http-8080-85" daemon [_thread_in_native, id=614, stack(0xfffffffdf3000000,0xfffffffdf3100000)]
0x0000000102d5a000 JavaThread "http-8080-84" daemon [_thread_in_native, id=613, stack(0xfffffffdf3200000,0xfffffffdf3300000)]
0x000000010247cc00 JavaThread "http-8080-83" daemon [_thread_in_native, id=612, stack(0xfffffffdf3400000,0xfffffffdf3500000)]
0x00000001028dfc00 JavaThread "http-8080-82" daemon [_thread_in_native, id=611, stack(0xfffffffdf3600000,0xfffffffdf3700000)]
0x000000010247e800 JavaThread "http-8080-81" daemon [_thread_in_native, id=610, stack(0xfffffffdf3800000,0xfffffffdf3900000)]
0x0000000102be3800 JavaThread "http-8080-80" daemon [_thread_in_native, id=609, stack(0xfffffffdf3a00000,0xfffffffdf3b00000)]
0x00000001017f9400 JavaThread "http-8080-79" daemon [_thread_in_native, id=608, stack(0xfffffffdf3c00000,0xfffffffdf3d00000)]
0x00000001014b6400 JavaThread "http-8080-78" daemon [_thread_in_native, id=607, stack(0xfffffffdf3e00000,0xfffffffdf3f00000)]
0x0000000102e10c00 JavaThread "http-8080-77" daemon [_thread_in_native, id=606, stack(0xfffffffdf4000000,0xfffffffdf4100000)]
0x0000000101bc0800 JavaThread "http-8080-76" daemon [_thread_in_native, id=605, stack(0xfffffffdf4200000,0xfffffffdf4300000)]
0x000000010267dc00 JavaThread "http-8080-75" daemon [_thread_in_native, id=602, stack(0xfffffffdf7c00000,0xfffffffdf7d00000)]
0x0000000103d7b000 JavaThread "http-8080-74" daemon [_thread_in_native, id=600, stack(0xfffffffdf4400000,0xfffffffdf4500000)]
0x0000000102808c00 JavaThread "http-8080-73" daemon [_thread_blocked, id=599, stack(0xfffffffdf4600000,0xfffffffdf4700000)]
0x0000000102ed0000 JavaThread "http-8080-72" daemon [_thread_in_native, id=598, stack(0xfffffffdf4800000,0xfffffffdf4900000)]
0x0000000101448000 JavaThread "http-8080-71" daemon [_thread_in_native, id=597, stack(0xfffffffdf5600000,0xfffffffdf5700000)]
0x0000000101e62c00 JavaThread "http-8080-70" daemon [_thread_in_Java, id=596, stack(0xfffffffdf4a00000,0xfffffffdf4b00000)]
0x0000000101aee000 JavaThread "http-8080-69" daemon [_thread_in_native, id=595, stack(0xfffffffdf4c00000,0xfffffffdf4d00000)]
0x000000010311a400 JavaThread "http-8080-68" daemon [_thread_in_native, id=584, stack(0xfffffffdf4e00000,0xfffffffdf4f00000)]
0x0000000103119000 JavaThread "http-8080-67" daemon [_thread_blocked, id=583, stack(0xfffffffdf5000000,0xfffffffdf5100000)]
0x0000000103117800 JavaThread "http-8080-66" daemon [_thread_blocked, id=582, stack(0xfffffffdf5200000,0xfffffffdf5300000)]
0x0000000102bffc00 JavaThread "http-8080-65" daemon [_thread_in_native, id=581, stack(0xfffffffdf5400000,0xfffffffdf5500000)]
0x000000010177b400 JavaThread "http-8080-64" daemon [_thread_in_native, id=578, stack(0xfffffffdf5800000,0xfffffffdf5900000)]
0x0000000103d6a800 JavaThread "http-8080-63" daemon [_thread_blocked, id=577, stack(0xfffffffdf5a00000,0xfffffffdf5b00000)]
0x0000000103d6a000 JavaThread "http-8080-62" daemon [_thread_in_native, id=576, stack(0xfffffffdf6600000,0xfffffffdf6700000)]
0x0000000101b3fc00 JavaThread "http-8080-61" daemon [_thread_in_native, id=574, stack(0xfffffffdf5c00000,0xfffffffdf5d00000)]
0x000000010270e400 JavaThread "http-8080-60" daemon [_thread_in_native, id=573, stack(0xfffffffdf5e00000,0xfffffffdf5f00000)]
0x00000001028dcc00 JavaThread "http-8080-59" daemon [_thread_in_native, id=571, stack(0xfffffffdf6000000,0xfffffffdf6100000)]
0x00000001028dc400 JavaThread "http-8080-58" daemon [_thread_in_Java, id=570, stack(0xfffffffdf6200000,0xfffffffdf6300000)]
0x00000001028db000 JavaThread "http-8080-57" daemon [_thread_in_native, id=569, stack(0xfffffffdf6400000,0xfffffffdf6500000)]
0x00000001014a4800 JavaThread "http-8080-56" daemon [_thread_in_native, id=566, stack(0xfffffffdf6a00000,0xfffffffdf6b00000)]
0x0000000102481c00 JavaThread "http-8080-55" daemon [_thread_blocked, id=549, stack(0xfffffffdf6c00000,0xfffffffdf6d00000)]
0x0000000100fc2000 JavaThread "http-8080-54" daemon [_thread_in_native, id=548, stack(0xfffffffdf6e00000,0xfffffffdf6f00000)]
0x0000000101877000 JavaThread "http-8080-53" daemon [_thread_in_native, id=547, stack(0xfffffffdf7000000,0xfffffffdf7100000)]
0x0000000103acac00 JavaThread "http-8080-52" daemon [_thread_blocked, id=546, stack(0xfffffffdf7200000,0xfffffffdf7300000)]
0x0000000103aca000 JavaThread "http-8080-51" daemon [_thread_in_native, id=545, stack(0xfffffffdf7400000,0xfffffffdf7500000)]
0x0000000101bc4400 JavaThread "http-8080-50" daemon [_thread_blocked, id=544, stack(0xfffffffdf7600000,0xfffffffdf7700000)]
0x0000000100fbfc00 JavaThread "http-8080-49" daemon [_thread_blocked, id=543, stack(0xfffffffdf7800000,0xfffffffdf7900000)]
0x0000000101d9c000 JavaThread "http-8080-48" daemon [_thread_in_Java, id=541, stack(0xfffffffdf7a00000,0xfffffffdf7b00000)]
0x00000001032b3000 JavaThread "http-8080-47" daemon [_thread_in_native, id=539, stack(0xfffffffdf7e00000,0xfffffffdf7f00000)]
0x00000001032b1c00 JavaThread "http-8080-46" daemon [_thread_in_native, id=538, stack(0xfffffffdf8000000,0xfffffffdf8100000)]
0x00000001032b5000 JavaThread "http-8080-45" daemon [_thread_blocked, id=537, stack(0xfffffffdf8200000,0xfffffffdf8300000)]
0x00000001032b3c00 JavaThread "http-8080-44" daemon [_thread_blocked, id=536, stack(0xfffffffdf8400000,0xfffffffdf8500000)]
0x0000000103a77c00 JavaThread "http-8080-43" daemon [_thread_in_native, id=535, stack(0xfffffffdf8600000,0xfffffffdf8700000)]
0x00000001028acc00 JavaThread "http-8080-42" daemon [_thread_in_native, id=534, stack(0xfffffffdfba00000,0xfffffffdfbb00000)]
0x00000001032b1000 JavaThread "http-8080-41" daemon [_thread_in_native, id=533, stack(0xfffffffdf8800000,0xfffffffdf8900000)]
0x0000000102e15c00 JavaThread "http-8080-40" daemon [_thread_in_native, id=532, stack(0xfffffffdf8a00000,0xfffffffdf8b00000)]
0x0000000102e15400 JavaThread "http-8080-39" daemon [_thread_in_native, id=531, stack(0xfffffffdf8e00000,0xfffffffdf8f00000)]
0x0000000101238800 JavaThread "http-8080-38" daemon [_thread_in_native, id=530, stack(0xfffffffdf8c00000,0xfffffffdf8d00000)]
0x0000000101ce1400 JavaThread "http-8080-37" daemon [_thread_in_native, id=528, stack(0xfffffffdf9000000,0xfffffffdf9100000)]
0x0000000101651400 JavaThread "http-8080-36" daemon [_thread_in_native, id=527, stack(0xfffffffdf9200000,0xfffffffdf9300000)]
0x0000000101650800 JavaThread "http-8080-35" daemon [_thread_in_native, id=526, stack(0xfffffffdf9400000,0xfffffffdf9500000)]
0x00000001023ddc00 JavaThread "http-8080-34" daemon [_thread_in_native, id=525, stack(0xfffffffdf9600000,0xfffffffdf9700000)]
0x00000001023dd000 JavaThread "http-8080-33" daemon [_thread_in_native, id=524, stack(0xfffffffdf9800000,0xfffffffdf9900000)]
0x00000001017fb800 JavaThread "http-8080-32" daemon [_thread_in_native, id=523, stack(0xfffffffdf9a00000,0xfffffffdf9b00000)]
0x00000001023df000 JavaThread "http-8080-31" daemon [_thread_in_native, id=522, stack(0xfffffffdf9c00000,0xfffffffdf9d00000)]
0x0000000101ce6c00 JavaThread "http-8080-30" daemon [_thread_in_native, id=521, stack(0xfffffffdf9e00000,0xfffffffdf9f00000)]
0x0000000101c00400 JavaThread "http-8080-29" daemon [_thread_blocked, id=520, stack(0xfffffffdfa000000,0xfffffffdfa100000)]
0x0000000102807800 JavaThread "http-8080-28" daemon [_thread_in_native, id=519, stack(0xfffffffdfa200000,0xfffffffdfa300000)]
0x0000000102807000 JavaThread "http-8080-27" daemon [_thread_blocked, id=518, stack(0xfffffffdfa400000,0xfffffffdfa500000)]
0x0000000101035c00 JavaThread "http-8080-26" daemon [_thread_in_native, id=492, stack(0xfffffffdfae00000,0xfffffffdfaf00000)]
0x0000000103d15c00 JavaThread "http-8080-25" daemon [_thread_in_native, id=366, stack(0xfffffffdfa600000,0xfffffffdfa700000)]
0x000000010271f000 JavaThread "http-8080-24" daemon [_thread_in_native, id=365, stack(0xfffffffdfa800000,0xfffffffdfa900000)]
0x0000000102973400 JavaThread "http-8080-23" daemon [_thread_in_native, id=364, stack(0xfffffffdfaa00000,0xfffffffdfab00000)]
0x0000000102972800 JavaThread "http-8080-22" daemon [_thread_in_native, id=363, stack(0xfffffffdfac00000,0xfffffffdfad00000)]
0x0000000102974400 JavaThread "http-8080-21" daemon [_thread_in_native, id=362, stack(0xfffffffdfb000000,0xfffffffdfb100000)]
0x0000000103249c00 JavaThread "http-8080-20" daemon [_thread_in_native, id=316, stack(0xfffffffdfb200000,0xfffffffdfb300000)]
0x0000000103249400 JavaThread "http-8080-19" daemon [_thread_in_native, id=315, stack(0xfffffffdfb400000,0xfffffffdfb500000)]
0x00000001016ac000 JavaThread "http-8080-18" daemon [_thread_in_native, id=314, stack(0xfffffffe00500000,0xfffffffe00600000)]
0x0000000101678000 JavaThread "http-8080-17" daemon [_thread_in_native, id=306, stack(0xfffffffdfb600000,0xfffffffdfb700000)]
0x0000000101677000 JavaThread "http-8080-16" daemon [_thread_in_native, id=305, stack(0xfffffffdfb800000,0xfffffffdfb900000)]
0x0000000102161400 JavaThread "http-8080-15" daemon [_thread_in_native, id=304, stack(0xfffffffdfe900000,0xfffffffdfea00000)]
0x0000000101165c00 JavaThread "http-8080-14" daemon [_thread_in_native, id=303, stack(0xfffffffe00300000,0xfffffffe00400000)]
0x0000000102575000 JavaThread "http-8080-13" daemon [_thread_in_native, id=235, stack(0xfffffffe00a00000,0xfffffffe00b00000)]
0x0000000103aa2c00 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=224, stack(0xfffffffdfc200000,0xfffffffdfc300000)]
0x0000000101293400 JavaThread "UCP-worker-thread-24" daemon [_thread_blocked, id=223, stack(0xfffffffdfc700000,0xfffffffdfc800000)]
0x0000000101293c00 JavaThread "UCP-worker-thread-23" daemon [_thread_blocked, id=222, stack(0xfffffffdfc900000,0xfffffffdfca00000)]
0x0000000102ab2000 JavaThread "UCP-worker-thread-22" daemon [_thread_blocked, id=221, stack(0xfffffffdfcb00000,0xfffffffdfcc00000)]
0x00000001019be400 JavaThread "UCP-worker-thread-21" daemon [_thread_blocked, id=220, stack(0xfffffffdfcd00000,0xfffffffdfce00000)]
0x0000000103aa4400 JavaThread "UCP-worker-thread-20" daemon [_thread_blocked, id=219, stack(0xfffffffe03e00000,0xfffffffe03f00000)]
0x0000000103d88800 JavaThread "UCP-worker-thread-19" daemon [_thread_blocked, id=218, stack(0xfffffffe04400000,0xfffffffe04500000)]
0x0000000100f57800 JavaThread "UCP-worker-thread-18" daemon [_thread_blocked, id=216, stack(0xfffffffdfcf00000,0xfffffffdfd000000)]
0x0000000101e21c00 JavaThread "UCP-worker-thread-17" daemon [_thread_blocked, id=215, stack(0xfffffffdfd100000,0xfffffffdfd200000)]
0x00000001038ac400 JavaThread "pool-1-thread-10" [_thread_blocked, id=214, stack(0xfffffffdfd300000,0xfffffffdfd400000)]
0x00000001020c9800 JavaThread "pool-1-thread-9" [_thread_blocked, id=213, stack(0xfffffffdfd500000,0xfffffffdfd600000)]
0x0000000103c34000 JavaThread "pool-1-thread-8" [_thread_blocked, id=212, stack(0xfffffffdfd700000,0xfffffffdfd800000)]
0x00000001015b5800 JavaThread "pool-1-thread-7" [_thread_blocked, id=211, stack(0xfffffffdfd900000,0xfffffffdfda00000)]
0x00000001040e0800 JavaThread "pool-1-thread-6" [_thread_blocked, id=210, stack(0xfffffffdfdb00000,0xfffffffdfdc00000)]
0x00000001020cf800 JavaThread "UCP-worker-thread-16" daemon [_thread_blocked, id=209, stack(0xfffffffdfdd00000,0xfffffffdfde00000)]
0x0000000101e23000 JavaThread "GC Daemon" daemon [_thread_blocked, id=208, stack(0xfffffffdfdf00000,0xfffffffdfe000000)]
0x0000000102677400 JavaThread "RMI RenewClean-[10.72.109.90:51744]" daemon [_thread_blocked, id=207, stack(0xfffffffdfe100000,0xfffffffdfe200000)]
0x0000000103d1c800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=206, stack(0xfffffffdfe300000,0xfffffffdfe400000)]
0x0000000103b35c00 JavaThread "UCP-worker-thread-15" daemon [_thread_blocked, id=205, stack(0xfffffffdfeb00000,0xfffffffdfec00000)]
0x0000000103d1dc00 JavaThread "UCP-worker-thread-14" daemon [_thread_blocked, id=204, stack(0xfffffffdfed00000,0xfffffffdfee00000)]
0x000000010250e800 JavaThread "UCP-worker-thread-13" daemon [_thread_blocked, id=203, stack(0xfffffffdfef00000,0xfffffffdff000000)]
0x0000000103c05400 JavaThread "UCP-worker-thread-12" daemon [_thread_blocked, id=202, stack(0xfffffffe00700000,0xfffffffe00800000)]
0x0000000101813000 JavaThread "UCP-worker-thread-24" daemon [_thread_blocked, id=201, stack(0xfffffffdff100000,0xfffffffdff200000)]
0x00000001037b7400 JavaThread "UCP-worker-thread-11" daemon [_thread_blocked, id=200, stack(0xfffffffdff300000,0xfffffffdff400000)]
0x0000000101679000 JavaThread "UCP-worker-thread-23" daemon [_thread_blocked, id=199, stack(0xfffffffe09300000,0xfffffffe09400000)]
0x00000001024a9400 JavaThread "UCP-worker-thread-22" daemon [_thread_blocked, id=198, stack(0xfffffffdff600000,0xfffffffdff700000)]
0x0000000103e53800 JavaThread "UCP-worker-thread-21" daemon [_thread_blocked, id=197, stack(0xfffffffdff800000,0xfffffffdff900000)]
0x00000001017ca000 JavaThread "UCP-worker-thread-20" daemon [_thread_blocked, id=196, stack(0xfffffffe09700000,0xfffffffe09800000)]
0x000000010376c400 JavaThread "UCP-worker-thread-19" daemon [_thread_blocked, id=195, stack(0xfffffffe0fe00000,0xfffffffe0ff00000)]
0x0000000103a81800 JavaThread "UCP-worker-thread-18" daemon [_thread_blocked, id=194, stack(0xfffffffe0c500000,0xfffffffe0c600000)]
0x000000010112e400 JavaThread "http-8080-12" daemon [_thread_in_native, id=193, stack(0xfffffffdffa00000,0xfffffffdffb00000)]
0x00000001021ad800 JavaThread "UCP-worker-thread-17" daemon [_thread_blocked, id=192, stack(0xfffffffdffc00000,0xfffffffdffd00000)]
0x0000000103062c00 JavaThread "UCP-worker-thread-16" daemon [_thread_blocked, id=191, stack(0xfffffffdffe00000,0xfffffffdfff00000)]
0x00000001039e4400 JavaThread "UCP-worker-thread-10" daemon [_thread_blocked, id=190, stack(0xfffffffe00100000,0xfffffffe00200000)]
0x0000000103107800 JavaThread "TP-Monitor" daemon [_thread_blocked, id=188, stack(0xfffffffe00c00000,0xfffffffe00d00000)]
0x00000001028e2400 JavaThread "TP-Processor4" daemon [_thread_in_native, id=187, stack(0xfffffffe00e00000,0xfffffffe00f00000)]
0x0000000100f1b000 JavaThread "TP-Processor3" daemon [_thread_blocked, id=186, stack(0xfffffffe01000000,0xfffffffe01100000)]
0x00000001041fa400 JavaThread "TP-Processor2" daemon [_thread_blocked, id=185, stack(0xfffffffe01200000,0xfffffffe01300000)]
0x00000001028e3000 JavaThread "TP-Processor1" daemon [_thread_blocked, id=184, stack(0xfffffffe01400000,0xfffffffe01500000)]
0x0000000101f84c00 JavaThread "http-8080-11" daemon [_thread_blocked, id=183, stack(0xfffffffe01600000,0xfffffffe01700000)]
0x0000000101ab8c00 JavaThread "http-8080-10" daemon [_thread_in_native, id=182, stack(0xfffffffe01800000,0xfffffffe01900000)]
0x0000000101ab8400 JavaThread "http-8080-9" daemon [_thread_blocked, id=181, stack(0xfffffffe01a00000,0xfffffffe01b00000)]
0x0000000101f85400 JavaThread "http-8080-8" daemon [_thread_in_native, id=180, stack(0xfffffffe01c00000,0xfffffffe01d00000)]
0x0000000101f84000 JavaThread "http-8080-7" daemon [_thread_in_native, id=179, stack(0xfffffffe01e00000,0xfffffffe01f00000)]
0x0000000102b28c00 JavaThread "http-8080-6" daemon [_thread_in_native, id=178, stack(0xfffffffe02000000,0xfffffffe02100000)]
0x0000000102b27c00 JavaThread "http-8080-5" daemon [_thread_in_native, id=177, stack(0xfffffffe02200000,0xfffffffe02300000)]
0x00000001012f5800 JavaThread "http-8080-4" daemon [_thread_in_native, id=176, stack(0xfffffffe02400000,0xfffffffe02500000)]
0x00000001037f8800 JavaThread "http-8080-3" daemon [_thread_in_native, id=175, stack(0xfffffffe02600000,0xfffffffe02700000)]
0x0000000103804c00 JavaThread "http-8080-2" daemon [_thread_in_native, id=174, stack(0xfffffffe02800000,0xfffffffe02900000)]
0x00000001037f7800 JavaThread "http-8080-1" daemon [_thread_in_native, id=173, stack(0xfffffffe02a00000,0xfffffffe02b00000)]
0x000000010103ac00 JavaThread "http-8080-Acceptor-0" daemon [_thread_in_native, id=172, stack(0xfffffffe02c00000,0xfffffffe02d00000)]
0x00000001012f6400 JavaThread "http-8443-Acceptor-0" daemon [_thread_in_native, id=171, stack(0xfffffffe02e00000,0xfffffffe02f00000)]
0x00000001037f7400 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=170, stack(0xfffffffe03000000,0xfffffffe03100000)]
0x0000000102147c00 JavaThread "UCP-worker-thread-9" daemon [_thread_blocked, id=169, stack(0xfffffffe03200000,0xfffffffe03300000)]
0x0000000101be6400 JavaThread "pool-1-thread-5" [_thread_blocked, id=168, stack(0xfffffffe03400000,0xfffffffe03500000)]
0x0000000101be5c00 JavaThread "pool-1-thread-4" [_thread_blocked, id=167, stack(0xfffffffe03600000,0xfffffffe03700000)]
0x0000000102108400 JavaThread "pool-1-thread-3" [_thread_blocked, id=166, stack(0xfffffffe03800000,0xfffffffe03900000)]
0x0000000101aafc00 JavaThread "pool-1-thread-2" [_thread_blocked, id=165, stack(0xfffffffe03a00000,0xfffffffe03b00000)]
0x00000001021b6400 JavaThread "pool-1-thread-1" [_thread_blocked, id=164, stack(0xfffffffe03c00000,0xfffffffe03d00000)]
0x0000000102c46400 JavaThread "collector" daemon [_thread_blocked, id=162, stack(0xfffffffe04000000,0xfffffffe04100000)]
0x0000000101ac9c00 JavaThread "UCP-worker-thread-15" daemon [_thread_blocked, id=161, stack(0xfffffffe04200000,0xfffffffe04300000)]
0x0000000103aa3400 JavaThread "UCP-worker-thread-14" daemon [_thread_blocked, id=159, stack(0xfffffffe04700000,0xfffffffe04800000)]
0x0000000102f1c400 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=158, stack(0xfffffffe04900000,0xfffffffe04a00000)]
0x0000000103c07400 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=157, stack(0xfffffffe04b00000,0xfffffffe04c00000)]
0x0000000103064000 JavaThread "javamelody mobile" daemon [_thread_blocked, id=156, stack(0xfffffffe04d00000,0xfffffffe04e00000)]
0x0000000103063800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=154, stack(0xfffffffe04f00000,0xfffffffe05000000)]
0x000000010195fc00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=153, stack(0xfffffffe05100000,0xfffffffe05200000)]
0x0000000103d59c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=152, stack(0xfffffffe05300000,0xfffffffe05400000)]
0x0000000103015000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=151, stack(0xfffffffe05500000,0xfffffffe05600000)]
0x0000000103014c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=150, stack(0xfffffffe05700000,0xfffffffe05800000)]
0x0000000102775800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=149, stack(0xfffffffe05900000,0xfffffffe05a00000)]
0x0000000102774400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=148, stack(0xfffffffe05b00000,0xfffffffe05c00000)]
0x0000000102773800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=147, stack(0xfffffffe05d00000,0xfffffffe05e00000)]
0x000000010232a400 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=146, stack(0xfffffffe05f00000,0xfffffffe06000000)]
0x00000001019de000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" [_thread_blocked, id=145, stack(0xfffffffe06100000,0xfffffffe06200000)]
0x00000001019dd400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10" [_thread_blocked, id=144, stack(0xfffffffe06300000,0xfffffffe06400000)]
0x0000000102ef2c00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9" [_thread_blocked, id=143, stack(0xfffffffe06500000,0xfffffffe06600000)]
0x0000000102ef2000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8" [_thread_blocked, id=142, stack(0xfffffffe06700000,0xfffffffe06800000)]
0x0000000102e63000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-7" [_thread_blocked, id=141, stack(0xfffffffe06900000,0xfffffffe06a00000)]
0x0000000102e62000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6" [_thread_blocked, id=140, stack(0xfffffffe06b00000,0xfffffffe06c00000)]
0x0000000102bf3400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-5" [_thread_blocked, id=139, stack(0xfffffffe06d00000,0xfffffffe06e00000)]
0x0000000102bf2c00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4" [_thread_blocked, id=138, stack(0xfffffffe06f00000,0xfffffffe07000000)]
0x0000000102bf2400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3" [_thread_blocked, id=137, stack(0xfffffffe07100000,0xfffffffe07200000)]
0x0000000103d05800 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2" [_thread_blocked, id=136, stack(0xfffffffe07300000,0xfffffffe07400000)]
0x0000000102d3b000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1" [_thread_blocked, id=135, stack(0xfffffffe07500000,0xfffffffe07600000)]
0x00000001029c2000 JavaThread "UCP-worker-thread-8" daemon [_thread_blocked, id=134, stack(0xfffffffe07700000,0xfffffffe07800000)]
0x000000010180b400 JavaThread "UCP-worker-thread-13" daemon [_thread_blocked, id=133, stack(0xfffffffe07900000,0xfffffffe07a00000)]
0x00000001026c8c00 JavaThread "UCP-worker-thread-7" daemon [_thread_blocked, id=132, stack(0xfffffffe07c00000,0xfffffffe07d00000)]
0x0000000101ecb400 JavaThread "UCP-worker-thread-12" daemon [_thread_blocked, id=131, stack(0xfffffffe0bf00000,0xfffffffe0c000000)]
0x00000001039f1800 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=130, stack(0xfffffffe07e00000,0xfffffffe07f00000)]
0x00000001039f1000 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=129, stack(0xfffffffe08000000,0xfffffffe08100000)]
0x0000000101da7400 JavaThread "UCP-worker-thread-6" daemon [_thread_blocked, id=128, stack(0xfffffffe08200000,0xfffffffe08300000)]
0x0000000101da5000 JavaThread "UCP-worker-thread-11" daemon [_thread_blocked, id=127, stack(0xfffffffe08400000,0xfffffffe08500000)]
0x000000010248f400 JavaThread "UCP-worker-thread-5" daemon [_thread_blocked, id=126, stack(0xfffffffe08600000,0xfffffffe08700000)]
0x00000001029e2000 JavaThread "UCP-worker-thread-10" daemon [_thread_blocked, id=125, stack(0xfffffffe0ef00000,0xfffffffe0f000000)]
0x0000000102d67800 JavaThread "UCP-worker-thread-4" daemon [_thread_blocked, id=124, stack(0xfffffffe08800000,0xfffffffe08900000)]
0x0000000101ab1800 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=123, stack(0xfffffffe08a00000,0xfffffffe08b00000)]
0x0000000101ab2000 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=122, stack(0xfffffffe08c00000,0xfffffffe08d00000)]
0x00000001025b5000 JavaThread "UCP-worker-thread-9" daemon [_thread_blocked, id=121, stack(0xfffffffe08f00000,0xfffffffe09000000)]
0x0000000101014800 JavaThread "UCP-worker-thread-3" daemon [_thread_blocked, id=120, stack(0xfffffffe09100000,0xfffffffe09200000)]
0x000000010182e400 JavaThread "UCP-worker-thread-8" daemon [_thread_blocked, id=118, stack(0xfffffffe09500000,0xfffffffe09600000)]
0x0000000102e9a800 JavaThread "UCP-worker-thread-2" daemon [_thread_blocked, id=116, stack(0xfffffffe09900000,0xfffffffe09a00000)]
0x0000000103d7cc00 JavaThread "Timer-3" daemon [_thread_blocked, id=115, stack(0xfffffffe09b00000,0xfffffffe09c00000)]
0x0000000102e99000 JavaThread "UCP-worker-thread-7" daemon [_thread_blocked, id=114, stack(0xfffffffe09d00000,0xfffffffe09e00000)]
0x0000000102479c00 JavaThread "UCP-worker-thread-1" daemon [_thread_blocked, id=113, stack(0xfffffffe09f00000,0xfffffffe0a000000)]
0x0000000103b61800 JavaThread "Timer-2" daemon [_thread_blocked, id=112, stack(0xfffffffe0a100000,0xfffffffe0a200000)]
0x0000000101060400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=110, stack(0xfffffffe0a300000,0xfffffffe0a400000)]
0x0000000103784c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=109, stack(0xfffffffe0a500000,0xfffffffe0a600000)]
0x0000000102c4b400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=108, stack(0xfffffffe0a700000,0xfffffffe0a800000)]
0x0000000103d4bc00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=107, stack(0xfffffffe0a900000,0xfffffffe0aa00000)]
0x0000000102b75c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=106, stack(0xfffffffe0ab00000,0xfffffffe0ac00000)]
0x0000000102d21c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=105, stack(0xfffffffe0ad00000,0xfffffffe0ae00000)]
0x0000000102b73c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=104, stack(0xfffffffe0af00000,0xfffffffe0b000000)]
0x0000000102b73000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=103, stack(0xfffffffe0b100000,0xfffffffe0b200000)]
0x0000000102b71000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=102, stack(0xfffffffe0b300000,0xfffffffe0b400000)]
0x0000000102b70c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=101, stack(0xfffffffe0b500000,0xfffffffe0b600000)]
0x0000000102b70400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=100, stack(0xfffffffe0b700000,0xfffffffe0b800000)]
0x0000000101f88400 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=99, stack(0xfffffffe0ba00000,0xfffffffe0bb00000)]
0x0000000102b2a000 JavaThread "UCP-worker-thread-6" daemon [_thread_blocked, id=98, stack(0xfffffffe0bc00000,0xfffffffe0bd00000)]
0x0000000101dee400 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=96, stack(0xfffffffe0c100000,0xfffffffe0c200000)]
0x0000000101dedc00 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=95, stack(0xfffffffe0c300000,0xfffffffe0c400000)]
0x0000000101def800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=92, stack(0xfffffffe0c700000,0xfffffffe0c800000)]
0x0000000102687c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=91, stack(0xfffffffe0c900000,0xfffffffe0ca00000)]
0x0000000103288c00 JavaThread "UCP-worker-thread-5" daemon [_thread_blocked, id=90, stack(0xfffffffe0cb00000,0xfffffffe0cc00000)]
0x0000000101ded400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=89, stack(0xfffffffe0cd00000,0xfffffffe0ce00000)]
0x0000000101f98000 JavaThread "Statistics Thread" daemon [_thread_blocked, id=88, stack(0xfffffffe0cf00000,0xfffffffe0d000000)]
0x0000000103581400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=87, stack(0xfffffffe0d100000,0xfffffffe0d200000)]
0x000000010390f800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=86, stack(0xfffffffe0d300000,0xfffffffe0d400000)]
0x000000010390e400 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=85, stack(0xfffffffe0d500000,0xfffffffe0d600000)]
0x0000000103a39400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" [_thread_blocked, id=84, stack(0xfffffffe0d700000,0xfffffffe0d800000)]
0x00000001030cb000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10" [_thread_blocked, id=83, stack(0xfffffffe0d900000,0xfffffffe0da00000)]
0x00000001040e9400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9" [_thread_blocked, id=82, stack(0xfffffffe0db00000,0xfffffffe0dc00000)]
0x00000001040e8400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8" [_thread_blocked, id=81, stack(0xfffffffe0dd00000,0xfffffffe0de00000)]
0x00000001033bf400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-7" [_thread_blocked, id=80, stack(0xfffffffe0df00000,0xfffffffe0e000000)]
0x00000001033bec00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6" [_thread_blocked, id=79, stack(0xfffffffe0e100000,0xfffffffe0e200000)]
0x00000001033be400 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-5" [_thread_blocked, id=78, stack(0xfffffffe0e300000,0xfffffffe0e400000)]
0x000000010225cc00 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4" [_thread_blocked, id=77, stack(0xfffffffe0e500000,0xfffffffe0e600000)]
0x000000010225c800 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3" [_thread_blocked, id=76, stack(0xfffffffe0e700000,0xfffffffe0e800000)]
0x000000010225c000 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2" [_thread_blocked, id=75, stack(0xfffffffe0e900000,0xfffffffe0ea00000)]
0x0000000102fdd800 JavaThread "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1" [_thread_blocked, id=74, stack(0xfffffffe0eb00000,0xfffffffe0ec00000)]
0x0000000103195800 JavaThread "UCP-worker-thread-4" daemon [_thread_blocked, id=73, stack(0xfffffffe0ed00000,0xfffffffe0ee00000)]
0x0000000102dfa800 JavaThread "UCP-worker-thread-3" daemon [_thread_blocked, id=71, stack(0xfffffffe0fc00000,0xfffffffe0fd00000)]
0x0000000101907800 JavaThread "UCP-worker-thread-2" daemon [_thread_blocked, id=69, stack(0xffffffff68200000,0xffffffff68300000)]
0x0000000101547c00 JavaThread "Timer-1" daemon [_thread_blocked, id=68, stack(0xffffffff68400000,0xffffffff68500000)]
0x0000000102115c00 JavaThread "UCP-worker-thread-1" daemon [_thread_blocked, id=67, stack(0xffffffff68600000,0xffffffff68700000)]
0x00000001017f4000 JavaThread "Timer-0" daemon [_thread_blocked, id=66, stack(0xffffffff68800000,0xffffffff68900000)]
0x0000000101ee4400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=64, stack(0xffffffff68a00000,0xffffffff68b00000)]
0x000000010170b400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=63, stack(0xffffffff68c00000,0xffffffff68d00000)]
0x00000001028c6800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=62, stack(0xffffffff68e00000,0xffffffff68f00000)]
0x00000001028c5c00 JavaThread "Statistics Thread" daemon [_thread_blocked, id=61, stack(0xffffffff69000000,0xffffffff69100000)]
0x0000000101a4e800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=60, stack(0xffffffff69200000,0xffffffff69300000)]
0x0000000101a4d400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=59, stack(0xffffffff69400000,0xffffffff69500000)]
0x0000000101de9800 JavaThread "Statistics Thread" daemon [_thread_blocked, id=58, stack(0xffffffff69600000,0xffffffff69700000)]
0x0000000102624400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=57, stack(0xffffffff69800000,0xffffffff69900000)]
0x00000001011cd400 JavaThread "Statistics Thread" daemon [_thread_blocked, id=56, stack(0xffffffff69a00000,0xffffffff69b00000)]
0x0000000102fad800 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=55, stack(0xffffffff69c00000,0xffffffff69d00000)]
0x0000000102203800 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=54, stack(0xffffffff69e00000,0xffffffff69f00000)]
0x0000000101f38c00 JavaThread "DefaultQuartzScheduler_Worker-10" [_thread_blocked, id=53, stack(0xffffffff6a000000,0xffffffff6a100000)]
0x00000001010d0c00 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=52, stack(0xffffffff6a200000,0xffffffff6a300000)]
0x00000001014d8000 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=51, stack(0xffffffff6a400000,0xffffffff6a500000)]
0x00000001019fcc00 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=50, stack(0xffffffff6a600000,0xffffffff6a700000)]
0x00000001019fc000 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=49, stack(0xffffffff6a800000,0xffffffff6a900000)]
0x0000000102a13c00 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=48, stack(0xffffffff6aa00000,0xffffffff6ab00000)]
0x00000001018c0c00 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=47, stack(0xffffffff6ac00000,0xffffffff6ad00000)]
0x00000001010d7400 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=46, stack(0xffffffff6ae00000,0xffffffff6af00000)]
0x0000000101e91800 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=45, stack(0xffffffff6b000000,0xffffffff6b100000)]
0x0000000101ff3c00 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=44, stack(0xffffffff6b200000,0xffffffff6b300000)]
0x00000001013fb800 JavaThread "RMI TCP Accept-8999" daemon [_thread_in_native, id=42, stack(0xffffffff6b600000,0xffffffff6b700000)]
0x00000001013fb000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=41, stack(0xffffffff6b800000,0xffffffff6b900000)]
0x0000000100ec6c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=40, stack(0xffffffff6c800000,0xffffffff6c900000)]
0x0000000100ec2c00 JavaThread "CompilerThread1" daemon [_thread_blocked, id=39, stack(0xffffffff6ca00000,0xffffffff6cb00000)]
0x0000000100ec1000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=38, stack(0xffffffff6cc00000,0xffffffff6cd00000)]
0x0000000100ebfc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=37, stack(0xffffffff6ce00000,0xffffffff6cf00000)]
0x0000000100ebec00 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=36, stack(0xffffffff6d000000,0xffffffff6d100000)]
0x0000000100e96000 JavaThread "Finalizer" daemon [_thread_blocked, id=35, stack(0xffffffff6d200000,0xffffffff6d300000)]
0x0000000100e8fc00 JavaThread "Reference Handler" daemon [_thread_blocked, id=34, stack(0xffffffff6d400000,0xffffffff6d500000)]
0x0000000100117000 JavaThread "main" [_thread_in_native, id=2, stack(0xffffffff7c600000,0xffffffff7c700000)]
Other Threads:
0x0000000100e8b000 VMThread [stack: 0xffffffff6d600000,0xffffffff6d700000] [id=33]
0x0000000101148400 WatcherThread [stack: 0xffffffff6b400000,0xffffffff6b500000] [id=43]
=>0x0000000100616800 (exited) GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=30]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
par new generation total 440896K, used 287771K [0xfffffffe10000000, 0xfffffffe2de60000, 0xfffffffe2de60000)
eden space 391936K, 60% used [0xfffffffe10000000, 0xfffffffe1ea00c18, 0xfffffffe27ec0000)
from space 48960K, 100% used [0xfffffffe27ec0000, 0xfffffffe2ae90000, 0xfffffffe2ae90000)
to space 48960K, 0% used [0xfffffffe2ae90000, 0xfffffffe2ae90000, 0xfffffffe2de60000)
concurrent mark-sweep generation total 4753024K, used 4396503K [0xfffffffe2de60000, 0xffffffff50000000, 0xffffffff50000000)
concurrent-mark-sweep perm gen total 393216K, used 195407K [0xffffffff50000000, 0xffffffff68000000, 0xffffffff68000000)
Dynamic libraries:
0x0000000100000000 /usr/jdk/instances/jdk1.6.0_06/bin/sparcv9/java
0xffffffff7f700000 /lib/64/libthread.so.1
0xffffffff7f200000 /usr/jdk/instances/jdk1.6.0_06/bin/sparcv9/../../jre/lib/sparcv9/jli/libjli.so
0xffffffff7f000000 /lib/64/libdl.so.1
0xffffffff7ed00000 /lib/64/libc.so.1
0xffffffff7ec00000 /platform/SUNW,Sun-Fire-T1000/lib/sparcv9/libc_psr.so.1
0xffffffff7e000000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/server/libjvm.so
0xffffffff7de00000 /lib/64/libsocket.so.1
0xffffffff7eb00000 /usr/lib/64/libsched.so.1
0xffffffff7dc00000 /lib/64/libm.so.1
0xffffffff7da00000 /usr/lib/64/libCrun.so.1
0xffffffff7d800000 /lib/64/libdoor.so.1
0xffffffff7d600000 /lib/64/libnsl.so.1
0xffffffff7d400000 /lib/64/libm.so.2
0xffffffff7d200000 /lib/64/libscf.so.1
0xffffffff7d000000 /lib/64/libuutil.so.1
0xffffffff7ce00000 /lib/64/libgen.so.1
0xffffffff7cc00000 /lib/64/libmd.so.1
0xffffffff7ca00000 /platform/SUNW,Sun-Fire-T1000/lib/sparcv9/libmd_psr.so.1
0xffffffff7c800000 /lib/64/libmp.so.2
0xffffffff7c400000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/native_threads/libhpi.so
0xffffffff7c000000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libverify.so
0xffffffff7be00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libjava.so
0xffffffff7bc00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libzip.so
0xffffffff6c600000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libmanagement.so
0xffffffff6c400000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libnet.so
0xffffffff6c200000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libj2pkcs11.so
0xffffffff6c000000 /usr/lib/sparcv9/libpkcs11.so
0xffffffff6be00000 /usr/lib/64/libcryptoutil.so.1
0xffffffff6bc00000 /usr/lib/security/64/pkcs11_kernel.so
0xffffffff6ba00000 /usr/lib/security/64/pkcs11_softtoken_extra.so
0xfffffffe0f900000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libawt.so
0xfffffffe0f700000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/headless/libmawt.so
0xfffffffe0f500000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libnio.so
0xfffffffe0f300000 /lib/64/librt.so.1
0xfffffffe0f100000 /lib/64/libaio.so.1
0xfffffffdfe600000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/librmi.so
0xfffffffdfc400000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libfontmanager.so
0xfffffffdfc000000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libdcpr.so
0xfffffffdfbe00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libcmm.so
0xfffffffdfbc00000 /usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/libjpeg.so
VM Arguments:
jvm_args: -Dcom.sun.management.jmxremote=8081 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djavax.net.ssl.keyStore=/rtiapp/apache-tomcat-6.0.16/conf/ASD-TREE-CA.keystore -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=/rtiapp/apache-tomcat-6.0.16/conf/ASD-TREE-CA.keystore -Djavax.net.ssl.trustStorePassword=changeit -Djava.util.logging.config.file=/rtiapp/apache-tomcat-6.0.16/conf/logging.properties -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xmx5120m -Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.endorsed.dirs=/rtiapp/apache-tomcat-6.0.16/endorsed -Dcatalina.base=/rtiapp/apache-tomcat-6.0.16 -Dcatalina.home=/rtiapp/apache-tomcat-6.0.16 -Djava.io.tmpdir=/rtiapp/apache-tomcat-6.0.16/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/bin
LD_LIBRARY_PATH=/usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9/server:/usr/jdk/instances/jdk1.6.0_06/jre/lib/sparcv9:/usr/jdk/instances/jdk1.6.0_06/jre/../lib/sparcv9
SHELL=/bin/sh
Signal Handlers:
SIGSEGV: [libjvm.so+0x733d70], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0x733d70], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x636530], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: [libjvm.so+0x636530], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x636530], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x639610], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x1e2028], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
--------------- S Y S T E M ---------------
OS: Solaris 10 8/07 s10s_u4wos_12b SPARC
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007
uname:SunOS 5.10 Generic_142900-02 sun4v (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 32000, AS infinity
load average:4.01 3.93 3.46
CPU:total 32 has_v8, has_v9, has_vis1, is_sun4v, is_niagara1
Memory: 8k page, physical 16646144k(8588672k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (10.0-b22) for solaris-sparc JRE (1.6.0_06-b02), built on Mar 25 2008 00:31:18 by "" with Workshop 5.8
time: Wed Mar 18 10:53:46 2015
elapsed time: 106039 seconds
REPRODUCIBILITY :
This bug can be reproduced occasionally.
CUSTOMER SUBMITTED WORKAROUND :
Have to restart the server.
SUPPORT :
YES