SEGV in MarkFromRootsClosure::scanOopsInOop

Customer encounters crash in ConcurrentMarkSweepThread when using CMS.
(dbx) where -l
  [1] libthread.so.1:__sigprocmask(0x3, 0xdc240444, 0x0), at 0xdfb66b94
  [2] libthread.so.1:_resetsig(0x0, 0x6, 0xdfb7a000, 0x0, 0x1, 0xdc240d00), at 0xdfb5d9d4
  [3] libthread.so.1:_sigon(0xdfb7a000, 0xdc2404a0, 0xdfb5f953, 0xdfb812f8, 0x6, 0x6), at 0xdfb5d2fd
  [4] libthread.so.1:_lmutex_unlock(0xdfb812f8), at 0xdfb5ac70
  [5] libthread.so.1:_thrp_kill(0xdc240d68, 0x6, 0x6, 0xdef9808d, 0xdefe08c4, 0xdfb35000), at 0xdfb5f953
  [6] libthread.so.1:_pthread_kill(0x6, 0x6), at 0xdfb5f843
  [7] libc.so.1:raise(0x6), at 0xdfadbd87
  [8] libc.so.1:abort(0xdefc8000, 0xdc2405c0, 0xdef39955, 0x1, 0x0, 0x0), at 0xdfacc14c
  [9] libjvm.so:os::abort(0x1), at 0xdeeec4ff
  [10] libjvm.so:VMError::report_and_die(0xdc2405d8, 0xdc2407bc, 0xdc240d68, 0xdefc8000, 0xb, 0xdef98562), at 0xdef39955
  [11] libjvm.so:JVM_handle_solaris_signal(0xb, 0xdc240a3c, 0xdc24083c, 0x1), at 0xdedd284b
  [12] libjvm.so:signalHandler(0xb, 0xdc240a3c, 0xdc24083c, 0xdc240828, 0xdfb65b55, 0xb), at 0xdedd20a0
  [13] libthread.so.1:__sighndlr(0xb, 0xdc240a3c, 0xdc24083c, 0xdedd2080), at 0xdfb57a8b
  ---- called from signal handler with signal 11 (SIGSEGV) ------
=>[14] libjvm.so:MarkFromRootsClosure::scanOopsInOop(0xdc240b70, 0xcb6519e8), at 0xdee17915
  [15] libjvm.so:MarkFromRootsClosure::do_bit(0xdc240b70, 0x89467a), at 0xdee1770b
  [16] libjvm.so:BitMap::iterate(0x8128bd4, 0xdc240b70, 0x0, 0x3f00000), at 0xdeddbf48
  [17] libjvm.so:CMSCollector::markFromRootsWork(0x8128b60, 0x1), at 0xdee12c79
  [18] libjvm.so:CMSCollector::markFromRoots(0x8128b60, 0x1), at 0xdee12ab7
  [19] libjvm.so:CMSCollector::collect_in_background(0x8128b60, 0x0), at 0xdee10c7c
  [20] libjvm.so:ConcurrentMarkSweepThread::run(0x8139060), at 0xdee1a2de
  [21] libjvm.so:_start(0x8139060, 0xdeeec230), at 0xdeeec302

The heap shows

Heap
 par new generation total 347328K, used 190482K [0xb4000000, 0xc9400000, 0xc9400000)
  eden space 346496K, 54% used [0xb4000000, 0xbfa04b78, 0xc9260000)
  from space 832K, 0% used [0xc9330000, 0xc9330000, 0xc9400000)
  to space 832K, 0% used [0xc9260000, 0xc9260000, 0xc9330000)
 concurrent mark-sweep generation total 176128K, used 43830K [0xc9400000, 0xd4000000, 0xd4000000)
 concurrent-mark-sweep perm gen total 81920K, used 38705K [0xd4000000, 0xd9000000, 0xd9000000)

jvm_args: -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -Xms512m -Xmx512m -verbose:gc -XX:+PrintGCTimeStamps -XX:PermSize=80m -XX:MaxPermSize=80m -XX:NewSize=340m -XX:MaxNewSize=340m -XX:SoftRefLRUPolicyMSPerMB=125 -XX:SurvivorRatio=400


###@###.### 2005-04-29 05:30:31 GMT
In Nokia we have seen a similar problem using Java 5:


Anyone any ideas why our jvm server crashed with the following stack trace ?
This is a Weblogic process, running version 9 of WL on Solaris 9 sparc.
The crashed happened in QA not PROD env.

Is this a crash from the CMS collector ? If the problem reoccurs I will open a CR.
There are no test cases which can reproduce the crash.

Many thanks,
Stefan


#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGBUS (0xa) at pc=0xfec282d8, pid=9234, tid=6
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_09-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x4282d8]
#

--------------- T H R E A D ---------------

Current thread (0x00125410): ConcurrentMarkSweepThread [id=6]

siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x00690077

Registers:
 O0=0x00000000 O1=0xc108c560 O2=0xfb17f8b4 O3=0x00000002
 O4=0x00000009 O5=0x00104f44 O6=0xfb17f850 O7=0x000038f0
 G1=0x000038f0 G2=0x61980000 G3=0x0000e3bc G4=0x000038ef
 G5=0x00000000 G6=0x00000000 G7=0xff350a00 Y=0x00000000
 PC=0xfec282d8 nPC=0xfec282dc


Top of Stack: (sp=0xfb17f850)
0xfb17f850: fb17f8b4 00000001 0069006f 00008000
0xfb17f860: feed26a8 00007000 ffffffff ff0171b4
0xfb17f870: fb17fa30 00009118 ff01407c ff014080
0xfb17f880: ff0174fc fefce000 fb17f8e0 febe40fc
0xfb17f890: 00000000 001258bc fb17f948 fec22f48
0xfb17f8a0: fb17f8c8 fb17f900 00000000 00000001
0xfb17f8b0: 00000001 ff013fb0 001b50b0 00104e50
0xfb17f8c0: 90c00000 1a000000 00104e84 00104f44

Instructions: (pc=0xfec282d8)
0xfec282c8: c8 23 60 3c 87 29 20 02 d2 00 80 03 e4 02 60 04
0xfec282d8: d6 04 a0 08 e8 02 e0 d0 9f c5 00 00 90 04 a0 08

Stack: [0xfb100000,0xfb180000), sp=0xfb17f850, free space=510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x4282d8]
V [libjvm.so+0x3e4104]
V [libjvm.so+0x421864]
V [libjvm.so+0x4215a4]
V [libjvm.so+0x41ee58]
V [libjvm.so+0x42bf2c]
V [libjvm.so+0x670740]


The thread 6 died. Its stack trace is below:
tid 6 died. Analyzing the core dump looks like
the CMS might be the problem here. Im not sure. It could be a
problem between CMS and the VM.

----------------- t@6 -----------------
0xff31fc54 _lwp_kill + 0x8
0xff2b6d50 abort + 0x100
0xfee70c1c void os::abort(int) + 0x58
0xfeeff200 void VMError::report_and_die() + 0xc84
0xfea73480 JVM_handle_solaris_signal + 0xaac
0xff385bac __sighndlr + 0xc
0xff37f804 call_user_handler + 0x234
0xff37f9b4 sigacthandler + 0x64
0xfec282d8 void MarkFromRootsClosure::scanOopsInOop(HeapWord*) + 0x174
0xfebe40fc void BitMap::iterate(BitMapClosure*,unsigned,unsigned) + 0x80
0xfec2185c int CMSCollector::markFromRootsWork(int) + 0x168
0xfec2159c int CMSCollector::markFromRoots(int) + 0x134
0xfec1ee50 void CMSCollector::collect_in_background(int) + 0x2d0
0xfec2bf24 void ConcurrentMarkSweepThread::run() + 0x4fc
0xfee70738 void*_start(void*) + 0x208
0xff385854 _lwp_start


Our VM options:


Other Threads:
  0x001b5bd8 VMThread [id=7]
  0x001cad68 WatcherThread [id=16]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 par new generation total 523840K, used 299620K [0x70c00000, 0x90c00000, 0x90c00000)
  eden space 523392K, 57% used [0x70c00000, 0x83099360, 0x90b20000)
  from space 448K, 0% used [0x90b20000, 0x90b20000, 0x90b90000)
  to space 448K, 0% used [0x90b90000, 0x90b90000, 0x90c00000)
 concurrent mark-sweep generation total 1572864K, used 1040001K [0x90c00000, 0xf0c00000, 0xf0c00000)
 concurrent-mark-sweep perm gen total 131072K, used 74795K [0xf0c00000, 0xf8c00000, 0xf8c00000)


VM Arguments:
jvm_args: -XX:+UseConcMarkSweepGC -Xms2048m -Xmx2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=128m -XX:MaxPermSize=128m -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0 -Dcommerce.properties=/opt/nokiacif/app/swd/current/config/swdDomain/properties/weblogiccommerce.properties -DmasterReference=/opt/nok
iacif/app/swd/current/config/swdDomain/properties/master.properties -Djava.awt.headless=true -Dweblogic.Name=swdServer2 -Dbea.home=/opt/nokiacif/bea9 -Dweblogic.man
agement.username=system -Dweblogic.management.password=weblogic -Dweblogic.ProductionModeEnabled=true -Dweblogic.management.discover=true -Djavax.xml.soap.MessageFa
ctory=com.sun.xml.messaging.saaj.soap.ver1_1.SOAPMessageFactory1_1Impl -Dweblogic.management.server=wl-swd-admin:7110 -Djava.security.policy=/opt/nokiacif/bea9/webl
ogic91/server/lib/weblogic.policy
java_command: weblogic.Server
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=/opt/nokiacif/app/swd/current/config/swdDomain/autodeploy/FDPNotify/WEB-INF/lib/saaj.jar:/opt/nokiacif/bea9/weblogic91/server/lib/weblogic.jar:/opt/nokiac
if/bea9/weblogic91/server/lib/webservices.jar::/opt/nokiacif/bea9/jdk1.5.0_09/lib/tools.jar:::/opt/nokiacif/app/swd/current/config/swdDomain/autodeploy/SWD/APP-INF/
lib/NCP.jar:/opt/nokiacif/app/swd/current/config/swdDomain/autodeploy/SWD/APP-INF/lib/nols_common.jar:/opt/nokiacif/app/swd/current/config/swdDomain/autodeploy/SWD/
APP-INF/lib/commons-lang-2.1.jar:/opt/nokiacif/app/swd/current/config/swdDomain/autodeploy/SWD/APP-INF/lib/commons-httpclient-2.0.jar:/opt/nokiacif/bea9/weblogic91/
server/lib/consoleapp/APP-INF/lib/commons-logging.jar:/opt/nokiacif/app/swd/current/config/swdDomain/autodeploy/SWD/APP-INF/lib/jdom.jar:/opt/nokiacif/bea9/weblogic
91/server/lib/consoleapp/APP-INF/lib/log4j.jar
PATH=/opt/nokiacif/bea9/weblogic91/server/bin/oci920_8:/opt/nokiacif/bea9/weblogic91/server/bin:/opt/nokiacif/bea9/jdk1.5.0_09/bin:/usr/bin::/opt/stlbox/bin:/usr/lo
cal/bin:/usr/ucb/bin:/opt/misctool/openssh/bin:/opt/misctool/sudo/bin
LD_LIBRARY_PATH=/opt/nokiacif/bea9/jdk1.5.0_09/jre/lib/sparc/server:/opt/nokiacif/bea9/jdk1.5.0_09/jre/lib/sparc:/opt/nokiacif/bea9/jdk1.5.0_09/jre/../lib/sparc:/op
t/nokiacif/bea9/weblogic91/server/lib/solaris:/opt/nokiacif/bea9/weblogic91/server/lib/solaris/oci920_8
SHELL=/bin/ksh
HOSTTYPE=sparc
OSTYPE=solaris2.9
MACHTYPE=sparc-sun-solaris2.9

Signal Handlers:
SIGSEGV: [libjvm.so+0x6ff5b8], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x6ff5b8], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x27299c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x27299c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x27299c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0x672ca4], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0x27299c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0x67191c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x67191c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x67191c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x67191c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004


--------------- S Y S T E M ---------------

OS: Solaris 9 9/04 s9s_u7wos_09 SPARC
           Copyright 2004 Sun Microsystems, Inc. All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 29 June 2004

uname:SunOS 5.9 Generic_118558-30 sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 8192, AS infinity
load average:0.71 0.19 0.09

CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

Memory: 8k page, physical 16777216k(10392456k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_09-b03) for solaris-sparc, built on Oct 12 2006 01:30:08 by unknown with unknown Workshop:0x550
Another customer (Intralinks) has run into the same issue with Java 5.
Actually, they see it with a 5.0u11-based FVB for CR 6492085 and
CR 6518092 provided by Java sustaining. We found no connection of
the newly observed crash to the original fix.

I recommended to use -XX:-CMSParallelRemarkEnabled option as a workaround
and asked to try to reproduce the crash with fastdebug version and
-Verify{Before,After,During}GC options specified.