kill -3 causes the JVM to core dump

8 in _cond_reltimedwait_cancel()
 
 t@40 _start() sleep on 0x367450 in _cond_wait_cancel()
 
 t@41 a l@12 _start() sleep on 0x4cba38 in __lwp_sema_wait()
 
 t@42 _start() sleep on 0x4cafe0 in _cond_wait_cancel()
 
 t@43 _start() sleep on 0x362c78 in _cond_wait_cancel()
 
 t@44 _start() sleep on 0x362260 in _cond_wait_cancel()
 
 t@45 _start() sleep on 0x48e448 in _cond_wait_cancel()
 
 t@46 _start() sleep on 0x3683f0 in _cond_wait_cancel()
 
 t@47 _start() sleep on 0x490380 in _cond_wait_cancel()
 
 t@48 _start() sleep on 0x48fc40 in _cond_wait_cancel()
 
 t@49 _start() sleep on 0x4b21a8 in _cond_wait_cancel()
 
 t@50 _start() sleep on 0x4b2f30 in _cond_wait_cancel()
 
 t@51 _start() sleep on 0x5ded48 in _cond_wait_cancel()
 
 t@52 _start() sleep on 0x5dfad0 in _cond_wait_cancel()
 
 t@53 _start() sleep on 0x5ce1b0 in _cond_wait_cancel()
 
 t@54 _start() sleep on 0x51afa0 in _cond_wait_cancel()
 
 t@55 _start() sleep on 0x51aca0 in _cond_wait_cancel()
 
 t@56 _start() sleep on 0x536f10 in _cond_wait_cancel()
 
 t@57 _start() sleep on 0x580c18 in _cond_reltimedwait_cancel()
 
 t@58 _start() sleep on 0x589900 in _cond_wait_cancel()
 
 t@59 _start() sleep on 0x58a030 in _cond_wait_cancel()
 
 t@60 _start() sleep on 0x5b2710 in _cond_wait_cancel()
 
 t@61 _start() sleep on 0x5b1c28 in _cond_wait_cancel()
 
 t@62 _start() sleep on 0x5b3220 in _cond_wait_cancel()
 
 t@63 _start() sleep on 0x5b3fa8 in _cond_wait_cancel()
 
 t@64 _start() sleep on 0x5b4d30 in _cond_wait_cancel()
 
 t@65 _start() sleep on 0x5b5ab8 in _cond_wait_cancel()
 
 t@66 _start() sleep on 0x5b6840 in _cond_wait_cancel()
 
 t@67 _start() sleep on 0x5b75c8 in _cond_wait_cancel()
 
 t@68 _start() sleep on 0x5b8350 in _cond_wait_cancel()
 
 t@69 _start() sleep on 0x49bcf0 in _cond_wait_cancel()
 
 t@70 _start() sleep on 0x49ca78 in _cond_wait_cancel()
 
 t@71 _start() sleep on 0x49d800 in _cond_wait_cancel()
 
 t@72 _start() sleep on 0x49e588 in _cond_wait_cancel()
Customer is using BEA weblogic 6.0 and JDK 1.3.1_05

the application starts up and the the kill -3 is done and they get the thread dump they are expecting. When the application gets to a point where a long running transaction is going on, a kill -3 is issued to get a thread dump so they can see where they are and it cores.

Now what may be something interesting only to me is that I've see problems like this before but the thread in the stack trace goes with a SIG 10 or something else, but in this stack trace you see the SIGQUIT.

there was a bug# 4485006 that looked similar, but this still happens with 1.3.l_05 with both the -server and -client.

They are not using -Xrs flag, file desciptors are set to 1024
the flags they use are :
  -verbose:gc -classpath $CLASSPATH -Djava.io.tmpdir=$domain_dir/tmp -Dweblogic.Domain=$domain_name -Dweblogic.Name=eagle -Dbea.home=$weblogic_root -Dweblogic.management.password=$WLS_PW -Dweblogic.ProductionModeEnabled=$STARTMODE -Djava.security.policy==$WL_HOME/lib/weblogic.policy -Dcovad.util.properties.file=$app_dir_srv/$srv_name/covad.properties -Dcovad.ejbfinder.connectionPool=bometaPool -Dcovad.dbpool.tx=bometaPool -Dcovad.dbpool.jdbc=txPool -Dcovad.home=$app_dir_srv/$srv_name -Dcovad.tt.debug=true weblogic.Server
There is no error log file that we can find.


gladstone% psig 15440
 15440: /opt/jdk-1.3.1_05/bin/sparc/native_threads/java_g -client -Did=bome
 
QUIT caught RESTART,SIGINFO HUP,INT,QUIT,ILL,TRAP,ABRT,EMT,FPE,BUS,SEGV,SYS,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,CONT,TTIN,TTOU,VTALRM,PROF,XCPU,XFSZ,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
 
gladstone% kill -3 15440
 (this produced a core dump)


detected a multithreaded program
 
t@1 (l@8) terminated by signal ABRT (Abort)
 
0xff369790: __sigprocmask+0x0008: jmp %o7 + 0x8
 
dbx: warning: can't find file "/export/d0/jdk131-update/jdk/ws/make/java/main/java/../../../../build/solaris-sparc/tmp/java/java/obj_g/java.o"
 
dbx: warning: see `help finding-files'
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/os_solaris.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/debug.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/mutex.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/threadLocalStorage.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/handles.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/sharedRuntime.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/javaCalls.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/jvm.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/reflection.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/jdk/ws/make/java/java/../../../build/solaris-sparc/tmp/java/java.lang/java/obj_g/Method.o"
 
dbx: warning: can't find file "/export/d0/jdk131-update/hotspot/ws/solaris-sparc/hotspot1.3.1fcs/build/solaris/solaris_sparc_compiler1/jvmg/jni.o"
 
(/opt/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx) where
 
current thread: t@1
 
=>[1] __sigprocmask(0x0, 0xffbea078, 0x0, 0x0, 0x0, 0x0), at 0xff369790
 
 [2] _resetsig(0xff36bf68, 0x0, 0x0, 0x25850, 0xff37e000, 0x0), at 0xff35e9ac
 
 [3] _sigon(0x25850, 0xff385938, 0x6, 0xffbea14c, 0x25850, 0x6), at 0xff35e14c
 
 [4] _thrp_kill(0x0, 0x1, 0x6, 0xff37e000, 0x1, 0xff2be460), at 0xff36118c
 
 [5] raise(0x6, 0x0, 0x0, 0xffffffff, 0xff2be3cc, 0x4), at 0xff24b738
 
 [6] abort(0xff2ba004, 0xffbea2a8, 0x0, 0xfffffff8, 0x4, 0xffbea2c9), at 0xff235aac
 
 [7] os::abort(0x1, 0xfe552fd8, 0xffbeab4c, 0x100, 0x0, 0x258e4), at 0xfe0c5878
 
 [8] report_error(0x1, 0xfe5e6cdc, 0xd5, 0xfe552e28, 0xfe552e3c, 0xfe5e6c64), at 0xfdda4920
 
 [9] report_assertion_failure(0xfe5e6c64, 0xfe5e6cdc, 0xd5, 0xfe5e6d40, 0x0, 0x0), at 0xfdda3cf0
 
 [10] Mutex::check_prelock_state(0x3a690, 0x3b490, 0x7e200, 0x0, 0x0, 0x0), at 0xfe05c578
 
 [11] Mutex::lock(0x3a690, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfe05b580
 
 [12] MutexLocker::MutexLocker(0xffbeae74, 0x3a690, 0x0, 0x0, 0x0, 0x0), at 0xfe0ce308
 
 [13] os::signal_notify(0x3, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfe0c67a0
 
 [14] UserHandler(0x3, 0xffbeb340, 0xffbeb088, 0x0, 0x0, 0x0), at 0xfe0c64d4
 
 [15] __sighndlr(0x3, 0xffbeb340, 0xffbeb088, 0xfe0c64a0, 0x258f4, 0x258e4), at 0xff36b82c
 
 [16] sigacthandler(0x3, 0x25850, 0x0, 0x0, 0x0, 0xff37e000), at 0xff368504
 
 ---- called from signal handler with signal 3 (SIGQUIT) ------
 
 [17] ..rem(0x7e202, 0x80, 0x7e200, 0xf443a1d8, 0x0, 0x20), at 0xff2beeb4
 
 [18] ThreadLocalStorage::get_thread_via_cache(0x3b490, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfe208060
 
 [19] ThreadLocalStorage::thread(0x6, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfde1b814
 
 [20] Thread::current(0xffbeb5f0, 0x1, 0x0, 0x0, 0x0, 0x0), at 0xfde1ad24
 
 [21] NoHandleMark::~NoHandleMark(0xffbeb604, 0x3a4b0689, 0x7fa06e53, 0x7fa06e53, 0x0, 0x0), at 0xfde1aa4c
 
 [22] SharedRuntime::lmul(0x0, 0x7fa06e53, 0x0, 0x38fd4433, 0x0, 0x0), at 0xfe151810
 
 [23] 0xfa827bf8(0xf443a1d8, 0xf4434038, 0x12, 0x20, 0xf, 0x0), at 0xfa827bf7
 
 [24] 0xfa8299d4(0xf443a1d8, 0xf4434038, 0x20, 0xf7045f55, 0x5159d842, 0x974c8707), at 0xfa8299d3
 
 [25] 0xfa8cf9e4(0xf4434290, 0xf4434038, 0x1e, 0xf443a1d8, 0xf4439460, 0x20), at 0xfa8cf9e3
 
 [26] 0xa1754(0xf4434290, 0xf85d64f0, 0x0, 0xbb7b0, 0xf4600000, 0x0), at 0xa1753
 
 [27] 0xa1754(0xf4438168, 0xf85dbd38, 0xf44386c0, 0xbb39c, 0xf4600000, 0x0), at 0xa1753
 
 [28] 0xa1754(0xf4438168, 0x3, 0xffbebb00, 0xbb5b0, 0xf4600000, 0x0), at 0xa1753
 
 [29] 0xa1754(0xf4438168, 0x0, 0xffbebba4, 0xbb39c, 0xf84f7f28, 0x0), at 0xa1753
 
 [30] 0xa1754(0xf4d1a9f8, 0xffbebc30, 0xffbebc34, 0xbb5b0, 0xf44318c8, 0x0), at 0xa1753
 
 [31] 0xa1864(0xf4d1a9f8, 0x0, 0x0, 0xbb7b0, 0xf4600000, 0x0), at 0xa1863
 
 [32] 0xa1754(0xf442db90, 0x0, 0xffbebd84, 0xbb5b0, 0x0, 0x0), at 0xa1753
 
 [33] 0xa1754(0x3b490, 0xb8, 0xf8441e60, 0xbb9b0, 0xf442dc58, 0x1), at 0xa1753
 
 [34] 0xa1754(0x3b490, 0xb8, 0xffbebe28, 0xb6268, 0x0, 0x0), at 0xa1753
 
 [35] 0xa1864(0xf4424c50, 0x3b490, 0x8, 0xb62c0, 0xffbec120, 0x0), at 0xa1863
 
 [36] 0xfe77c364(0xffbebf3c, 0xfe7395c0, 0xa, 0xf8eeb578, 0xa4134, 0xffbec114), at 0xfe77c363
 
 [37] JavaCalls::call_helper(0xffbec110, 0xfe77c2e0, 0xffbec108, 0x3b490, 0x0, 0x3ba58), at 0xfdec1774
 
 [38] os::os_exception_wrapper(0xfdec1028, 0xfe7395b8, 0xffbec148, 0xffbec108, 0x3b490, 0xffffffff), at 0xfe0c969c
 
 [39] JavaCalls::call(0xfe7395b8, 0xffbec148, 0xffbec108, 0x3b490, 0xffbec110, 0x1), at 0xfdec0fbc
 
 [40] JavaCalls::call_default_constructor(0x3b490, 0xffbec244, 0xffbec240, 0x3b490, 0x1, 0xffbec1b4), at 0xfdebfde0
 
 [41] JVM_NewInstance(0x3b530, 0xffbec3a4, 0x8, 0x18, 0x3b490, 0x0), at 0xfdf4cf24
 
 [42] 0xa5574(0xf8eed9f0, 0xffbec424, 0xffbec428, 0xffffffff, 0x0, 0x0), at 0xa5573
 
 [43] 0xa1754(0xf8eed9f0, 0xb6, 0xffbec4b0, 0xbb7b0, 0x0, 0x0), at 0xa1753
 
 [44] 0xa1754(0xf441cfc0, 0xb7, 0xffbec52c, 0xb6268, 0x0, 0x0), at 0xa1753
 
 [45] 0xa1754(0xf441cfc0, 0xb7, 0x54, 0xb6018, 0x0, 0x0), at 0xa1753
 
 [46] 0xa1864(0x3b490, 0xb8, 0xffbec6e4, 0xb6018, 0xf8441e60, 0x0), at 0xa1863
 
 [47] 0xa1754(0x3b490, 0xb8, 0xffbec768, 0xb6268, 0xf4b4bda0, 0x0), at 0xa1753
 
 [48] 0xa1754(0xf4f7f938, 0xa4134, 0xffbec7f0, 0xb6268, 0xb9, 0x0), at 0xa1753
 
 [49] 0xa1b0c(0xf4ecffb8, 0xf4f00bd0, 0xffbec878, 0xbbba0, 0x0, 0x0), at 0xa1b0b
 
 [50] 0xa1754(0xf4ecffb8, 0xb7, 0x26, 0xb5dc8, 0x0, 0x0), at 0xa1753
 
 [51] 0xa1864(0xf4ecffb8, 0xb7, 0xffbec98c, 0xb6018, 0x0, 0x0), at 0xa1863
 
 [52] 0xa1864(0xf4ecffb8, 0x3b490, 0x8, 0xb6018, 0xffbecd58, 0x0), at 0xa1863
 
 [53] 0xfe77c364(0xffbeca1c, 0xffbecd38, 0xa, 0xf8861f40, 0xa43d8, 0xffbecd4c), at 0xfe77c363
 
 [54] JavaCalls::call_helper(0xffbecd48, 0xfe77c2e0, 0xffbecd40, 0x3b490, 0x0, 0x0), at 0xfdec1774
 
 [55] os::os_exception_wrapper(0xfdec1028, 0xffbecd30, 0xffbeccc4, 0xffbecd40, 0x3b490, 0x0), at 0xfe0c969c
 
 [56] JavaCalls::call(0xffbecd30, 0xffbeccc4, 0xffbecd40, 0x3b490, 0xf8441e60, 0x0), at 0xfdec0fbc
 
 [57] Reflection::invoke(0xffbece34, 0xffbece30, 0xffbece2c, 0x0, 0xffbece28, 0xe), at 0xfe0fa190
 
 [58] Reflection::invoke_method(0xffbece24, 0x1, 0xffbeced0, 0x3b490, 0xf4ee5ee0, 0x0), at 0xfe0fa6d4
 
 [59] JVM_InvokeMethod(0x3b530, 0xffbed06c, 0xffbed068, 0xffbed064, 0xf4e99988, 0x0), at 0xfdf5ca1c
 
 [60] Java_java_lang_reflect_Method_invoke(0x3b530, 0xffbed06c, 0xffbed068, 0xffbed064, 0xf4600000, 0x0), at 0xfdbbd2f8
 
 [61] 0xa5574(0xf4e70a38, 0xffbed0fc, 0xffbed100, 0xb4b2c, 0xfffffff8, 0x0), at 0xa5573
 
 [62] 0xa1754(0xf4ecffb8, 0x0, 0xffbed198, 0xbb9b0, 0xf8845e78, 0x0), at 0xa1753
 
 [63] 0xa1754(0xf4ecffb8, 0xf86c3c98, 0xffbed230, 0xbb39c, 0xf86c3c98, 0x0), at 0xa1753
 
 [64] 0xa1754(0xf4ecffb8, 0xa4134, 0xffbed2d4, 0xbb7b0, 0xf8441e60, 0x0), at 0xa1753
 
 [65] 0xa19fc(0xf4e13b78, 0xf4ed0760, 0xffbed364, 0xbbba0, 0xf8487e80, 0x0), at 0xa19fb
 
 [66] 0xa1754(0xf4e13b78, 0xa4134, 0x39, 0xbb39c, 0xf4ee5ee0, 0x0), at 0xa1753
 
 [67] 0xa19fc(0xf4ee3038, 0xf4ef99e0, 0x0, 0xbbba0, 0xf864ad80, 0xffbed34c), at 0xa19fb
 
 [68] 0xfa913d48(0xf4ee3038, 0xf4ef72f8, 0xf4ef36c8, 0x0, 0xb9, 0x0), at 0xfa913d47
 
 [69] 0xa19fc(0xf4ef72f8, 0xa4134, 0xffbed5c4, 0xb64b8, 0xb9, 0x0), at 0xa19fb
 
 [70] 0xa1b0c(0xf4ef0d78, 0xa4134, 0x51, 0xb64b8, 0xf8845e78, 0x0), at 0xa1b0b
 
 [71] 0xa1b0c(0xf4ed0450, 0xb7, 0xffbed6dc, 0xb64b8, 0x0, 0x0), at 0xa1b0b
 
 [72] 0xa1864(0xf4ed0450, 0x3b490, 0x8, 0xb6018, 0xffbedaa8, 0x0), at 0xa1863
 
 [73] 0xfe77c364(0xffbed76c, 0xffbeda88, 0xa, 0xf88454a0, 0xa4134, 0xffbeda9c), at 0xfe77c363
 
 [74] JavaCalls::call_helper(0xffbeda98, 0xfe77c2e0, 0xffbeda90, 0x3b490, 0x0, 0x0), at 0xfdec1774
 
 [75] os::os_exception_wrapper(0xfdec1028, 0xffbeda80, 0xffbeda14, 0xffbeda90, 0x3b490, 0x0), at 0xfe0c969c
 
 [76] JavaCalls::call(0xffbeda80, 0xffbeda14, 0xffbeda90, 0x3b490, 0xf8441e60, 0x0), at 0xfdec0fbc
 
 [77] Reflection::invoke(0xffbedb84, 0xffbedb80, 0xffbedb7c, 0x0, 0xffbedb78, 0xe), at 0xfe0fa190
 
 [78] Reflection::invoke_method(0xffbedb74, 0x1, 0xffbedc20, 0x3b490, 0xf4f53a48, 0x0), at 0xfe0fa6d4
 
 [79] JVM_InvokeMethod(0x3b530, 0xffbeddbc, 0xffbeddb8, 0xffbeddb4, 0x0, 0x0), at 0xfdf5ca1c
 
 [80] Java_java_lang_reflect_Method_invoke(0x3b530, 0xffbeddbc, 0xffbeddb8, 0xffbeddb4, 0xf4600000, 0x0), at 0xfdbbd2f8
 
 [81] 0xa5574(0xf4e39290, 0xffbede4c, 0xffbede50, 0xb4b2c, 0xfffffff8, 0x0), at 0xa5573
 
 [82] 0xa1754(0xf4ed0450, 0x0, 0xffbedee8, 0xbb9b0, 0x0, 0x0), at 0xa1753
 
 [83] 0xa1754(0xf4ed0450, 0xf86c3c98, 0xffbedf80, 0xbb39c, 0xf86c3c98, 0x0), at 0xa1753
 
 [84] 0xa1754(0xf4ed0450, 0xa4134, 0xffbee024, 0xbb7b0, 0xf8441e60, 0x0), at 0xa1753
 
 [85] 0xa19fc(0xf4e13b78, 0xf4ed1988, 0xffbee0b4, 0xbbba0, 0xf8487e80, 0x0), at 0xa19fb
 
 [86] 0xa1754(0xf4e13b78, 0xa4134, 0x39, 0xbb39c, 0xf4f53a48, 0x0), at 0xa1753
 
 [87] 0xa19fc(0xf4f2e6f8, 0xf864ad80, 0xffbee210, 0xbbba0, 0xf864ad80, 0x0), at 0xa19fb
 
 [88] 0xa1754(0xf4f2e6f8, 0xa4134, 0xffbee29c, 0xbb7b0, 0xb9, 0x0), at 0xa1753
 
 [89] 0xa19fc(0xf4f2e568, 0xa4134, 0xffbee314, 0xb64b8, 0xb9, 0x0), at 0xa19fb
 
 [90] 0xa1b0c(0xf4f15b50, 0xa4134, 0xffbee3a0, 0xb64b8, 0x0, 0x0), at 0xa1b0b
 
 [91] 0xa1b0c(0x3b490, 0xb8, 0xc2, 0xbb7b0, 0x0, 0x0), at 0xa1b0b
 
 [92] 0xa1864(0x3b490, 0xb8, 0xffbee4cc, 0xb3274, 0x0, 0x0), at 0xa1863
 
 [93] 0xa1864(0xf4b5b9d0, 0xb6, 0x9c, 0xb5dc8, 0x0, 0x0), at 0xa1863
 
 [94] 0xa1864(0x3b490, 0xb8, 0xffbee5dc, 0xb5dc8, 0x0, 0x0), at 0xa1863
 
 [95] 0xa1754(0xf44bb890, 0x3b490, 0x8, 0xb6268, 0xffbee870, 0x309098), at 0xa1753
 
 [96] 0xfe77c364(0xffbee66c, 0xffbee948, 0xa, 0xf84d16f0, 0xa4134, 0xffbee864), at 0xfe77c363
 
 [97] JavaCalls::call_helper(0xffbee860, 0xfe77c2e0, 0xffbee858, 0x3b490, 0x5, 0x10), at 0xfdec1774
 
 [98] os::os_exception_wrapper(0xfdec1028, 0xffbee940, 0xffbee840, 0xffbee858, 0x3b490, 0xff0000), at 0xfe0c969c
 
 [99] JavaCalls::call(0xffbee940, 0xffbee840, 0xffbee858, 0x3b490, 0xffbee8a4, 0x1), at 0xfdec0fbc
 
 [100] jni_invoke(0x3b530, 0xffbee940, 0x0, 0x0, 0x306870, 0xffbee924), at 0xfdee1b98
 
(/opt/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx) threads
 
o> t@1 a l@8 ?() signal SIGABRT in __sigprocmask()
 
 t@2 b l@2 ?() LWP suspended in __signotifywait()
 
 t@3 ?() sleep on (unknown) in _reap_wait()
 
 t@4 a l@14 _start() sleep on 0x3ade8 in __lwp_sema_wait()
 
 t@5 b l@4 _co_timerset() LWP suspended in __lwp_sema_wait()
 
 t@6 a l@15 _start() sleep on 0xea4e0 in __lwp_sema_wait()
 
 t@7 a l@13 _start() sleep on 0xec630 in __lwp_sema_wait()
 
 t@8 a l@5 _start() LWP suspended in _poll()
 
 t@9 _start() sleep on 0x3b1e8 in _cond_wait_cancel()
 
 t@10 _start() sleep on 0x3a718 in _cond_wait_cancel()
 
 t@11 a l@7 _start() sleep on 0x3a0d8 in __lwp_sema_wait()
 
 t@12 a l@1 _start() LWP suspended in _poll()
 
 t@13 _start() sleep on 0x46f9b0 in _cond_wait_cancel()
 
 t@14 a l@11 _start() sleep on 0x46dfb0 in __lwp_sema_wait()
 
 t@15 _start() sleep on 0x3a69e8 in _cond_wait_cancel()
 
 t@16 _start() sleep on 0x3aafd0 in _cond_wait_cancel()
 
 t@17 _start() sleep on 0x3abd58 in _cond_wait_cancel()
 
 t@18 _start() sleep on 0x3acae0 in _cond_reltimedwait_cancel()
 
 t@19 a l@9 _start() sleep on 0x3ad868 in __lwp_sema_wait()
 
 t@20 a l@3 _start() sleep on 0x3a7840 in __lwp_sema_wait()
 
 t@21 _start() sleep on 0x3a85c8 in _cond_wait_cancel()
 
 t@22 _start() sleep on 0x3a9350 in _cond_wait_cancel()
 
 t@23 _start() sleep on 0x3aa0d8 in _cond_wait_cancel()
 
 t@24 _start() sleep on 0x491178 in _cond_wait_cancel()
 
 t@25 _start() sleep on 0x491f00 in _cond_wait_cancel()
 
 t@26 _start() sleep on 0x492c88 in _cond_wait_cancel()
 
 t@27 _start() sleep on 0x493a10 in _cond_wait_cancel()
 
 t@28 _start() sleep on 0x494798 in _cond_wait_cancel()
 
 t@29 _start() sleep on 0x495520 in _cond_wait_cancel()
 
 t@30 _start() sleep on 0x4962a8 in _cond_wait_cancel()
 
 t@31 _start() sleep on 0x497030 in _cond_wait_cancel()
 
 t@32 _start() sleep on 0x497db8 in _cond_wait_cancel()
 
 t@33 a l@10 _start() LWP suspended in _poll()
 
 t@34 a l@6 _start() LWP suspended in _poll()
 
 t@35 _start() sleep on 0x4c7228 in _cond_wait_cancel()
 
 t@36 _start() sleep on 0x4c7fb0 in _cond_wait_cancel()
 
 t@37 _start() sleep on 0x4c9140 in _cond_wait_cancel()
 
 t@38 _start() sleep on 0x46d190 in _cond_wait_cancel()
 
 t@39 _start() sleep on 0x4cbde