FULL PRODUCT VERSION :
java version "1.4.2_05"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04)
Java HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode)


FULL OS VERSION :
SunOS 5.9 Generic_117171-02 sun4u sparc SUNW,Sun-Fire-880

A DESCRIPTION OF THE PROBLEM :
This particular code that we're running starts up 1 thread (so we have main and this new thread). The thread reads data from several files and puts the data into an object that is shared by both threads. While this is happening, the main thread is reading the data from that shared object.

Now, we consistently get a hotspot error in the same general location of the code. We successfully ran this against 30 other sets of data files, but for this one set of data files we always get the hotspot error and always in the same general area of our code.

I attempted to add some printlns to narrow this down to any data of our that may be causing a problem, but it seems the printlns are affecting the code enough that it can cause the process to work instead of generating the hotspot error. But always once I remove the printlns, I get the hotspot error again.



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
I can't really say how to reproduce, except to run our code against our data.

Here's the command line used:

/sysarch/bison/mhansen/j2sdk1.4.2_05/bin/java -d64 -Xmx14000m -Xms1200
0m -XX:NewSize=4200m -XX:MaxNewSize=4200m -XX:SurvivorRatio=2 -XX:+UseParNewGC -XX:+ShowMessageBoxOnError -Djava.util.logging.config.class=com.navtech.util.LoggerConfig -Dpropsfile=/tmp/propsfile.8220 -jar /vobs/bison_core/lib/vsamtordb.jar



ERROR MESSAGES/STACK TRACES THAT OCCUR :

Unexpected Signal : 11 occurred at PC=0xFFFFFFFF7E631818
Function=[Unknown. Nearest: JVM_Clone+0x80B0]
Library=/sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/server/libjvm.so

Current Java thread:

Dynamic libraries:
0x100000000 /sysarch/bison/mhansen/j2sdk1.4.2_05/bin/sparcv9/java
0xffffffff7f300000 /usr/lib/64/libthread.so.1
0xffffffff7f200000 /usr/lib/64/libdl.so.1
0xffffffff7ef00000 /usr/lib/64/libc.so.1
0xffffffff7f500000 /usr/platform/SUNW,Sun-Fire-880/lib/sparcv9/libc_psr.so.1
0xffffffff7e400000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/server/
libjvm.so
0xffffffff7e200000 /usr/lib/64/libCrun.so.1
0xffffffff7e000000 /usr/lib/64/libsocket.so.1
0xffffffff7de00000 /usr/lib/64/libnsl.so.1
0xffffffff7dc00000 /usr/lib/64/libm.so.1
0xffffffff7da00000 /usr/lib/64/libsched.so.1
0xffffffff7ed00000 /usr/lib/64/libw.so.1
0xffffffff7d600000 /usr/lib/64/libmp.so.2
0xffffffff7d400000 /usr/lib/64/librt.so.1
0xffffffff7d200000 /usr/lib/64/libaio.so.1
0xffffffff7d000000 /usr/lib/64/libmd5.so.1
0xffffffff7cd00000 /usr/platform/SUNW,Sun-Fire-880/lib/sparcv9/libmd5_psr.so.1
0xffffffff7c400000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/native_
threads/libhpi.so
0xffffffff7c100000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/libveri
fy.so
0xffffffff7bf00000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/libjava
.so
0xffffffff7bc00000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/libzip.
so
0xfffffffbc5100000 /usr/lib/locale/en_US.ISO8859-1/sparcv9/en_US.ISO8859-1.so.2


Heap at VM Abort:
Heap
 par new generation total 3225600K, used 2195225K [0xfffffffbc8800000, 0xfffff
ffccf000000, 0xfffffffccf000000)
  eden space 2150400K, 82% used [0xfffffffbc8800000, 0xfffffffc35168098, 0xffff
fffc4bc00000)
  from space 1075200K, 38% used [0xfffffffc4bc00000, 0xfffffffc6525e3e0, 0xffff
fffc8d600000)
  to space 1075200K, 0% used [0xfffffffc8d600000, 0xfffffffc8d600000, 0xffff
fffccf000000)
 tenured generation total 7987200K, used 4973332K [0xfffffffccf000000, 0xfffff
ffeb6800000, 0xffffffff33800000)
   the space 7987200K, 62% used [0xfffffffccf000000, 0xfffffffdfe8c5130, 0xffff
fffdfe8c5200, 0xfffffffeb6800000)
 compacting perm gen total 16384K, used 7352K [0xffffffff33800000, 0xffffffff34
800000, 0xffffffff37800000)
   the space 16384K, 44% used [0xffffffff33800000, 0xffffffff33f2e1a8, 0xffffff
ff33f2e200, 0xffffffff34800000)

Local Time = Mon Aug 16 12:04:59 2004
Elapsed Time = 9996
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.4.2_05-b04 mixed mode)
#
--------------------
INFO: Done performing Merge of Admin AREA and CARTO Boundaries
        [VsamServer] [tstamp: 3947 time: 3944.8 (1:05:44) split: 984.5 (16:24)] [Memory: 4.777G total: 7.759G free: 2.982G]
INFO: 7600000 packets. Current packet is class com.navtech.data.packet.CartoPathPacket
        [Loader] [tstamp: 3987 time: 3985.8 (1:06:25) split: 60.8 (01:00)] [Memory: 4.973G total: 7.759G free: 2.787G]
INFO: 7650000 packets. Current packet is class com.navtech.kernel.flat.FlatPointOfInterest
        [Loader] [tstamp: 4661 time: 4659.6 (1:17:39) split: 673.9 (11:13)] [Memory: 4.512G total: 7.759G free: 3.248G]
INFO: 7700000 packets. Current packet is class com.navtech.kernel.flat.FlatPointOfInterest
        [Loader] [tstamp: 4667 time: 4665.5 (1:17:45) split: 05.8] [Memory: 4.787G total: 7.759G free: 2.972G]
INFO: 7750000 packets. Current packet is class com.navtech.kernel.flat.FlatPointOfInterest
        [Loader] [tstamp: 4678 time: 4676.9 (1:17:56) split: 11.4] [Memory: 5.190G total: 7.759G free: 2.569G]
INFO: 7800000 packets. Current packet is class com.navtech.kernel.flat.FlatPointOfInterest
        [Loader] [tstamp: 4779 time: 4778.1 (1:19:38) split: 101.2 (01:41)] [Memory: 5.115G total: 7.759G free: 2.645G]
INFO: Done processing Pois file.
        [VsamServer] [tstamp: 4782 time: 4779.9 (1:19:39) split: 835.1 (13:55)] [Memory: 5.243G total: 7.759G free: 2.517G]
Unexpected Signal: 11, PC: ffffffff7e631818, PID: 3512
An error has just occurred.
To debug, use 'dbx - 3512'; then switch to thread t@12
Otherwise, press RETURN to abort...

Unexpected Signal : 11 occurred at PC=0xFFFFFFFF7E631818
Function=[Unknown. Nearest: JVM_Clone+0x80B0]
Library=/sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/server/libjvm.so

Current Java thread:

Dynamic libraries:
0x100000000 /sysarch/bison/mhansen/j2sdk1.4.2_05/bin/sparcv9/java
0xffffffff7f300000 /usr/lib/64/libthread.so.1
0xffffffff7f200000 /usr/lib/64/libdl.so.1
0xffffffff7ef00000 /usr/lib/64/libc.so.1
0xffffffff7f500000 /usr/platform/SUNW,Sun-Fire-880/lib/sparcv9/libc_psr.so.1
0xffffffff7e400000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/server/libjvm.so
0xffffffff7e200000 /usr/lib/64/libCrun.so.1
0xffffffff7e000000 /usr/lib/64/libsocket.so.1
0xffffffff7de00000 /usr/lib/64/libnsl.so.1
0xffffffff7dc00000 /usr/lib/64/libm.so.1
0xffffffff7da00000 /usr/lib/64/libsched.so.1
0xffffffff7ed00000 /usr/lib/64/libw.so.1
0xffffffff7d600000 /usr/lib/64/libmp.so.2
0xffffffff7d400000 /usr/lib/64/librt.so.1
0xffffffff7d200000 /usr/lib/64/libaio.so.1
0xffffffff7d000000 /usr/lib/64/libmd5.so.1
0xffffffff7cd00000 /usr/platform/SUNW,Sun-Fire-880/lib/sparcv9/libmd5_psr.so.1
0xffffffff7c400000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/native_threads/libhpi.so
0xffffffff7c100000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/libverify.so
0xffffffff7bf00000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/libjava.so
0xffffffff7bc00000 /sysarch/bison/mhansen/j2sdk1.4.2_05/jre/lib/sparcv9/libzip.so
0xfffffffd3d500000 /usr/lib/locale/en_US.ISO8859-1/sparcv9/en_US.ISO8859-1.so.2

Heap at VM Abort:
Heap
 par new generation total 1843200K, used 1221619K [0xfffffffd3f800000, 0xfffffffdd5800000, 0xfffffffdd5800000)
  eden space 1228800K, 70% used [0xfffffffd3f800000, 0xfffffffd74038030, 0xfffffffd8a800000)
  from space 614400K, 58% used [0xfffffffd8a800000, 0xfffffffda08c4ce0, 0xfffffffdb0000000)
  to space 614400K, 0% used [0xfffffffdb0000000, 0xfffffffdb0000000, 0xfffffffdd5800000)
 tenured generation total 5734400K, used 4111451K [0xfffffffdd5800000, 0xffffffff33800000, 0xffffffff33800000)
   the space 5734400K, 71% used [0xfffffffdd5800000, 0xfffffffed0716c20, 0xfffffffed0716e00, 0xffffffff33800000)
 compacting perm gen total 16384K, used 7618K [0xffffffff33800000, 0xffffffff34800000, 0xffffffff37800000)
   the space 16384K, 46% used [0xffffffff33800000, 0xffffffff33f70ae0, 0xffffffff33f70c00, 0xffffffff34800000)

Local Time = Wed Sep 15 09:38:31 2004
Elapsed Time = 5525
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.4.2_05-b04 mixed mode)
#
# An error report file has been saved as hs_err_pid3512.log.
# Please refer to the file for further information.
#
/vobs/bison_core/bin/launch[323]: 3512 Abort



----------------- lwp# 12 / thread# 12 --------------------
 ffffffff7efa7930 read (0, ffffffff7f0c4948, 400)
 ffffffff7ef98b94 _filbuf (ffffffff7f0bf5a8, 1, ffffffff7f0bf5a8, 0, 73, 0) + c4
 ffffffff7ef9b448 fgets (fffffffd3c7fc550, 400, ffffffff7f0bf5a8, ffffffff7eb84f3c, db8, c) + 208
 ffffffff7e9073fc __1cCosLmessage_box6Fpkc2_i_ (fffffffd3c7fd230, ffffffff7eb84f3c, b, ffffffff7e631818, db8, 0) + 44
 ffffffff7e903d78 __1cCosbBhandle_unexpected_exception6FpnGThread_ipCpv_v_ (10030c440, b, ffffffff7e631818, fffffffd3c7
fd9d0, ffffffff7e69b7c0, 0) + 140
 ffffffff7e69b8c8 JVM_handle_solaris_signal (fffffffd3c7fd9d0, ffffffff7e9cde6e, fffffffd3c7fd6f0, 1, 0, 1) + 8e0
 ffffffff7f318ae4 __sighndlr (b, fffffffd3c7fd9d0, fffffffd3c7fd6f0, ffffffff7e69bc64, 0, 0) + c
 ffffffff7f312718 call_user_handler (ffffffff7ee02c00, fffffffd3c7fd9d0, fffffffd3c7fd6f0, 0, 0, 0) + 25c
 ffffffff7f3128dc sigacthandler (b, fffffffd3c7fd9d0, fffffffd3c7fd6f0, 1, 100a5e1f8, ffffffff7e631b5c) + 68
 --- called from signal handler with signal 11 (SIGSEGV) ---
 ffffffff7e631818 __1cMPhaseChaitinQgather_lrg_masks6Mi_v_ (1, 1, 1, 2, 0, 2) + 3ac
 ffffffff7e66d8a0 __1cMPhaseChaitinRRegister_Allocate6M_v_ (ffffffff7eb795d0, ffffffff7eb66310, fffffffd3c7fdfa8, a400,
 a450, fffffffffffffff8) + 3d8
 ffffffff7e7d1780 __1cHCompileICode_Gen6M_v_ (fffffffd3c7feca0, fffffffd3c7fe8d0, fffffffd3c7feb20, 1, 0, fffffffd3c7fe
b29) + 234
 ffffffff7e7d0f70 __1cHCompile2t6MpnFciEnv_pnHciScope_pnIciMethod_iii_v_ (fffffffd3c7fe8d0, fffffffd3c7fe908, 0, 1006df
698, ffffffffffffffff, 1) + 998
 ffffffff7e7a7340 __1cKC2CompilerOcompile_method6MpnFciEnv_pnHciScope_pnIciMethod_ii_v_ (1001192c0, fffffffd3c7ffb50, 0
, 1006df698, ffffffffffffffff, 0) + 68
 ffffffff7e68c210 __1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_ (0, ffffffff, ffffffff7eb62ad0, 4b5,
 ffffffff7eb5dce8, 10030c558) + 530
 ffffffff7e7126c8 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (ffffffff7eb630e8, 10030c440, 10030cf10, 10019c290, ffff
ffff7e6d196c, 0) + 328
 ffffffff7e6d1994 __1cKJavaThreadRthread_main_inner6M_v_ (10030c440, 40, 0, 0, 0, 0) + 3c
 ffffffff7e6cc4fc _start (10030c440, 0, 0, 0, 0, 0) + ec
 ffffffff7f318814 _lwp_start (0, 0, 0, 0, 0, 0)
----------------- lwp# 13 / thread# 13 --------------------


When we use +PrintCompilation flag, the failure doesn't
happen. The extra printing made the problem go away.

Here is the dbx output from core file

 t@null (l@12) terminated by signal ABRT (Abort)
0xffffffff7efa808c: __lwp_kill+0x0008: bad opcode
(dbx) where
=>[1] __lwp_kill(0x0, 0x6, 0xffffffffffffffe6, 0x0, 0x0, 0x0), at 0xffffffff7efa808c
  [2] raise(0x6, 0x0, 0xfffffffcfdbfc850, 0x7fbffeff00003ff6, 0x0, 0x2), at 0xffffffff7ef588fc
  [3] abort(0x0, 0xfffffffcfdbfc930, 0x0, 0xfffffffffffffff8, 0x0, 0xfffffffcfdbfc959), at 0xffffffff7ef3e3f8
  [4] os::abort(0x1, 0xffffffff7e9cc38c, 0xfffffffcfdbfca30, 0xffffffff7e9cbed9, 0x4ab25c, 0xffffffff7eb72148), at 0xffffffff7e9055bc
  [5] os::handle_unexpected_exception(0x10030c440, 0xb, 0xffffffff7e631818, 0xfffffffcfdbfd9d0, 0xffffffff7e69b7c0, 0x0), at 0xffffffff7e903ea8
  [6] JVM_handle_solaris_signal(0xfffffffcfdbfd9d0, 0xffffffff7e9cde6e, 0xfffffffcfdbfd6f0, 0x1, 0x0, 0x1), at 0xffffffff7e69b8c8
  [7] __sighndlr(0xb, 0xfffffffcfdbfd9d0, 0xfffffffcfdbfd6f0, 0xffffffff7e69bc64, 0x0, 0x0), at 0xffffffff7f318ae4
  [8] call_user_handler(0xffffffff7ee02c00, 0xfffffffcfdbfd9d0, 0xfffffffcfdbfd6f0, 0x0, 0x0, 0x0), at 0xffffffff7f312718
  [9] sigacthandler(0xb, 0xfffffffcfdbfd9d0, 0xfffffffcfdbfd6f0, 0x1, 0x100815d08, 0xffffffff7e631b5c), at 0xffffffff7f3128dc
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [10] PhaseChaitin::gather_lrg_masks(0x1, 0x1, 0x1, 0x2, 0x0, 0x2), at 0xffffffff7e631818
  [11] PhaseChaitin::Register_Allocate(0xffffffff7eb795d0, 0xffffffff7eb66310, 0xfffffffcfdbfdfa8, 0xa400, 0xa450, 0xfffffffffffffff8), at 0xffffffff7e66d8a0
  [12] Compile::Code_Gen(0xfffffffcfdbfeca0, 0xfffffffcfdbfe8d0, 0xfffffffcfdbfeb20, 0x1, 0x0, 0xfffffffcfdbfeb29), at 0xffffffff7e7d1780
  [13] Compile::Compile(0xfffffffcfdbfe8d0, 0xfffffffcfdbfe908, 0x0, 0x10051a478, 0xffffffffffffffff, 0x1), at 0xffffffff7e7d0f70
  [14] C2Compiler::compile_method(0x1001192c0, 0xfffffffcfdbffb50, 0x0, 0x10051a478, 0xffffffffffffffff, 0x0), at 0xffffffff7e7a7340
  [15] CompileBroker::invoke_compiler_on_method(0x0, 0xffffffff, 0xffffffff7eb62ad0, 0x4b3, 0xffffffff7eb5dce8, 0x10030c558), at 0xffffffff7e68c210
  [16] CompileBroker::compiler_thread_loop(0xffffffff7eb630e8, 0x10030c440, 0x10030cf10, 0x10019c290, 0xffffffff7e6d196c, 0x0), at 0xffffffff7e7126c8
  [17] JavaThread::thread_main_inner(0x10030c440, 0x40, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7e6d1994
  [18] _start(0x10030c440, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7e6cc4fc
(dbx) help lwps
lwps (command)
lwps # List all LWPs in the current process
(dbx) lwps
o>l@12 signal SIGABRT in __lwp_kill()
  l@13 LWP suspended in ___lwp_cond_wait()
  l@14 LWP suspended in ___lwp_mutex_lock()
  l@15 LWP suspended in _libc_nanosleep()
  l@1 LWP suspended in _libc_nanosleep()
  l@2 LWP suspended in ___lwp_cond_wait()
  l@3 LWP suspended in ___lwp_cond_wait()
  l@4 LWP suspended in ___lwp_cond_wait()
  l@5 LWP suspended in ___lwp_cond_wait()
  l@6 LWP suspended in ___lwp_cond_wait()
  l@7 LWP suspended in ___lwp_cond_wait()
  l@8 LWP suspended in ___lwp_cond_wait()
  l@9 LWP suspended in ___lwp_cond_wait()
  l@10 LWP suspended in __lwp_park()
  l@11 LWP suspended in ___lwp_cond_wait()
(dbx) help lwp
lwp (command)
lwp # Display current LWP
lwp <lwpid> # Switch to LWP <lwpid>
(dbx) lwp l@12
t@null (l@12) stopped in __lwp_kill at 0xffffffff7efa808c
0xffffffff7efa808c: __lwp_kill+0x0008: bad opcode
(dbx)