JVM crash - Java 6 on Solaris 10 / T2000

The problem is only seen in production, it is not seen in QA or development.
Also, the crashes happen quite frequently and stacks seem consistent.


---- called from signal handler with signal 4 (SIGILL) ------
  [8] 0xfc408354(0xbc30fa48, 0x2ca7f6e0, 0x382b39e0, 0x162fa34, 0x1, 0x2ca7f5c0), at 0xfc408354
  [9] 0xfc4058c0(0xbc30fa48, 0xb7, 0x3917e790, 0x162fa30, 0x2, 0x2ca7f658), at 0xfc4058c0
  [10] 0xfc405a30(0xbc30fa48, 0x2ca7f6e0, 0x382b3a60, 0xfc418ea0, 0x1, 0x2ca7f6e8), at 0xfc405a30
  [11] 0xfc405908(0xbc1a94e0, 0xb6, 0x0, 0xfc418c20, 0xfc40c340, 0x2ca7f760), at 0xfc405908
  [12] 0xfc405908(0x1, 0x38b6e120, 0x0, 0xfc418c20, 0x167bd58, 0x2ca7f7e8), at 0xfc405908
  [13] 0xfc40021c(0x2ca7f8f0, 0x2ca7fb98, 0xa, 0x38b6e120, 0xfc40c340, 0x2ca7fa80), at 0xfc40021c
  [14] libjvm.so:JavaCalls::call_helper(0x1, 0x38b6e120, 0x2ca7fa78, 0x1688800, 0x167bd58, 0x2ca7f8d0), at 0xfe8e1460
  [15] libjvm.so:JavaCalls::call_virtual(0x2ca7fb90, 0x167bd5c, 0xfefd3200, 0x167bd68, 0x2ca7fa78, 0x1688800), at 0xfeb98c90
  [16] libjvm.so:JavaCalls::call_virtual(0x2ca7fb90, 0x2ca7fb8c, 0x2ca7fb88, 0x2ca7fb84, 0x2ca7fb80, 0x167bd5c), at 0xfe98d054
  [17] libjvm.so:thread_entry(0x38030878, 0x1688800, 0x3ac00, 0xfefd32e0, 0xfefd3200, 0xfefd2c80), at 0xfe9a127c
  [18] libjvm.so:JavaThread::thread_main_inner(0x1688800, 0x3d3e, 0xbcfb6758, 0xfef98000, 0xfef98000, 0x0), at 0xfee21eb4
  [19] libjvm.so:java_start(0x1688800, 0x3d3c, 0x455, 0xfef98000, 0xfef20270, 0x2a400), at 0xfed8ecb0

I saw some resemblance to what's reported in CR 6670667, but the stack from the hs_err log (below) doesn't seem to match at the CR at all. Not to mention the issue in the CR was seen with 1.4.2 rather than 6.

Stack: [0x2ca00000,0x2ca80000), sp=0x2ca7f538, free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
j java.io.BufferedInputStream.getBufIfOpen()[B+6
j java.io.BufferedInputStream.fill()V+1
j java.io.BufferedInputStream.read()I+12
j com.adp.e.server.rpc.WebServerTask.read()I+7
j com.adp.e.server.alert.AlertTask.run()V+4
v ~StubRoutines::call_stub
V [libjvm.so+0xe1468];; void JavaCalls::call_helper(JavaValue*,methodHandle*,J
avaCallArguments*,Thread*)+0x1ec
V [libjvm.so+0x398c98];; void JavaCalls::call_virtual(JavaValue*,KlassHandle,s
ymbolHandle,symbolHandle,JavaCallArguments*,Thread*)+0xf4
V [libjvm.so+0x18d05c];; void JavaCalls::call_virtual(JavaValue*,Handle,KlassH
andle,symbolHandle,symbolHandle,Thread*)+0x7c
V [libjvm.so+0x1a1284];; void thread_entry(JavaThread*,Thread*)+0x118
V [libjvm.so+0x621ebc];; void JavaThread::thread_main_inner()+0x50
V [libjvm.so+0x58ecb8];; java_start+0x140

From dissassemble we see:

0xfc408338: save %sp, -608, %sp
0xfc40833c: sethi %hi(0xfeef0000), %o1
0xfc408340: inc 666, %o1
0xfc408344: sethi %hi(0xfefc5800), %o5
0xfc408348: ld [%o5 + 0x00000110], %o5 ! 0xfefc5910
0xfc40834c: call %o5 <------ call into debug should stop
0xfc408350: nop
0xfc408354: ta 16 <----- this is breakpoint trap in JVM 0xfc408358: st %sp, [%g2 + 216]
0xfc40835c: mov %g2, %l7
0xfc408360: call throw_pending_exception ! 0xfe96b708
0xfc408364: mov %g2, %o0
0xfc408368: mov %l7, %g2

Also around the time of the crash, we do see some IO related exceptions from the application:

java.lang.Exception: StackTrace for IO
        at com.adp.e.persist.DBAccess.a(Unknown Source)
        at com.adp.e.persist.DBAccess.<init>(Unknown Source)
        at com.adp.e.persist.DBAccess.<init>(Unknown Source)
        at com.adp.e.persist.DataStoreAccessFactory.getDataAccess(Unknown Source
)
        at com.adp.e.ramsam.SqlAccHandler.a(Unknown Source)
        at com.adp.e.ramsam.SqlAccHandler.SamCreate(Unknown Source)
        at com.adp.e.ramsam.RecAccHandler.RamStart(Unknown Source)
        at com.adp.e.ramsam.RecAccHandler.read(Unknown Source)
        at com.adp.e.ramsam.RecAccHandler.read(Unknown Source)
        at com.adp.e.server.ServerMasterHandler.read(Unknown Source)
        at com.adp.e.common.RecAccMgr.read(Unknown Source)
        at com.adp.e.pgbuffer.PGRead.readLevel(Unknown Source)
        at com.adp.e.pgbuffer.PGRead.readLevel(Unknown Source)
        at com.adp.e.pgbuffer.PGRead.readData(Unknown Source)
        at com.adp.e.server.buffer.PGHandler.getPGData(Unknown Source)
        at com.adp.e.server.ServerMasterHandler.getPGData(Unknown Source)
        at com.adp.e.pgbuffer.PGMgr.getPGData(Unknown Source)
        at com.adp.e.server.messageagent.MessageAgentHandler.a(Unknown Source)
        at com.adp.e.server.messageagent.MessageAgentHandler.processMessage(Unkn
own Source)
        at com.adp.e.server.ServerMasterHandler.processMessage(Unknown Source)
        at com.adp.e.server.MessageAgentMgr.processMessage(Unknown Source)
        at com.adp.e.server.webservice.ADPMessage.b(Unknown Source)
        at com.adp.e.server.webservice.ADPMessage.a(Unknown Source)
        at com.adp.e.server.webservice.ADPMessage.process(Unknown Source)
        at com.adp.e.server.rpc.ADPWebServices.doPost(Unknown Source)
        at com.adp.e.server.rpc.WebServerTask.run(Unknown Source)
        at com.adp.e.server.TaskThread.run(Unknown Source)