java crash in generateOopMap.cpp

jsp written by a customer crashes a few times per day.
It is application which works with
- BEA Weblogic 5.1 sp12 wlcs 3.2sp2
- HotSpot Client VM 1.3.1_04-b02 (and 1.3.1_03-b03)
- Ultra Enterprise 450 5.7 (and 5.8)

# HotSpot Virtual Machine Error, Internal Error
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 47454E45524154452F4F502D41500E4350500265 01
#
# Problematic Thread: prio=5 tid=0x8a4e00 nid=0x1b runnable
#

% /usr/j2se/bin/java -classpath . ErrorID 47454E45524154452F4F502D41500E4350500265
generateOopMap.cpp, 613

Application makes some large table in the memory
This problem is only on multi CPU machines.
Customer checked his application on few machines and Solaris verions.
It looks like it does not depend on architecture (Sparc II/ Sparc III) nor Solaris version (7 nor 8).
It worked fine on one cpu server.

Location of core files and explorer output for solaris 7,8:
/net/uranus.poland/export/mnt/CALLS/36626546
They are too big to be attched to bug:
sol7/jdk131_04:
-rw-r--r-- 1 claudiom answer 800182016 Jul 26 16:07 core
sol8/jdk131_04:
-rw-r--r-- 1 claudiom answer 459316856 Jul 26 16:25 core


Using following startup options:

java -client -XX:MaxPermSize=256m -Xms512m -Xmx512m -noverify -noclassgc
-verbose:gc -classpath $JAVA_CLASSPATH
-Dweblogic.class.path=$WEBLOGIC_CLASSPATH
-Dweblogic.cluster.multicastAddress=225.0.0.3
-Dweblogic.system.name=$SYSTEM_NAME
-Dweblogic.ejb.tempDir=/exp/log/weblogic/$SYSTEM_NAME/tmp_www2
-Dweblogic.system.home=$CFG_DIR
-Dweblogic.system.logFile=$LOG_DIR/$SYSTEM_NAME/weblogic.www2.log
-Dweblogic.httpd.logFileName=$LOG_DIR/$SYSTEM_NAME/access.www2.log
-Dweblogic.jdbc.logFileName=$LOG_DIR/$SYSTEM_NAME/jdbc.www2.log
-Dweblogic.home=$WEBLOGIC_HOME -Djava.security.manager
-Djava.security.policy=$CFG_DIR/weblogic.policy
-Dcommerce.properties=$CFG_DIR/weblogiccommerce.properties
-Dweblogic.properties=$CFG_DIR/weblogic.properties
-Dpipeline.properties=$CFG_DIR/pipeline.properties
-Dwebflow.properties=$CFG_DIR/webflow.properties weblogic.Server

Using the sol8 core I see they have tested -server too:

% strings - core | grep sparc/native_threads/java
/usr/java/bin/../bin/sparc/native_threads/java -server -XX:MaxPermSize=128m -Xm

Checked patches for sol8; see also complete report in attachment webpatchdiag-sol8.html
The following is the list of patches installed now at customer site (sol8) compared to the list of suggested solaris 8 cluster for j2se: http://sunsolve.Central.Sun.COM/cgi/retrieve.pl?doctype=patch&doc=J2SE_Solaris_8_Recommended.README

Now Cluster
============================================================================
-55 108652-56 X11 6.4.1: Xsun patch
-11 108921-15 CDE 1.4: dtwm patch
none 112003-03 SunOS 5.8: Unable to load fontset in 64-bit Solaris 8 iso-1 or iso-15
-08 108773-15 SunOS 5.8: IIIM and X Input & Output Method patch
ok 111293-04 SunOS 5.8: /usr/lib/libdevinfo.so.1 patch
ok 111310-01 SunOS 5.8: /usr/lib/libdhcpagent.so.1 patch
none 112472-01 SunOS 5.8: Font2DTest2 abort when Lucida Sans Thai Typewriter
selected
-05 108714-07 CDE 1.4: libDtWidget patch
ok 111111-03 SunOS 5.8: /usr/bin/nawk patch
ok 112396-02 SunOS 5.8: /usr/bin/fgrep patch
-16 108940-43 Motif 1.2.7 and 2.1.1: Runtime library patch for Solaris 8
ok 108987-09 SunOS 5.8: Patch for patchadd and patchrm
ok 108528-15 SunOS 5.8: kernel update patch
ok 108989-02 SunOS 5.8: /usr/kernel/sys/acctctl and /usr/kernel/sys/exacctsys patch
-25 108827-26 SunOS 5.8: /usr/lib/libthread.so.1 patch

Using dbx 7.0 and 1.3.1_04-b02 on a 5.8 Sun-Fire I see they are using lib/lwp/libthread and I get the stack with the following calls GenerateOopMap::compute_map -> GenerateOopMap::init_basic_blocks -> GenerateOopMap::verify_error -> GenerateOopMap::error_work :

# dbx /usr/j2se/bin/sparc/native_threads/java core
Reading java
core file header read successfully
Reading ld.so.1
Reading libthread.so.1
Reading libdl.so.1
Reading libc.so.1
Reading libc_psr.so.1
Reading libjvm.so
Reading libCrun.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libm.so.1
Reading libw.so.1
Reading libmp.so.2
Reading librt.so.1
Reading libaio.so.1
Reading libhpi.so
Reading libverify.so
Reading libjava.so
Reading libzip.so
Reading libnet.so
Reading libgen.so.1
Reading libsched.so.1
Reading nss_files.so.1
WARNING!!
A loadobject was found with an unexpected checksum value.
See `help core mismatch' for details, and run `proc -map'
to see what checksum values were expected and found.
dbx: warning: Some symbolic information might be incorrect.
detected a multithreaded program
t@30 (l@30) terminated by signal ABRT (Abort)
0xff31bdc4: _lwp_kill+0x0010: cmp %o0, 0x5b
(dbx) pathmap
(1) -c /tmp_mnt /
(dbx) proc -map
Loadobject mappings for current core file:
0x00010000 /usr/j2se/bin/sparc/native_threads/java
0xff360000 /usr/lib/lwp/libthread.so.1
  Warning: checksum in file(8177) doesn't match image(8a1a)
0xff390000 /usr/lib/libdl.so.1
           is being filtered by: /usr/lib/ld.so.1
0xff280000 /usr/lib/libc.so.1
           is being filtered by: /usr/platform/SUNW,Sun-Fire/lib/libc_psr.so.1
  Warning: checksum in file(d661) doesn't match image(314e)
0xff270000 /usr/platform/SUNW,Ultra-4/lib/libc_psr.so.1
0xfec00000 /usr/j2se/jre/lib/sparc/server/libjvm.so
0xff230000 /usr/lib/libCrun.so.1
  Warning: checksum in file(58ba) doesn't match image(ad48)
0xff200000 /usr/lib/libsocket.so.1
  Warning: checksum in file(b85d) doesn't match image(c331)
0xfeb00000 /usr/lib/libnsl.so.1
  Warning: checksum in file(283e) doesn't match image(1cac)
0xff1d0000 /usr/lib/libm.so.1
0xff260000 /usr/lib/libw.so.1
           is being filtered by: libc.so.1
0xff1a0000 /usr/lib/libmp.so.2
0xff180000 /usr/lib/librt.so.1
  Warning: checksum in file(ea15) doesn't match image(eb91)
0xff160000 /usr/lib/libaio.so.1
0xfebd0000 /usr/j2se/jre/lib/sparc/native_threads/libhpi.so
0xfead0000 /usr/j2se/jre/lib/sparc/libverify.so
0xfea90000 /usr/j2se/jre/lib/sparc/libjava.so
0xfea60000 /usr/j2se/jre/lib/sparc/libzip.so
0xfd810000 /usr/j2se/jre/lib/sparc/libnet.so
0xfb7e0000 /usr/local/weblogic/weblogic510sp12/lib/solaris/libweblogicunix1.so (not loaded)
0xd8700000 /usr/local/weblogic/weblogic510sp12/lib/solaris/oci817_8/libweblogicoci36.so (not loaded)
0xd8000000 /opt/oracle/lib/libclntsh.so.8.0 (not loaded)
0xfb7c0000 /opt/oracle/lib/libwtc8.so (not loaded)
0xfb790000 /usr/lib/libgen.so.1
0xfb6e0000 /usr/lib/libsched.so.1
0xdb1b0000 /usr/lib/nss_files.so.1
0xdb190000 /usr/local/weblogic/weblogic510sp12/lib/solaris/libmuxer.so (not loaded)
0xff3b0000 /usr/lib/ld.so.1 [LM_ID_LDSO]
(dbx) where
current thread: t@30
=>[1] _lwp_kill(0x0, 0x1e, 0x0, 0xff33a000, 0xff386000, 0xff33e428), at 0xff31bdc4
  [2] rand(0x6, 0x0, 0x0, 0xffffffff, 0xff33e3b4, 0x0), at 0xff2cb758
  [3] addsev(0xff33a000, 0xd6d7d648, 0x0, 0x4, 0x0, 0xd6d7d669), at 0xff2b5a7c
  [4] os::abort(0x1, 0xff0ce000, 0x1, 0xff0ce000, 0x6ee660, 0xd6d7d664), at 0xfefc7da4
  [5] report_error(0xdc, 0xd6d7deec, 0x842, 0xff054858, 0xff13e4b0, 0xff0ce000), at 0xfef18c0c
  [6] report_fatal(0x842, 0xff0ce000, 0xff056a20, 0xd6d7eb34, 0xff0e26b4, 0xff0ce000), at 0xfef184dc
  [7] GenerateOopMap::error_work(0xff0ce000, 0xff0564c8, 0xd6d7eb34, 0xff0ce000, 0x24ec758, 0x11ff4c), at 0xfef26444
  [8] GenerateOopMap::verify_error(0xd6d7ed04, 0xff0564c8, 0xd8c, 0xd0e910, 0x1b1a0, 0x1b180), at 0xfef264a4
  [9] GenerateOopMap::init_basic_blocks(0x24ec760, 0xf3dc2ae4, 0xff13fd2c, 0xff1404dc, 0xbfc3, 0x0), at 0xfeddc0a0
  [10] GenerateOopMap::compute_map(0xff0d9698, 0xff0ce000, 0xff0ce000, 0xffffffff, 0x1, 0xd6d7ec04), at 0xfedeb384
  [11] ResolveOopMapConflicts::do_potential_rewrite(0xd6d7ee00, 0xd6d7ee00, 0xd6d7ed04, 0xffffffff, 0xfffffff8, 0xf3dd06a8), at 0xfee0bf0c
  [12] Rewriter::rewrite_method(0xd6d7ee00, 0xff0ce000, 0xd6d7ee04, 0x6ee660, 0xff13fd2c, 0xff1404dc), at 0xfed187f4
  [13] Rewriter::rewrite(0xff1404d0, 0xff1404c8, 0xd6d7ee9c, 0x4, 0x8, 0xf3dc2860), at 0xfed8aca8
  [14] instanceKlass::link_class_impl(0xf3dc2850, 0x6ee660, 0xd6d7f020, 0xff0ce000, 0x6ee660, 0xd6d7ef3c), at 0xfed639d8
  [15] instanceKlass::initialize_impl(0xd6d7f128, 0xf3dcfff0, 0xff0ce000, 0x6ee660, 0x6ee660, 0xd6d7f0c4), at 0xfed4e938
  [16] instanceKlass::initialize(0xf3dcfff0, 0x6ee660, 0x6cb138, 0xff0ce000, 0x6ee660, 0xd6d7f144), at 0xfed2bf6c
  [17] JVM_NewInstance(0xf3dcfff0, 0x6ee660, 0x6ee6ec, 0xff0ce000, 0xff0e4960, 0x0), at 0xfecb9178
  [18] 0xa6604(0xd6d7f334, 0x0, 0xf3dd0118, 0xac6f0, 0x0, 0x109a0), at 0xa6603
  [19] 0xa3790(0xd6d7f3bc, 0xd6d7f460, 0xd6d7f464, 0xaf2fc, 0x4, 0xd6d7f2d0), at 0xa378f
  [20] 0xa3790(0xd6d7f464, 0xd6d7f500, 0xd6d7f504, 0xaf228, 0x4, 0xd6d7f350), at 0xa378f
  [21] 0xa3790(0xd6d7f504, 0x0, 0xd6d7f594, 0xaf120, 0x4, 0xd6d7f3f8), at 0xa378f
  [22] 0xa376c(0xd6d7f59c, 0xd6d7f638, 0xd6d7f63c, 0xaf120, 0x4, 0xd6d7f490), at 0xa376b
  [23] 0xa376c(0xd6d7f644, 0x1dec430e, 0xd6d7f648, 0xaf2fc, 0xc, 0xd6d7f530), at 0xa376b
  [24] 0xa376c(0xd6d7f6dc, 0x0, 0xd6d7f77c, 0xaf120, 0xc, 0xd6d7f5d8), at 0xa376b
  [25] 0xa3790(0xd6d7f784, 0xd6d7f788, 0xd6d7f788, 0xaf120, 0xc, 0xd6d7f670), at 0xa378f
  [26] 0xa376c(0xd6d7f834, 0xd6d7f8cc, 0xd6d7f8d0, 0xaf120, 0xc, 0xd6d7f718), at 0xa376b
  [27] 0xa376c(0xd6d7f8dc, 0xd6d7f968, 0xd6d7f96c, 0xaf120, 0xc, 0xd6d7f7c8), at 0xa376b
  [28] 0xa376c(0xd6d7f974, 0x1dec430b, 0xff0ce000, 0xaf120, 0x10, 0xd6d7f870), at 0xa376b
  [29] 0xa376c(0xd6d7fa34, 0x198, 0xd6d7fac4, 0xaf120, 0xc, 0xd6d7f908), at 0xa376b
  [30] 0xa3790(0xd6d7fac4, 0xe085b0d8, 0xd6d7fb50, 0xaf120, 0x8, 0xd6d7f9d0), at 0xa378f
  [31] 0xa3790(0xd6d7fb54, 0xf374ada8, 0xd6d7fb58, 0xaf120, 0x4, 0xd6d7fa60), at 0xa378f
  [32] 0xa38d4(0xd6d7fbdc, 0x1, 0xff0dca68, 0xaf494, 0x8, 0xd6d7fae8), at 0xa38d3
  [33] 0xff138048(0xd6d7fc68, 0xd6d7fea0, 0xa, 0xf3770260, 0x4, 0xd6d7fb80), at 0xff138047
  [34] JavaCalls::call_helper(0xd6d7fe98, 0xff0ce000, 0xd6d7fde4, 0x6ee660, 0xa55bc, 0xd6d7fea0), at 0xfeccd0c8
  [35] JavaCalls::call_virtual(0xf3770540, 0xd6d7fdd0, 0xd6d7fdd4, 0xff0ce000, 0xd6d7fe98, 0xd6d7fde4), at 0xfedf82e8
  [36] JavaCalls::call_virtual(0xd6d7fe98, 0xd6d7fe94, 0xd6d7fe90, 0xd6d7fe84, 0xd6d7fe7c, 0x6ee660), at 0xfedfe2b8
  [37] thread_entry(0xf3417f78, 0x6ee660, 0xff0ce000, 0xd6d7ffa0, 0x1e, 0xe), at 0xfee1caec
  [38] JavaThread::run(0xd6d00000, 0xff0d9e1c, 0xff0ce000, 0x80000, 0x6ee660, 0x80000), at 0xfee175ec
  [39] _start(0xff0ce000, 0xfebc3800, 0x0, 0x0, 0x0, 0x0), at 0xfee15320
(dbx) threads
      t@1 a l@1 ?() LWP suspended in ___lwp_cond_wait()
      t@2 b l@2 _start() LWP suspended in ___lwp_cond_wait()
      t@3 b l@3 _start() LWP suspended in ___lwp_cond_wait()
      t@4 b l@4 _start() LWP suspended in ___lwp_cond_wait()
      t@5 b l@5 _start() LWP suspended in _poll()
      t@6 b l@6 _start() LWP suspended in ___lwp_cond_wait()
      t@7 b l@7 _start() LWP suspended in ___lwp_cond_wait()
      t@8 b l@8 _start() LWP suspended in ___lwp_cond_wait()
      t@9 b l@9 _start() LWP suspended in ___lwp_cond_wait()
     t@10 b l@10 _start() LWP suspended in ___lwp_cond_wait()
     t@11 b l@11 _start() LWP suspended in ___lwp_cond_wait()
     t@12 b l@12 _start() LWP suspended in ___lwp_cond_wait()
     t@13 b l@13 _start() LWP suspended in ___lwp_cond_wait()
     t@14 b l@14 _start() LWP suspended in ___lwp_cond_wait()
     t@15 b l@15 _start() LWP suspended in ___lwp_cond_wait()
     t@16 b l@16 _start() LWP suspended in ___lwp_cond_wait()
     t@17 b l@17 _start() LWP suspended in ___lwp_cond_wait()
     t@18 b l@18 _start() LWP suspended in ___lwp_cond_wait()
     t@19 b l@19 _start() LWP suspended in ___lwp_cond_wait()
     t@20 b l@20 _start() LWP suspended in ___lwp_cond_wait()
     t@21 b l@21 _start() LWP suspended in ___lwp_cond_wait()
     t@22 b l@22 _start() LWP suspended in ___lwp_cond_wait()
     t@23 b l@23 _start() LWP suspended in ___lwp_cond_wait()
     t@24 b l@24 _start() LWP suspended in ___lwp_cond_wait()
     t@25 b l@25 _start() LWP suspended in ___lwp_cond_wait()
     t@26 b l@26 _start() LWP suspended in ___lwp_cond_wait()
     t@27 b l@27 _start() LWP suspended in ___lwp_cond_wait()
     t@28 b l@28 _start() LWP suspended in ___lwp_cond_wait()
     t@29 b l@29 _start() LWP suspended in ___lwp_cond_wait()
o> t@30 b l@30 _start() signal SIGABRT in _lwp_kill()
     t@31 b l@31 _start() LWP suspended in ___lwp_cond_wait()
     t@32 b l@32 _start() LWP suspended in ___lwp_cond_wait()
     t@33 b l@33 _start() LWP suspended in ___lwp_cond_wait()
     t@34 b l@34 _start() LWP suspended in ___lwp_cond_wait()
     t@35 b l@35 _start() LWP suspended in ___lwp_cond_wait()
     t@36 b l@36 _start() LWP suspended in ___lwp_cond_wait()
     t@37 b l@37 _start() LWP suspended in ___lwp_cond_wait()
     t@38 b l@38 _start() LWP suspended in _poll()
     t@39 b l@39 _start() LWP suspended in ___lwp_mutex_lock()
     t@40 b l@40 _start() LWP suspended in ___lwp_mutex_lock()
     t@41 b l@41 _start() LWP suspended in _poll()
     t@42 b l@42 _start() LWP suspended in ___lwp_cond_wait()
     t@43 b l@43 _start() LWP suspended in ___lwp_cond_wait()
     t@47 b l@47 _start() LWP suspended in _so_accept()