Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-4724304

java crash in generateOopMap.cpp

XMLWordPrintable

    • sparc
    • solaris_8

      jsp written by a customer crashes a few times per day.
      It is application which works with
      - BEA Weblogic 5.1 sp12 wlcs 3.2sp2
      - HotSpot Client VM 1.3.1_04-b02 (and 1.3.1_03-b03)
      - Ultra Enterprise 450 5.7 (and 5.8)

      # HotSpot Virtual Machine Error, Internal Error
      # Please report this error at
      # http://java.sun.com/cgi-bin/bugreport.cgi
      #
      # Error ID: 47454E45524154452F4F502D41500E4350500265 01
      #
      # Problematic Thread: prio=5 tid=0x8a4e00 nid=0x1b runnable
      #

      % /usr/j2se/bin/java -classpath . ErrorID 47454E45524154452F4F502D41500E4350500265
      generateOopMap.cpp, 613

      Application makes some large table in the memory
      This problem is only on multi CPU machines.
      Customer checked his application on few machines and Solaris verions.
      It looks like it does not depend on architecture (Sparc II/ Sparc III) nor Solaris version (7 nor 8).
      It worked fine on one cpu server.

      Location of core files and explorer output for solaris 7,8:
      /net/uranus.poland/export/mnt/CALLS/36626546
      They are too big to be attched to bug:
      sol7/jdk131_04:
      -rw-r--r-- 1 claudiom answer 800182016 Jul 26 16:07 core
      sol8/jdk131_04:
      -rw-r--r-- 1 claudiom answer 459316856 Jul 26 16:25 core


      Using following startup options:

      java -client -XX:MaxPermSize=256m -Xms512m -Xmx512m -noverify -noclassgc
      -verbose:gc -classpath $JAVA_CLASSPATH
      -Dweblogic.class.path=$WEBLOGIC_CLASSPATH
      -Dweblogic.cluster.multicastAddress=225.0.0.3
      -Dweblogic.system.name=$SYSTEM_NAME
      -Dweblogic.ejb.tempDir=/exp/log/weblogic/$SYSTEM_NAME/tmp_www2
      -Dweblogic.system.home=$CFG_DIR
      -Dweblogic.system.logFile=$LOG_DIR/$SYSTEM_NAME/weblogic.www2.log
      -Dweblogic.httpd.logFileName=$LOG_DIR/$SYSTEM_NAME/access.www2.log
      -Dweblogic.jdbc.logFileName=$LOG_DIR/$SYSTEM_NAME/jdbc.www2.log
      -Dweblogic.home=$WEBLOGIC_HOME -Djava.security.manager
      -Djava.security.policy=$CFG_DIR/weblogic.policy
      -Dcommerce.properties=$CFG_DIR/weblogiccommerce.properties
      -Dweblogic.properties=$CFG_DIR/weblogic.properties
      -Dpipeline.properties=$CFG_DIR/pipeline.properties
      -Dwebflow.properties=$CFG_DIR/webflow.properties weblogic.Server

      Using the sol8 core I see they have tested -server too:

      % strings - core | grep sparc/native_threads/java
      /usr/java/bin/../bin/sparc/native_threads/java -server -XX:MaxPermSize=128m -Xm

      Checked patches for sol8; see also complete report in attachment webpatchdiag-sol8.html
      The following is the list of patches installed now at customer site (sol8) compared to the list of suggested solaris 8 cluster for j2se: http://sunsolve.Central.Sun.COM/cgi/retrieve.pl?doctype=patch&doc=J2SE_Solaris_8_Recommended.README

      Now Cluster
      ============================================================================
      -55 108652-56 X11 6.4.1: Xsun patch
      -11 108921-15 CDE 1.4: dtwm patch
      none 112003-03 SunOS 5.8: Unable to load fontset in 64-bit Solaris 8 iso-1 or iso-15
      -08 108773-15 SunOS 5.8: IIIM and X Input & Output Method patch
      ok 111293-04 SunOS 5.8: /usr/lib/libdevinfo.so.1 patch
      ok 111310-01 SunOS 5.8: /usr/lib/libdhcpagent.so.1 patch
      none 112472-01 SunOS 5.8: Font2DTest2 abort when Lucida Sans Thai Typewriter
      selected
      -05 108714-07 CDE 1.4: libDtWidget patch
      ok 111111-03 SunOS 5.8: /usr/bin/nawk patch
      ok 112396-02 SunOS 5.8: /usr/bin/fgrep patch
      -16 108940-43 Motif 1.2.7 and 2.1.1: Runtime library patch for Solaris 8
      ok 108987-09 SunOS 5.8: Patch for patchadd and patchrm
      ok 108528-15 SunOS 5.8: kernel update patch
      ok 108989-02 SunOS 5.8: /usr/kernel/sys/acctctl and /usr/kernel/sys/exacctsys patch
      -25 108827-26 SunOS 5.8: /usr/lib/libthread.so.1 patch

      Using dbx 7.0 and 1.3.1_04-b02 on a 5.8 Sun-Fire I see they are using lib/lwp/libthread and I get the stack with the following calls GenerateOopMap::compute_map -> GenerateOopMap::init_basic_blocks -> GenerateOopMap::verify_error -> GenerateOopMap::error_work :

      # dbx /usr/j2se/bin/sparc/native_threads/java core
      Reading java
      core file header read successfully
      Reading ld.so.1
      Reading libthread.so.1
      Reading libdl.so.1
      Reading libc.so.1
      Reading libc_psr.so.1
      Reading libjvm.so
      Reading libCrun.so.1
      Reading libsocket.so.1
      Reading libnsl.so.1
      Reading libm.so.1
      Reading libw.so.1
      Reading libmp.so.2
      Reading librt.so.1
      Reading libaio.so.1
      Reading libhpi.so
      Reading libverify.so
      Reading libjava.so
      Reading libzip.so
      Reading libnet.so
      Reading libgen.so.1
      Reading libsched.so.1
      Reading nss_files.so.1
      WARNING!!
      A loadobject was found with an unexpected checksum value.
      See `help core mismatch' for details, and run `proc -map'
      to see what checksum values were expected and found.
      dbx: warning: Some symbolic information might be incorrect.
      detected a multithreaded program
      t@30 (l@30) terminated by signal ABRT (Abort)
      0xff31bdc4: _lwp_kill+0x0010: cmp %o0, 0x5b
      (dbx) pathmap
      (1) -c /tmp_mnt /
      (dbx) proc -map
      Loadobject mappings for current core file:
      0x00010000 /usr/j2se/bin/sparc/native_threads/java
      0xff360000 /usr/lib/lwp/libthread.so.1
        Warning: checksum in file(8177) doesn't match image(8a1a)
      0xff390000 /usr/lib/libdl.so.1
                 is being filtered by: /usr/lib/ld.so.1
      0xff280000 /usr/lib/libc.so.1
                 is being filtered by: /usr/platform/SUNW,Sun-Fire/lib/libc_psr.so.1
        Warning: checksum in file(d661) doesn't match image(314e)
      0xff270000 /usr/platform/SUNW,Ultra-4/lib/libc_psr.so.1
      0xfec00000 /usr/j2se/jre/lib/sparc/server/libjvm.so
      0xff230000 /usr/lib/libCrun.so.1
        Warning: checksum in file(58ba) doesn't match image(ad48)
      0xff200000 /usr/lib/libsocket.so.1
        Warning: checksum in file(b85d) doesn't match image(c331)
      0xfeb00000 /usr/lib/libnsl.so.1
        Warning: checksum in file(283e) doesn't match image(1cac)
      0xff1d0000 /usr/lib/libm.so.1
      0xff260000 /usr/lib/libw.so.1
                 is being filtered by: libc.so.1
      0xff1a0000 /usr/lib/libmp.so.2
      0xff180000 /usr/lib/librt.so.1
        Warning: checksum in file(ea15) doesn't match image(eb91)
      0xff160000 /usr/lib/libaio.so.1
      0xfebd0000 /usr/j2se/jre/lib/sparc/native_threads/libhpi.so
      0xfead0000 /usr/j2se/jre/lib/sparc/libverify.so
      0xfea90000 /usr/j2se/jre/lib/sparc/libjava.so
      0xfea60000 /usr/j2se/jre/lib/sparc/libzip.so
      0xfd810000 /usr/j2se/jre/lib/sparc/libnet.so
      0xfb7e0000 /usr/local/weblogic/weblogic510sp12/lib/solaris/libweblogicunix1.so (not loaded)
      0xd8700000 /usr/local/weblogic/weblogic510sp12/lib/solaris/oci817_8/libweblogicoci36.so (not loaded)
      0xd8000000 /opt/oracle/lib/libclntsh.so.8.0 (not loaded)
      0xfb7c0000 /opt/oracle/lib/libwtc8.so (not loaded)
      0xfb790000 /usr/lib/libgen.so.1
      0xfb6e0000 /usr/lib/libsched.so.1
      0xdb1b0000 /usr/lib/nss_files.so.1
      0xdb190000 /usr/local/weblogic/weblogic510sp12/lib/solaris/libmuxer.so (not loaded)
      0xff3b0000 /usr/lib/ld.so.1 [LM_ID_LDSO]
      (dbx) where
      current thread: t@30
      =>[1] _lwp_kill(0x0, 0x1e, 0x0, 0xff33a000, 0xff386000, 0xff33e428), at 0xff31bdc4
        [2] rand(0x6, 0x0, 0x0, 0xffffffff, 0xff33e3b4, 0x0), at 0xff2cb758
        [3] addsev(0xff33a000, 0xd6d7d648, 0x0, 0x4, 0x0, 0xd6d7d669), at 0xff2b5a7c
        [4] os::abort(0x1, 0xff0ce000, 0x1, 0xff0ce000, 0x6ee660, 0xd6d7d664), at 0xfefc7da4
        [5] report_error(0xdc, 0xd6d7deec, 0x842, 0xff054858, 0xff13e4b0, 0xff0ce000), at 0xfef18c0c
        [6] report_fatal(0x842, 0xff0ce000, 0xff056a20, 0xd6d7eb34, 0xff0e26b4, 0xff0ce000), at 0xfef184dc
        [7] GenerateOopMap::error_work(0xff0ce000, 0xff0564c8, 0xd6d7eb34, 0xff0ce000, 0x24ec758, 0x11ff4c), at 0xfef26444
        [8] GenerateOopMap::verify_error(0xd6d7ed04, 0xff0564c8, 0xd8c, 0xd0e910, 0x1b1a0, 0x1b180), at 0xfef264a4
        [9] GenerateOopMap::init_basic_blocks(0x24ec760, 0xf3dc2ae4, 0xff13fd2c, 0xff1404dc, 0xbfc3, 0x0), at 0xfeddc0a0
        [10] GenerateOopMap::compute_map(0xff0d9698, 0xff0ce000, 0xff0ce000, 0xffffffff, 0x1, 0xd6d7ec04), at 0xfedeb384
        [11] ResolveOopMapConflicts::do_potential_rewrite(0xd6d7ee00, 0xd6d7ee00, 0xd6d7ed04, 0xffffffff, 0xfffffff8, 0xf3dd06a8), at 0xfee0bf0c
        [12] Rewriter::rewrite_method(0xd6d7ee00, 0xff0ce000, 0xd6d7ee04, 0x6ee660, 0xff13fd2c, 0xff1404dc), at 0xfed187f4
        [13] Rewriter::rewrite(0xff1404d0, 0xff1404c8, 0xd6d7ee9c, 0x4, 0x8, 0xf3dc2860), at 0xfed8aca8
        [14] instanceKlass::link_class_impl(0xf3dc2850, 0x6ee660, 0xd6d7f020, 0xff0ce000, 0x6ee660, 0xd6d7ef3c), at 0xfed639d8
        [15] instanceKlass::initialize_impl(0xd6d7f128, 0xf3dcfff0, 0xff0ce000, 0x6ee660, 0x6ee660, 0xd6d7f0c4), at 0xfed4e938
        [16] instanceKlass::initialize(0xf3dcfff0, 0x6ee660, 0x6cb138, 0xff0ce000, 0x6ee660, 0xd6d7f144), at 0xfed2bf6c
        [17] JVM_NewInstance(0xf3dcfff0, 0x6ee660, 0x6ee6ec, 0xff0ce000, 0xff0e4960, 0x0), at 0xfecb9178
        [18] 0xa6604(0xd6d7f334, 0x0, 0xf3dd0118, 0xac6f0, 0x0, 0x109a0), at 0xa6603
        [19] 0xa3790(0xd6d7f3bc, 0xd6d7f460, 0xd6d7f464, 0xaf2fc, 0x4, 0xd6d7f2d0), at 0xa378f
        [20] 0xa3790(0xd6d7f464, 0xd6d7f500, 0xd6d7f504, 0xaf228, 0x4, 0xd6d7f350), at 0xa378f
        [21] 0xa3790(0xd6d7f504, 0x0, 0xd6d7f594, 0xaf120, 0x4, 0xd6d7f3f8), at 0xa378f
        [22] 0xa376c(0xd6d7f59c, 0xd6d7f638, 0xd6d7f63c, 0xaf120, 0x4, 0xd6d7f490), at 0xa376b
        [23] 0xa376c(0xd6d7f644, 0x1dec430e, 0xd6d7f648, 0xaf2fc, 0xc, 0xd6d7f530), at 0xa376b
        [24] 0xa376c(0xd6d7f6dc, 0x0, 0xd6d7f77c, 0xaf120, 0xc, 0xd6d7f5d8), at 0xa376b
        [25] 0xa3790(0xd6d7f784, 0xd6d7f788, 0xd6d7f788, 0xaf120, 0xc, 0xd6d7f670), at 0xa378f
        [26] 0xa376c(0xd6d7f834, 0xd6d7f8cc, 0xd6d7f8d0, 0xaf120, 0xc, 0xd6d7f718), at 0xa376b
        [27] 0xa376c(0xd6d7f8dc, 0xd6d7f968, 0xd6d7f96c, 0xaf120, 0xc, 0xd6d7f7c8), at 0xa376b
        [28] 0xa376c(0xd6d7f974, 0x1dec430b, 0xff0ce000, 0xaf120, 0x10, 0xd6d7f870), at 0xa376b
        [29] 0xa376c(0xd6d7fa34, 0x198, 0xd6d7fac4, 0xaf120, 0xc, 0xd6d7f908), at 0xa376b
        [30] 0xa3790(0xd6d7fac4, 0xe085b0d8, 0xd6d7fb50, 0xaf120, 0x8, 0xd6d7f9d0), at 0xa378f
        [31] 0xa3790(0xd6d7fb54, 0xf374ada8, 0xd6d7fb58, 0xaf120, 0x4, 0xd6d7fa60), at 0xa378f
        [32] 0xa38d4(0xd6d7fbdc, 0x1, 0xff0dca68, 0xaf494, 0x8, 0xd6d7fae8), at 0xa38d3
        [33] 0xff138048(0xd6d7fc68, 0xd6d7fea0, 0xa, 0xf3770260, 0x4, 0xd6d7fb80), at 0xff138047
        [34] JavaCalls::call_helper(0xd6d7fe98, 0xff0ce000, 0xd6d7fde4, 0x6ee660, 0xa55bc, 0xd6d7fea0), at 0xfeccd0c8
        [35] JavaCalls::call_virtual(0xf3770540, 0xd6d7fdd0, 0xd6d7fdd4, 0xff0ce000, 0xd6d7fe98, 0xd6d7fde4), at 0xfedf82e8
        [36] JavaCalls::call_virtual(0xd6d7fe98, 0xd6d7fe94, 0xd6d7fe90, 0xd6d7fe84, 0xd6d7fe7c, 0x6ee660), at 0xfedfe2b8
        [37] thread_entry(0xf3417f78, 0x6ee660, 0xff0ce000, 0xd6d7ffa0, 0x1e, 0xe), at 0xfee1caec
        [38] JavaThread::run(0xd6d00000, 0xff0d9e1c, 0xff0ce000, 0x80000, 0x6ee660, 0x80000), at 0xfee175ec
        [39] _start(0xff0ce000, 0xfebc3800, 0x0, 0x0, 0x0, 0x0), at 0xfee15320
      (dbx) threads
            t@1 a l@1 ?() LWP suspended in ___lwp_cond_wait()
            t@2 b l@2 _start() LWP suspended in ___lwp_cond_wait()
            t@3 b l@3 _start() LWP suspended in ___lwp_cond_wait()
            t@4 b l@4 _start() LWP suspended in ___lwp_cond_wait()
            t@5 b l@5 _start() LWP suspended in _poll()
            t@6 b l@6 _start() LWP suspended in ___lwp_cond_wait()
            t@7 b l@7 _start() LWP suspended in ___lwp_cond_wait()
            t@8 b l@8 _start() LWP suspended in ___lwp_cond_wait()
            t@9 b l@9 _start() LWP suspended in ___lwp_cond_wait()
           t@10 b l@10 _start() LWP suspended in ___lwp_cond_wait()
           t@11 b l@11 _start() LWP suspended in ___lwp_cond_wait()
           t@12 b l@12 _start() LWP suspended in ___lwp_cond_wait()
           t@13 b l@13 _start() LWP suspended in ___lwp_cond_wait()
           t@14 b l@14 _start() LWP suspended in ___lwp_cond_wait()
           t@15 b l@15 _start() LWP suspended in ___lwp_cond_wait()
           t@16 b l@16 _start() LWP suspended in ___lwp_cond_wait()
           t@17 b l@17 _start() LWP suspended in ___lwp_cond_wait()
           t@18 b l@18 _start() LWP suspended in ___lwp_cond_wait()
           t@19 b l@19 _start() LWP suspended in ___lwp_cond_wait()
           t@20 b l@20 _start() LWP suspended in ___lwp_cond_wait()
           t@21 b l@21 _start() LWP suspended in ___lwp_cond_wait()
           t@22 b l@22 _start() LWP suspended in ___lwp_cond_wait()
           t@23 b l@23 _start() LWP suspended in ___lwp_cond_wait()
           t@24 b l@24 _start() LWP suspended in ___lwp_cond_wait()
           t@25 b l@25 _start() LWP suspended in ___lwp_cond_wait()
           t@26 b l@26 _start() LWP suspended in ___lwp_cond_wait()
           t@27 b l@27 _start() LWP suspended in ___lwp_cond_wait()
           t@28 b l@28 _start() LWP suspended in ___lwp_cond_wait()
           t@29 b l@29 _start() LWP suspended in ___lwp_cond_wait()
      o> t@30 b l@30 _start() signal SIGABRT in _lwp_kill()
           t@31 b l@31 _start() LWP suspended in ___lwp_cond_wait()
           t@32 b l@32 _start() LWP suspended in ___lwp_cond_wait()
           t@33 b l@33 _start() LWP suspended in ___lwp_cond_wait()
           t@34 b l@34 _start() LWP suspended in ___lwp_cond_wait()
           t@35 b l@35 _start() LWP suspended in ___lwp_cond_wait()
           t@36 b l@36 _start() LWP suspended in ___lwp_cond_wait()
           t@37 b l@37 _start() LWP suspended in ___lwp_cond_wait()
           t@38 b l@38 _start() LWP suspended in _poll()
           t@39 b l@39 _start() LWP suspended in ___lwp_mutex_lock()
           t@40 b l@40 _start() LWP suspended in ___lwp_mutex_lock()
           t@41 b l@41 _start() LWP suspended in _poll()
           t@42 b l@42 _start() LWP suspended in ___lwp_cond_wait()
           t@43 b l@43 _start() LWP suspended in ___lwp_cond_wait()
           t@47 b l@47 _start() LWP suspended in _so_accept()
           

            robm Robert Mckenna
            cmassi Claudio Massi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:
              Imported:
              Indexed: