JVM crash during C2 compilation/PhaseIdealLoop: PrintCompilation
tells us the method being compiled is com/ibm/sslite/g.c

This is an obfuscated class in IBM's security library, a jar which
has caused problems in a few other bug reports, e.g. 4724509.

No problem in -client, or in 1.4.x.

  [11] libthread.so.1:sigacthandler(0xb, 0xe6501d98, 0x0, 0x0, 0x0, 0xff38c000), at 0xff37811c
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [12] libc_psr.so.1:_memcpy(0x815b2000, 0x415b2040, 0x3fffe060, 0x415b2040, 0x7, 0x0), at 0xff3406f0
  [13] libjvm.so:Arena::Arealloc(0x815b00a0, 0x815b00a0, 0x40000000, 0x80000000, 0xfe4dc628, 0x40000000), at 0xfe0bfbf0
  [14] libjvm.so:Node_Array::grow(0xe6500ef0, 0x0, 0x10000000, 0xe6500ef0, 0x12cb2c4, 0xe6501260), at 0xfe0e5898
  [15] libjvm.so:PhaseIdealLoop::build_loop_late(0x0, 0x0, 0x12cca0c, 0xe0b8a8, 0xe6500f70, 0x12cca0c), at 0xfe0bc9f8
  [16] libjvm.so:PhaseIdealLoop::PhaseIdealLoop(0xe6500ef0, 0x3f8224, 0x0, 0xc6f00, 0x1, 0xe6500fb0), at 0xfe196920
  [17] libjvm.so:Compile::Optimize(0xe65017a0, 0xe65015d8, 0xe65017a0, 0xe65015fc, 0x0, 0xe65013fc), at 0xfe1cf444
  [18] libjvm.so:Compile::Compile(0x10952c8, 0xe6501820, 0x12a04cc, 0xfe53df2c, 0xe65018c0, 0xe65018d0), at 0xfe1ce190
  [19] libjvm.so:C2Compiler::compile_method(0x27ad0, 0xe6501b1c, 0x12a04cc, 0x12a03e0, 0xffffffff, 0x1), at 0xfe1c9d68
  [20] libjvm.so:CompileBroker::invoke_compiler_on_method(0x12a03e0, 0x12a04cc, 0xfe4ff5a0, 0x0, 0x0, 0x1ff), at 0xfe1ca36c
  [21] libjvm.so:CompileBroker::compiler_thread_loop(0x28c20, 0xf4f50, 0xfe4dc628, 0xe6501d38, 0xfe4dc628, 0xffffffff), at 0xfe2862a8
  [22] libjvm.so:JavaThread::run(0xe6302000, 0xfe4e84f0, 0xfe4dc628, 0x1ffd98, 0xf4f50, 0x1ffd98), at 0xfe219270
  [23] libjvm.so:_start(0xfe4dc628, 0xff1d5d38, 0x0, 0x5, 0x1, 0xfe401000), at 0xfe216f98

Java heap size is only 256mb, but the core file is 2.4gb:

Why is this core so big...? (and why does it compress so well, down to 20-something mb..)

The native heap is what makes it large:

  3: 00026000 - 815B2000 = 2119216K (264902 pages) [heap]

Why so large a C heap? The number "0x01315434" is written in the heap starting here...

0x017b0ae0: 0x012cc900 0x01315434 0x01315434 0x01315434
0x017b0af0: 0x01315434 0x01315434 0x01315434 0x01315434
0x017b0b00: 0x01315434 0x01315434 0x01315434 0x01315434

...and it goes right to the end of the heap...

2 GIGABYTES of it.... This explains the large core file, and it's very regular so
compresses well.

That number is a pointer to a JVM object:

dbx>x 0x01315434/32X
0x01315434: 0xfe533230 0x01315454 0x00000003 0x00000003
0x01315444: 0x0000130b 0x00000000 0x000b0000 0x00001071
0x01315454: 0x01315434 0x012cca0c 0x01315460 0xfe52e368
0x01315464: 0x0131547c 0x00000001 0x00000001 0x0000130c
0x01315474: 0x012b36fd 0x00000001 0x01315268 0x00000000
0x01315484: 0x00000000 0x0017a014 0x00000000 0x0017c6ec
0x01315494: 0x00000000 0x0017c698 0x00000000 0x0017c644
0x013154a4: 0x00000000 0x0017c5ec 0x00000000 0x0017c598

dbx>x 0xfe533230
0xfe533230: __1cPCountedLoopNodeG__vtbl_ : 0xfe4b7a34
dbx>dem __1cPCountedLoopNodeG__vtbl_

__1cPCountedLoopNodeG__vtbl_ == CountedLoopNode::__vtbl

By sneaky trick, that object looks like:

dbx>print -r *(CountedLoopNode*) XXXXXXX
*((class CountedLoopNode *) XXXXXXXX) = {
    CountedLoopNode::LoopNode::RegionNode::Node::_in = 0x1315454
    CountedLoopNode::LoopNode::RegionNode::Node::_cnt = 3U
    CountedLoopNode::LoopNode::RegionNode::Node::_max = 3U
    CountedLoopNode::LoopNode::RegionNode::Node::_idx = 4875U
    CountedLoopNode::LoopNode::RegionNode::Node::_pad_to_16_bytes = 0
    CountedLoopNode::LoopNode::RegionNode::Node::_debug_idx = 720896
    CountedLoopNode::LoopNode::RegionNode::Node::NotAMachineReg = 4294901760U
    CountedLoopNode::LoopNode::_flags = 0
    CountedLoopNode::_main_idx = 20010036U
}


This object may or may not be the cause of the problem, but it's address is printed all over the heap...


-------------

Extracting the class that causes a problem, and it's dependencies, these classes also show
a problem with java_g: we don't get as far as massively overwriting the heap, we hit an assertion
earlier:

java_g -server -Xbatch -Xbootclasspath/p:ibm.jar -XX:+CompileTheWorld

...where ibm.jar contains the required ibm classes.


(5.9)$ java_g -server -Xbatch -Xbootclasspath/p:ibm.jar -XX:+CompileTheWorld

VM option '+CompileTheWorld'
CompileTheWorld : Compiling all classes in /opt/calls/64018573.citibank/Test/ibm.jar

CompileTheWorld (1) : com/ibm/sslite/g
Preloading failed for (1) com/ibm/sslite/g
#
# HotSpot Virtual Machine Error, assertion failure
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# assert(found || old == du->C->_top_node, "missing du edge")
#
# Error ID: /export/sail1/jdk131-update/ws/fcs/hotspot/src/share/vm/opto/phaseX.cpp, 1540 [ Patch
ed ]
#
# Problematic Thread: prio=5 tid=0x13d240 nid=0x9 runnable
#
Dumping core....
Abort(coredump)



The assertion matches with:

4724509 [1.3.1_04] Exception in "CompileThread0" java.lang.OutOfMemoryError: alloc 512MB

...which became non-reproducible.

The testcase here is reproducible with 1.3.1_10 (the customer's version), FCS ad also the latest 1.3.1_11.