Hotspot server VM in 1.3.1_06 crashed because of compiler thread failure

Hotspot server VM in 1.3.1_06 crashed at our customer site.

CONFIGURATION :
OS : Solaris 9
VM : Hotspot Server VM 1.3.1_06

The followings are the stack trace at the crash.

---------------------------------------------
.......
#8 <signal handler called>
#9 0xfecbf864 in const Type*Type::meet(const Type*)const
(0x1770c94, 0xc39ff384, 0xc39ff384, 0xff11c100, 0xc39ff384, 0x0)
#10 0xfeddee78 in const Type*CMoveNode::Value(PhaseTransform*)const
(0x1770c94, 0xc39ff384, 0xc39ff384, 0xff11c100, 0xc39ff384, 0x0)
#11 0xff08db9c in Node*PhaseIterGVN::transform_old(Node*,int)
(0xc39ff384, 0x1, 0xc39ff384, 0x1, 0x63, 0x2000000)
#12 0xfed7c204 in void PhaseIterGVN::optimize()
(0x20, 0x0, 0xff11c100, 0xc39ff350, 0xc39ff340, 0x0)
#13 0xfed9d4ac in PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&,const
PhaseIdealLoop*) (0xc39ff128,
0x19fc7d4, 0x1, 0x19fede4, 0x4000, 0x180cc10)
#14 0xfedd6924 in void Compile::Optimize()
(0xff180538, 0xc39ff7f8, 0x1998198, 0xc39ff9c0, 0x19a96ec, 0xc39ff61c)
#15 0xfedd56b0 in Compile::Compile(ciEnv*,ciScope*,ciMethod*,int,long,long)
(0xff180538, 0x198b2ac, 0x11c7e64, 0xff180570, 0xc39ff81c, 0xc39ffb08)
......
---------------------------------------------


INVESTIGATION:
The crash scenario is ,

The value of "phase->type(in(IfFalse))" in connode.cpp becomes zero
in some reason.
That causes the program fails to refer meet() function.
Then the compiler thread terminates abnormally.

- CMoveNode.in(IfTrue) was ConNode
- CMoveNode.in(IfFalse) was CMoveINode

src/share/vm/opto/type.cpp
------------------------------------------------------------------
const Type *Type::meet( const Type *t ) const {
const Type *mt = xmeet(t);
.......
return mt;
}
-------------------------------------------------------------------

src/share/vm/opto/connode.cpp
-------------------------------------------------------------------
// Result is the meet of inputs
const Type *CMoveNode::Value( PhaseTransform *phase ) const {
const Type *t = phase->type(in(IfFalse))->meet(phase->type(in(IfTrue))); <==(a)
return phase->widen(t, phase->_types[_idx]); // Widen once per Phi
}
--------------------------------------------------------------------

The behavior is very similar to bugid:4807962.


APPENDIX 1 :

The followings are the log when we investigated core file.

-------------------------------------------------------------------------------
(gdb) frame 10
#10 0xfeddee78 in const Type*CMoveNode::Value(PhaseTransform*)const
(0x1770c94, 0xc39ff384, 0xc39ff384, 0xff11c100, 0xc39ff384, 0x0)
(gdb) disassemble
Dump of assembler code for function CMoveNode::Value(PhaseTransform*):
0xfeddee44 <>: save %sp, -96, %sp
0xfeddee48 <+4>: ld [ %i0 + 4 ], %o0 // %o0(0x01770cb0) is Node._in
0xfeddee4c <+8>: mov %i1, %i2
0xfeddee50 <+12>: ld [ %i1 + 0x24 ], %g2 // %g2(0x0141c070) is
PhaseTransform::_types._types
0xfeddee54 <+16>: ld [ %o0 + 8 ], %o1 // %o1(0x01770c94) is in(IfFalse)
0xfeddee58 <+20>: ld [ %o0 + 0xc ], %o0 // %o0(0x01770c94) is in(IfTrue)
0xfeddee5c <+24>: ld [ %o1 + 0x10 ], %g3 // %g3(0x5337) is in(IfFasle)->_idx
0xfeddee60 <+28>: ld [ %o0 + 0x10 ], %g4
0xfeddee64 <+32>: sll %g3, 2, %g3
0xfeddee68 <+36>: ld [ %g2 + %g3 ], %o0 // %o0(0x0) is
phase->type(in(IfFalse))
0xfeddee6c <+40>: sll %g4, 2, %g4
0xfeddee70 <+44>: call 0xfecbf864 <const Type*Type::meet(const Type*)const>
0xfeddee74 <+48>: ld [ %g2 + %g4 ], %o1

The "phase->type()" in connode.cpp is corresponding
to the line, PhaseTransform::_types._types[] .
Data has been set before index (0x5337) of in(IfFasle)->_idx
But after the index, data has not.(all the data is 0)

Please see the following list.

[%g2]0x0141c070 [%g3]0x00014cdc
(gdb) x/20 0x0141c070+0x00014cdc phase->type(in(IfFalse))
0x1430d4c: 0x00000000 0x00000000 0x00000000 0x00000000
          ^^^^^^^^^^
           The data from here is set to 0.
0x1430d5c: 0x00000000 0x00000000 0x00000000 0x00000000
0x1430d6c: 0x00000000 0x00000000 0x00000000 0x00000000
0x1430d7c: 0x00000000 0x00000000 0x00000000 0x00000000
0x1430d8c: 0x00000000 0x00000000 0x00000000 0x00000000

As to data before the above block, data seems set correctly.

(gdb) x/20 0x0141c070+0x00014cdc-80
0x1430cfc: 0x000ec560 0x000ec560 0x000ec560 0x000ec560
0x1430d0c: 0x0147fc60 0x0147fc60 0x000ec0d8 0x0147fc60
0x1430d1c: 0x0147fc60 0x0147fc60 0x0147fc60 0x0147fc60
0x1430d2c: 0x000ec0d8 0x000ec0d8 0x000ec0d8 0x000ec0d8
0x1430d3c: 0x000ec0d8 0x000ec370 0x000ec0d8 0x000ec0d8

As the below dump information says,
in(IfFalse) is CMoveINode
in(IfTrue) is ConNode

(gdb) p/x $i0 (this)
$6 = 0x1770c94
(gdb) x/8wx $i0
0x1770c94: 0xff16c928 0x01770cb0 0x00000004 0x00000004
                     ^^^^^^^^^^^_in
0x1770ca4: 0x00005337 0x00000000 0x000ec560 0x00000000
(gdb) x/8wx 0x01770cb0 _in
0x1770cb0: 0x00000000 0x019fc7d4 0x01770c94 0x019fc7d4
0x1770cc0: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/4wx 0x01770c94 in(IfFalse)
0x1770c94: 0xff16c928 0x01770cb0 0x00000004 0x00000004
0x1770ca4: 0x00005337 0x00000000 0x000ec560 0x00000000
           ^^^^^^^^^_idx
(gdb) x/x 0xff16c928
0xff16c928 <CMoveINode::__vtbl>: 0xff0f2a44

(gdb) x/4wx 0x019fc7d4
0x19fc7d4: 0xff16b628 0x019fc7f0 0x00000001 0x00000001
(gdb) x/x 0xff16b628
0xff16b628 <ConNode::__vtbl>: 0xff0f20d8


APPENDIX2 :

We track the same process in source code.
The process of phase->type(in(IfFalse)) and the process where crash occurs
are as follows.

-- process of phase->type(in(IfFalse)) --
1) PhaseIdealLoop::PhaseIdealLoop()
2) PhaseIdealLoop::split_if_with_blocks()
3) PhaseIdealLoop::conditional_move()
4) CMoveNode::make()

The type is set by calling
register_new_node( cmov, cmov_ctrl );
after CMoveNode::make().

-- the process where crash occurs --
Abnormal terminations occurs after the above
PhaseIdealLoop::split_if_with_blocks()
and following 1) - 4)
1') PhaseIdealLoop::PhaseIdealLoop()
2')PhaseIterGVN::optimize()
3')PhaseIterGVN::transform_old()
4')CMoveNode::Value()