Hi,
                                                                          
I sometimes get a crash during deoptimzation in a debug build of the
Hotspot 1.5.0_11. The problem is that an invalid oop is extracted and
the VM crashes in an assertion when that invalid oop is stored in a
handle. The opt version crashes too in the GC when the invalid oop is
processed, but this happens naturally less often. You can reproduce this
with the attached program.

The crash can be reproduced on either Linux/x64 and Solaris x64.

Please compile the attached program and run it with bash:
                                                                          
while true; do java_g -agentlib:jdwp=transport=dt_socket,server=y,
address=8000,suspend=n -XX:+ShowMessageBoxOnError DeoptBugTest; done
                                                                          
It will probably take a few minutes (5 - 30) until the error pops up. The active
debugging at least makes the bug more likely to appear.

Solaris stacktrace:

  [1] _read(0x0, 0xb4126044, 0x10), at 0xfef50a27
  [2] read(0x0, 0xb4126044, 0x10), at 0xfef441a2
  [3] os::message_box(0xfeb6a106, 0xfecbf948), at 0xfe6793a9
  [4] VMError::show_message_box(0xb41261b4, 0xfecbf948, 0x7d0), at 0xfe89c814
  [5] VMError::report_and_die(0xb41261b4), at 0xfe89b243
  [6] report_assertion_failure(0xfe9e12b8, 0x12, 0xfe9e12f9), at 0xfe243d61
=>[7] HandleArea::allocate_handle(0x820f498, 0xf0819eb0), at 0xfe3350c7
  [8] Handle::Handle(0xb4126280, 0xf0819eb0), at 0xfe896742
  [9] compiledVFrame::create_stack_value(0x820f0f0, 0x820f260), at 0xfe894828
  [10] compiledVFrame::locals(0x820f0f0), at 0xfe893e1d
  [11] vframeArrayElement::fill_in(0x82c3638, 0x820f0f0), at 0xfe8901c3
  [12] vframeArray::fill_in(0x82c3500, 0x8212280, 0xe, 0x820f078, 0xb4126820, 0x0), at 0xfe8913b0
  [13] vframeArray::allocate(0x8212280, 0xe, 0x820f078, 0xb4126820, 0xb4126b04, 0xf7802e71, 0xb4126b24, 0xb4126ad8, 0xf78af6c0, 0x8212280, 0xb4126aa0, 0xf78c43e0, 0xb86009c
8, 0xb4126b04, 0xf7802e71, 0xb4126b24), at 0xfe891322
  [14] Deoptimization::create_vframeArray(0x8212280, 0xb4126aa0, 0xf78c43e0, 0xb86009c8, 0xb4126820), at 0xfe257070
  [15] Deoptimization::fetch_unroll_info_helper(0x8212280), at 0xfe25557d
  [16] Deoptimization::uncommon_trap(0x8212280, 0xffffffb5, 0x0, 0xfebcdf34, 0x2a92a4e, 0x24), at 0xfe25944a
  [17] 0xf78ad519(0x31, 0xb46259e8, 0xb8600b00, 0x0, 0x0, 0x0), at 0xf78ad519


Linux stacktrace:
                                                                          
HandleArea::allocate_handle at handles.cpp:18
Handle at handles.inline.hpp:18
compiledVFrame::create_stack_value at vframe_hp.cpp:208
compiledVFrame::locals at vframe_hp.cpp:40
vframeArrayElement::fill_in at vframeArray.cpp:63
vframeArray::fill_in at vframeArray.cpp:416
vframeArray::allocate at vframeArray.cpp:402
Deoptimization::create_vframeArray at deoptimization.cpp:675
Deoptimization::fetch_unroll_info_helper at deoptimization.cpp:149
Deoptimization::uncommon_trap at deoptimization.cpp:1417


The Test-Systems i used are:

Solaris:

SunOS shapeshifter 5.10 Generic_118855-33 i86pc i386 i86pc
v40z
                        Solaris 10 11/06 s10x_u3wos_10 X86
           Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 14 November 2006

Linux:

Linux baldur 2.6.17-10-generic #2 SMP Fri Oct 13 15:34:39 UTC 2006 x86_64 GNU/Linux
Ubuntu 6.10 AMD64
Since this CR seems to hate file attachments, here's the java-proggy inline:

public class DeoptBugTest {

    public static void main(String[] args) throws InterruptedException {
        // Start the alloction threads first.
        Thread[] allocators = new Thread[10];
        
        for (int i = 0; i < allocators.length; ++i) {
            allocators[i] = new Thread(new Runnable() {

                public void run() {
                    while (true) {
                        byte[] dummy = new byte[(int) (Math.random() * 2 * 1024)];
                        
                        if ((dummy != null) && (Math.random() < 0.00001)) {
                            Runtime.getRuntime().halt(0);
                        }
                    }
                }
            });
        }

        for (int i = 0; i < allocators.length; ++i) {
            allocators[i].start();
        }

        for (int i = 0; i < allocators.length; ++i) {
            allocators[i].join();
        }
    }
}