Customer Problem Description:

Our program is an automatic stock-trading system, and for the most part runs unattended.

We have only been running the Client VM. We have not tried Server mode.
The production systems are using the default install of the JRE, which does not
seem to have a -server option. Running in pure interpreted mode is not feasible because we barely have enough speed as it is in production.

With the instrumented jvm.dll for 1.3.1_05 C1 in place, we have seen
the following crash twice at the same point(see attached core files
(hoyt2-20021016-tw.zip, hoyt3-20021014-tw1.zip ) :

ChildEBP RetAddr Args to Child
02defd08 080e7798 02defd8c 030d7ec8 02defd98
jvm!Scavenge::scavenge_oop_with_check+0x28
[F:\1.3.1_05\hotspot\src\share\vm\memory\scavenge.cpp @ 105]
02defd28 0806570c 02defd8c 00af71d0 02defd98 jvm!OopMapSet::oops_do+0x28
[F:\1.3.1_05\hotspot\src\share\vm\compiler\oopMap.cpp @ 349]
02defd4c 08065b5c 080f9f80 02defd98 080f9f80
jvm!frame::oops_code_blob_do+0x2c
[F:\1.3.1_05\hotspot\src\share\vm\runtime\frame.cpp @ 630]
02defd60 08115f6b 080f9f80 02defd98 080f9f80 jvm!frame::oops_do+0x6c
[F:\1.3.1_05\hotspot\src\share\vm\runtime\frame.cpp @ 741]
02defdd0 08118028 080f9f80 00000000 00000000 jvm!JavaThread::oops_do+0xbb
[F:\1.3.1_05\hotspot\src\share\vm\runtime\thread.cpp @ 1791]
02defde0 080facda 080f9f80 080f9f80 080f9f80 jvm!Threads::oops_do+0x18
[F:\1.3.1_05\hotspot\src\share\vm\runtime\thread.cpp @ 2601]
02defe8c 081250af 0008cd94 00000000 0643f984
jvm!Scavenge::invoke_at_safepoint+0x50a
[F:\1.3.1_05\hotspot\src\share\vm\memory\scavenge.cpp @ 558]
02defe9c 08124da6 00830040 0643f958 0812728c jvm!VM_Scavenge::doit+0xf
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vm_operations.cpp @ 98]
02defec8 08124593 00830040 0643f958 0812728c jvm!VM_Operation::evaluate+0x76
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vm_operations.cpp @ 26]
02defef4 0812480b 0643f958 0082fc40 02bfcda8
jvm!VMThread::evaluate_operation+0x53
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vmThread.cpp @ 222]
02bfcda8 00000000 00000000 00000000 00000000 jvm!VMThread::loop+0x1ab
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vmThread.cpp @ 298]

ChildEBP RetAddr
02defd08 080e7798 jvm!Scavenge::scavenge_oop_with_check(oopDesc** p =
02defd8c )+0x28 [F:\1.3.1_05\hotspot\src\share\vm\memory\scavenge.cpp @ 105]
02defd28 0806570c jvm!OopMapSet::oops_do(frame* fr = 02defd8c , CodeBlob* cb
= 00af71d0 , RegisterMap* reg_map = 02defd98 , <function>* f = 080f9f80
)+0x28 [F:\1.3.1_05\hotspot\src\share\vm\compiler\oopMap.cpp @ 349]
02defd4c 08065b5c jvm!frame::oops_code_blob_do(<function>* f = 080f9f80 ,
RegisterMap* reg_map = 02defd98 )+0x2c
[F:\1.3.1_05\hotspot\src\share\vm\runtime\frame.cpp @ 630]
02defd60 08115f6b jvm!frame::oops_do(<function>* f = 080f9f80 , RegisterMap*
map = 02defd98 )+0x6c [F:\1.3.1_05\hotspot\src\share\vm\runtime\frame.cpp @
741]
02defdd0 08118028 jvm!JavaThread::oops_do(<function>* f = 080f9f80 )+0xbb
[F:\1.3.1_05\hotspot\src\share\vm\runtime\thread.cpp @ 1791]
02defde0 080facda jvm!Threads::oops_do(<function>* f = 080f9f80 )+0x18
[F:\1.3.1_05\hotspot\src\share\vm\runtime\thread.cpp @ 2601]
02defe8c 081250af jvm!Scavenge::invoke_at_safepoint(int size_to_be_allocated
= 0x8cd94, long deferred = 0, long* notify_ref_lock = 0643f984 )+0x50a
[F:\1.3.1_05\hotspot\src\share\vm\memory\scavenge.cpp @ 558]
02defe9c 08124da6 jvm!VM_Scavenge::doit()+0xf
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vm_operations.cpp @ 98]
02defec8 08124593 jvm!VM_Operation::evaluate()+0x76
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vm_operations.cpp @ 26]
02defef4 0812480b jvm!VMThread::evaluate_operation(VM_Operation* op =
0643f958 )+0x53 [F:\1.3.1_05\hotspot\src\share\vm\runtime\vmThread.cpp @
222]
02bfcda8 00000000 jvm!VMThread::loop()+0x1ab
[F:\1.3.1_05\hotspot\src\share\vm\runtime\vmThread.cpp @ 298]

Again the oop is bad.It's value is 0x841, which does not make a
good pointer. This time the jvm!CarTable::_table_base is 0084feb0,
and there is memory backing that address, so the should_scavenge check
happens to succeed, and we AV while trying to evaluate obj->is_forwarded().

080f9f9f 8b7cc114 mov edi,[ecx+eax*8+0x14]
080f9fa3 85ff test edi,edi
080f9fa5 5f pop edi
080f9fa6 741d jz jvm!Scavenge::scavenge_oop_with_check+0x45
(080f9fc5)
080f9fa8 8b02 mov eax,[edx]
ds:0023:00000841=???????? <------
080f9faa 8bc8 mov ecx,eax
080f9fac 83e103 and ecx,0x3
080f9faf 80f903 cmp cl,0x3
080f9fb2 7506 jnz jvm!Scavenge::scavenge_oop_with_check+0x3a
(080f9fba)

void Scavenge::scavenge_oop_with_check(oop* p) {
  oop obj = *p;
  if (should_scavenge(obj)) {
    if (obj->is_forwarded()) {
      *p = obj->forwardee();
    } else {
      *p = obj->copy_to_survivor_space(NULL);
    }
  }
}

inline bool Scavenge::should_scavenge(oop p) {
  return p == NULL ? false : CarTable::should_scavenge(p);
}


  static bool CarTable::should_scavenge(oop p) { return
desc_for((oop*)p)->should_scavenge(); }


  static CarTableDesc* CarTable::desc_for(oop* p) {
    CarTableDesc* entry = &(_table_base[(juint) p >> LogOfCarSpaceSize]);
    assert(entry >= _table && entry < _table + _table_size, "Invalid oop");
    return entry;
  }
The "this" pointer for JavaThread::oops_do is 035ad300 for the 2002-10-14
crash, and 032fa5d0 for the 2002-10-16 crash.