java/lang/StackWalker/LocalsAndOperands.java crashes when run with -XX:+UseJVMCICompiler with g1 and parallel, with and without class unloading.

Stack trace is typically like this:

V [libjvm.dylib+0x6ea177] G1ParScanThreadState::copy_to_survivor_space(G1HeapRegionAttr, oop, markWord)+0x67
V [libjvm.dylib+0x68585c] void G1ParScanThreadState::do_oop_evac<unsigned int>(unsigned int*)+0x1dc
V [libjvm.dylib+0x6e755b] G1ParScanThreadState::trim_queue_to_threshold(unsigned int)+0x15b
V [libjvm.dylib+0x6e9dca] G1ParScanThreadState::trim_queue()+0x1a
V [libjvm.dylib+0x680bbb] G1ParEvacuateFollowersClosure::do_void()+0x6b
V [libjvm.dylib+0x689ab4] G1EvacuateRegionsBaseTask::evacuate_live_objects(G1ParScanThreadState*, unsigned int, G1GCPhaseTimes::GCParPhases, G1GCPhaseTimes::GCParPhases)+0x94
V [libjvm.dylib+0x68984f] G1EvacuateRegionsBaseTask::work(unsigned int)+0x14f
V [libjvm.dylib+0x1105f80] GangWorker::run_task(WorkData)+0x60
V [libjvm.dylib+0x110603c] GangWorker::loop()+0x2c

Investigation showed that the broken oop has a invalid/broken Klass; it is always part of (first and only element of) an objArrayOop, and the test exercises code to generate and retrieve java thread stack frames

Attached patch adds an oop_iterate() call (in check_obj()) on all oops passed to user data structures when filling in stack frames; that crashes as follows, indicating some race with MonitorInfo::_owner.

V [libjvm.so+0x15a8ed8] check_obj(oop)+0x88
V [libjvm.so+0x15aa5db] LiveFrameStream::monitors_to_object_array(GrowableArray<MonitorInfo*>*, Thread*)+0xfb
V [libjvm.so+0x15aa916] LiveFrameStream::fill_live_stackframe(Handle, methodHandle const&, Thread*)+0xe6
V [libjvm.so+0x15aad5f] LiveFrameStream::fill_frame(int, objArrayHandle, methodHandle const&, Thread*)+0x12f
V [libjvm.so+0x15ab8a1] StackWalk::fill_in_frames(long, BaseFrameStream&, int, int, objArrayHandle, int&, Thread*)+0x361
V [libjvm.so+0x15ac575] StackWalk::fetchFirstBatch(BaseFrameStream&, Handle, long, int, int, int, objArrayHandle, Thread*)+0x2b5
V [libjvm.so+0x15aca20] StackWalk::walk(Handle, long, int, int, int, objArrayHandle, Thread*)+0x120
V [libjvm.so+0xe8d923] JVM_CallStackWalk+0x1d3

[The attached patch also adds some additional code to verify eden space (where the original crash always occurred) by calling oop_iterate() on all objects in it - it always crashes with above or quite similar stack trace *before* GC operation starts - i.e. the bad oop is created outside of gc. This verification code is needed because VerifyBeforeGC otherwise does not check Klasses.]

[~dlong] mentioned that this crash occurs in jdk 11/15/16; I only ever tried 15 and 16.

The crash also looks very much the recent crashes caused by integration of JDK-8244603 (later backed out in JDK-8248650).

Reproduction with that test and -XX:+UseJVMCICompiler is ~1/100 on jdk16 (also without the patch).

Kindly asking for investigation by the runtime team.