DirectNativeCallWrapper::verify_resolve_call fails with "assert(db != __null && !db->is_adapter_blob()) failed: must use stub!"

The following test failed in the JDK20 CI:

runtime/cds/appcds/redefineClass/RedefineRunningMethods_Shared.java

Here's a snippet from the log file:

----------System.err:(155/12572)----------
 stdout: [The class is shared as expected: RedefineRunningMethods_Shared
The class is shared as expected: RedefineRunningMethods_SharedHelper
The class is shared as expected: RedefineRunningMethods
The class is shared as expected: RedefineRunningMethods_B
The class is shared as expected: RedefineClassHelper
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler$FileManagerWrapper
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler$FileManagerWrapper$1
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler$MemoryJavaFileObject
[2.624s][trace][redefine,class,iklass,add] adding previous version ref for RedefineRunningMethods_B, EMCP_cnt=4
[2.624s][trace][redefine,class,iklass,add] scratch class added; one of its methods is on_stack.
[2.654s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.654s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.654s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.654s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
infinite called
[2.693s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.693s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.693s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.693s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.715s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.715s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.715s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.715s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.740s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.740s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.740s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.740s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.762s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.762s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.762s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.762s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.787s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.787s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.787s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.787s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.811s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.811s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.811s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.811s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.834s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.834s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.834s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.834s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.856s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.856s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.856s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.856s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.878s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.878s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.878s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.879s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.902s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.902s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.902s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.902s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.943s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.943s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.943s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.943s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.966s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.966s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.966s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.966s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.988s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.988s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.988s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.988s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.024s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.024s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.024s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.024s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.048s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.048s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.048s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.048s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.073s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.073s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.073s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.073s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.097s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.097s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.097s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.097s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.121s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.121s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.121s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.121s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.144s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.144s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.144s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.144s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.172s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.172s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.172s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.172s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.295s][trace][redefine,class,iklass,add ] adding previous version ref for RedefineRunningMethods_B, EMCP_cnt=2
[3.295s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.295s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.295s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.295s][trace][redefine,class,iklass,add ] infinite_emcp(()V): flush obsolete method @3 in version @0
[3.295s][trace][redefine,class,iklass,add ] localSleep(()V): flush obsolete method @4 in version @0
[3.295s][trace][redefine,class,iklass,add ] scratch class added; one of its methods is on_stack.
[3.328s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.328s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.328s][trace][redefine,class,iklass,purge] previous version 0x000000080113b178 is alive
[3.328s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.328s][trace][redefine,class,iklass,purge] previous version stats: live=2, deleted=0
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=/nmethod.cpp:3249
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S94905/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/5ecec92a-60d9-4c7e-9510-cf8bac54efe9/runs/863af1fa-8803-489d-829f-2e54063e48d5/workspace/open/src/hotspot/share/code/nmethod.cpp:3249), pid=3351329, tid=3351333
# assert(db != __null && !db->is_adapter_blob()) failed: must use stub!
#
# JRE version: Java(TM) SE Runtime Environment (20.0+19) (fastdebug build 20-ea+19-1326)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-ea+19-1326, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0x1448424][thread 3351361 also had an error]
  DirectNativeCallWrapper::verify_resolve_call(unsigned char*) const+0x54
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S94889/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6ca99daf-28b6-456d-84e1-2511b8bcab33/runs/e3985560-a056-4d43-82fc-761d1adaf63f/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/3/core.3351329)
#
Unsupported internal testing APIs have been used.

# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S94889/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6ca99daf-28b6-456d-84e1-2511b8bcab33/runs/e3985560-a056-4d43-82fc-761d1adaf63f/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/3/hs_err_pid3351329.log
[thread 3351359 also had an error]
[thread 3351357 also had an error]
[thread 3351394 also had an error]
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
];
 stderr: [java version "20-ea" 2023-03-21
Java(TM) SE Runtime Environment (fastdebug build 20-ea+19-1326)
Java HotSpot(TM) 64-Bit Server VM (fastdebug build 20-ea+19-1326, mixed mode, sharing)
]
 exitValue = 134

java.lang.RuntimeException: Hotspot crashed
at jdk.test.lib.cds.CDSTestUtils.checkCommonExecExceptions(CDSTestUtils.java:320)
at jdk.test.lib.cds.CDSTestUtils.checkExec(CDSTestUtils.java:473)
at jdk.test.lib.cds.CDSTestUtils.checkExec(CDSTestUtils.java:460)
at RedefineRunningMethods_Shared.main(RedefineRunningMethods_Shared.java:78)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
at java.base/java.lang.Thread.run(Thread.java:1591)

JavaTest Message: Test threw exception: java.lang.RuntimeException
JavaTest Message: shutting down test

result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Hotspot crashed


Here's the crashing thread's stack:

--------------- T H R E A D ---------------

Current thread (0x0000fffd08054ce0): WorkerThread "ZWorker#0" [stack: 0x0000fffd0cbe0000,0x0000fffd0cde0000] [id=3351333]

Stack: [0x0000fffd0cbe0000,0x0000fffd0cde0000], sp=0x0000fffd0cdde5c0, free space=2041k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1448424] DirectNativeCallWrapper::verify_resolve_call(unsigned char*) const+0x54 (nmethod.cpp:3249)
V [libjvm.so+0x9a4778] CompiledIC::is_call_to_interpreted() const+0x98 (compiledIC.cpp:353)
V [libjvm.so+0x9a71d4] CompiledIC::verify()+0x54 (compiledIC.cpp:691)
V [libjvm.so+0x9ad8d0] CompiledMethod::cleanup_inline_caches_impl(bool, bool)+0x1f0 (compiledIC.hpp:303)
V [libjvm.so+0x9ae738] CompiledMethod::unload_nmethod_caches(bool)+0xc8 (compiledMethod.cpp:550)
V [libjvm.so+0x194ae4c] ZNMethodUnlinkClosure::do_nmethod(nmethod*)+0xbc (zNMethod.cpp:309)
V [libjvm.so+0x194d0ac] ZNMethodTableIteration::nmethods_do(NMethodClosure*)+0x8c (zNMethodTableIteration.cpp:72)
V [libjvm.so+0x194aa48] ZNMethodUnlinkTask::work()+0x28 (zNMethod.cpp:338)
V [libjvm.so+0x197f708] ZTask::Task::work(unsigned int)+0x28 (zTask.cpp:34)
V [libjvm.so+0x18f3678] WorkerThread::run()+0x98 (workerThread.cpp:69)
V [libjvm.so+0x17af638] Thread::call_run()+0xf8 (thread.cpp:224)
V [libjvm.so+0x14c4824] thread_native_entry(Thread*)+0x100 (os_linux.cpp:710)
C [libpthread.so.0+0x7908] start_thread+0x188


This is a ZGC WorkerThread calling compiler code and failing
an assertion. I'm starting this bug off in hotspot/gc for initial triage.