JDK HEAD commit: 476d2ae69d6

Testcase:
1. Add javaagent which will change the class file (CFLH will be touched)
2. A virtual thread need to be pinned
3. Add the option `-Djdk.tracePinnedThreads=full`
I add the testcase in the attachment.
I think this and JDK-8330146 should be backport for JDK21. JDK21 will run into deadlock now.
Update: I pull some new commits and now HEAD is 657c0bddf90. The release version will run into deadlock instead of causing jvm crash. The slowdebug version will run into jvm crash.

Cause:
When the virtual thread is pinned, it will use the class `PinnedThreadPrinter` to report. When loading this class, javaagent will change its content and post the CFLH. However, JvmtiClassFileLoadHookPoster should not call `post` for vthreads which are in VTMS transition.

Solution:
Add condition judegement for `JvmtiClassFileLoadHookPoster::post`

The crash log:

# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f861178c8b1, pid=41748, tid=41772
#
# JRE version: OpenJDK Runtime Environment (24.0) (build 24-internal-adhoc.tjw378335.jdk)
# Java VM: OpenJDK 64-Bit Server VM (24-internal-adhoc.tjw378335.jdk, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x2f58b1] AccessInternal::PostRuntimeDispatch<G1BarrierSet::AccessBarrier<548964ul, G1BarrierSet>, (AccessInternal::BarrierType)2, 548964ul>::oop_access_barrier(void*)+0x1
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

--------------- S U M M A R Y ------------


Host: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz, 96 cores, 503G, Alibaba Group Enterprise Linux Server release 7.2 (Paladin)
Time: Mon Jul 29 15:08:54 2024 CST elapsed time: 0.128565 seconds (0d 0h 0m 0s)

--------------- T H R E A D ---------------

Current thread (0x00007f8528061400): JavaThread "pool-1-thread-1" [_thread_in_vm, id=41772, stack(0x00007f85ab13a000,0x00007f85ab23b000) (1028K)]

Stack: [0x00007f85ab13a000,0x00007f85ab23b000], sp=0x00007f85ab237c10, free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x2f58b1] AccessInternal::PostRuntimeDispatch<G1BarrierSet::AccessBarrier<548964ul, G1BarrierSet>, (AccessInternal::BarrierType)2, 548964ul>::oop_access_barrier(void*)+0x1 (access.inline.hpp:81)
V [libjvm.so+0xb2e933] JvmtiClassFileLoadHookPoster::post_to_env(JvmtiEnv*, bool)+0x163 (jvmtiExport.cpp:270)
V [libjvm.so+0xb23ce5] JvmtiExport::post_class_file_load_hook(Symbol*, Handle, Handle, unsigned char**, unsigned char**, JvmtiCachedClassFileData**)+0x1d5 (jvmtiExport.cpp:987)
V [libjvm.so+0xb53c26] KlassFactory::create_from_stream(ClassFileStream*, Symbol*, ClassLoaderData*, ClassLoadInfo const&, JavaThread*)+0x176 (klassFactory.cpp:148)
V [libjvm.so+0x62e33e] ClassLoader::load_class(Symbol*, PackageEntry*, bool, JavaThread*)+0x24e (classLoader.cpp:1205)
V [libjvm.so+0xf707f5] SystemDictionary::load_instance_class_impl(Symbol*, Handle, JavaThread*)+0x235 (systemDictionary.cpp:1279)
V [libjvm.so+0xf6f148] SystemDictionary::load_instance_class(Symbol*, Handle, JavaThread*)+0x18 (systemDictionary.cpp:1343)
V [libjvm.so+0xf6f80f] SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, JavaThread*)+0x5ef (systemDictionary.cpp:702)
V [libjvm.so+0xf6fc50] SystemDictionary::resolve_or_null(Symbol*, Handle, Handle, JavaThread*)+0x40 (systemDictionary.cpp:343)
V [libjvm.so+0xf6fdae] SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, JavaThread*)+0x1e (systemDictionary.cpp:321)
V [libjvm.so+0x6b2616] ConstantPool::klass_ref_at(int, Bytecodes::Code, JavaThread*)+0x296 (constantPool.cpp:599)
V [libjvm.so+0xb83726] LinkInfo::LinkInfo(constantPoolHandle const&, int, Bytecodes::Code, JavaThread*)+0x46 (linkResolver.cpp:278)
V [libjvm.so+0xb8fcaa] LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle const&, int, Bytecodes::Code, JavaThread*)+0x28a (linkResolver.cpp:1745)
V [libjvm.so+0x9155e6] InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x1c6 (interpreterRuntime.cpp:842)
V [libjvm.so+0x915ab5] InterpreterRuntime::resolve_from_cache(JavaThread*, Bytecodes::Code)+0xd5 (interpreterRuntime.cpp:1004)
j java.lang.VirtualThread$VThreadContinuation.onPinned(Ljdk/internal/vm/Continuation$Pinned;)V+42 java.base@24-internal
j jdk.internal.vm.Continuation.onPinned0(I)V+5 java.base@24-internal
j jdk.internal.vm.Continuation.yield0(Ljdk/internal/vm/ContinuationScope;Ljdk/internal/vm/Continuation;)Z+325 java.base@24-internal
j jdk.internal.vm.Continuation.yield(Ljdk/internal/vm/ContinuationScope;)Z+69 java.base@24-internal
j java.lang.VirtualThread.yieldContinuation()Z+8 java.base@24-internal
j java.lang.VirtualThread.parkNanos(J)V+64 java.base@24-internal
j java.lang.VirtualThread.sleepNanos(J)V+70 java.base@24-internal
j java.lang.Thread.sleepNanos(J)V+26 java.base@24-internal
j java.lang.Thread.sleep(J)V+25 java.base@24-internal
j TestPinCaseWithTrace.native2Java(I)I+11
v ~StubRoutines::call_stub 0x00007f85fb9b5c86
V [libjvm.so+0x91e014] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x2f4 (javaCalls.cpp:415)
V [libjvm.so+0x9d3241] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) [clone .isra.148] [clone .constprop.253]+0x161 (jni.cpp:888)
V [libjvm.so+0x9d3d5f] jni_CallStaticIntMethod+0x14f (jni.cpp:1580)
C [libPinJNI.so+0x677] Java_TestPinCaseWithTrace_nativeFuncPin+0x47 (libPinJNI.c:29)
j TestPinCaseWithTrace.nativeFuncPin(I)I+0
j TestPinCaseWithTrace.lambda$main$0()V+4
j TestPinCaseWithTrace$$Lambda+0x00007f856b003438.run()V+0
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base@24-internal
j java.lang.VirtualThread.run(Ljava/lang/Runnable;)V+66 java.base@24-internal
j java.lang.VirtualThread$VThreadContinuation$1.run()V+8 java.base@24-internal
j jdk.internal.vm.Continuation.enter0()V+4 java.base@24-internal
j jdk.internal.vm.Continuation.enter(Ljdk/internal/vm/Continuation;Z)V+1 java.base@24-internal
J 240 jdk.internal.vm.Continuation.enterSpecial(Ljdk/internal/vm/Continuation;ZZ)V java.base@24-internal (0 bytes) @ 0x00007f85fc22445c [0x00007f85fc224320+0x000000000000013c]
j jdk.internal.vm.Continuation.run()V+122 java.base@24-internal
j java.lang.VirtualThread.runContinuation()V+72 java.base@24-internal
j java.lang.VirtualThread$$Lambda+0x00007f856b064cb8.run()V+4 java.base@24-internal
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@24-internal
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@24-internal
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base@24-internal
j java.lang.Thread.run()V+19 java.base@24-internal
v ~StubRoutines::call_stub 0x00007f85fb9b5c86
V [libjvm.so+0x91e014] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x2f4 (javaCalls.cpp:415)
V [libjvm.so+0x91fc43] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x223 (javaCalls.cpp:329)
V [libjvm.so+0x9fa5d3] thread_entry(JavaThread*, JavaThread*)+0x73 (jvm.cpp:2910)
V [libjvm.so+0x937c0c] JavaThread::run()+0x15c (javaThread.cpp:757)
V [libjvm.so+0xfbd18f] Thread::call_run()+0x9f (thread.cpp:225)
V [libjvm.so+0xd160aa] thread_native_entry(Thread*)+0xda (os_linux.cpp:858)
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.lang.VirtualThread$VThreadContinuation.onPinned(Ljdk/internal/vm/Continuation$Pinned;)V+42 java.base@24-internal
j jdk.internal.vm.Continuation.onPinned0(I)V+5 java.base@24-internal
j jdk.internal.vm.Continuation.yield0(Ljdk/internal/vm/ContinuationScope;Ljdk/internal/vm/Continuation;)Z+325 java.base@24-internal
j jdk.internal.vm.Continuation.yield(Ljdk/internal/vm/ContinuationScope;)Z+69 java.base@24-internal
j java.lang.VirtualThread.yieldContinuation()Z+8 java.base@24-internal
j java.lang.VirtualThread.parkNanos(J)V+64 java.base@24-internal
j java.lang.VirtualThread.sleepNanos(J)V+70 java.base@24-internal
j java.lang.Thread.sleepNanos(J)V+26 java.base@24-internal
j java.lang.Thread.sleep(J)V+25 java.base@24-internal
j TestPinCaseWithTrace.native2Java(I)I+11
v ~StubRoutines::call_stub 0x00007f85fb9b5c86
j TestPinCaseWithTrace.nativeFuncPin(I)I+0
j TestPinCaseWithTrace.lambda$main$0()V+4
j TestPinCaseWithTrace$$Lambda+0x00007f856b003438.run()V+0
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base@24-internal
j java.lang.VirtualThread.run(Ljava/lang/Runnable;)V+66 java.base@24-internal
j java.lang.VirtualThread$VThreadContinuation$1.run()V+8 java.base@24-internal
j jdk.internal.vm.Continuation.enter0()V+4 java.base@24-internal
j jdk.internal.vm.Continuation.enter(Ljdk/internal/vm/Continuation;Z)V+1 java.base@24-internal
J 240 jdk.internal.vm.Continuation.enterSpecial(Ljdk/internal/vm/Continuation;ZZ)V java.base@24-internal (0 bytes) @ 0x00007f85fc22445c [0x00007f85fc224320+0x000000000000013c]
j jdk.internal.vm.Continuation.run()V+122 java.base@24-internal
j java.lang.VirtualThread.runContinuation()V+72 java.base@24-internal
j java.lang.VirtualThread$$Lambda+0x00007f856b064cb8.run()V+4 java.base@24-internal
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@24-internal
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@24-internal
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base@24-internal
j java.lang.Thread.run()V+19 java.base@24-internal
v ~StubRoutines::call_stub 0x00007f85fb9b5c86

--------------- Dead Lock Stack Trace (the other condition) ------------
The deadlock related stacktrace:

Thread 24 "pool-1-thread-1" hit Breakpoint 5, JvmtiVTMSTransitionDisabler::JvmtiVTMSTransitionDisabler (this=0x7fff8fde5940, thread=0x0) at src/hotspot/share/prims/jvmtiThreadState.cpp:256
256 JvmtiVTMSTransitionDisabler::JvmtiVTMSTransitionDisabler(jthread thread)
(gdb) bt
#0 JvmtiVTMSTransitionDisabler::JvmtiVTMSTransitionDisabler (this=0x7fff8fde5940, thread=0x0) at src/hotspot/share/prims/jvmtiThreadState.cpp:256
#1 0x00007ffff64c691d in JvmtiEnv::SetThreadLocalStorage (this=this@entry=0x7ffff001a2a0, thread=thread@entry=0x0, data=data@entry=0x7effc0bb) at src/hotspot/share/prims/jvmtiEnv.cpp:152
#2 0x00007ffff6479b8e in jvmti_SetThreadLocalStorage (env=0x7ffff001a2a0, thread=0x0, data=0x7effc0bb) at build/linux-x86_64-server-release/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:1049
#3 0x00007ffff508ab0f in confirmingTLSSet (jvmtienv=jvmtienv@entry=0x7ffff001a2a0, thread=thread@entry=0x0, newValue=newValue@entry=0x7effc0bb) at src/java.instrument/share/native/libinstrument/Reentrancy.c:73
#4 0x00007ffff508abbc in tryToAcquireReentrancyToken (jvmtienv=0x7ffff001a2a0, thread=thread@entry=0x0) at src/java.instrument/share/native/libinstrument/Reentrancy.c:130
#5 0x00007ffff50883c2 in transformClassFile (agent=0x7ffff001a8e0, jnienv=jnienv@entry=0x7ffff02a89a0, loaderObject=loaderObject@entry=0x0, name=name@entry=0x7fff24105478 "java/lang/PinnedThreadPrinter",
    classBeingRedefined=classBeingRedefined@entry=0x0, protectionDomain=protectionDomain@entry=0x0, class_data_len=8100, class_data=0x7fff241034a0 "\312\376\272\276", new_class_data_len=0x7fff8fde5c64, new_class_data=0x7fff8fde5c68,
    is_retransformer=0 '\000') at src/java.instrument/share/native/libinstrument/JPLISAgent.c:829
#6 0x00007ffff50872ef in eventHandlerClassFileLoadHook (jvmtienv=<optimized out>, jnienv=0x7ffff02a89a0, class_being_redefined=0x0, loader=0x0, name=0x7fff24105478 "java/lang/PinnedThreadPrinter", protectionDomain=0x0,
    class_data_len=8100, class_data=0x7fff241034a0 "\312\376\272\276", new_class_data_len=0x7fff8fde5c64, new_class_data=0x7fff8fde5c68) at src/java.instrument/share/native/libinstrument/InvocationAdapter.c:650
#7 0x00007ffff64ec742 in JvmtiClassFileLoadHookPoster::post_to_env (this=0x7fff8fde5d80, env=0x7ffff001a2a0, caching_needed=<optimized out>) at src/hotspot/share/prims/jvmtiExport.cpp:1014
#8 0x00007ffff64e1965 in JvmtiClassFileLoadHookPoster::post_all_envs (this=0x7fff8fde5d80) at src/hotspot/share/prims/jvmtiExport.cpp:987
#9 JvmtiClassFileLoadHookPoster::post (this=0x7fff8fde5d80) at src/hotspot/share/prims/jvmtiExport.cpp:971
#10 JvmtiExport::post_class_file_load_hook (h_name=h_name@entry=0x7fffac09c2d8, class_loader=..., h_protection_domain=..., data_ptr=data_ptr@entry=0x7fff8fde5e90, end_ptr=end_ptr@entry=0x7fff8fde5e98,
    cache_ptr=cache_ptr@entry=0x7fff8fde5e88) at src/hotspot/share/prims/jvmtiExport.cpp:1105
#11 0x00007ffff65119d6 in check_class_file_load_hook (__the_thread__=0x7ffff02a85e0, cached_class_file=0x7fff8fde5e88, protection_domain=..., loader_data=0x7ffff0205b60, name=0x7fffac09c2d8, stream=0x7fff24105448)
    at src/hotspot/share/classfile/klassFactory.cpp:148
#12 KlassFactory::create_from_stream (stream=0x7fff24105448, name=name@entry=0x7fffac09c2d8, loader_data=0x7ffff0205b60, cl_info=..., __the_thread__=__the_thread__@entry=0x7ffff02a85e0)
    at src/hotspot/share/classfile/klassFactory.cpp:189
#13 0x00007ffff5fea6de in ClassLoader::load_class (name=name@entry=0x7fffac09c2d8, pkg_entry=pkg_entry@entry=0x7ffff02128e0, search_append_only=search_append_only@entry=false, __the_thread__=__the_thread__@entry=0x7ffff02a85e0)
    at src/hotspot/share/classfile/classLoader.cpp:1205
#14 0x00007ffff6928a35 in SystemDictionary::load_instance_class_impl (class_name=class_name@entry=0x7fffac09c2d8, class_loader=..., __the_thread__=__the_thread__@entry=0x7ffff02a85e0)
    at src/hotspot/share/classfile/systemDictionary.cpp:1279
#15 0x00007ffff6927388 in SystemDictionary::load_instance_class (name=name@entry=0x7fffac09c2d8, class_loader=class_loader@entry=..., __the_thread__=__the_thread__@entry=0x7ffff02a85e0)
    at src/hotspot/share/classfile/systemDictionary.cpp:1343
#16 0x00007ffff6927a4f in SystemDictionary::resolve_instance_class_or_null (name=name@entry=0x7fffac09c2d8, class_loader=..., class_loader@entry=..., protection_domain=..., __the_thread__=0x7ffff02a85e0)
    at src/hotspot/share/classfile/systemDictionary.cpp:702
#17 0x00007ffff6927e90 in SystemDictionary::resolve_or_null (class_name=class_name@entry=0x7fffac09c2d8, class_loader=..., protection_domain=..., __the_thread__=__the_thread__@entry=0x7ffff02a85e0)
    at src/hotspot/share/classfile/systemDictionary.cpp:343
#18 0x00007ffff6927fee in SystemDictionary::resolve_or_fail (class_name=class_name@entry=0x7fffac09c2d8, class_loader=..., class_loader@entry=..., protection_domain=..., throw_error=throw_error@entry=true,
    __the_thread__=__the_thread__@entry=0x7ffff02a85e0) at src/hotspot/share/classfile/systemDictionary.cpp:321
#19 0x00007ffff606e526 in ConstantPool::klass_at_impl (__the_thread__=0x7ffff02a85e0, cp_index=44, this_cp=...) at src/hotspot/share/oops/constantPool.cpp:599
#20 ConstantPool::klass_at (__the_thread__=0x7ffff02a85e0, cp_index=44, this=<optimized out>) at src/hotspot/share/oops/constantPool.hpp:381
#21 ConstantPool::klass_ref_at (this=<optimized out>, which=which@entry=5, code=code@entry=Bytecodes::_invokestatic, __the_thread__=__the_thread__@entry=0x7ffff02a85e0) at src/hotspot/share/oops/constantPool.cpp:840
#22 0x00007ffff6541216 in LinkInfo::LinkInfo (this=0x7fff8fde6830, pool=..., index=5, code=Bytecodes::_invokestatic, __the_thread__=0x7ffff02a85e0) at src/hotspot/share/interpreter/linkResolver.cpp:278
#23 0x00007ffff654d79a in LinkResolver::resolve_invokestatic (__the_thread__=0x7ffff02a85e0, index=-1881247760, pool=..., result=...) at src/hotspot/share/interpreter/linkResolver.cpp:1745
#24 LinkResolver::resolve_invoke (result=..., recv=recv@entry=..., pool=..., index=index@entry=5, byte=byte@entry=Bytecodes::_invokestatic, __the_thread__=__the_thread__@entry=0x7ffff02a85e0)
    at src/hotspot/share/interpreter/linkResolver.cpp:1705
#25 0x00007ffff62d2556 in InterpreterRuntime::resolve_invoke (current=0x7ffff02a85e0, bytecode=Bytecodes::_invokestatic) at src/hotspot/share/interpreter/interpreterRuntime.cpp:842
#26 0x00007ffff62d2a25 in InterpreterRuntime::resolve_from_cache (current=0x7ffff02a85e0, bytecode=<optimized out>) at src/hotspot/share/interpreter/interpreterRuntime.cpp:1004


The `JvmtiVTMSTransitionDisabler::JvmtiVTMSTransitionDisabler` cannot jump out the loop since the vthread-1 itself is in VTMS Transition. (VTMS_transition_mark is true)