JvmtiVTMSTransitionDisabler only accounts for virtual threads in a transition, i.e. it doesn't do anything for platform threads. This means that a JVMTI operation that targets a platform thread acting as a carrier can see changing values of the JvmtiThreadState fetched from the JavaThread associated with the platform thread oop. In particular trying to switch the platform thread to interpreted only mode can face issues such as the wrong thread being set to interpreted mode or even lead to crashes.

Attached a simpler reproducer that crashes with:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/scratch/pchilano/random2/open/src/hotspot/share/prims/jvmtiEventController.cpp:214), pid=235469, tid=235493
# assert(state != nullptr) failed: sanity check
#
# JRE version: OpenJDK Runtime Environment (22.0) (fastdebug build 22-internal-2023-06-29-0153314.pchilano...)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 22-internal-2023-06-29-0153314.pchilano..., mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)


Stack: [0x00007f2a4eaa8000,0x00007f2a4eba8000], sp=0x00007f2a4eba49f0, free space=1010k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x11784a5] EnterInterpOnlyModeClosure::do_thread(Thread*)+0x5a5 (jvmtiEventController.cpp:214)
V [libjvm.so+0xd9bd06] HandshakeOperation::do_handshake(JavaThread*)+0x46 (handshake.cpp:326)
V [libjvm.so+0xd9c2aa] HandshakeState::try_process(HandshakeOperation*) [clone .part.0]+0xfa (handshake.cpp:662)
V [libjvm.so+0xd9ca6a] Handshake::execute(HandshakeClosure*, ThreadsListHandle*, JavaThread*)+0x49a (handshake.cpp:390)
V [libjvm.so+0x1173666] JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState*)+0x96 (jvmtiEventController.cpp:375)
V [libjvm.so+0x11739ee] JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState*) [clone .part.0]+0xee (jvmtiEventController.cpp:597)
V [libjvm.so+0x1174c00] JvmtiEventControllerPrivate::recompute_enabled()+0x110 (jvmtiEventController.cpp:558)
V [libjvm.so+0x1176f74] JvmtiEventController::set_user_enabled(JvmtiEnvBase*, JavaThread*, oop, jvmtiEvent, bool)+0x1b4 (jvmtiEventController.cpp:1060)
V [libjvm.so+0x1150bb6] JvmtiEnv::SetEventNotificationMode(jvmtiEventMode, jvmtiEvent, _jobject*, ...)+0x1f6 (jvmtiEnv.cpp:603)
V [libjvm.so+0x1104577] jvmti_SetEventNotificationMode+0x107 (jvmtiEnter.cpp:5321)
C [libRepro8311177.so+0x2356] Java_Repro8311177_setSingleSteppingMode+0x226 (jvmti.h:2523)
j Repro8311177.setSingleSteppingMode(Z)V+0
j Repro8311177.runTest()V+53
j Repro8311177.main([Ljava/lang/String;)V+9