During testing JDK 11, we noticed a 'deadlock' situation (with a stress test) with 'reentrant locking' of a class when the current thread is already in the middle of redefining the same class. The issue is reproducible withe the latest jdk/jdk repo.

When redefining class, the VM 'locks' (VM_RedefineClasses::lock_classes()) a class by setting the InstanceKlass' '_is_being_redefined' field to true if it's not currently being redefined in other threads. Class file load hook event may trigger new class redefinition when a thread is redefining a class (after lock_classes()). In that case, it causes the thread being blocked indefinitely while trying to wait for the class being 'unlocked'.

The author extracted the test into a standalone test case to help reproduce/verify the bug. Please see attached test case.

Following is my analysis of the original issue (running the original stress test on JDK 11):

When attaching it from gdb, it shows the main thread is waiting and blocked. For a given class, when sun.instrument.InstrumentationImpl.retransformClasses0() (marked with '>>>>>>' below in the Java stack) is called (which calls JvmtiEnv::RetransformClasses), the VM 'locks' the class if it's not currently being redefined in other threads by setting the '_is_being_redefined' to true in the InstanceKlass. The operation is done by VM_RedefineClasses::lock_classes().

void VM_RedefineClasses::lock_classes() {
...
     if (ik->is_being_redefined()) {
        RedefineClasses_lock->wait();
        has_redefined = true;
        break; // for loop
      }

The VM then calls SystemDictionary::parse_stream to parse the new class stream, which calls KlassFactory::create_from_stream. KlassFactory::create_from_stream posts class_file_load_hook if there is a JVMTI agent and the capability is enabled. Apparently, the agent has a new version of the class, then the VM tries to transform the same class again, which goes through VM_RedefineClasses::lock_classes() again for the class. Since '_is_being_redefined' is already set by the same thread and the first retransformClasses is not completed yet, the thread is never able to 'lock' the same class again. That causes the hang.

Please feel free to change the subcomponent to 'runtime' if that's more appropriate.