A JavaThread can escape back to Java from an ongoing handshake given the right timing of handshakes and external suspend/resume.

After a JavaThread that was suspended is resumed, there is a small window(after it’s resumed and before it restores its state in java_suspend_self_with_safepoint_check()) where the VMThread trying to handshake that JavaThread can see that it is in the blocked state and so proceed to execute the handshake. Back in java_suspend_self_with_safepoint_check() the resumed JavaThread could skip blocking for the handshake in block_if_requested() if this call to handle_special_runtime_exit_condition() came from a previous ~ThreadInVMForHandshake(). Here is a trace that shows the previous scenario with the state of the JavaThread as it goes through the different transitions until it escapes the handshake:

       [1] = _thread_in_Java
       ThreadInVMfromJava()
              [2] = _thread_in_vm
       ~ThreadInVMfromJava()
              [3] = _thread_in_vm_trans
              SafepointMechanism::block_if_requested() -> block_if_requested_slow() -> handshake_process_by_self() -> process_by_self() -> process_self_inner()
                     ThreadInVMForHandshake()
                            [4] = _thread_in_vm
                     ~ThreadInVMForHandshake()
                            [5] = _thread_in_vm_trans
                            [6] = _thread_in_vm_trans
                            handle_special_runtime_exit_condition() -> java_suspend_self_with_safepoint_check()
                                   [7] = _thread_blocked <—— HERE VMTHREAD SEES IT’S SAFE FOR HANDSHAKE (JavaThread was already resumed)
                                   [8] = _thread_in_vm_trans
                                   SafepointMechanism::block_if_requested() -> block_if_requested_slow() -> handshake_process_by_self() -> process_by_self() <—— WE DONT BLOCK FOR HANDSHAKE BECAUSE FLAG IS STILL SET FOR PREVIOUS HANDSHAKE (We are inside ~ThreadInVMForHandshake()). JAVATHREAD ESCAPES BACK TO JAVA WHILE HANDSHAKE IS IN PROGRESS.

I’ve seen test SuspendAtExit.java failing because of this while working on JDK-8232733. Removing the need to grab the Threads_lock while processing the handshake just makes this small window bigger. In my Mac, adding a sleep of os::naked_short_nanosleep(100000) before set_thread_state_fence(state) in java_suspend_self_with_safepoint_check() makes the test fail.