In PrintDeoptimizationDetails, ttyLock is held to print out information about the compiledVFrame. This printing with ZGC takes out the StackWatermark_lock which makes StackWatermark_lock have a lower lock ranking than ttyLock. ttyLock is used to keep xml printing together.
Using a stringStream to compose messages outside the ttyLock, reduces the scope of the lock, at least during deoptimization.

This is one place where the inversion is that requires StackWatermark_lock to be a lower rank than tty.

Stack: [0x00007f5e3e11c000,0x00007f5e3e21d000], sp=0x00007f5e3e2173e0, free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1500cde] Mutex::check_rank(Thread*)+0xfe
V [libjvm.so+0x1501661] Mutex::lock_without_safepoint_check(Thread*)+0x51
V [libjvm.so+0x18c4e6c] StackWatermark::process_one()+0x1c
V [libjvm.so+0x18c5d29] StackWatermarkSet::on_iteration(JavaThread*, frame const&)+0x89
V [libjvm.so+0xbf4afd] frame::frame_size(RegisterMap*) const+0x11d
V [libjvm.so+0x1abc977] javaVFrame::print_value() const+0x3b7
V [libjvm.so+0xaceb2c] Deoptimization::create_vframeArray(JavaThread*, frame, RegisterMap*, GrowableArray<compiledVFrame*>*, bool)+0x1bc

Here's another:

V [libjvm.so+0x175889c] StackWatermark::process_one()+0x1c
V [libjvm.so+0x1759769] StackWatermarkSet::on_iteration(JavaThread*, frame const&)+0x89
V [libjvm.so+0xbc85a5] frame::sender(RegisterMap*) const+0x105
V [libjvm.so+0x19d70cd] WB_VerifyFrames+0x2dd
j sun.hotspot.WhiteBox.verifyFrames(ZZ)V+0