There are several restrictions on when nested locks should be optimized. One of them is that NLO requires one BoxLock node per one lock/unlock region.
Unfortunately loop unswitching optimization can create such a case because loop cloning does not clone BoxLock nodes which are outside loops (they are pinned to the Root node). Note, Lock/Unlock nodes are a subclass of Call node - all loop optimizations are skipped, except unswitching.

NLO happens during Macro nodes elimination. But before that Lock/Unlock coarsening optimization happens during IGVN as early as right after parsing - even before Escape analysis.

Now back to the failing case:

while (a) {
    synchronized(obj) { // ORn (outer lock region)
          if (loop_invar_check) {
              synchronized(obj) { // NR1 } // (nested lock region 1)
          } else {
             // no synchronization
          }
          synchronized(obj) { // NR2 } // (nested lock region 2)
    }
}

After loop unswitching:

 if (loop_invar_check) {
    while (a) {
        synchronized(obj) { // ORn (outer lock region)
             synchronized(obj) { // NR1 } // (nested lock region 1)
             synchronized(obj) { // NR2 } // (nested lock region 2)
        }
    }
} else {
    while (a) {
        synchronized(obj) { // CORn (Clone of outer lock region)
             synchronized(obj) { // CNR2 } // (Clone of nested lock region 2)
        }
    }
}

After loop unswitching for the code in the first branch Lock/Unlock coarsening optimization marked Unlock node from NR1 and Lock from NR2 as "Coarsened".

When NLO (Nested Lock Optimization) started, it overwrote the state for Unlock node from NR1 as "Nested" because it passed all conditions: it belonged to a simple region with one BoxLock node and was inside ORn region.
But NLO did not overwrite the state of Lock node from NR2 because this region shares BoxLock node with its clone CNR2 in second branch of unswitched loop. As a result this Lock node stayed "Coarsened". Later, when locks are eliminated, Lock/Unlock nodes in NR1 are eliminated as "Nested". And Lock node in NR2 is eliminated as "Coarsened" leaving behind an un-matching Unlock node from NR2.