There are 2 problems: broken bailout message and C2 can't compile.

JDK 22 and latest 23 produces strange message:

java -XX:CICompilerCount=1 -XX:-TieredCompilation X:+PrintCompilation -Xbatch -Xmx128m TestNestedLocks
154 1 b java.util.ArrayList::add (25 bytes)
159 2 b java.util.ArrayList::add (23 bytes)
162 3 !b TestNestedLocks::test (102 bytes)
162 3 !b TestNestedLocks::test (102 bytes) COMPILE SKIPPED: ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
182 4 % !b TestNestedLocks::main @ 18 (39 bytes)
184 5 !b TestNestedLocks::main (39 bytes)
637 4 % ! TestNestedLocks::main @ 18 (39 bytes) made not entrant
OOM caught in test

It looks like something was broken recently with bailout messages.

JDK 11u produces:

    259 4 !b TestNestedLocks::test (102 bytes) COMPILE SKIPPED: cannot parse method (not retryable)

JDK 21 has more detailed info:

    426 3 !b TestNestedLocks::test (102 bytes) COMPILE SKIPPED: cannot parse method: not compilable (unbalanced monitors) (retry at different tier)

We need to fix message in JDK 22 and 23.

And find why C2 think it is unbalanced.

Actually C1 is also can't compile it:
     28 6 !b 1 TestNestedLocks::test (102 bytes) COMPILE SKIPPED: invalid parsing (retry at different tier)

It works fine if we put inner 2 locks into separate method and inline it.