The following situation can happen with libgraal and -Xcomp or -Xbatch:

"PmdThread 8" #22 prio=5 os_prio=31 cpu=68.62ms elapsed=526.77s tid=0x00007fdb73828200 nid=0x6e03 waiting on condition [0x00007000116ea000]
   java.lang.Thread.State: RUNNABLE
        at jdk.internal.jimage.ImageStringsReader.charsFromByteBufferLength(java.base@15.0.1/ImageStringsReader.java:199)
        at jdk.internal.jimage.ImageStringsReader.stringFromByteBuffer(java.base@15.0.1/ImageStringsReader.java:245)
        at jdk.internal.jimage.BasicImageReader.getString(java.base@15.0.1/BasicImageReader.java:334)
        at jdk.internal.jimage.ImageStringsReader.get(java.base@15.0.1/ImageStringsReader.java:51)
        at jdk.internal.jimage.ImageLocation.verify(java.base@15.0.1/ImageLocation.java:158)
        at jdk.internal.jimage.BasicImageReader.findLocation(java.base@15.0.1/BasicImageReader.java:273)
        - locked <0x00000001ccbc42f8> (a jdk.internal.jimage.ImageReader$SharedImageReader)

This thread is blocked waiting for the compilation of jdk.internal.jimage.ImageStringsReader::charsFromByteBufferLength. This is the libgraal thread compiling it:

"JVMCI-native CompilerThread1" #23 daemon prio=9 os_prio=35 cpu=245.11ms elapsed=31250.04s tid=0x00007fb655915e00 nid=0x8303 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   Compiling: 1087 % b 4 jdk.internal.jimage.ImageStringsReader::charsFromByteBufferLength @ 2 (47 bytes)

That thread is blocked on the SVM lock in HotSpotReplacementsImpl.getEncodedSnippet, which is held by:

"JVMCI-native CompilerThread2" #24 daemon prio=9 os_prio=35 cpu=1126.48ms elapsed=31250.04s tid=0x00007fb636049400 nid=0x7f03 waiting for monitor entry [0x0000700009b59000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   Compiling: 1089 b 4 java.util.ArrayList::add (25 bytes)
        at jdk.internal.jimage.BasicImageReader.findLocation(java.base@15.0.1/BasicImageReader.java:253)
        - waiting to lock <0x00000001cb3c4278> (a jdk.internal.jimage.ImageReader$SharedImageReader)
        at jdk.internal.jimage.ImageReader.findLocation(java.base@15.0.1/ImageReader.java:148)
        at jdk.internal.module.SystemModuleFinders$SystemModuleReader.findImageLocation(java.base@15.0.1/SystemModuleFinders.java:428)
        at jdk.internal.module.SystemModuleFinders$SystemModuleReader.read(java.base@15.0.1/SystemModuleFinders.java:464)
        at jdk.internal.loader.BuiltinClassLoader.defineClass(java.base@15.0.1/BuiltinClassLoader.java:772)
        at jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(java.base@15.0.1/BuiltinClassLoader.java:705)
        at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@15.0.1/BuiltinClassLoader.java:630)
        - locked <0x00000001245e8ac0> (a java.lang.Object)

That is, "JVMCI-native CompilerThread2" is waiting for the lock held by "PmdThread 8". With jargraal (i.e. -XX:-UseJVMCINativeLibrary), this deadlock is resolved by seeing that "JVMCI-native CompilerThread1" is blocked on a lock in which case "PmdThread 8" is allowed to continue without waiting for the compilation to finish [1].

However, with libgraal "JVMCI-native CompilerThread2" can be blocked on a lock inside libgraal which is invisible to HotSpot. As far as HotSpot is concerned, CompilerThread2 is not blocked on a lock.

To handle this, a new mechanism is required for libgraal compiler threads to communicate to HotSpot that they are not blocked. The design of such a mechanism is as follows:

* Each libgraal thread has a "compilation ticks" counter.
* There is also a global compilation ticks counter for libgraal.
* Each libgraal-to-HotSpot call increments the libgraal thread-local compilation ticks counter.
* Every 512 increments to any libgraal thread-local compilation ticks counter also increments the global compilation ticks counter.
* A thread waiting on a blocking libgraal compilation will continue without waiting for the compilation to complete if polling of these counters indicates no compilation progress is being made.

[1] https://github.com/openjdk/jdk/blob/c79e6346d0396b72bc4f7c4371683e7e06ae59e4/src/hotspot/share/compiler/compileBroker.cpp#L1594-L1603