The CompilerToVM.lookupType method uses a class loader to resolve a String to a class. Currently, the system class loader is used if no class loader is provided. When the system class loader is not parallel capable (e.g. when -Djava.system.class.loader is used) and -Xcomp is specified, this can cause deadlock if the thread that is waiting for the blocking compilation holds the system class loader lock.

The attached all-threads.txt shows such a deadlock between the 2 following threads:
* thread #11, name = 'Java: JVMCI-native CompilerThread0'
* thread #3

The solution is to use a more appropriate class loader. For example, when resolving a boot or platform class, the boot or platform class loader should be used.