Bad Thread Utilization in ForkJoinPool

The thread utilization in ForkJoinPool seems to drop significantly from time to time. I saw this issue while testing the performance of continuations. The actual issue is probably related to the ForkJoinPool and unrelated to the continuation support itself.

The attached application schedules 8000 virtual threads. Each thread yields rapidly, which stresses the continuation and the ForkJoinPool implementation. This workload is executed for multiple iterations.

The example output below shows that there are some iterations that are slower. This happens pretty randomly, there are also runs where all iterations are fast.

For the iteration that is especially slow, profiling shows that only 4 of the 30+ ForkJoinPool worker threads were actually used in this iteration, even though there should be sufficient work in the queue. The CPU usage of the whole machine drops significantly during that particular iteration.

Additionally, I also tested this application with GraalVM Native Image on both JDK 17 and 19 (GraalVM Native Image supports virtual threads on JDK 17 as well). The JDK 17 ForkJoinPool does not show this problematic behavior, while the JDK 19 implementation shows the same problematic behavior as on HotSpot. The overall performance is also higher on JDK 17 (the ForkJoinPool is the major bottleneck on JDK 19).

How to reproduce:
$JAVA_HOME/bin/javac --enable-preview --release 19 ContinuationPerformance.java
$JAVA_HOME/bin/java --enable-preview ContinuationPerformance

Example output, executed with JDK 19 (HotSpot, C2):
Finished after 8489 ms
Finished after 3918 ms
Finished after 3926 ms
Finished after 4040 ms
Finished after 4145 ms
Finished after 31332 ms
Finished after 4010 ms
Finished after 4244 ms
Finished after 4111 ms
Finished after 4198 ms
Finished after 7713 ms
Finished after 5378 ms
Finished after 4447 ms
Finished after 4365 ms
Finished after 5803 ms
Finished after 4549 ms
Finished after 4312 ms
Finished after 4373 ms
Finished after 4470 ms
Finished after 4402 ms