On 8/09/2023 12:38 am, Simone Bordet wrote:> Hello,
>
> We switched the Jetty builds to Java 21 a while ago, and they randomly
> fail with a hard deadlock during class initialization.
>
> We tried to understand if we were doing something wrong, but the code
> compiles fine, and at first glance there seems to be nothing wrong
> with it, but it may well be a popular Java puzzler that I am not aware
> of :)
>
> We did not see these failures with Java 20, and we do not see them with Java 17.
>
> I was wondering if something changed in Java 21 to cause this deadlock?

Taking a thread dump we see one thread in this state:

java.lang.Thread.State: RUNNABLE
at org.eclipse.jetty.http.HttpFields.build(org.eclipse.jetty.http@12.0.2-SNAPSHOT/HttpFields.java:76)
- waiting on the Class initialization monitor for
org.eclipse.jetty.http.MutableHttpFields
at org.eclipse.jetty.http.HttpFields.<clinit>(org.eclipse.jetty.http@12.0.2-SNAPSHOT/HttpFields.java:67)
"ForkJoinPool-1-worker-1" #23 [1457161] daemon prio=5 os_prio=0
cpu=320.77ms elapsed=8870.98s allocated=18414K defined_classes=928
tid=0x00007f6ec0c6a6f0 nid=1457161 waiting on condition
[0x00007f6e91ffb000]

And other threads are in this state (or similar -- the bottom frame is
different but still triggering the clinit of HttpFields):

java.lang.Thread.State: RUNNABLE
at org.eclipse.jetty.http.HttpTester.parseResponse(org.eclipse.jetty.http@12.0.2-SNAPSHOT/HttpTester.java:274)
- waiting on the Class initialization monitor for
org.eclipse.jetty.http.HttpFields
at org.eclipse.jetty.http.HttpTester.parseResponse(org.eclipse.jetty.http@12.0.2-SNAPSHOT/HttpTester.java:261)
"ForkJoinPool-1-worker-2" #24 [1457162] daemon prio=5 os_prio=0
cpu=136.15ms elapsed=8870.96s allocated=3303K defined_classes=340
tid=0x00007f6e4c01ddb0 nid=1457162 waiting on condition
[0x00007f6e91efc000]

Class hierarchy:

MutableHttpFields implements HttpFields.Mutable
HttpFields.Mutable extends HttpFields

Full thread dump:
https://gist.github.com/olamy/7e883adcf51b2410337abfa775a79a0b

Code:
https://github.com/eclipse/jetty.project/blob/jetty-12.0.1/jetty-core/jetty-http/src/main/java/org/eclipse/jetty/http/HttpFields.java

--
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless. Victoria Livschitz