Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8355036

Java 24: All Virtual Threads Hang

XMLWordPrintable

    • x86_64
    • linux_ubuntu

      ADDITIONAL SYSTEM INFORMATION :
      Any Linux-based distro. Could be for all OS's

      A DESCRIPTION OF THE PROBLEM :
      We have a quarkus application using virtual threads. I don't think this has anything to do with Quarkus, Vertx, Netty etc but more within the JVM itself. With Java 24 we expected pinning deadlocks to be eradicated but it seems like this is not the case. We have instances where all virtual threads seem to stop being scheduled as the current carrier threads are all stuck with pinned virtual threads and do not seem to become unstuck. Platform threads are all fine and can continue, but all virtual threads are blocked from being scheduled as the carriers are all stuck.
      A thread dump shows some interesting things like:
      #12376 "v-uk-es-5181" virtual
            java.base/jdk.internal.misc.Unsafe.park(Native Method)
            java.base/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:821)
            java.base/java.lang.VirtualThread.park(VirtualThread.java:759)
            java.base/java.lang.System$1.parkVirtualThread(System.java:2279)
            java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:367)
            java.base/sun.nio.ch.Poller.poll(Poller.java:197)
            java.base/sun.nio.ch.Poller.poll(Poller.java:142)
            java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:174)
            java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:200)
            java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:308)
            java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:345)
            java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:790)
            java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:983)
            java.base/java.net.Socket$SocketInputStream.read(Socket.java:970)
            java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
            java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)
            java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
            java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461)
            java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)
            java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
            com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:64)
            com.mysql.cj.protocol.a.SimplePacketReader.readHeaderLocal(SimplePacketReader.java:81)
            com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
            com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
            com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
            com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
            com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
            com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
            com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:576)
            com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:762)
            com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:701)
            com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1050)
            com.mysql.cj.NativeSession.execSQL(NativeSession.java:660)
            com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:889)
            com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:968)...

      And
      #41 "ForkJoinPool-1-worker-1"
            java.base/jdk.internal.vm.Continuation.run(Continuation.java:254)
            java.base/java.lang.VirtualThread.runContinuation(VirtualThread.java:303)
            java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1735)
            java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
            java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1650)
            java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1394)
            java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1970)
            java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
      #46 "ForkJoinPool-1-worker-2"
            java.base/jdk.internal.vm.Continuation.run(Continuation.java:254)
            java.base/java.lang.VirtualThread.runContinuation(VirtualThread.java:303)
            java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1735)
            java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
            java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1650)
            java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1394)
            java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1970)
            java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
      #5331 "ForkJoinPool-1-worker-3"
            java.base/jdk.internal.misc.Unsafe.park(Native Method)
            java.base/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:2059)
            java.base/java.util.concurrent.ForkJoinPool.deactivate(ForkJoinPool.java:2013)
            java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1978)
            java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

      Thread dumps taken minutes apart show all virtual threads stuck in the same state.

      This happens almost instantly if -Djdk.virtualThreadScheduler.parallelism is set to 1, with it at 2 (or default on a 2 core VM then it happens within 10 minutes of the JVM servicing traffic, and if -Djdk.virtualThreadScheduler.parallelism is set to say 8 on the same server then it does not lock up.

      My understanding is that with Java 24, pinning should only happen in very low level code and thus unlikely to cause deadlock issues, but this seems to not be the case?

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      I have not been able to create a simple reproducer but would happily work with the assigned engineer to reproduce on our test environment and share detailed thread dumps and JFR events of pinning

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      JVM should not lock up
      ACTUAL -
      All virtual threads lock up

        1. 12056.txt
          7 kB
        2. threaddump.txt
          7 kB
        3. threadprint.txt
          11 kB

            Unassigned Unassigned
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: