Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8227375

TIMED_WAIT often hangs forever

XMLWordPrintable

    • x86_64
    • linux

      ADDITIONAL SYSTEM INFORMATION :
      CentOS Linux release 7.6.1810 (Core)
      Linux edimq-consumer-fs2fts-az1-1 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

      openjdk version "11.0.3" 2019-04-16
      OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)
      OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.3+7, mixed mode)

      A DESCRIPTION OF THE PROBLEM :
      Object.wait() with timeout set (30 seconds in my case) often (at least every couple of hours) hangs forever. While debugging the issue, I discovered that it suddenly unlocks after I run `jhsdb jstack --pid <pid>` and the application then continues operation.

      Example stack trace (see thread `pool-2-thread-3`):

      2019-06-27 06:00:16
      Full thread dump OpenJDK 64-Bit Server VM (11.0.3+7 mixed mode):

      Threads class SMR info:
      _java_thread_list=0x00007f7e50005ef0, length=56, elements={
      0x00007f7e9813f000, 0x00007f7e98141000, 0x00007f7e98155800, 0x00007f7e98158000,
      0x00007f7e9815a000, 0x00007f7e9815c800, 0x00007f7e981eb000, 0x00007f7e981f9800,
      0x00007f7e98c1c800, 0x00007f7e98c31800, 0x00007f7e98e41000, 0x00007f7e98be6000,
      0x00007f7e98e6f800, 0x00007f7e98f38800, 0x00007f7e98f56000, 0x00007f7e98011000,
      0x00007f7e0807e800, 0x00007f7e0807f800, 0x00007f7e08081800, 0x00007f7e08083000,
      0x00007f7e08085800, 0x00007f7e08087800, 0x00007f7e08089000, 0x00007f7e0808b000,
      0x00007f7e08091000, 0x0000000002083800, 0x00007f7e00530000, 0x00007f7e005e6000,
      0x00007f7e005f0000, 0x00007f7e006f3800, 0x00007f7e1c001800, 0x00007f7e0071d000,
      0x00007f7e20009000, 0x00007f7e2000a800, 0x00007f7e2000c800, 0x00007f7e2000e800,
      0x00007f7e20010800, 0x00007f7e20012800, 0x00007f7e20014800, 0x00007f7e20016800,
      0x00007f7e20018800, 0x00007f7e2001a800, 0x00007f7e2001c800, 0x00007f7e2001e800,
      0x00007f7e20021000, 0x00007f7e20023000, 0x00007f7e20025000, 0x00007f7e20027000,
      0x00007f7e20029000, 0x00007f7e2002b000, 0x00007f7e2002c800, 0x00007f7e2002f000,
      0x00007f7e1c003000, 0x00007f7e1805c800, 0x00007f7e18061000, 0x00007f7e50020800
      }

      "Reference Handler" #2 daemon prio=10 os_prio=0 cpu=14.21ms elapsed=72297.44s tid=0x00007f7e9813f000 nid=0x806 waiting on condition [0x00007f7e9c201000]
         java.lang.Thread.State: RUNNABLE
              at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.3/Native Method)
              at java.lang.ref.Reference.processPendingReferences(java.base@11.0.3/Unknown Source)
              at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.3/Unknown Source)

      "Finalizer" #3 daemon prio=8 os_prio=0 cpu=7.27ms elapsed=72297.44s tid=0x00007f7e98141000 nid=0x807 in Object.wait() [0x00007f7e9c100000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(java.base@11.0.3/Native Method)
              - waiting on <no object reference available>
              at java.lang.ref.ReferenceQueue.remove(java.base@11.0.3/Unknown Source)
              - waiting to re-lock in wait() <0x000000008b832db0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(java.base@11.0.3/Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.3/Unknown Source)

      "Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=1.13ms elapsed=72297.43s tid=0x00007f7e98155800 nid=0x808 runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=28235.13ms elapsed=72297.43s tid=0x00007f7e98158000 nid=0x809 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
         No compile task

      "C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=3375.83ms elapsed=72297.43s tid=0x00007f7e9815a000 nid=0x80a waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
         No compile task

      "Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=1915.46ms elapsed=72297.43s tid=0x00007f7e9815c800 nid=0x80b runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "Service Thread" #9 daemon prio=9 os_prio=0 cpu=0.06ms elapsed=72297.40s tid=0x00007f7e981eb000 nid=0x80c runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "Common-Cleaner" #10 daemon prio=8 os_prio=0 cpu=33.12ms elapsed=72297.40s tid=0x00007f7e981f9800 nid=0x80e in Object.wait() [0x00007f7e8129f000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(java.base@11.0.3/Native Method)
              - waiting on <no object reference available>
              at java.lang.ref.ReferenceQueue.remove(java.base@11.0.3/Unknown Source)
              - waiting to re-lock in wait() <0x000000008bce30a0> (a java.lang.ref.ReferenceQueue$Lock)
              at jdk.internal.ref.CleanerImpl.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)
              at jdk.internal.misc.InnocuousThread.run(java.base@11.0.3/Unknown Source)

      "Log4j2-TF-2-Scheduled-1" #12 daemon prio=5 os_prio=0 cpu=151.49ms elapsed=72296.56s tid=0x00007f7e98c1c800 nid=0x811 waiting on condition [0x00007f7e80165000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008bce33e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Log4j2-TF-3-AsyncLoggerConfig-2" #13 daemon prio=5 os_prio=0 cpu=142220.05ms elapsed=72296.54s tid=0x00007f7e98c31800 nid=0x812 waiting on condition [0x00007f7e55ad4000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008bce3848> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
              at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
              at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
              at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Log4j2-TF-5-Scheduled-3" #14 daemon prio=5 os_prio=0 cpu=150.43ms elapsed=72296.40s tid=0x00007f7e98e41000 nid=0x813 waiting on condition [0x00007f7e552d3000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc8afb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Log4j2-TF-6-AsyncLoggerConfig-4" #15 daemon prio=5 os_prio=0 cpu=98920.11ms elapsed=72296.39s tid=0x00007f7e98be6000 nid=0x814 waiting on condition [0x00007f7e551d2000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc8b1f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
              at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
              at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
              at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Timer-0" #16 prio=5 os_prio=0 cpu=94.60ms elapsed=72296.32s tid=0x00007f7e98e6f800 nid=0x815 in Object.wait() [0x00007f7e2f9d8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(java.base@11.0.3/Native Method)
              - waiting on <no object reference available>
              at java.util.TimerThread.mainLoop(java.base@11.0.3/Unknown Source)
              - waiting to re-lock in wait() <0x000000008cc1fbc8> (a java.util.TaskQueue)
              at java.util.TimerThread.run(java.base@11.0.3/Unknown Source)

      "maintenance" #19 prio=5 os_prio=0 cpu=44546.83ms elapsed=72296.16s tid=0x00007f7e98f38800 nid=0x81a in Object.wait() [0x00007f7e2f3d1000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(java.base@11.0.3/Native Method)
              - waiting on <no object reference available>
              at java.lang.Object.wait(java.base@11.0.3/Unknown Source)
              at java_lang_Object$wait.call(Unknown Source)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupMaintenanceThread_closure2.doCall(ClouEDIMQPoll.groovy:84)
              - waiting to re-lock in wait() <0x000000008cc8b310> (a java.lang.Object)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupMaintenanceThread_closure2.doCall(ClouEDIMQPoll.groovy)
              at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.3/Native Method)
              at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:399)
              at groovy.lang.Closure.run(Closure.java:486)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Thread-1" #20 prio=5 os_prio=0 cpu=72.36ms elapsed=72296.10s tid=0x00007f7e98f56000 nid=0x81b in Object.wait() [0x00007f7e2f4d3000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(java.base@11.0.3/Native Method)
              - waiting on <0x000000008d3544a0> (a java.lang.Object)
              at java.lang.Object.wait(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:404)
              - waiting to re-lock in wait() <0x000000008d3544a0> (a java.lang.Object)
              at org.eclipse.jetty.server.Server.join(Server.java:617)
              at spark.embeddedserver.jetty.EmbeddedJettyServer.join(EmbeddedJettyServer.java:158)
              at spark.Service.lambda$init$2(Service.java:508)
              at spark.Service$$Lambda$95/0x00000001002f5c40.run(Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "DestroyJavaVM" #22 prio=5 os_prio=0 cpu=1268.35ms elapsed=72296.07s tid=0x00007f7e98011000 nid=0x7ff waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "qtp1919147235-23" #23 prio=5 os_prio=0 cpu=552.46ms elapsed=72296.02s tid=0x00007f7e0807e800 nid=0x81c waiting on condition [0x00007f7e2f2cf000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008e28da68> (a java.util.concurrent.SynchronousQueue$TransferStack)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.SynchronousQueue.put(java.base@11.0.3/Unknown Source)
              at cz.aimtec.mq.concurrent.ReusableWorker.submitJob(ReusableWorker.groovy:46)
              at cz.aimtec.mq.concurrent.Worker$submitJob.call(Unknown Source)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupHttpServer_closure6$_closure17.doCall(ClouEDIMQPoll.groovy:195)
              at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:421)
              at cz.aimtec.mq.concurrent.SpecializedWorkerPool.withWorker(SpecializedWorkerPool.groovy:42)
              at jdk.internal.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:179)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:70)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupHttpServer_closure6.doCall(ClouEDIMQPoll.groovy:194)
              at jdk.internal.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:50)
              at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:122)
              at com.sun.proxy.$Proxy22.handle(Unknown Source)
              at spark.RouteImpl$1.handle(RouteImpl.java:72)
              at spark.http.matching.Routes.execute(Routes.java:61)
              at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:130)
              at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1568)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.Server.handle(Server.java:564)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
              at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
              at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-24" #24 prio=5 os_prio=0 cpu=258.35ms elapsed=72296.02s tid=0x00007f7e0807f800 nid=0x81d runnable [0x00007f7e2f1d0000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPoll.wait(java.base@11.0.3/Native Method)
              at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.3/Unknown Source)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.3/Unknown Source)
              - locked <0x000000008d354cd0> (a sun.nio.ch.Util$2)
              - locked <0x000000008d354c78> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
              at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-25-acceptor-0@37c891ed-ServerConnector@7102e363{HTTP/1.1,[http/1.1]}{127.0.0.1:8088}" #25 prio=3 os_prio=0 cpu=251.90ms elapsed=72296.02s tid=0x00007f7e08081800 nid=0x81e runnable [0x00007f7e2f0cf000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.ServerSocketChannelImpl.accept0(java.base@11.0.3/Native Method)
              at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.3/Unknown Source)
              at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352)
              at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-26" #26 prio=5 os_prio=0 cpu=253.47ms elapsed=72296.02s tid=0x00007f7e08083000 nid=0x81f runnable [0x00007f7e2efce000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPoll.wait(java.base@11.0.3/Native Method)
              at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.3/Unknown Source)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.3/Unknown Source)
              - locked <0x000000008d300bd8> (a sun.nio.ch.Util$2)
              - locked <0x000000008d300b80> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
              at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-27" #27 prio=5 os_prio=0 cpu=274.29ms elapsed=72296.02s tid=0x00007f7e08085800 nid=0x820 waiting on condition [0x00007f7e2eecd000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d32b2e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:565)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:50)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:628)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-28" #28 prio=5 os_prio=0 cpu=248.07ms elapsed=72296.02s tid=0x00007f7e08087800 nid=0x821 waiting on condition [0x00007f7e2edca000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008dda89f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.SynchronousQueue.take(java.base@11.0.3/Unknown Source)
              at cz.aimtec.mq.concurrent.ReusableWorker.submitJob(ReusableWorker.groovy:48)
              at cz.aimtec.mq.concurrent.Worker$submitJob.call(Unknown Source)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupHttpServer_closure6$_closure17.doCall(ClouEDIMQPoll.groovy:195)
              at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:421)
              at cz.aimtec.mq.concurrent.SpecializedWorkerPool.withWorker(SpecializedWorkerPool.groovy:42)
              at jdk.internal.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:179)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:70)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupHttpServer_closure6.doCall(ClouEDIMQPoll.groovy:194)
              at jdk.internal.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:50)
              at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:122)
              at com.sun.proxy.$Proxy22.handle(Unknown Source)
              at spark.RouteImpl$1.handle(RouteImpl.java:72)
              at spark.http.matching.Routes.execute(Routes.java:61)
              at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:130)
              at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1568)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.Server.handle(Server.java:564)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
              at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
              at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-29" #29 prio=5 os_prio=0 cpu=260.82ms elapsed=72296.02s tid=0x00007f7e08089000 nid=0x822 waiting on condition [0x00007f7e2eccb000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d300e30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:173)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "qtp1919147235-30" #30 prio=5 os_prio=0 cpu=257.32ms elapsed=72296.02s tid=0x00007f7e0808b000 nid=0x823 waiting on condition [0x00007f7e2ebca000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d32b6a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:173)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Scheduler-1081622769" #31 prio=5 os_prio=0 cpu=12.43ms elapsed=72296.02s tid=0x00007f7e08091000 nid=0x824 waiting on condition [0x00007f7e2eac9000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d308d78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Scheduler-1011590512" #32 prio=5 os_prio=0 cpu=18.34ms elapsed=72255.30s tid=0x0000000002083800 nid=0x846 waiting on condition [0x00007f7e80c73000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d32a290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Thread-0 (ActiveMQ-client-netty-threads)" #33 daemon prio=5 os_prio=0 cpu=1319.96ms elapsed=72255.09s tid=0x00007f7e00530000 nid=0x848 runnable [0x00007f7e2e2b9000]
         java.lang.Thread.State: RUNNABLE
              at io.netty.channel.epoll.Native.epollWait0(Native Method)
              at io.netty.channel.epoll.Native.epollWait(Native.java:114)
              at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
              at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
              at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-1 (ActiveMQ-client-netty-threads)" #34 daemon prio=5 os_prio=0 cpu=1312.50ms elapsed=72255.08s tid=0x00007f7e005e6000 nid=0x849 runnable [0x00007f7e2e1b8000]
         java.lang.Thread.State: RUNNABLE
              at io.netty.channel.epoll.Native.epollWait0(Native Method)
              at io.netty.channel.epoll.Native.epollWait(Native.java:114)
              at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
              at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
              at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-2 (ActiveMQ-client-netty-threads)" #35 daemon prio=5 os_prio=0 cpu=72129.20ms elapsed=72255.07s tid=0x00007f7e005f0000 nid=0x84a runnable [0x00007f7e2e0b7000]
         java.lang.Thread.State: RUNNABLE
              at io.netty.channel.epoll.Native.epollWait0(Native Method)
              at io.netty.channel.epoll.Native.epollWait(Native.java:114)
              at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
              at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
              at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-0 (ActiveMQ-client-global-scheduled-threads)" #36 daemon prio=5 os_prio=0 cpu=57.83ms elapsed=72255.02s tid=0x00007f7e006f3800 nid=0x84b waiting on condition [0x00007f7e2cdb5000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d3e5ac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-1 (ActiveMQ-client-global-scheduled-threads)" #37 daemon prio=5 os_prio=0 cpu=82.20ms elapsed=72255.01s tid=0x00007f7e1c001800 nid=0x84c waiting on condition [0x00007f7e2ccb4000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d3e5ac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-11" #39 prio=5 os_prio=0 cpu=5889.83ms elapsed=72254.96s tid=0x00007f7e0071d000 nid=0x84e waiting on condition [0x00007f7e2cab1000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008eec1258> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ExecutorCompletionService.take(java.base@11.0.3/Unknown Source)
              at cz.aimtec.mq.concurrent.DefaultParallelWorker$_performTask_closure3.doCall(DefaultParallelWorker.groovy:44)
              at cz.aimtec.mq.concurrent.DefaultParallelWorker$_performTask_closure3.call(DefaultParallelWorker.groovy)
              at org.codehaus.groovy.runtime.DefaultGroovyMethods.times(DefaultGroovyMethods.java:15584)
              at cz.aimtec.mq.concurrent.DefaultParallelWorker.performTask(DefaultParallelWorker.groovy:43)
              at cz.aimtec.mq.concurrent.CoordinatedParallelWorker.performTask(CoordinatedParallelWorker.groovy:32)
              at cz.aimtec.mq.concurrent.ParallelWorker$performTask.call(Unknown Source)
              at cz.aimtec.mq.poll.ClouEDIMQPoll.multithreadedPoll(ClouEDIMQPoll.groovy:275)
              at jdk.internal.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:351)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:64)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupHttpServer_closure6$_closure17$_closure18.doCall(ClouEDIMQPoll.groovy:198)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_setupHttpServer_closure6$_closure17$_closure18.doCall(ClouEDIMQPoll.groovy)
              at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:399)
              at cz.aimtec.mq.concurrent.ReusableWorker$_closure1.doCall(ReusableWorker.groovy:35)
              at cz.aimtec.mq.concurrent.ReusableWorker$_closure1.call(ReusableWorker.groovy)
              at groovy.lang.Closure.run(Closure.java:486)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-1" #40 prio=5 os_prio=0 cpu=97689.78ms elapsed=72254.94s tid=0x00007f7e20009000 nid=0x84f waiting on condition [0x00007f7e2c9b1000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-2" #41 prio=5 os_prio=0 cpu=95434.52ms elapsed=72254.94s tid=0x00007f7e2000a800 nid=0x850 waiting on condition [0x00007f7e2c8b0000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-3" #42 prio=5 os_prio=0 cpu=95887.10ms elapsed=72254.94s tid=0x00007f7e2000c800 nid=0x851 in Object.wait() [0x00007f7e2c7ae000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(java.base@11.0.3/Native Method)
              - waiting on <no object reference available>
              at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:302)
              - eliminated <0x000000008ec77060> (a org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
              at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:276)
              - waiting to re-lock in wait() <0x000000008ec77060> (a org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
              at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.saveToOutputStream(ClientLargeMessageImpl.java:115)
              at org.apache.activemq.artemis.api.core.client.ClientMessage$saveToOutputStream.call(Unknown Source)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_receiveTo_closure11.doCall(ClouEDIMQPoll.groovy:391)
              at jdk.internal.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:421)
              at org.codehaus.groovy.runtime.IOGroovyMethods.withStream(IOGroovyMethods.java:1297)
              at org.codehaus.groovy.runtime.ResourceGroovyMethods.withOutputStream(ResourceGroovyMethods.java:1828)
              at org.codehaus.groovy.runtime.dgm$1079.invoke(Unknown Source)
              at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
              at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
              at cz.aimtec.mq.poll.ClouEDIMQPoll.receiveTo(ClouEDIMQPoll.groovy:390)
              at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:351)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:64)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:192)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_poll_closure10.doCall(ClouEDIMQPoll.groovy:300)
              at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:421)
              at cz.aimtec.mq.ClouEDIMQConsumerPool$1.receiveMessage(ClouEDIMQConsumerPool.groovy:72)
              at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:179)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:70)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
              at cz.aimtec.mq.poll.ClouEDIMQPoll.poll(ClouEDIMQPoll.groovy:297)
              at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:351)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:64)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_multithreadedPoll_closure8.doCall(ClouEDIMQPoll.groovy:263)
              at cz.aimtec.mq.poll.ClouEDIMQPoll$_multithreadedPoll_closure8.doCall(ClouEDIMQPoll.groovy)
              at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/Unknown Source)
              at java.lang.reflect.Method.invoke(java.base@11.0.3/Unknown Source)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
              at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
              at groovy.lang.Closure.call(Closure.java:405)
              at groovy.lang.Closure.call(Closure.java:399)
              at cz.aimtec.mq.concurrent.DefaultParallelWorker$_performTask_closure1.doCall(DefaultParallelWorker.groovy:29)
              at cz.aimtec.mq.concurrent.DefaultParallelWorker$_performTask_closure1.call(DefaultParallelWorker.groovy)
              at java.util.concurrent.FutureTask.run(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.FutureTask.run(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-4" #43 prio=5 os_prio=0 cpu=96026.49ms elapsed=72254.93s tid=0x00007f7e2000e800 nid=0x852 waiting on condition [0x00007f7e2c6ae000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-5" #44 prio=5 os_prio=0 cpu=96019.11ms elapsed=72254.93s tid=0x00007f7e20010800 nid=0x853 waiting on condition [0x00007f7e2c5ad000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-6" #45 prio=5 os_prio=0 cpu=96527.35ms elapsed=72254.93s tid=0x00007f7e20012800 nid=0x854 waiting on condition [0x00007f7e2c4ac000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-7" #46 prio=5 os_prio=0 cpu=95650.97ms elapsed=72254.93s tid=0x00007f7e20014800 nid=0x855 waiting on condition [0x00007f7e2c3ab000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-8" #47 prio=5 os_prio=0 cpu=96623.17ms elapsed=72254.93s tid=0x00007f7e20016800 nid=0x856 waiting on condition [0x00007f7e2c2aa000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-9" #48 prio=5 os_prio=0 cpu=95652.77ms elapsed=72254.93s tid=0x00007f7e20018800 nid=0x857 waiting on condition [0x00007f7e2c1a9000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-10" #49 prio=5 os_prio=0 cpu=95589.40ms elapsed=72254.93s tid=0x00007f7e2001a800 nid=0x858 waiting on condition [0x00007f7df5ffc000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-11" #50 prio=5 os_prio=0 cpu=97121.34ms elapsed=72254.93s tid=0x00007f7e2001c800 nid=0x859 waiting on condition [0x00007f7df5efb000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-12" #51 prio=5 os_prio=0 cpu=96606.84ms elapsed=72254.93s tid=0x00007f7e2001e800 nid=0x85a waiting on condition [0x00007f7df5dfa000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-13" #52 prio=5 os_prio=0 cpu=96339.57ms elapsed=72254.93s tid=0x00007f7e20021000 nid=0x85b waiting on condition [0x00007f7df5cf9000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-14" #53 prio=5 os_prio=0 cpu=96468.08ms elapsed=72254.93s tid=0x00007f7e20023000 nid=0x85c waiting on condition [0x00007f7df5bf8000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-15" #54 prio=5 os_prio=0 cpu=96236.45ms elapsed=72254.93s tid=0x00007f7e20025000 nid=0x85d waiting on condition [0x00007f7df5af7000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-16" #55 prio=5 os_prio=0 cpu=95913.72ms elapsed=72254.93s tid=0x00007f7e20027000 nid=0x85e waiting on condition [0x00007f7df59f6000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-17" #56 prio=5 os_prio=0 cpu=96528.20ms elapsed=72254.93s tid=0x00007f7e20029000 nid=0x85f waiting on condition [0x00007f7df58f5000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-18" #57 prio=5 os_prio=0 cpu=95982.01ms elapsed=72254.92s tid=0x00007f7e2002b000 nid=0x860 waiting on condition [0x00007f7df57f4000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-19" #58 prio=5 os_prio=0 cpu=95514.35ms elapsed=72254.92s tid=0x00007f7e2002c800 nid=0x861 waiting on condition [0x00007f7df56f3000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "pool-2-thread-20" #59 prio=5 os_prio=0 cpu=95956.35ms elapsed=72254.92s tid=0x00007f7e2002f000 nid=0x862 waiting on condition [0x00007f7df55f2000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008cc38f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at java.lang.Thread.run(java.base@11.0.3/Unknown Source)

      "Thread-2 (ActiveMQ-client-global-scheduled-threads)" #60 daemon prio=5 os_prio=0 cpu=20.24ms elapsed=72225.02s tid=0x00007f7e1c003000 nid=0x868 waiting on condition [0x00007f7e2e3ba000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d3e5ac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-3 (ActiveMQ-client-global-scheduled-threads)" #61 daemon prio=5 os_prio=0 cpu=35.59ms elapsed=72195.01s tid=0x00007f7e1805c800 nid=0x86d waiting on condition [0x00007f7df02c8000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d3e5ac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Thread-4 (ActiveMQ-client-global-scheduled-threads)" #82 daemon prio=5 os_prio=0 cpu=32.97ms elapsed=72165.01s tid=0x00007f7e18061000 nid=0x4b50 waiting on condition [0x00007f7deffc7000]
         java.lang.Thread.State: WAITING (parking)
              at jdk.internal.misc.Unsafe.park(java.base@11.0.3/Native Method)
              - parking to wait for <0x000000008d3e5ac0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/Unknown Source)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      "Attach Listener" #1485 daemon prio=9 os_prio=0 cpu=2.66ms elapsed=119.40s tid=0x00007f7e50020800 nid=0x4191 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "VM Thread" os_prio=0 cpu=8107.94ms elapsed=72297.44s tid=0x00007f7e98136800 nid=0x805 runnable

      "GC Thread#0" os_prio=0 cpu=26397.47ms elapsed=72297.46s tid=0x00007f7e98038800 nid=0x800 runnable

      "GC Thread#1" os_prio=0 cpu=26077.26ms elapsed=72297.22s tid=0x00007f7e68001000 nid=0x80f runnable

      "GC Thread#2" os_prio=0 cpu=26095.48ms elapsed=72296.21s tid=0x00007f7e6805e800 nid=0x816 runnable

      "GC Thread#3" os_prio=0 cpu=25745.73ms elapsed=72296.21s tid=0x00007f7e68060000 nid=0x817 runnable

      "G1 Main Marker" os_prio=0 cpu=319.49ms elapsed=72297.46s tid=0x00007f7e98067800 nid=0x801 runnable

      "G1 Conc#0" os_prio=0 cpu=133822.11ms elapsed=72297.46s tid=0x00007f7e98069000 nid=0x802 runnable

      "G1 Refine#0" os_prio=0 cpu=11.14ms elapsed=72297.45s tid=0x00007f7e980dc800 nid=0x803 runnable

      "G1 Young RemSet Sampling" os_prio=0 cpu=6886.61ms elapsed=72297.45s tid=0x00007f7e980de800 nid=0x804 runnable
      "VM Periodic Task Thread" os_prio=0 cpu=18383.54ms elapsed=72297.40s tid=0x00007f7e981ed800 nid=0x80d waiting on condition

      JNI global refs: 27, weak refs: 0

      #####

      The code of waitCompletion() (see https://github.com/apache/activemq-artemis/blob/951a0615559e9a3b417a432a2415dcbd1252827c/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/impl/LargeMessageControllerImpl.java#L284):

          public synchronized boolean waitCompletion(long timeWait) throws ActiveMQException {
              if (this.outStream == null) {
                  return false;
              } else {
                  long timeOut;
                  if (timeWait != 0L) {
                      timeOut = System.currentTimeMillis() + timeWait;
                  } else {
                      timeOut = System.currentTimeMillis() + this.readTimeout;
                  }

                  while(!this.streamEnded && this.handledException == null) {
                      try {
                          this.wait(timeWait == 0L ? this.readTimeout : timeWait);
                      } catch (InterruptedException var6) {
                          throw new ActiveMQInterruptedException(var6);
                      }

                      if (!this.streamEnded && this.handledException == null) {
                          if (timeWait != 0L && System.currentTimeMillis() > timeOut) {
                              throw ActiveMQClientMessageBundle.BUNDLE.timeoutOnLargeMessage();
                          }

                          if (System.currentTimeMillis() > timeOut && !this.packetAdded) {
                              throw ActiveMQClientMessageBundle.BUNDLE.timeoutOnLargeMessage();
                          }
                      }
                  }

                  this.checkException();
                  return this.streamEnded;
              }

      #####

      I've extracted the heap dump as well and observed, that `this.readTimeout` was set correctly (30000).

      So far, I've been unable to reproduce the issue manually.


      FREQUENCY : often


            tongwan Andrew Wang
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: