Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8130025

Tomcat enter stuck status, one thread hold the lock, and all others are blocked

XMLWordPrintable

      FULL PRODUCT VERSION :
      java version "1.7.0_72"
      Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
      Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)

      ADDITIONAL OS VERSION INFORMATION :
      CentOS release 6.5 (Final)

      EXTRA RELEVANT SYSTEM CONFIGURATION :
      Tomcat 7.0.57

      A DESCRIPTION OF THE PROBLEM :
      Hold the lock thread:
      "http-bio-10886-exec-919" daemon prio=10 tid=0x00007fa54c57b800 nid=0x4489 runnable [0x00007fa4579ee000]
         java.lang.Thread.State: RUNNABLE
      at java.util.HashMap.removeEntryForKey(Unknown Source)
      at java.util.HashMap.remove(Unknown Source)
      at sun.security.util.MemoryCache.emptyQueue(Unknown Source)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - locked <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

      all other threads are in the status as below:
      "http-bio-10886-exec-921" daemon prio=10 tid=0x00007fa54c3fd000 nid=0x448e waiting for monitor entry [0x00007fa4576b5000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)



      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      it happened in Prod env with high traffic

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      Tomcat Server response normally
      ACTUAL -
      server stuck there.

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      "http-bio-10886-exec-921" daemon prio=10 tid=0x00007fa54c3fd000 nid=0x448e waiting for monitor entry [0x00007fa4576b5000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa3a416730> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa3a41f3c0> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa3a41f398> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa3a42d0f0> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      "http-bio-10886-exec-919" daemon prio=10 tid=0x00007fa54c57b800 nid=0x4489 runnable [0x00007fa4579ee000]
         java.lang.Thread.State: RUNNABLE
      at java.util.HashMap.removeEntryForKey(Unknown Source)
      at java.util.HashMap.remove(Unknown Source)
      at sun.security.util.MemoryCache.emptyQueue(Unknown Source)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - locked <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa489031f0> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa4890be80> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa4890be58> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa48919bb0> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      "http-bio-10886-exec-916" daemon prio=10 tid=0x00007fa54c0d3000 nid=0x4463 waiting for monitor entry [0x00007fa59d1d3000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa4a9e6b48> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa4a9ef7d8> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa4a9ef7b0> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa4a9fd508> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      "http-bio-10886-exec-913" daemon prio=10 tid=0x00007fa54c1a5800 nid=0x445a waiting for monitor entry [0x00007fa4598d7000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa406b2ce8> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa406bb978> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa406bb950> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa406c96a8> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      "http-bio-10886-exec-915" daemon prio=10 tid=0x00007fa54c13e000 nid=0x4459 waiting for monitor entry [0x00007fa44e7fd000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa448210e0> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa44829d70> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa44829d48> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa44837aa0> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      "http-bio-10886-exec-914" daemon prio=10 tid=0x00007fa460080000 nid=0x4458 waiting for monitor entry [0x00007fa58065a000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa3e5c52a8> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa3e5cdf38> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa3e5cdf10> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa3e5dbc68> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      "http-bio-10886-exec-912" daemon prio=10 tid=0x00007fa460079800 nid=0x4455 waiting for monitor entry [0x00007fa58135f000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.security.util.MemoryCache.put(Unknown Source)
      - waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
      at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
      at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
      at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
      at sun.security.ssl.Handshaker.processLoop(Unknown Source)
      at sun.security.ssl.Handshaker.process_record(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
      - locked <0x00007faa3c4a8aa8> (a sun.security.ssl.SSLSocketImpl)
      - locked <0x00007faa3c4b1738> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
      - locked <0x00007faa3c4b1710> (a java.lang.Object)
      at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
      at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
      at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
      - locked <0x00007faa3c4bf468> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

      REPRODUCIBILITY :
      This bug can be reproduced occasionally.

      CUSTOMER SUBMITTED WORKAROUND :
      Not sure if this solution (https://github.com/netty/netty/issues/832) is workable

      SUPPORT :
      YES

            igerasim Ivan Gerasimov
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: