FULL PRODUCT VERSION :
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Very occasionally (maybe once every few days) we see a condition where all of our application threads are in WAITING state trying to acquire a ReentrantLock, and a single thread is in RUNNABLE state in CompletableFuture.cleanStack(). The thread that is in RUNNABLE state has already acquired the lock that all of the other threads are trying to acquire, but once it calls CompletableFuture.cleanStack() it appears to get stuck in an infinite loop. We also see one other thread with CompletableFuture in its stack, but I'm not sure whether that is significant. Once our application enters this state no threads make any progress, and the JVM has to be restarted. Example stack trace with most threads (and some details) omitted for brevity:

#693 daemon prio=10 os_prio=2 tid=0x00007f7462f52000 nid=0x5825 runnable [0x00007f73c7670000]
   java.lang.Thread.State: RUNNABLE
at java.util.concurrent.CompletableFuture.cleanStack(CompletableFuture.java:483)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1768)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
...
at java.lang.Thread.run(Thread.java:745)

#1239 daemon prio=5 os_prio=2 tid=0x00007f779c028000 nid=0x5a43 waiting on condition [0x00007f7340c12000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005346abdf0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
...
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

#698 daemon prio=10 os_prio=2 tid=0x00007f7462f56000 nid=0x582a waiting on condition [0x00007f73c716c000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005346abdf0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
...
at java.lang.Thread.run(Thread.java:745)

#929 daemon prio=5 os_prio=2 tid=0x00007f750400a000 nid=0x590d waiting on condition [0x00007f7354948000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005346abdf0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
...
at java.lang.Thread.run(Thread.java:745)


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The RUNNABLE thread should exit CompletableFuture.cleanStack() quickly
ACTUAL -
RUNNABLE thread is stuck in CompletableFuture.cleanStack() indefinitely until the JVM is killed

REPRODUCIBILITY :
This bug can be reproduced rarely.

CUSTOMER SUBMITTED WORKAROUND :
When our application gets into this state the only fix is to restart the JVM