bash-2.05$ head -615 pstack.1 | tail -4
----------------- lwp# 62 / thread# 62 --------------------
 008af0d8 ???????? (1084e, ffffffff, 6a4d2, 7f2c0fb0, 7d6741b4, 7f46a300)
 f9c79414 ???????? (839a20d0, b7, 78142388, 78142388, 8, 0)
 f9c05850 ???????? (652fbb40, b6, 0, f9c16230, 18, 652fba40)
bash-2.05$ head -615 pstack.2 | tail -4
----------------- lwp# 62 / thread# 62 --------------------
 008af0d4 ???????? (105fc, ffffffff, 6a4d2, 7f2c0fb0, 7d7b21d4, 7f46a300)
 f9c79414 ???????? (839a20d0, b7, 78142388, 78142388, 8, 0)
 f9c05850 ???????? (652fbb40, b6, 0, f9c16230, 18, 652fba40)
bash-2.05$ head -615 pstack.3 | tail -4
----------------- lwp# 62 / thread# 62 --------------------
 008af0a4 ???????? (103ac, ffffffff, 6a4d2, 7f2c0fb0, 7d7788bc, 7f46a300)
 f9c79414 ???????? (839a20d0, b7, 78142388, 78142388, 8, 0)
 f9c05850 ???????? (652fbb40, b6, 0, f9c16230, 18, 652fba40)

Diffs between pstacks show that the Thread t@62 is moving very slowly.

If this hasnt changed, thread 62 has been running for a long while.
bash-2.05$ grep 12157 prstat.out | grep 62
 12157 wmadmin1 2425M 2391M cpu30 0 10 1:51:47 0.0% java/62


hsdb> where t@62

Java Stack Trace for HTTP Handler 171.177.166.223
Thread state = IN_JAVA
 - public java.math.BigInteger pow(int) @0xf09f07e0 @bci = 88, pc = 0xf9c05858 (Interpreted)
 - private static java.math.BigInteger timesTenToThe(java.math.BigInteger, int) @0xf11974f0 @bci = 8, pc = 0xf9c05858 (Interpreted)
 - public void <init>(java.lang.String) @0xf1196568 @bci = 448, pc = 0xfa48e228 (Compiled)
 - private java.math.BigDecimal getBigDecimal(java.lang.Object) @0xf1bb67f0 @bci = 26, line = 404, pc = 0xfa501aa4 (Compiled)
 - protected void evalSymComparator(com.wm.lang.flow.ExpressionEvaluator$Token) @0xf1bb8928 @bci = 2, line = 1199, pc = 0xfa43a150 (Compiled)
...

We are again in BigInteger.pow(int) ...

(dbx) print __1cUSafepointSynchronizeG_state_
__1cUSafepointSynchronizeG_state_ = 1
036 _synchronizing = 1, // Synchronizing in progress

(dbx) print __1cUSafepointSynchronizeR_waiting_to_block_
__1cUSafepointSynchronizeR_waiting_to_block_ = 1

(dbx) print __1cUSafepointSynchronizeU_safepointing_thread_
__1cUSafepointSynchronizeU_safepointing_thread_ = 1
042 _vm_thread = 1,

(dbx) print -fu __1cUSafepointSynchronizeP_last_safepoint_
__1cUSafepointSynchronizeP_last_safepoint_ = 3409970318

In the past, we wonder if a slow java.math.BigInteger pow can cause the JVM to be very slow coming to a safepoint. There was a bug 4646474 on slow algorithm of BigInteger.pow.