Kitchensink was considered hanged, but in fact it was too long GC. Heap was 16Gb, and pause was more than 30 minutes, after time limit application was killed.

run time:16hours

RULE bigapps/Kitchensink/stability Unknown Server is hang: no server activity noticed.

ILW=M(can't say if it is hang or just very long pause) L(rest of batch jobs are running ok for 26+hours for the moment ) H(no workaround), => priority=P3


pstack shows 1 active GC tread
----------------- lwp# 7 / thread# 7 --------------------
 ffff80ff3f82b5ef __1cOFreeRegionListLadd_ordered6Mp0_v_ () + 7f
 ffff80ff3f678ea6 __1cQG1ParNoteEndTaskEwork6MI_v_ () + 196
 ffff80ff40025844 __1cKGangWorkerEloop6M_v_ () + a4
 ffff80ff3fd5f5c0 java_start () + 230
 ffff80ffbcc41545 _thrp_setup () + a5
 ffff80ffbcc417f0 _lwp_start ()

almost all other lwps are running lwp_cond_wait

gc log ends up with unfinished "[GC cleanup":

60093.184: #226056: [GC pause (G1 Evacuation Pause) (young) (initial-mark), 0.0504610 secs]
   [Parallel Time: 40.3 ms, GC Workers: 6]
      [GC Worker Start (ms): Min: 60093184.7, Avg: 60093184.8, Max: 60093184.8, Diff: 0.0]
      [Ext Root Scanning (ms): Min: 1.0, Avg: 1.2, Max: 1.2, Diff: 0.2, Sum: 7.0]
      [Update RS (ms): Min: 9.7, Avg: 10.1, Max: 10.7, Diff: 0.9, Sum: 60.3]
         [Processed Buffers: Min: 8, Avg: 10.2, Max: 12, Diff: 4, Sum: 61]
      [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.4, Diff: 0.3, Sum: 1.4]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.3]
      [Object Copy (ms): Min: 28.0, Avg: 28.5, Max: 28.7, Diff: 0.6, Sum: 170.8]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4]
      [GC Worker Total (ms): Min: 40.0, Avg: 40.1, Max: 40.1, Diff: 0.1, Sum: 240.4]
      [GC Worker End (ms): Min: 60093224.8, Avg: 60093224.8, Max: 60093224.9, Diff: 0.0]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.8 ms]
   [Other: 9.2 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 6.8 ms]
      [Ref Enq: 0.3 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.4 ms]
   [Eden: 3500.0M(3500.0M)->0.0B(3500.0M) Survivors: 108.0M->96.0M Heap: 7307.2M(8292.0M)->3819.0M(8292.0M)]
 [Times: user=0.25 sys=0.00 real=0.05 secs]
60093.235: #226057: [GC concurrent-root-region-scan-start]
60093.265: #226057: [GC concurrent-root-region-scan-end, 0.0303055 secs]
60093.265: #226057: [GC concurrent-mark-start]
60095.477: #226057: [GC concurrent-mark-end, 2.2117385 secs]
60095.479: #226057: [GC remark 60095.479: #226057: [Finalize Marking, 0.0004639 secs] 60095.480: #226057: [GC ref-proc, 1.3226148 secs] 60096.803: #226057: [Unloading, 0.0389114 secs] 60096.841: #226057: [GC aggregate-data, 0.0042423 secs], 1.3672145 secs]
 [Times: user=1.54 sys=0.00 real=1.37 secs]
60096.847: #226057: [GC cleanup