FULL PRODUCT VERSION :
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)


FULL OS VERSION :
3.2.45-0.6.wd.865.49.315.metal1.x86_64 #1 SMP Fri Apr 1 10:12:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
We have found that the 'C1 CompilerThread' named child processes of our JVM process begin using lots of CPU cycles when the CodeCache memory area becomes full.

Here's the output from `top` and then the associated output from jstack. These combined show that it is the 'C1 CompilerThreads' that are spinning on the CPU cores.

24739 nobody 20 0 52.3g 35g 35m S 24.0 60.5 8:50.97 java
 24740 nobody 20 0 52.3g 35g 35m S 20.3 60.5 8:55.73 java
 24741 nobody 20 0 52.3g 35g 35m S 20.3 60.5 8:47.82 java
 24738 nobody 20 0 52.3g 35g 35m S 18.4 60.5 8:52.77 java
 24742 nobody 20 0 52.3g 35g 35m S 18.4 60.5 8:59.38 java
 26397 nobody 20 0 52.3g 35g 35m R 14.7 60.5 8:41.18 java
 26510 nobody 20 0 52.3g 35g 35m S 14.7 60.5 8:52.96 java
 26504 nobody 20 0 52.3g 35g 35m S 12.9 60.5 9:06.56 java
...
...
...
"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f9130107800 nid=0x60a6 waiting on condition [0x0000000000000000]
"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f9130105800 nid=0x60a5 waiting on condition [0x0000000000000000]
"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f9130103800 nid=0x60a4 waiting on condition [0x0000000000000000]
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f9130101800 nid=0x60a3 waiting on condition [0x0000000000000000]
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f91300ff800 nid=0x60a2 waiting on condition [0x0000000000000000]
"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f91300fd000 nid=0x60a1 runnable [0x0000000000000000]
"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f91300fb000 nid=0x60a0 waiting on condition [0x0000000000000000]
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f91300f8800 nid=0x609f waiting on condition [0x0000000000000000]
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f91300f6800 nid=0x609e waiting on condition [0x0000000000000000]
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f91300f4000 nid=0x609d waiting on condition [0x0000000000000000]
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f91300ea000 nid=0x609c waiting on condition [0x0000000000000000]
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f91300e8000 nid=0x609b waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f91300e5800 nid=0x609a runnable [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f91300e3800 nid=0x6099 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f91300e0800 nid=0x6098 waiting on condition [0x0000000000000000]


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: No

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Our production server, a Tomcat application, falls into the high CPU usage mode after several days of continuous execution. The problem occurs after JVM CodeCache memory usage reaches the JVM's default-configured CodeCache memory size. This problem was re-occurring very consistently on our Prod fleet.

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Our workaround was to disabled 'Tiered Compilation' via the -XX:-TieredCompilation JVM option. Doing this greatly reduced our CodeCache memory usage and thereby avoided us reaching capacity and triggering this problem. Turning off Tiered Compilation did come at a performance cost unfortunately.