FULL PRODUCT VERSION :
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
FULL OS VERSION :
CentOS Linux release 7.1.1503 (Core)
Linux 75-25-26-yf-core.jpool.sinaimg.cn 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
A DESCRIPTION OF THE PROBLEM :
When -XX:-CMSClassUnloadingEnabled flag is set, the objects in StringTable that created by intern() method of String class can not be cleaned up by CMS collector, but can be cleaned by Full GC.
THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes
THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Set jvm options to:
-Xms256m
-Xmx256m
-Xmn128m
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSParallelInitialMarkEnabled
-XX:+CMSParallelRemarkEnabled
-XX:-CMSClassUnloadingEnabled
2. Create a large amount of strings by intern() method and store a certain number of them
3. Use jstat -gcutil <pid> to watch the gc information
EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected:
When the usage of old generation reaches 60%, CMS collector can clean up the interned strings that won't be used.
Actual:
The interned strings can not be cleaned up, until a Full GC is triggered when the old generation is almost full.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
The output of jstat -gcutil:
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 16.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 22.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 28.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 34.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 40.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 46.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 52.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 58.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 64.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 70.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 76.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 82.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 88.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 92.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 98.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 100.00 3.96 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 7.92 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 13.86 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 19.80 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 25.74 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 29.70 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 35.64 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 41.58 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 45.54 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 51.47 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 57.41 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 61.37 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 67.31 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 73.25 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 79.19 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 85.13 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 91.07 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 97.01 20.54 66.77 72.01 1 0.183 0 0.000 0.183
100.00 0.00 1.99 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 7.95 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 13.91 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 19.87 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 23.84 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 29.80 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 35.76 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 41.73 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 45.70 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 51.66 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 55.63 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 61.59 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 67.55 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 73.52 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 77.49 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 83.45 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 89.41 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 95.37 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 99.35 51.05 66.77 72.01 2 0.268 0 0.000 0.268
0.00 100.00 3.98 80.30 66.77 72.01 3 0.364 1 0.033 0.398
0.00 100.00 9.96 80.30 66.77 72.01 3 0.364 1 0.033 0.398
0.00 100.00 15.93 79.97 66.77 72.01 3 0.364 2 0.069 0.433 <= CMS should work but only cleaned up few of them
0.00 100.00 21.91 79.97 66.77 72.01 3 0.364 2 0.069 0.433
0.00 100.00 27.89 79.97 66.77 72.01 3 0.364 4 0.147 0.511
0.00 100.00 33.86 79.97 66.77 72.01 3 0.364 4 0.147 0.511
0.00 100.00 39.84 79.97 66.77 72.01 3 0.364 6 0.228 0.592
0.00 100.00 43.82 79.97 66.77 72.01 3 0.364 6 0.228 0.592
0.00 100.00 49.80 79.97 66.77 72.01 3 0.364 6 0.228 0.592
0.00 100.00 55.77 79.97 66.77 72.01 3 0.364 8 0.327 0.691
0.00 100.00 59.75 79.97 66.77 72.01 3 0.364 8 0.327 0.691
0.00 100.00 65.73 79.97 66.77 72.01 3 0.364 10 0.415 0.779
0.00 100.00 71.70 79.97 66.77 72.01 3 0.364 10 0.415 0.779
0.00 100.00 75.69 79.97 66.77 72.01 3 0.364 12 0.506 0.871
0.00 100.00 79.67 79.97 66.77 72.01 3 0.364 12 0.506 0.871
0.00 100.00 85.64 79.97 66.77 72.01 3 0.364 14 0.606 0.970
0.00 100.00 89.63 79.97 66.77 72.01 3 0.364 14 0.606 0.970
0.00 100.00 93.61 79.97 66.77 72.01 3 0.364 14 0.606 0.970
0.00 100.00 99.58 79.95 66.77 72.01 3 0.364 16 0.719 1.084
0.00 0.00 1.98 0.70 66.77 72.01 4 0.364 17 1.074 1.439 <= Full GC cleaned up all
0.00 0.00 7.92 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 13.86 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 19.80 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 25.75 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 31.69 0.70 66.77 72.01 4 0.364 17 1.074 1.439
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
public class InternString extends LinkedHashMap<String, String> {
private static final int MAX_SIZE = 512;
private InternString() {
super(MAX_SIZE, 0.8f, true);
}
private String intern(String input) {
String result = get(input);
if (result == null) {
result = input.intern();
put(result, result);
}
return result;
}
@Override
protected boolean removeEldestEntry(Map.Entry<String, String> eldest) {
return size() > MAX_SIZE;
}
private static String genRandomNumberString(int length) {
if (length <= 0) {
return "";
} else {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < length; ++i) {
sb.append(ThreadLocalRandom.current().nextInt(10));
}
return sb.toString();
}
}
public static void main(String[] args) throws Exception {
InternString instance = new InternString();
for (int i = 0; i < 1000000000; i++) {
if (i % 100 == 0) {
// slow down the growth of the usage of jvm heap
Thread.sleep(1);
}
instance.intern(genRandomNumberString(10));
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
set -XX:+CMSClassUnloadingEnabled flag to jvm
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
FULL OS VERSION :
CentOS Linux release 7.1.1503 (Core)
Linux 75-25-26-yf-core.jpool.sinaimg.cn 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
A DESCRIPTION OF THE PROBLEM :
When -XX:-CMSClassUnloadingEnabled flag is set, the objects in StringTable that created by intern() method of String class can not be cleaned up by CMS collector, but can be cleaned by Full GC.
THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes
THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Set jvm options to:
-Xms256m
-Xmx256m
-Xmn128m
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSParallelInitialMarkEnabled
-XX:+CMSParallelRemarkEnabled
-XX:-CMSClassUnloadingEnabled
2. Create a large amount of strings by intern() method and store a certain number of them
3. Use jstat -gcutil <pid> to watch the gc information
EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected:
When the usage of old generation reaches 60%, CMS collector can clean up the interned strings that won't be used.
Actual:
The interned strings can not be cleaned up, until a Full GC is triggered when the old generation is almost full.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
The output of jstat -gcutil:
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 16.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 22.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 28.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 34.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 40.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 46.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 52.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 58.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 64.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 70.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 76.06 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 82.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 88.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 92.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 0.00 98.07 0.00 17.29 19.76 0 0.000 0 0.000 0.000
0.00 100.00 3.96 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 7.92 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 13.86 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 19.80 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 25.74 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 29.70 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 35.64 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 41.58 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 45.54 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 51.47 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 57.41 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 61.37 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 67.31 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 73.25 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 79.19 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 85.13 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 91.07 20.54 66.77 72.01 1 0.183 0 0.000 0.183
0.00 100.00 97.01 20.54 66.77 72.01 1 0.183 0 0.000 0.183
100.00 0.00 1.99 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 7.95 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 13.91 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 19.87 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 23.84 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 29.80 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 35.76 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 41.73 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 45.70 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 51.66 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 55.63 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 61.59 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 67.55 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 73.52 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 77.49 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 83.45 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 89.41 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 95.37 51.05 66.77 72.01 2 0.268 0 0.000 0.268
100.00 0.00 99.35 51.05 66.77 72.01 2 0.268 0 0.000 0.268
0.00 100.00 3.98 80.30 66.77 72.01 3 0.364 1 0.033 0.398
0.00 100.00 9.96 80.30 66.77 72.01 3 0.364 1 0.033 0.398
0.00 100.00 15.93 79.97 66.77 72.01 3 0.364 2 0.069 0.433 <= CMS should work but only cleaned up few of them
0.00 100.00 21.91 79.97 66.77 72.01 3 0.364 2 0.069 0.433
0.00 100.00 27.89 79.97 66.77 72.01 3 0.364 4 0.147 0.511
0.00 100.00 33.86 79.97 66.77 72.01 3 0.364 4 0.147 0.511
0.00 100.00 39.84 79.97 66.77 72.01 3 0.364 6 0.228 0.592
0.00 100.00 43.82 79.97 66.77 72.01 3 0.364 6 0.228 0.592
0.00 100.00 49.80 79.97 66.77 72.01 3 0.364 6 0.228 0.592
0.00 100.00 55.77 79.97 66.77 72.01 3 0.364 8 0.327 0.691
0.00 100.00 59.75 79.97 66.77 72.01 3 0.364 8 0.327 0.691
0.00 100.00 65.73 79.97 66.77 72.01 3 0.364 10 0.415 0.779
0.00 100.00 71.70 79.97 66.77 72.01 3 0.364 10 0.415 0.779
0.00 100.00 75.69 79.97 66.77 72.01 3 0.364 12 0.506 0.871
0.00 100.00 79.67 79.97 66.77 72.01 3 0.364 12 0.506 0.871
0.00 100.00 85.64 79.97 66.77 72.01 3 0.364 14 0.606 0.970
0.00 100.00 89.63 79.97 66.77 72.01 3 0.364 14 0.606 0.970
0.00 100.00 93.61 79.97 66.77 72.01 3 0.364 14 0.606 0.970
0.00 100.00 99.58 79.95 66.77 72.01 3 0.364 16 0.719 1.084
0.00 0.00 1.98 0.70 66.77 72.01 4 0.364 17 1.074 1.439 <= Full GC cleaned up all
0.00 0.00 7.92 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 13.86 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 19.80 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 25.75 0.70 66.77 72.01 4 0.364 17 1.074 1.439
0.00 0.00 31.69 0.70 66.77 72.01 4 0.364 17 1.074 1.439
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
public class InternString extends LinkedHashMap<String, String> {
private static final int MAX_SIZE = 512;
private InternString() {
super(MAX_SIZE, 0.8f, true);
}
private String intern(String input) {
String result = get(input);
if (result == null) {
result = input.intern();
put(result, result);
}
return result;
}
@Override
protected boolean removeEldestEntry(Map.Entry<String, String> eldest) {
return size() > MAX_SIZE;
}
private static String genRandomNumberString(int length) {
if (length <= 0) {
return "";
} else {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < length; ++i) {
sb.append(ThreadLocalRandom.current().nextInt(10));
}
return sb.toString();
}
}
public static void main(String[] args) throws Exception {
InternString instance = new InternString();
for (int i = 0; i < 1000000000; i++) {
if (i % 100 == 0) {
// slow down the growth of the usage of jvm heap
Thread.sleep(1);
}
instance.intern(genRandomNumberString(10));
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
set -XX:+CMSClassUnloadingEnabled flag to jvm