-
Bug
-
Resolution: Incomplete
-
P3
-
None
-
8u162
-
x86_64
-
linux
ADDITIONAL SYSTEM INFORMATION :
linux/jdk1.8.0_311
A DESCRIPTION OF THE PROBLEM :
jvm suddely went down in production environment
REGRESSION : Last worked in version 8u311
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
suddenly jvm went down
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
jvm should not crash and work as expected
ACTUAL -
jvm went down
---------- BEGIN SOURCE ----------
hs_err_pid1921152.log
========================================
OS:
----------------------------------------
Version: Red Hat Enterprise Linux release 8.4 (Ootpa)
ARCH: X86_64
CPUs (cpu x cpu cores x hyperthreading): 8
Memory: 94391M
Memory Free: 25835M (27%)
Memory Available: 54458M (58%)
Swap: 2048M
Swap Free: 2048M (100%)
========================================
JVM:
----------------------------------------
Version: 1.8.0_162-b12
Vendor: ORACLE
Application: JBOSS_EAP7
JVM User: jboss
VM State: at safepoint (normal execution)
Crash Date: Fri Nov 26 07:32:17 2021
Run Time: 6d 7h 10m 52s
Garbage Collector(s): PARALLEL_SCAVENGE, PARALLEL_OLD
Heap Max: 6144M
Heap Allocation: 6130M (100% Heap Max)
Heap Used: 763M (12% Heap Allocation)
Compressed oops mode: UNKNOWN
Metaspace Max: 512M
Metaspace Allocation: 254M (50% Metaspace Max)
Metaspace Used: 230M (91% Metaspace Allocation)
Thread Stack Size: 1024K
Thread Stack Memory: 627M
Code Cache Max: 420M
JVM Memory Max: >7703M (8% Available Memory)
========================================
Threads:
----------------------------------------
Current thread: VMThread [stack: 0x000014909f1f3000,0x000014909f2f3000] [id=1921168]
VM operation: ParallelGCSystemGC, mode: safepoint, requested by thread 0x0000149034c5c800
# Java threads: 627
========================================
Error(s):
----------------------------------------
# SIGSEGV (0xb) at pc=0x00001490d19a80a6, pid=1921152, tid=0x000014909f2f2700
# V [libjvm.so+0x98c0a6] PSParallelCompact::IsAliveClosure::do_object_b(oopDesc*)+0x36
========================================
Stack:
----------------------------------------
Stack: [0x000014909f1f3000,0x000014909f2f3000], sp=0x000014909f2f11f0, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x98c0a6] PSParallelCompact::IsAliveClosure::do_object_b(oopDesc*)+0x36
V [libjvm.so+0x8c8f6c] MethodData::clean_extra_data(BoolObjectClosure*)+0x1dc
V [libjvm.so+0x648cf6] InstanceKlass::clean_weak_instanceklass_links(BoolObjectClosure*)+0x56
V [libjvm.so+0x7d0d9e] Klass::clean_weak_klass_links(BoolObjectClosure*, bool)+0x18e
V [libjvm.so+0x98dea1] PSParallelCompact::marking_phase(ParCompactionManager*, bool, ParallelOldTracer*)+0x4e1
V [libjvm.so+0x99303e] PSParallelCompact::invoke_no_policy(bool)+0x40e
V [libjvm.so+0x9938b3] PSParallelCompact::invoke(bool)+0x63
V [libjvm.so+0xad16e8] VM_ParallelGCSystemGC::doit()+0xa8
...
========================================
ANALYSIS:
----------------------------------------
error
----------------------------------------
*JVM crash inside libjvm.so. Reference: https://access.redhat.com/solutions/442903.
*The crash occurred dereferencing a null pointer (si_addr: 0x0000000000000000).
*Duplicate jvm options: -XX:MetaspaceSize=96m -XX:MetaspaceSize=256m.
*Duplicate jvm options: -XX:MaxMetaspaceSize=256m -XX:MaxMetaspaceSize=512m.
----------------------------------------
warn
----------------------------------------
*MaxMetaspaceSize is less than CompressedClassSpaceSize. MaxMetaspaceSize includes CompressedClassSpaceSize, so MaxMetaspaceSize should be larger than CompressedClassSpaceSize. If MaxMetaspaceSize is set smaller than CompressedClassSpaceSize, the JVM auto adjusts CompressedClassSpaceSize as follows: CompressedClassSpaceSize = MaxMetaspaceSize - (2 * InitialBootClassLoaderMetaspaceSize).
*Number of GC log files is defined (-XX:NumberOfGCLogFiles), yet GC log file rotation is disabled. Either remove -XX:NumberOfGCLogFiles or enable GC log file rotation with -XX:+UseGCLogFileRotation.
*The gc log file has a static name and will be overwritten on JVM startup. Enable log file rotation and/or include process id or datestamp in the file name (e.g. -Xloggc:gc_%p_%t.log).
*GC log file size (-XX:GCLogFileSize=N[K|M|G]) is small (< 5M). Data needed for troubleshooting could be lost due to excessive log rotation. Consider increasing the size.
*The gc log file has a static name and will be overwritten on JVM startup. Enable log file rotation and/or include process id or datestamp in the file name (e.g. -Xloggc:gc_%p_%t.log).
----------------------------------------
info
----------------------------------------
*JDK does not appear to be a Red Hat build.
*Signal number SIGSEGV: Segmentation fault. Accessing valid memory in an invalid way.
*Signal code SI_KERNEL: Sent by the kernel.
*Initial and/or max metaspace size is being set. This is generally not recommended. Reference: https://access.redhat.com/solutions/1489263.
*Consider adding -XX:+HeapDumpOnOutOfMemoryError, a standard recommended option to generate a heap dump when the first thread throws OutOfMemoryError. It does not impact performance (until the heap is actually written out) and generally should always be used, as it provides critical information in case of a memory error.
*Metaspace includes class metadata plus compressed class space.
*Instrumentation is being used: -javaagent:/opt/ca_introscope/rh_jboss/wily/Agent.jar.
*GC log file rotation is disabled (-XX:-UseGCLogFileRotation). Consider enabling rotation to protect disk space.
========================================
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
restart the jvm
FREQUENCY : often
linux/jdk1.8.0_311
A DESCRIPTION OF THE PROBLEM :
jvm suddely went down in production environment
REGRESSION : Last worked in version 8u311
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
suddenly jvm went down
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
jvm should not crash and work as expected
ACTUAL -
jvm went down
---------- BEGIN SOURCE ----------
hs_err_pid1921152.log
========================================
OS:
----------------------------------------
Version: Red Hat Enterprise Linux release 8.4 (Ootpa)
ARCH: X86_64
CPUs (cpu x cpu cores x hyperthreading): 8
Memory: 94391M
Memory Free: 25835M (27%)
Memory Available: 54458M (58%)
Swap: 2048M
Swap Free: 2048M (100%)
========================================
JVM:
----------------------------------------
Version: 1.8.0_162-b12
Vendor: ORACLE
Application: JBOSS_EAP7
JVM User: jboss
VM State: at safepoint (normal execution)
Crash Date: Fri Nov 26 07:32:17 2021
Run Time: 6d 7h 10m 52s
Garbage Collector(s): PARALLEL_SCAVENGE, PARALLEL_OLD
Heap Max: 6144M
Heap Allocation: 6130M (100% Heap Max)
Heap Used: 763M (12% Heap Allocation)
Compressed oops mode: UNKNOWN
Metaspace Max: 512M
Metaspace Allocation: 254M (50% Metaspace Max)
Metaspace Used: 230M (91% Metaspace Allocation)
Thread Stack Size: 1024K
Thread Stack Memory: 627M
Code Cache Max: 420M
JVM Memory Max: >7703M (8% Available Memory)
========================================
Threads:
----------------------------------------
Current thread: VMThread [stack: 0x000014909f1f3000,0x000014909f2f3000] [id=1921168]
VM operation: ParallelGCSystemGC, mode: safepoint, requested by thread 0x0000149034c5c800
# Java threads: 627
========================================
Error(s):
----------------------------------------
# SIGSEGV (0xb) at pc=0x00001490d19a80a6, pid=1921152, tid=0x000014909f2f2700
# V [libjvm.so+0x98c0a6] PSParallelCompact::IsAliveClosure::do_object_b(oopDesc*)+0x36
========================================
Stack:
----------------------------------------
Stack: [0x000014909f1f3000,0x000014909f2f3000], sp=0x000014909f2f11f0, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x98c0a6] PSParallelCompact::IsAliveClosure::do_object_b(oopDesc*)+0x36
V [libjvm.so+0x8c8f6c] MethodData::clean_extra_data(BoolObjectClosure*)+0x1dc
V [libjvm.so+0x648cf6] InstanceKlass::clean_weak_instanceklass_links(BoolObjectClosure*)+0x56
V [libjvm.so+0x7d0d9e] Klass::clean_weak_klass_links(BoolObjectClosure*, bool)+0x18e
V [libjvm.so+0x98dea1] PSParallelCompact::marking_phase(ParCompactionManager*, bool, ParallelOldTracer*)+0x4e1
V [libjvm.so+0x99303e] PSParallelCompact::invoke_no_policy(bool)+0x40e
V [libjvm.so+0x9938b3] PSParallelCompact::invoke(bool)+0x63
V [libjvm.so+0xad16e8] VM_ParallelGCSystemGC::doit()+0xa8
...
========================================
ANALYSIS:
----------------------------------------
error
----------------------------------------
*JVM crash inside libjvm.so. Reference: https://access.redhat.com/solutions/442903.
*The crash occurred dereferencing a null pointer (si_addr: 0x0000000000000000).
*Duplicate jvm options: -XX:MetaspaceSize=96m -XX:MetaspaceSize=256m.
*Duplicate jvm options: -XX:MaxMetaspaceSize=256m -XX:MaxMetaspaceSize=512m.
----------------------------------------
warn
----------------------------------------
*MaxMetaspaceSize is less than CompressedClassSpaceSize. MaxMetaspaceSize includes CompressedClassSpaceSize, so MaxMetaspaceSize should be larger than CompressedClassSpaceSize. If MaxMetaspaceSize is set smaller than CompressedClassSpaceSize, the JVM auto adjusts CompressedClassSpaceSize as follows: CompressedClassSpaceSize = MaxMetaspaceSize - (2 * InitialBootClassLoaderMetaspaceSize).
*Number of GC log files is defined (-XX:NumberOfGCLogFiles), yet GC log file rotation is disabled. Either remove -XX:NumberOfGCLogFiles or enable GC log file rotation with -XX:+UseGCLogFileRotation.
*The gc log file has a static name and will be overwritten on JVM startup. Enable log file rotation and/or include process id or datestamp in the file name (e.g. -Xloggc:gc_%p_%t.log).
*GC log file size (-XX:GCLogFileSize=N[K|M|G]) is small (< 5M). Data needed for troubleshooting could be lost due to excessive log rotation. Consider increasing the size.
*The gc log file has a static name and will be overwritten on JVM startup. Enable log file rotation and/or include process id or datestamp in the file name (e.g. -Xloggc:gc_%p_%t.log).
----------------------------------------
info
----------------------------------------
*JDK does not appear to be a Red Hat build.
*Signal number SIGSEGV: Segmentation fault. Accessing valid memory in an invalid way.
*Signal code SI_KERNEL: Sent by the kernel.
*Initial and/or max metaspace size is being set. This is generally not recommended. Reference: https://access.redhat.com/solutions/1489263.
*Consider adding -XX:+HeapDumpOnOutOfMemoryError, a standard recommended option to generate a heap dump when the first thread throws OutOfMemoryError. It does not impact performance (until the heap is actually written out) and generally should always be used, as it provides critical information in case of a memory error.
*Metaspace includes class metadata plus compressed class space.
*Instrumentation is being used: -javaagent:/opt/ca_introscope/rh_jboss/wily/Agent.jar.
*GC log file rotation is disabled (-XX:-UseGCLogFileRotation). Consider enabling rotation to protect disk space.
========================================
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
restart the jvm
FREQUENCY : often