Aurora central feeder started to fail once a week. It's a simple jetty based http process
which transfers files from sca to spb and back.
It is run on 7u17/b02 on Solaris 11
with -XX:+HeapDumpOnOutOfMemoryError -Xmx2g -Xloggc:logs/gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
See attached hs errors.
Somehow there's no core file, even though
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:222), pid=18173, tid=11
#
# JRE version: 7.0_17-b02
# Java VM: Java HotSpot(TM) Server VM (23.7-b01 mixed mode solaris-x86 )
# Core dump written. Default location: /export/home/auroradmin/feeder-central/core or core.18173
#
--------------- T H R E A D ---------------
Current thread (0x0832b800): VMThread [stack: 0xf97b9000,0xf9839000] [id=11]
Stack: [0xf97b9000,0xf9839000], sp=0xf9837b80, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xc9a6da] void VMError::report(outputStream*)+0x8f2
V [libjvm.so+0xc9b8bf] void VMError::report_and_die()+0x557
V [libjvm.so+0x6a9e9c] void report_vm_out_of_memory(const char*,int,unsigned,const char*)+0x70
V [libjvm.so+0x13bac5] void*Arena::grow(unsigned)+0x1d9
V [libjvm.so+0xfe2ad] char*resource_allocate_bytes(unsigned)+0x89
V [libjvm.so+0xafa69b] void OopMapSet::update_register_map(const frame*,RegisterMap*)+0xb3
V [libjvm.so+0x7265b4] frame frame::sender(RegisterMap*)const+0x1ec
V [libjvm.so+0x174598] frame frame::real_sender(RegisterMap*)const+0x28
V [libjvm.so+0xc9035a] vframe*vframe::sender()const+0x23e
V [libjvm.so+0x20fe88] vframe*compiledVFrame::sender()const+0xcc
V [libjvm.so+0xc907c6] javaVFrame*vframe::java_sender()const+0x22
V [libjvm.so+0x48d873] GrowableArray<MonitorInfo*>*get_or_compute_monitor_info(JavaThread*)+0x20b
V [libjvm.so+0x492097] void VM_BulkRevokeBias::doit()+0x837
V [libjvm.so+0x1f2a11] void VM_Operation::evaluate()+0x71
V [libjvm.so+0x2dab78] void VMThread::run()+0x52c
V [libjvm.so+0xb0354d] java_start+0xaf5
C [libc.so.1+0xc443c] _thrp_setup+0x9d
C [libc.so.1+0xc46e0] _lwp_start+0x0
which transfers files from sca to spb and back.
It is run on 7u17/b02 on Solaris 11
with -XX:+HeapDumpOnOutOfMemoryError -Xmx2g -Xloggc:logs/gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
See attached hs errors.
Somehow there's no core file, even though
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:222), pid=18173, tid=11
#
# JRE version: 7.0_17-b02
# Java VM: Java HotSpot(TM) Server VM (23.7-b01 mixed mode solaris-x86 )
# Core dump written. Default location: /export/home/auroradmin/feeder-central/core or core.18173
#
--------------- T H R E A D ---------------
Current thread (0x0832b800): VMThread [stack: 0xf97b9000,0xf9839000] [id=11]
Stack: [0xf97b9000,0xf9839000], sp=0xf9837b80, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xc9a6da] void VMError::report(outputStream*)+0x8f2
V [libjvm.so+0xc9b8bf] void VMError::report_and_die()+0x557
V [libjvm.so+0x6a9e9c] void report_vm_out_of_memory(const char*,int,unsigned,const char*)+0x70
V [libjvm.so+0x13bac5] void*Arena::grow(unsigned)+0x1d9
V [libjvm.so+0xfe2ad] char*resource_allocate_bytes(unsigned)+0x89
V [libjvm.so+0xafa69b] void OopMapSet::update_register_map(const frame*,RegisterMap*)+0xb3
V [libjvm.so+0x7265b4] frame frame::sender(RegisterMap*)const+0x1ec
V [libjvm.so+0x174598] frame frame::real_sender(RegisterMap*)const+0x28
V [libjvm.so+0xc9035a] vframe*vframe::sender()const+0x23e
V [libjvm.so+0x20fe88] vframe*compiledVFrame::sender()const+0xcc
V [libjvm.so+0xc907c6] javaVFrame*vframe::java_sender()const+0x22
V [libjvm.so+0x48d873] GrowableArray<MonitorInfo*>*get_or_compute_monitor_info(JavaThread*)+0x20b
V [libjvm.so+0x492097] void VM_BulkRevokeBias::doit()+0x837
V [libjvm.so+0x1f2a11] void VM_Operation::evaluate()+0x71
V [libjvm.so+0x2dab78] void VMThread::run()+0x52c
V [libjvm.so+0xb0354d] java_start+0xaf5
C [libc.so.1+0xc443c] _thrp_setup+0x9d
C [libc.so.1+0xc46e0] _lwp_start+0x0