One of Intentias customers(Flextronics) reported that their system worked very slow and that their technicians (approx 1000) couldn't report into the system. It seemed like their requests somehow qued up.

Later, when people started to logoff of the system, the queued jobs was executed. They belived it was some kind of resource problem and suspected filehandles, file descriptors and to many open files.

Their solution was to increase the parameter RLIM_FD_MAX from 1024 to 32000.

For a couple of days this seemed to solve the performance problems, but then they got this JVM coredumps instead. The cust believes the coredumps are related to the RLIM_FD_MAX tuning and the increased throughput to the JVM.

The JVM 1.4.1_05 crash output looked like this:

# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002E6 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Client VM (1.4.1_05-b01 mixed mode)
#
# An error report file has been saved as hs_err_pid29362.log.
# Please refer to the file for further information.

Location of files: /net/sponge.sweden/export/calls/37003272

Same problem at 2 sites Flextronics(Sweden) and Gautier(France)

Servers @ Flextronics
---------------------
knaux41.kna.flextronics.com Production server no 1
knaux42.kna.flextronics.com Production server no 2
knaux45.kna.flextronics.com Explorer outputs from Prodservers systemcontrollers
knaux49.kna.flextronics.com Test
knaux50.kna.flextronics.com Test

Servers @ Gautier
-----------------
SERV-MOVEX Production server

Servers @ Intentia
------------------
soldis1 Intentias test server (works ok)

FILES
=====

Info
----
info/flextronics_system.txt Flextronics knaux41 system info
info/gautier_system.txt Gautier SERV-MOVEX system info
info/intentia_test_system.txt Intentias MOVEX test system info.

Explorers
---------
explorers/Intentia-Flex-SunWExplo-out.tar.gz All explorers
explorers/explorer.830f68b1.knaux41-2003.11.26.11.56
explorers/explorer.830743e4.knaux42-2003.11.23.21.00
explorers/explorer.83194f27.knaux49-2003.11.16.20.00
explorers/explorer.831953b5.knaux50-2003.11.16.21.00
explorers/explorer.83278183.knaux45-2003.11.16.19.00
explorers/explorer.8308db6a.SERV-MOVEX-2003.11.25.15.14
explorers/explorer.831263a2.soldis1-2003.09.29.23.54

_d.txt files are results from suncheckup.

...and more compressed explorers in directories for each server.

Logs
----
logs/Flextronics-appl-log-031117 Application log Flextronics from 17 Nov
logs/Flextronics-messages-log-031117 Messagesfile Flextronics from 17 Nov
logs/Flextronics-oracle-alert-logs-031117 Oracle alert log Flextronics from 17 Nov
logs/Gautier-appl-log-031117 Application log. Gautier from 17 Nov
logs/Gautier-messages-log-031117 Messagesfile Gautier from 17 Nov

Cores
-----
cores/Intentia-Flextronics.core Core Flextronics case
cores/Intentia-Gautier.core Core Gautier case
cores/core_Mar_040310_0159_MVXEDU02.gz Latest Flextronics core from JVM 1.4.2

Dbx
---
dbx_Intentia-Flextronics_core.txt Dbx output from Flextronics core
dbx_Intentia-Gautier_core.txt Dbx output from Gautier core

Jre
---
Some different downloaded java versions

Bin
---
T3 extractor sent to customer

Debugjava
---------
debugjava/libjvm.so.Z Debug JVM from Andy's contact at engineering.
debugjava/README A debug java install "HowTo"