Synopsis : JVM sometimes would suddenly consume significant amount of memory

Description :

  Customer has recently experienced a severe Java memory problem on the Linux platform.
The customer has noticed that JVM sometimes would suddenly consume significant amount of memory.
When this happens, the operating system starts to get noticeably slower, and eventually starts to
terminate other low priority processes. Customer has experienced this issue several times.
They consider this a critical issue.

Application is a Java program that monitors network devices.
Customer noticed that sometimes when the program would suddenly consume a large amount of memory.

Normally, from 'top' output, the JVM consumes ~3G virtual mem, and ~500M res mem as below :


  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

27400 root 25 0 2969m 452m 38m S 0.0 45.2 5:03.36 java.sgm

 

However, when this problem occurs, the JVM memory utilization would suddenly increase significantly.
From 'top' command, customer observed the JVM is using 7.2G virtual memory and 3.5G res memory.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

13925 root 19 0 7206m 3.5g 5808 S 2 90.4 6:13.79 java.sgm

 
Furthermore, at the time when this problem occurred, the Java API "java.lang.Runtime.totalMemory()"
and "java.lang.Runtime.freeMemory()" reported used memory is less than 500M.

Have worked with the customer to take a core file snapshot of the JVM when this problem occurred.
Following steps were used for collecting core file :
   Customer keeps observing memory usage of the JVM. When they noticed a significant memory usage jump,
they will issue a command "kill -11 <pid>", where the PID is the Java process with the issue.
This creates a 'core' file and an 'hs_err' file when this issue occurred.


Following are some basic information about this issue:


* Operating System: Red Hat Enterprise Linux WS release 4 (linux-amd64)

   uname:Linux 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64



* Java version: 1.5.0_13-b05

 

* JVM Parameters:

./java -DPROC_NAME=sgmProcessManager -Xmx2048M -XX:MaxPermSize=512m -server -Djava.endorsed.dirs=/opt/CSCOsgm/server/lib/endorsed -Djboss.server.name=MWTM -Djboss.server.home.dir=/opt/CSCOsgm/server -Djboss.server.home.url=file:/opt/CSCOsgm/server -Djboss.server.config.dir=/opt/CSCOsgm/server/conf -Dprogram.name=sgmServer.sh -Djava.security.egd= -Djava.awt.headless=true -Djava.protocol.handler.pkgs=org.apache.naming.resources -Ddrools.compiler=JANINO -Djava.util.prefs.userRoot=/opt/CSCOsgm/prefs/ -cp /opt/CSCOsgm/properties:/opt/CSCOsgm/etc:/opt/CSCOsgm/server/lib/run.jar:/opt/CSCOsgm/images org.jboss.Main
 


Location of Core dumps :
=========================

 Path : /net/cores.central/cores/dir33/831645

1) core-cisco.zip ---->> Core File
2) hs_err_pid13925.log ---->> Error Logs


Need to troubleshoot this problem.