FULL PRODUCT VERSION :
$ java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode)

FULL OS VERSION :
$ uname -a
Linux 176282-app1 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
We have 2 Tomcat instances running on the same server. The only difference between the 2 instances is the GC algorithm.

The JVM running with the G1 GC seems to leak memory over time, from looking at what is reported by top.

$ ps aux --sort -vsz | head -3
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
tomcat 7634 208 49.0 3159628 1985012 ? Sl 10:44 1088:28 /opt/java/bin/java -server -Xms512m -Xmx1408m -XX:MaxPermSize=128m -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError -Djava.rmi.server.hostname={the.server.ip.address} -Dcom.sun.management.jmxremote.port={portB} -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/mobileiq/domains/ms2/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath :/opt/tomcat/bin/bootstrap.jar -Dcatalina.base=/var/mobileiq/domains/ms2/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/var/mobileiq/domains/ms2/tomcat/temp org.apache.catalina.startup.Bootstrap start
tomcat 19274 80.7 36.3 1845148 1469940 ? Sl Jul28 1571:09 /opt/java/bin/java -server -Xms512m -Xmx1408m -XX:MaxPermSize=128m -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError -Djava.rmi.server.hostname={the.server.ip.address} -Dcom.sun.management.jmxremote.port={portA} -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/mobileiq/domains/ms1/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath :/opt/tomcat/bin/bootstrap.jar -Dcatalina.base=/var/mobileiq/domains/ms1/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/var/mobileiq/domains/ms1/tomcat/temp org.apache.catalina.startup.Bootstrap start

Please note the VSZ, RSS and TIME values. Both JVM had been up a while, were thoroughly warmed up and had served similar volumes of traffic in tomcat.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run an application over time and observe the memory growth of the process.

EXPECTED VERSUS ACTUAL BEHAVIOR :
VSZ and RSS should not be so dramatically larger when using the G1 GC versus Parallel.
REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Don't use the G1 GC. There is no impact - we can use Parallel (which we've preferred since it is more stable) or CMS (which is a better fit and gives us better peformance, but we've seen a lot of core dumps in the where the active thread was GC, at least in Java 6 update 7).