In Mustang build 103, while I was running VolanoTest with 64 bit JVM in Solaris AMD, there is a OutOfMemoryError with ParallelGC after 30 minutes or so.

I have been using 256M max heap size for the app, but pmap shows more than 1G for the process.

So far the problem is happening to Solaris AMD 64 bit VM with newer version of Solaris 10 update, such as s10x_u1wos_19a and s10x_u2wos_05 on sunfire x4100/x4200 machines. But the problem didn't happen to v20z machines running earlier version s10x_u1wos_06

Note that in those x4100/x4200 machines, I have previously found sunlibc bug- 6391358.
I applied the patches that fix libC leaks and maybe there reveals different issue after the fix.

I attached the pmap file and the findleaks report generated by libumem.so.
I found that libjvm.so looks very suspicious from the findleaks report.

Here is the test information:
-----------------------------
hostname: jtg-x4100-01

cat /etc/*release*
                        Solaris 10 6/06 s10x_u2wos_05 X86
           Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 14 March 2006

/usr/j2se/bin/java -d64 -version
java version "1.6.0-rc"
Java(TM) SE Runtime Environment (build 1.6.0-rc-b103)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b103, mixed mode)

vm flags: "-d64 -XX:+UseParallelGC -XX:+PrintHeapAtGC -Xmx256m"

From the /bt/VolanoTestrun.7659.-d64/vtestserver.out:
-----------------------------------------------------
{Heap before GC invocations=2444 (full 964):
 PSYoungGen total 58752K, used 29440K [0xfffffd7ff5800000, 0xfffffd7ffae00000, 0xfffffd7ffae00000)
  eden space 29440K, 100% used [0xfffffd7ff5800000,0xfffffd7ff74c0000,0xfffffd7ff74c0000)
  from space 29312K, 0% used [0xfffffd7ff74c0000,0xfffffd7ff74c0000,0xfffffd7ff9160000)
  to space 29312K, 0% used [0xfffffd7ff9160000,0xfffffd7ff9160000,0xfffffd7ffae00000)
 PSOldGen total 176128K, used 176126K [0xfffffd7feac00000, 0xfffffd7ff5800000, 0xfffffd7ff5800000)
  object space 176128K, 99% used [0xfffffd7feac00000,0xfffffd7ff57ffaf8,0xfffffd7ff5800000)
 PSPermGen total 22528K, used 8205K [0xfffffd7fe5800000, 0xfffffd7fe6e00000, 0xfffffd7feac00000)
  object space 22528K, 36% used [0xfffffd7fe5800000,0xfffffd7fe6003558,0xfffffd7fe6e00000)
4751.506: [Full GC [PSYoungGen: 29440K->27774K(58752K)] [PSOldGen: 176126K->176125K(176128K)] 205566K->203899K(234880K) [PSPermGen: 8205K->8205K(22528K)] GC time would exceed GCTimeLimit of 98%
, 0.5470262 secs]
Heap after GC invocations=2444 (full 964):
 PSYoungGen total 58752K, used 27774K [0xfffffd7ff5800000, 0xfffffd7ffae00000, 0xfffffd7ffae00000)
  eden space 29440K, 94% used [0xfffffd7ff5800000,0xfffffd7ff731f818,0xfffffd7ff74c0000)
  from space 29312K, 0% used [0xfffffd7ff74c0000,0xfffffd7ff74c0000,0xfffffd7ff9160000)
  to space 29312K, 0% used [0xfffffd7ff9160000,0xfffffd7ff9160000,0xfffffd7ffae00000)
 PSOldGen total 176128K, used 176125K [0xfffffd7feac00000, 0xfffffd7ff5800000, 0xfffffd7ff5800000)
  object space 176128K, 99% used [0xfffffd7feac00000,0xfffffd7ff57ff540,0xfffffd7ff5800000)
 PSPermGen total 22528K, used 8205K [0xfffffd7fe5800000, 0xfffffd7fe6e00000, 0xfffffd7feac00000)
  object space 22528K, 36% used [0xfffffd7fe5800000,0xfffffd7fe6003558,0xfffffd7fe6e00000)
}
{Heap before GC invocations=2445 (full 965):
 PSYoungGen total 58752K, used 29440K [0xfffffd7ff5800000, 0xfffffd7ffae00000, 0xfffffd7ffae00000)
  eden space 29440K, 100% used [0xfffffd7ff5800000,0xfffffd7ff74c0000,0xfffffd7ff74c0000)
  from space 29312K, 0% used [0xfffffd7ff74c0000,0xfffffd7ff74c0000,0xfffffd7ff9160000)
  to space 29312K, 0% used [0xfffffd7ff9160000,0xfffffd7ff9160000,0xfffffd7ffae00000)
 PSOldGen total 176128K, used 176125K [0xfffffd7feac00000, 0xfffffd7ff5800000, 0xfffffd7ff5800000)
  object space 176128K, 99% used [0xfffffd7feac00000,0xfffffd7ff57ff540,0xfffffd7ff5800000)
 PSPermGen total 22528K, used 8205K [0xfffffd7fe5800000, 0xfffffd7fe6e00000, 0xfffffd7feac00000)
  object space 22528K, 36% used [0xfffffd7fe5800000,0xfffffd7fe6003558,0xfffffd7fe6e00000)
4752.090: [Full GC [PSYoungGen: 29440K->3280K(58752K)] [PSOldGen: 176125K->176124K(176128K)] 205565K->179404K(234880K) [PSPermGen: 8205K->8205K(22528K)], 0.4140487 secs]
Heap after GC invocations=2445 (full 965):
 PSYoungGen total 58752K, used 3280K [0xfffffd7ff5800000, 0xfffffd7ffae00000, 0xfffffd7ffae00000)
  eden space 29440K, 11% used [0xfffffd7ff5800000,0xfffffd7ff5b341c8,0xfffffd7ff74c0000)
  from space 29312K, 0% used [0xfffffd7ff74c0000,0xfffffd7ff74c0000,0xfffffd7ff9160000)
  to space 29312K, 0% used [0xfffffd7ff9160000,0xfffffd7ff9160000,0xfffffd7ffae00000)
 PSOldGen total 176128K, used 176124K [0xfffffd7feac00000, 0xfffffd7ff5800000, 0xfffffd7ff5800000)
  object space 176128K, 99% used [0xfffffd7feac00000,0xfffffd7ff57ff030,0xfffffd7ff5800000)
 PSPermGen total 22528K, used 8205K [0xfffffd7fe5800000, 0xfffffd7fe6e00000, 0xfffffd7feac00000)
  object space 22528K, 36% used [0xfffffd7fe5800000,0xfffffd7fe6003558,0xfffffd7fe6e00000)
}

From the application log:
-------------------------
-bash-3.00# tail /bt/VolanoTestrun.7659.-d64/logs/error-2006-10-23.log
[Mon Oct 23 13:43:29 PDT 2006] Error reading from 10.5.17.209. (java.lang.OutOfMemoryError: GC overhead limit exceeded)
[Mon Oct 23 13:43:31 PDT 2006] Error handling packet from 10.5.17.209. (java.lang.OutOfMemoryError: GC overhead limit exceeded)
[Mon Oct 23 13:43:34 PDT 2006] Error reading from 10.5.17.209. (java.lang.OutOfMemoryError: GC overhead limit exceeded)
[Mon Oct 23 13:43:35 PDT 2006] Error handling packet from 10.5.17.209. (java.lang.OutOfMemoryError: GC overhead limit exceeded)
Exception in thread "Receiver-201919"
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
at java.lang.StringBuffer.<init>(StringBuffer.java:79)
at COM.volano.e.run(DashoA5383)
at java.lang.Thread.run(Thread.java:619)

Backup core file location from a another run that set to a much larger heap size:
/net/jtg-x4100-01.sfbay/export/logs/core.1321