The symptom is, as soon parameter -Xprof is used at (as it seems) any java program, the native heap of the process is growing slowly but endless. This seems a indication for a memory leak.

The original used case is Oracle Coherence product, where it is described at the manual that customers can use -Xprof "for ever" at production environment.

The issue is reproducible on Solaris x86 and Linux. Other platforms not tested so far.

=======================================
Howto reproduce with Oracle Coherence:
=======================================
1. Installation of Coherence
- Download Coherence for Java
http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html?ssSourceSiteId=ocomen

2. Unzip to a free folder

3. Modify COHERENCE_HOME/bin/cache-server.sh

…
MEMORY=64m
…
JAVA_OPT="-Xprof -Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.ttl=0"
…

(This will not allow multicast packets to leave your machine and it will switch on the profiler)

4. Run one Coherence cache server instance from COHERENCE_HOME with

bin/cache-server.sh

5. Identify the PID of the JVM running "DefaultCacheServer" with JDK command "jps"

6. Run attached script GetMemUsagePID.sh <PID>
(This will log total virtual memory size and resident size of the prozess every 2 sec into memusage_pid<PID>.csv)

7. You can watch the consumption of virtual memory with
tail –f memusage_pid<PID>.csv
(The memory consumption is growing extremely slowly. It takes hours or days until the process runs into OutOfMemoryError.)

=======================================
Howto reproduce with a Hello World loop
using Solaris 10:
=======================================
1.) compile this Hello.java program (it's a endless loop and print every 15min a timestamp)

import java.util.Date;
import java.text.SimpleDateFormat;
public class Hello
{
    public static void main (String argv[]) throws Exception
    {
        int count = 1;
        do {
            Date todaysDate = new java.util.Date();
            SimpleDateFormat formatter = new SimpleDateFormat("EEE, dd-MMM-yyyy HH:mm:ss");
            String formattedDate = formatter.format(todaysDate);
            System.out.println("java thread sleep for 15min, starting at "+formattedDate);
            Thread.currentThread().sleep( 900000 );
        } while (count <= 2);
    }
}

2) create a shell script to monitor the process with the help of Solaris pmap like this (pwatch.sh)

#!/bin/ksh

if [ $# -eq 0 ]; then
    print "$0 <pid>"
    exit 1
fi

SLEEP=900

while [ 1 ]
do
    DATE=`/usr/bin/date +%Y%m%d-%H%M%S`
    print "Gathering data on $DATE ..."
    /usr/bin/pmap -r $1 > pmap.$DATE 2> pmap.$DATE.err
    print "Sleeping for $SLEEP seconds (Ctrl+C breaks) ..."
    /usr/bin/sleep $SLEEP
done

3) Start the Hello Java program with
/path/to/your/java -Xms64m -Xmx64m -Xprof Hello

4) pickup the process ID and let run your pwatch script at another terminal

5) let it run some hours or days or weeks and grep from time to time
at the pmap output files for "[ heap ]" and "total".
We will see a constant growing of overall process size on native heap
(don't mix it with java heap! java heap is fine.).

Suggested environment to reproduce:
At a Oracle Solaris 10 update 9 x86 Solaris, running
$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) Server VM (build 17.0-b16, mixed mode)

I was able to measure this growing (grep-ed from pmap)
05.10.2010_16:19:32|2200K|104128K
...
12.10.2010_17:35:08|2278000K|2379948K

Which means that at program start, the native heap was small as 2200K and after 7 days 2278000K.
And the overall process total size grows from 104128K up to 2379948K.