Customer is on a X4140 system with Solaris 10.
This problem only occurs on their x4140 systems and only if they have Kernel patch (141445-09) installed. There is only one and only zone which is the global zone:

java -Xms3g -Xmx3g -version

   bash-3.00# java -Xms3g -Xmx3g -version
   Error occurred during initialization of VM
   Could not reserve enough space for object heap
   Could not create the Java virtual machine.

java -d64 -Xms3g -Xmx3g -version

   bash-3.00# java -d64 -Xms3g -Xmx3g -version
   java version "1.5.0_21"
   Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_21-b01)
   Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_21-b01, mixed mode)

prtconf

   bash-3.00# prtconf
   System Configuration: Sun Microsystems i86pc
   Memory size: 8192 Megabytes
   System Peripherals (Software Nodes):

   i86pc


swap -s

   bash-3.00# swap -s
   total: 301920k bytes allocated + 292916k reserved = 594836k used, 20597376k available

prstat -t

    NPROC USERNAME SWAP RSS MEMORY TIME CPU
        64 root 93M 115M 1.4% 0:00:23 0.0%
        1 noaccess 107M 178M 2.2% 0:00:13 0.0%
        4 samadmin 369M 130M 1.6% 0:00:18 0.0%
        1 smmsp 1200K 5544K 0.1% 0:00:00 0.0%
        6 daemon 6556K 6092K 0.1% 0:00:00 0.0%

   Total: 76 processes, 266 lwps, load averages: 0.02, 0.05, 0.12
prstat

   bash-3.00# prstat
      PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
      907 noaccess 188M 125M sleep 59 0 0:00:13 0.0% java/19
      866 samadmin 384M 126M sleep 60 0 0:00:19 0.0% java/43
      751 root 12M 11M sleep 59 0 0:00:02 0.0% Xvnc/1
     1222 root 3792K 2932K cpu0 59 0 0:00:00 0.0% prstat/1
      718 root 25M 13M sleep 59 0 0:00:16 0.0% Xorg/1
     1039 root 2328K 1376K sleep 59 0 0:00:00 0.0% rpc.rstatd/1
      133 root 5036K 1976K sleep 59 0 0:00:00 0.0% syseventd/15
      430 root 2140K 1000K sleep 59 0 0:00:00 0.0% sac/1
      444 root 2468K 1348K sleep 59 0 0:00:00 0.0% ttymon/1
      389 daemon 4708K 1152K sleep 59 0 0:00:00 0.0% nfsmapid/3
      458 root 1436K 660K sleep 59 0 0:00:00 0.0% utmpd/1
      472 root 2068K 964K sleep 59 0 0:00:00 0.0% smcboot/1
      387 daemon 2444K 1320K sleep 60 -20 0:00:00 0.0% nfs4cbd/2
      277 root 2276K 760K sleep 59 0 0:00:00 0.0% iscsid/2
     1142 root 4348K 2228K sleep 60 0 0:00:00 0.0% dtexec/1
      162 root 3756K 2488K sleep 59 0 0:00:00 0.0% picld/5
      474 root 2068K 668K sleep 59 0 0:00:00 0.0% smcboot/1
      412 daemon 2424K 1364K sleep 60 -20 0:00:00 0.0% lockd/2
      363 root 2844K 1000K sleep 59 0 0:00:00 0.0% cron/1
      382 daemon 2808K 1192K sleep 59 0 0:00:00 0.0% rpcbind/1
      473 root 2068K 668K sleep 59 0 0:00:00 0.0% smcboot/1
   Total: 76 processes, 267 lwps, load averages: 0.02, 0.04, 0.1

vmstat 5 10

   bash-3.00# vmstat 5 10
    kthr memory page disk faults cpu
    r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
    0 0 0 20348804 7391952 175 398 347 4 3 0 550 -0 16 2 2 833 2309 936 1 2 97
    0 0 0 20597232 7363308 0 11 0 0 0 0 0 0 0 0 0 678 471 375 0 0 100
    0 0 0 20597112 7363188 0 1 0 0 0 0 0 0 0 0 0 731 611 413 0 0 100
    0 0 0 20597112 7363192 0 0 0 0 0 0 0 0 0 0 0 641 331 343 0 0 100
    0 0 0 20597112 7363192 0 0 0 0 0 0 0 0 0 0 0 665 313 343 0 0 100

This makes no difference:

bash-3.00# java -Xss256k -Xms3g -Xmx3g -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

ulimit -a

    bash-3.00# ulimit -a
    core file size (blocks, -c) unlimited
    data seg size (kbytes, -d) unlimited
    file size (blocks, -f) unlimited
    open files (-n) 256
    pipe size (512 bytes, -p) 10
    stack size (kbytes, -s) 10240
    cpu time (seconds, -t) unlimited
    max user processes (-u) 16357
    virtual memory (kbytes, -v) unlimited

Attached you will find:

  * Success.truss from a successful run of
    java -Xms3g -Xmx3g -version
    without aforementioned Kernel Patch

  * Failed.truss from a failed run of
    java -Xms3g -Xmx3g -version
    with aforementioned Kernel Patch

  * output of pmap and pmap -r
    ran against the pid of their application process.
    Their application starts with
    -Xms2400m -Xmx2400m -XX:PermSize=650m Hence the need
    for 3Gig test to pass