System description:
X4800 oaf657 128 x 2267MHz 262135 Megabytes

oaf657# java -d64 -version 2>/dev/stdout
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for E in /HUDSON/workspace/jdk7u7-2-build-solaris-amd64-product/jdk7u7/hotspot/src/share/vm/utilities/taskqueue.hpp
# An error report file with more information is saved as:
# /export/bench/autobench/5.x/Results/hs_err_pid7393.log

oaf657# java -version 2>/dev/stdout
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) Server VM (build 23.3-b01, mixed mode)

oaf657# swap -lh
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 274,2 4K 218G 218G

oaf657# echo ::memstat |mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 4236512 16548 6%
ZFS File Data 269796 1053 0%
Anon 48217526 188349 72%
Exec and libs 54653 213 0%
Page cache 6824 26 0%
Free (cachelist) 32705 127 0%
Free (freelist) 14288517 55814 21%
Total 67106533 262134
using 1.6.0 installed in the box it doesn't have the same problem:

oaf657# ./java -d64 -version
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)

but I've installed 1.7.0_05 and it fails:

oaf657# ./java -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b06)
Java HotSpot(TM) Server VM (build 23.1-b03, mixed mode)

oaf657# ./java -d64 -version
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for E in /export/HUDSON/workspace/jdk7u5-2-build-solaris-amd64-product/jdk7u5/hotspot/src/share/vm/utilities/taskqueue.hpp
# An error report file with more information is saved as:
# /root/7u5/solaris-amd64/bin/hs_err_pid1353.log

I'm copying older updates, I'll be updating this bug once I've got more information.
The problem starts in 7u4.

7u3:

oaf657# ./java -version
java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b04)
Java HotSpot(TM) Server VM (build 22.1-b02, mixed mode)

oaf657# ./java -d64 -version
java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b04)
Java HotSpot(TM) 64-Bit Server VM (build 22.1-b02, mixed mode)

7u4:

oaf657# ./java -version
java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b20)
Java HotSpot(TM) Server VM (build 23.0-b21, mixed mode)

oaf657# ./java -d64 -version
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for E in /export/HUDSON/workspace/jdk7u4-2-build-solaris-amd64-product/jdk7u4/hotspot/src/share/vm/utilities/taskqueue.hpp
# An error report file with more information is saved as:
# /root/7u4/solaris-amd64/bin/hs_err_pid2894.log
So far I haven't found this problem anywhere else.

oaf657# echo ::memstat |mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 3938332 15384 6%
ZFS File Data 237030 925 0%
Anon 54886 214 0%
Exec and libs 1215 4 0%
Page cache 4649 18 0%
Free (cachelist) 120686 471 0%
Free (freelist) 62749735 245116 94%
Total 67106533 262134

oaf657# ./java -d64 -version
java version "1.7.0_03-fastdebug"
Java(TM) SE Runtime Environment (build 1.7.0_03-fastdebug-b04)
Java HotSpot(TM) 64-Bit Server VM (build 22.1-b02-fastdebug, mixed mode)

oaf657# ./java -d64 -version
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for E in /export/HUDSON/workspace/jdk7u4-2-build-solaris-amd64-product/jdk7u4/hotspot/src/share/vm/utilities/taskqueue.hpp
# An error report file with more information is saved as:
# /root/7u4/solaris-amd64/fastdebug/bin/hs_err_pid2616.log

I've attached the log file.
I've just reproduced the same problem in another x4800:

X4800 oaf690 128 x 2267MHz 524279 Megabytes

oaf690# java -d64 -version
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for E in /HUDSON/workspace/jdk7u7-2-build-solaris-amd64-product/jdk7u7/hotspot/src/share/vm/utilities/taskqueue.hpp
# An error report file with more information is saved as:
# /root/hs_err_pid20757.log

both boxes are running s11u1_24a.