Solaris malloc can only handle one contigues chunk of memory. The class data sharing file gets mapped at address 0x10000000 on 32 bit Solaris. This means the the malloc heap can not grow beyond that address.

Below is an example where the malloc heap starts at 0x08061000 and now has to end at 0x10000000, which limits the malloc heap to 127 MB. This in turn gives us a native out of memory long before we run out of address space.

$ pmap -xs 6877
6877: /scratch/brutisso/jdk8/jdk1.8.0/bin/java -client -Xcomp -Xmx256m -Xmn1
 Address Kbytes RSS Anon Locked Pgsz Mode Mapped File
08044000 16 16 16 - 4K rw--- [ stack ]
08050000 4 4 - - 4K r-x-- java
08060000 4 4 4 - 4K rwx-- java
08061000 120 120 120 - 4K rwx-- [ heap ]
0807F000 8 - - - - rwx-- [ heap ]
08081000 460 460 460 - 4K rwx-- [ heap ]
080F4000 8 - - - - rwx-- [ heap ]
080F6000 44 44 44 - 4K rwx-- [ heap ]
08101000 24 - - - - rwx-- [ heap ]
08107000 1160 1160 1160 - 4K rwx-- [ heap ]
08229000 4 - - - - rwx-- [ heap ]
0822A000 128852 128852 128852 - 4K rwx-- [ heap ]
10000000 748 748 - - 4K r--s- classes.jsa
100BB000 4 - - - - r--s- classes.jsa
100BC000 232 232 - - 4K r--s- classes.jsa
100F6000 104 - - - - r--s- classes.jsa
10110000 36 36 - - 4K r--s- classes.jsa
10119000 4 - - - - r--s- classes.jsa