On Solaris, the process is normally layed out like this:

0x0
text
data
heap

On SPARC, the text segment is by default >4GB, and on AMD64 <4GB. This is a problem for example for the GC. The GC wants to put the Java heap in the lower 32 GB if possible to get optimized compressed oops. This clashes with where native memory is allocated with a malloc implementation using sbrk.

This problem is easily fixed by placing the text segment above 32 GB virtual address so that compressed oops may be used without running into malloc problems.

One way of fixing this is by linking the launchers with this mapfile:

$mapfile_version 2
$if _ELF64
LOAD_SEGMENT heap { VADDR = 0x800000000 };
$endif

This places the text segment at 32 GB, and hence dodges all known Solaris malloc issues. This works great on SPARC, but currently the compiler produces messed up ELF files for AMD64 that crop the high order 32 bits of references to external functions, resulting in SIGSEGV.

Ideally we would also be able to do this in a map file:

RESERVE_SEGMENT JavaHeap {
        VADDR = 0x0;
        SIZE = 0x800000000;
};

This feature works on Solaris 11.3 and basically reserves the lower 32 GB VA in a segment in the ELF file so the OS won't put any funny things in there. I can compile this on Solaris 11.3 with studio 12.4 with no problem, but have not been able to do so with current compilers.

Once the studio people have worked out a bug fix for the linker and it is backported to 12.4 that we use, I propose applying the patches to our compilers to make a world of Solaris bugs disappear.