After JDK-8010722 the maximum heap allowed for supporting COOPs changed. In case of sparcv9, instead of Coops with base address the VM now is able to use zero base compressed oops.

I.e. the VM successfully maps the heap at 4G+4M (4M being the max large page size); however on sparcv9 the default base address for the application image is at 4G too. Since the malloc() heap always starts after the application image, there is very little room for it, leading to OOME.

E.g. with the changes in JDK-8010722 the maximum heap supporting Coops is now 28668 MB instead of 28672 MB. This difference is enough to cause this issue. Previously the VM used coops with base address instead of zero based oops (wrongly), so this did not trigger.

It also only triggers on machines with >= 32G of physical memory.

Note that the same issue can be reproduced by manually setting the heap to 28668 MB using -Xmx.

This has only been seen on solaris-sparcv9.

Workaround:
 - specify maximum heap size that is reasonably low (-Xmx) or larger than the value above (e.g. outside of (OopEncodingHeapMax - HeapBaseMinAddress - X, OopEncodingHeapMax - HeapBaseMinAddress).
 - set LargePageSizeInBytes to something high (e.g. 1G) - this also increases the available C heap
 - set heap base minimum address to something > 4G (e.g. 6G to get 2G C heap)

P2: high impact (practically guaranteed OOME), occurrence medium (needs machine with >= 32G, special -Xmx settings), workaround simple (HHL)