For Java and Compiler threads the return code of pthread_attr_setstacksize
is overwritten. If pthread_attr_setstacksize does not succeed, a tiny
stack size with not enough space for the yellow/red pages is allocated.
VM crashes with SIGSEGV right away. This fixes tests TestOptionWithRanges.java
and TestThreadStackSizes.java.

Behaviour differs from linux x86_64 because linux (glibc) accepts any sizes for
pthread_attr_setstacksize and only the later pthread_create fails.

Messages:

before:
bin/java -XX:CompilerThreadStackSize=8000000000000
[0.423s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.423s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.424s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.424s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.424s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.424s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.425s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
[0.425s][warning][os,thread] The thread stack size specified is invalid: 8000000000000k
Segmentation fault (core dumped)

now:

bin/java -XX:CompilerThreadStackSize=80000000000000
[1.989s][warning][os,thread] The compiler thread stack size specified is invalid: 80000000000000k
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

linux x86_64:

bin/java -XX:CompilerThreadStackSize=8000000000000
[0.271s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 8000000000000k, guardsize: 0k, detached.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached