HotSpot generates code with unaligned stack, crashes on SSE operation

FULL PRODUCT VERSION :
JRE version: (8.0_162-b12) (build )
Java VM: Java HotSpot(TM) Client VM (25.162-b12 mixed mode linux-x86 )
jdk-8u162-linux-i586.tar.gz.

'java -version' crashes immediately due to this bug.


FULL OS VERSION :
32-bit x86 Gentoo Linux, vanilla kernel 4.15.10.

A DESCRIPTION OF THE PROBLEM :
At least on 32-bit x86 HotSpot dynamic compiler generates a machine code that
does not keep the stack pointer aligned to a 16-byte boundary (as required by
the i386 psABI).

When such generated code calls back a gcc-compiled C/C++ code (in JVM,
glibc or some other library) this misaligned stack travels along a call chain
until it hits something that assumes a proper stack alignement.
For example, hitting a SSE operation results in a segfault due to a misaligned
address.

For example let's consider a code generated by
InterpreterGenerator::generate_native_entry() in
src/cpu/x86/vm/templateInterpreter_x86_32.cpp.
This function explicitly generates a stack-realigning instruction
("__ andptr(rsp, -(StackAlignmentInBytes));"), then uses the
MacroAssembler::call_VM() method with a one register argument in
src/cpu/x86/vm/macroAssembler_x86.cpp to generate a function call.

This method generates a "call" instruction, then a "push" instruction to load
the argument, then the call_VM_base() method in the same class will generate
an additional "push" instruction loading a JavaThread pointer before
finally generating a "call" instruction to
InterpreterRuntime::prepare_native_call().
This will result in total of 3 words being pushed to the stack before the
call, misaligning it.

It is easy to see (some) call paths that call C/C++ code with the stack not
aligned correctly - it turns out that HotSpot already has
os::verify_stack_alignment() method for this.
However, it is unexpectedly ifdef'ed out on 32-bit x86 - I would strongly
recommend enabling it there, too.

This issue is also tracked as IcedTea bug 3533
( https://icedtea.classpath.org/bugzilla/show_bug.cgi?id=3533 )
and Gentoo bug 647954 ( https://bugs.gentoo.org/647954 ).
These bug reports contain some additional information, for example
possible fixes to the code generator.


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Enable the stack alignement check in os::verify_stack_alignment()
also for 32-bit x86, rebuild JVM and see how fast assert there gets triggered.

Note that not all entry points call this function so it's not enough
to only the fix call chains that trigger that assert.

16-byte stack alignement is required by i386 psABI so every such assertion
failure signifies a violation of this ABI (these violations now started to bite
since newer GCCs do make use of this requirement to generate code with
SSE instructions in things like glibc).


EXPECTED VERSUS ACTUAL BEHAVIOR :
The HotSpot code generator should be fixed to generate i386
psABI-conforming code.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (os_linux_x86.cpp:291), pid=226557, tid=0xa42ffb40
# fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.
#
# JRE version: (8.0_162-b12) (build )
# Java VM: Java HotSpot(TM) Server VM (25.162-b12 mixed mode linux-x86 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid226557.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Aborted


REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
1) Recompile the JVM with -mincoming-stack-boundary=2
(obviously possible only for IcedTea),

Or:

2) Recompile all the libraries in the system that are ever called from JVM with {C,CXX}FLAGS that prohibit generation of a SSE code (there might be
other reasons the compiler makes assumptions about the stack alignement,
however).