Since JDK-20, this seems to reproduce:

/oracle-work/jdk-20.0.2/fastdebug/bin/java -XX:UseAVX=3 -XX:+UnlockDiagnosticVMOptions -XX:+UseKNLSetting -XX:CompileCommand=compileonly,Test2::test -XX:CompileCommand=printcompilation,Test2::* -XX:-TieredCompilation Test2.java

Looks like bad combo of ConvF2HF and KNL.

-------------- Original Info ------------
During work for JDK-8308606 and JDK-8303113 I wanted to whitelist "AlignVector" in the IR framework. Then I hit this issue with test
compiler/vectorization/TestFloatConversionsVector.java

The "-XX:+UseKNLSetting -XX:+AlignVector" combo is not a real hardware.

I reduced it down to Test.java

./java -XX:UseAVX=3 -XX:+UnlockDiagnosticVMOptions -XX:+UseKNLSetting -XX:+AlignVector -XX:CompileCommand=compileonly,Test::test -XX:CompileCommand=printcompilation,Test::* -XX:-TieredCompilation -XX:+TraceSuperWord Test.java

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/oracle-work/jdk-fork0/open/src/hotspot/cpu/x86/assembler_x86.cpp:11506), pid=1769078, tid=1769091
# assert(((!attributes->uses_vl()) || (attributes->get_vector_len() == AVX_512bit) || (!_legacy_mode_vl) || (attributes->is_legacy_mode()))) failed: XMM register should be 0-15
#
# JRE version: Java(TM) SE Runtime Environment (21.0) (slowdebug build 21-internal-LTS-2023-06-07-1344422.emanuel...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 21-internal-LTS-2023-06-07-1344422.emanuel..., mixed mode, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x655b2a] Assembler::vex_prefix_and_encode(int, int, int, Assembler::VexSimdPrefix, Assembler::VexOpcode, InstructionAttr*)+0x15a
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /oracle-work/jdk-fork0/build/linux-x64-slowdebug/jdk/bin/core.1769078)
#
# An error report file with more information is saved as:
# /oracle-work/jdk-fork0/build/linux-x64-slowdebug/jdk/bin/hs_err_pid1769078.log
#
# Compiler replay data is saved as:
# /oracle-work/jdk-fork0/build/linux-x64-slowdebug/jdk/bin/replay_pid1769078.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#