Seems more likely after JDK-8323582, but I see similar failures on a while back as well, in the valhalla repo. So I don't think that JDK-8323582 is the cause, but it may now make the use of some XMM registers more likely.

Probably only reproduces with AVX512, and possibly only with a subset of those machines (not on my laptop that has only a subset of AVX512 functionality):
jdk/incubator/vector/Long256VectorTests.java

# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/opt/mach5/mesos/work_dir/slaves/03ecc23a-edd5-4bb5-a333-4ff8ea07fd7c-S1547/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/66737b21-4c8d-4877-aa96-f180c9760371/runs/5a6d0c67-5874-4459-9cc4-f30183b0fd30/workspace/open/src/hotspot/cpu/x86/assembler_x86.cpp:13620), pid=2098796, tid=2098815
# assert(((!is_extended) || (!attributes->is_legacy_mode()))) failed: XMM register should be 0-15
#
# JRE version: Java(TM) SE Runtime Environment (25.0) (fastdebug build 25-internal-LTS-2025-02-27-0711127.emanuel.peter.jdk-fork7)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 25-internal-LTS-2025-02-27-0711127.emanuel.peter.jdk-fork7, mixed mode, sharing, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x66cd78] Assembler::vex_prefix_and_encode(int, int, int, Assembler::VexSimdPrefix, Assembler::VexOpcode, InstructionAttr*, bool, bool, bool) [clone .constprop.2]+0x2b8
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/03ecc23a-edd5-4bb5-a333-4ff8ea07fd7c-S1941/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/e1d8c66f-7708-45a7-bc8e-d4e603b0dda9/runs/401b2f42-2f9e-4a1f-a29b-cbb74a09981a/testoutput/test-support/jtreg_open_test_jdk_jdk_vector/scratch/0/core.2098796)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp


Current CompileTask:
C2:17089 1722 b Long256VectorTests::SADDLong256VectorTests (152 bytes)

Stack: [0x00007f42ebafc000,0x00007f42ebbfc000], sp=0x00007f42ebbf7680, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x66cd78] Assembler::vex_prefix_and_encode(int, int, int, Assembler::VexSimdPrefix, Assembler::VexOpcode, InstructionAttr*, bool, bool, bool) [clone .constprop.2]+0x2b8 (assembler_x86.cpp:13620)
V [libjvm.so+0x6794dd] Assembler::vpcmpeqq(XMMRegister, XMMRegister, XMMRegister, int)+0xbd (assembler_x86.cpp:5104)
V [libjvm.so+0x8a88a3] C2_MacroAssembler::vpgenmax_value(BasicType, XMMRegister, XMMRegister, int, bool)+0x33 (c2_MacroAssembler_x86.cpp:6862)
V [libjvm.so+0x8a905b] C2_MacroAssembler::vector_addsub_dq_saturating_evex(int, BasicType, XMMRegister, XMMRegister, XMMRegister, XMMRegister, XMMRegister, KRegister, KRegister, int)+0x18b (c2_MacroAssembler_x86.cpp:6953)
V [libjvm.so+0x4552a7] vector_addsub_saturating_reg_evexNode::emit(C2_MacroAssembler*, PhaseRegAlloc*) const+0x2c7 (x86.ad:10685)
V [libjvm.so+0x159482c] PhaseOutput::scratch_emit_size(Node const*)+0x3dc (output.cpp:3385)
V [libjvm.so+0x158cc2a] PhaseOutput::shorten_branches(unsigned int*)+0x33a (output.cpp:539)
V [libjvm.so+0x159e776] PhaseOutput::Output()+0xa36 (output.cpp:339)
V [libjvm.so+0xa6b3da] Compile::Code_Gen()+0x5aa (compile.cpp:3017)
V [libjvm.so+0xa6e18f] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1c9f (compile.cpp:881)
V [libjvm.so+0x8ba2e5] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1d5 (c2compiler.cpp:141)
V [libjvm.so+0xa7a958] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x928 (compileBroker.cpp:2331)
V [libjvm.so+0xa7b698] CompileBroker::compiler_thread_loop()+0x528 (compileBroker.cpp:1975)
V [libjvm.so+0xf4423e] JavaThread::thread_main_inner()+0xee (javaThread.cpp:776)
V [libjvm.so+0x189b466] Thread::call_run()+0xb6 (thread.cpp:231)
V [libjvm.so+0x1573d08] thread_native_entry(Thread*)+0x128 (os_linux.cpp:877)


Test is randomized, so it might require the seed:

For random generator using seed: 9171110224325618755
To re-run test with same seed value please add "-Djdk.test.lib.random.seed=9171110224325618755" to command line.