I have been noticing lately (last 2-3 weeks maybe but hard to pin down) that we have a number of crashing tests in the CI that don't seem to produce a hs_err file. I initially thought it was due to an issue in hotspot where recursive errors stopped the production of the file. However @skarlsso dug into this and discovered what is really happening. It seems that somehow when test A experiences a crash, the hs_err output mostly appears in the jtr file for a different test B, and the hs_err_pidxxx.log file gets put into the scratch directory for test B. Because test B passes, its scratch directory gets deleted and we never find the hs_err file.

Here's an example I just found in the CI:

Failing test is java/lang/Class/forName/modules/TestDriver.java

The log file for the failing test contains this tail of the hs_err output:
#
# Compiler replay data is saved as:
# /opt/mach5/mesos/work_dir/slaves/e32bbe1d-e945-4ded-9d0a-e114d727b0db-S1923/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2fd69187-e5b3-4d1c-821f-491f3183881a/runs/a63c5f4c-bba0-4672-b8c6-ad389da3c798/testoutput/test-support/jtreg_open_test_jdk_jdk_lang/scratch/3/replay_pid976580.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

The log file for passing test java/lang/Character/CheckProp.java
contains the start of the hs_err output at its end:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/opt/mach5/mesos/work_dir/slaves/03ecc23a-edd5-4bb5-a333-4ff8ea07fd7c-S855/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/fa757e9a-cc57-4f4d-8336-cd7d91e8d55a/runs/378168e2-9a45-4ea6-9815-8ac06726c10c/workspace/open/src/hotspot/share/opto/phaseX.cpp:1855), pid=976580, tid=976741
# assert(!failure) failed: PhaseCCP not at fixpoint: analysis result may be unsound.
#
# JRE version: Java(TM) SE Runtime Environment (25.0+15) (fastdebug build 25-ea+15-LTS-1603)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 25-ea+15-LTS-1603, mixed mode, compressed oops, compressed class ptrs, serial gc, linux-aarch64)
# Problematic frame:
# V [libjvm.so+0x144c190] PhaseCCP::analyze()+0x700
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/e32bbe1d-e945-4ded-9d0a-e114d727b0db-S1923/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2fd69187-e5b3-4d1c-821f-491f3183881a/runs/a63c5f4c-bba0-4672-b8c6-ad389da3c798/testoutput/test-support/jtreg_open_test_jdk_jdk_lang/scratch/3/core.976580)
#
# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/e32bbe1d-e945-4ded-9d0a-e114d727b0db-S1923/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2fd69187-e5b3-4d1c-821f-491f3183881a/runs/a63c5f4c-bba0-4672-b8c6-ad389da3c798/testoutput/test-support/jtreg_open_test_jdk_jdk_lang/scratch/3/hs_err_pid976580.log