Consider this trivial jtreg test:

/*
 * @test
 * @summary Test to verify that jtreg's jtr file contains System.out and System.err messages when
 * the test times out when run in agent vm mode
 * @run main/timeout=15 TimeoutTest
 */
public class TimeoutTest {

    public static void main(final String[] args) throws Exception {
        System.out.println("**** A System.out message");
        System.err.println("**** A System.err message");
        final int numSecs = 600;
        System.out.println("Sleeping for " + numSecs + " seconds");
        Thread.sleep(numSecs * 1000);
        System.out.println("Done sleeping");
    }
}

The test writes to System.out and System.err and (intentionally) sleeps for 10 minutes to simulate a test that times out. The test is marked with "@run main/timeout=15" just to complete the test execution quickly. Other than that it plays no role in the issue being reported here.

When this test is run in either implicit or explicit agent vm mode:

jtreg -avm -report:files -jdk:build/macosx-aarch64/images/jdk <paht-to-test-file>

The test fails with a timeout as expected. When the corresponding jtr file is checked, it doesn't contain any of the System.out or System.err messages that the test had logged before it timed out.
If the same test is run in othervm mode (either implicitly or explicitly), the jtr file does indeed contain the System.out and System.err messages. So this only affects agent vm mode.

This affects many tests that timeout on mach5.

Locally, the jtreg version in use is:

jtreg -version
jtreg 7.1.1+1-1
Installed in /home/me/jtreg/jtreg-7.1.1+1/lib/jtreg.jar
Running on platform version 19.0.2 from /home/me/jdk-19.0.2.jdk/Contents/Home.
Built with Java(TM) 2 SDK, Version 11.0.16+11-LTS-199 on December 06, 2022.
Copyright (c) 1999, 2022, Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
JT Harness, version 6.0 ea b24 (September 27, 2022)
Java Assembler Tools, version 7.0 ea b09 (June 04, 2020)
TestNG: testng-7.3.0.jar, jcommander-1.78.jar, guice-4.2.3.jar
JUnit: junit-platform-console-standalone-1.8.2.jar