It looks like some +UseSHM test cases added by JDK-8213269 reliably blow up the VM log reader with OOME. There are lots of "OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory." in the log, if you increase the test heap size. AFAIU, many of those messages are expected from the new test cases.

$ make run-test TEST=jtreg:gtest/LargePageGtests.java

jdk.test.lib.process.OutputBuffer$OutputBufferException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at jdk.test.lib.process.OutputBuffer$LazyOutputBuffer$StreamTask.get(OutputBuffer.java:115)
at jdk.test.lib.process.OutputBuffer$LazyOutputBuffer.getStderr(OutputBuffer.java:144)
at jdk.test.lib.process.OutputAnalyzer.getStderr(OutputAnalyzer.java:557)
at jdk.test.lib.process.ProcessTools.getProcessLog(ProcessTools.java:494)
at jdk.test.lib.process.ProcessTools.executeProcess(ProcessTools.java:470)
at jdk.test.lib.process.ProcessTools.executeProcess(ProcessTools.java:417)
at jdk.test.lib.process.ProcessTools.executeProcess(ProcessTools.java:404)
at jdk.test.lib.process.ProcessTools.executeCommand(ProcessTools.java:553)
at GTestWrapper.main(GTestWrapper.java:91)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at jdk.test.lib.process.OutputBuffer$LazyOutputBuffer$StreamTask.get(OutputBuffer.java:109)
... 14 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3536)
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
at jdk.test.lib.process.StreamPumper.run(StreamPumper.java:111)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
... 1 more


This seems to help:

diff --git a/test/hotspot/jtreg/gtest/LargePageGtests.java b/test/hotspot/jtreg/gtest/LargePageGtests.java
index 87ce5db8064..4fdcf655703 100644
--- a/test/hotspot/jtreg/gtest/LargePageGtests.java
+++ b/test/hotspot/jtreg/gtest/LargePageGtests.java
@@ -61,5 +61,5 @@
  * @modules java.base/jdk.internal.misc
  * java.xml
  * @requires vm.flagless
- * @run main/native GTestWrapper --gtest_filter=os* -XX:+UseLargePages -XX:+UseSHM
+ * @run main/othervm/native -Xmx2g GTestWrapper --gtest_filter=os* -XX:+UseLargePages -XX:+UseSHM
  */

Dropping the number of concurrent threads from 30 (!) to 4 helps as well:

diff --git a/test/hotspot/gtest/runtime/test_os_linux.cpp b/test/hotspot/gtest/runtime/test_os_linux.cpp
index f4841efe342..35e98d57a27 100644
--- a/test/hotspot/gtest/runtime/test_os_linux.cpp
+++ b/test/hotspot/gtest/runtime/test_os_linux.cpp
@@ -415,7 +415,7 @@ public:
 
 TEST_VM(os_linux, reserve_memory_special_concurrent) {
   ReserveMemorySpecialRunnable runnable;
- ConcurrentTestRunner testRunner(&runnable, 30, 15000);
+ ConcurrentTestRunner testRunner(&runnable, 4, 15000);
   testRunner.run();
 }

I believe ultimately this test produces a virtually unbounded number of warning messages, which would eventually blow out the Java heap. ConcurrentTestRunner runs a time-bound number of iterations, which means the faster machine is, the more warning messages would be printed. Maybe the way out is to make ConcurrentTestRunner do the static number of iterations, so that VM output length is more predictable.