This seems to manifest on the following test with x86_32. The bisection shows it started with JDK-8267612, but that one probably just triggers it. Confusingly, this only happens with fastdebug, not slowdebug, not release. Maybe the stack is just a little bit too small for fastdebug config.

$ CONF=linux-x86-server-fastdebug make run-test TEST=java/lang/invoke/VarHandles/VarHandleTestAccessShort.java

configuration:
STDOUT:
config VarHandleTestAccessShort.setup(): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Instance final field", VarHandleBaseTest$VarHandleAccessTestCase@3d664d): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Instance final field unsupported", VarHandleBaseTest$VarHandleAccessTestCase@ac964b): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Static final field", VarHandleBaseTest$VarHandleAccessTestCase@b82994): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Static final field unsupported", VarHandleBaseTest$VarHandleAccessTestCase@1b983b5): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Instance field", VarHandle -> Instance field): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Instance field unsupported", VarHandleBaseTest$VarHandleAccessTestCase@1922a2b): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Static field", VarHandle -> Static field): success
test VarHandleTestAccessShort.testAccess("VarHandle -> Static field unsupported", VarHandleBaseTest$VarHandleAccessTestCase@22ff6b): success
STDERR:


TEST RESULT: Failed. Unexpected exit from test [exit code: 139]
--------------------------------------------------
Test results: failed: 1


There is no hs_err, because it is a native stack overflow.

Core parse:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0xf6867f63 in DUIterator_Fast::DUIterator_Fast (offset=188, node=0xbee8ca98, this=0xbe783038)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/node.hpp:1398
1398 { _outp = node->_out + offset; debug_only(sample(node)); }
[Current thread is 1 (Thread 0xbe7ffb40 (LWP 811246))]

(gdb) info stack
#0 0xf6867f63 in DUIterator_Fast::DUIterator_Fast (offset=188, node=0xbee8ca98, this=0xbe783038)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/node.hpp:1398
#1 Node::fast_outs (imax=..., this=0xbee8ca98) at /home/shade/trunks/jdk/src/hotspot/share/opto/node.hpp:1429
#2 Compile::start (this=0xbe7fe41c) at /home/shade/trunks/jdk/src/hotspot/share/opto/compile.cpp:1080
#3 0xf69ea5bb in ConnectionGraph::find_inst_mem (this=0xb8879438, orig_mem=0xb89deba0, alias_idx=32, orig_phis=...)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/escape.cpp:2833
#4 0xf69eab83 in ConnectionGraph::find_inst_mem (this=<optimized out>, orig_mem=<optimized out>, alias_idx=<optimized out>, orig_phis=...)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/escape.cpp:2899
#5 0xf69eab83 in ConnectionGraph::find_inst_mem (this=<optimized out>, orig_mem=<optimized out>, alias_idx=<optimized out>, orig_phis=...)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/escape.cpp:2899
#6 0xf69eab83 in ConnectionGraph::find_inst_mem (this=<optimized out>, orig_mem=<optimized out>, alias_idx=<optimized out>, orig_phis=...)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/escape.cpp:2899
#7 0xf69eab83 in ConnectionGraph::find_inst_mem (this=<optimized out>, orig_mem=<optimized out>, alias_idx=<optimized out>, orig_phis=...)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/escape.cpp:2899
#8 0xf69eb953 in ConnectionGraph::split_memory_phi (this=0xb8879438, orig_phi=0xbbf86fa8, alias_idx=32, orig_phi_worklist=...)
    at /home/shade/trunks/jdk/src/hotspot/share/opto/node.hpp:391

The last 6 frames repeat themselves for about 150 more times until they run out of stack. The `orig_phi` are all different, which suggest this is not a guaranteed-recursive stack overflow, but just the recursive optimization running out of stack.