The attached Java Fuzzer test hit the memory limit assert. I then run it with the new CompileTaskTimeout and it timed out after 10s. I reduced the test a bit (Reduced.java) which now still takes 10s but will succeed a few seconds later.

There still seems to be an inefficiency which we should investigate further.

# Intermittent - depends on StressLoopPeeling taking the "right" decisions to peel. Otherwise, the compilation is completed instantly. By running with TraceLoopOpts, one can see where it suddenly becomes stuck.
$ java -XX:CompileCommand=compileonly,*Reduced*::test -XX:-TieredCompilation -Xcomp -XX:CompileCommand=dontinline,*::* -XX:+StressLoopPeeling -XX:PerMethodTrapLimit=0 -XX:+VerifyLoopOptimizations -XX:CompileTaskTimeout=10000 -XX:+TraceLoopOpts Reduced.java

$ java -XX:CompileOnly=Test::* -XX:-TieredCompilation -Xcomp -XX:CompileCommand=dontinline,*::* -XX:+StressLoopPeeling -XX:PerMethodTrapLimit=0 -XX:+VerifyLoopOptimizations -XX:CompileCommand=printcompilation,*::* -XX:CompileTaskTimeout=10000 -XX:+TraceLoopOpts Test.java


Stuck at timeout:
V [libjvm.so+0xaae5f1] CompilerThreadTimeoutLinux::compiler_signal_handler(int, siginfo*, void*)+0xe3 (compilerThreadTimeout_linux.cpp:47)
V [libjvm.so+0xaae50b] compiler_signal_handler(int, siginfo*, void*)+0x35 (compilerThreadTimeout_linux.cpp:37)
C [libc.so.6+0x3ebf0]
V [libjvm.so+0x13b82e9] PhaseIdealLoop::get_late_ctrl_with_anti_dep(LoadNode*, Node*, Node*)+0xb8d (loopnode.cpp:6510)
V [libjvm.so+0x13b76e2] PhaseIdealLoop::get_late_ctrl(Node*, Node*)+0x132 (loopnode.cpp:6435)
V [libjvm.so+0x13b9404] PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0x42a (loopnode.cpp:6823)
V [libjvm.so+0x13b8fd6] PhaseIdealLoop::build_loop_late_post(Node*)+0x28 (loopnode.cpp:6711)
V [libjvm.so+0x13b8bb8] PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x272 (loopnode.cpp:6656)
V [libjvm.so+0x13b1f12] PhaseIdealLoop::build_and_optimize()+0xbfc (loopnode.cpp:5093)
V [libjvm.so+0xa7df1e] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, PhaseIdealLoop const*)+0x14e (loopnode.hpp:1204)
V [libjvm.so+0x13b3195] PhaseIdealLoop::verify() const+0x135 (loopnode.cpp:5332)
V [libjvm.so+0x13c623e] PhaseIdealLoop::split_if_with_blocks_post(Node*)+0x8c4 (loopopts.cpp:1527)
V [libjvm.so+0x13c8c4b] PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&)+0x1f3 (loopopts.cpp:1984)
V [libjvm.so+0x13b2894] PhaseIdealLoop::build_and_optimize()+0x157e (loopnode.cpp:5206)
V [libjvm.so+0xa7ddc5] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x191 (loopnode.hpp:1189)
V [libjvm.so+0xa7e264] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x10c (loopnode.hpp:1283)
V [libjvm.so+0xa6c017] Compile::optimize_loops(PhaseIterGVN&, LoopOptsMode)+0x9d (compile.cpp:2254)
V [libjvm.so+0xa6d711] Compile::Optimize()+0x1519 (compile.cpp:2503)
V [libjvm.so+0xa648a2] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1916 (compile.cpp:859)
V [libjvm.so+0x90c489] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x48f (c2compiler.cpp:147)
V [libjvm.so+0xa8cddd] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xe01 (compileBroker.cpp:2340)
V [libjvm.so+0xa8b1d6] CompileBroker::compiler_thread_loop()+0x596 (compileBroker.cpp:1984)
V [libjvm.so+0xaae1c1] CompilerThread::thread_entry(JavaThread*, JavaThread*)+0x89 (compilerThread.cpp:69)
V [libjvm.so+0xf94cfc] JavaThread::thread_main_inner()+0x1b2 (javaThread.cpp:775)
V [libjvm.so+0xf94b47] JavaThread::run()+0x1e3 (javaThread.cpp:760)
V [libjvm.so+0x189e66d] Thread::call_run()+0x1b9 (thread.cpp:243)
V [libjvm.so+0x159bbd1] thread_native_entry(Thread*)+0x1f5 (os_linux.cpp:868)