Please mention me in the PR, I put a lot of work into getting a nice JASM reproducer from some nasty compiled class files.

emanuel@emanuel-oracle:/oracle-work/triage/next$ java -jar ~/Documents/asmtools-7.0-build/release/lib/asmtools.jar jasm X.jasm
emanuel@emanuel-oracle:/oracle-work/triage/next$ /oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/java -XX:+UnlockExperimentalVMOptions -Xcomp -XX:PerMethodTrapLimit=0 -XX:PerMethodSpecTrapLimit=0 -XX:StressLongCountedLoop=2000000 -XX:CompileCommand=compileonly,X::* -XX:CompileCommand=printcompilation,X::* X
CompileCommand: compileonly X.* bool compileonly = true
CompileCommand: PrintCompilation X.* bool PrintCompilation = true
492 26 b 3 X::main (11 bytes)
493 27 b 4 X::main (11 bytes)
493 28 b 3 X::test (33 bytes)
494 29 b 4 X::test (33 bytes)
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/oracle-work/jdk-fork4/open/src/hotspot/share/opto/compile.cpp:3776), pid=3982454, tid=3982468
# assert(!n->as_Loop()->is_loop_nest_inner_loop() || _loop_opts_cnt == 0) failed: should have been turned into a counted loop
#
# JRE version: Java(TM) SE Runtime Environment (23.0) (fastdebug build 23-internal-2024-05-22-1054280.emanuel...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 23-internal-2024-05-22-1054280.emanuel..., compiled mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x9d8634] Compile::final_graph_reshaping_main_switch(Node*, Final_Reshape_Counts&, unsigned int, Unique_Node_List&)+0x2144
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /oracle-work/triage/next/core.3982454)
#
# An error report file with more information is saved as:
# /oracle-work/triage/next/hs_err_pid3982454.log
#
# Compiler replay data is saved as:
# /oracle-work/triage/next/replay_pid3982454.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)

Current CompileTask:
C2:519 29 b 4 X::test (33 bytes)

Stack: [0x00007ff53c140000,0x00007ff53c241000], sp=0x00007ff53c23be80, free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x9d8634] Compile::final_graph_reshaping_main_switch(Node*, Final_Reshape_Counts&, unsigned int, Unique_Node_List&)+0x2144 (compile.cpp:3776)
V [libjvm.so+0x9d88ef] Compile::final_graph_reshaping_impl(Node*, Final_Reshape_Counts&, Unique_Node_List&) [clone .part.0]+0x15f (compile.cpp:3171)
V [libjvm.so+0x9d91f4] Compile::final_graph_reshaping_walk(Node_Stack&, Node*, Final_Reshape_Counts&, Unique_Node_List&)+0x174 (compile.cpp:3120)
V [libjvm.so+0x9e4afd] Compile::final_graph_reshaping()+0x53d (compile.cpp:4059)
V [libjvm.so+0x9e68dd] Compile::Optimize()+0x10bd (compile.cpp:2502)
V [libjvm.so+0x9e9c46] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1af6 (compile.cpp:853)
V [libjvm.so+0x83e1f5] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1d5 (c2compiler.cpp:142)
V [libjvm.so+0x9f5a78] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x928 (compileBroker.cpp:2303)
V [libjvm.so+0x9f6708] CompileBroker::compiler_thread_loop()+0x478 (compileBroker.cpp:1961)
V [libjvm.so+0xe9f6bc] JavaThread::thread_main_inner()+0xcc (javaThread.cpp:759)
V [libjvm.so+0x17b2de6] Thread::call_run()+0xb6 (thread.cpp:225)
V [libjvm.so+0x149f1b7] thread_native_entry(Thread*)+0x127 (os_linux.cpp:846)


Having a quick look with RR:

rr /oracle-work/jdk-fork4/build/linux-x64-slowdebug/jdk/bin/java -XX:+UnlockExperimentalVMOptions -Xcomp -XX:PerMethodTrapLimit=0 -XX:PerMethodSpecTrapLimit=0 -XX:StressLongCountedLoop=2000000 -XX:CompileCommand=compileonly,X::* -XX:CompileCommand=printcompilation,X::* X

(rr) p n->dump_bfs(1000,n,"#cA")
dist apd dump
---------------------------------------------
   9 15 99 IfFalse === 363 [[ 120 ]] #0 !jvms: X::test @ bci:17
   8 11 406 IfFalse === 387 [[ 405 ]] #0 !orig=[364],99 !jvms: X::test @ bci:17
   8 18 431 IfTrue === 429 [[ 405 ]] #1 !orig=[404],[364],99 !jvms: X::test @ bci:17
   8 15 120 Loop === 120 5 99 [[ 120 65 432 ]] !orig=[26] !jvms: X::test @ bci:2
   7 11 405 Region === 405 431 406 [[ 405 363 408 ]] !orig=[364],99 !jvms: X::test @ bci:17
   7 15 432 NeverBranch === 120 [[ 433 434 ]]
   6 11 363 If === 405 412 [[ 99 362 ]] P=0.900000, C=-1.000000
   6 15 433 CProj === 432 [[ 60 ]] #0
   5 11 362 IfTrue === 363 [[ 365 ]] #1 !orig=98 !jvms: X::test @ bci:17
   5 15 60 SafePoint === 433 1 7 1 1 65 1 21 21 21 59 [[ 365 ]] SafePoint !jvms: X::test @ bci:17
   4 11 365 Loop === 365 60 362 [[ 365 388 366 379 397 437 ]]
   3 4 90 SafePoint === 386 1 7 1 1 94 1 21 144 144 59 [[ 387 ]] SafePoint !jvms: X::test @ bci:17
   3 11 397 SafePoint === 365 1 7 1 1 388 1 21 389 389 59 [[ 429 ]] SafePoint !orig=90 !jvms: X::test @ bci:17
   2 4 387 If === 90 383 [[ 98 406 ]] P=0.900000, C=-1.000000 !orig=[166],[97] !jvms: X::test @ bci:17
   2 11 429 If === 397 428 [[ 430 431 ]] P=0.100000, C=-1.000000 !orig=[396],387,[166],[97] !jvms: X::test @ bci:17
   1 4 98 IfTrue === 387 [[ 386 ]] #1 !jvms: X::test @ bci:17
   1 11 430 IfFalse === 429 [[ 386 ]] #0 !orig=[395],98 !jvms: X::test @ bci:17
   0 0 386 Loop === 386 430 98 [[ 386 380 94 90 ]] !orig=[167],[122],[78] !jvms: X::test @ bci:20

Looks like an infinite-loop bug (NeverBranch), where we do not do things as expected. This seems to happen often with infinite-loops.

Also, these flags seem to be required:

-Xcomp -XX:PerMethodTrapLimit=0 -XX:PerMethodSpecTrapLimit=0 -XX:StressLongCountedLoop=2000000

The effect of these trap limit flags is that there are no alternative exits, hence the infinite-loop. I have seen this before.