I attached a simple reproducer (reproducer.jasm). To reproduce, run:
- java org.openjdk.asmtools.Main jasm reproducer.jasm
- java -XX:CompileCommand=quiet -XX:CompileCommand=compileonly,Test::test* -XX:-TieredCompilation -XX:CompileCommand=dontinline,Test::m Test
 
The underlying issue is that the target of the second exception table entry is inside the try/catch scope:
 
      Exception table:
         from to target type
             7 10 13 Class java/lang/Exception
             0 18 15 Class java/lang/Throwable
 
We actually have code in C1 to handle this case:
 
          // It's acceptable for an exception handler to cover itself
          // but we don't handle that in the parser currently. It's
          // very rare so we bailout instead of trying to handle it.
 
https://github.com/openjdk/jdk/blob/7277bb19f128b84094400cb4262b2e0432e559c5/src/hotspot/share/c1/c1_GraphBuilder.cpp#L2504
 
My understanding is that the JVM Specification does not impose any restrictions on an exception handler covering the same range of instructions as itself.

I discussed this with [~dholmes] and he agreed that it's legal for an exception handler to cover itself. The only JVMS restrictions on from/to are that from < to and that they are both in the bci range.
 
In fact, the following code generates such an exception handler with itself as target around monitorexit even with javac:
 
void test() { synchronized(this) { } }
 
         9: astore_2
        10: aload_1
        11: monitorexit
        12: aload_2
 
      Exception table:
         from to target type
             4 6 9 any
             9 12 9 any
 
Since there is only a limited set of cases where javac would create such an exception table, C2 is not well tested with these scenarios. I updated my attached reproducer to cover another problematic case.
 
Failure modes are:
 
   0 26 Catch === 21 22 [[ 27 ]] !jvms: Test::test1 @ bci:0
   1 27 CatchProj === 26 [[ 105 ]] #0@bci -1 !jvms: Test::test1 @ bci:0
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error
(/oracle/jdk2/open/src/hotspot/share/opto/compile.cpp:4003), pid=3085146,
tid=3085159
# assert(false) failed: malformed control flow
#
# JRE version: Java(TM) SE Runtime Environment (21.0) (fastdebug build
21-internal-LTS-2023-05-12-0924511.tobias...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug
21-internal-LTS-2023-05-12-0924511.tobias..., mixed mode, compressed oops,
compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x9b8c24] Compile::final_graph_reshaping()+0x11d4
 
Current CompileTask:
C2: 345 24 ! Test::test1 (5 bytes)
 
Stack: [0x00007f1d57057000,0x00007f1d57158000], sp=0x00007f1d57152f50, free
space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V [libjvm.so+0x9b8c24] Compile::final_graph_reshaping()+0x11d4
(compile.cpp:4003)
V [libjvm.so+0x9ba774] Compile::Optimize()+0x1aa4 (compile.cpp:2450)
V [libjvm.so+0x9bc4a5] Compile::Compile(ciEnv*, ciMethod*, int, Options,
DirectiveSet*)+0x1aa5 (compile.cpp:840)
V [libjvm.so+0x813664] C2Compiler::compile_method(ciEnv*, ciMethod*, int,
bool, DirectiveSet*)+0x3c4 (c2compiler.cpp:118)
V [libjvm.so+0x9c8290]
CompileBroker::invoke_compiler_on_method(CompileTask*)+0x9e0
(compileBroker.cpp:2265)
V [libjvm.so+0x9c9138] CompileBroker::compiler_thread_loop()+0x628
(compileBroker.cpp:1944)
V [libjvm.so+0xe6418e] JavaThread::thread_main_inner()+0xee
(javaThread.cpp:717)
V [libjvm.so+0x172fc98] Thread::call_run()+0xb8 (thread.cpp:216)
V [libjvm.so+0x1432a4a] thread_native_entry(Thread*)+0x11a
(os_linux.cpp:740)
 
 
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/oracle/jdk2/open/src/hotspot/share/opto/multnode.cpp:77),
pid=3085184, tid=3085197
# assert((Opcode() != Op_If && Opcode() != Op_RangeCheck) || outcnt() == 2)
failed: bad if #1
#
# JRE version: Java(TM) SE Runtime Environment (21.0) (fastdebug build
21-internal-LTS-2023-05-12-0924511.tobias...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug
21-internal-LTS-2023-05-12-0924511.tobias..., mixed mode, compressed oops,
compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x1397c0c] MultiNode::proj_out(unsigned int) const+0x8c
 
Current CompileTask:
C2: 306 24 ! Test::test2 (6 bytes)
 
Stack: [0x00007f3577335000,0x00007f3577436000], sp=0x00007f3577430fd0, free
space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V [libjvm.so+0x1397c0c] MultiNode::proj_out(unsigned int) const+0x8c
(multnode.cpp:77)
V [libjvm.so+0x12635fa]
PhaseMacroExpand::expand_subtypecheck_node(SubTypeCheckNode*)+0x1fa
(macro.cpp:2275)
V [libjvm.so+0x1269c15] PhaseMacroExpand::expand_macro_nodes()+0xba5
(macro.cpp:2530)
V [libjvm.so+0x9ba5ae] Compile::Optimize()+0x18de (compile.cpp:2409)
V [libjvm.so+0x9bc4a5] Compile::Compile(ciEnv*, ciMethod*, int, Options,
DirectiveSet*)+0x1aa5 (compile.cpp:840)
V [libjvm.so+0x813664] C2Compiler::compile_method(ciEnv*, ciMethod*, int,
bool, DirectiveSet*)+0x3c4 (c2compiler.cpp:118)
V [libjvm.so+0x9c8290]
CompileBroker::invoke_compiler_on_method(CompileTask*)+0x9e0
(compileBroker.cpp:2265)
V [libjvm.so+0x9c9138] CompileBroker::compiler_thread_loop()+0x628
(compileBroker.cpp:1944)
V [libjvm.so+0xe6418e] JavaThread::thread_main_inner()+0xee
(javaThread.cpp:717)
V [libjvm.so+0x172fc98] Thread::call_run()+0xb8 (thread.cpp:216)
V [libjvm.so+0x1432a4a] thread_native_entry(Thread*)+0x11a
(os_linux.cpp:740)