With the WhiteBoxAPI, one can directly "WHITE_BOX.enqueueMethodForCompilation". However, the compiler checks that we are not invoking compilation on abstract or native methods. Thus, when we "enqueueMethodForCompilation" an abstract or native method, we hit an assert (or guarantee in product).

I know that JDK24-22 are affected, did not check for older versions.

[~thartmann] suggested this:
I think we should fix this [...] by simply returning false from WB_EnqueueMethodForCompilation. We do something similar in other methods for unhandled cases, for example in WB_EnqueueInitializerForCompilation we return false if the method holder is not initialized. Or even better, handle in WhiteBox::compile_method and print a diagnostic output like we do for other unhandled cases.

I created this JTREG test:
/oracle-work/jdk-fork4/open/test/hotspot/jtreg/compiler/whitebox/TestCompileAbstractMethod.java

~/Documents/jtreg/bin/jtreg -va -s -jdk:/oracle-work/jdk-fork4/build/linux-x64-debug/jdk -javaoptions:"" -J-Djavatest.maxOutputSize=10000000 /oracle-work/jdk-fork4/open/test/hotspot/jtreg/compiler/whitebox/TestCompileAbstractMethod.java

And got this: hs_err_pid545324.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/oracle-work/jdk-fork4/open/src/hotspot/share/compiler/compileBroker.cpp:1363), pid=545324, tid=545347
# assert(!method->is_abstract() && (osr_bci == InvocationEntryBci || !method->is_native())) failed: cannot compile abstract/native methods
#
# JRE version: Java(TM) SE Runtime Environment (23.0) (fastdebug build 23-internal-2024-05-22-1054280.emanuel...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 23-internal-2024-05-22-1054280.emanuel..., mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x9f75d6] CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, DirectiveSet*, JavaThread*)+0x246
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/scratch/core.545324)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

--------------- S U M M A R Y ------------

Command Line: -Dtest.vm.opts= -Dtest.tool.vm.opts= -Dtest.compiler.opts= -Dtest.java.opts= -Dtest.jdk=/oracle-work/jdk-fork4/build/linux-x64-debug/jdk -Dcompile.jdk=/oracle-work/jdk-fork4/build/linux-x64-debug/jdk -Dtest.timeout.factor=1.0 -Dtest.root=/oracle-work/jdk-fork4/open/test/hotspot/jtreg -Dtest.name=compiler/whitebox/TestCompileAbstractMethod.java -Dtest.file=/oracle-work/jdk-fork4/open/test/hotspot/jtreg/compiler/whitebox/TestCompileAbstractMethod.java -Dtest.src=/oracle-work/jdk-fork4/open/test/hotspot/jtreg/compiler/whitebox -Dtest.src.path=/oracle-work/jdk-fork4/open/test/hotspot/jtreg/compiler/whitebox:/oracle-work/jdk-fork4/open/test/lib:/oracle-work/jdk-fork4/open/test/hotspot/jtreg -Dtest.classes=/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes/compiler/whitebox/TestCompileAbstractMethod.d -Dtest.class.path=/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes/compiler/whitebox/TestCompileAbstractMethod.d:/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes/test/lib:/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes -Dtest.class.path.prefix=/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes/compiler/whitebox/TestCompileAbstractMethod.d:/oracle-work/jdk-fork4/open/test/hotspot/jtreg/compiler/whitebox:/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes/test/lib:/oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/classes -ea -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:-BackgroundCompilation com.sun.javatest.regtest.agent.MainWrapper /oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/JTwork/compiler/whitebox/TestCompileAbstractMethod.d/main.0.jta

Host: emanuel-oracle, 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz, 16 cores, 30G, Ubuntu 20.04.6 LTS
Time: Fri Jul 19 15:58:43 2024 CEST elapsed time: 1.752858 seconds (0d 0h 0m 1s)

--------------- T H R E A D ---------------

Current thread (0x00007f93cc47a1f0): JavaThread "MainThread" [_thread_in_vm, id=545347, stack(0x00007f93a5428000,0x00007f93a5529000) (1028K)]

Stack: [0x00007f93a5428000,0x00007f93a5529000], sp=0x00007f93a5527210, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x9f75d6] CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, DirectiveSet*, JavaThread*)+0x246 (compileBroker.cpp:1363)
V [libjvm.so+0x9f7da6] CompileBroker::compile_method(methodHandle const&, int, int, methodHandle const&, int, CompileTask::CompileReason, JavaThread*)+0x96 (compileBroker.cpp:1348)
V [libjvm.so+0x18eabee] WhiteBox::compile_method(Method*, int, int, JavaThread*)+0xbe (whitebox.cpp:1093)
V [libjvm.so+0x18eb011] WB_EnqueueMethodForCompilation+0x161 (whitebox.cpp:1120)
j jdk.test.whitebox.WhiteBox.enqueueMethodForCompilation0(Ljava/lang/reflect/Executable;II)Z+0
j jdk.test.whitebox.WhiteBox.enqueueMethodForCompilation(Ljava/lang/reflect/Executable;II)Z+9
j jdk.test.whitebox.WhiteBox.enqueueMethodForCompilation(Ljava/lang/reflect/Executable;I)Z+4
j TestCompileAbstractMethod.main([Ljava/lang/String;)V+35
j java.lang.invoke.LambdaForm$DMH+0x00000000910c6000.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)V+10 java.base
j java.lang.invoke.LambdaForm$MH+0x00000000910c7c00.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+33 java.base
j java.lang.invoke.LambdaForm$MH+0x00000000910c8000.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+20 java.base
j jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+55 java.base
j jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+23 java.base
j java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+102 java.base
j com.sun.javatest.regtest.agent.MainWrapper$MainTask.run()V+134
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base
j java.lang.Thread.run()V+19 java.base
v ~StubRoutines::call_stub 0x00007f93c445fd01
V [libjvm.so+0xe69379] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x4a9 (javaCalls.cpp:415)
V [libjvm.so+0xe69a0c] JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*)+0x34c (javaCalls.cpp:329)
V [libjvm.so+0xe69c26] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x76 (javaCalls.cpp:191)
V [libjvm.so+0xfd7693] thread_entry(JavaThread*, JavaThread*)+0x93 (jvm.cpp:2937)
V [libjvm.so+0xe9f6bc] JavaThread::thread_main_inner()+0xcc (javaThread.cpp:759)
V [libjvm.so+0x17b2de6] Thread::call_run()+0xb6 (thread.cpp:225)
V [libjvm.so+0x149f1b7] thread_native_entry(Thread*)+0x127 (os_linux.cpp:846)
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j jdk.test.whitebox.WhiteBox.enqueueMethodForCompilation0(Ljava/lang/reflect/Executable;II)Z+0
j jdk.test.whitebox.WhiteBox.enqueueMethodForCompilation(Ljava/lang/reflect/Executable;II)Z+9
j jdk.test.whitebox.WhiteBox.enqueueMethodForCompilation(Ljava/lang/reflect/Executable;I)Z+4
j TestCompileAbstractMethod.main([Ljava/lang/String;)V+35
j java.lang.invoke.LambdaForm$DMH+0x00000000910c6000.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)V+10 java.base
j java.lang.invoke.LambdaForm$MH+0x00000000910c7c00.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+33 java.base
j java.lang.invoke.LambdaForm$MH+0x00000000910c8000.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+20 java.base
j jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+55 java.base
j jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+23 java.base
j java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+102 java.base
j com.sun.javatest.regtest.agent.MainWrapper$MainTask.run()V+134
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base
j java.lang.Thread.run()V+19 java.base
v ~StubRoutines::call_stub 0x00007f93c445fd01



In product, we hit this:

# guarantee(!method->is_abstract()) failed: cannot compile abstract methods