An out-of-spec classfile can trigger an OOME in the ClassFileAPi, as demonstrated with the javap tool.

javap -J-Xmx8G -c OOME_ClassFileAPI.class
public class org.openjdk.jmh.samples.JMHSample_35_Profilers$Classy$XLoader extends java.net.URLClassLoader {
  public org.openjdk.jmh.samples.JMHSample_35_Profilers$Classy$XLoader();
    Code:
       0: aload_0
       1: iconst_0
       2: anewarray #1 // class java/net/URL
       5: invokestatic #3 // Method java/lang/ClassLoader.getSystemClassLoader:()Ljava/lang/ClassLoader;
       8: invokespecial #9 // Method java/net/URLClassLoader."<init>":([LCava/net/URL;Ljava/lang/ClassLoader;)V
      11: return

  protected java.lang.Class<?> findClass(java.lang.String) throws java.lang.ClassNotFoundException;
    Code:
       0: aload_0
       1: aload_1
       2: getstatic #15 // Field X_BYTECODE:[B
       5: iconst_0
       6: getstatic #15 // Field X_BYTECODE:[B
       9: arraylength
      10: invokevirtual #21 // Method defineClass:(Ljava/lang/String;[BII)Ljava/lang/Class;
      13: areturn

  static {};
    Code:
Error: Insufficient memory. To increase memory use -J-Xmx option.

When forcing a crash log with -XX:+CrashOnOutOfMemory the following native stack trace shows that an allocation in jdk.internal.classfile.impl.CodeImpl.bcToInstruction causes an oversized memory allocation.

V [libjvm.dylib+0x3a29e3] report_java_out_of_memory(char const*)+0xc3
V [libjvm.dylib+0x8870c3] MemAllocator::Allocation::check_out_of_memory()+0x63
V [libjvm.dylib+0x887b85] MemAllocator::allocate() const+0xd5
V [libjvm.dylib+0x5608d1] InstanceKlass::allocate_instance(JavaThread*)+0x41
V [libjvm.dylib+0x9a0227] OptoRuntime::new_instance_C(Klass*, JavaThread*)+0xf7
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v ~RuntimeStub::_new_instance_Java 0x0000000115ab72a5
J 802 c2 jdk.internal.classfile.impl.CodeImpl.bcToInstruction(II)Ljava/lang/classfile/Instruction; java.base@22.0.1 (1940 bytes) @ 0x000000011601371c [0x0000000116012e40+0x00000000000008dc]
J 814% c2 jdk.internal.classfile.impl.CodeImpl.forEachElement(Ljava/util/function/Consumer;)V java.base@22.0.1 (252 bytes) @ 0x0000000116014738 [0x00000001160144a0+0x0000000000000298]
j java.lang.classfile.CompoundElement.elementList()Ljava/util/List;+18 java.base@22.0.1
j java.lang.classfile.CompoundElement.elements()Ljava/lang/Iterable;+1 java.base@22.0.1
j java.lang.classfile.CompoundElement.iterator()Ljava/util/Iterator;+1 java.base@22.0.1
j com.sun.tools.javap.CodeWriter.writeInstrs(Ljava/lang/classfile/attribute/CodeAttribute;)V+9 jdk.jdeps@22.0.1
j com.sun.tools.javap.ClassWriter.writeMethod(Ljava/lang/classfile/MethodModel;)V+892 jdk.jdeps@22.0.1
j com.sun.tools.javap.ClassWriter.writeMethods()V+36 jdk.jdeps@22.0.1
j com.sun.tools.javap.ClassWriter.write(Ljava/lang/classfile/ClassModel;)Z+1135 jdk.jdeps@22.0.1
j com.sun.tools.javap.JavapTask.write(Lcom/sun/tools/javap/JavapTask$ClassFileInfo;)Z+78 jdk.jdeps@22.0.1
j com.sun.tools.javap.JavapTask.writeClass(Lcom/sun/tools/javap/ClassWriter;Ljava/lang/String;)I+115 jdk.jdeps@22.0.1
j com.sun.tools.javap.JavapTask.run()I+184 jdk.jdeps@22.0.1