So, I debug a small example program doing a lot of reflection in IntelliJ 2018.1.5 CE.

Program link: https://github.com/tstuefe/repros/blob/master/src/test3/ReflectionStress.java

I start the program with a current-tip fastdebug build and arguments "100 100" (100 classes generated with each 100*3 methods, which are then invoked via core reflection).

When I stop the program during the invocation phase (After "Loading 100 classes..."), I usually break at Unsafe.defineClass0, where a Generated{Method|Constructor}Accessor is just being generated.

When I then step out, the VM will crash when attempting to leave the MethodAccessorGenerator::generate() method:

/shared/projects/openjdk/jdk-jdk/output-fastdebug/images/jdk/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:42909,suspend=y,server=n -Dfile.encoding=UTF-8 -classpath /shared/projects/openjdk/test/repros/out/production/repros:/shared/projects/idea/idea-IC-181.5281.24/lib/idea_rt.jar test3.ReflectionStress
Connected to the target VM, address: '127.0.0.1:42909', transport: 'socket'
classes loaded: 0
Loaded 100 classes...Invoking...# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=/jvmtiThreadState.cpp:277
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/prims/jvmtiThreadState.cpp:277), pid=17656, tid=17661
# assert(_cur_stack_depth == count_frames()) failed: cur_stack_depth out of sync
#
# JRE version: OpenJDK Runtime Environment (11.0) (fastdebug build 11-internal+0-adhoc.thomas.source)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 11-internal+0-adhoc.thomas.source, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /shared/projects/openjdk/test/repros/core.17656)
#
# An error report file with more information is saved as:
# /shared/projects/openjdk/test/repros/hs_err_pid17656.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Current thread is 17661

 24 Current thread (0x00007f6c6001c000): JavaThread "main" [_thread_in_vm, id=15202, stack(0x00007f6c684a3000,0x00007f6c685a4000)]
  25
  26 Stack: [0x00007f6c684a3000,0x00007f6c685a4000], sp=0x00007f6c6859f6c0, free space=1009k
  27 Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
  28 V [libjvm.so+0x18c1d92] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x532
  29 V [libjvm.so+0x18c29ff] VMError::report_and_die(Thread*, void*, char const*, int, char const*, char const*, __va_list_tag*)+0x2f
  30 V [libjvm.so+0xb276ea] report_vm_error(char const*, int, char const*, char const*, ...)+0x12a
  31 V [libjvm.so+0x120e340] JvmtiThreadState::cur_stack_depth()+0xd0
  32 V [libjvm.so+0x11d18e9] JvmtiExport::post_method_exit(JavaThread*, Method*, frame)+0x179
  33 V [libjvm.so+0xed1e40] InterpreterRuntime::post_method_exit(JavaThread*)+0xd0
  34 j java.lang.Object.<init>()V+0 java.base@11-internal
  35 j java.lang.AbstractStringBuilder.<init>(I)V+1 java.base@11-internal
  36 j java.lang.StringBuilder.<init>()V+3 java.base@11-internal
  37 j java.lang.Class.toString()Ljava/lang/String;+4 java.base@11-internal
  38 v ~StubRoutines::call_stub
  39 V [libjvm.so+0xef7f22] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x8a2
  40 V [libjvm.so+0xfd708a] jni_invoke_nonstatic(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .constprop.223]+0x24a
  41 V [libjvm.so+0x100a206] jni_CallObjectMethodA+0x1d6
  42 C [libjdwp.so+0x1d3d1] invoker_doInvoke+0x961
  43 C [libjdwp.so+0x154bc] reportEvents.part.2+0x18c
  44 C [libjdwp.so+0x157d0] event_callback+0x1f0
  45 C [libjdwp.so+0x18b7f] cbSingleStep+0xcf
  46 V [libjvm.so+0x11d22fd] JvmtiExport::post_single_step(JavaThread*, Method*, unsigned char*)+0x27d
  47 V [libjvm.so+0x11d281c] JvmtiExport::at_single_stepping_point(JavaThread*, Method*, unsigned char*)+0xac
  48 V [libjvm.so+0xed0d1f] InterpreterRuntime::at_safepoint(JavaThread*)+0xdf
  49 j jdk.internal.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Ljdk/internal/reflect/MagicAccessorImpl;+758 java.base@11-internal
  50 J 5083 c1 jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(Ljava/lang/reflect/Constructor;)Ljdk/internal/reflect/ConstructorAccessor; java.base@11-internal (154 bytes) @ 0x00007f6c49cf49a4 [0x00007f6c49cf3200+0x00000000000017a4]
  51 J 5096 c1 java.lang.reflect.Constructor.acquireConstructorAccessor()Ljdk/internal/reflect/ConstructorAccessor; java.base@11-internal (44 bytes) @ 0x00007f6c49cfba24 [0x00007f6c49cfb7e0+0x0000000000000244]
  52 J 5066 c1 java.lang.Class.newInstance()Ljava/lang/Object; java.base@11-internal (160 bytes) @ 0x00007f6c49cdf3bc [0x00007f6c49cdde80+0x000000000000153c]
  53 J 5113 c1 jdk.internal.reflect.MethodAccessorGenerator$1.run()Ljdk/internal/reflect/MagicAccessorImpl; java.base@11-internal (41 bytes) @ 0x00007f6c49d6b05c [0x00007f6c49d6a180+0x0000000000000edc]
  54 J 5112 c1 jdk.internal.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; java.base@11-internal (5 bytes) @ 0x00007f6c49d69adc [0x00007f6c49d69a60+0x000000000000007c]
  55 v ~StubRoutines::call_stub
  56 V [libjvm.so+0xef7f22] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x8a2
  57 V [libjvm.so+0x1075fc6] JVM_DoPrivileged+0x846
  58 J 656 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; java.base@11-internal (0 bytes) @ 0x00007f6c502c8459 [0x00007f6c502c8300+0x0000000000000159]
  59 J 5101 c1 jdk.internal.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Ljdk/internal/reflect/MagicAccessorImpl; java.base@11-internal (762 bytes) @ 0x00007f6c49d53144 [0x00007f6c49d4a240+0x0000000000008f04]
  60 J 5156 c1 jdk.internal.reflect.ReflectionFactory.newMethodAccessor(Ljava/lang/reflect/Method;)Ljdk/internal/reflect/MethodAccessor; java.base@11-internal (119 bytes) @ 0x00007f6c49e02fac [0x00007f6c49e01440+0x0000000000001b6c]
  61 J 5186% c2 test3.ReflectionStress.main([Ljava/lang/String;)V (350 bytes) @ 0x00007f6c5080c574 [0x00007f6c5080c2e0+0x0000000000000294]
  62 v ~StubRoutines::call_stub
  63 V [libjvm.so+0xef7f22] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x8a2
  64 V [libjvm.so+0xfd5a6f] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.90] [clone .constprop.234]+0x26f
  65 V [libjvm.so+0xffb48d] jni_CallStaticVoidMethod+0x22d
  66 C [libjli.so+0x4cde] JavaMain+0xa9e
  67
  68 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
  69 j java.lang.Object.<init>()V+0 java.base@11-internal
  70 j java.lang.AbstractStringBuilder.<init>(I)V+1 java.base@11-internal
  71 j java.lang.StringBuilder.<init>()V+3 java.base@11-internal
  72 j java.lang.Class.toString()Ljava/lang/String;+4 java.base@11-internal
  73 v ~StubRoutines::call_stub
  74 j jdk.internal.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Ljdk/internal/reflect/MagicAccessorImpl;+758 java.base@11-internal
  75 J 5083 c1 jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(Ljava/lang/reflect/Constructor;)Ljdk/internal/reflect/ConstructorAccessor; java.base@11-internal (154 bytes) @ 0x00007f6c49cf49a4 [0x00007f6c49cf3200+0x00000000000017a4]
  76 J 5096 c1 java.lang.reflect.Constructor.acquireConstructorAccessor()Ljdk/internal/reflect/ConstructorAccessor; java.base@11-internal (44 bytes) @ 0x00007f6c49cfba24 [0x00007f6c49cfb7e0+0x0000000000000244]
  77 J 5066 c1 java.lang.Class.newInstance()Ljava/lang/Object; java.base@11-internal (160 bytes) @ 0x00007f6c49cdf3bc [0x00007f6c49cdde80+0x000000000000153c]
  78 J 5113 c1 jdk.internal.reflect.MethodAccessorGenerator$1.run()Ljdk/internal/reflect/MagicAccessorImpl; java.base@11-internal (41 bytes) @ 0x00007f6c49d6b05c [0x00007f6c49d6a180+0x0000000000000edc]
  79 J 5112 c1 jdk.internal.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; java.base@11-internal (5 bytes) @ 0x00007f6c49d69adc [0x00007f6c49d69a60+0x000000000000007c]
  80 v ~StubRoutines::call_stub
  81 J 656 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; java.base@11-internal (0 bytes) @ 0x00007f6c502c83e6 [0x00007f6c502c8300+0x00000000000000e6]
  82 J 5101 c1 jdk.internal.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Ljdk/internal/reflect/MagicAccessorImpl; java.base@11-internal (762 bytes) @ 0x00007f6c49d53144 [0x00007f6c49d4a240+0x0000000000008f04]
  83 J 5156 c1 jdk.internal.reflect.ReflectionFactory.newMethodAccessor(Ljava/lang/reflect/Method;)Ljdk/internal/reflect/MethodAccessor; java.base@11-internal (119 bytes) @ 0x00007f6c49e02fac [0x00007f6c49e01440+0x0000000000001b6c]
  84 J 5186% c2 test3.ReflectionStress.main([Ljava/lang/String;)V (350 bytes) @ 0x00007f6c5080c574 [0x00007f6c5080c2e0+0x0000000000000294]
  85 v ~StubRoutines::call_stub

---

Note that I ran the VM with -Dsun.reflect.noInflation=true, but that is not necessary to reproduce the problem.