Pasting the following code into JShell will crash it:
class O { class I {} }
var i = new O().new I();
class O { static class I {} }

The direct cause is a ClassFormatError that is thrown when redefining O.I.

It seems like this bug was introduced with JDK-8282714, although I assume that the underlying problem exists for longer.

Javac sees a mixed state during compilation - it has the old classfile for O.I which has an enclosing instance, while the new O.I class is static.
This mixed state results in https://github.com/openjdk/jdk/blob/8a4ea09fa220f74f2236fc85e197eadf83b65875/src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/ClassWriter.java#L383 producing a MethodType that has one parameter for the enclosing instance, but we don't actually write a parameter for it. This results in a malformed classfile.

I wasn't able to figure out what a good solution to this problem could look like.

Full stacktrace of the crash:
Exception in thread "main" java.lang.ClassFormatError: class not in class file format
at jdk.jdi/com.sun.tools.jdi.VirtualMachineImpl.redefineClasses(VirtualMachineImpl.java:396)
at jdk.jshell/jdk.jshell.execution.JdiExecutionControl.redefine(JdiExecutionControl.java:90)
at jdk.jshell/jdk.jshell.Unit.doRedefines(Unit.java:312)
at jdk.jshell/jdk.jshell.Eval.lambda$compileAndLoad$27(Eval.java:1133)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:196)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:636)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:656)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:662)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:667)
at jdk.jshell/jdk.jshell.Eval.lambda$compileAndLoad$29(Eval.java:1134)
at jdk.jshell/jdk.jshell.TaskFactory.lambda$runTask$4(TaskFactory.java:218)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskPool.getTask(JavacTaskPool.java:193)
at jdk.jshell/jdk.jshell.TaskFactory.runTask(TaskFactory.java:211)
at jdk.jshell/jdk.jshell.TaskFactory.compile(TaskFactory.java:191)
at jdk.jshell/jdk.jshell.Eval.compileAndLoad(Eval.java:1113)
at jdk.jshell/jdk.jshell.Eval.declare(Eval.java:914)
at jdk.jshell/jdk.jshell.Eval.eval(Eval.java:141)
at jdk.jshell/jdk.jshell.JShell.eval(JShell.java:513)
at jdk.jshell/jdk.internal.jshell.tool.JShellTool.processSource(JShellTool.java:3648)
at jdk.jshell/jdk.internal.jshell.tool.JShellTool.processSourceCatchingReset(JShellTool.java:1366)
at jdk.jshell/jdk.internal.jshell.tool.JShellTool.processInput(JShellTool.java:1264)
at jdk.jshell/jdk.internal.jshell.tool.JShellTool.run(JShellTool.java:1235)
at jdk.jshell/jdk.internal.jshell.tool.JShellTool.start(JShellTool.java:1018)
at jdk.jshell/jdk.internal.jshell.tool.JShellToolBuilder.start(JShellToolBuilder.java:273)
at jdk.jshell/jdk.internal.jshell.tool.JShellToolProvider.main(JShellToolProvider.java:120)