We have scrip we use to test jaotc which compiles graal module jdk.internal.vm.compiler.
In JDK 10 it run very long time and at the end OutOfMemory (even with 20g heap) when compiling next method:
org.graalvm.compiler.phases.common.DeoptimizationGroupingPhase.run(Lorg/graalvm/compiler/nodes/StructuredGraph;Lorg/graalvm/compiler/phases/tiers/MidTierContext;)V

to reproduce:

1. Build fastdebug JDK from http://hg.openjdk.java.net/jdk10/hs sources.
2. Run jaotc from that JDK:
$ cat > compile.txt
compileOnly org.graalvm.compiler.phases.common.DeoptimizationGroupingPhase.run(Lorg/graalvm/compiler/nodes/StructuredGraph;Lorg/graalvm/compiler/phases/tiers/MidTierContext;)V

$ JAVA_HOME/bin/jaotc -J-Xms2g -J-Xmx3g -J-XX:+UseParallelGC -J-XX:-UseCompressedOops -J-XX:ParallelGCThreads=4 -J-XX:CICompilerCount=3 -J-Dgraal.PrintCompilation=true --threads 1 --verbose --compile-commands compile.txt --module jdk.internal.vm.compiler
Compiling unnamed.so...
3124 classes found (908 ms)
.
.
.
20278 methods total, 1 methods to compile (554 ms)
Compiling with 1 threads
1 JVMCI Lorg/graalvm/compiler/phases/common/DeoptimizationGroupingPhase; run (Lorg/graalvm/compiler/nodes/StructuredGraph;Lorg/graalvm/compiler/phases/tiers/MidTierContext;)V ...

...Error: Failed compilation: org.graalvm.compiler.phases.common.DeoptimizationGroupingPhase.run(Lorg/graalvm/compiler/nodes/StructuredGraph;Lorg/graalvm/compiler/phases/tiers/MidTierContext;)V: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.LinkedList.linkLast(LinkedList.java:146)
    at java.base/java.util.LinkedList.add(LinkedList.java:342)
    at jdk.internal.vm.compiler/org.graalvm.compiler.loop.LoopEx.nodesInLoopBranch(LoopEx.java:313)
    at jdk.internal.vm.compiler/org.graalvm.compiler.loop.DefaultLoopPolicies.shouldUnswitch(DefaultLoopPolicies.java:127)
    at jdk.internal.vm.compiler/org.graalvm.compiler.loop.phases.LoopUnswitchingPhase.run(LoopUnswitchingPhase.java:59)
    at jdk.internal.vm.compiler/org.graalvm.compiler.loop.phases.ContextlessLoopPhase.run(ContextlessLoopPhase.java:47)
    at jdk.internal.vm.compiler/org.graalvm.compiler.loop.phases.ContextlessLoopPhase.run(ContextlessLoopPhase.java:29)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:206)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:135)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.PhaseSuite.run(PhaseSuite.java:179)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:206)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:135)
    at jdk.internal.vm.compiler/org.graalvm.compiler.core.GraalCompiler.emitFrontEnd(GraalCompiler.java:231)
    at jdk.internal.vm.compiler/org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:174)
    at jdk.internal.vm.compiler/org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:160)
    at jdk.aot/jdk.tools.jaotc.AOTBackend.compileGraph(AOTBackend.java:130)
    at jdk.aot/jdk.tools.jaotc.AOTBackend.compileMethod(AOTBackend.java:99)
    at jdk.aot/jdk.tools.jaotc.AOTCompilationTask.run(AOTCompilationTask.java:121)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)