Profiling a simple scenario:

$ ~/Install/jdk9u20/bin/java -jar dist/nashorn.jar -Dnashorn.typeInfo.disabled=false --class-cache-size=0 --persistent-code-cache=false -scripting --log=time test/script/basic/compile-octane.js -- --iterations 5

...yields a few simple low-hanging fruits, here is one of them. Out of 12 seconds spent in class installation time, at least 2 seconds are spent here:

12.690 jdk.nashorn.internal.codegen.CompilationPhase$12.transform(jdk.nashorn.internal.codegen.Compiler, jdk.nashorn.internal.codegen.Compiler$CompilationPhases, jdk.nashorn.internal.ir.FunctionNode)
2.110 jdk.nashorn.internal.codegen.CompilationPhase.access$100(jdk.nashorn.internal.ir.FunctionNode, jdk.nashorn.internal.ir.FunctionNode$CompilationState)
2.110 jdk.nashorn.internal.codegen.CompilationPhase.setStates(jdk.nashorn.internal.ir.FunctionNode, jdk.nashorn.internal.ir.FunctionNode$CompilationState)
2.100 jdk.nashorn.internal.ir.FunctionNode.accept(jdk.nashorn.internal.ir.visitor.NodeVisitor)

Attila says setStates should not be hot, since it is apparently the internal invariant-checking thing. The dumbest patch of all:

diff -r 494092ee7a01 src/jdk.scripting.nashorn/share/classes/jdk/nashorn/internal/codegen/CompilationPhase.java
--- a/src/jdk.scripting.nashorn/share/classes/jdk/nashorn/internal/codegen/CompilationPhase.java Mon Aug 25 22:36:05 2014 +0200
+++ b/src/jdk.scripting.nashorn/share/classes/jdk/nashorn/internal/codegen/CompilationPhase.java Tue Aug 26 12:24:39 2014 +0400
@@ -567,7 +567,7 @@
                 log.fine(sb.toString());
             }
 
- return setStates(fn.setRootClass(null, rootClass), BYTECODE_INSTALLED);
+ return fn.setRootClass(null, rootClass);
         }
 
         @Override

...on the current jdk9/dev and this simple benchmark:

$ for I in `seq 1 5`; do ~/trunks/jdk9-dev/build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/java -jar ~/trunks/jdk9-dev/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/ext/nashorn.jar -Dnashorn.typeInfo.disabled=false --class-cache-size=0 --persistent-code-cache=false -scripting --log=time test/script/basic/compile-octane.js -- --iterations 1 2>&1 | grep "Class Installation"; done

...yields a big improvement.

Before (setStates is called):
 [time] 'Class Installation' 2926 ms
 [time] 'Class Installation' 3426 ms
 [time] 'Class Installation' 3079 ms
 [time] 'Class Installation' 2844 ms
 [time] 'Class Installation' 2893 ms

After (setStates is not called):
 [time] 'Class Installation' 1668 ms
 [time] 'Class Installation' 1680 ms
 [time] 'Class Installation' 1674 ms
 [time] 'Class Installation' 1681 ms
 [time] 'Class Installation' 1720 ms

This is with JDK-8055954 applied, so may be the relative improvement without JDK-8055954 is worse.