The VM can get into a loop of uncommon traps without recompilation.

This causes the VM to thrash between compiled code and the interpreter on each method call.

This shows up as a pattern in LogCompilation output where uncommon traps continue forever, with no decompilation or recompilation to address them. Look for action='none' in the log. For example:

<uncommon_trap thread='51' reason='unreached' action='none' compile_id='6992' compiler='C2' stamp='2426.928'>
<jvms bci='55' method='FooItem addInfoHelper (Ljava/lang/Object;Z)V' bytes='224' count='325' iicount='1427' decompiles='2' class_check_traps='28' overflow_recompiles='24'/>
<jvms bci='23' method='FooItem addInfoHelper (Ljava/lang/Object;)V' bytes='27' count='292' iicount='1429' decompiles='2' class_check_traps='27' overflow_recompiles='23'/>
<jvms bci='203' method='FooItem addInfo (Ljava/lang/Object;)V' bytes='309' count='326' iicount='1425' decompiles='2'/>
</uncommon_trap>

In this case, the addInfoHelper routines have gone through an early episode of recompiles (a couple dozen) due to an unrelated uncommon trap (it was a class_check in this case). When the enclosing method addInfo inlines the recompilation-weary addInfoHelper, the code in GraphKit::uncommon_trap sees that too_many_recompiles is true, and substitutes the evil Action_none. After that, the method gets slow and stays slow.

Note that the top-level method only needs to be recompiled once, before it gets into the final bad state. All the excessive recompilation that triggers this is in the helper methods.

There are several problems here. (Not sure what the priority is for handling them.)

1. Something is causing addInfoHelper to get a cycle of class_check recompiles, which makes it compilation-weary. I suspect it may be the monomorphic call handling in doCall.cpp, where we've seen bugs like this (6225440, 6419652).

2. The excessive recompilations of addInfoHelper (both of them) are causing the top-level addInfo method to emit an Action_none. Probably this extremely desperate bail-out should only be attempted if the top-level method itself has been recompiling too much.

3. When GraphKit::uncommon_trap tests for too many recompiles, it does not try to determine if the current trap reason has anything to do with the recompilation loop. If the reason is 'unreached', it is certain that trapping to the interpreter will update the MDO so that the next compilation will *not* see an unreached state. Thus, a loop is impossible. (There are only so many bytecodes that will ever be unreached, and each can only be trapped as unreached once.) Perhaps if the current reason has never happened, then the code should not try to defend against a recompilation loop.

4. The compiler probably relies too much on the interpreter to update the MDO. If the MDO update fails, the compiler will get caught in a Groundhog Day type of loop, and keep making the same bad compilation decision, until the recompilation limit causes a bad bailout. (That was 6225440.) It is probably safer, but more laborious, to have the compiler update the MDO directly, before it calls the uncommon trap stub. At the very least, we should make the uncommon trap runtime routine (deoptimization.cpp) update the MDO in C code. Perhaps this update should be coded in opto, so we can better coordinate the MDO update logic with the compilation logic that is relying on it.

5. The Action_none bailout is dangerous. GraphKit::uncommon_trap should bail out to Action_make_not_compilable. That way the log will print an interesting failure event, and performance will degrade into the interpreter, which is faster than the deoptimizer.