C2 compiled checkcast of non-null object triggers endless deoptimization/recompilation cycle

When C2 determines at compile time that there is no subtype relation between the objects involved in a checkcast, it inserts a null assert because the checkcast can only pass if the value is null. This will trigger deoptimization and recompilation if the object is non-null:

   1151 636 ! 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes) made not entrant
   1151 638 !b 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)
   1152 638 ! 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes) made not entrant
   1152 639 !b 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)
   1152 639 ! 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes) made not entrant
   1153 640 !b 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)

[...]

   1475 1030 ! 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes) made not entrant
   1475 1031 !b 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)
made not compilable on level 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes) decompile_count > PerMethodRecompilationCutoff
   1476 1031 ! 4 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes) made not entrant
   1476 1032 !b 1 compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)

However, we can do better here if we already know that the object is non-null (i.e. the check always fails) or if we hit too_many_traps_or_recompiles (which suggests that the object is often null) to avoid an endless deoptimization/recompilation cycle until we hit the PerMethodRecompilationCutoff and starve at lower tiers.

In the former case we can simply throw right away and in the latter case we should fall back to a full check in case there is an exception handler in compiled code that handles the failing (non-null) case. That way we avoid deoptimization.

I've spotted this with our test framework for inline types in project Valhalla, while investigating an intermittent failure where several test methods suddenly became non-compilable at tier 4.