Context:
We encountered sporadic signal 11 crashes without any stack trace or hs-err log file. With the patch of -XX:+UseSigAltStack (a feature proposed for JDK-8364654), we managed to get the stack trace of these crashes. The stack trace contains over 1000 frames of `Parse::jump_switch_ranges` from C2, causing stack overflow in the compiler thread. We found that C2 was compiling a method with a huge switch statement, with a size > 38KiB.

The application runs with -XX:-TieredCompilation. The JVM should not compile this large method to begin with, due to -XX:+DontCompileHugeMethods and HugeMethodLimit defaulting to 8000.

The bug is in CompilationPolicy::compile() for the logic below:

  if (!CompilationModeFlag::disable_intermediate()) {
    if ((bci == InvocationEntryBci && !can_be_compiled(mh, level))) {
      ...
      return;
    }
    if ((bci != InvocationEntryBci && !can_be_osr_compiled(mh, level))) {
      ...
      return;
    }
  }

The can_be_compiled() and can_be_osr_compiled() are the places that check for DontCompileHugeMethods, but they are guarded by !CompilationModeFlag::disable_intermediate(). disable_intermediate() is true under -XX:-TieredCompilation.

This bug was introduced by JDK-8251462 in JDK 17. Behavior differences for -XX:-TieredCompilation:

- before JDK-8251462, in compilationPolicy.cpp, can_be_compiled() and can_be_osr_compiled() checks are called from SimpleCompPolicy::method_invocation_event() and SimpleCompPolicy::method_back_branch_event(). See attached "before.png".

- after JDK-8251462, SimpleCompPolicy was removed, and -XX:-TieredCompilation also uses CompilationPolicy::method_invocation_event() and CompilationPolicy::method_back_branch_event(), which lacks the can_be_compiled() and can_be_osr_compiled() checks (see after.png). JDK-8251462 moved most of TieredThresholdPolicy::compile() into CompilationPolicy::compile(), which is where the CompilationModeFlag::disable_intermediate() checks came from.

Besides not respecting -XX:+DontCompileHugeMethods, it is possible that other checks in can_be_compiled() and can_be_osr_compiled() are not respected under -XX:-TieredCompilation, e.g. check for AbstractInterpreter::can_be_compiled().