Discovered this originally when testing Persistent Profiles PR in Leyden. In short, the transition from level=0 would not go to level=4 ever for trivial methods, if we disable intermediate levels. I believe this is one the causes for at least some performance artifacts I seen in Leyden testing last week.

Reproducer:

$ build/linux-x86_64-server-release/images/jdk/bin/java -XX:+PrintTieredEvents -XX:+UseParallelGC -Xmn7g -Xms8g -Xmx8g -XX:+AlwaysPreTouch -XX:-TieredCompilation -cp JavacBenchApp.jar -XX:AOTCache=app.aot JavacBenchApp 1000 | grep com.sun.tools.javac.code.Type.isPrimitive | nl
...
 20672 43.784202: [call level=0 [com.sun.tools.javac.code.Type.isPrimitive()Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2617088,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=0(0), 0(0), deps=0]
 20673 43.784681: [call level=0 [com.sun.tools.javac.code.Type.isPrimitive()Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2617216,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=0(0), 0(0), deps=0]
 20674 43.786087: [call level=0 [com.sun.tools.javac.code.Type.isPrimitive()Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2617344,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=0(0), 0(0), deps=0]
 20675 43.786708: [call level=0 [com.sun.tools.javac.code.Type.isPrimitive()Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2617472,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=0(0), 0(0), deps=0]
 20676 43.787414: [call level=0 [com.sun.tools.javac.code.Type.isPrimitive()Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2617600,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=0(0), 0(0), deps=0]

Note there are 20K+ events for this method, and counters are in millions already. There is no MDO. Yet, no compilation happens. That seems to be because we have lost a short-cut for trivial methods. Once the shortcut is reinstated, we run into another bug:

$ build/linux-x86_64-server-release/images/jdk//bin/java -XX:+PrintTieredEvents -XX:+UseParallelGC -Xmn7g -Xms8g -Xmx8g -XX:+AlwaysPreTouch -XX:-TieredCompilation -cp JavacBenchApp.jar -XX:AOTCache=app.aot JavacBenchApp 1000 | grep "java.util.regex.Pattern\$Slice.match" | nl
...
 21471 38.216459: [call level=0 [java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2748288,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=5053(0), 0(0), deps=null]
 21472 38.216671: [call level=0 [java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2748416,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=5053(0), 0(0), deps=null]
 21473 38.218676: [call level=0 [java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2748544,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=5053(0), 0(0), deps=null]
 21474 38.218903: [call level=0 [java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2748672,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=5053(0), 0(0), deps=null]
 21475 38.219143: [call level=0 [java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2748800,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=5053(0), 0(0), deps=null]
 21476 38.234896: [call level=0 [java.util.regex.Pattern$Slice.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z] @-1 queues=0,0 rate=n/a k=1.00,1.00 total=2748928,0 mdo=0(0),0(0) max levels=0,0 compilable=c1,c1-osr,c2,c2-osr status=idle mtd: mdo=5053(0), 0(0), deps=null]


In here, the method has MTD/MDO, but the MCS counters are updated! So compilation policy that only looks at MDO has no idea it needs to transit method further. This second bug is a superset of the first bug.

This only happens with -XX:AOTCache=app.aot, so this is a Leyden bug.