We use CTW testing for making sure compilers behave well. But we compile the code that is not executed at all, and since our inlining heuristics often looks back at profiles, we end up not actually inlining all too much! This means CTW testing likely misses lots of bugs that normal code is exposed to, especially e.g. in loop optimizations.

See for example:

$ test/hotspot/jtreg/testlibrary/ctw/dist $ JAVA_OPTIONS="-XX:+PrintCompilation -XX:+PrintInlining -XX:-TieredCompilation -XX:CICompilerCount=1 -XX:+CITime" ./ctw.sh modules:jdk.compiler

65439 12668 b com.sun.tools.javac.tree.TreeMaker::Literal (362 bytes)
                            @ 14 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 25 com.sun.tools.javac.code.Type::constType (8 bytes) failed to inline: virtual call
                            @ 28 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 47 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 58 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 61 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 80 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 91 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 94 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 113 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 124 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 127 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 149 java.lang.Character::toString (8 bytes) failed to inline: never executed
                            @ 153 java.lang.String::charAt (25 bytes) failed to inline: already compiled into a medium method
                            @ 164 java.lang.Integer::valueOf (32 bytes) failed to inline: call site not reached
                            @ 167 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 179 java.lang.Integer::valueOf (32 bytes) failed to inline: call site not reached
                            @ 182 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 185 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 204 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 215 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 218 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 237 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 248 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 251 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 270 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 281 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 284 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency
                            @ 304 java.lang.Boolean::booleanValue (5 bytes) accessor
                            @ 323 java.lang.Integer::valueOf (32 bytes) failed to inline: call site not reached
                            @ 326 com.sun.tools.javac.tree.TreeMaker::Literal (20 bytes) failed to inline: low call site frequency
                            @ 338 java.lang.Integer::valueOf (32 bytes) failed to inline: call site not reached
                            @ 341 com.sun.tools.javac.code.Type$JCPrimitiveType::constType (13 bytes) failed to inline: low call site frequency
                            @ 344 com.sun.tools.javac.tree.JCTree$JCLiteral::setType (8 bytes) failed to inline: low call site frequency


AFAICS, we only inline trivial methods, accessors and some code that happens to be hot since JDK itself touches it. We should configure CTW to inline quite a bit more.

There is an intrinsic tradeoff in this. If we improve inlining, we will likely reach more interesting cases, but it would also take more time to compile. Remains to be seen where is the sweet spot in this.