C2: LCM may prioritize CheckCastPP nodes over projections

LCM should prioritize projections over other nodes when selecting among the ready list, to ensure that nodes are directly followed by their projections in local schedules.

Currently, LCM gives equal priority to projections, constants, CreateEx, and CheckCastPP nodes (see PhaseCFG::select()), effectively relying on the order in which these nodes appear in the ready list for tie-breaking.

In ARM32, this leads to the assertion failure reported below, where a CheckCastPP node is scheduled between a node and its projection. Even if this assertion failure has only showed up for ARM32, it could potentially happen for other platforms if the order of insertion in the ready list is altered.

An example subgraph for x86-64 that could potentially suffer from the same issue is attached. Depending on the node order within the ready list, the following local schedule could be produced:

461 membar_storestore
460 MachProj
504 checkCastPP
498 MachProj

This local schedule is problematic because all projections of 461 membar_storestore (460 MachProj, 498 MachProj) are not scheduled directly after 461.

ORIGINAL REPORT:

As we're part of the OpenJDK Quality Outreach program I recently started running the JaCoCo builds on a fastdebug build of OpenJDK 18-ea and observed the following crash during the execution of beanshell-maven-plugin task:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/workspace/src/hotspot/share/opto/block.cpp:1281), pid=7, tid=2051
# assert(pred == parent || (pred->is_Proj() && pred->in(0) == parent)) failed: projections must follow their parents or other sibling projections
#
# JRE version: OpenJDK Runtime Environment (18.0) (fastdebug build 18-internal+0-adhoc..workspace)
# Java VM: OpenJDK Server VM (fastdebug 18-internal+0-adhoc..workspace, mixed mode, sharing, g1 gc, linux-arm)
# Problematic frame:
# V [libjvm.so+0x22e72a] PhaseCFG::verify() const+0x4e9
#
# Core dump will be written. Default location: /workspace/core
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

The problem occures reproducible on our ARM32 build environment.
The JDK is based on commit 4fbcce119b1736455cb74d0a585097eca617593c