JDK-8351833 added an assert checking that single IGVN iterations do not add an excessive number of new nodes. The assert initially turned out to be slightly too optimistic, expecting an increase of at most 4000 nodes per iterations (by default, but also depending on NodeLimitFudgeFactor). We saw this limit surpassed in JDK-8354767, and decided to increase the limit to 6000 (by default). However, it is still unclear if the violating transformations (with >4000, but <6000 new nodes) are actually reasonable.

The issue in JDK-8354767 is highly intermittent (1% of runs), but reproduces with java/lang/invoke/VarHandles/VarHandleTestByteArrayAsLong.java on debug builds on all major platforms when reverting the changeset for JDK-8354767.

Consider the following piece of code in PhaseIterGVN::optimize, tracking the live node increase after the changeset for JDK-8351833.

DEBUG_ONLY(int live_nodes_before = C->live_nodes();)
Node* nn = transform_old(n);
DEBUG_ONLY(int live_nodes_after = C->live_nodes();)

Printing 'n' and 'nn' in violating iterations, we get the below.

19517 Phi === 1444 17794 19566 19568 19570 14402 19571 19573 19575 19576 19580 19584 19585 19591 19597 19598 19606 19617 19720 19704 19826 19703 19837 19618 19851 19701 19956 19700 19970 19698 19987 19697 20004 19695 20024 19694 20044 19692 20067 19691 20090 19689 20116 19688 20142 19686 20171 19685 20200 19683 20232 19682 20264 20301 19620 17860 20405 19680 20441 20480 19621 19678 20583 19677 20622 20666 20768 17855 20869 19675 20912 19623 20958 19673 21058 19672 21104 19624 17850 17850 21155 19670 21206 19668 21260 19667 21314 17845 21373 19665 21431 19663 21492 19662 21553 21615 19660 21678 19659 21742 21807 19657 21873 19656 21940 22008 19654 22077 19653 22147 22218 19651 22290 19650 22363 22437 19648 22512 19647 22588 22665 19645 22743 19644 22822 22902 19642 22983 19641 23065 23148 19639 23232 19638 23317 23403 19636 23490 19635 23578 23667 19633 23757 19632 23848 23940 19630 24033 19629 24127 24222 19627 24318 19626 24415 24513 [[ 19529 ]] #memory Memory: @java/lang/Long (java/io/Serializable,java/lang/Comparable,java/lang/constant/Constable,java/lang/constant/ConstantDesc):NotNull:exact+16 *,iid=4638, name=value, idx=32; !orig=19506,[2618],[39511],[764] !jvms: VarHandleTestByteArrayAsLong::testArrayReadWrite @ bci:83 (line 1059)

19517 Phi === 1444 17794 19566 19568 19570 14402 19571 19573 19575 19576 19580 19584 19585 19591 19597 19598 19606 19617 19720 19704 19826 19703 19837 19618 19851 19701 19956 19700 19970 19698 19987 19697 20004 19695 20024 19694 20044 19692 20067 19691 20090 19689 20116 19688 20142 19686 20171 19685 20200 19683 20232 19682 20264 20301 19620 17860 20405 19680 20441 20480 19621 19678 20583 19677 20622 20666 20768 17855 20869 19675 20912 19623 20958 19673 21058 19672 21104 19624 17850 17850 21155 19670 21206 19668 21260 19667 21314 17845 21373 19665 21431 19663 21492 19662 21553 21615 19660 21678 19659 21742 21807 19657 21873 19656 21940 22008 19654 22077 19653 22147 22218 19651 22290 19650 22363 22437 19648 22512 19647 22588 22665 19645 22743 19644 22822 22902 19642 22983 19641 23065 23148 19639 23232 19638 23317 23403 19636 23490 19635 23578 23667 19633 23757 19632 23848 23940 19630 24033 19629 24127 24222 19627 24318 19626 24415 24513 [[ 19529 ]] #memory Memory: @java/lang/Long (java/io/Serializable,java/lang/Comparable,java/lang/constant/Constable,java/lang/constant/ConstantDesc):NotNull:exact+16 *,iid=4638, name=value, idx=32; !orig=19506,[2618],[39511],[764] !jvms: VarHandleTestByteArrayAsLong::testArrayReadWrite @ bci:83 (line 1059)

That is, a (locally unchanged) large Phi node where PhiNode::Ideal added 4k new nodes somewhere further up the inputs. Speculating wildly, a guess is that the new nodes are added as part of the call to MemNode::optimize_memory_chain in PhiNode::Ideal.

We should investigate if the above is sane and profitable. If neither, we should determine if we can catch such cases before creating all the new nodes and instead bail out from compilation.