[ubsan]: The assert in layout_helper_boolean_diffbit() in klass.hpp needs UB to fail

XMLWordPrintable

    • Type: Enhancement
    • Resolution: Unresolved
    • Priority: P4
    • tbd
    • Affects Version/s: None
    • Component/s: hotspot
    • In Review

      In layout_helper_boolean_diffbit(), inside the loop that is calculating the result, there is an assert that diffbit (the in-progress calculated result) is not zero. But it can never be zero in the absence of UB. It is initialized to 1, and then left shifted 1 for each iteration. That bit could be shifted all the way to the sign bit, so that the value is INT_MIN. But another left shift, which naively would result in zero, is actually UB.

            Assignee:
            Afshin Zafari
            Reporter:
            Afshin Zafari
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: