Incorrect overflow test in Toom-Cook branch of BigInteger multiplication

XMLWordPrintable

    • b13
    • generic
    • generic
    • Verified

        In this code

        if (bitLength(mag, mag.length) +
           bitLength(val.mag, val.mag.length) >
           32L*MAX_MAG_LENGTH) {
           reportOverflow();
        }

        In the above, bitLength() returns 'int', hence the left operand of the '>' comparison is 'int' too, but the right operand of the '>' comparison is 'long' with value == Integer.MAX_VALUE + 1 hence this condition is always false.

              Assignee:
              Brian Burkhalter
              Reporter:
              Brian Burkhalter
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: