Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8160425

Vectorization with signalling NaN returns wrong result

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • P2
    • 9
    • 9
    • hotspot
    • None
    • b131
    • x86
    • Verified

    Description

      Attached TestNaNVector.java triggers the problem on 32-bit x86.

      We vectorize the following loop:
      for (int i = 0; i < 1024; ++i) {
        array[i] = (char)0xfff7;
      }
      The array store will be replaced by a 64-bit vector store to four subsequent array elements. The vector should look like this '0xfff7fff7fff7fff7' and is read from the constant table. However, in floating point arithmetic this is a signalling NaN which may be converted to a quiet NaN when processed by the x87 FPU. If the signalling bit is set, the vector ends up in the constant table as '0xfffffff7fff7fff7' which leads to an incorrect result.

      The problem is that the constant vector is passed around as a double in the C code. On x86 32-bit, floating point values are returned via the FPU stack and since the value is a signalling NaN, it's converted to a quiet NaN by the FPU instructions.

      Attachments

        Issue Links

          Activity

            People

              thartmann Tobias Hartmann
              thartmann Tobias Hartmann
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: