Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8280703

CipherCore.doFinal(...) causes potentially massive byte[] allocations during decryption

XMLWordPrintable

        As announced on the security-dev mailing list, I'd like to (re)introduce a conditional statement that was first introduced in the context of JDK-8253821 and removed in JDK-8253821:

        It used to be this [1]:

        if (getMode() != GCM_MODE || outputCapacity < estOutSize) {
            // create temporary output buffer if the estimated size is larger
            // than the user-provided buffer.
            internalOutput = new byte[estOutSize];
            offset = 0;
        }

        The assumption in the original issue was, that only GCM would benefit from working on the original buffer. However, the temporary internal buffer can be avoided altogether if the output buffer is large enough.

        Furthermore, the need to copy the results back from the temporary buffer to the output buffer can be circumvented.

        The current implementation will _always_ create a temporary buffer, despite the output buffer being sufficient. For every call to `Cipher.doFinal(ByteBuffer, ByteBuffer)` this causes the allocation of a byte[] with the size depending on the supplied ciphertext.

        A small test program for showing these allocations is available on [2].

        [1] https://github.com/openjdk/jdk/blame/e546ae27ffc6c19ae078a41ab6e1741a104958c1/src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java#L946-L951

        [2] https://gist.github.com/overheadhunter/f3969950c0fdbaecaa77c857b2246cc5

              sstenzel Sebastian Stenzel
              sstenzel Sebastian Stenzel
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: