Customer reported a minor bug in the JCE, resulting in incorrect output from encryption/decryption operations under specific conditions. This bug can result in multiple successive encryption calls to doFinal or doUpdate that overwrites the results of the initial encryption/decryption call.

CipherSpi has two functions, engineDoFinal and engineUpdate, which take ByteBuffers as both inputs and outputs. As documented in the Cipher javadoc for both doFinal and engineUpdate:

    Note: this method should be copy-safe, which means the input and output
buffers can reference the same byte array and no unprocessed input data is
overwritten when the result is copied into the output buffer.

Although this statement is not reflected in CipherSpi, in practice Cipher
directly delegates to CipherSpi and therefore CipherSpi must implement the
same contract.

If not overridden by the Cipher implementation, CipherSpi provides a default implementation of engineDoFinal(ByteBuffer, ByteBuffer) and engineUpdate(ByteBuffer, ByteBuffer) that detects if the ByteBuffers have the same backing array (by checking whether .array() is the same if .hasArray() is true), and if so, sends the entire buffer down the implementation to perform the entire operation. The implementation will take care of any overlap.

However, it is possible to construct overlapping ByteBuffers for which this
test fails. For example, if the input ByteBuffer is read-only, hasArray()
will return false. Alternatively, overlapping direct byte buffers can also trigger this issue.

In these cases,the array is copied out and encrypted/decrypted in 4K chunks. If the output location is positioned later in the same buffer, and the message size exceeds the 4K bufffer size used in the copy loop, the output will overwrite some of input before it is read.

A modified test case for this bug is attached, and has been tested (and found to fail) on 1.8.0u131 as well as on Java 9 early access builds.