Our developers reported a regression on JDK 25 where AES-GCM encryption fails for payloads of approximately 1MB in size. The developers did investigations and identified the issue was caused by the intrinsic implementation of the AES-GCM encryption algorithm for implGCMCrypt0 on AVX512 enabled machine.

The intrinsic method incorrectly reports the number of bytes processed if input length is not a multiple of the block size.

Analysis from our developers:
==========
... in particular for implGCMCrypt0 which is emitted by generate_galoisCounterMode_AESCrypt. This emitter internally uses aesgcm_avx512 or aesgcm_avx2, depending on the available instruction set extension.

AVX512
-----------
void StubGenerator::aesgcm_avx512(Register in, Register len, Register ct, Register out, Register key, Register state,
                                  Register subkeyHtbl, Register avx512_subkeyHtbl, Register counter) {
  // [...]
  const Register pos = rax; // result in rax (processed bytes)

  // [...]
  __ movl(pos, 0);
  __ cmpl(len, 256);
  __ jcc(Assembler::lessEqual, ENC_DEC_DONE); // line 3399

  // [...]
  initial_blocks_16_avx512(in, out, ct, pos, key, avx512_subkeyHtbl, CTR_CHECK, rounds, CTR_BLOCKx, AAD_HASHx, ADDBE_4x4, ADDBE_1234, ADD_1234, SHUF_MASK, stack_offset);
  __ addl(pos, 16 * 16);
  __ cmpl(len, 32 * 16);
  __ jcc(Assembler::below, MESG_BELOW_32_BLKS); // line 3461

  // [...]
  __ bind(MESG_BELOW_32_BLKS);
  __ subl(len, 16 * 16);
  __ addl(pos, 16 * 16); // line 3512
  gcm_enc_dec_last_avx512(len, in, pos, AAD_HASHx, SHUF_MASK, avx512_subkeyHtbl, ghashin_offset, HashKey_16, true, true);

  // [...]
  __ bind(ENC_DEC_DONE);
}

For an input length <= 256 bytes, we immediately return in line 3399. For an input length < 512, we jump to the label MESG_BELOW_32_BLKS. Finally, we can see that at the highlighted line 3512, the result is always incremented by 256, independently of whether the actual input length was aligned to 256.

This means, that for any unaligned input length in [257, 511], the function returns that it processed 512 bytes. This is OK as long as the calling Java code aligns any lengths appropriately.
==========

I can also reproduce the failure on my local machine using both release and slowdebug binaries built from JDK mainline. On my local machine, the AVX2 variant of generate_avx2_galoisCounterMode_AESCrypt is enabled by default.

StubGenerator::aesgcm_avx2 assumes the input size must be 128 bytes or more: https://github.com/openjdk/jdk/blob/6322aaba63b235cb6c73d23a932210af318404ec/src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp#L4021. However, that assumption may not be true when implGCMCrypt0 is called for handling any remaining data after the main loop in implGCMCrypt:

https://github.com/openjdk/jdk/blob/6322aaba63b235cb6c73d23a932210af318404ec/src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java#L595