Summary

Update the ML-KEM and ML-DSA private key encodings to align with Section 6 of draft-ietf-lamps-kyber-certificates-11 and Section 6 of RFC 9881. Specifically, the privateKey field inside the PKCS #8 encoding will follow the DER-encoded ASN.1 CHOICE structure as defined in these specifications. All 3 CHOICE options ("seed", "expandedKey", and "both") will be supported at both encoding and decoding. The default encoding format is "seed".

For interoperability with latest version of JDK and other PQC vendors, we'd like to backport this function to all update releases where there are already ML-KEM and ML-DSA or we have a plan to backport them to. The default encoding format on all updates release will also be "seed".

Problem

When ML-KEM and ML-DSA were introduced in JDK 24, the RFCs had not been published yet. The drafts at the time, draft-ietf-lamps-kyber-certificates and draft-ietf-lamps-dilithium-certificates, described the private key only as “an opaque byte sequence,” without specifying a concrete format.

Based on that, the JDK 24 implementation adopted the encodings defined in NIST FIPS 203 and 204 — specifically, using the private key output from the ML-KEM.KeyGen function (Section 7.1 of FIPS 203), and the ML-DSA.KeyGen function (Section 5.1 of FIPS 204).

As of today, draft-ietf-lamps-kyber-certificates has been published as RFC 9881 and draft-ietf-lamps-kyber-certificates has been approved by IESG and is now waiting in the RFC Editor Queue. Their private key formats are formally defined as DER-encoded ASN.1 CHOICE types. For example, for ML-KEM-512:

ML-KEM-512-PrivateKey ::= CHOICE {
	seed [0] OCTET STRING (SIZE (64)),
	expandedKey OCTET STRING (SIZE (1632)),
	both SEQUENCE {
  		seed OCTET STRING (SIZE (64)),
  		expandedKey OCTET STRING (SIZE (1632))
  	}
}

Similar structures are defined for ML-KEM-768, ML-KEM-1024, ML-DSA-44, ML-DSA-65, and ML-DSA-87.

The JDK 24 implementation currently uses the second option, expandedKey OCTET STRING. To comply with the final RFCs, we need to update our implementation to support the other two CHOICEs as well.

Solution

For reading, all three CHOICEs are supported when parsing an existing PKCS #8 encoded private key.

For writing, new security properties are introduced to control which CHOICE is used when creating a new private key. When KeyPairGenerator::generateKey or KeyFactory::translateKey is called, the resulting private key will be encoded according to the specified security property setting.

The default value for the new security properties for both ML-KEM and ML-DSA is "seed", for several reasons:

1. It is the smallest encoding.
2. Since the seed can be arbitrary bytes, there is no need to validate the content except for confirming the size, which is always 64 bytes for ML-KEM and 32 bytes for ML-DSA.
3. The seed format contains enough information to derive the other 2 formats.

Both draft-ietf-lamps-kyber-certificates-11 and RFC 9881 recommend using the seed format.

Note: If KeyFactory::translateKey is used to convert a key that does not contain a seed into a format that requires one, the operation will fail with an InvalidKeyException.

Specification

New security properties, jdk.mlkem.pkcs8.encoding and jdk.mldsa.pkcs8.encoding, are introduced to control the PKCS #8 encoding of newly created ML-KEM and ML-DSA private keys.

The following is added to the java.security file:

#
# PKCS #8 encoding format for newly created ML-KEM and ML-DSA private keys
#
# draft-ietf-lamps-kyber-certificates-11 and RFC 9881 define three possible formats for a private key:
# a seed (64 bytes for ML-KEM, 32 bytes for ML-DSA), an expanded private key,
# or a sequence containing both.
#
# The following security properties determine the encoding format used when a
# new keypair is generated with a KeyPairGenerator, and the output of the
# translateKey method on an existing key using a ML-KEM or ML-DSA KeyFactory.
#
# Valid values for these properties are "seed", "expandedKey", and "both"
# (case-insensitive). The default is "seed".
#
# If a system property of the same name is also specified, it supersedes the
# security property value defined here.
#
# Note: These properties are currently used by the SunJCE (for ML-KEM) and SUN
# (for ML-DSA) providers in the JDK Reference implementation. They are not
# guaranteed to be supported by other implementations or third-party security
# providers.
#
#jdk.mlkem.pkcs8.encoding = seed
#jdk.mldsa.pkcs8.encoding = seed