PEM API (Preview)

Summary

Introduce an API for encoding and decoding the Privacy-Enhanced Mail (PEM) format. The PEM format is used for storing and sending cryptographic keys, certificates, and certificate revocations lists. This is a preview API.

Goals

• Ease of use - Define a concise API that converts between PEM data and asymmetric key, certificate, and certificate revocation lists objects.
• Binary format support with PEM:
  
  • PrivateKey and encrypted private keys using PKCS#8
  • Certificate, CRL, and PublicKey objects support using X.509
  • KeyPair and EncryptedPrivateKeyInfo object support using PKCS#8 v2.0

Non-Goals

• It is not a goal to support types in RFC 7468 that cannot be represented as a cryptographic Java object.

Motivation

PEM is a textual encoding used for storing and transferring security objects, such as asymmetric keys, certificates, and certificate revocation lists (CRLs). Defined in RFC 1421 and RFC 7468, PEM consists of a Base64-formatted binary encoding surrounded by a header and footer identifying the type. In the PEM-formatted RSA public key example below, the header and footer start with five dashes followed by a "BEGIN" or "END", respectively. Then each follow with an identifier that describes the encoded security object, in this example "PUBLIC KEY". Finally, each ends with five dashes. Details about the key, such as the algorithm, can be obtained by parsing the binary encoding.

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDrYzJwsz8uIwnmWznTr5r1N23e
/nLzxYC+TH6gwjOWvMiNMiJoP4c4mySRy4N3plFQUp3pIB7wqshi1t6hkdg7gRGj
MtJpIPIXynEqRy2mIw2GrKTtu3dqrW+ndarbD6D4yRY1hWHluiuOtzhxuueCuf9h
XCYEHZS1cqd8wokFPwIDAQAB
-----END PUBLIC KEY-----

PEM was designed for sending security objects over email, and over time has been used in different services. Certificate Authorities issue certificate chains in PEM. Microservices use PEM for key and/or certificate stores when replicating multiple server instances that require pre-configured security objects. Cryptographic libraries, like OpenSSL, support security object generation and format conversion with PEM. Key Management applications initialize and update security objects with PEM.

After the Java Cryptographic Extensions Survey in April 2022, Key encodings were identified as a top feature lacking for security libraries. The Java API does not provide an easy-to-use implementation of PEM. It leaves much of the work to the developer; from using trial-and-error techniques to discover the binary encoded key algorithm, to stripping or adding PEM headers and footers, to creating all the encryption objects necessary to encrypt or decrypt a private key. This complexity drives users to find other solutions. Java can simplify these tasks by providing a means to obtain the key algorithm and a simple API to read and write PEM textual data. The next two sections will illustrate how the current Java APIs are lacking and tedious to use for reading and writing PEM textual data.

Decoding

Java lacks an API to directly decode the above PEM. Developers must piece together security and non-security classes to convert the PEM text into a PublicKey object:

String pemData = <PEM public key>
String base64Data = pemData.replace("-----BEGIN PUBLIC KEY-----", "") (1)
    .replaceAll(System.lineSeparator(), "")
    .replace("-----END PUBLIC KEY-----", "");

byte[] encoded = Base64.getDecoder().decode(base64Data);              (2)
X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);

KeyFactory keyFactory = KeyFactory.getInstance("RSA");                (3)
var key = keyFactory.generatePublic(keySpec);

This code is very fragile, prone to parsing errors, and uses System.lineSeparator() which is platform-specific. The developer must manually strip the header, footer, and line separators before passing the encoding into the Base64 API (1). The developer must know which EncodedKeySpec is associated with a public key to create the correct intermediate object from the decoded Base64 result (2). Finally, a RSA KeyFactory is used to generate a PublicKey object from the intermediate X509EncodedKeySpec object (3). This example makes two assumptions about the PEM text. First, it only supports public keys by assuming a constant header and footer, and a particular intermediate object class. Secondly, it assumes the key is RSA. If the key algorithm is not known by either prior knowledge or user-provided parsing of the binary encoding, an iteration over asymmetric KeyFactory instances would replace "(3)" with code like the following:

String[] algorithms = { "RSA", "DSA", "EC", "EdDSA", "RSASSA-PSS" };  (1)

int i = 0;
PublicKey pubKey = null;                                             

while (algorithms.length > i) {                                       (2)
    try {
        KeyFactory kf = KeyFactory.getInstance(algorithms[i++]);
        pubKey = kf.generatePublic(keySpec);
        break;

    } catch (InvalidKeySpecException e) {                             (3)
        // continue loop
    }
}
if (pubKey == null) {
    throw new InvalidKeySpecException("unable to generate key");
}

The sample code contains a hardcoded algorithm list because there is no API that returns a supported list of asymmetric algorithms or KeyFactory instances (1). The developer must instantiate a KeyFactory for each supported algorithm until a public key is generated (2). The code must catch any unsuccessful public key generation and throw an exception if none were successful (3). The loop is an inefficient way find the key's algorithm and the complexity grows as the application supports more security objects. Other security objects are encoded with different PEM headers/footers and require different security APIs to decode the data into Java objects.

Encoding

Encoding an asymmetric key is easier than decoding because the key object contains the needed information:

StringBuilder sb = new StringBuilder();
sb.append("-----BEGIN PUBLIC KEY-----");                            (1)
sb.append(Base64.getEncoder().encodeToString(pubKey.getEncoded())); (2)
sb.append("-----END PUBLIC KEY-----");
String s = sb.toString();

The encoding example is fairly straightforward and does not require a KeyFactory for encoding the key. The developer writes the proper key type header and footer (1) and passes the key’s binary encoding through the Base64 encoder (2). Nevertheless, this task should be easier for the developer to implement.

The most complicated PEM use case with the current Java API is encrypting a PrivateKey. In order to encode a private key to PEM using the “ENCRYPTED PRIVATE KEY” identifier, the key must first be encrypted with a password or secret key. The developer's code may look like this:

char[] password;
PBEKeySpec pbeKeySpec = new PBEKeySpec(password);                    (1)
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(pbeAlgo);
SecretKey pbeKey = keyFactory.generateSecret(pbeKeySpec);

Cipher pbeCipher = Cipher.getInstance(pbeAlgo);                      (2)
pbeCipher.init(Cipher.ENCRYPT_MODE, pbeKey);
byte[] ciphertext = pbeCipher.doFinal(privateKey.getEncoded());

AlgorithmParameters aps = pbeCipher.getParameters();                 (3)
EncryptedPrivateKeyInfo epki = new EncryptedPrivateKeyInfo(aps, ciphertext);

StringBuilder sb = new StringBuilder();                              (4)
Base64.Encoder e = Base64.getEncoder();
sb.append("-----BEGIN ENCRYPTED PRIVATE KEY-----");
sb.append(e.encodeToString(epki.getEncoded()));
sb.append("-----END ENCRYPTED PRIVATE KEY-----");
String s = sb.toString();

As the long example shows, PBEKeySpec is first used with the password to generate a secret key from the SecretKeyFactory(1). That SecretKey is then used to encrypt the private key's binary encoded data (2). Then, the encrypted data and encryption parameters are used to create an EncryptedPrivateKeyInfo object, which contains the correct binary encoding format (3). Finally, we finish by Base64 encoding the encoded bytes of the EncryptedPrivateKeyInfo and surrounding it with the proper PEM header and footer (4). This use case requires detailed knowledge of the format of the encrypted private key PEM structure beyond what should be expected; as well as requiring six security classes and two provider service instances to encrypt and encode a PrivateKey to PEM.

Description

A new interface and two new classes will provide an easier to use experience with PEM. These additions include:

• A DEREncodeable interface that will be extended by Java classes that contain Key or Certificate material.
• PEMEncoder and PEMDecoder classes to provide support for PEM. These APIs are immutable and reusable and do not keep state of the previously used security object.

DEREncodable

This new sealed interface marks the permitted classes and interfaces that return security data in a binary encoded representation. Distinguished Encoding Rules (DER) are used with binary encoding formats, X.509 and PKCS#8, and supported by the Java. PEM uses DEREncodable to identify which objects are supported for Base64 encoding and decoding. This generic interface type simplifies the API and minimizes defined methods. The permitted classes are: AsymmetricKey, X509Certificate, X509CRL, KeyPair, EncryptedPrivateKeyInfo, PKCS8EncodedKeySpec, and X509EncodedKeySpec.

package java.security;

public sealed interface DEREncodable permits AsymmetricKey, KeyPair, X509CRL,
    X509Certificate, PKCS8EncodedKeySpec, X509EncodedKeySpec,
    EncryptedPrivateKeyInfo {}

With this new sealed class, some of the permitted classes will change slightly:

public non-sealed interface AsymmetricKey ...
public non-sealed abstract class X509Certificate ...
public non-sealed abstract class X509CRL ...
public final class EncryptedPrivateKeyInfo ...
public final class PKCS8EncodedKeySpec ...
public final class X509EncodedKeySpec ...

Encoding

The PEMEncoder class defines methods for encoding DEREncodable types to PEM:

package java.security;

public final class PEMEncoder {
    public static PEMEncoder of();
    public byte[] encode(DEREncodable so) throws IOException;

    public String encodeToString(DEREncodable so) throws IOException;

    public PEMEncoder withEncryption(char[] password);
}

To encode a DEREncodable, get a PEMEncoder instance by calling of(). There are two methods to complete the encoding process. The first is encode(DEREncodable) that returns PEM data in a byte[]. The other is encodeToString(DEREncodable) that returns PEM data as a String. A PEMEncoder instance is reusable, allowing encode methods to be used repeatedly. If the DEREncodable to be encoded is a PrivateKey, there is an option to encrypt. The withEncryption(char[] password) method returns a new immutable PEMEncoder instance, configured with encryption using a default algorithm. Only PrivateKey objects can be encoded by an encrypted PEMEncoder. To use non-default encryption parameters or encrypt with a different JCE Provider, encode with an EncryptedPrivateKeyInfo object. See the EncryptedPrivateKeyInfo section below for more details.

Here are examples using the PEMEncoder class:

Encoding a PrivateKey into PEM:

PEMEncoder pe = PEMEncoder.of();
byte[] pemData = pe.encode(privKey);</code></pre>

<p>Encoding a <code class="prettyprint" >PrivateKey</code> into PEM with encryption:</p>

<pre class="prettyprint" ><code>String pemString = pe.withEncryption(password).encodeToString(privKey);</code></pre>

<p>Encoding both a public and private key into the same PEM:</p>

<pre class="prettyprint" ><code>byte[] pemData = pe.encode(new KeyPair(publicKey, privateKey));

Decoding

The PEMDecoder subclass defines methods for decoding PEM to a DEREncodable:

 package java.security;

 public final class PEMDecoder {
     public static PEMDecoder of();
     public PEMDecoder withDecryption(char[] password);
     public PEMDecoder withFactory(Provider provider);
     public DEREncodable decode(String str) throws IOException;
     public DEREncodable decode(InputStream is) throws IOException;
     public <S extends DEREncodable> S decode(String string,
         Class<S> sClass) throws IOException;
     public <S extends DEREncodable> S decode(InputStream is,
         Class<S> sClass) throws IOException;
 }

To decode PEM data, get a PEMDecoder instance by calling of(). There are four methods to complete the decoding process. They each return a DEREncodable for which the caller can use pattern matching when processing the result. If the developer knows the class type being decoded, the two decode methods that take a Class<S> argument can be used to specify the returned object's class. If the class does not match the PEM type, an IOException is thrown. When passing input data, the application is responsible for processing data ahead of the PEM header as it will be ignored by decode.

If the PEM data is an encrypted private key, withDecryption(char[] password) is a helper method that returns a new immutable PEMDecoder configured for decryption with the given password. The decode methods called from that configured instance return a PrivateKey object, but throw an IOException if the password is incorrect. Other PEM data types can be decoded by a configured instance, as decryption is not relevant. On non-configured instances, the decode methods return an EncryptedPrivateKeyInfo object for encrypted private keys, which can then be used to decrypt and generate the PrivateKey object (see EncryptedPrivateKeyInfo below for more details).

Some DEREncodable types may need to be generated from a particular JCE provider. The withFactory(Provider) method returns a new PEMDecoder instance that uses the specified JCE provider to generate the security object with a particular factory (for example, a KeyFactory). If the provider does not support the PEM data being decoded, an IOException is thrown.

Here are some examples:

Decoding a PublicKey from PEM:

PEMDecoder pd = PEMDecoder.of();
PublicKey key = pd.decode(pemData, PublicKey.class);

Decoding PEM data of an unknown type:

switch (pd.decode(pemData)) {
    case PublicKey pubkey -> ...
    case PrivateKey privkey -> ...
    ...
    }

Decoding an encrypted ECPrivateKey from PEM:

ECPrivateKey eckey = pd.withDecryption(password).
    decode(pemData, ECPrivateKey.class);

Decoding a Certificate with a specific Factory provider:

PEMDecoder d = pd.withFactory(providerF);
Certificate c = d.decode(pemData, Certificate.class);

EncryptedPrivateKeyInfo

A common use case with private keys is to encrypt the key before encoding to PEM or decrypt the key after decoding from PEM. However, this operation may require additional encryption parameters than just a password. To keep the PEM APIs simple, the EncryptedPrivateKeyInfo class has been enhanced with additional methods that support these operations:

 EncryptedPrivateKeyInfo {
     ...
     public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
         char[] password) throws IOException;
     public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
         char[] password, String algorithm, AlgorithmParameterSpec params,
         Provider p) throws IOException;
     public PrivateKey getKey(char[] password) throws IOException;

     public PrivateKey getKey(char[] password, Provider provider)
         throws IOException;
 }

The two new encryptKey() methods encrypt the given PrivateKey with the given password. For advanced usage, the second method allows all the cryptographic parameters to be specified if the defaults are not sufficient. The returned EncryptedPrivateKeyInfo object can then be passed to PEMEncoder.encode() to encode to PEM:

ekpi = EncryptedPrivateKeyInfo.encryptKey(privkey, password);
byte[] pemData = PEMEncoder.of().encode(epki);

The getKey() methods are used for decrypting an initialized EncryptedPrivateKeyInfo, such as when PEMDecoder.decode() has returned an EncryptedPrivateKeyInfo. The methods will return a PrivateKey with a given password and an optional JCE provider:

EncryptedPrivateKeyInfo epki = PEMDecoder.of().decode(pemData);
PrivateKey key = epki.getKey(password);

The default PBE algorithm used when encrypting a PrivateKey with PEMEncoder and EncryptedPrivateKeyInfo is stored in the java.security file. The jdk.epkcs8.defaultAlgorithm security property defines the default algorithm to be "PBEWithHmacSHA256AndAES_128". There is no compatibility issue if the default algorithm is changed as the encrypted PKCS#8 encoding contains the algorithm and parameters necessary for decrypting.

This is a Preview API, disabled by default

To use this new API in JDK 24, you must enable preview features:

• Compile the program with javac --release 24 --enable-preview Main.java and run it with java --enable-preview Main; or,
• When using the source code launcher, run the program with java --enable-preview Main.java; or,
• When using jshell, start it with jshell --enable-preview.

Alternatives

A PEM API is a bridge between Base64 and JCE security objects. How to layer that on top of a complex JCE with an easy to use API is important. Retrofitting existing APIs can be awkward when all the pieces don't fit in the right classes. Building a new API brings a clean slate, but still needs to work in the existing structure. Here are some of the designs explored:

Extending the EncodedKeySpec API

The EncodedKeySpec API is used to encapsulate binary encoded key data for KeyFactory instances and other security classes which require the data type identification. Java provides two sub-classes, PKCS8EncodedKeySpec for PrivateKey encodings and X509EncodedKeySpec for PublicKey encodings. A new PEMEncodedKeySpec sub-class was considered which would type identify encapsulated PEM data, while providing encoding and decoding operations between PEM and the appropriate private or public key EncodedKeySpec. This new EncodedKeySpec sub-class could also be an input for KeyFactory instances, simplifying the user experience.

This design had a few deficiencies. This new PEMEncodedKeySpec sub-class would be a misuse of a translation class and would be Key-centric. It cannot support encoding certificates or CRLs to PEM. Also, the existing EncodedKeySpec sub-classes mentioned above are already used by KeyFactory instances for public and private key generation. A new EncodedKeySpec sub-class would add compatibility risks and ease of use issues with existing third-party providers.

Enhancing the CertificateFactory and KeyFactory APIs

Key and certificate factory classes are used to convert and generate their respective objects. The CertificateFactory API already supports decoding of PEM certificate and CRL data. Adding encoding to CertificateFactory and KeyFactory would be consistent with the existing design, but it complicates JCE providers. CertificateFactory makes the design look easy as there is one industry standard encoding for certificates. KeyFactory is much more complicated as key encoding formats differ between algorithm and key type and different JCE providers support different asymmetric algorithms. PEM adds an extra layer of complexity if each provider is responsible for its conversion; as well as, handling encrypted private keys.

To avoid this provider complexity, another design idea was to add static PEM methods to KeyFactory, but this deviates from the pluggable design of CertificateFactory and creates different solutions between the two factories.

Single Class API for PEM

When considering a new API for PEM, a single class that could encode and decode PEM text was examined. Ultimately, the API lacked distinct lanes for encode and decode operations and was felt not to be as user-friendly.

Creating a new JCE API/SPI for Encodings

The Java Crypto Architecture (JCA) is built on a pluggable API/SPI that allows third party providers to enable cryptographic services not included in the JDK. Binary encodings have not been part of that infrastructure. Java JCE providers limit the binary encodings used to import and export keys. PEM is another option for both importing and exporting that could be a provider service. Defining a SPI for encoding and decoding encoding formats could be beneficial beyond PEM.

Unfortunately this alternative provides nothing beyond a lot of infrastructure to get a provider PEM instance with a more complicated API than java.util.Base64. Additional API/SPIs would be needed for encrypted private keys, extracting the key algorithm, and other details to use with a factory to generate a Key or Certificate object. These PEM-specific changes would eliminate any benefit of using the JCA. With third-party APIs existing for many years, a pluggable infrastructure offers nothing but making a simple task hard.

Testing

Tests should include:

• Verifying all supported DEREncodable classes can encode and decode PEM.
• Verifying RSA, EC, and EdDSA security objects can be encoded and decoded.
• Reading PEM generated from third-party applications, and vice-versa.
• Negative testing with bad PEM data.