PEM API (Preview)

Summary

Introduce an API for encoding and decoding the Privacy-Enhanced Mail (PEM) format. The PEM format is used for storing and sending cryptographic keys and certificates. This is a preview API.

Goals

• Ease of use - Define a concise API that converts between Java objects and PEM.
• Standards - Support for PKCS#8, X.509, PKCS#8 v2.0 (OneAsymmetricKey), and Encrypted PKCS#8 binary encoding formats.

Non-Goals

• It is not a goal to support the optional OneAsymmetricKey Attributes field defined in PKCS#8 v2.0

Motivation

PEM is a textual encoding used for storing and transferring security objects, such as asymmetric keys, certificates, and certificate revocation lists (CRL). Defined in RFC 1421 and RFC7468, PEM consists of a Base64-formatted binary encoding surrounded by a type identifying header and footer. In the PEM-formatted RSA public key example below, the header and footer start with five dashes followed by a "BEGIN" or "END", respectively. Then each follow with an identifier that describes the encoded security object, in this example "PUBLIC KEY". Finally, each ends with five dashes. Details about the key, such as the algorithm, can be obtained by parsing the binary encoding.

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDrYzJwsz8uIwnmWznTr5r1N23e
/nLzxYC+TH6gwjOWvMiNMiJoP4c4mySRy4N3plFQUp3pIB7wqshi1t6hkdg7gRGj
MtJpIPIXynEqRy2mIw2GrKTtu3dqrW+ndarbD6D4yRY1hWHluiuOtzhxuueCuf9h
XCYEHZS1cqd8wokFPwIDAQAB
-----END PUBLIC KEY-----

PEM was designed for sending security objects over email, and over time has been used in different services. Certificate Authorities issue certificate chains in PEM. Microservices use PEM for key and/or certificate stores when replicating multiple server instances that require pre-configured and/or consistent security objects. Cryptographic libraries, like OpenSSL, support security object generation and format conversion with PEM. Key Management applications can initialize and update security objects with PEM.

Decoding

Java lacks an API to directly decode the above PEM. Developers must piece together security and non-security classes to convert the PEM text into a PublicKey object:

String pemData = <PEM public key>
String base64Data = pemData.replace("-----BEGIN PUBLIC KEY-----", "") (1)
    .replaceAll(System.lineSeparator(), "")
    .replace("-----END PUBLIC KEY-----", "");

byte[] encoded = Base64.getDecoder().decode(base64Data);              (2)
X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);

KeyFactory keyFactory = KeyFactory.getInstance("RSA");                (3)
var key = keyFactory.generatePublic(keySpec);

This code is very fragile, prone to parsing errors, and uses System.lineSeparator() which is platform-specific. The developer must manually strip the header, footer, and line separators before passing the encoding into the Base64 API (1). The developer must know which EncodedKeySpec is associated with a public key to create the correct intermediate object from the decoded Base64 result (2). Next, a RSA KeyFactory is used to generate a PublicKey object from the intermediate X509EncodedKeySpec object (3). This example makes two assumptions about the PEM text. First, it only supports public keys by assuming a constant header and footer, and a particular intermediate object class. Secondly, it assumes the key is RSA. If the key algorithm is not known by either prior knowledge or user-provided parsing of the binary encoding, an iteration over asymmetric KeyFactory instances would replace "(3)" with code like the following:

String[] algorithms = { "RSA", "DSA", "EC", "EdDSA", "RSASSA-PSS" };  (1)

int i = 0;
KeyFactory keyFactory;
PublicKey pubKey = null;                                             

while (algorithms.length > i) {                                       (2)
    try {
        keyFactory = KeyFactory.getInstance(algorithms[i++]);
        pubKey = keyFactory.generatePublic(keySpec);
        break;

    } catch (InvalidKeySpecException e) {                             (3)
        // continue loop
    }
}
if (pubKey == null) {
    throw new InvalidKeySpecException("unable to generate key");
}

The sample code shows a hardcoded algorithm list because there is no method that returns a supported list of asymmetric algorithms or KeyFactories (1). Loop through each algorithm's KeyFactory until a public key is generated (2). Catch every KeyFactory's unsuccessful key object generation and throwing an error if no KeyFactory was successful (3). The loop is not an efficient way find the key's algorithm and the complexity grows the more security objects the application supports. Other security objects require different KeyFactories, object generation method calls, header/footer, and/or intermediate objects.

Encoding

Encoding an asymmetric key is easier than decoding because the key object contains the needed information:

StringBuilder sb = new StringBuilder();
sb.append("-----BEGIN PUBLIC KEY-----");                            (1)
sb.append(Base64.getEncoder().encodeToString(pubKey.getEncoded())); (2)
sb.append("-----END PUBLIC KEY-----");
String s = sb.toString();

Without needing a KeyFactory instance, the encoding example is straightforward. The developer writes the proper key type header and footer (1) and running the key’s binary encoding through the Base64 encoder (2). Nevertheless, this task should not be left for the developer to implement.

The most complicated PEM type with the current Java API is encrypting aPrivateKey. The specification defines that the header and footer type identifier is "ENCRYPTED PRIVATE KEY", the encryption key uses password-based encryption (PBE), and the private key is encrypted before encoding with Base64. The developer's code may look like this:

char[] password;
PBEKeySpec pbeKeySpec = new PBEKeySpec(password);                    (1)
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(pbeAlgo);
SecretKey pbeKey = keyFactory.generateSecret(pbeKeySpec);

Cipher pbeCipher = Cipher.getInstance(pbeAlgo);                      (2)
pbeCipher.init(Cipher.ENCRYPT_MODE, pbeKey);
byte[] ciphertext = pbeCipher.doFinal(privateKey.getEncoded());

AlgorithmParameters aps = pbeCipher.getParameters();                 (3)
EncryptedPrivateKeyInfo epki = EncryptedPrivateKeyInfo(aps, ciphertext);

StringBuilder sb = new StringBuilder();                              (4)
Base64.Encoder e = Base64.getEncoder();
sb.append("-----BEGIN ENCRYPTED PRIVATE KEY-----");
sb.append(e.encodeToString(ekpi.getEncoded()));
sb.append("-----END ENCRYPTED PRIVATE KEY-----");
String s = sb.toString()

As the long example shows, PBEKeySpec is used with the password to generate a secret from the SecretKeyFactory(1). That SecretKey is used to encrypt the private key's binary encoded data (2). The encrypted data and encryption parameters are used to create an EncryptedPrivateKeyInfo object. That object will output the correct binary encoding format (3). Finish by Base64 encoding the EncryptedPrivateKeyInfo encoded bytes and surrounding it with the proper PEM header and footer (4). This requires detailed knowledge of the encrypted private key PEM structure beyond what should be expected; as well as, using six security classes and two provider service instances to go from PrivateKey to Encrypted Private Key PEM.

After the JCE Survey in April 2022, Key encodings was identified as a top feature lacking for security libraries. The Java API does not provide an easy-to-use implementation of PEM. It leaves much of the work to the developer. From trial-and-error with discovering the binary encoded key algorithm, to stripping or adding PEM headers and footers, to creating all the encryption objects necessary to encrypt or decrypt a private key. This complexity drives users to find other solutions. Java can do better by simplifying the tasks, providing an internal parser to obtain the key algorithm, and to read and write PEM textual data.

Description

A set of new interfaces and classes will provide an easier to use experience with PEM and a foundation for future encodings. These additions include:

• A SecurityObject interface that will be extended by Java classes that contain Key or Certificate material.
• An Encoder and Decoder interface to provide future encoding support.
• A PEMEncoder and PEMDecoder classes to provide support for PEM. They are immutable, reusable objects that do not keep state previously used security object.

SecurityObject

Security classes and sub-classes of Key, Certificate, and CRL are used directly with cryptographic operation, such as Signature.initSign(PrivateKey). KeyPair, EncryptedPrivateKeyInfo, and sub-classes of KeySpec store data in an encoded state; such as KeyFactory(KeySpec) to generate a PrivateKey. All these classes can be given or returned by PEM methods, and specifying a method signature for each class would bloat the API. To solve this, a new interface called SecurityObject will be implemented by those classes. This generic type simplifies the API, minimizing the number of methods. This interface will contain no method.

package java.security;

public interface SecurityObject {}

Encoding

The Encoder interface defines basic methods for encoding security objects. The PEMEncoder subclass defines methods for encoding SecurityObjects to PEM:

package java.security;

public interface Encoder<T> {
    String encode(T tClass) throws IOException;
}

public final class PEMEncoder implements Encoder<SecurityObject> {
    public PEMEncoder();
    public String encode(SecurityObject so) throws IOException;
    public PEMEncoder withEncryption(char[] password) throws IOException;
}

To encode a SecurityObject, an instance of PEMEncoder must first be created by the constructor. To complete the encoding, call encode(SecurityObject) which will return a String containing the PEM data. PEMEncoder is reusable, allowing encode() to be used repeatedly. If the SecurityObject to be encoded is a PrivateKey, there is an option to encrypt it. withEncryption(char[] password) will return a new immutable PEMEncoder, configured with encryption, a default algorithm, and the given password. Non-PrivateKey's can use an encrypted PEMEncoder as if it were an unencrypted instance. To use non-default encryption parameters or encrypt with a different JCE Provider, encode with an EncryptedPrivateKeyInfo object. See the EncryptedPrivateKeyInfo section for more details.

Here are examples using the Encoder class: Encoding a PrivateKey into PEM:

String pemData = new PEMEncoder().encode(privKey);

Encoding a PrivateKey into PEM with encryption:

String pemData = new PEMEncoder().withEncryption(password).encode(privKey);

Encoding both a public and private key into the same PEM:

String pemData = new PEMEncoder().encode(new KeyPair(publicKey, privateKey));

Decoding

The Decoder interface defines basic methods for encoding security objects. The PEMDecoder subclass defines methods for encoding SecurityObjects to PEM:

 package java.security;

 public interface Decoder<T> {
     <S extends T> S decode(String string, Class <S> tClass)
         throws IOException;
     <S extends T> S decode(Reader reader, Class <S> tClass)
         throws IOException;
     T decode(String string) throws IOException;
     T decode(Reader reader) throws IOException;
 }

 public final class PEMDecoder implements Decoder<SecurityObject> {
     public PEMDecoder();
     public PEMDecoder withDecryption(char[] password);
     public PEMDecoder withFactory(Provider provider);
     public SecurityObject decode(String str) throws IOException;
     public SecurityObject decode(Reader reader) throws IOException;
     public <S extends SecurityObject> S decode(Reader reader,
         Class<S> sClass) throws IOException;
     public <S extends SecurityObject> S decode(String string,
         Class<S> sClass) throws IOException;
 }

To decode, get an instance from the PEMDecoder constructor and provide the PEM data in a String or Reader to one of the decode() methods. The return value will be a SecurityObject that the caller can use instanceof to check the class type. If the developer knows the class type being decoded, the generics decode methods can be used to cast the returned object class. An IOException is thrown when the class given, and the return class do not match. A returned class type of EncryptedPrivateKeyInfo means the PEM was an encrypted private key. EncryptedPrivateKeyInfo methods must be used to generate the PrivateKey. If the developer knows the the PEM was encrypted, withDecryption(char[] password) will return a new immutable PEMDecoder configured with the given password. This allows decode() to directly return the PrivateKey. withFactory(Provider) will return a new PEMDecoder instance which will limit the security object generation to the given JCE Provider's Factory. If the Provider does not support the security object, an IOException is thrown.

Here are some examples:

Decoding a PublicKey from PEM:

PublicKey key = new PEMDecoder().decode(pemData, PublicKey.class);

Decoding PEM data of unknown type:

switch (new PEMDecoder().decode(pemData)) {
    case PublicKey pubkey -> ...
    case PrivateKey privkey -> ...
    ...

    }

Decoding an Encrypted EC PrivateKey PEM:

ECPrivateKey eckey = new PEMDecoder().withDecryption(password).
    decode(pemData, ECPrivateKey.class);

Decoding with a specific Factory provider Decoder:

PEMDecoder d = new PEMDecoder().withFactory(providerF);
Certificate c = d.decode(pemData, Certificate.class);

EncryptedPrivateKeyInfo

A common use case with private keys is to encrypt the key before encoding to PEM or decrypt the key after decoding from PEM. However, this operation requires the PEM encoders be configured and it may require additional encryption parameters. To keep the PEM APIs simple, the EncryptedPrivateKeyInfo class was enhanced with additional methods that support these operations.

 EncryptedPrivateKeyInfo {
     ...
     public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
         char[] password) throws IOException;
     public static EncryptedPrivateKeyInfo encryptKey(PrivateKey key,
         char[] password, String pbeAlgo, AlgorithmParameterSpec aps,
         Provider p) throws IOException;
     public PrivateKey getKey(char[] password) throws IOException;

     public PrivateKey getKey(char[] password, Provider provider)
         throws IOException;
 }

The new encryptKey() methods will encrypted the given PrivateKey with the given password. A String algorithm, a AlgorithmParameterSpec and a Provider are optional to not use the default encryption parameters. EncryptedPrivateKeyInfo can be given to PEMEncoder.encode() to create a PEM.

ekpi = EncryptedPrivateKeyInfo.encryptKey(privkey, password);
String pemData = new PEMEncoder().encode(epki);

getKey() is used for decrypting a initialized EncryptedPrivateKeyInfo, such as when PEMDecoder.decode() has returned an EncryptedPrivateKeyInfo. The methods will return a PrivateKey with a given password and optional JCE Provider.

EncryptedPrivateKeyInfo epki = new PEMDecoder().decode(pemData);
PrivateKey key = epki.getKey(password);

The default PBE algorithm used when encrypting a PrivateKey with PEMEncoder and EncryptedPrivateKeyInfo is stored in the java.security file. The jdk.epkcs8.defaultAlgorithm security property defines the default algorithm to PBEWithHmacSHA256AndAES_128. There is no compatibility issue if the default algorithm is changed as Encrypted PKCS#8 encoding contains the algorithm and parameters necessary for decrypting.

Alternatives

A PEM API is a bridge between Base64 and JCE security objects. How to layer that on top of a complex JCE with an easy to use design is important to creating the right API. Retrofitting existing APIs can be awkward when all the pieces don't fit in the right classes. Building a new API brings a clean slate, but still needs to work in the existing structure. Here are some of the designs explored:

Extending the EncodedKeySpec API

The EncodedKeySpec is used to encapsulate binary encoded key data for KeyFactories and other security classes which require the data type identification. Java provides two sub-classes, PKCS8EncodedKeySpec for PrivateKey and X509EncodedKeySpec for PublicKey. A new PEMEncodedKeySpec would type identify encapsulated PEM data, while providing encoding and decoding operations between PEM and the appropriate private or public key EncodedKeySpec. This new EncodedKeySpec could also be a new input for KeyFactories, simplifying the user experience.

This design had a few deficiencies. EncodedKeySpec was being misused as a translation class and was Key-centric. It can not support Certificates or CRLs. Also, KeyFactory's use EncodedKeySpec during public and private key generation. A new EncodedKeySpec added compatibility risks and ease of use issues with existing third-party providers.

Enhancing the CertificateFactory and KeyFactory APIs

Key and certificate factory classes are used to convert and generate their respective objects. CertifcateFactory already decodes PEM certificate and CRL data. Adding encoding to CertificateFactory and both to KeyFactory would be consistent with the existing design, but it complicates JCE providers. CertificateFactory makes the design look easy as there is one industry standard encoding for certificates. KeyFactory is much more complicated as keys encoding formats differ between algorithm and key type with different JCE providers supporting different asymmetric algorithms. PEM adds an extra layer of complexity if each provider is responsible for its conversion; as well as, handling encrypted private keys.

To avoid this provider complexity, another design idea was to add static PEM methods in KeyFactory, but this deviates from the CertificateFactory's pluggable design and creates different solutions between the two factories.

Single Class API for PEM

When considering a new API for PEM, a single class that could encode and decode PEM text was examined. Ultimately, the API lacked distinct lanes for encode and decode operations and was felt not to be as user-friendly.

Creating a new JCE API/SPI for Encodings

The Java Crypto Architecture (JCA) is built on a pluggable API/SPI that allows third party providers to enable cryptographic services not included in the JDK. Binary encodings have not been part of that infrastructure. Java JCE providers limit the binary encoding used to import and export keys. PEM is another option for both importing and exporting that could be a provider service. Defining a SPI for encoding and decoding encoding formats could be beneficial beyond PEM.

Unfortunately this alternative provides nothing beyond a lot of infrastructure to get a provider PEM instance with a more complicated API than java.util.Base64. Additional API/SPIs would be needed for encrypted private keys, extracting the key algorithm, and other details to use with a factory to generate a Key or Certificate object. These PEM-specific changes would eliminate any benefit of using the JCA. With third-party APIs existing for many years, a pluggable infrastructure offers nothing but making a simple task hard.

Testing

Tests should include:

• Verify all supported SecurityObject classes can encode and decode PEM.
• Verify RSA, EC, and EdDSA security objects can be encoded and decoded.
• Reading PEM generated from third-party application
• Negative testing with bad PEM data.

Risks and Assumptions

There is no risk to existing applications because none of that is being removed.

Dependencies

None