Better support ALPN byte wire values in SunJSSE

Summary

Certain TLS ALPN values can't be properly read or written by the SunJSSE provider. This is due to the choice of Strings as the API interface and the undocumented internal use of the UTF-8 Character Set which converts characters larger than U+00007F into multi-byte arrays that may not be expected by a peer.

Problem

RFC 7301 Application-Layer Protocol Negotiation (ALPN) is a protocol extension for the various versions of the Transport Layer Security (TLS). The RFC states that byte arrays are used for data transport, but in Section 6, these values could be the UTF-8 encodings of the Strings, but don't have to be.

The Java ALPN APIs selected Strings for ease of use, but internal to SunJSSE, these Strings are converted to byte arrays using UTF-8 as suggested in in RFC 7301. This encoding convention was not required by the RFC or Java documentation/APIs.

Because of the way UTF-8 encodes characters, It is currently not possible for ALPN characters in the range of (U+0080-U+00FF) to be properly output in SunJSSE, and are instead converted to a multi-byte representation.

Recently, a new mechanism called GREASE (RFC 8701) was introduced to help prevent extensibility failures in the TLS ecosystem. Unfortunately, 1/2 of the defined GREASE values fall into the (U+0080-U+00FF) range, and thus can't be represented by SunJSSE (client or server side).

A new API could be defined to use byte arrays, but this approach would be not be helpful for earlier Java releases without Maintenance Releases (MR). e.g.

https://jcp.org/aboutJava/communityprocess/mrel/jsr337/index3.html

This CSR presents an alternate approach using the existing APIs.

Solution

The proposed workaround/fix is to have the SunJSSE implementation encode Strings directly as ISO_8859_1/LATIN-1 characters which correctly handles the full range of byte values (U+0000-U+00FF). (i.e. string.getBytes(Charset.forName("ISO_8859_1");) This change will enhance interoperabibility with other implementations such as Chromium/OpenSSL/etc.

Other UNICODE String values in the range of U+0080-U+10FFFF must now be correctly encoded/decoded (e.g. UTF-8) by applications before sending/receiving, rather than depending on SunJSSE to automatically perform the (possibly incorrect) encoding. We don't anticipate this to be a significant interoperability issue, since all known/current values in the IETF/IANA TLS ALPN extension list can be encoded as ISO_8859_1/LATIN-1 (of which the 7-bit US-ASCII Charset is a proper subset). While UTF-8 is the logical default choice for character encoding, it is possible other formats are being used and therefore the onus of correctly encoding the values should rest with the application. These characters must now be converted to the format required by the peer.

It was also discovered during code development that if a selected ALPN value contained any characters that map to a multi-byte UTF-8 representation, the server-side code would throw an Exception. Since we haven't received any bug reports for this issue, we feel it is unlikely that this will be a problem in the field. But for compatibility concerns, we also introduce a new Java Security property to reverse this change.

Specification

When encoding ALPN values from Strings to/from network bytes, use the ISO_8859_1/LATIN-1 encoding instead of UTF-8.

For the current JDK under development (currently targeting 16), we will add some clarifying verbiage to the javadoc and JSSE documentation. We will add similar behavior and documentation to the various backports, but will not update the javadocs.

In the javax.net.ssl package:

In SSLEngine/SSLSocket/SSLParameters, add text to indicate that the String will be represented using the network byte representation, and may need conversion to the proper Unicode value before use.

SSLEngine.java: Add the following to the package description.

  The ApplicationProtocol {@code String} values returned by the methods
  in this class are in the network byte representation sent by the peer.
  The bytes could be directly compared, or converted to its Unicode
  code String} format for comparison.

  <blockquote><pre>
      String networkString = sslEngine.getHandshakeApplicationProtocol();
      byte[] bytes = networkString.getBytes(StandardCharsets.ISO_8859_1);

      //

      // Match using bytes:

      //

      //   "http/1.1"                       (7-bit ASCII values same in UTF-8)

      //   MEETEI MAYEK LETTERS "HUK UN I"  (Unicode 0xabcd->0xabcf)

      //

      String HTTP1_1 = "http/1.1";

      byte[] HTTP1_1_BYTES = HTTP1_1.getBytes(StandardCharsets.UTF_8);

      byte[] HUK_UN_I_BYTES = new byte[] {
          (byte) 0xab, (byte) 0xcd,
          (byte) 0xab, (byte) 0xce,
          (byte) 0xab, (byte) 0xcf};

      if ((Arrays.compare(bytes, HTTP1_1_BYTES) == 0 )
              || Arrays.compare(bytes, HUK_UN_I_BYTES) == 0) {
         ...
      }

      //
      // Alternatively match using string.equals() if we know the ALPN value
      // was encoded from a {@code String} using a certain character set,
      // for example {@code UTF-8}.  The ALPN value must first be properly
      // decoded to a Unicode {@code String} before use.
      //
      String unicodeString = new String(bytes, StandardCharsets.UTF_8);
      if (unicodeString.equals(HTTP1_1)
              || unicodeString.equals("\u005cuabcd\u005cuabce\u005cuabcf")) {
          ...
      }
   /pre></blockquote>

SSLSocket.java: Add the following to the package description (identical to SSLSocket.java except for sslEngine-sslSocket).

  The ApplicationProtocol {@code String} values returned by the methods
  in this class are in the network byte representation sent by the peer.
  The bytes could be directly compared, or converted to its Unicode
  code String} format for comparison.

  <blockquote><pre>
      String networkString = sslSocket.getHandshakeApplicationProtocol();
      byte[] bytes = networkString.getBytes(StandardCharsets.ISO_8859_1);

      //

      // Match using bytes:

      //

      //   "http/1.1"                       (7-bit ASCII values same in UTF-8)

      //   MEETEI MAYEK LETTERS "HUK UN I"  (Unicode 0xabcd->0xabcf)

      //

      String HTTP1_1 = "http/1.1";

      byte[] HTTP1_1_BYTES = HTTP1_1.getBytes(StandardCharsets.UTF_8);

      byte[] HUK_UN_I_BYTES = new byte[] {
          (byte) 0xab, (byte) 0xcd,
          (byte) 0xab, (byte) 0xce,
          (byte) 0xab, (byte) 0xcf};

      if ((Arrays.compare(bytes, HTTP1_1_BYTES) == 0 )
              || Arrays.compare(bytes, HUK_UN_I_BYTES) == 0) {
         ...
      }

      //
      // Alternatively match using string.equals() if we know the ALPN value
      // was encoded from a {@code String} using a certain character set,
      // for example {@code UTF-8}.  The ALPN value must first be properly
      // decoded to a Unicode {@code String} before use.
      //
      String unicodeString = new String(bytes, StandardCharsets.UTF_8);
      if (unicodeString.equals(HTTP1_1)
              || unicodeString.equals("\u005cuabcd\u005cuabce\u005cuabcf")) {
          ...
      }
   /pre></blockquote>

SSLParameters.java: Add the following to the end of the setApplicationProtocols(String[] protocols) method.

  The {@code String} values must be presented using the network
  byte representation expected by the peer.  For example, if an ALPN
  {@code String} should be exchanged using {@code UTF-8}, the
  {@code String} should be converted to its {@code byte[]} representation

  and stored as a byte-oriented 
{@code String}
 before calling this method.

  <blockquote><pre>

      // MEETEI MAYEK LETTERS HUK UN I (Unicode 0xabcd->0xabcf): 2 bytes

      byte[] bytes = "\u005cuabcd\u005cuabce\u005cuabcf"
              .getBytes(StandardCharsets.UTF_8);
      String HUK_UN_I = new String(bytes, StandardCharsets.ISO_8859_1);

      // 0x00-0xFF:  1 byte
      String rfc7301Grease8F = "\u005c008F\u005c008F";

      SSLParameters p = sslSocket.getSSLParameters();
      p.setApplicationProtocols(new String[] {
              "h2", "http/1.1", rfc7301Grease8F, HUK_UN_I});
      sslSocket.setSSLParameters(p);
  </pre></blockquote>

For compatibility concerns with older SunJSSEs, we introduce a Java Security property to reverse this change:

#
# The default Character set name (java.nio.charset.Charset.forName()) for
# converting TLS ALPN values between byte arrays and Strings.
#
# jdk.tls.alpnCharset=UTF-8
jdk.tls.alpnCharset=ISO_8859_1

which can be overridden to restore the previous encoding process.