Better support ALPN byte wire values in SunJSSE

Summary

A backport of the JDK 16. The original CSR is at JDK-8256817 The main difference with the 11u and 8u backports is that the extra API documentation added to the SSLEngine/SSLSocket/SSLParameters Java SE classes will be dropped. Instead, it should be sufficient to document the changes via release note.

Certain TLS ALPN values can't be properly read or written by the SunJSSE provider. This is due to the choice of Strings as the API interface and the undocumented internal use of the UTF-8 Character Set which converts characters larger than U+00007F into multi-byte arrays that may not be expected by a peer.

Problem

RFC 7301 Application-Layer Protocol Negotiation (ALPN) is a protocol extension for the various versions of the Transport Layer Security (TLS). The RFC states that byte arrays are used for data transport, but in Section 6, these values could be the UTF-8 encodings of the Strings, but don't have to be.

The Java ALPN APIs selected Strings for ease of use, but internal to SunJSSE, these Strings are converted to byte arrays using UTF-8 as suggested in in RFC 7301. This encoding convention was not required by the RFC or Java documentation/APIs.

Because of the way UTF-8 encodes characters, It is currently not possible for ALPN characters in the range of (U+0080-U+00FF) to be properly output in SunJSSE, and are instead converted to a multi-byte representation.

Recently, a new mechanism called GREASE (RFC 8701) was introduced to help prevent extensibility failures in the TLS ecosystem. Unfortunately, 1/2 of the defined GREASE values fall into the (U+0080-U+00FF) range, and thus can't be represented by SunJSSE (client or server side).

A new API could be defined to use byte arrays, but this approach would be not be helpful for earlier Java releases without Maintenance Releases (MR). e.g. https://jcp.org/aboutJava/communityprocess/mrel/jsr337/index3.html

This CSR presents an alternate approach using the existing APIs.

Solution

The proposed workaround/fix is to have the SunJSSE implementation encode Strings directly as ISO_8859_1/LATIN-1 characters which correctly handles the full range of byte values (U+0000-U+00FF). (i.e. string.getBytes(Charset.forName("ISO_8859_1");) This change will enhance interoperabibility with other implementations such as Chromium/OpenSSL/etc.

Other UNICODE String values in the range of U+0080-U+10FFFF must now be correctly encoded/decoded (e.g. UTF-8) by applications before sending/receiving, rather than depending on SunJSSE to automatically perform the (possibly incorrect) encoding. We don't anticipate this to be a significant interoperability issue, since all known/current values in the IETF/IANA TLS ALPN extension list can be encoded as ISO_8859_1/LATIN-1 (of which the 7-bit US-ASCII Charset is a proper subset). While UTF-8 is the logical default choice for character encoding, it is possible other formats are being used and therefore the onus of correctly encoding the values should rest with the application. These characters must now be converted to the format required by the peer.

It was also discovered during code development that if a selected ALPN value contained any characters that map to a multi-byte UTF-8 representation, the server-side code would throw an Exception. Since we haven't received any bug reports for this issue, we feel it is unlikely that this will be a problem in the field. But for compatibility concerns, we also introduce a new Java Security property to reverse this change.

Specification

When encoding ALPN values from Strings to/from network bytes, use the ISO_8859_1/LATIN-1 encoding instead of UTF-8.

For the JDK 11u and JDK 8u backports, a release note will be added highlighting this change. The release note will highlight that :

The ApplicationProtocol String values returned by the methods in the javax.net.ssl.SSLSocket and javax.net.ssl.SSLEngine classes are in the network byte representation sent by the peer. The bytes could be directly compared, or converted to its Unicode String format for comparison.

The release note will also highlight that the String values used in the javax.net.ssl.SSLParameters.setApplicationProtocols(String[] protocols)method must be presented using the network byte representation expected by the peer. For example, if an ALPN String should be exchanged using UTF-8, the String should be converted to its byte[] representation and stored as a byte-oriented String.

Coding examples will be included as per JDK 16 javadoc comments.

For compatibility concerns with older SunJSSEs, we introduce a Java Security property to reverse this change:

#
# The default Character set name (java.nio.charset.Charset.forName()) for
# converting TLS ALPN values between byte arrays and Strings.
#
# jdk.tls.alpnCharset=UTF-8
jdk.tls.alpnCharset=ISO_8859_1

which can be overridden to restore the previous encoding process.