ADDITIONAL SYSTEM INFORMATION :
Windows 10 / Java 11.0.2
OS agnostic.

A DESCRIPTION OF THE PROBLEM :
As requested on OpenJDK security-dev, I'd like to create an RFE to improve Java 11 JSSE interoperability in TLS 1.2 mode with existing and perhaps older and less tolerant TLS stacks. It's currently not possible to make a TLS 1.2 client hello in the handshake to look exactly like the one from Java 8, 9, 10. Some of the current behavior is hardcoded and not exposed for configuration, I'd like to open a request to make all of the aspects of the handshake configurable.

I have a sample socket client that can handshake with TLS 1.2 on most servers, however with some devices it cannot negotiate due to inclusion of newer TLS extensions even in TLS 1.2 mode which help with conformance with TLS 1.3 but at a cost of incompatibility with older servers.
With TLS 1.3 support in JDK some of the extensions are not controllable via properties and hence the only way to ensure interop is to make code changes to drop TLS level for these devices possibly compromising security to boot.

It appears some of the issues can come from
- inclusion of RSASSA-PSS alg in TLS 1.2 handshakes but these can disabled if required

-signature_algorithms_cert and supported_versions extensions which seem to be hardcoded for TLS 1.2 (I was not able to conclusively identify which of these caused my troubles)

https://tools.ietf.org/html/rfc8446#section-1.3 does say that TLS 1.2 clients are affected but in an optional manner.

To reproduce I've used as sample socket client created on top of JDK but any such client will do, the problem was apparent only when talking to a specific hardware device which refused to negotiate TLS session (I've seen several odd TLS implementations that were intolerant to Java changes in various ways over the years and compatibility could always be assured through config changes, this time around less so).

Some of the hardware TLS stacks can range from small oddities to being completely broken by small changes as they can contain outdated and poorly implemented TLS stacks that are very sensitive so even a small change can break them and thats why its always important to have levers provided to control almost every aspect of the handshake.

I have a sample in my gist (https://gist.github.com/amir-khassaia/04347ca88526f4b958b3326968a905c0), which is in Kotlin. When ran with java 8, 9, 10 there were no issues. With java 11 this worked on most devices but I've had a device at a remote location that was not in my control that I've had to diagnose the handshake failure on using java 11 it was intolerant to TLS 1.2 client hello from Java 11 but fine with TLS 1.1 as the new extensions are not present. It would be fine with TLS 1.2 client hello from Java 10 and earlier as I mentioned.

Javax.net.debug output
-------------------------------
Wireshark TLS 1.2 Java 8 client hello
-------------------------------------------------
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 157
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 153
            Version: TLS 1.2 (0x0303)
            Random: 5c34044c709feae39585e4db8e41b0170fbf9fa428b38941...
                GMT Unix Time: Jan 8, 2019 13:00:44.000000000 AUS Eastern Daylight Time
                Random Bytes: 709feae39585e4db8e41b0170fbf9fa428b38941983ddb53...
            Session ID Length: 0
            Cipher Suites Length: 44
            Cipher Suites (22 suites)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02d)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 (0xc031)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 (0x00a2)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 68
            Extension: supported_groups (len=22)
                Type: supported_groups (10)
                Length: 22
                Supported Groups List Length: 20
                Supported Groups (10 groups)
                    Supported Group: secp256r1 (0x0017)
                    Supported Group: secp384r1 (0x0018)
                    Supported Group: secp521r1 (0x0019)
                    Supported Group: sect283k1 (0x0009)
                    Supported Group: sect283r1 (0x000a)
                    Supported Group: sect409k1 (0x000b)
                    Supported Group: sect409r1 (0x000c)
                    Supported Group: sect571k1 (0x000d)
                    Supported Group: sect571r1 (0x000e)
                    Supported Group: secp256k1 (0x0016)
            Extension: ec_point_formats (len=2)
                Type: ec_point_formats (11)
                Length: 2
                EC point formats Length: 1
                Elliptic curves point formats (1)
                    EC point format: uncompressed (0)
            Extension: signature_algorithms (len=28)
                Type: signature_algorithms (13)
                Length: 28
                Signature Hash Algorithms Length: 26
                Signature Hash Algorithms (13 algorithms)
                    Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA256 DSA (0x0402)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: SHA224 ECDSA (0x0303)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: SHA224 RSA (0x0301)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA224 DSA (0x0302)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: ecdsa_sha1 (0x0203)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA1 DSA (0x0202)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: DSA (2)
            Extension: extended_master_secret (len=0)
                Type: extended_master_secret (23)
                Length: 0



Wireshark Java 11 TLS 1.2 Client hello
----------------------------------------------------
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 185
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 181
            Version: TLS 1.2 (0x0303)
            Random: 37f32691301b6b9d45bb62c6268915819881b8ebd95f152c...
                GMT Unix Time: Sep 30, 1999 19:00:01.000000000 AUS Eastern Standard Time
                Random Bytes: 301b6b9d45bb62c6268915819881b8ebd95f152c41c7e483...
            Session ID Length: 0
            Cipher Suites Length: 10
            Cipher Suites (5 suites)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 130
            Extension: supported_groups (len=10)
                Type: supported_groups (10)
                Length: 10
                Supported Groups List Length: 8
                Supported Groups (4 groups)
                    Supported Group: secp256r1 (0x0017)
                    Supported Group: secp384r1 (0x0018)
                    Supported Group: secp521r1 (0x0019)
                    Supported Group: secp160k1 (0x000f)
            Extension: ec_point_formats (len=2)
                Type: ec_point_formats (11)
                Length: 2
                EC point formats Length: 1
                Elliptic curves point formats (1)
                    EC point format: uncompressed (0)
            Extension: signature_algorithms (len=42)
                Type: signature_algorithms (13)
                Length: 42
                Signature Hash Algorithms Length: 40
                Signature Hash Algorithms (20 algorithms)
                    Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (4)
                    Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (5)
                    Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (6)
                    Signature Algorithm: rsa_pss_pss_sha256 (0x0809)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (9)
                    Signature Algorithm: rsa_pss_pss_sha384 (0x080a)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (10)
                    Signature Algorithm: rsa_pss_pss_sha512 (0x080b)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (11)
                    Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA256 DSA (0x0402)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: SHA224 ECDSA (0x0303)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: SHA224 RSA (0x0301)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA224 DSA (0x0302)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: ecdsa_sha1 (0x0203)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA1 DSA (0x0202)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: MD5 RSA (0x0101)
                        Signature Hash Algorithm Hash: MD5 (1)
                        Signature Hash Algorithm Signature: RSA (1)
            Extension: signature_algorithms_cert (len=42)
                Type: signature_algorithms_cert (50)
                Length: 42
                Signature Hash Algorithms Length: 40
                Signature Hash Algorithms (20 algorithms)
                    Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (4)
                    Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (5)
                    Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (6)
                    Signature Algorithm: rsa_pss_pss_sha256 (0x0809)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (9)
                    Signature Algorithm: rsa_pss_pss_sha384 (0x080a)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (10)
                    Signature Algorithm: rsa_pss_pss_sha512 (0x080b)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (11)
                    Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA256 DSA (0x0402)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: SHA224 ECDSA (0x0303)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: SHA224 RSA (0x0301)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA224 DSA (0x0302)
                        Signature Hash Algorithm Hash: SHA224 (3)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: ecdsa_sha1 (0x0203)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: SHA1 DSA (0x0202)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Algorithm: MD5 RSA (0x0101)
                        Signature Hash Algorithm Hash: MD5 (1)
                        Signature Hash Algorithm Signature: RSA (1)
            Extension: extended_master_secret (len=0)
                Type: extended_master_secret (23)
                Length: 0
            Extension: supported_versions (len=5)
                Type: supported_versions (43)
                Length: 5
                Supported Versions length: 4
                Supported Version: TLS 1.2 (0x0303)
                Supported Version: TLS 1.1 (0x0302)
            Extension: renegotiation_info (len=1)
                Type: renegotiation_info (65281)
                Length: 1
                Renegotiation Info extension
                    Renegotiation info extension length: 0