FULL PRODUCT VERSION :
java version "1.8.0_172-ea"
Java(TM) SE Runtime Environment (build 1.8.0_172-ea-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b03, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux 32-<host name>.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Setting system property jdk.tls.ephemeralDHKeySize to value "2048" does not result in a DH key size of 2048 being used for TLS handshake.

Used the code below to attempt a connection to a server with jdk.tls.disabledAlgorithms set to "DH keySize < 2048" and the connection fails. When the required key size is reduced to 1024 the connection is successful.

Debug output appears to show that the client key size is 1024 even with jdk.tls.ephemeralDHKeySize property set to 2048

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Set up an HTTPS server running on one system with the security property jdk.tls.disabledAlgorithms set to include the restriction: "DH keySize < 2048"

Create a simple Java client using the HttpsUrlConnection class that attempts to connect to the server with cipher suites limited to the list shown below by the system property https.cipherSuites and the jdk.tls.ephemeralDHKeySize property set to the value "2048".



Enabled ciphers:

TLS_DHE_DSS_WITH_AES_128_CBC_SHA",
         "TLS_DHE_DSS_WITH_AES_128_CBC_SHA256", "TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
         "TLS_DHE_DSS_WITH_AES_256_CBC_SHA", "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256",
         "TLS_DHE_DSS_WITH_AES_256_GCM_SHA384", "TLS_DHE_RSA_WITH_AES_128_CBC_SHA",
         "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256",
         "TLS_DHE_RSA_WITH_AES_256_CBC_SHA", "TLS_DHE_RSA_WITH_AES_256_CBC_SHA256",
         "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The connection from client to server should be successful.
ACTUAL -
The connection fails during the SSL handshake.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Using javax.net.debug=all, the following output is captured:

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
main, setSoTimeout(0) called
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 for TLSv1
No available cipher suite for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 for TLSv1.1
No available cipher suite for TLSv1.1
%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie: GMT: 1501971517 bytes = { 89, 190, 4, 122, 58, 169, 14, 250, 222, 157, 237, 45, 135, 117, 168, 32, 235, 34, 242, 163, 202, 114, 95, 181, 26, 212, 91, 192 }
Session ID: {}
Cipher Suites: [TLS_DHE_RSA_WITH_AES_256_GCM_SHA384]
Compression Methods: { 0 }
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
Extension extended_master_secret
Extension renegotiation_info, renegotiated_connection: <empty>
***
main, WRITE: TLSv1.2 Handshake, length = 88
main, READ: TLSv1.2 Handshake, length = 1881
*** ServerHello, TLSv1.2
RandomCookie: GMT: 1501971517 bytes = { 123, 32, 173, 43, 14, 134, 186, 113, 92, 204, 175, 43, 169, 223, 200, 84, 37, 243, 35, 239, 24, 86, 144, 239, 208, 139, 24, 36 }
Session ID: {90, 134, 68, 61, 120, 205, 134, 122, 71, 49, 235, 46, 196, 117, 178, 138, 238, 59, 5, 243, 8, 131, 68, 153, 60, 89, 105, 36, 161, 72, 114, 52}
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
Extension extended_master_secret
***
%% Initialized: [Session-6, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384]
** TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=plynch-ha-vm1.prx.eng.westminster.polycom.com, DC=prx, DC=eng, DC=westminster, DC=polycom, DC=com, OU=Self Signed Certificate, O=Polycom DMA 7000
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key: Sun RSA public key, 2048 bits
  modulus: 27022885122629718910841909624801932077655142347767466818127750551462029003395659244461274932605663817128669689625553078731224353408431444574446140582082183401056643137675603135920202737572672993874580152457696253836279742081505869466965541185327756432878332312431642613119913952025656240058633559660265402008708208927191737761946619822122323378473325890051910866726232890718194754632318062867984995593873310299233290740205486128797638453806221900808715182693337931083362370616728579014026592270180771421082247948920034435335386088795310887295610307103283114592580329927330872772202854863880334639453656657965448345633
  public exponent: 65537
  Validity: [From: Fri Feb 02 07:29:40 MST 2018,
               To: Sun Mar 13 08:29:40 MDT 2022]
  Issuer: CN=plynch-ha-vm1.prx.eng.westminster.polycom.com, DC=prx, DC=eng, DC=westminster, DC=polycom, DC=com, OU=Self Signed Certificate, O=Polycom DMA 7000
  SerialNumber: [ f951d114 275e9f95]

Certificate Extensions: 4
[1]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[2]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

[3]: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

[4]: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: plynch-ha-vm1.prx.eng.westminster.polycom.com
  DNSName: plynch-ha-vm1
  IPAddress: 10.47.17.140
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 7B 6B C4 B8 C3 14 5F F7 66 99 FD E1 53 C6 AF 10 .k...._.f...S...
0010: FA 7F E0 55 74 C1 8A 03 29 9A EF C3 ED A2 97 39 ...Ut...)......9
0020: 2A 2D FA B3 F9 4A D5 11 1C 38 AB 63 69 11 6C F4 *-...J...8.ci.l.
0030: B3 AB CD B1 51 AC 0A E0 35 30 6F 5A F9 D6 DB FE ....Q...50oZ....
0040: 14 3A E0 99 BD AB 07 F5 68 72 B7 9D 87 EA BD F1 .:......hr......
0050: FA 9F DA 9D 0B C1 90 D5 18 41 C8 E0 96 07 80 C9 .........A......
0060: 15 D8 C7 84 13 00 B9 F9 73 1B 3C DA C8 6C 9D 5B ........s.<..l.[
0070: 18 79 EB 66 D6 47 6F FC E6 2A 2A 20 E1 3E A2 42 .y.f.Go..** .>.B
0080: 46 43 0E EC F5 90 F3 E3 29 1F 81 AC 5F 64 67 4E FC......)..._dgN
0090: 81 BD 11 3E D0 30 41 83 67 A7 0E DB 18 FC A4 AE ...>.0A.g.......
00A0: E1 7E 45 CB D9 77 16 9E E4 9C DC F9 B4 2A 36 94 ..E..w.......*6.
00B0: 92 21 AC 5A E3 86 9F 29 78 DC 0B 12 DD 00 82 80 .!.Z...)x.......
00C0: 70 19 C2 A7 D3 FE F3 B2 3E 42 A9 6A E3 1A 3D 95 p.......>B.j..=.
00D0: 8F 49 C9 74 F1 EE 9E 2C 57 21 BE 61 37 6A 85 D1 .I.t...,W!.a7j..
00E0: 2A 0F 2E 5D 64 7F B8 C0 E6 A0 BF 77 4F 9F BF 70 *..]d......wO..p
00F0: 4A 82 99 F6 0B D5 AC 16 05 05 DA F4 8A 59 AB AD J............Y..

]
***
*** Diffie-Hellman ServerKeyExchange
DH Modulus: { 255, 255, 255, 255, 255, 255, 255, 255, 201, 15, 218, 162, 33, 104, 194, 52, 196, 198, 98, 139, 128, 220, 28, 209, 41, 2, 78, 8, 138, 103, 204, 116, 2, 11, 190, 166, 59, 19, 155, 34, 81, 74, 8, 121, 142, 52, 4, 221, 239, 149, 25, 179, 205, 58, 67, 27, 48, 43, 10, 109, 242, 95, 20, 55, 79, 225, 53, 109, 109, 81, 194, 69, 228, 133, 181, 118, 98, 94, 126, 198, 244, 76, 66, 233, 166, 55, 237, 107, 11, 255, 92, 182, 244, 6, 183, 237, 238, 56, 107, 251, 90, 137, 159, 165, 174, 159, 36, 17, 124, 75, 31, 230, 73, 40, 102, 81, 236, 230, 83, 129, 255, 255, 255, 255, 255, 255, 255, 255 }
DH Base: { 2 }
Server DH Public Key: { 165, 251, 168, 172, 3, 223, 75, 2, 148, 203, 60, 236, 239, 150, 242, 146, 126, 182, 168, 186, 97, 158, 214, 10, 164, 15, 228, 18, 216, 134, 151, 206, 247, 33, 72, 111, 45, 62, 44, 169, 211, 58, 101, 204, 202, 88, 152, 232, 181, 187, 21, 27, 235, 73, 116, 42, 61, 203, 45, 149, 80, 152, 246, 245, 124, 78, 202, 209, 210, 203, 195, 197, 85, 8, 213, 178, 78, 91, 40, 114, 154, 52, 128, 155, 193, 177, 235, 120, 90, 34, 173, 5, 233, 25, 4, 234, 120, 157, 56, 180, 125, 25, 16, 27, 91, 234, 156, 17, 178, 63, 73, 134, 134, 157, 177, 177, 245, 170, 105, 50, 50, 43, 213, 89, 220, 209, 244, 124 }
Anonymous
*** ServerHelloDone
*** ClientKeyExchange, DH
DH Public key: { 44, 206, 109, 189, 80, 20, 236, 243, 36, 80, 134, 226, 41, 190, 10, 61, 217, 58, 216, 157, 187, 235, 67, 225, 85, 52, 39, 152, 254, 146, 46, 134, 127, 216, 90, 87, 28, 114, 96, 240, 87, 82, 191, 58, 206, 201, 224, 241, 179, 107, 193, 192, 30, 35, 104, 59, 174, 36, 33, 141, 48, 242, 86, 161, 43, 191, 141, 171, 8, 123, 195, 112, 76, 76, 210, 252, 7, 53, 115, 55, 79, 215, 112, 251, 220, 221, 12, 106, 63, 136, 230, 220, 29, 221, 106, 218, 106, 216, 153, 193, 157, 144, 78, 244, 51, 72, 34, 68, 228, 161, 36, 104, 28, 33, 176, 134, 53, 90, 116, 34, 98, 130, 3, 183, 210, 170, 214, 21 }
main, WRITE: TLSv1.2 Handshake, length = 134
SESSION KEYGEN:
PreMaster Secret:
0000: FE 1F F0 15 D0 06 5F 90 AC 50 47 00 7E 09 AE FF ......_..PG.....
0010: CD CE 00 82 79 25 7E 59 1A 5E 9B 14 21 5A 72 FF ....y%.Y.^..!Zr.
0020: D2 47 F0 E9 20 44 07 D6 FB DD 23 F3 D3 58 D9 C9 .G.. D....#..X..
0030: 2E 10 4F B6 19 8A 55 9E 36 6A 44 1D 59 51 44 6C ..O...U.6jD.YQDl
0040: 5D DE 44 33 7B 9C 58 3B CD 3B F0 8D 5D 47 82 7D ].D3..X;.;..]G..
0050: 3D 16 36 C7 8B 22 FD D3 7F B7 78 39 2E DA CB 64 =.6.."....x9...d
0060: 6C 13 AC 32 AE 0B EF AA 65 53 21 AB A9 6B 54 07 l..2....eS!..kT.
0070: 4C B3 7B 02 11 76 D1 F5 3C 6E D0 F6 87 DF 98 54 L....v..<n.....T
CONNECTION KEYGEN:
Client Nonce:
0000: 5A 86 44 3D 59 BE 04 7A 3A A9 0E FA DE 9D ED 2D Z.D=Y..z:......-
0010: 87 75 A8 20 EB 22 F2 A3 CA 72 5F B5 1A D4 5B C0 .u. ."...r_...[.
Server Nonce:
0000: 5A 86 44 3D 7B 20 AD 2B 0E 86 BA 71 5C CC AF 2B Z.D=. .+...q\..+
0010: A9 DF C8 54 25 F3 23 EF 18 56 90 EF D0 8B 18 24 ...T%.#..V.....$
Master Secret:
0000: 3E 48 F7 A3 B7 E0 DA 77 96 68 8A 19 D0 4D 15 5B >H.....w.h...M.[
0010: 8C F1 E0 DF A0 80 3F E9 3D 4E 28 90 29 32 3C 0B ......?.=N(.)2<.
0020: 11 80 C3 8B 1A 32 78 51 D2 82 FE 5F 18 2F 46 0B .....2xQ..._./F.
... no MAC keys used for this cipher
Client write key:
0000: 4F 03 C2 CB DB 00 2F 9B 44 0A DF B1 A5 7E 7A 63 O...../.D.....zc
0010: ED 0C BF 2C 7E C8 CC 45 D7 C8 B6 CB A0 AE B1 B7 ...,...E........
Server write key:
0000: 28 41 1B 68 43 F9 E3 81 34 DE 90 13 C3 9A 7F 87 (A.hC...4.......
0010: E6 C7 30 66 10 C5 7D 35 58 5F 1F 47 0F B1 BA 5B ..0f...5X_.G...[
Client write IV:
0000: 72 35 83 23 r5.#
Server write IV:
0000: 3F 1C C0 D6 ?...
main, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 234, 236, 113, 223, 228, 48, 151, 171, 73, 202, 126, 172 }
***
main, WRITE: TLSv1.2 Handshake, length = 40
main, waiting for close_notify or alert: state 1
main, received EOFException: error
main, Exception while waiting for close javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
%% Invalidated: [Session-6, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384]
main, SEND TLSv1.2 ALERT: fatal, description = handshake_failure
main, WRITE: TLSv1.2 Alert, length = 26
main, Exception sending alert: java.net.SocketException: Broken pipe (Write failed)
main, called closeSocket()


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Assumption 1: Server is a Java 8 based HTTPS server listening on port 8443 with jdk.tls.disabledAlgorithms set to include the restriction: DH keySize < 2048

Assumption 2: HttpsURLConnection class has been configured with a default SSL socket factory that will trust the server certificate and has been provided with a default HostnameVerifier that will successfully verify the server hostname.

Code Snippet:

         URL url = new URL("https",<server ip-address>,8443, "/");
         HttpsURLConnection conn;
         try
         {
            conn = (HttpsURLConnection) url.openConnection();
            conn.connect();
            try (final InputStream is = conn.getInputStream())
            {
               System.out.println("Connected with cipher: " + conn.getCipherSuite());
            }
         }
         catch (IOException e)
         {
            e.printStackTrace(System.out);
         }


---------- END SOURCE ----------