FULL PRODUCT VERSION :
java version " 1.7.0_17 "
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) Client VM (build 23.7-b01, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Tested on Windows 7 (Microsoft Windows [Version 6.1.7601]), but occurs in e..g OpenJDK 7 as well.

A DESCRIPTION OF THE PROBLEM :
When performing Diffie-Hellman key agreement for SSL/TLS, the TLS specification (RFC 5246) says that " Leading bytes of Z that contain all zero bits are stripped before it is used as the pre_master_secret. "

However, com.sun.crypto.provider.DHKeyAgreement.java does not strip leading zero bytes. This causes approximately 1 out 256 SSL/TLS handshakes with DH/DHE cipher suites to fail (when the leading byte happens, by chance, to be zero).


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Start a simple JSSE socket server with -Djavax.net.debug=all.

2. Connect to the server with e.g. OpenSSL command line tool, ensuring that DHE cipher suite gets selected (e.g. " openssl s_client -cipher DHE-RSA-AES128-SHA -connect 192.168.81.1:9999 " ) repeatedly. Other SSL clients can be used -- this is not an OpenSSL bug (see below).

3. Repeat the connection. After a couple of hundred successful connections, the connection will fail with handshake_failure alert.

4. Examine the JSSE debug logs produced by the server: the failed connection will have a PreMaster secret that begins with zero byte
(while all other connections have non-zero byte here). For example:

SESSION KEYGEN:
PreMaster Secret:
0000: 00 70 C5 7E 91 38 C8 DE ED 75 3D 76 8A B5 44 69 .p...8...u=v..Di
0010: E7 32 1C EE 80 77 50 C7 A9 51 24 2E E3 15 11 30 .2...wP..Q$....0
0020: 9D F6 9F BC 9D EB 5C 18 F7 A4 19 ED 1A AC 2E 0C ......\.........
0030: E3 18 C5 11 B1 80 07 7D B1 C6 70 A8 D7 EB CF DD ..........p.....
0040: 2D B5 1D BC 01 3E 28 2A 2B 5B 38 8F EB 20 F2 A2 -....>(*+[8.. ..
0050: 00 07 47 F7 87 B8 99 CB EF B4 13 04 C8 8B 82 FB ..G.............


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Expected result is that every connection succeed.
ACTUAL -
Roughly one out of 256 connections fail.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Java server:

import javax.net.ssl.SSLServerSocket;
import javax.net.ssl.SSLServerSocketFactory;
import javax.net.ssl.SSLSocket;

public class TestServer {
    public static void main(String args[]) throws Exception {
SSLServerSocketFactory ssf = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
SSLServerSocket ss = (SSLServerSocket) ssf.createServerSocket(9999);
System.out.println( " Listening on port 9999 " );
for (String cs : ss.getEnabledCipherSuites()) {
    System.out.println(cs);
}
while (true) {
    SSLSocket s = (SSLSocket) ss.accept();
    System.out.println( " Connected with " +s.getSession().getCipherSuite());
    s.close();
}
    }
}

Run as as follows:

keytool -storepass " password " -keypass " password " -genkey -keyalg RSA -keystore test_keystore.jks -dname CN=test
javac TestServer.java
java -Djavax.net.debug=all -Djavax.net.ssl.keyStore=./test_keystore.jks -Djavax.net.ssl.keyStorePassword=password TestServer

OpenSSL client:

set -e
while true; do
  openssl s_client -cipher DHE-RSA-AES128-SHA -connect 127.0.0.1:9999 -quiet -no_ign_eof < /dev/null
done



---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Disable Diffie-Hellman cipher suites.