FULL PRODUCT VERSION :
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
I use HttpURLConnection to connect from a Swing GUI to an application server. The connection supports tunneling HTTPS through an NTLM proxy.

The GUI uses Java JRE, and uses the HttpURLConnection class

The connection should use keep-alive by default since the client, proxy and server are all using HTTP/1.1. A single socket should be opened from the GUI and multiple HTTP requests should reuse the same socket.

Before Java 1.7.0_40, the HTTPS tunnel through the NTLM proxy would do NTLM authentication on the initial CONNECT request, but not on the following HTTP requests. The GUI would open a single socket and reuse it.

In Java 1.7.0_40 and later JRE versions, each request results in the GUI sending a new CONNECT request to the proxy server, doing the NTLM authentication (successfully), and then creating a new tunnel and new socket to the server.

The impact of this is that each GUI connection to the server takes longer to create (~3000 ms vs ~50 ms if socket was reused) and critical messages like heartbeats can get backed up, leading to disconnects or perceived unresponsiveness.

The HTTP headers between the GUI and the proxy servers appear identical in all the JRE versions, but the Java client's behavior is different. The headers appear to follow what is supposed to be sent, as detailed in this link: http://msdn.microsoft.com/en-us/library/dd925287%28v=office.12%29.aspx

I downloaded and tested the available JREs from the Oracle download site, and found that the behavior appears first in 1.7.0_40, but is OK in 1.7.0_25. It exists in 1.7.0_40, 1.7.0_45, 1.7.0_51, 1.7.0_67 and 1.8.0_20.

I tested using three different NTLM proxy servers although one was a external proxy for which I don't have details.

Squid 3.1.19 for x86_64 11:44 AM
I think the ntlm is squid-2.5-ntlmssp






REGRESSION. Last worked in version 7u25

ADDITIONAL REGRESSION INFORMATION:
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
See the attached sample code. There is a sample command line in the comments.

You need to provide a target HTTPS URL, an NTLM proxy server and port and user info.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
In the logging, I would expect to see that the local socket number does not change, so the count of the local sockets remains at 1.

Also, the CONNECT and NTLM authentication headers should appear in the first request, but not in the subsequent ones.
ACTUAL -
I see that the local socket number changes after each request, indicating a new socket is used for each request.

Also the headers show that the CONNECT to the proxy and NTLM authentication happens each time as well.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.IOException;
import java.lang.reflect.Field;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.PasswordAuthentication;
import java.net.Socket;
import java.net.URL;
import java.net.URLConnection;
import java.util.logging.Handler;
import java.util.logging.Level;
import java.util.logging.Logger;

import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSession;

/**
 * Run HTTPKeepAliveTester to connect to a HTTP server and log all HTTP headers.
 * It also counts the number of local sockets opened to the server to check if keepalive
 * is being used, and displays the time each connection request took.
 *
 * You can specify various -D JVM args:
 *
 * -Durl=<url>
 * -Dhttp.proxyHost=<host>
 * -Dhttp.proxyPort=<port>
 * -Dhttp.proxyUser=<user>
 * -Dhttp.proxyPassword=<pass>
 * -Dhttp.auth.ntlm.domain=<domain>
 * -Dn=<number of iterations. Defaults to 5. Use 0 for continuous>
 * -Ddelay=<seconds between iterations>
 *
 * Sample command line to run this test:
 *
 * %JAVA_HOME%\bin\java.exe -Durl=https://&lt;your_ssl_server.com> -Dhttp.proxyHost=10.6.160.94 -Dhttp.proxyPort=8080 -Dhttp.proxyUser=steve -Dhttp.proxyPassword=hello -Dhttp.auth.ntlm.domain=SAMPLENTDOMAIN.com -cp HTTPKeepAliveTester.jar HTTPKeepAliveTester
 *
 */
public class HTTPKeepAliveTester {

private static final Logger LOGGER = Logger.getLogger("HTTPKeepAliveTester");

static {
// Set this to ignore server certificate problems while testing.
setDefaultHostnameVerifier();

// Enable HttpURLConnection debug logging to print out the outgoing and
// incoming HTTP header. Get the root logger
Logger rootLogger = Logger.getLogger("");
for (Handler handler : rootLogger.getHandlers()) {
// Change log level of default handler(s) of root logger
handler.setLevel(Level.FINEST);
}
// Set root logger level to FINE to allow HttpURLConnection debug
// logging to appear
rootLogger.setLevel(Level.FINE);
}

/**
* This class will make a HTTP/S connection to the server.
*/
public static void main(String[] args) throws Exception {

// Usage help
if (args.length > 0 && (args[0].equals("?") || args[0].contains("help"))) {
LOGGER.info("Options: -Durl=<url> -Dhttp.proxyHost=<host> -Dhttp.proxyPort=<port> -Dhttp.proxyUser=<user> -Dhttp.proxyPassword=<pass> -Dhttp.auth.ntlm.domain=<domain> -Dn=<number of iterations. Defaults to 5. Use 0 for continuous> -Ddelay=<seconds between iterations>");
}

String urlString = System.getProperty("url");
String proxyHost = System.getProperty("http.proxyHost");
String proxyPort = System.getProperty("http.proxyPort");
String proxyUser = System.getProperty("http.proxyUser");
String proxyPassword = System.getProperty("http.proxyPassword");
String authNtlmDomain = System.getProperty("http.auth.ntlm.domain");
String numStr = System.getProperty("n", "5");
String delayStr = System.getProperty("delay", "1");

LOGGER.info("--------------------------------------------------------------------------------------------------------------\n\n");
LOGGER.info("url: " + urlString);
LOGGER.info("http.proxyHost: " + proxyHost);
LOGGER.info("http.proxyPort: " + proxyPort);
LOGGER.info("http.proxyUser: " + proxyUser);
LOGGER.info("http.proxyPassword: " + (proxyPassword == null ? null : "****"));
LOGGER.info("http.auth.ntlm.domain: " + authNtlmDomain);
LOGGER.info("n: " + numStr);
LOGGER.info("delay: " + delayStr + "\n\n");
LOGGER.info("--------------------------------------------------------------------------------------------------------------\n\n");

// Set https proxy settings from http settings
if (proxyHost != null) {
System.setProperty("https.proxyHost", proxyHost);
}
if (proxyPort != null) {
System.setProperty("https.proxyPort", proxyPort);
}

// URL
URL url = new URL(urlString);

// Set proxy authenticator if required.
if (proxyHost != null && proxyUser != null) {
Authenticator.setDefault(new ProxyAuthenticator(proxyUser, proxyPassword));
}

// Variables to track number of local sockets created for sending.
// Should only be one if keepalive is used.
int currentPort = -1;
int localPortCount = 0;

int num = Integer.parseInt(numStr);
// If n=0, continue for a long time...
if (num == 0) {
num = Integer.MAX_VALUE;
}
int delay = Integer.parseInt(delayStr);
// Prevent delay < 1 second
if (delay < 1) {
delay = 1;
}

for (int i = 0; i < num; i++) {
Long startTime = System.currentTimeMillis();

URLConnection urlConnection = url.openConnection();

urlConnection.setUseCaches(false);
urlConnection.setAllowUserInteraction(false);
urlConnection.setRequestProperty("Cache-Control", "no-cache");
urlConnection.setRequestProperty("accept", "*");

// Use GET protocol to connect.
urlConnection.connect();

// After making the connection, check the local port number to see
// if we are using keepalive or creating new sockets...
Socket newSocket = null;
if (urlConnection instanceof HttpsURLConnection) {
newSocket = extractHttpsSock((HttpsURLConnection) urlConnection);
} else if (urlConnection instanceof HttpURLConnection) {
newSocket = extractHttpSock((HttpURLConnection) urlConnection);
}

if (newSocket.getLocalPort() != currentPort) {
currentPort = newSocket.getLocalPort();
localPortCount++;
}

// Process the response.
HttpURLConnection conn = (HttpURLConnection) urlConnection;

// NOTE: You MUST call getResponseCode and getInputStream, or the
// request will not actually get sent!!!!
try {
conn.getResponseCode();
conn.getInputStream();
} catch (IOException e) {
// If a bad response code is returned, like a 401, an
// IOException will be thrown when you try
// to get the response code or the input stream. In that case,
// read from the error stream instead.
conn.getResponseCode();
conn.getErrorStream();
}

// You must close the connection's inputstream in order for the socket to
// be reused for keep-alive. See: http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
conn.getInputStream().close();

LOGGER.info("\nRequest duration (ms): " + (System.currentTimeMillis() - startTime) + ". Local port creation count: " + localPortCount + "\n\n");
LOGGER.info("--------------------------------------------------------------------------------------------------------------\n\n");

// Sleep between requests
Thread.sleep(delay * 1000);
}
}

/**
* Override default certificate host name verifier to ignore hostname check when testing.
*/
static public void setDefaultHostnameVerifier() {
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
});
}

/**
* Get the socket from the HttpsURLConnection object.
*/
static private Socket extractHttpsSock(Object obj) throws Exception {
Field f = extractField(obj, "delegate");
if (f == null)
throw new NoSuchFieldException("Declared field 'delegate' not found");
return extractHttpSock(f.get(obj));
}

/**
* Get the socket from the HttpURLConnection object.
*/
static private Socket extractHttpSock(Object obj) throws Exception {
Field f = extractField(obj, "http");
if (f == null)
throw new NoSuchFieldException("Declared field 'http' not found");
Object o = f.get(obj);
f = extractField(o, "serverSocket");
if (f == null)
throw new NoSuchFieldException("Declared field 'serverSocket' not found");
return (Socket) f.get(o);
}

/**
* Use reflection to get access to a private field.
*/
static private Field extractField(Object obj, String name) {
Class<?> c = obj.getClass();
while (c != null) {
try {
Field f = c.getDeclaredField(name);
f.setAccessible(true);
return f;
} catch (Exception ex) {
}
;
c = c.getSuperclass();
}
return null;
}

/**
* Proxy Authenticator - Used to provide a real password authenticator.
* Default Authenticator.getPasswordAuthentication() method just returns
* null.
*/
private static final class ProxyAuthenticator extends Authenticator {

private String username;
private String password;

private ProxyAuthenticator(String username, String password) {
this.username = username;
this.password = password;
}

protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(username, password.toCharArray());
}
}

}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Need to keep GUI on earlier java versions which work.