URLConnection for HTTPS connection through Proxy w/ Digest Authentication gives 400 Bad Request

URLConnection fails with error 400 (Bad Request) for an HTTPS connection
through a Proxy server, that is configured to request Digest Authentication.

1. Testcase
-----------
Please find attached the following testcase: proxy-bug.jar

% jar tvf proxy-bug.jar
     0 Wed Apr 09 14:25:58 MEST 2008 META-INF/
    44 Wed Apr 09 14:25:58 MEST 2008 META-INF/MANIFEST.MF
  3526 Tue Feb 05 13:32:14 MET 2008 org/jsantander/proxybug/ProxyBug.java
 19325 Tue Feb 05 14:19:10 MET 2008 org/jsantander/proxybug/StoreCertTrustManager.java
%

The tescase will need a Forward Proxy server with "Digest Authentication".
Customer used Apache 2.2.6.

The testcase needs to be called like follows:

java -classpath . -DPROXY_USER=<user> -DPROXY_PASSWORD=<password> \
                  -DPROXY_HOST=<proxy-address> -DPROXY_PORT=<proxy-port> \
                  org.jsantander.proxybug.ProxyBug <https-url>

where <user>, <password>, <proxy-address>, <proxy-port> and <https-url> are the parameters
appropriate for the environment.


2. Behaviour
------------
The testcase does the following :

 1.- Create an HTTPS URL.
 2.- Obtain an URLConnection with url.openConnection() providing a Proxy
 3.- Setup the SSLSocketFactory.
 4.- Setup the default Authenticator.
 5.- Read result from connection

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
 1.- Send CONNECT HTTP Request to the Proxy
 2.- Receive a 407 Proxy Authentication Required
 3.- Send new CONNECT with authentication credentials.
 4.- Receive 200 OK. The tunnel is established.
 5.- Send GET through the HTTPS tunnel
 6.- Receive HTTPS Response
ACTUAL -
 1.- Send CONNECT HTTP Request to the Proxy
 2.- Receive a 407 Proxy Authentication Required
 3.- Send new CONNECT with authentication credentials.
 4.- Receive 400 Bad Request


Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\>"c:\Program Files\Java\jre1.6.0_05\bin\java"
-cp . -DPROXY_USER=jsantander -DPROXY_PASSWORD=jsantander123
-DPROXY_HOST=127.0.0.1 -DPROXY_PORT=8000 org.jsantander.proxybug.ProxyBug https://eas3nc1.ascc.lucent.com:8943/msb
09-abr-2008 14:56:07 org.jsantander.proxybug.StoreCertTrustManager <init>
INFO: StoreCertTrustManager is beginning!
09-abr-2008 14:56:07 org.jsantander.proxybug.StoreCertTrustManager <init>
INFO: Leaving StoreCertTrustManager!
java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 400 Bad Request"
        at sun.net.www.protocol.http.HttpURLConnection.doTunneling(Unknown Source)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
        at org.jsantander.proxybug.ProxyBug.main(ProxyBug.java:128)

C:\>
C:\>"c:\Program Files\Java\jre1.6.0_05\bin\java" -version
java version "1.6.0_05"
Java(TM) SE Runtime Environment (build 1.6.0_05-b13)
Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode, sharing)
      

3. Symptoms
-----------
3.1 Apache log
--------------
 Apache's error log shows:
[Tue Feb 05 15:00:56 2008] [error] [client 135.92.88.74] Digest: uri mismatch
 - </wps/portal/ntchangepasswd> does not match request-uri <all.alcatel-lucent.com:443>


3.2 Etherreal trace
-------------------
Ethereal trace of the exchange

>> Frame 4 (220 bytes on wire, 220 bytes captured)
>> Ethernet II, Src: AlcatelN_8a:be:77 (00:d0:95:8a:be:77), Dst: Foxconn_7b:53:e1
>> (00:15:58:7b:53:e1)
>> Internet Protocol, Src: 135.92.88.74 (135.92.88.74), Dst: 159.23.107.247 (159.23.107.247)
>> Transmission Control Protocol, Src Port: 56859 (56859), Dst Port: 8000 (8000),
>> Seq: 1, Ack: 1, Len: 166
>> Hypertext Transfer Protocol
>> CONNECT all.alcatel-lucent.com:443 HTTP/1.1\r\n
>> User-Agent: Java/1.5.0_13\r\n
>> Host: all.alcatel-lucent.com\r\n
>> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n
>> \r\n
>>
>> Frame 5 (881 bytes on wire, 881 bytes captured)
>> Ethernet II, Src: Foxconn_7b:53:e1 (00:15:58:7b:53:e1), Dst: IETF-VRRP-virtual-router-VRID_cf
>> (00:00:5e:00:01:cf)
>> Internet Protocol, Src: 159.23.107.247 (159.23.107.247), Dst: 135.92.88.74 (135.92.88.74)
>> Transmission Control Protocol, Src Port: 8000 (8000), Dst Port: 56859 (56859),
>> Seq: 1, Ack: 167, Len: 827
>> Hypertext Transfer Protocol
>> HTTP/1.1 407 Proxy Authentication Required\r\n
>> Date: Tue, 05 Feb 2008 14:00:56 GMT\r\n
>> Proxy-Authenticate: Digest realm="Restricted Files",
>> nonce="K/Bnq2lFBAA=5da90dd82f45ce069133f035478844b5a08819c5", algorithm=MD5, qop="auth"\r\n
>> Content-Length: 536
>> Content-Type: text/html; charset=iso-8859-1\r\n
>> \r\n
>> Line-based text data: text/html
>>
>> Frame 14 (531 bytes on wire, 531 bytes captured)
>> Ethernet II, Src: AlcatelN_8a:be:77 (00:d0:95:8a:be:77), Dst: Foxconn_7b:53:e1
>> (00:15:58:7b:53:e1)
>> Internet Protocol, Src: 135.92.88.74 (135.92.88.74), Dst: 159.23.107.247 (159.23.107.247)
>> Transmission Control Protocol, Src Port: 56860 (56860), Dst Port: 8000 (8000),
>> Seq: 1, Ack: 1, Len: 477
>> Hypertext Transfer Protocol
>> CONNECT all.alcatel-lucent.com:443 HTTP/1.1\r\n
>> User-Agent: Java/1.5.0_13\r\n
>> Host: all.alcatel-lucent.com\r\n
>> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n
>> Proxy-Authorization: Digest username="jsantander", realm="Restricted Files",
>> nonce="K/Bnq2lFBAA=5da90dd82f45ce069133f035478844b5a08819c5", nc=00000001,
>> uri="/wps/portal/ntchangepasswd",
>> response="5fb851590d4cdabd402613cd429815d2", algorith
>> \r\n
>>
>> Frame 15 (551 bytes on wire, 551 bytes captured)
>> Ethernet II, Src: Foxconn_7b:53:e1 (00:15:58:7b:53:e1), Dst: IETF-VRRP-virtual-router-VRID_cf
>> (00:00:5e:00:01:cf)
>> Internet Protocol, Src: 159.23.107.247 (159.23.107.247), Dst: 135.92.88.74 (135.92.88.74)
>> Transmission Control Protocol, Src Port: 8000 (8000), Dst Port: 56860 (56860),
>> Seq: 1, Ack: 478, Len: 497
>> Hypertext Transfer Protocol
>> HTTP/1.1 400 Bad Request\r\n
>> Date: Tue, 05 Feb 2008 14:00:56 GMT\r\n
>> Content-Length: 347
>> Connection: close\r\n
>> Content-Type: text/html; charset=iso-8859-1\r\n
>> \r\n
>> Line-based text data: text/html
 

3.3 Apache's HttpClient
-----------------------
Problem can be cured by using Apache's HttpClient (http://hc.apache.org/httpclient-3.x/).
4. How to set up Apache2 as Forward Proxy Server w/ Digest Authentication
-------------------------------------------------------------------------

 I used Solaris 10 Apache2 2.0.52:

# /usr/apache2/bin/httpd -V
Server version: Apache/2.0.52
[ ... ]

 4.1 configure apache2
 ----------------------
# cd /etc/apache2
# cp httpd.conf-example httpd.conf

 make the following changes to httpd.conf

 4.1.1 change the following three lines
 --------------------------------------
 4.1.1.1
    Listen 80
 into
    Listen 8080

 4.1.1.2
     ServerAdmin ###@###.###
 into
     ServerAdmin <your_name>@sun.com

 4.1.1.3
     ServerName 127.0.0.1
 into
     ServerName <your_server_name>


 4.1.2 add the following lines after the "<IfModule mod_ssl.c>" block
 ---------------------------------------------------------------------
 add before line "### Section 3: Virtual Hosts"


  ProxyRequests On

<Proxy *>

  AuthType Digest
  AuthName "Private"
  AuthDigestFile /usr/local/apache/passwd/digest
  Require valid-user

</Proxy>


 4.2 configure Digest Authentication user
 -----------------------------------------
# mkdir -p /usr/local/apache/passwd
# /usr/apache2/bin/htdigest -c /usr/local/apache/passwd/digest Private dummy
Adding password for dummy in realm Private.
New password: dummypw
Re-type new password:
#

# more /usr/local/apache/passwd/digest
dummy:Private:f6bad2cc54d7f1b5667e78ab0a35e839
#

 4.3 Launch apache2 server
 -------------------------
# svcadm enable apache2

 Apache2 server should start up.
 Make sure apache server is running:
# ps -ef | grep apache
webservd 9849 9847 0 16:00:37 ? 0:00 /usr/apache2/bin/httpd -k start
    root 9847 1 0 16:00:36 ? 0:00 /usr/apache2/bin/httpd -k start
    root 9852 9847 0 16:00:37 ? 0:00 /usr/apache2/bin/httpd -k start
webservd 9850 9847 0 16:00:37 ? 0:00 /usr/apache2/bin/httpd -k start
webservd 9848 9847 0 16:00:37 ? 0:00 /usr/apache2/bin/httpd -k start
    root 9854 7283 0 16:00:37 pts/2 0:00 grep apache
webservd 9851 9847 0 16:00:37 ? 0:00 /usr/apache2/bin/httpd -k start
#


 4.4 Run client against apache proxy server
 -------------------------------------------
 possibly from a different host:

 /jdk1.6.0_04/bin/java -classpath . -DPROXY_USER=dummy -DPROXY_PASSWORD=dummypw \
 -DPROXY_HOST=<your_server_name>.<domain>.sun.com -DPROXY_PORT=8080 \
 org/jsantander/proxybug/ProxyBug https://www.sun.com/

 should give

Apr 17, 2008 3:46:41 PM org.jsantander.proxybug.StoreCertTrustManager <init>
INFO: StoreCertTrustManager is beginning!
Apr 17, 2008 3:46:41 PM org.jsantander.proxybug.StoreCertTrustManager <init>
INFO: Leaving StoreCertTrustManager!
java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 400 Bad Request"
        at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:1423)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:164)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:977)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:234)
        at org.jsantander.proxybug.ProxyBug.main(ProxyBug.java:128)
%