Name: mc57594 Date: 12/16/99

On Linux:
java version "1.2"
Classic VM (build Linux_JDK_1.2_pre-release-v2, native threads, nojit)

On Windows NT:
java version "1.2"
Classic VM (build JDK-1.2-V, native threads)


1.Exact steps to reproduce the problem

I am currently developing a kind of a web crawler in JAVA which fills out
forms with data provided by a database.
Some HTTP servers do not accept the form data I send them, e.g. Lotus Notes.
When this happens, I get a 'java.io.FileNotFoundException' (although the URL
where I send the data to is correct). I know that the error code provided by the
HTTP server is NOT correct.
I used a tool called 'hunt' on a Linux system to create a dump of the data my
program sends to the HTTP server.
I also dumped the data sent by Netscape 4.61.
The dumps have only two differences:
a) the order of the header fields
b) after the content, Netscape sends two CR/LF line delimiters, my crawler sends
two LF line delimiters.
I have tried several work arounds, but none solved the problem:
a) adding a "\r\n" th the _outData before I write it out results in a wrong
Content-length header field. Explicitly setting that property "by hands" doesn't
work because the library classes ALWAYS determine the length of the data I send
to the DataOutputStream and ignore my setting.
b) System.setProperty( "line.separator", "\r\n" ); was also ignored.
c) Not sending any data to the HTTP server by commenting out the lines between
/* BEGIN */ and /* END */ without changing anything else, the Exception does not
occur. But of course, the program does not make any sense without providing the
data to the server.

According to the RFCs I read, CR/LF is the prescribed line delimiter in HTTP
protocol and therefore I consider the described problem a bug.



2.Java SOURCE CODE that demonstrates the problem

String _outString = "FIELD1=VALUE+1&FIELD2=VALUE+2"

URL url = new URL("http","myhost",80,"mypath");
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");

conn.setRequestProperty( "Content-type", "application/x-www-form-urlencoded" );
conn.setDoOutput(true);

conn.connect();

/* BEGIN */
URLoutput = new DataOutputStream( conn.getOutputStream() );
URLoutput.writeBytes( _outString );
URLoutput.close();
/* END */

BufferedReader URLinput = new BufferedReader( new
InputStreamReader(conn.getInputStream()));

... reading the data ...

URLinput.close();

conn.disconnect();



3.Exact text of any error message(s) that appeared.

java.io.FileNotFoundException: http://myhost:80/mypath
        at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.jav
a:530)
        at MyProgram ...



4.Any trace information.

see description at 1.



5.Include additional configuration information

NONE.

===================================================
From: "Markus <###@###.###>
To: "chamness"
Date: Mon, 06 Dec 1999 12:50:37 +0100
MIME-Version: 1.0
Subject: Re: (Review ID: 98133) Wrong line delimiters in DataOutputStream of HttpURLConnection

Hi Mark,

Well, here is some more info:
I changed my code accorting to your suggestion, it didn't help.
I attached a ZIP archive containing the complete test program and two
.sniff files that I created on a Linux PC with the network sniffer 'hunt 1.3'
I know that Linux is not the best supported OS by Java, but 'hunt' only
works on Linux and the problem occurs on every OS I tried it (Solaris,
Windows NT, Lunix) similarly. So I think it's OK to use Linux to create
the network dump files.

The test programm can be started with or without transmitting data to the
server.
If no Field name=value pairs are specified at the command line, no output
is done. If started this way, the program receives a correct HTML page.
Transmitting data, the program receives an error message.
I tested it with a Lotus Domino server 4.6.6.b (international) running on
Windows NT.
This is the only platform where the server doesn't like the data, but
my crawler should be able to talk to every HTTP server.

Looking at the Java.sniff file with a hex viewer, you will notice that the
lines are correctly delimited by 0D0A, except of the line containing the
CGI String. It is delimited by 0A, and also the blank line after it.

Can you take another look at this problem ?

Thank you.
(Review ID: 98133)
======================================================================