From: Jarl Fransson <###@###.###>
This does not look like form output to me.


There seems to be a bug with the line counter in StreamTokenizer
class JDK 1.0. ver 1.10.

Diagnostics: For every character ' (ascii 39) in the
input the rest of the line is skipped and lineno is not
updated.

Work-around: define ' as an ordinary character.





The documentation of pushBack seems to be unclear: it is
not appearant that pushBack should be called before you
can do nextToken 2 times and get the same token, as
is my experience, right?

Regards,

Jarl Fransson


Name: saC57035 Date: 12/15/96


Here is more precise description of this bug:

If EOL is used as string terminator, the line counter is not incremented.

In fact, JLS allows EOL being string terminator (see it.22.14.21):
  "If a string quote character is encountered, then a string is recognized,
   consisting of all characters after (but not including) the string quote
   character, up to (but not including) the next occurrence of that same
   string quote character, or a line terminator, or end of file."

Here is the example demonstrating the bug:
-----Test.java---------------------------
import java.io.*;

public class Test {

public static void main( String argv[] ) {
 StreamTokenizer st = new StreamTokenizer
               (new StringBufferInputStream("line1 \' \n line2 \n line3"));
 try {
   while(st.nextToken() != st.TT_EOF) System.out.println(st);
 } catch (Throwable e) {
   System.out.println("Test failed: "+e+" is thrown");
 }
}

}
-----Output-------------------------------
Token[line1], line 1
Token['''], line 1
Token[line2], line 1
Token[line3], line 2
------------------------------------------
======================================================================