from fp.bugs:
>From: ###@###.###
 
StreamTokenizer.peekc is initialized to ' ', and is again set to ' '
when an ordinaryChar token is encountered. This has the effect of
prepending a space character to the stream being tokenized, and
inserting a space character after each ordinaryChar token. This is
fine and dandy in the default case, where ' ' is a whitespaceChar and
so gets skipped by the next call. But I need ' ' to be a wordChar,
and now I get a superfluous TT_WORD token at the beginning of my
stream, and an extra space at the beginning of each TT_WORD which
follows an ordinaryChar.
##Note: if you call ordinaryChars(' ', ' '), nextToken returns ' ' forever.
##( fp.bugs 3403 )... sounds like the same problem...

For example, the code below should produce the following output:

% java TokenizerBug
"null" (40)
"here are" (-3)
"some tokens" (-3)
"null" (41)
"another" (-3)

In fact, it produces the following:

% java TokenizerBug
" " (-3)
"null" (40)
" here are" (-3)
"some tokens" (-3)
"null" (41)
" another" (-3)

-------------------- TokenizerBug.java --------------------

import java.io.*;

class TokenizerBug {
    public static void main(String argv[]) {
String theString = "(here are*some tokens)another";
StringBufferInputStream sbis = new StringBufferInputStream(theString);
StreamTokenizer st = new StreamTokenizer(sbis);

st.resetSyntax();
st.wordChars(0, 255); // everything's a word
st.ordinaryChar('('); // parens are their own tokens
st.ordinaryChar(')');
st.whitespaceChars('*', '*'); // * is a separator

try {
while (st.nextToken() != st.TT_EOF) {
System.out.println("\\"" + st.sval + "\\" (" + st.ttype + ")");
}
} catch (Exception e) {
}
    }
}