Name: paC48320 Date: 11/10/97


If the tokenizer is set to parseNumbers() and it encounters a token such as
-word, it returns the hyphen '-' as an ordinary character, regardless of whether
the '-' has been set as a wordChar.

Here's the relevant code from StreamTokenizer.java
if ((ctype & CT_DIGIT) != 0) {
boolean neg = false;
if (c == '-') {
c = read();
if (c != '.' && (c < '0' || c > '9')) {
peekc = c;
return ttype = '-';
}
neg = true;
}

It should check if '-' is a wordChar() and if so it should continue looking for wordChars.

Also there is no easy way to stop the tokenizer from parsing numbers; parseNumbers() has no inverse !!

You have to do the following:
// We need to reset the Tokenizer since it parses number by default and there's no way to turn it off
in.resetSyntax();
in.wordChars('0', '9');
in.wordChars('-', '.');
in.wordChars('a', 'z');
in.wordChars('A', 'Z');
in.wordChars(128 + 32, 255);
in.whitespaceChars(0, ' ');
in.commentChar('/');
in.quoteChar('"');
in.quoteChar('\'');

Which kind of sucks !!

I hope that is clear enough
(Review ID: 19161)
======================================================================