StreamTokenizer Does Not Balance Quote Chars

What I’ve always feared was just proved true: Java’s StreamTokenizer does not balance quote characters. If the string is "abc'123"456"789'def", I would want the entire string to be one token, because everything is enclosed in the same pair of " quotes. What StreamTokenizer produces, however, is simply "abc'123". It ends the quoted string at the first matching quote character.

It’s also impossible to define different characters for beginning and ending quotation marks, like { for the beginning and } for the end.

Looks like I have to do quote a bit of work here myself…

Share

About Mathias

Software development engineer. Principal developer of DrJava. Recent Ph.D. graduate from the Department of Computer Science at Rice University.
This entry was posted in DrJava, Ramblings. Bookmark the permalink.

Leave a Reply