9.10 The TokenStringSource Utility

Building Parsers with Java
By Steven  John  Metsker

Table of Contents
Chapter  9.   Advanced Tokenizing


The TokenStringSource class in sjm.parse.tokens is a utility that helps divide an input stream into lines that a parser can parse one at a time. Figure 9.16 shows this class.

Figure 9.16. The TokenStringSource class. The TokenStringSource nextTokenString() method returns all the tokens up to the specified delimiter as a TokenString .


This class provides an enumeration over the tokens of a tokenizer. Each call to nextTokenString() results in a TokenString that contains all the tokens up to a specified delimiter. This is useful for languages that consist of statements that are separated by a delimiter, such as " ; ". By using a TokenStringSource , the author of a parser need only parse individual statements of the language. The main() method of TokenStringSource gives the following example:

 public static void main(String args[]) {      String s = "I came; I saw; I left in peace;";     TokenStringSource tss =         new TokenStringSource(new Tokenizer(s), ";");     while (tss.hasMoreTokenStrings()) {         System.out.println(tss.nextTokenString());     } } 

This prints the following:

 I came  I saw I left in peace 

The Logikus parser in Chapter 14, "Parsing a Logic Language," uses TokenStringSource to break up lines of input. Statements in a Logikus program are rules and facts, separated by semicolons. The Logikus parser does not parse an entire program. Instead, it parses a single statement and relies on a TokenStringSource object to divide the input into statements. Using TokenStringSource can make your parser easier to write and faster, too.


Building Parsers with Java
Building Parsers With Javaв„ў
ISBN: 0201719622
EAN: 2147483647
Year: 2000
Pages: 169

Similar book on Amazon

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net