3.8 Tokenizing a Character Stream

Example 3-6 was a Reader implementation wrapped around another Reader. ReaderTokenizer (Example 3-7) is a Tokenizer implementation wrapped around a Reader. The Tokenizer interface was shown in Example 2-8, and the ReaderTokenizer class shown here is a subclass of the AbstractTokenizer class of Example 2-9.

As its name implies, ReaderTokenizer tokenizes the text it reads from a Reader stream. The class implements the abstract createBuffer( ) and fillBuffer( ) methods of its superclass, and you may want to reread Example 2-9 to refresh your memory about the interactions between ReaderTokenizer and AbstractTokenizer.

Example 3-7 includes an inner class named Test that reads and tokenizes characters from a FileReader, listing the tokens read on the standard output. It also writes the text of each token to a FileWriter, producing a copy of the input file and demonstrating that the tokenizer accounts for every character of the input file (as long as it is not configured to discard spaces, that is). Like its superclass, ReaderTokenizer uses the assert keyword, and must be compiled with the -source 1.4 option to javac.

Example 3-7. ReaderTokenizer.java

package je3.io; import je3.classes.Tokenizer; import je3.classes.AbstractTokenizer; import java.io.*; /**  * This Tokenizer implementation extends AbstractTokenizer to tokenize a stream  * of text read from a java.io.Reader.  It implements the createBuffer( ) and  * fillBuffer( ) methods required by AbstractTokenizer.  See that class for  * details on how these methods must behave.  Note that a buffer size may  * be selected, and that this buffer size also determines the maximum token  * length.  The Test class is a simple test that tokenizes a file and uses  * the tokens to produce a copy of the file  **/ public class ReaderTokenizer extends AbstractTokenizer {     Reader in;     // Create a ReaderTokenizer with a default buffer size of 16K characters     public ReaderTokenizer(Reader in) { this(in, 16*1024); }     public ReaderTokenizer(Reader in, int bufferSize) {         this.in = in;  // Remember the reader to read input from         // Tell our superclass about the selected buffer size.         // The superclass will pass this number to createBuffer( )         maximumTokenLength(bufferSize);     }     // Create a buffer to tokenize.     protected void createBuffer(int bufferSize) {         // Make sure AbstractTokenizer only calls this method once         assert text == null;         this.text = new char[bufferSize];  // the new buffer         this.numChars = 0;                 // how much text it contains     }     // Fill or refill the buffer.     // See AbstractTokenizer.fillBuffer( ) for what this method must do.     protected boolean fillBuffer( ) throws IOException {         // Make sure AbstractTokenizer is upholding its end of the bargain         assert text!=null && 0 <= tokenStart && tokenStart <= tokenEnd &&             tokenEnd <= p && p <= numChars && numChars <= text.length;         // First, shift already tokenized characters out of the buffer         if (tokenStart > 0) {             // Shift array contents             System.arraycopy(text, tokenStart, text, 0, numChars-tokenStart);             // And update buffer indexes             tokenEnd -= tokenStart;              p -= tokenStart;             numChars -= tokenStart;             tokenStart = 0;          }         // Now try to read more characters into the buffer         int numread = in.read(text, numChars, text.length-numChars);         // If there are no more characters, return false         if (numread == -1) return false;         // Otherwise, adjust the number of valid characters in the buffer         numChars += numread;         return true;       }     // This test class tokenizes a file, reporting the tokens to standard out     // and creating a copy of the file to demonstrate that every input     // character is accounted for (since spaces are not skipped).     public static class Test {         public static void main(String[  ] args) throws java.io.IOException {             Reader in = new FileReader(args[0]);             PrintWriter out = new PrintWriter(new FileWriter(args[0]+".copy"));             ReaderTokenizer t = new ReaderTokenizer(in);             t.tokenizeWords(true).tokenizeNumbers(true).tokenizeSpaces(true);             while(t.next( ) != Tokenizer.EOF) {                 switch(t.tokenType( )) {                 case Tokenizer.EOF:                     System.out.println("EOF"); break;                 case Tokenizer.WORD:                     System.out.println("WORD: " + t.tokenText( )); break;                 case Tokenizer.NUMBER:                     System.out.println("NUMBER: " + t.tokenText( )); break;                 case Tokenizer.SPACE:                     System.out.println("SPACE"); break;                 default:                     System.out.println((char)t.tokenType( ));                 }                 out.print(t.tokenText( ));  // Copy token to the file             }             out.close( );         }     } }