You have several choices regarding how you acquire a tokenizer to use for your parsers. For one thing, you can use a tokenizer that comes with the freely available Java software development kit (JDK). However, as of this writing, these tokenizers lack features that the languages in this book require. To overcome these limitations, a new tokenizer was developed for this book, and this chapter explains how to use it.
This discussion opens other approaches to acquiring a tokenizer. You can use the tokenizer that comes with this book as is, without changing its code. The Tokenizer class in sjm.parse.tokens is highly customizable, and you may find that it meets all your needs for tokenization. Another choice is to compare StreamTokenizer in java.io with Tokenizer in sjm.parse.tokens to develop ideas for writing your own tokenizer. You can learn from the strengths and weaknesses of existing tokenizers if you choose to design your own.
