9.2 Acquiring a Tokenizer

Building Parsers with Java
By Steven  John  Metsker

Table of Contents
Chapter  9.   Advanced Tokenizing


You have several choices regarding how you acquire a tokenizer to use for your parsers. For one thing, you can use a tokenizer that comes with the freely available Java software development kit (JDK). However, as of this writing, these tokenizers lack features that the languages in this book require. To overcome these limitations, a new tokenizer was developed for this book, and this chapter explains how to use it.

This discussion opens other approaches to acquiring a tokenizer. You can use the tokenizer that comes with this book as is, without changing its code. The Tokenizer class in sjm.parse.tokens is highly customizable, and you may find that it meets all your needs for tokenization. Another choice is to compare StreamTokenizer in java.io with Tokenizer in sjm.parse.tokens to develop ideas for writing your own tokenizer. You can learn from the strengths and weaknesses of existing tokenizers if you choose to design your own.


Building Parsers with Java
Building Parsers With Javaв„ў
ISBN: 0201719622
EAN: 2147483647
Year: 2000
Pages: 169

Similar book on Amazon

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net