Chapter 9. Advanced Tokenizing

Building Parsers with Java
By Steven  John  Metsker

Table of Contents


Tokenizing a string means dividing the string into logical chunks , or tokens . This lets you define a grammar in terms of token terminals, something that is much simpler than defining a grammar in terms of every character in an input stream. Simpler grammars translate into simpler parsers, so tokenizing reduces the complexity of a parser. This chapter explains how tokenizing works and describes techniques for controlling in detail how sequences of characters form tokens.


Building Parsers with Java
Building Parsers With Javaв„ў
ISBN: 0201719622
EAN: 2147483647
Year: 2000
Pages: 169 © 2008-2017.
If you may any questions please contact us: