A Tour of the StringTokenizer Class

The StringTokenizer class is used to store and handle groups of strings, known as tokens, that are combined into one long string. An example of its advantage is when reading in data from a file. Let's say you store a high-score list in a file for a game that you have made; each line in the file consists of three data strings used to store information about the username, e-mail address, and high score, respectively, with any two of these strings being separated by a comma character. For example:

String line1 =     new String("Glenn,[e-mail],12000"); String line2 =     new String("Andrew,[e-mail],9000");

The individual data strings, such as the username, can then be treated as tokens of the full string, and the comma can be treated as the delimiter of the tokens. The delimiter is simply a character used to separate tokens. The StringTokenizer class belongs to the package java.util (which we will look at in detail in Chapter 5). For the time being, we will use the class by specifying its complete path name, including the package to which it belongs. (Again, we will learn about packages in Chapter 5, so do not worry about them for now.)

In the following example, SimpleTokens.java, we cycle through an array of strings. Each string is imagined to be a line of text read in from a file containing a username, e-mail address, and high score gained, with each value separated by a comma character. (We are not actually reading in from a file in this example but just creating an array of strings that may have been data read in from a file. We will look at reading in files for real in Chapter 6.) For each line of text, a new StringTokenizer object is created to handle extracting the individual string tokens and printing their values to the console screen. Here is the code:

public class SimpleTokens {     public static void main(String args[])     {           String[] data =             {                 "Glenn,[e-mail],12000",                 "Andrew,[e-mail],9000"             };                String[] tokenType = {"username", "email", "high score"};                         for(int i=0; i<data.length; i++)         {             int tokenIndex = 0;             System.out.println("New line of data...");             java.util.StringTokenizer tokenizer = new                 java.util.StringTokenizer(data[i],",");             while(tokenizer.hasMoreTokens())             {                 String token = tokenizer.nextToken();                 System.out.println("\t"+tokenType[tokenIndex]+"                     "+token);                 tokenIndex++;             }         }     } }

When you compile and run this example code, you should get output similar to the following screen shot.

Figure 3-11:

In this example code, we use the most common constructor of the StringTokenizer class that takes two arguments.

java.util.StringTokenizer tokenizer =     new java.util.StringTokenizer(data[i],",");

The first argument is the String object containing the data that we wish to "tokenize." The second is a string representing the delimiter of the tokens. The delimiter does not need to be one character but may be a number of characters in the string where each character acts as a delimiter. You should note if the delimiter consists of multiple characters; then the string value itself is not a delimiter, but its characters are all individual delimiters. Once the StringTokenizer object is created, we then extract the tokens using the methods hasMoreTokens and nextToken. StringTokenizer contains an index position, which moves along the string each time a token is found. The method hasMoreTokens returns true if there is at least one more token available from the index position onward. From here, we use the method nextToken to get this next available token from the index position, which nextToken returns as a String object, moving the index position forward in the process.

To find out how many more tokens are left to get in the StringTokenizer as a numeric value, you can use the method countTokens if necessary, which returns the number of tokens left from the index position to the end.