The tokenize Module

The tokenize module splits a Python source file into individual tokens. It can be used for syntax highlighting or for various kinds of code-analysis tools.

In Example 13-17, we simply print the tokens.

Example 13-17. Using the tokenize Module

File: tokenize-example-1.py

import tokenize

file = open("tokenize-example-1.py")

def handle_token(type, token, (srow, scol), (erow, ecol), line):
 print "%d,%d-%d,%d:	%s	%s" % 
 (srow, scol, erow, ecol, tokenize.tok_name[type], repr(token))

tokenize.tokenize(
 file.readline,
 handle_token
 )

1,0-1,6: NAME import
1,7-1,15: NAME 	okenize
1,15-1,16: NEWLINE \012
2,0-2,1: NL \012
3,0-3,4: NAME file
3,5-3,6: OP =
3,7-3,11: NAME open
3,11-3,12: OP (
3,12-3,35: STRING "tokenize-example-1.py"
3,35-3,36: OP )
3,36-3,37: NEWLINE \012
...

Note that the tokenize function takes two callable objects: the first argument is called repeatedly to fetch new code lines, and the second argument is called for each token.