Python tokenize Module
Example
Tokenize Python code:
import tokenize
import io
code = 'x = 42'
tokens = list(tokenize.generate_tokens(io.StringIO(code).readline))
print(f'Found {len(tokens)} tokens')
Try it Yourself »
Definition and Usage
The tokenize module provides a lexical scanner for Python source code.
Use it to convert Python code into tokens, analyze source code structure, or build code analysis tools.
Members
| Member | Description |
|---|---|
| TokenInfo | Named tuple representing a token. |
| generate_tokens() | Tokenize a Python source code string. |
| tokenize() | Tokenize a Python source from bytes. |
| untokenize() | Convert tokens back to Python source code. |
