Python token Module
Example
Display token types used by Python tokenizer:
import token
print(f'NAME token: {token.NAME}')
print(f'NUMBER token: {token.NUMBER}')
print(f'STRING token: {token.STRING}')
Try it Yourself »
Definition and Usage
The token module defines constants representing the numeric values of Python's internal token types.
Use it when working with the tokenize module to parse Python source code into tokens.
Members
| Member | Description |
|---|---|
| AMPER | Token for '&' operator. |
| ASYNC | Token for 'async' keyword. |
| AWAIT | Token for 'await' keyword. |
| DEDENT | Token for dedentation. |
| ENDMARKER | Token marking end of input. |
| INDENT | Token for indentation. |
| NAME | Token for identifiers. |
| NEWLINE | Token for newline character. |
| NUMBER | Token for numeric literals. |
| OP | Token for operators. |
| STRING | Token for string literals. |
