Python AST Module

Python AST Module

AST (Abstract Syntax Tree) is a module present in the python standard library. Before transforming python code to “byte code”(.pyc files), it is converted to an AST. The most important function of the AST module is to generate this AST.

The interpreter follows pre-written instructions that translate the Python code into instructions that a machine can run. AST follows the following steps to generate machine code from Python code.

• At the initial step, code is parsed into smaller chunks called tokens. These tokens are based on a set of pre-defined rules for things that should be treated differently. For instance, the token for the keyword if is different from the token of a numeric value like 42.
• An Abstract Syntax tree (AST) is generated from these raw lists of tokens.
• From the AST, the Python online compiler produces lower-level instruction known as bytecode, which is very generic and can be run by the computer efficiently.
• The interpreter can finally run the code using the bytecode instructions.

Compilation Modes

The three modes available to compile the code are:

• exec mode to execute the normal Python code.
• eval mode to return a result after evaluating Python's expression.
• single mode to execute one statement at a time, like the Python shell.

Code Execution and Expression Evaluation

Python Code Compilation in single mode

In the following example, the Python code is compiled in a single mode.

Code

Output

Python Code Execution

In the following example, the Python code is run, using the AST module. The compile() and exec() functions execute the AST.

Code

Output

Expression Evaluation

In the following example, the AST module evaluates the Python expression exp and returns the result after evaluation using the eval() function.

Code

Output

Multiline ASTs Creation

The following example shows how the AST module is used to create multi-line ASTs and how it handles and distributes more than one operation. The output of the dump() function reflects how the code is distributed into different components.

Code

Output

NodeTransformer and NodeVisitor Classes

In the initial step, the code is parsed. The ast module has the NodeTransformer and NodeVisitor. The former is used to take different types and modify them as per requirement. The NodeVisitor class calls the visit function every time the tree is traversed to get more control over the nodes. The DemoVisitor class and the DemoTransformer class inherit the NodeVisitor and NodeTransformer classes, respectively.

Code

Output

Analyzing AST

After getting the tree, the Analyzer follows the visitor pattern. The NodeVisitor class can track any node in Python. However, to visit a particular type of node a method similar to visit_<node type> is implemented. The visit() method works just like the visit_<node type> method whenever that type of node is encountered while traversing through the tree structure.

AST as an Analysis Tool

The byte code generated from the Python code is primitive and designed for machines as it makes the interpreter fast. Humans can not read this. Though ASTs are not very human-friendly, they are more legible than the byte code representation. ASTs contain structured information that makes drawing an idea about the original Python code easier.

When to Use Python AST Module?

• Abstract Syntax Tree parses the source code and finds the errors in the code.
• It is used as a custom Python interpreter.
• Python AST module can analyze the static code.

Examples of Python AST Module

The following examples illustrate how ASTs are created and executed. There are two fundamental functions:

• parse - interpret and create an AST from a string containing Python code
• dump - prints the abstract syntax tree as a string The formatted string contains information about the fields and values of the tree. The compile() and exec() functions execute the AST.

Handling Multiple Operations with Python AST

The following example shows how the AST module is used to create multi-line ASTs and how it handles and distributes more than one operation. works on two numbers and then shows the output.

Code

Output

Arithmetic Operation Visualization by Python AST Module

In this example, the tree shows how the Mult() operation works on two numbers and then shows the output.

Code

Output

AST Visualization for a List

The following example illustrates an AST for printing a list.

Code

Output

AST Visualization for a String

The following example illustrates an AST for printing a string.

Code

Output