How Indentation is Parsed?

• Whitespace is handled by the lexical analyzer before being parsed.
• The lexical analyzer uses a stack to store indentation levels. At the beginning, the stack contains just the value 0,

which is the leftmost position. Whenever a nested block begins, the new indentation level is pushed on the stack,

and an "INDENT" token is inserted into the token stream which is passed to the parser. There can never be more

than one "INDENT" token in a row ( IndentationError ).

• When a line is encountered with a smaller indentation level, values are popped from the stack until a value is on top which is equal to the new indentation level (if none is found, a syntax error occurs). For each value popped, a "DEDENT" token is generated. Obviously, there can be multiple "DEDENT" tokens in a row.
• The lexical analyzer skips empty lines (those containing only whitespace and possibly comments), and will never generate either "INDENT" or "DEDENT" tokens for them.
• At the end of the source code, "DEDENT" tokens are generated for each indentation level left on the stack, until just the 0 is left.
• For example:

if foo:
  if bar:
      x = 42
else:
  print foo

is analyzed as:

<if> <foo> <:> [0]
<INDENT> <if> <bar> <:> [0, 4]
<INDENT> <x> <=> <42> [0, 4, 8]
<DEDENT> <DEDENT> <else> <:> [0]
<INDENT> <print> <foo> [0, 2]
<DEDENT>

The parser than handles the "INDENT" and "DEDENT" tokens as block delimiters.