Changing Python's Grammer

A couple of months back, I was tinkering with the cpython’s source code. While going through the developer documentation I came across Changing CPython’s Grammar. I wanted to try it out.

Using this documentation, I modified the import statement.

Here is how I did it.

It’s a two step process,

• Grammer modification.
• token and parser regenration.

After cloning Cpython’s repo and configuring python , I modified Grammer/python.gram. This file contains the python grammer.

Grammer modification

The modifications in Grammer/python.gram are as follows.

• In simple_stmt grammer,

  The default grammer for simple_stmt looks like

  # NOTE: assignment MUST precede expression, else parsing a simple assignment
  # will throw a SyntaxError.
  simple_stmt[stmt_ty] (memo):
      | assignment
      | e=star_expressions { _PyAST_Expr(e, EXTRA) }
      | &'return' return_stmt
      | &('import' | 'from') import_stmt
      | &'raise' raise_stmt
      | 'pass' { _PyAST_Pass(EXTRA) }
      | &'del' del_stmt
      | &'yield' yield_stmt
      | &'assert' assert_stmt
      | 'break' { _PyAST_Break(EXTRA) }
      | 'continue' { _PyAST_Continue(EXTRA) }
      | &'global' global_stmt
      | &'nonlocal' nonlocal_stmt
  

  I added the bring keyword to it.

  # NOTE: assignment MUST precede expression, else parsing a simple assignment
  # will throw a SyntaxError.
  simple_stmt[stmt_ty] (memo):
      | assignment
      | e=star_expressions { _PyAST_Expr(e, EXTRA) }
      | &'return' return_stmt
      | &('bring'| 'import' | 'from') import_stmt
      | &'raise' raise_stmt
      | 'pass' { _PyAST_Pass(EXTRA) }
      | &'del' del_stmt
      | &'yield' yield_stmt
      | &'assert' assert_stmt
      | 'break' { _PyAST_Break(EXTRA) }
      | 'continue' { _PyAST_Continue(EXTRA) }
      | &'global' global_stmt
      | &'nonlocal' nonlocal_stmt
  

• The import_stmt.

  The default import_stmt looks like

  import_stmt[stmt_ty]: import_name | import_from
  

  Which is modified to

  import_stmt[stmt_ty]: bring_name | import_name | import_from
  

  We have added one more token to the default import_stmt. So we need to define it.

• Under Import statements added one more thing

  bring_name[stmt_ty]: 'bring' a=dotted_as_names {_PyAST_Import(a, EXTRA)}
  

• The import from statements also needs a modification

  So the default import_from statement

  import_from[stmt_ty]:
      | 'from' a=('.' | '...')* b=dotted_name 'import' c=import_from_targets {
          _PyAST_ImportFrom(b->v.Name.id, c, _PyPegen_seq_count_dots(a), EXTRA) }
      | 'from' a=('.' | '...')+ 'import' b=import_from_targets {
          _PyAST_ImportFrom(NULL, b, _PyPegen_seq_count_dots(a), EXTRA) }
  

  is modified to

  import_from[stmt_ty]:
      | 'from' a=('.' | '...')* b=dotted_name 'import' c=import_from_targets {
          _PyAST_ImportFrom(b->v.Name.id, c, _PyPegen_seq_count_dots(a), EXTRA) }
      | 'from' a=('.' | '...')+ 'import' b=import_from_targets {
          _PyAST_ImportFrom(NULL, b, _PyPegen_seq_count_dots(a), EXTRA) }
      | 'from' a=('.' | '...')* b=dotted_name 'bring' c=import_from_targets {
          _PyAST_ImportFrom(b->v.Name.id, c, _PyPegen_seq_count_dots(a), EXTRA) }
      | 'from' a=('.' | '...')+ 'bring' b=import_from_targets {
          _PyAST_ImportFrom(NULL, b, _PyPegen_seq_count_dots(a), EXTRA) }
  

Token regeneration

The tokens are re-generated by running,

make regen-token

Parser regeneration

Once the tokens are generated, the parser needs to be regenerated. To do that, run

make regen-pegen

This regenerates the parser Parser/parser.c from the modified grammer.

After all of this is done, You will have regenerated Include/token.h, Parser/token.c, Lib/token.py and Doc/library/token-list.inc.

Building python

Now we are all set to build python with the modified grammer.

make -j

You will have python binary.

With this binary, the bring keyword can be used instead of import.

Which looks something like:

[girish@fedora cpython]$ ./python 
Python 3.12.0a0 (heads/main-dirty:330f1d5828, Aug 21 2022, 17:52:16) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> bring this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>> from json bring loads