|
1 | 1 | .. _grammar: |
2 | 2 |
|
3 | | -========================== |
4 | | -Changing CPython's grammar |
5 | | -========================== |
6 | | - |
7 | | -Abstract |
8 | | -======== |
9 | | - |
10 | | -There's more to changing Python's grammar than editing |
11 | | -:cpy-file:`Grammar/python.gram`. Here's a checklist. |
12 | | - |
13 | | -.. note:: |
14 | | - These instructions are for Python 3.9 and beyond. Earlier |
15 | | - versions use a different parser technology. You probably shouldn't |
16 | | - try to change the grammar of earlier Python versions, but if you |
17 | | - really want to, use GitHub to track down the earlier version of this |
18 | | - file in the devguide. |
19 | | - |
20 | | -For more information on how to use the new parser, check the |
21 | | -:ref:`section on how to use CPython's parser <parser>`. |
22 | | - |
23 | | -Checklist |
24 | | -========= |
25 | | - |
26 | | -Note: sometimes things mysteriously don't work. Before giving up, try ``make clean``. |
27 | | - |
28 | | -* :cpy-file:`Grammar/python.gram`: The grammar, with actions that build AST nodes. |
29 | | - After changing it, run ``make regen-pegen`` (or ``build.bat --regen`` on Windows), |
30 | | - to regenerate :cpy-file:`Parser/parser.c`. |
31 | | - (This runs Python's parser generator, :cpy-file:`Tools/peg_generator`). |
32 | | - |
33 | | -* :cpy-file:`Grammar/Tokens` is a place for adding new token types. After |
34 | | - changing it, run ``make regen-token`` to regenerate |
35 | | - :cpy-file:`Include/internal/pycore_token.h`, :cpy-file:`Parser/token.c`, |
36 | | - :cpy-file:`Lib/token.py` and :cpy-file:`Doc/library/token-list.inc`. |
37 | | - If you change both ``python.gram`` and ``Tokens``, |
38 | | - run ``make regen-token`` before ``make regen-pegen``. |
39 | | - On Windows, ``build.bat --regen`` will regenerate both at the same time. |
40 | | - |
41 | | -* :cpy-file:`Parser/Python.asdl` may need changes to match the grammar. |
42 | | - Then run ``make regen-ast`` to regenerate |
43 | | - :cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`. |
44 | | - |
45 | | -* :cpy-file:`Parser/lexer/` contains the tokenization code. |
46 | | - This is where you would add a new type of comment or string literal, for example. |
47 | | - |
48 | | -* :cpy-file:`Python/ast.c` will need changes to validate AST objects |
49 | | - involved with the grammar change. |
50 | | - |
51 | | -* :cpy-file:`Python/ast_unparse.c` will need changes to unparse AST |
52 | | - involved with the grammar change ("unparsing" is used to turn annotations |
53 | | - into strings per :pep:`563`). |
54 | | - |
55 | | -* The :ref:`compiler` has its own page. |
56 | | - |
57 | | -* ``_Unparser`` in the :cpy-file:`Lib/ast.py` file may need changes |
58 | | - to accommodate any modifications in the AST nodes. |
59 | | - |
60 | | -* :cpy-file:`Doc/library/ast.rst` may need to be updated to reflect changes |
61 | | - to AST nodes. |
62 | | - |
63 | | -* Add some usage of your new syntax to ``test_grammar.py``. |
64 | | - |
65 | | -* Certain changes may require tweaks to the library module :mod:`pyclbr`. |
66 | | - |
67 | | -* :cpy-file:`Lib/tokenize.py` needs changes to match changes to the tokenizer. |
68 | | - |
69 | | -* Documentation must be written! Specifically, one or more of the pages in |
70 | | - :cpy-file:`Doc/reference/` will need to be updated. |
| 3 | +This document is now part of the |
| 4 | +`CPython Internals Docs <https://github.com/python/cpython/blob/main/InternalDocs/changing_grammar.md>`_. |
0 commit comments