Why doesn't the Python grammar spec include docstrings and comments?

Section 1

What happens to the comments?

Comments (anything preceding #

) are ignored during tokenization / lexical analysis, so there is no need to write rules to parse them. They do not provide semantic information to the interpreter / compiler, as they only serve to improve the verbosity of your program for the reader and are therefore ignored.

Here's the lex specification for the ANSI C programming language: http://www.quut.com/c/ANSI-C-grammar-l-1998.html . I would like to draw your attention to the way comments are handled here:

"/*"            { comment(); }
"//"[^\n]*      { /* consume //-comment */ }

      

        
        
        
      

    

Now let's take a look at the rule for int

.

"int"           { count(); return(INT); }

      

        
        
        
      

    

Here's a lex function to handle int

other tokens as well:

void count(void)
{
    int i;

    for (i = 0; yytext[i] != '\0'; i++)
        if (yytext[i] == '\n')
            column = 0;
        else if (yytext[i] == '\t')
            column += 8 - (column % 8);
        else
            column++;

    ECHO;
}

      

        
        
        
      

    

You see here that it ends with a statement ECHO

, which means that it is a valid token and needs to be parsed.

Now, here's the lex function handling comments:

void comment(void)
{
    char c, prev = 0;

    while ((c = input()) != 0)      /* (EOF maps to 0) */
    {
        if (c == '/' && prev == '*')
            return;
        prev = c;
    }
    error("unterminated comment");
}

      

        
        
        
      

    

Not here ECHO

. So nothing is returned.

This is a typical example, but python does the same.

Section 2

What's going on with docstrings?

Note. This section of my answer is intended to complement @MartijnPieters' answer. It is not intended to replicate any of the information he provided in his post. Now, having said this, ...

I originally assumed that comments and docstrings are removed in the first pass to the CPython compiler [...]

Docstrings (string literals that are not assigned to any variable name, nothing in the '...'

, "..."

, '''...'''

and """..."""

) actually processed. They are parsed as simple string literals (token STRING+

), as Martijn Pieters mentions in his answer . As far as the current docs are concerned, it is only mentioned in passing that docstrings are assigned to the function / class / module attribute __doc__

. How this is done is not actually mentioned anywhere.

What actually happens is that they are marked and parsed as string literals and the resulting parse tree will contain them. A bytecode is generated from the parse tree, with the docstrings in their rightful place in the attribute __doc__

(they are clearly not part of the bytecode as shown below). I won't go into details, as the answer I linked above describes the same thing in very nice detail.

Of course, they can be completely ignored. If you use python -OO

(the flag -OO

means "optimize intensively" as opposed to -O

which means "optimize"), with the resulting bytecode stored in files .pyo

, which excludes dock sequences,

An illustration can be seen below:

Create a file test.py

with the following code:

def foo():
    """ docstring """
    pass

      

        
        
        
      

    

We will now compile this code with the normal flags set.

>>> code = compile(open('test.py').read(), '', 'single')
>>> import dis
>>> dis.dis(code)
  1           0 LOAD_CONST               0 (<code object foo at 0x102b20ed0, file "", line 1>)
              2 LOAD_CONST               1 ('foo')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (foo)
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE

      

        
        
        
      

    

As you can see, there is no mention of our docstring in the bytecode. However, they are. To get the docstring you can do ...

>>> code.co_consts[0].co_consts
(' docstring ', None)

      

        
        
        
      

    

So, as you can see, the docstring remains, just not as part of the main bytecode. Now let's recompile this code, but with an optimization level of 2 (switch equivalent -OO

):

>>> code = compile(open('test.py').read(), '', 'single', optimize=2)
>>> dis.dis(code)
  1           0 LOAD_CONST               0 (<code object foo at 0x102a95810, file "", line 1>)
              2 LOAD_CONST               1 ('foo')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (foo)
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE

      

        
        
        
      

    

No, it makes a difference, but ...

>>> code.co_consts[0].co_consts
(None,)

      

        
        
        
      

    

The docks are now gone.

Flags -O

and -OO

only remove things ( bytecode optimization is done by default ... -O

removes asserts and if __debug__:

bundles from the generated bytecode, while -OO

ignoring docstrings in appendix). The resulting compilation time will decrease slightly. In addition, the execution speed remains the same unless you have a large number of statements assert

and if __debug__:

, otherwise, it does not affect performance.

Also remember that docstrings are only saved if they are the first in the function / class / module definition. All extra lines are simply removed at compile time. If you change test.py

to the following:

def foo():
    """ docstring """

    """test"""
    pass

      

        
        
        
      

    

And then repeat the same process with optimization=0

, it gets stored in a variable co_consts

when compiled:

>>> code.co_consts[0].co_consts
(' docstring ', None)

      

        
        
        
      

    

The meaning """ test """

was ignored. You will be interested to know that this removal is done as part of the basic optimization for bytecode.

Section 3

Additional reading

(You may find these links as interesting as I am.)

• What does Python optimization (-O or PYTHONOPTIMIZE) do?

• What do the python file extensions, .pyc.pyd.pyo mean?

• Are the Docstrings and Python comments kept in memory when the module is loaded?

• Working with compilation ()

• dis
  
  module

• peephole.c
  
  (courtesy Martijn) - Source code for all compiler optimizations. It's especially fun if you can figure it out!