Does CPython include the mentioned optimizations from PEP 380?
PEP 380 mentions that the syntax yield from expr
can be optimized in Python.
PEP 380 - Optimization
The use of specialized syntax opens up opportunities for optimization when there is a long chain of generators. Such chains can arise, for example, during the recursive movement of a tree structure. The overhead of passing calls
__next__()
and values given down and up the chain can cause what should be an O (n) operation to become, in the worst case, O (n ** 2).A possible strategy is to add a slot to generator objects to allow for generator delegation. When calls to
__next__()
or are made in the generatorsend()
, this slot is first checked, and if it is not empty, the generator it referenced is resumed instead. If it risesStopIteration
, the slot is cleared and the main generator resumes.This will reduce the redundancy of delegation to chaining C function calls that do not involve executing Python code. A possible reinforcement would be to loop through the entire generator chain and directly resume at the end, although the processing is
StopIteration
more complicated.
Does CPython use this optimization?
Does not look like. As of Python 3.6, the structure of a generator structure does not have a suggested field, and the code path for yield from
through the chain of generators always goes through the process of resuming their Python stack frames individually. (This code path goes from the opcode through / to , and from there to and on to the next generator in the chain.) YIELD_FROM
_PyGen_Send
gen_iternext
gen_send_ex
PyEval_EvalFrameEx
YIELD_FROM