Skip to content

gh-141984: Reword the "yield expressions"` section #159421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
encukou wants to merge 16 commits into
base: main
Choose a base branch
from
+511 −181
Open

gh-141984: Reword the "yield expressions"` section #159421

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
4b67787
Work on yield expression
encukou Apr 22, 2026
590b50f
Merge "yield statements" into "expression statements"
encukou Sep 16, 2026
7b96ea2
Reword the intro a bit
encukou Sep 23, 2026
c74219d
Work more on the intro
encukou Sep 23, 2026
cb4bfc9
WIP
encukou Sep 23, 2026
971a2bc
"Using yield as an expression"
encukou Sep 23, 2026
0f6ae31
Move "yield expressions in try" to stdtypes
encukou Sep 23, 2026
68fa794
Go over "yield from expressions"
encukou Oct 7, 2026
827852d
The last two sections before acyncgen
encukou Oct 7, 2026
e79cd80
Remove bad edit
encukou Oct 14, 2026
e69cea3
Cleanups
encukou Oct 14, 2026
8ae0306
Fixups
encukou Oct 14, 2026
3ab5380
Merge in the main branch
encukou Oct 28, 2026
f2e85c4
Fix lint issues
encukou Oct 28, 2026
0d7dd2a
Merge in the main branch
encukou Nov 4, 2026
3865475
Bring back or record old HTML IDs
encukou Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 158 additions & 37 deletions Doc/builtins/stdtypes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -962,6 +962,12 @@ Once an iterator's :meth:`~iterator.__next__` method raises
Implementations that do not obey this property are deemed broken.


.. index::
single: generator
single: generator iterator
single: generator function

.. _generator-iterators:
.. _generator-types:

Generator types
Expand All @@ -976,32 +982,84 @@ A function that contains one or more :ref:`yield expressions <yieldexpr>`
is a :term:`generator function`.
For example::

>>> def count_to_three():
... yield 0
... yield 1
... yield 2
... yield 3
>>> def generator_function():
... print("generator starting")
... yield 123
... print("generator at line 4")
... yield 456
... print("generator ending")

Generator functions behave as regular
:ref:`user-defined functions <user-defined-funcs>`
(for example, they have the same attributes), except that calling a generator
function returns a :ref:`generator iterator <generator-methods>`::
function returns a :term:`generator iterator`::

>>> generator_iterator = generator_function()
>>> generator_iterator
<generator object generator_function at ...>

The iterator then allows control over the execution of the underlying function.
Initially, the function is *suspended*.
Execution starts when a :dfn:`consumer` asks the iterator to produce the next
value, for example using the :func:`next` function (which calls
the generator's :meth:`~generator.__next__` method).
The execution starts at the beginning of the function body
and proceeds to the first :keyword:`!yield` expression.
The expression after the :keyword:`!yield` keyword is evaluated and returned
to the consumer (here, the caller of ``next``)::

>>> first_value = next(generator_iterator)
generator starting
>>> print(first_value)
123

At this point, execution of the generator function is *suspended* again.
By *suspended*, we mean that all local execution state is retained, including the
current bindings of local variables, the "instruction pointer" (index of
the :keyword:`!yield` expression at which execution is suspended),
the internal evaluation stack, and the state of any exception handling.

When the iterator is asked to produce another value, execution is *resumed*:
the function proceeds from the :keyword:`!yield` expression where it was
suspended and continues until the next :keyword:`!yield` expression
is encountered::

>>> second_value = next(generator_iterator)
generator at line 4
>>> print(second_value)
456

If the generator function returns before a :keyword:`!yield` expression is
encountered (either with a :keyword:`return` statement, or implicitly at the
end of the function), the generator iterator becomes :term:`exhausted`.
All further attempts to get the next value from it will raise
a :exc:`StopIteration` exception::

>>> third_value = next(generator_iterator)
Traceback (most recent call last):
...
StopIteration
>>> third_value = next(generator_iterator)
Traceback (most recent call last):
...
StopIteration

>>> count_to_three()
<generator object count_to_three at 0x7f33a2305000>
Commonly, an iterator is consumed via a :keyword:`for` loop, which implicitly
calls :func:`next` repeatedly, and handles :exc:`StopIteration`::

Iterating a generator iterator executes code of the underlying
generator function, producing each :keyword:`yield`\ed value in turn::
>>> for value in generator_function():
... print("value from generator:", value)
generator starting
value from generator: 123
generator at line 4
value from generator: 456
generator ending

>>> for number in count_to_three():
... print(number)
0
1
2
3
.. note::

>>> list(count_to_three())
[0, 1, 2, 3]
In this example, the ``generator_function()`` call creates a new iterator
with its own local execution state.
The exhausted ``generator_iterator`` from earlier examples is not reused.

One common use for generator functions is implementing the
:meth:`~object.__iter__` method of custom iterable objects.
Expand All @@ -1015,12 +1073,71 @@ For example::
>>> list(CardDeck())
['three of clubs', 'ace of hearts']

.. seealso::

:pep:`255` - Simple Generators
The proposal for adding generators and the :keyword:`yield` statement to Python.

Generator finalization
^^^^^^^^^^^^^^^^^^^^^^

If a generator iterator is not resumed before it is destroyed
(:term:`garbage collected <garbage collection>`),
its :meth:`~generator.close` method will be called,
raising a :exc:`GeneratorExit` at the :keyword:`!yield` expression where the
function is suspended.
If that :keyword:`!yield` expression is in a :keyword:`try` construct,
pending :keyword:`finally` clauses execute.

.. admonition:: CPython implementation detail

In the following example, ``iterator`` is garbage collected immediately
after the ``del iterator`` statement, triggering the ``finally`` block::

>>> def gen():
... try:
... yield 123
... finally:
... print('shutting down')
...
>>> iterator = gen()
>>> next(iterator)
123
>>> del iterator
shutting down

On non-CPython implementations, or future versions of CPython,
garbage collection (and thus printing the message) may happen at any
later point, or even not at all.

After :meth:`~generator.close` raises :exc:`!GeneratorExit`,
the underlying function must not yield.
If it does, :exc:`RuntimeError` is raised instead of suspending the function::

>>> def gen():
... try:
... yield 123
... except GeneratorExit:
... pass # ignore the exception
... yield 456
...
>>> iterator = gen()
>>> next(iterator)
123
>>> iterator.close()
Traceback (most recent call last):
File "<python-input-4>", line 1, in <module>
iterator.close()
~~~~~~~~^^
RuntimeError: generator ignored GeneratorExit



.. index:: pair: object; generator
.. _generator-methods:

Generator iterators
^^^^^^^^^^^^^^^^^^^
Generator iterator methods
^^^^^^^^^^^^^^^^^^^^^^^^^^

Generator iterators implement the
:ref:`iterator protocol <stdtypes-iterators>`.
Expand All @@ -1032,20 +1149,19 @@ Iterating them drives execution of the underlying generator function.

Starts the execution of a generator function or resumes it at the
:ref:`yield expression <yieldexpr>` where the function is currently suspended.
When a generator function is resumed with a :meth:`~generator.__next__`
method, the current yield expression always evaluates to :const:`None`.
The execution then continues to the next yield expression, where the
generator is suspended again, and the value of the expression after the
:keyword:`yield` keyword is returned to :meth:`~generator.__next__`'s
caller.
The :keyword:`!yield` expression (if any) evaluates to :const:`!None`.
The execution then continues to the next :keyword:`!yield` expression,
where the generator is suspended again, and the value of the expression
after the :keyword:`!yield` keyword is returned to
:meth:`!__next__`'s caller.
If the generator exits without yielding another value,
:meth:`~generator.__next__` raises a :exc:`StopIteration` exception,
:meth:`!__next__` raises a :exc:`StopIteration` exception,
signalling that iteration has completed.

This method is normally called implicitly, for example by a :keyword:`for`
loop, or by the built-in :func:`next` function.

Generator iterators have a few more methods than generic iterators, which
Generator iterators have a few more methods than generic iterators. These
can be used to control the execution of the underlying generator function:

.. method:: generator.send(value)
Expand All @@ -1057,8 +1173,8 @@ can be used to control the execution of the underlying generator function:
the underlying function and either returns the next yielded value or raises
:exc:`StopIteration`.

When :meth:`send` is called to start the generator, it must be called
with :const:`None` as the argument, because there is no current yield
When :meth:`!send` is called to start the generator, it must be called
with :const:`!None` as the argument, because there is no current yield
expression that could receive the value.


Expand All @@ -1073,10 +1189,9 @@ can be used to control the execution of the underlying generator function:
If the generator function does not catch the passed-in exception, or
raises a different exception, then that exception propagates to the caller.

When :meth:`throw` is called to start the generator, the generator
immediately exits (that is, subsequent calls to :meth:`~generator.__next__`
will raise :exc:`StopIteration`) and the thrown exception is propagated to
:meth:`throw`'s caller.
When :meth:`throw` is called to start the generator, the generator is
immediately :term:`exhausted`, the underlying function is not executed,
and the thrown exception is propagated to :meth:`throw`'s caller.

In typical use, this is called with a single argument, an exception instance,
similar to the way the :keyword:`raise` keyword is used.
Expand All @@ -1102,9 +1217,9 @@ can be used to control the execution of the underlying generator function:
Raises a :exc:`GeneratorExit` exception at the point where the generator
function is currently suspended (equivalent to calling ``throw(GeneratorExit)``).

If the generator function has already exited (due to an exception or
normal return), or raises :exc:`GeneratorExit` (by not catching the
exception), :meth:`close` returns :const:`None`.
Then, if the generator function has already exited (due to an exception or
normal return), or raises :exc:`GeneratorExit` (including by not catching
the exception), :meth:`close` returns :const:`None`.
If the generator yields a value, a :exc:`RuntimeError` is raised.
If the generator raises any other exception, it is propagated to the caller.
If a generator returns a value upon being closed, that value is returned
Expand All @@ -1125,6 +1240,12 @@ Calling any of the generator methods (:meth:`~generator.__next__`,
while one of these methods is already executing
raises a :exc:`ValueError` exception.

.. seealso::

:pep:`342` - Coroutines via Enhanced Generators
The proposal to add :meth:`!send`, :meth:`!throw` and :meth:`!close`,
making generators usable as simple coroutines.


.. index:: pair: object; asynchronous-generator
.. _asynchronous-generator-methods:
Expand Down
10 changes: 5 additions & 5 deletions Doc/glossary.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -649,25 +649,25 @@ Glossary
collector that is able to detect and break reference cycles. The
garbage collector can be controlled using the :mod:`gc` module.

.. index:: single: generator
.. index:: single: generator

generator
Informally used to mean either a :term:`generator function` or a
:term:`generator iterator`, depending on context. The formal terms
:term:`generator function` and :term:`generator iterator` are uncommon
in practice; "generator" alone is almost always sufficient.

.. index:: single: generator function
.. index:: single: generator function

generator function
A function which returns a :term:`generator` object. It looks like a
A function which returns a :term:`generator iterator`. It looks like a
normal function except that it contains :keyword:`yield` expressions
for producing a series of values usable in a :keyword:`for`\-loop or
that can be retrieved one at a time with the :func:`next` function.
See :ref:`yieldexpr`.

generator iterator
An object created by a :term:`generator function` or a
An :term:`iterator` created by a :term:`generator function` or a
:term:`generator expression`.

Each :keyword:`yield` temporarily suspends processing, remembering the
Expand All @@ -680,7 +680,7 @@ Glossary
:meth:`~generator.throw` method to raise an exception at the point
where the generator was paused. See :ref:`generator-methods`.

.. index:: single: generator expression
.. index:: single: generator expression

generator expression
An :term:`expression` that returns an :term:`iterator`. It looks like a normal expression
Expand Down
Loading
Loading