From 896be6fa2e29f956256963ac9e6e4c3cee8585d2 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:25:26 -0500 Subject: [PATCH 01/12] Write the motivation and rationale. --- peps/pep-0852.rst | 210 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 peps/pep-0852.rst diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst new file mode 100644 index 00000000000..313d7138491 --- /dev/null +++ b/peps/pep-0852.rst @@ -0,0 +1,210 @@ +PEP: 852 +Title: Asynchronous functions in the C API +Author: Peter Bierma +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 02-Nov-2026 +Python-Version: 3.16 +Post-History: `14-Jan-2023 `__, + `09-Jan-2024 `__, + `12-Apr-2024 `__ + + +Abstract +======== + +This PEP describes a new suite of functions in the C API to interact with +:term:`coroutines `. For example: + + +Motivation +========== + + +The C API has no easy way to invoke asynchronous code +----------------------------------------------------- + +Currently, Python's C API provides no simple interface for defining :term:`coroutine +functions `. This lack of API support has not gone unnoticed by +users. For example, ``mypyc`` still `does not support all asynchronous features +`__ as a result of the lack of support +in CPython. + +For extension developers, the current solution to this problem is to do one +of three things: + +1. Don't write the asynchronous code in C, and instead define a pure-Python + function that handles the :keyword:`await`\ s, and then call an + event-loop-specific API from C. Given that a common use case for building + extension modules is to improve performance, this can be a step backwards + for developers. +2. Implement a type that implements the :term:`coroutine` protocol on its own, + complete with all the necessary logic for a state machine. This is possible, + but non-trivial and bad for extension maintenance; C API users should not + have to learn about the inner workings of generators in order to use a + built-in syntax construct. +3. Use a binding generator that has done one of the above. To `quote + `__ + Petr Viktorin: "CPython should have a usable and stable C API for all + stakeholders, and a list of a few blessed users goes against that goal." + In other words, CPython should aim to make its C API usable (and ergonomic) + where possible -- not encourage users to switch to binding generators. + +All of these options are less than ideal, and as a result, there is support +from both Python users and core developers for adding such a C API to Python. +To `quote `__ Marc-André Lemburg: + + I very much do think that we need a core maintained async C API in CPython. + + We currently don’t have a way to write Python C extensions which work in + async mode without using a shim written in Python which takes care of + the interactions with the asyncio (and other) event loops. + + +A third-party library is bad for both maintainers and users +----------------------------------------------------------- + +The API described in this PEP is heavily influenced by the third-party +`PyAwaitable `__ library -- but why does that +need to come into CPython at all? + +First and foremost, adding a dependency to a C extension typically has a very +high bar. In most cases, it's deemed safer to pick one of the solutions described +above, which each come with their own set of problems. + +Second, distributing native dependencies is extremely difficult in Python's extension +ecosystem. As of recently, extension modules are increasingly being written +using C API binding libraries and not the C API itself. Unfortunately, these +binding libraries cannot wrap APIs that don't come directly from CPython, so +native libraries are effectively unusable to projects using other C API bindings. +In fact, PyAwaitable uses a vendoring technique that would make it functionally +impossible to use it from something such as PyO3. + +There are have been `some `__ `discussions +`__ about making this easier (both of whic +arised *because* PyAwaitable was so difficult to distribute), but in general, fixing +this problem seems like a much more massive proposal than what is currently being +proposed. An asynchronous C API in Python's core would be useful, and adding +one to CPython is much simpler and easier for both maintainers and users than +trying to solve a more generic problem in the packaging ecosystem; in other words, +to quote :pep:`20`, "practicality beats purity". + + +Why was a third-party library created in the first place? +********************************************************** + +When PyAwaitable was first created in early 2024, the author of the implementation +(who is also the author of this proposal) was new to CPython contribution and +thus not in a great place to author a PEP or oversee an implementation of PyAwaitable +in CPython. A relatively simple implementation was possible without changes to +the CPython core, and as such, PyAwaitable was better fit as a library for the +time being -- though always with the intent of eventually becoming part of the +public C API. + +The author has since become a core developer, and as such, it is now much more +feasible for them to maintain PyAwaitable in CPython directly. + + +Specification +============= + + +Rationale +========= + + +An API for coroutine functions, not futures +------------------------------------------- + +This proposal aims to add a C API for generic ``async``/``await`` constructs, *not* +an API for describing all capabilities of :mod:`asyncio`. While this PEP may +be reason for adding an ``asyncio`` C API in the future, its only goal is to +make :keyword:`async def` functions possible to implement in the C API. + + +Deviations from PyAwaitable +--------------------------- + + +``PyAsync`` instead of ``PyAwaitable`` +************************************** + +The most obvious deviation from the original library is the use of the +``PyAsync`` prefix over ``PyAwaitable``. This is because the original +"PyAwaitable" name is a historical artifact; it was chosen as +a parallel to the ``PyCoro`` prefix, with the idea that a ``PyCoro`` was +analogous to a native ``coroutine`` object, while a ``PyAwaitable`` would +be analogous to a native ``awaitable`` object. Given that this API (and new +versions of PyAwaitable) support all of the :class:`~collections.abc.Coroutine` +protocol (in contrast to the :class:`~collections.abc.Awaitable` protocol), +this name is now misleading, and thus the rename to ``PyAsync`` felt necessary. + + +Simpler storage for callback values +*********************************** + +This API uses a single ``PyObject *`` reference and ``void *`` pointer for value +storage between callbacks instead of variadic argument lists that were stored as +arrays. PyAwaitable used the latter because it was optimized for ergonomics when +writing pure C code, but since this API is also designed to be accessible from +bindings generators (and FFIs) as well, a single pointer for each felt like the +better call. PyAwaitable's array system also turned out to be a source of +complexity and bugs, and avoiding extra complexity for CPython where possible +is ideal. + + +Backwards Compatibility +======================= + +This PEP has no known backwards compatibility concerns. + + +Security Implications +===================== + +This PEP has no known security concerns. + + +How to Teach This +================= + +[How to teach users, new and experienced, how to apply the PEP to their work.] + + +Reference Implementation +======================== + +[Link to any existing implementation and details about its state, e.g. proof-of-concept.] + + +Rejected Ideas +============== + +TBD + + +Open Issues +=========== + +TBD + + +Acknowledgements +================ + +Thanks to Petr Viktorin for his help with the original PyAwaitable design +discussion, and to all users of PyAwaitable over the past few years. + + +Change History +============== + +TBD. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 73b66430a3f83722d904d94d98097557a9cca38c Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:41:15 -0500 Subject: [PATCH 02/12] Add the specification based on PyAwaitable's docs. --- peps/pep-0852.rst | 207 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 207 insertions(+) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 313d7138491..0808980a942 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -109,6 +109,203 @@ feasible for them to maintain PyAwaitable in CPython directly. Specification ============= +The following APIs are available through ``Python.h`` and require an +:term:`attached thread state`. Unless otherwise specified, an *awaitable* +argument must be a :c:type:`PyAsync` object. Callbacks receive borrowed +references to their arguments. + + +Async objects +------------- + +.. c:type:: PyAsync + + An opaque Python object that implements the :class:`~collections.abc.Coroutine` + protocol. A :c:type:`PyAsync` object stores a sequence of asynchronous + operations and callbacks to execute when it is awaited. + + +.. c:var:: PyTypeObject PyAsync_Type + + The type object for :c:type:`PyAsync` objects. + + +.. c:function:: int PyAsync_CheckExact(PyObject *obj) + + Return nonzero if *obj* is a :c:type:`PyAsync` object, and ``0`` otherwise. + This function cannot fail. + + +.. c:function:: PyObject *PyAsync_New(void) + + Create a new, empty :c:type:`PyAsync` object. Return a new + :term:`strong reference` on success, or ``NULL`` with an exception set + on failure. + + By default, awaiting the object returns ``None``. The result can be changed + with :c:func:`PyAsync_SetResult`. An object can only be awaited once; + attempting to reuse a completed object raises :exc:`RuntimeError`. + + Destroying an object that was never awaited or cancelled with + :c:func:`PyAsync_Cancel` emits a :exc:`ResourceWarning`. + + +.. c:function:: int PyAsync_SetResult(PyObject *awaitable, PyObject *result) + + Set the object returned by awaiting *awaitable* to *result*. + *result* must not be ``NULL``. + + This function stores a strong reference to *result*, releasing any + previously stored result. It cannot fail and returns ``0``. + + +Awaiting and callbacks +---------------------- + +.. c:type:: int (*PyAsync_Callback)(PyObject *awaitable, PyObject *result) + + The type of a result callback. *awaitable* is the :c:type:`PyAsync` + object executing the callback, and *result* is the value returned by + the awaited operation. + + The callback must return one of the following: + + * ``1`` on success. + * ``0`` with an exception set to invoke the associated error callback, + or propagate the exception if no error callback was provided. + * ``-1`` with an exception set to propagate the exception directly, + skipping the associated error callback. + + +.. c:type:: int (*PyAsync_Error)(PyObject *awaitable, PyObject *exception) + + The type of an error callback. *exception* is the exception raised by + an awaited operation or its result callback. The error callback is + invoked with no exception set. + + The callback must return one of the following: + + * ``1`` to handle the exception and continue executing queued operations. + * ``0`` to restore and propagate the original exception. + * ``-1`` with a new exception set to propagate that exception instead. + + +.. c:type:: int (*PyAsync_Defer)(PyObject *awaitable) + + The type of a deferred callback, registered with + :c:func:`PyAsync_DeferAwait`. The callback must return ``1`` on success, + or ``0`` with an exception set to stop execution and propagate the + exception to the caller awaiting *awaitable*. + + +.. c:function:: int PyAsync_AddAwait(PyObject *awaitable, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Append *coroutine* to the operations to await when *awaitable* executes. + Operations execute in the order they were added. This function stores a + strong reference to *coroutine*, which must be a non-``NULL`` awaitable + object distinct from *awaitable*. + + On successful completion, *result_callback* is invoked with *awaitable* + and the operation's return value. If *result_callback* is ``NULL``, the + return value is discarded. + + If the operation raises an exception, or *result_callback* returns ``0``, + *error_callback* is invoked. If *error_callback* is ``NULL``, the exception + propagates to the caller awaiting *awaitable*. + + Callbacks may append further operations to *awaitable*. Adding operations + to a completed object raises :exc:`RuntimeError`. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +.. c:function:: int PyAsync_AddExpr(PyObject *awaitable, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Like :c:func:`PyAsync_AddAwait`, but steals a reference to *expr* on + both success and failure. If *expr* is ``NULL``, return ``-1`` without + changing the current exception. + + Otherwise, return ``0`` on success, or ``-1`` with an exception set + on failure. + + +.. c:function:: int PyAsync_DeferAwait(PyObject *awaitable, PyAsync_Defer callback) + + Append a synchronous callback to the operations executed when *awaitable* + is awaited. *callback* must not be ``NULL``. It is invoked with *awaitable* + after previously queued operations complete, and may queue further + operations. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +.. c:function:: void PyAsync_Cancel(PyObject *awaitable) + + Clear the queued operations and callbacks, releasing their references + and discarding the current await iterator. This also suppresses the + :exc:`ResourceWarning` for an object that was never awaited. + + This function may be called from a callback. Further operations may be + added to *awaitable* until it completes. This function cannot fail. + + +.. c:function:: int PyAsync_AsyncWith(PyObject *awaitable, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Append an :keyword:`async with` operation using *context_manager*. + Await its ``__aenter__`` method, invoke *result_callback*, and await its + ``__aexit__`` method. *result_callback* receives a :c:type:`PyAsync` + object representing the context manager operation and the value + returned by ``__aenter__``. It may be ``NULL``. + + If *result_callback* fails, its exception is passed to ``__aexit__``, + which may suppress it by returning a true value. An exception from + ``__aenter__`` propagates without invoking ``__aexit__``. + + *error_callback* handles exceptions from the operation, including any + exception left unsuppressed by ``__aexit__``. It receives *awaitable* + and the exception. If it is ``NULL``, the exception propagates to the + caller awaiting *awaitable*. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +Value storage +------------- + +Each :c:type:`PyAsync` object has one object slot and one arbitrary pointer +slot for storing state between callbacks. Both slots are initially ``NULL``. + +.. c:function:: int PyAsync_SetValue(PyObject *awaitable, PyObject *value) + + Store a strong reference to *value* in the object slot, releasing the + previously stored reference. If *value* is ``NULL``, clear the slot. + + This function cannot fail and returns ``0``. + + +.. c:function:: PyObject *PyAsync_GetValue(PyObject *awaitable) + + Return a :term:`borrowed reference` to the object stored in the object + slot, or ``NULL`` if the slot is empty. This function cannot fail and + does not set an exception when the slot is empty. + + +.. c:function:: int PyAsync_SetArbValue(PyObject *awaitable, void *value) + + Store *value* in the arbitrary pointer slot, replacing the previous + pointer. *value* may be ``NULL``. The caller is responsible for the + lifetime of any memory referenced by the pointer. + + This function cannot fail and returns ``0``. + + +.. c:function:: void *PyAsync_GetArbValue(PyObject *awaitable) + + Return the pointer stored in the arbitrary pointer slot, or ``NULL`` + if the slot is empty. This function cannot fail and does not set an + exception when the slot is empty. + + Rationale ========= @@ -154,6 +351,16 @@ complexity and bugs, and avoiding extra complexity for CPython where possible is ideal. +Different callback return values +******************************** + +In PyAwaitable, callbacks may return ``0``, ``-1``, and ``-2``, which correspond +to ``1``, ``0``, and ``-1`` in this proposal, respectively. This is also a historical +artifact; PyAwaitable did not originally have the ``-2`` return value, and changing +the meaning of ``0`` and ``-1`` could not be done for the sake of backwards +compatibility. + + Backwards Compatibility ======================= From c445a84ded9ccdd588e94d06ecf5f89a952c58ed Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:48:59 -0500 Subject: [PATCH 03/12] Use strong references instead of borrowed references. --- peps/pep-0852.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 0808980a942..a8135087f15 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -285,7 +285,7 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. .. c:function:: PyObject *PyAsync_GetValue(PyObject *awaitable) - Return a :term:`borrowed reference` to the object stored in the object + Return a new :term:`strong reference` to the object stored in the object slot, or ``NULL`` if the slot is empty. This function cannot fail and does not set an exception when the slot is empty. @@ -350,6 +350,11 @@ better call. PyAwaitable's array system also turned out to be a source of complexity and bugs, and avoiding extra complexity for CPython where possible is ideal. +This API also returns a :term:`strong reference` from ``GetValue`` instead of a +:term:`borrowed reference`. PyAwaitable used the latter because it predated the +acceptance of :pep:`703`, and thus any concurrency implications were not +considered. + Different callback return values ******************************** From 3009471eefab4f381ff0ba0165bc61eb9c10b16a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:52:06 -0500 Subject: [PATCH 04/12] Avoid the outdated argument names. --- peps/pep-0852.rst | 54 +++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index a8135087f15..c3317fa72a1 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -110,7 +110,7 @@ Specification ============= The following APIs are available through ``Python.h`` and require an -:term:`attached thread state`. Unless otherwise specified, an *awaitable* +:term:`attached thread state`. Unless otherwise specified, an *asyncobj* argument must be a :c:type:`PyAsync` object. Callbacks receive borrowed references to their arguments. @@ -150,9 +150,9 @@ Async objects :c:func:`PyAsync_Cancel` emits a :exc:`ResourceWarning`. -.. c:function:: int PyAsync_SetResult(PyObject *awaitable, PyObject *result) +.. c:function:: int PyAsync_SetResult(PyObject *asyncobj, PyObject *result) - Set the object returned by awaiting *awaitable* to *result*. + Set the object returned by awaiting *asyncobj* to *result*. *result* must not be ``NULL``. This function stores a strong reference to *result*, releasing any @@ -162,9 +162,9 @@ Async objects Awaiting and callbacks ---------------------- -.. c:type:: int (*PyAsync_Callback)(PyObject *awaitable, PyObject *result) +.. c:type:: int (*PyAsync_Callback)(PyObject *asyncobj, PyObject *result) - The type of a result callback. *awaitable* is the :c:type:`PyAsync` + The type of a result callback. *asyncobj* is the :c:type:`PyAsync` object executing the callback, and *result* is the value returned by the awaited operation. @@ -177,7 +177,7 @@ Awaiting and callbacks skipping the associated error callback. -.. c:type:: int (*PyAsync_Error)(PyObject *awaitable, PyObject *exception) +.. c:type:: int (*PyAsync_Error)(PyObject *asyncobj, PyObject *exception) The type of an error callback. *exception* is the exception raised by an awaited operation or its result callback. The error callback is @@ -190,36 +190,36 @@ Awaiting and callbacks * ``-1`` with a new exception set to propagate that exception instead. -.. c:type:: int (*PyAsync_Defer)(PyObject *awaitable) +.. c:type:: int (*PyAsync_Defer)(PyObject *asyncobj) The type of a deferred callback, registered with :c:func:`PyAsync_DeferAwait`. The callback must return ``1`` on success, or ``0`` with an exception set to stop execution and propagate the - exception to the caller awaiting *awaitable*. + exception to the caller awaiting *asyncobj*. -.. c:function:: int PyAsync_AddAwait(PyObject *awaitable, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AddAwait(PyObject *asyncobj, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) - Append *coroutine* to the operations to await when *awaitable* executes. + Append *coroutine* to the operations to await when *asyncobj* executes. Operations execute in the order they were added. This function stores a strong reference to *coroutine*, which must be a non-``NULL`` awaitable - object distinct from *awaitable*. + object distinct from *asyncobj*. - On successful completion, *result_callback* is invoked with *awaitable* + On successful completion, *result_callback* is invoked with *asyncobj* and the operation's return value. If *result_callback* is ``NULL``, the return value is discarded. If the operation raises an exception, or *result_callback* returns ``0``, *error_callback* is invoked. If *error_callback* is ``NULL``, the exception - propagates to the caller awaiting *awaitable*. + propagates to the caller awaiting *asyncobj*. - Callbacks may append further operations to *awaitable*. Adding operations + Callbacks may append further operations to *asyncobj*. Adding operations to a completed object raises :exc:`RuntimeError`. Return ``0`` on success, or ``-1`` with an exception set on failure. -.. c:function:: int PyAsync_AddExpr(PyObject *awaitable, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AddExpr(PyObject *asyncobj, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) Like :c:func:`PyAsync_AddAwait`, but steals a reference to *expr* on both success and failure. If *expr* is ``NULL``, return ``-1`` without @@ -229,27 +229,27 @@ Awaiting and callbacks on failure. -.. c:function:: int PyAsync_DeferAwait(PyObject *awaitable, PyAsync_Defer callback) +.. c:function:: int PyAsync_DeferAwait(PyObject *asyncobj, PyAsync_Defer callback) - Append a synchronous callback to the operations executed when *awaitable* - is awaited. *callback* must not be ``NULL``. It is invoked with *awaitable* + Append a synchronous callback to the operations executed when *asyncobj* + is awaited. *callback* must not be ``NULL``. It is invoked with *asyncobj* after previously queued operations complete, and may queue further operations. Return ``0`` on success, or ``-1`` with an exception set on failure. -.. c:function:: void PyAsync_Cancel(PyObject *awaitable) +.. c:function:: void PyAsync_Cancel(PyObject *asyncobj) Clear the queued operations and callbacks, releasing their references and discarding the current await iterator. This also suppresses the :exc:`ResourceWarning` for an object that was never awaited. This function may be called from a callback. Further operations may be - added to *awaitable* until it completes. This function cannot fail. + added to *asyncobj* until it completes. This function cannot fail. -.. c:function:: int PyAsync_AsyncWith(PyObject *awaitable, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AsyncWith(PyObject *asyncobj, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) Append an :keyword:`async with` operation using *context_manager*. Await its ``__aenter__`` method, invoke *result_callback*, and await its @@ -262,9 +262,9 @@ Awaiting and callbacks ``__aenter__`` propagates without invoking ``__aexit__``. *error_callback* handles exceptions from the operation, including any - exception left unsuppressed by ``__aexit__``. It receives *awaitable* + exception left unsuppressed by ``__aexit__``. It receives *asyncobj* and the exception. If it is ``NULL``, the exception propagates to the - caller awaiting *awaitable*. + caller awaiting *asyncobj*. Return ``0`` on success, or ``-1`` with an exception set on failure. @@ -275,7 +275,7 @@ Value storage Each :c:type:`PyAsync` object has one object slot and one arbitrary pointer slot for storing state between callbacks. Both slots are initially ``NULL``. -.. c:function:: int PyAsync_SetValue(PyObject *awaitable, PyObject *value) +.. c:function:: int PyAsync_SetValue(PyObject *asyncobj, PyObject *value) Store a strong reference to *value* in the object slot, releasing the previously stored reference. If *value* is ``NULL``, clear the slot. @@ -283,14 +283,14 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. This function cannot fail and returns ``0``. -.. c:function:: PyObject *PyAsync_GetValue(PyObject *awaitable) +.. c:function:: PyObject *PyAsync_GetValue(PyObject *asyncobj) Return a new :term:`strong reference` to the object stored in the object slot, or ``NULL`` if the slot is empty. This function cannot fail and does not set an exception when the slot is empty. -.. c:function:: int PyAsync_SetArbValue(PyObject *awaitable, void *value) +.. c:function:: int PyAsync_SetArbValue(PyObject *asyncobj, void *value) Store *value* in the arbitrary pointer slot, replacing the previous pointer. *value* may be ``NULL``. The caller is responsible for the @@ -299,7 +299,7 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. This function cannot fail and returns ``0``. -.. c:function:: void *PyAsync_GetArbValue(PyObject *awaitable) +.. c:function:: void *PyAsync_GetArbValue(PyObject *asyncobj) Return the pointer stored in the arbitrary pointer slot, or ``NULL`` if the slot is empty. This function cannot fail and does not set an From cec7a1493ba5a4415fa99e856bcc86a68803cccf Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 15:12:36 -0500 Subject: [PATCH 05/12] Add an examples section. --- peps/pep-0852.rst | 182 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 181 insertions(+), 1 deletion(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index c3317fa72a1..206e297df2b 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -381,7 +381,187 @@ This PEP has no known security concerns. How to Teach This ================= -[How to teach users, new and experienced, how to apply the PEP to their work.] +The new APIs will be documented in the C API documentation. +A tutorial can also be added based on the tutorial present in PyAwaitable's +documentation. + +Examples +-------- + +The first three examples adapt patterns from PyAwaitable's documentation. +Callbacks return ``1`` on success, while functions such as +:c:func:`PyAsync_AddExpr` return ``0`` on success. + + +Example: Forwarding an asynchronous result +****************************************** + +An extension method can call an asynchronous function and forward the +result of awaiting it. :c:type:`PyAsync` handles suspension and resumption +for the extension. + +.. code-block:: c + + static int + return_result(PyObject *asyncobj, PyObject *result) + { + PyAsync_SetResult(asyncobj, result); + return 1; + } + + // Equivalent to def call_async(func): return await func() + static PyObject * + call_async(PyObject *self, PyObject *func) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AddExpr(asyncobj, PyObject_CallNoArgs(func), + return_result, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + + +Example: Keeping state across an await +************************************** + +To implement ``return (await get_number()) * factor``, store *factor* with +:c:func:`PyAsync_SetValue` before queuing ``get_number()`` with this callback. +The :c:type:`PyAsync` object keeps the factor alive while execution is +suspended. The callback releases the strong reference returned by +:c:func:`PyAsync_GetValue` after using it. + +.. code-block:: c + + static int + multiply_result(PyObject *asyncobj, PyObject *value) + { + PyObject *factor = PyAsync_GetValue(asyncobj); + PyObject *result = PyNumber_Multiply(value, factor); + Py_DECREF(factor); + if (result == NULL) { + return 0; + } + PyAsync_SetResult(asyncobj, result); + Py_DECREF(result); + return 1; + } + + +Example: Handling a timeout +*************************** + +In ``call_async`` above, replace the final ``NULL`` argument to +:c:func:`PyAsync_AddExpr` with ``timeout_to_none``. Awaiting the result +then returns ``None`` on :exc:`TimeoutError`. Returning ``0`` from the +callback lets other exceptions propagate. + +.. code-block:: c + + static int + timeout_to_none(PyObject *asyncobj, PyObject *exception) + { + if (!PyErr_GivenExceptionMatches(exception, PyExc_TimeoutError)) { + return 0; + } + PyAsync_SetResult(asyncobj, Py_None); + return 1; + } + +For example, :func:`asyncio.timeout` can impose a deadline on an operation: + +.. code-block:: python + + import asyncio + + async def slow_operation(): + async with asyncio.timeout(0.01): + return await asyncio.sleep(1, result=42) + + assert asyncio.run(call_async(slow_operation)) is None + + +Example: Scheduling a dependent read +************************************ + +Suppose a protocol reader exposes asynchronous ``read_size()`` and +``readexactly(size)`` methods. Store the reader with +:c:func:`PyAsync_SetValue`, then queue ``reader.read_size()`` with this +callback. The first read's result determines the next asynchronous call, +which reuses ``return_result`` to return the completed packet. + +.. code-block:: c + + static int + read_body(PyObject *asyncobj, PyObject *size) + { + PyObject *reader = PyAsync_GetValue(asyncobj); + PyObject *coro = PyObject_CallMethod(reader, "readexactly", "O", size); + Py_DECREF(reader); + return PyAsync_AddExpr(asyncobj, coro, return_result, NULL) == 0; + } + + +Example: Acquiring an async lock for native work +************************************************ + +An asynchronous context manager can acquire a lock and return the protected +dictionary to the result callback: + +.. code-block:: python + + import asyncio + from contextlib import asynccontextmanager + + @asynccontextmanager + async def locked_cache(lock, cache): + async with lock: + yield cache + +The C callback updates that dictionary while the lock is held: + +.. code-block:: c + + static int + mark_ready(PyObject *asyncobj, PyObject *cache) + { + return PyDict_SetItemString(cache, "ready", Py_True) == 0; + } + + static PyObject * + update_locked(PyObject *self, PyObject *guard) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AsyncWith(asyncobj, guard, mark_ready, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + +Python code passes the lock and dictionary through the context manager: + +.. code-block:: python + + async def main(): + lock, cache = asyncio.Lock(), {} + await update_locked(locked_cache(lock, cache)) + assert cache["ready"] is True + assert not lock.locked() + + asyncio.run(main()) + +:c:func:`PyAsync_AsyncWith` invokes the result callback after acquiring the +lock, and releases the lock even if the dictionary update raises an exception. Reference Implementation From 3a37ca43e0377f5b3672f09505cf998b8db64d50 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 15:28:21 -0500 Subject: [PATCH 06/12] Some cleanup. --- peps/pep-0852.rst | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 206e297df2b..590cf0b60e9 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -17,6 +17,33 @@ Abstract This PEP describes a new suite of functions in the C API to interact with :term:`coroutines `. For example: +.. code-block:: c + + #include + + static int + return_result(PyObject *asyncobj, PyObject *result) + { + PyAsync_SetResult(asyncobj, result); + return 1; + } + + /* Equivalent to: async def await_from_c(coro): return await coro */ + static PyObject * + await_from_c(PyObject *self, PyObject *coro) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AddAwait(asyncobj, coro, return_result, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + Motivation ========== @@ -81,9 +108,9 @@ native libraries are effectively unusable to projects using other C API bindings In fact, PyAwaitable uses a vendoring technique that would make it functionally impossible to use it from something such as PyO3. -There are have been `some `__ `discussions -`__ about making this easier (both of whic -arised *because* PyAwaitable was so difficult to distribute), but in general, fixing +There have been `some `__ `discussions +`__ about making this easier (both of which +arose *because* PyAwaitable was so difficult to distribute), but in general, fixing this problem seems like a much more massive proposal than what is currently being proposed. An asynchronous C API in Python's core would be useful, and adding one to CPython is much simpler and easier for both maintainers and users than @@ -385,13 +412,10 @@ The new APIs will be documented in the C API documentation. A tutorial can also be added based on the tutorial present in PyAwaitable's documentation. + Examples -------- -The first three examples adapt patterns from PyAwaitable's documentation. -Callbacks return ``1`` on success, while functions such as -:c:func:`PyAsync_AddExpr` return ``0`` on success. - Example: Forwarding an asynchronous result ****************************************** From 204c7fd0148da58df6f7e52dbd15c22f641585aa Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 15:41:08 -0500 Subject: [PATCH 07/12] Add reference implementation. --- peps/pep-0852.rst | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 590cf0b60e9..502e95b073a 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -55,8 +55,8 @@ The C API has no easy way to invoke asynchronous code Currently, Python's C API provides no simple interface for defining :term:`coroutine functions `. This lack of API support has not gone unnoticed by users. For example, ``mypyc`` still `does not support all asynchronous features -`__ as a result of the lack of support -in CPython. +`__, because they are forced to +build coroutine types from scratch, which is difficult to do correctly. For extension developers, the current solution to this problem is to do one of three things: @@ -73,10 +73,11 @@ of three things: built-in syntax construct. 3. Use a binding generator that has done one of the above. To `quote `__ - Petr Viktorin: "CPython should have a usable and stable C API for all - stakeholders, and a list of a few blessed users goes against that goal." - In other words, CPython should aim to make its C API usable (and ergonomic) - where possible -- not encourage users to switch to binding generators. + Petr Viktorin (on a separate proposal): "CPython should have a usable and + stable C API for all stakeholders, and a list of a few blessed users goes + against that goal." In other words, CPython should aim to make its C API + usable (and ergonomic) where possible -- not encourage users to switch to + binding generators. All of these options are less than ideal, and as a result, there is support from both Python users and core developers for adding such a C API to Python. @@ -132,6 +133,11 @@ public C API. The author has since become a core developer, and as such, it is now much more feasible for them to maintain PyAwaitable in CPython directly. +It is also worth noting that the reference implementation of this proposal does +require the use of some internal helper functions (that PyAwaitable, as a +third-party, had to copy and work around), which is more evidence that PyAwaitable +is a good fit for CPython. + Specification ============= @@ -333,6 +339,11 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. exception when the slot is empty. +Limited API +----------- + +All APIs in this PEP will be made part of the :ref:`Stable ABI` in Python 3.16. + Rationale ========= @@ -346,6 +357,11 @@ an API for describing all capabilities of :mod:`asyncio`. While this PEP may be reason for adding an ``asyncio`` C API in the future, its only goal is to make :keyword:`async def` functions possible to implement in the C API. +As such, this proposal doesn't come with any generic mechanism to implement +suspension of a C function, though it is still possible to implement non-blocking +native operations using other parts of Python's C API (such as by manually creating +:class:`asyncio.Future` instances through one of the call APIs) + Deviations from PyAwaitable --------------------------- @@ -591,8 +607,8 @@ lock, and releases the lock even if the dictionary update raises an exception. Reference Implementation ======================== -[Link to any existing implementation and details about its state, e.g. proof-of-concept.] - +A reference implementation of this PEP can be found +`here `__. Rejected Ideas ============== From 35ebb40c5ffdce2af2442f2295989778420b1e8b Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 18:54:22 -0500 Subject: [PATCH 08/12] Touch ups. --- peps/pep-0852.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 502e95b073a..7496adaa716 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -30,7 +30,7 @@ This PEP describes a new suite of functions in the C API to interact with /* Equivalent to: async def await_from_c(coro): return await coro */ static PyObject * - await_from_c(PyObject *self, PyObject *coro) /* METH_O */ + await_from_c(PyObject *self, PyObject *coro) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { @@ -70,7 +70,7 @@ of three things: complete with all the necessary logic for a state machine. This is possible, but non-trivial and bad for extension maintenance; C API users should not have to learn about the inner workings of generators in order to use a - built-in syntax construct. + built-in syntax construct. This is the root of ``mypyc``'s problem, for example. 3. Use a binding generator that has done one of the above. To `quote `__ Petr Viktorin (on a separate proposal): "CPython should have a usable and @@ -451,7 +451,7 @@ for the extension. // Equivalent to def call_async(func): return await func() static PyObject * - call_async(PyObject *self, PyObject *func) /* METH_O */ + call_async(PyObject *self, PyObject *func) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { @@ -574,7 +574,7 @@ The C callback updates that dictionary while the lock is held: } static PyObject * - update_locked(PyObject *self, PyObject *guard) /* METH_O */ + update_locked(PyObject *self, PyObject *guard) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { @@ -610,16 +610,17 @@ Reference Implementation A reference implementation of this PEP can be found `here `__. + Rejected Ideas ============== -TBD +TBD. Open Issues =========== -TBD +TBD. Acknowledgements From 0e1ec9b93f72a83467d9420c86cd4237fe550e96 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 18:57:19 -0500 Subject: [PATCH 09/12] Add codeowner. --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 944a91ed755..c30f6960125 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -730,6 +730,7 @@ peps/pep-0849.rst @JelleZijlstra peps/pep-0850/ @willingc @rgommers peps/pep-0850.rst @willingc @rgommers peps/pep-0851.rst @StanFromIreland +peps/pep-0852.rst @ZeroIntensity # ... peps/pep-2026.rst @hugovk # ... From f08e5a35d3d44fcb17dcb2337735ce112af203fc Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 18:59:26 -0500 Subject: [PATCH 10/12] Remove misleading paragraph. --- peps/pep-0852.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 7496adaa716..c2fa8b8606d 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -357,11 +357,6 @@ an API for describing all capabilities of :mod:`asyncio`. While this PEP may be reason for adding an ``asyncio`` C API in the future, its only goal is to make :keyword:`async def` functions possible to implement in the C API. -As such, this proposal doesn't come with any generic mechanism to implement -suspension of a C function, though it is still possible to implement non-blocking -native operations using other parts of Python's C API (such as by manually creating -:class:`asyncio.Future` instances through one of the call APIs) - Deviations from PyAwaitable --------------------------- From e107cb300ba772491d6ed896fadc6379744407db Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 19:08:26 -0500 Subject: [PATCH 11/12] Strengthen motivation a little bit. --- peps/pep-0852.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index c2fa8b8606d..7a9780526b4 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -128,15 +128,15 @@ thus not in a great place to author a PEP or oversee an implementation of PyAwai in CPython. A relatively simple implementation was possible without changes to the CPython core, and as such, PyAwaitable was better fit as a library for the time being -- though always with the intent of eventually becoming part of the -public C API. - -The author has since become a core developer, and as such, it is now much more -feasible for them to maintain PyAwaitable in CPython directly. +public C API. The author has since become a core developer, and as such, it is +now much more feasible for them to maintain a variant of PyAwaitable in CPython +directly. It is also worth noting that the reference implementation of this proposal does require the use of some internal helper functions (that PyAwaitable, as a third-party, had to copy and work around), which is more evidence that PyAwaitable -is a good fit for CPython. +is a good fit for CPython, as one of the original arguments against adding such +an API was that it was implementable using only the public C API. Specification From d03f89576a5217c95fe5bfe766e628630bf1aa6e Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 20:36:11 -0500 Subject: [PATCH 12/12] Fix Sphinx errors. --- peps/pep-0852.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 7a9780526b4..150fa525db9 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -342,7 +342,7 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. Limited API ----------- -All APIs in this PEP will be made part of the :ref:`Stable ABI` in Python 3.16. +All APIs in this PEP will be made part of the :ref:`Stable ABI ` in Python 3.16. Rationale @@ -569,13 +569,13 @@ The C callback updates that dictionary while the lock is held: } static PyObject * - update_locked(PyObject *self, PyObject *guard) /* METH_O */ + update_locked(PyObject *self, PyObject *lock) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { return NULL; } - if (PyAsync_AsyncWith(asyncobj, guard, mark_ready, NULL) < 0) { + if (PyAsync_AsyncWith(asyncobj, lock, mark_ready, NULL) < 0) { PyAsync_Cancel(asyncobj); Py_DECREF(asyncobj); return NULL;