diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 944a91ed755..c30f6960125 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -730,6 +730,7 @@ peps/pep-0849.rst @JelleZijlstra peps/pep-0850/ @willingc @rgommers peps/pep-0850.rst @willingc @rgommers peps/pep-0851.rst @StanFromIreland +peps/pep-0852.rst @ZeroIntensity # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst new file mode 100644 index 00000000000..150fa525db9 --- /dev/null +++ b/peps/pep-0852.rst @@ -0,0 +1,638 @@ +PEP: 852 +Title: Asynchronous functions in the C API +Author: Peter Bierma +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 02-Nov-2026 +Python-Version: 3.16 +Post-History: `14-Jan-2023 `__, + `09-Jan-2024 `__, + `12-Apr-2024 `__ + + +Abstract +======== + +This PEP describes a new suite of functions in the C API to interact with +:term:`coroutines `. For example: + +.. code-block:: c + + #include + + static int + return_result(PyObject *asyncobj, PyObject *result) + { + PyAsync_SetResult(asyncobj, result); + return 1; + } + + /* Equivalent to: async def await_from_c(coro): return await coro */ + static PyObject * + await_from_c(PyObject *self, PyObject *coro) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AddAwait(asyncobj, coro, return_result, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + + +Motivation +========== + + +The C API has no easy way to invoke asynchronous code +----------------------------------------------------- + +Currently, Python's C API provides no simple interface for defining :term:`coroutine +functions `. This lack of API support has not gone unnoticed by +users. For example, ``mypyc`` still `does not support all asynchronous features +`__, because they are forced to +build coroutine types from scratch, which is difficult to do correctly. + +For extension developers, the current solution to this problem is to do one +of three things: + +1. Don't write the asynchronous code in C, and instead define a pure-Python + function that handles the :keyword:`await`\ s, and then call an + event-loop-specific API from C. Given that a common use case for building + extension modules is to improve performance, this can be a step backwards + for developers. +2. Implement a type that implements the :term:`coroutine` protocol on its own, + complete with all the necessary logic for a state machine. This is possible, + but non-trivial and bad for extension maintenance; C API users should not + have to learn about the inner workings of generators in order to use a + built-in syntax construct. This is the root of ``mypyc``'s problem, for example. +3. Use a binding generator that has done one of the above. To `quote + `__ + Petr Viktorin (on a separate proposal): "CPython should have a usable and + stable C API for all stakeholders, and a list of a few blessed users goes + against that goal." In other words, CPython should aim to make its C API + usable (and ergonomic) where possible -- not encourage users to switch to + binding generators. + +All of these options are less than ideal, and as a result, there is support +from both Python users and core developers for adding such a C API to Python. +To `quote `__ Marc-André Lemburg: + + I very much do think that we need a core maintained async C API in CPython. + + We currently don’t have a way to write Python C extensions which work in + async mode without using a shim written in Python which takes care of + the interactions with the asyncio (and other) event loops. + + +A third-party library is bad for both maintainers and users +----------------------------------------------------------- + +The API described in this PEP is heavily influenced by the third-party +`PyAwaitable `__ library -- but why does that +need to come into CPython at all? + +First and foremost, adding a dependency to a C extension typically has a very +high bar. In most cases, it's deemed safer to pick one of the solutions described +above, which each come with their own set of problems. + +Second, distributing native dependencies is extremely difficult in Python's extension +ecosystem. As of recently, extension modules are increasingly being written +using C API binding libraries and not the C API itself. Unfortunately, these +binding libraries cannot wrap APIs that don't come directly from CPython, so +native libraries are effectively unusable to projects using other C API bindings. +In fact, PyAwaitable uses a vendoring technique that would make it functionally +impossible to use it from something such as PyO3. + +There have been `some `__ `discussions +`__ about making this easier (both of which +arose *because* PyAwaitable was so difficult to distribute), but in general, fixing +this problem seems like a much more massive proposal than what is currently being +proposed. An asynchronous C API in Python's core would be useful, and adding +one to CPython is much simpler and easier for both maintainers and users than +trying to solve a more generic problem in the packaging ecosystem; in other words, +to quote :pep:`20`, "practicality beats purity". + + +Why was a third-party library created in the first place? +********************************************************** + +When PyAwaitable was first created in early 2024, the author of the implementation +(who is also the author of this proposal) was new to CPython contribution and +thus not in a great place to author a PEP or oversee an implementation of PyAwaitable +in CPython. A relatively simple implementation was possible without changes to +the CPython core, and as such, PyAwaitable was better fit as a library for the +time being -- though always with the intent of eventually becoming part of the +public C API. The author has since become a core developer, and as such, it is +now much more feasible for them to maintain a variant of PyAwaitable in CPython +directly. + +It is also worth noting that the reference implementation of this proposal does +require the use of some internal helper functions (that PyAwaitable, as a +third-party, had to copy and work around), which is more evidence that PyAwaitable +is a good fit for CPython, as one of the original arguments against adding such +an API was that it was implementable using only the public C API. + + +Specification +============= + +The following APIs are available through ``Python.h`` and require an +:term:`attached thread state`. Unless otherwise specified, an *asyncobj* +argument must be a :c:type:`PyAsync` object. Callbacks receive borrowed +references to their arguments. + + +Async objects +------------- + +.. c:type:: PyAsync + + An opaque Python object that implements the :class:`~collections.abc.Coroutine` + protocol. A :c:type:`PyAsync` object stores a sequence of asynchronous + operations and callbacks to execute when it is awaited. + + +.. c:var:: PyTypeObject PyAsync_Type + + The type object for :c:type:`PyAsync` objects. + + +.. c:function:: int PyAsync_CheckExact(PyObject *obj) + + Return nonzero if *obj* is a :c:type:`PyAsync` object, and ``0`` otherwise. + This function cannot fail. + + +.. c:function:: PyObject *PyAsync_New(void) + + Create a new, empty :c:type:`PyAsync` object. Return a new + :term:`strong reference` on success, or ``NULL`` with an exception set + on failure. + + By default, awaiting the object returns ``None``. The result can be changed + with :c:func:`PyAsync_SetResult`. An object can only be awaited once; + attempting to reuse a completed object raises :exc:`RuntimeError`. + + Destroying an object that was never awaited or cancelled with + :c:func:`PyAsync_Cancel` emits a :exc:`ResourceWarning`. + + +.. c:function:: int PyAsync_SetResult(PyObject *asyncobj, PyObject *result) + + Set the object returned by awaiting *asyncobj* to *result*. + *result* must not be ``NULL``. + + This function stores a strong reference to *result*, releasing any + previously stored result. It cannot fail and returns ``0``. + + +Awaiting and callbacks +---------------------- + +.. c:type:: int (*PyAsync_Callback)(PyObject *asyncobj, PyObject *result) + + The type of a result callback. *asyncobj* is the :c:type:`PyAsync` + object executing the callback, and *result* is the value returned by + the awaited operation. + + The callback must return one of the following: + + * ``1`` on success. + * ``0`` with an exception set to invoke the associated error callback, + or propagate the exception if no error callback was provided. + * ``-1`` with an exception set to propagate the exception directly, + skipping the associated error callback. + + +.. c:type:: int (*PyAsync_Error)(PyObject *asyncobj, PyObject *exception) + + The type of an error callback. *exception* is the exception raised by + an awaited operation or its result callback. The error callback is + invoked with no exception set. + + The callback must return one of the following: + + * ``1`` to handle the exception and continue executing queued operations. + * ``0`` to restore and propagate the original exception. + * ``-1`` with a new exception set to propagate that exception instead. + + +.. c:type:: int (*PyAsync_Defer)(PyObject *asyncobj) + + The type of a deferred callback, registered with + :c:func:`PyAsync_DeferAwait`. The callback must return ``1`` on success, + or ``0`` with an exception set to stop execution and propagate the + exception to the caller awaiting *asyncobj*. + + +.. c:function:: int PyAsync_AddAwait(PyObject *asyncobj, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Append *coroutine* to the operations to await when *asyncobj* executes. + Operations execute in the order they were added. This function stores a + strong reference to *coroutine*, which must be a non-``NULL`` awaitable + object distinct from *asyncobj*. + + On successful completion, *result_callback* is invoked with *asyncobj* + and the operation's return value. If *result_callback* is ``NULL``, the + return value is discarded. + + If the operation raises an exception, or *result_callback* returns ``0``, + *error_callback* is invoked. If *error_callback* is ``NULL``, the exception + propagates to the caller awaiting *asyncobj*. + + Callbacks may append further operations to *asyncobj*. Adding operations + to a completed object raises :exc:`RuntimeError`. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +.. c:function:: int PyAsync_AddExpr(PyObject *asyncobj, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Like :c:func:`PyAsync_AddAwait`, but steals a reference to *expr* on + both success and failure. If *expr* is ``NULL``, return ``-1`` without + changing the current exception. + + Otherwise, return ``0`` on success, or ``-1`` with an exception set + on failure. + + +.. c:function:: int PyAsync_DeferAwait(PyObject *asyncobj, PyAsync_Defer callback) + + Append a synchronous callback to the operations executed when *asyncobj* + is awaited. *callback* must not be ``NULL``. It is invoked with *asyncobj* + after previously queued operations complete, and may queue further + operations. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +.. c:function:: void PyAsync_Cancel(PyObject *asyncobj) + + Clear the queued operations and callbacks, releasing their references + and discarding the current await iterator. This also suppresses the + :exc:`ResourceWarning` for an object that was never awaited. + + This function may be called from a callback. Further operations may be + added to *asyncobj* until it completes. This function cannot fail. + + +.. c:function:: int PyAsync_AsyncWith(PyObject *asyncobj, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Append an :keyword:`async with` operation using *context_manager*. + Await its ``__aenter__`` method, invoke *result_callback*, and await its + ``__aexit__`` method. *result_callback* receives a :c:type:`PyAsync` + object representing the context manager operation and the value + returned by ``__aenter__``. It may be ``NULL``. + + If *result_callback* fails, its exception is passed to ``__aexit__``, + which may suppress it by returning a true value. An exception from + ``__aenter__`` propagates without invoking ``__aexit__``. + + *error_callback* handles exceptions from the operation, including any + exception left unsuppressed by ``__aexit__``. It receives *asyncobj* + and the exception. If it is ``NULL``, the exception propagates to the + caller awaiting *asyncobj*. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +Value storage +------------- + +Each :c:type:`PyAsync` object has one object slot and one arbitrary pointer +slot for storing state between callbacks. Both slots are initially ``NULL``. + +.. c:function:: int PyAsync_SetValue(PyObject *asyncobj, PyObject *value) + + Store a strong reference to *value* in the object slot, releasing the + previously stored reference. If *value* is ``NULL``, clear the slot. + + This function cannot fail and returns ``0``. + + +.. c:function:: PyObject *PyAsync_GetValue(PyObject *asyncobj) + + Return a new :term:`strong reference` to the object stored in the object + slot, or ``NULL`` if the slot is empty. This function cannot fail and + does not set an exception when the slot is empty. + + +.. c:function:: int PyAsync_SetArbValue(PyObject *asyncobj, void *value) + + Store *value* in the arbitrary pointer slot, replacing the previous + pointer. *value* may be ``NULL``. The caller is responsible for the + lifetime of any memory referenced by the pointer. + + This function cannot fail and returns ``0``. + + +.. c:function:: void *PyAsync_GetArbValue(PyObject *asyncobj) + + Return the pointer stored in the arbitrary pointer slot, or ``NULL`` + if the slot is empty. This function cannot fail and does not set an + exception when the slot is empty. + + +Limited API +----------- + +All APIs in this PEP will be made part of the :ref:`Stable ABI ` in Python 3.16. + + +Rationale +========= + + +An API for coroutine functions, not futures +------------------------------------------- + +This proposal aims to add a C API for generic ``async``/``await`` constructs, *not* +an API for describing all capabilities of :mod:`asyncio`. While this PEP may +be reason for adding an ``asyncio`` C API in the future, its only goal is to +make :keyword:`async def` functions possible to implement in the C API. + + +Deviations from PyAwaitable +--------------------------- + + +``PyAsync`` instead of ``PyAwaitable`` +************************************** + +The most obvious deviation from the original library is the use of the +``PyAsync`` prefix over ``PyAwaitable``. This is because the original +"PyAwaitable" name is a historical artifact; it was chosen as +a parallel to the ``PyCoro`` prefix, with the idea that a ``PyCoro`` was +analogous to a native ``coroutine`` object, while a ``PyAwaitable`` would +be analogous to a native ``awaitable`` object. Given that this API (and new +versions of PyAwaitable) support all of the :class:`~collections.abc.Coroutine` +protocol (in contrast to the :class:`~collections.abc.Awaitable` protocol), +this name is now misleading, and thus the rename to ``PyAsync`` felt necessary. + + +Simpler storage for callback values +*********************************** + +This API uses a single ``PyObject *`` reference and ``void *`` pointer for value +storage between callbacks instead of variadic argument lists that were stored as +arrays. PyAwaitable used the latter because it was optimized for ergonomics when +writing pure C code, but since this API is also designed to be accessible from +bindings generators (and FFIs) as well, a single pointer for each felt like the +better call. PyAwaitable's array system also turned out to be a source of +complexity and bugs, and avoiding extra complexity for CPython where possible +is ideal. + +This API also returns a :term:`strong reference` from ``GetValue`` instead of a +:term:`borrowed reference`. PyAwaitable used the latter because it predated the +acceptance of :pep:`703`, and thus any concurrency implications were not +considered. + + +Different callback return values +******************************** + +In PyAwaitable, callbacks may return ``0``, ``-1``, and ``-2``, which correspond +to ``1``, ``0``, and ``-1`` in this proposal, respectively. This is also a historical +artifact; PyAwaitable did not originally have the ``-2`` return value, and changing +the meaning of ``0`` and ``-1`` could not be done for the sake of backwards +compatibility. + + +Backwards Compatibility +======================= + +This PEP has no known backwards compatibility concerns. + + +Security Implications +===================== + +This PEP has no known security concerns. + + +How to Teach This +================= + +The new APIs will be documented in the C API documentation. +A tutorial can also be added based on the tutorial present in PyAwaitable's +documentation. + + +Examples +-------- + + +Example: Forwarding an asynchronous result +****************************************** + +An extension method can call an asynchronous function and forward the +result of awaiting it. :c:type:`PyAsync` handles suspension and resumption +for the extension. + +.. code-block:: c + + static int + return_result(PyObject *asyncobj, PyObject *result) + { + PyAsync_SetResult(asyncobj, result); + return 1; + } + + // Equivalent to def call_async(func): return await func() + static PyObject * + call_async(PyObject *self, PyObject *func) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AddExpr(asyncobj, PyObject_CallNoArgs(func), + return_result, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + + +Example: Keeping state across an await +************************************** + +To implement ``return (await get_number()) * factor``, store *factor* with +:c:func:`PyAsync_SetValue` before queuing ``get_number()`` with this callback. +The :c:type:`PyAsync` object keeps the factor alive while execution is +suspended. The callback releases the strong reference returned by +:c:func:`PyAsync_GetValue` after using it. + +.. code-block:: c + + static int + multiply_result(PyObject *asyncobj, PyObject *value) + { + PyObject *factor = PyAsync_GetValue(asyncobj); + PyObject *result = PyNumber_Multiply(value, factor); + Py_DECREF(factor); + if (result == NULL) { + return 0; + } + PyAsync_SetResult(asyncobj, result); + Py_DECREF(result); + return 1; + } + + +Example: Handling a timeout +*************************** + +In ``call_async`` above, replace the final ``NULL`` argument to +:c:func:`PyAsync_AddExpr` with ``timeout_to_none``. Awaiting the result +then returns ``None`` on :exc:`TimeoutError`. Returning ``0`` from the +callback lets other exceptions propagate. + +.. code-block:: c + + static int + timeout_to_none(PyObject *asyncobj, PyObject *exception) + { + if (!PyErr_GivenExceptionMatches(exception, PyExc_TimeoutError)) { + return 0; + } + PyAsync_SetResult(asyncobj, Py_None); + return 1; + } + +For example, :func:`asyncio.timeout` can impose a deadline on an operation: + +.. code-block:: python + + import asyncio + + async def slow_operation(): + async with asyncio.timeout(0.01): + return await asyncio.sleep(1, result=42) + + assert asyncio.run(call_async(slow_operation)) is None + + +Example: Scheduling a dependent read +************************************ + +Suppose a protocol reader exposes asynchronous ``read_size()`` and +``readexactly(size)`` methods. Store the reader with +:c:func:`PyAsync_SetValue`, then queue ``reader.read_size()`` with this +callback. The first read's result determines the next asynchronous call, +which reuses ``return_result`` to return the completed packet. + +.. code-block:: c + + static int + read_body(PyObject *asyncobj, PyObject *size) + { + PyObject *reader = PyAsync_GetValue(asyncobj); + PyObject *coro = PyObject_CallMethod(reader, "readexactly", "O", size); + Py_DECREF(reader); + return PyAsync_AddExpr(asyncobj, coro, return_result, NULL) == 0; + } + + +Example: Acquiring an async lock for native work +************************************************ + +An asynchronous context manager can acquire a lock and return the protected +dictionary to the result callback: + +.. code-block:: python + + import asyncio + from contextlib import asynccontextmanager + + @asynccontextmanager + async def locked_cache(lock, cache): + async with lock: + yield cache + +The C callback updates that dictionary while the lock is held: + +.. code-block:: c + + static int + mark_ready(PyObject *asyncobj, PyObject *cache) + { + return PyDict_SetItemString(cache, "ready", Py_True) == 0; + } + + static PyObject * + update_locked(PyObject *self, PyObject *lock) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AsyncWith(asyncobj, lock, mark_ready, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + +Python code passes the lock and dictionary through the context manager: + +.. code-block:: python + + async def main(): + lock, cache = asyncio.Lock(), {} + await update_locked(locked_cache(lock, cache)) + assert cache["ready"] is True + assert not lock.locked() + + asyncio.run(main()) + +:c:func:`PyAsync_AsyncWith` invokes the result callback after acquiring the +lock, and releases the lock even if the dictionary update raises an exception. + + +Reference Implementation +======================== + +A reference implementation of this PEP can be found +`here `__. + + +Rejected Ideas +============== + +TBD. + + +Open Issues +=========== + +TBD. + + +Acknowledgements +================ + +Thanks to Petr Viktorin for his help with the original PyAwaitable design +discussion, and to all users of PyAwaitable over the past few years. + + +Change History +============== + +TBD. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.