[pull] main from python:main

Doc/c-api/arg.rst

@@ -524,7 +524,7 @@ API Functions
    Returns true on success; on failure, it returns false and raises the
    appropriate exception.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 
 .. c:function:: int PyArg_ParseArrayAndKeywords(PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames, const char *format, const char * const *kwlist, ...)
@@ -535,7 +535,7 @@ API Functions
    Returns true on success; on failure, it returns false and raises the
    appropriate exception.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 
 .. c:function:: int PyArg_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssize_t max, ...)

Doc/c-api/dict.rst

@@ -495,7 +495,7 @@ Dictionary view objects
 Frozen dictionary objects
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. versionadded:: next
+.. versionadded:: 3.15
 
 
 .. c:var:: PyTypeObject PyFrozenDict_Type

Doc/c-api/exceptions.rst

@@ -716,7 +716,7 @@ Signal Handling
       This function may now execute a remote debugger script, if remote
       debugging is enabled.
 
-   .. versionchanged:: next
+   .. versionchanged:: 3.15
       The exception set by :c:func:`PyThreadState_SetAsyncExc` is now raised.
 
 

Doc/c-api/import.rst

@@ -350,14 +350,14 @@ Importing Modules
 
    Gets the current lazy imports mode.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 .. c:function:: PyObject* PyImport_GetLazyImportsFilter()
 
    Return a :term:`strong reference` to the current lazy imports filter,
    or ``NULL`` if none exists. This function always succeeds.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 .. c:function:: int PyImport_SetLazyImportsMode(PyImport_LazyImportsMode mode)
 
@@ -366,7 +366,7 @@ Importing Modules
 
    This function always returns ``0``.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 .. c:function:: int PyImport_SetLazyImportsFilter(PyObject *filter)
 
@@ -377,7 +377,7 @@ Importing Modules
 
    Return ``0`` on success and ``-1`` with an exception set otherwise.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 .. c:type:: PyImport_LazyImportsMode
 
@@ -396,7 +396,7 @@ Importing Modules
       Disable lazy imports entirely. Even explicit ``lazy`` statements become
       eager imports.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15
 
 .. c:function:: PyObject* PyImport_CreateModuleFromInitfunc(PyObject *spec, PyObject* (*initfunc)(void))
 

Doc/c-api/object.rst

@@ -817,4 +817,4 @@ Object Protocol
    Returns 1 if the object was made immortal and returns 0 if it was not.
    This function cannot fail.
 
-   .. versionadded:: next
+   .. versionadded:: 3.15

Doc/c-api/structures.rst

@@ -48,6 +48,19 @@ under :ref:`reference counting <countingrefs>`.
       Do not use this field directly; use :c:macro:`Py_TYPE` and
       :c:func:`Py_SET_TYPE` instead.
 
+   .. c:member:: PyMutex ob_mutex
+
+      A :ref:`per-object lock <per-object-locks>`, present only in the :term:`free-threaded <free threading>`
+      build (when :c:macro:`Py_GIL_DISABLED` is defined).
+
+      This field is **reserved for use by the critical section API**
+      (:c:macro:`Py_BEGIN_CRITICAL_SECTION` / :c:macro:`Py_END_CRITICAL_SECTION`).
+      Do **not** lock it directly with ``PyMutex_Lock``; doing so can cause
+      deadlocks.  If you need your own lock, add a separate :c:type:`PyMutex`
+      field to your object struct.
+
+      .. versionadded:: 3.13
+
 
 .. c:type:: PyVarObject
 

Doc/c-api/threads.rst

@@ -10,43 +10,63 @@ Thread states and the global interpreter lock
    single: interpreter lock
    single: lock, interpreter
 
-Unless on a :term:`free-threaded <free threading>` build of :term:`CPython`,
-the Python interpreter is not fully thread-safe.  In order to support
+Unless on a :term:`free-threaded build` of :term:`CPython`,
+the Python interpreter is generally not thread-safe.  In order to support
 multi-threaded Python programs, there's a global lock, called the :term:`global
-interpreter lock` or :term:`GIL`, that must be held by the current thread before
-it can safely access Python objects. Without the lock, even the simplest
-operations could cause problems in a multi-threaded program: for example, when
+interpreter lock` or :term:`GIL`, that must be held by a thread before
+accessing Python objects. Without the lock, even the simplest operations
+could cause problems in a multi-threaded program: for example, when
 two threads simultaneously increment the reference count of the same object, the
 reference count could end up being incremented only once instead of twice.
 
+As such, only a thread that holds the GIL may operate on Python objects or
+invoke Python's C API.
+
 .. index:: single: setswitchinterval (in module sys)
 
-Therefore, the rule exists that only the thread that has acquired the
-:term:`GIL` may operate on Python objects or call Python/C API functions.
-In order to emulate concurrency of execution, the interpreter regularly
-tries to switch threads (see :func:`sys.setswitchinterval`).  The lock is also
-released around potentially blocking I/O operations like reading or writing
-a file, so that other Python threads can run in the meantime.
+In order to emulate concurrency, the interpreter regularly tries to switch
+threads between bytecode instructions (see :func:`sys.setswitchinterval`).
+This is why locks are also necessary for thread-safety in pure-Python code.
+
+Additionally, the global interpreter lock is released around blocking I/O
+operations, such as reading or writing to a file. From the C API, this is done
+by :ref:`detaching the thread state <detaching-thread-state>`.
+
 
 .. index::
    single: PyThreadState (C type)
 
-The Python interpreter keeps some thread-specific bookkeeping information
-inside a data structure called :c:type:`PyThreadState`, known as a :term:`thread state`.
-Each OS thread has a thread-local pointer to a :c:type:`PyThreadState`; a thread state
+The Python interpreter keeps some thread-local information inside
+a data structure called :c:type:`PyThreadState`, known as a :term:`thread state`.
+Each thread has a thread-local pointer to a :c:type:`PyThreadState`; a thread state
 referenced by this pointer is considered to be :term:`attached <attached thread state>`.
 
 A thread can only have one :term:`attached thread state` at a time. An attached
-thread state is typically analogous with holding the :term:`GIL`, except on
-:term:`free-threaded <free threading>` builds.  On builds with the :term:`GIL` enabled,
-:term:`attaching <attached thread state>` a thread state will block until the :term:`GIL`
-can be acquired. However, even on builds with the :term:`GIL` disabled, it is still required
-to have an attached thread state to call most of the C API.
+thread state is typically analogous with holding the GIL, except on
+free-threaded builds.  On builds with the GIL enabled, attaching a thread state
+will block until the GIL can be acquired. However, even on builds with the GIL
+disabled, it is still required to have an attached thread state, as the interpreter
+needs to keep track of which threads may access Python objects.
+
+.. note::
+
+   Even on the free-threaded build, attaching a thread state may block, as the
+   GIL can be re-enabled or threads might be temporarily suspended (such as during
+   a garbage collection).
+
+Generally, there will always be an attached thread state when using Python's
+C API, including during embedding and when implementing methods, so it's uncommon
+to need to set up a thread state on your own. Only in some specific cases, such
+as in a :c:macro:`Py_BEGIN_ALLOW_THREADS` block or in a fresh thread, will the
+thread not have an attached thread state.
+If uncertain, check if :c:func:`PyThreadState_GetUnchecked` returns ``NULL``.
 
-In general, there will always be an :term:`attached thread state` when using Python's C API.
-Only in some specific cases (such as in a :c:macro:`Py_BEGIN_ALLOW_THREADS` block) will the
-thread not have an attached thread state. If uncertain, check if :c:func:`PyThreadState_GetUnchecked` returns
-``NULL``.
+If it turns out that you do need to create a thread state, call :c:func:`PyThreadState_New`
+followed by :c:func:`PyThreadState_Swap`, or use the dangerous
+:c:func:`PyGILState_Ensure` function.
+
+
+.. _detaching-thread-state:
 
 Detaching the thread state from extension code
 ----------------------------------------------
@@ -86,28 +106,37 @@ The block above expands to the following code::
 
 Here is how these functions work:
 
-The :term:`attached thread state` holds the :term:`GIL` for the entire interpreter. When detaching
-the :term:`attached thread state`, the :term:`GIL` is released, allowing other threads to attach
-a thread state to their own thread, thus getting the :term:`GIL` and can start executing.
-The pointer to the prior :term:`attached thread state` is stored as a local variable.
-Upon reaching :c:macro:`Py_END_ALLOW_THREADS`, the thread state that was
-previously :term:`attached <attached thread state>` is passed to :c:func:`PyEval_RestoreThread`.
-This function will block until another releases its :term:`thread state <attached thread state>`,
-thus allowing the old :term:`thread state <attached thread state>` to get re-attached and the
-C API can be called again.
-
-For :term:`free-threaded <free threading>` builds, the :term:`GIL` is normally
-out of the question, but detaching the :term:`thread state <attached thread state>` is still required
-for blocking I/O and long operations. The difference is that threads don't have to wait for the :term:`GIL`
-to be released to attach their thread state, allowing true multi-core parallelism.
+The attached thread state implies that the GIL is held for the interpreter.
+To detach it, :c:func:`PyEval_SaveThread` is called and the result is stored
+in a local variable.
+
+By detaching the thread state, the GIL is released, which allows other threads
+to attach to the interpreter and execute while the current thread performs
+blocking I/O. When the I/O operation is complete, the old thread state is
+reattached by calling :c:func:`PyEval_RestoreThread`, which will wait until
+the GIL can be acquired.
 
 .. note::
-   Calling system I/O functions is the most common use case for detaching
-   the :term:`thread state <attached thread state>`, but it can also be useful before calling
-   long-running computations which don't need access to Python objects, such
-   as compression or cryptographic functions operating over memory buffers.
+   Performing blocking I/O is the most common use case for detaching
+   the thread state, but it is also useful to call it over long-running
+   native code that doesn't need access to Python objects or Python's C API.
    For example, the standard :mod:`zlib` and :mod:`hashlib` modules detach the
-   :term:`thread state <attached thread state>` when compressing or hashing data.
+   :term:`thread state <attached thread state>` when compressing or hashing
+   data.
+
+On a :term:`free-threaded build`, the :term:`GIL` is usually out of the question,
+but **detaching the thread state is still required**, because the interpreter
+periodically needs to block all threads to get a consistent view of Python objects
+without the risk of race conditions.
+For example, CPython currently suspends all threads for a short period of time
+while running the garbage collector.
+
+.. warning::
+
+   Detaching the thread state can lead to unexpected behavior during interpreter
+   finalization. See :ref:`cautions-regarding-runtime-finalization` for more
+   details.
+
 
 APIs
 ^^^^
@@ -149,73 +178,84 @@ example usage in the Python source distribution.
    declaration.
 
 
-.. _gilstate:
-
 Non-Python created threads
 --------------------------
 
 When threads are created using the dedicated Python APIs (such as the
-:mod:`threading` module), a thread state is automatically associated to them
-and the code shown above is therefore correct.  However, when threads are
-created from C (for example by a third-party library with its own thread
-management), they don't hold the :term:`GIL`, because they don't have an
-:term:`attached thread state`.
+:mod:`threading` module), a thread state is automatically associated with them,
+However, when a thread is created from native code (for example, by a
+third-party library with its own thread management), it doesn't hold an
+attached thread state.
 
 If you need to call Python code from these threads (often this will be part
 of a callback API provided by the aforementioned third-party library),
 you must first register these threads with the interpreter by
-creating an :term:`attached thread state` before you can start using the Python/C
-API.  When you are done, you should detach the :term:`thread state <attached thread state>`, and
-finally free it.
+creating a new thread state and attaching it.
 
-The :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` functions do
-all of the above automatically.  The typical idiom for calling into Python
-from a C thread is::
+The most robust way to do this is through :c:func:`PyThreadState_New` followed
+by :c:func:`PyThreadState_Swap`.
 
-   PyGILState_STATE gstate;
-   gstate = PyGILState_Ensure();
+.. note::
+   ``PyThreadState_New`` requires an argument pointing to the desired
+   interpreter; such a pointer can be acquired via a call to
+   :c:func:`PyInterpreterState_Get` from the code where the thread was
+   created.
+
+For example::
+
+   /* The return value of PyInterpreterState_Get() from the
+      function that created this thread. */
+   PyInterpreterState *interp = thread_data->interp;
+
+   /* Create a new thread state for the interpreter. It does not start out
+      attached. */
+   PyThreadState *tstate = PyThreadState_New(interp);
+
+   /* Attach the thread state, which will acquire the GIL. */
+   PyThreadState_Swap(tstate);
 
    /* Perform Python actions here. */
    result = CallSomeFunction();
    /* evaluate result or handle exception */
 
-   /* Release the thread. No Python API allowed beyond this point. */
-   PyGILState_Release(gstate);
+   /* Destroy the thread state. No Python API allowed beyond this point. */
+   PyThreadState_Clear(tstate);
+   PyThreadState_DeleteCurrent();
 
-Note that the ``PyGILState_*`` functions assume there is only one global
-interpreter (created automatically by :c:func:`Py_Initialize`).  Python
-supports the creation of additional interpreters (using
-:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the
-``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure`
-and similar functions default to :term:`attaching <attached thread state>` a
-:term:`thread state` for the main interpreter, meaning that the thread can't safely
-interact with the calling subinterpreter.
+.. warning::
 
-Supporting subinterpreters in non-Python threads
-------------------------------------------------
+   If the interpreter finalized before ``PyThreadState_Swap`` was called, then
+   ``interp`` will be a dangling pointer!
 
-If you would like to support subinterpreters with non-Python created threads, you
-must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*``
-API.
+.. _gilstate:
 
-In particular, you must store the interpreter state from the calling
-function and pass it to :c:func:`PyThreadState_New`, which will ensure that
-the :term:`thread state` is targeting the correct interpreter::
+Legacy API
+----------
 
-   /* The return value of PyInterpreterState_Get() from the
-      function that created this thread. */
-   PyInterpreterState *interp = ThreadData->interp;
-   PyThreadState *tstate = PyThreadState_New(interp);
-   PyThreadState_Swap(tstate);
+Another common pattern to call Python code from a non-Python thread is to use
+:c:func:`PyGILState_Ensure` followed by a call to :c:func:`PyGILState_Release`.
 
-   /* GIL of the subinterpreter is now held.
-      Perform Python actions here. */
+These functions do not work well when multiple interpreters exist in the Python
+process. If no Python interpreter has ever been used in the current thread (which
+is common for threads created outside Python), ``PyGILState_Ensure`` will create
+and attach a thread state for the "main" interpreter (the first interpreter in
+the Python process).
+
+Additionally, these functions have thread-safety issues during interpreter
+finalization. Using ``PyGILState_Ensure`` during finalization will likely
+crash the process.
+
+Usage of these functions look like such::
+
+   PyGILState_STATE gstate;
+   gstate = PyGILState_Ensure();
+
+   /* Perform Python actions here. */
    result = CallSomeFunction();
    /* evaluate result or handle exception */
 
-   /* Destroy the thread state. No Python API allowed beyond this point. */
-   PyThreadState_Clear(tstate);
-   PyThreadState_DeleteCurrent();
+   /* Release the thread. No Python API allowed beyond this point. */
+   PyGILState_Release(gstate);
 
 
 .. _fork-and-threads: