Merge branch 'main' into ssl-keylogfile-always-supported

Author: robsdedude

Doc/c-api/complex.rst

@@ -130,7 +130,7 @@ rather than dereferencing them through pointers.
 
 Please note, that these functions are :term:`soft deprecated` since Python
 3.15.  Avoid using this API in a new code to do complex arithmetic: either use
-the `Number Protocol <number>`_ API or use native complex types, like
+the :ref:`Number Protocol <number>` API or use native complex types, like
 :c:expr:`double complex`.
 
 

Doc/extending/first-extension-module.rst

@@ -164,7 +164,7 @@ Then, create ``meson.build`` containing the following:
 
 .. note::
 
-   See `meson-python documentation <meson-python>`_ for details on
+   See the `meson-python documentation <meson-python_>`_ for details on
    configuration.
 
 Now, build install the *project in the current directory* (``.``) via ``pip``:

Doc/library/io.rst

@@ -38,6 +38,7 @@ will raise a :exc:`TypeError`.  So will giving a :class:`bytes` object to the
    Operations that used to raise :exc:`IOError` now raise :exc:`OSError`, since
    :exc:`IOError` is now an alias of :exc:`OSError`.
 
+.. _text-io:
 
 Text I/O
 ^^^^^^^^
@@ -65,6 +66,7 @@ In-memory text streams are also available as :class:`StringIO` objects::
 The text stream API is described in detail in the documentation of
 :class:`TextIOBase`.
 
+.. _binary-io:
 
 Binary I/O
 ^^^^^^^^^^
@@ -103,6 +105,13 @@ stream by opening a file in binary mode with buffering disabled::
 
 The raw stream API is described in detail in the docs of :class:`RawIOBase`.
 
+.. warning::
+   Raw I/O is a low-level interface and methods generally must have their return
+   values checked and be explicitly retried to ensure an operation completes.
+   For instance :meth:`~RawIOBase.write` returns the number of bytes written
+   which may be less than the number of bytes provided (a partial write).
+   High-level I/O objects like :ref:`binary-io` and :ref:`text-io` implement
+   retry behavior.
 
 .. _io-text-encoding:
 
@@ -478,8 +487,11 @@ I/O Base Classes
 
       Read up to *size* bytes from the object and return them.  As a convenience,
       if *size* is unspecified or -1, all bytes until EOF are returned.
-      Otherwise, only one system call is ever made.  Fewer than *size* bytes may
-      be returned if the operating system call returns fewer than *size* bytes.
+
+      Attempts to make only one system call but will retry if interrupted and
+      the signal handler does not raise an exception (see :pep:`475` for the
+      rationale). This means fewer than *size* bytes may be returned if the
+      operating system call returns fewer than *size* bytes.
 
       If 0 bytes are returned, and *size* was not 0, this indicates end of file.
       If the object is in non-blocking mode and no bytes are available,
@@ -493,13 +505,19 @@ I/O Base Classes
       Read and return all the bytes from the stream until EOF, using multiple
       calls to the stream if necessary.
 
+      If ``0`` bytes are returned this indicates end of file. If the object is in
+      non-blocking mode and the underlying :meth:`read` returns ``None``
+      indicating no bytes are available, ``None`` is returned.
+
    .. method:: readinto(b, /)
 
       Read bytes into a pre-allocated, writable
       :term:`bytes-like object` *b*, and return the
       number of bytes read.  For example, *b* might be a :class:`bytearray`.
-      If the object is in non-blocking mode and no bytes
-      are available, ``None`` is returned.
+
+      If ``0`` is returned and ``len(b)`` is not ``0``, this indicates end of file. If
+      the object is in non-blocking mode and no bytes are available, ``None`` is
+      returned.
 
    .. method:: write(b, /)
 
@@ -513,6 +531,13 @@ I/O Base Classes
       this method returns, so the implementation should only access *b*
       during the method call.
 
+      .. warning::
+
+         This function does not ensure all bytes are written or an exception is
+         thrown. Callers may implement that behavior by checking the return
+         value and, if it is less than the length of *b*, looping with additional
+         write calls until all unwritten bytes are written. High-level I/O
+         objects like :ref:`binary-io` and :ref:`text-io` implement retry behavior.
 
 .. class:: BufferedIOBase
 
@@ -641,7 +666,11 @@ Raw File I/O
 .. class:: FileIO(name, mode='r', closefd=True, opener=None)
 
    A raw binary stream representing an OS-level file containing bytes data.  It
-   inherits from :class:`RawIOBase`.
+   inherits from :class:`RawIOBase` and implements its low-level access design.
+   This means :meth:`~RawIOBase.write` does not guarantee all bytes are written
+   and :meth:`~RawIOBase.read` may read less bytes than requested even when more
+   bytes may be present in the underlying file. To get "write all" and
+   "read at least" behavior, use :ref:`binary-io`.
 
    The *name* can be one of two things:
 
@@ -661,10 +690,6 @@ Raw File I/O
    implies writing, so this mode behaves in a similar way to ``'w'``. Add a
    ``'+'`` to the mode to allow simultaneous reading and writing.
 
-   The :meth:`~RawIOBase.read` (when called with a positive argument),
-   :meth:`~RawIOBase.readinto` and :meth:`~RawIOBase.write` methods on this
-   class will only make one system call.
-
    A custom opener can be used by passing a callable as *opener*. The underlying
    file descriptor for the file object is then obtained by calling *opener* with
    (*name*, *flags*). *opener* must return an open file descriptor (passing
@@ -676,6 +701,13 @@ Raw File I/O
    See the :func:`open` built-in function for examples on using the *opener*
    parameter.
 
+   .. warning::
+      :class:`FileIO` is a low-level I/O object and members, such as
+      :meth:`~RawIOBase.read` and :meth:`~RawIOBase.write`, need to have their
+      return values checked explicitly in a retry loop to implement "write all"
+      and "read at least" behavior. High-level I/O objects :ref:`binary-io` and
+      :ref:`text-io` implement retry behavior.
+
    .. versionchanged:: 3.3
       The *opener* parameter was added.
       The ``'x'`` mode was added.

Doc/library/shlex.rst

@@ -44,12 +44,15 @@ The :mod:`!shlex` module defines the following functions:
    .. versionadded:: 3.8
 
 
-.. function:: quote(s)
+.. function:: quote(s, *, force=False)
 
    Return a shell-escaped version of the string *s*.  The returned value is a
    string that can safely be used as one token in a shell command line, for
    cases where you cannot use a list.
 
+   If *force* is :const:`True`, then *s* is unconditionally quoted,
+   even if it is already safe for a shell without being quoted.
+
    .. _shlex-quote-warning:
 
    .. warning::
@@ -91,8 +94,23 @@ The :mod:`!shlex` module defines the following functions:
       >>> command
       ['ls', '-l', 'somefile; rm -rf ~']
 
+   The *force* keyword can be used to produce consistent behavior when
+   escaping multiple strings:
+
+      >>> from shlex import quote
+      >>> filenames = ['my first file', 'file2', 'file 3']
+      >>> filenames_some_escaped = [quote(f) for f in filenames]
+      >>> filenames_some_escaped
+      ["'my first file'", 'file2', "'file 3'"]
+      >>> filenames_all_escaped = [quote(f, force=True) for f in filenames]
+      >>> filenames_all_escaped
+      ["'my first file'", "'file2'", "'file 3'"]
+
    .. versionadded:: 3.3
 
+   .. versionchanged:: next
+      The *force* keyword was added.
+
 The :mod:`!shlex` module defines the following class:
 
 

Doc/library/stdtypes.rst

@@ -2616,7 +2616,9 @@ expression support in the :mod:`re` module).
    :func:`re.split`). Splitting an empty string with a specified separator
    returns ``['']``.
 
-   For example::
+   For example:
+
+   .. doctest::
 
       >>> '1,2,3'.split(',')
       ['1', '2', '3']
@@ -2634,7 +2636,9 @@ expression support in the :mod:`re` module).
    string or a string consisting of just whitespace with a ``None`` separator
    returns ``[]``.
 
-   For example::
+   For example:
+
+   .. doctest::
 
       >>> '1 2 3'.split()
       ['1', '2', '3']
@@ -2646,7 +2650,9 @@ expression support in the :mod:`re` module).
    If *sep* is not specified or is ``None`` and  *maxsplit* is ``0``, only
    leading runs of consecutive whitespace are considered.
 
-   For example::
+   For example:
+
+   .. doctest::
 
       >>> "".split(None, 0)
       []
@@ -2655,7 +2661,7 @@ expression support in the :mod:`re` module).
       >>> "   foo   ".split(maxsplit=0)
       ['foo   ']
 
-   See also :meth:`join`.
+   See also :meth:`join` and :meth:`rsplit`.
 
 
 .. index::

Doc/using/windows.rst

@@ -455,6 +455,12 @@ customization.
        Python runtimes, or false to prevent it.
        By default, true.
 
+   * - ``shebang_templates``
+     - (none)
+     - Mapping from shebang line template to alternative command, such as
+       ``py -V:<tag>`` or a substitute string.
+       See :ref:`pymanager-shebang` for more details.
+
    * - ``log_level``
      - ``PYMANAGER_VERBOSE``, ``PYMANAGER_DEBUG``
      - Set the default level of output (0-50).
@@ -568,6 +574,30 @@ which the path to the script and any additional arguments will be appended.
 This functionality may be disabled by the ``shebang_can_run_anything``
 configuration option.
 
+Since version 26.3 of the Python install manager, custom shebang templates may
+be added to your configuration file. Add the ``shebang_templates`` object with
+one member for each template (the string to match) and the command to use when
+the template is matched. Most commands should be ``py -V:<tag>`` (or ``pyw``) to
+launch one of your installed runtimes. The ``py -3.<version>`` form is also
+allowed, as is a plain ``py`` to launch the default. No other arguments are
+supported.
+
+.. code:: json5
+
+   {
+       "shebang_templates": {
+           "/usr/bin/python": "py",
+           "/usr/bin/my_custom_python": "py -V:MyCustomPython/3"
+       }
+   }
+
+If the substitute command is not ``py`` or ``pyw``, it will be written back into
+the shebang and regular handling continues. If launching arbitrary executables
+is permitted, then providing a full path will allow you to redirect from Python
+to any executable. The template should match either the entire line (ignoring
+leading and trailing whitespace), or up to the first space in the shebang line.
+
+
 .. note::
 
    The behaviour of shebangs in the Python install manager is subtly different

Doc/whatsnew/3.16.rst

@@ -109,6 +109,13 @@ os
   process via a pidfd.  Available on Linux 5.6+.
   (Contributed by Maurycy Pawłowski-Wieroński in :gh:`149464`.)
 
+shlex
+-----
+
+* Add keyword-only parameter *force* to :func:`shlex.quote` to force quoting
+  a string, even if it is already safe for a shell without being quoted.
+  (Contributed by Jay Berry in :gh:`148846`.)
+
 xml
 ---
 

Include/dynamic_annotations.h

@@ -461,6 +461,7 @@ int RunningOnValgrind(void);
 
 #if DYNAMIC_ANNOTATIONS_ENABLED != 0 && defined(__cplusplus)
 
+extern "C++" {
   /* _Py_ANNOTATE_UNPROTECTED_READ is the preferred way to annotate racey reads.
 
      Instead of doing
@@ -476,6 +477,8 @@ int RunningOnValgrind(void);
     _Py_ANNOTATE_IGNORE_READS_END();
     return res;
   }
+}
+
   /* Apply _Py_ANNOTATE_BENIGN_RACE_SIZED to a static variable. */
 #define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)        \
     namespace {                                                       \