diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 7707fccd25a..b5ff0ff538e 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -719,6 +719,7 @@ peps/pep-0841.rst @corona10 @sobolevn peps/pep-0842.rst @ZeroIntensity peps/pep-0843.rst @ZeroIntensity peps/pep-0844.rst @warsaw +peps/pep-0847.rst @dstufft # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0847.rst b/peps/pep-0847.rst new file mode 100644 index 00000000000..0e8931306c4 --- /dev/null +++ b/peps/pep-0847.rst @@ -0,0 +1,489 @@ +PEP: 847 +Title: Problem Details for the Simple Repository API +Author: Luis Gonzalez , + William Woodruff , + Zsolt Dollenstein +Sponsor: Donald Stufft +PEP-Delegate: Donald Stufft +Discussions-To: https://discuss.python.org/t/pre-pep-discussion-rfc-9457-error-responses-for-package-registries/105453 +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 06-Aug-2026 +Post-History: `29-Dec-2025 `__ + +Abstract +======== + +This PEP proposes standardizing the format of error responses returned +by the :ref:`simple repository API `. + +In particular, this PEP proposes using :rfc:`9457` ("Problem Details for HTTP APIs") +as a baseline, uniform representation for error responses. + +The mechanism and approach defined in this PEP is intended to be backwards-compatible +with existing assumptions around simple repository API error responses, while +giving installers the ability to render richer, more useful error messages to users. + +.. _rationale: + +Rationale and Motivation +======================== + +The :ref:`simple repository API ` defines two +representations (HTML and JSON) for *success* responses. Installers (like pip +and uv) may perform `content negotiation `__ +to select between the representations. + +Unlike success responses, the simple repository API does **not** define any standard representation +for *error* responses. As a result, installers have historically been unable to make any assumptions +about the body of the response when handling an error. + +To compensate for this, installers have conventionally rendered just the HTTP status code +(e.g. ``403``, ``501``) along with the "reason phrase" specified in HTTP/1.1 +(:rfc:`RFC 2616 6.1.1 <2616#section-6.1.1>`). HTTP/1.1 origins can customize this phrase +beyond its default value; however, many origins choose to leave it as the default, +resulting in vague error messages like ``401 Unauthorized`` with no additional context. +Furthermore, the HTTP reason phrase is specified as unstructured text and is subject +to interoperability constraints (such as being truncated or rewritten across proxies). + +To make matters more complicated, HTTP/2 removes the "reason phrase" entirely and retains just +the HTTP status code. As a result, an installer that encounters an error when +requesting a simple index response will see only ``401`` (for example), with no space +in the protocol itself for additional context. + +This problem of missing context affects both PyPI as well as third-party indices: + +- Third party indices are typically authenticated or otherwise access controlled, + and would like to return useful error messages when an installer request + can't be honored. +- Private indices may serve as restricted mirrors of upstream indices such + as PyPI. They may reject requests for packages or distributions that exist + upstream for various reasons: because an administrator has blocked them, a + security scan has flagged them as malicious, mirroring has failed, a release + has not yet met a minimum age requirement, etc. + + An HTTP status code alone + cannot explain all of these distinctions. Error details can tell users why + the request failed, what action they can take, and, where applicable, when + the package or distribution is expected to become available. +- PyPI currently serves all error responses with HTML bodies, even if the + installer's request negotiates JSON for the index response. This response is large + and ultimately discarded for the overwhelming majority of requests, since + installers have no ability to interpret it. +- The inability to convey structured error information constrains PyPI's + (and Python packaging's) ability to perform other modernization efforts. + For example, PyPI may wish to express metadata like + :ref:`project status markers ` + as error responses in the future, but cannot do so usefully without + a way to convey error context. + +Consequently, package registries need a mechanism for properly representing and +transmitting context in error responses. This mechanism should be: + +- Machine readable: installers (and HTTP clients more generally) should be able + to parse and interpret the error response with minimal ambiguity. +- Generalizable: Python package indices are distinct services, and may fail + for distinct reasons that aren't necessarily shared between them. Consequently, + the mechanism should not assume common error codes or failure modes across + services, and should allow services to express their error states + with full generality. +- Future proof: Python package indices currently have a narrow standardized + surface, limited largely to the simple repository API. However, + future extensions of that surface *should* be able to make use of the same + error reporting primitives, so that installers and other clients do not + need multiple unique error handling pathways when interacting with + standards-conforming services. +- (Ideally) Established as prior art: Python packaging should not reinvent the + wheel with respect to conveying error messages over HTTP; we should strive + to adopt a well-known and already widely adopted mechanism. + +This PEP proposes the adoption of :rfc:`9457` because it satisfies these considerations. + +Specification +============= + +This PEP only applies to *error responses*, meaning HTTP responses with status codes +in the range ``400-499`` or ``500-599``. + +Furthermore, this PEP only applies to error responses produced by HTTP origins when serving +the :ref:`simple repository API `. + +Package indices +--------------- + +When preparing to send an error response to a requester (e.g., an installer client), +the package index **SHOULD** format its response as an +:rfc:`RFC 9457 Problem Details object <9457#section-3>`. + +Implementers should consult :rfc:`9457` for a fully detailed description of the +Problem Details object format. The following is an abbreviated description: + +- Each Problem Details object is a JSON (:rfc:`8259`) object. +- Each Problem Details object **MAY** have the following members. All members are optional. + + - ``type`` is a JSON string containing a URI reference. It **MAY** be a "locator," + i.e. an HTTP or HTTPS URI, in which case it **SHOULD** reference human-readable + documentation for the error being presented. + - ``status`` is a JSON number containing the HTTP status code for the response. + If present, the value of ``status`` is purely advisory. + - ``title`` is a JSON string containing a short, human-readable summary of the problem *type*. + In other words, the ``title`` is covariant with ``type``, and should not vary based on + the individual details of a specific occurrence. + - ``detail`` is a JSON string containing a human-readable explanation of the problem + specific to this occurrence. + - ``instance`` is a JSON string containing a URI reference. Like ``type``, it **MAY** + be a "locator," in which case it **SHOULD** reference human-readable information + about the problem specific to this occurrence. + +- Additionally, each Problem Details object **MAY** have additional members, deemed + "extensions." All extensions are optional. + +Examples of Problem Details objects are provided in :ref:`Appendix 1 `. + +When formatting a response as a Problem Details object, the package index **MUST** +additionally send the ``Content-Type: application/problem+json`` response header. + +Clients +------- + +Upon receipt of an error response from an origin, the client **SHOULD**: + +- Confirm that the ``Content-Type`` is ``application/problem+json``. If the ``Content-Type`` + is not ``application/problem+json``, the client **MUST NOT** process the response + as if it contains a Problem Details object. +- Deserialize the response body as JSON, and validate it as a Problem Details object. +- Use the contents of the Problem Details object to present a contextually + appropriate error message to the user. + +If the process above fails at any step, the client **MAY** handle the original HTTP +error response as it sees fit. This can include handling the error using any pre-existing, +generic HTTP error handling logic. + +An example of how a client may choose to handle a Problem Details response +(along with appropriate error/fallback handling) is provided in :ref:`Appendix 2 `. + +Backwards Compatibility +======================= + +Because Python packaging as a whole never identified a specific error response +format for the simple repository API, installer clients as a whole are +resilient to arbitrary responses from Python package indices (as well +as changes to those responses over time). + +Consequently, this PEP deems the backwards compatibility risk associated +with standardizing an error response format to be **very low**. + +To increase our confidence in that determination, we conducted a review +of popular Python package installers to determine how they currently +handle index error responses: + +- pip handles index error responses in ``raise_for_status`` + (`permalink `__), + which consults only the HTTP status phrase and status code. + +- Poetry handles index error responses in ``HTTPRepository._get_response`` + (`permalink `__), which inspects the status code and then defers to ``raise_for_status`` from the + ``requests`` library. The latter holds onto the response body, but nothing parses it. + +- uv handles index error responses in ``CachedClient::fresh_request`` + (`permalink `__), + and supports :rfc:`9457` error responses as of October 2025 + (`uv 0.9.4 `__). + Prior to that, uv consults only the HTTP status phrase and error code (and still consults those, as the fallback). + +In effect, this means that older versions of all of pip, Poetry, and uv will gracefully degrade +(or, in the case of uv, enhance) in the presence of Problem Details responses, as none currently +attempt to parse or interpret error response bodies *except* where doing so is already consistent +with this PEP. + +Future Considerations +===================== + +As mentioned in the :ref:`rationale`, one consideration for selecting :rfc:`9457` is +its future-proofedness: it's foreseeable (and expected) that Python packaging +will future expand the standardize interfaces associated with a Python package index over time. +Consequently, we should select an error representation that's sufficiently general. + +As of this PEP's authorship, there are several other open Packaging-track PEPs +that propose the use of :rfc:`9457` for other, non-index error responses: + +- :pep:`694#errors` ("Upload 2.0 API for Python Package Indexes") +- :pep:`807#constraints` ("Index support for Trusted Publishing") + +Security Implications +===================== + +This PEP does not identify any positive or negative security implications +associated with standardizing the error response format for the simple +repository API. + +How to Teach This +================= + +This PEP affects users only indirectly: once adopted by both indices and clients, +the only visible impact to users is improved error messages. + +Consequently, the primary audience for teaching this PEP is not individual users, +but implementation parties (both indices and clients). This PEP proposes +the following if accepted: + +- The authors of this PEP will coordinate with the maintainers + of PyPI on appropriate public-facing documentation and communication, + including an announcement on the `PyPI blog `__ + if deemed appropriate. + +- The authors of this PEP will make appropriate changes to the + :ref:`living standard ` for the simple + repository API, including admonitions and callouts where appropriate + to indicate that both indices and clients can progressively enhance + their error behavior by adopting Problem Details. + +Rejected Ideas +============== + +Do nothing +---------- + +One option would be to retain the status quo, and continue to allow +indices to return whatever error responses they please. +Clients could then progressively enhance by handling :rfc:`9457` +responses *if* a given index happens to respond with a valid +Problem Details response. + +We consider this option unsuitable because it doesn't *clearly* help +both indices and clients make user-friendly error messaging decisions, +and will further expose gaps in error reporting as HTTP/2 (and beyond) +adoption continues to increase. + +Pick or invent a new error format +--------------------------------- + +Another option is to diverge from :rfc:`9457`, and pick (or invent) another +error format. An argument in favor of this is specificity: +a custom error format could, for example, provide dedicated error +codes that communicate failure modes that are common/shared +across many index implementations and hosts. + +We consider this option unsuitable for two reasons: + +#. In practice, indices may diverge widely in terms of error states that + require representation. For example, third party indices will almost certainly + need custom representations for various authorization and authentication error + states. +#. :rfc:`9457` is *already* extensible, and a future PEP *could* added shared error + codes as a well-known extension in the future. In other words, any foreseeable + benefit from a custom format is already subsumable within the Problem Details format. + +.. _problem-details-appendix-1: + +Appendix 1: Problem Details Object Examples +=========================================== + +The following examples demonstrate the ways in which a Problem Details +object can vary and how a client *might* choose to present those state +variations. + +The simplest Problem Details object is the empty JSON object: + +.. code-block:: json + + {} + +This is a valid Problem Details because :rfc:`9457` specifies that all members +of a Problem Details object are optional. + +In practice, this is not a very common response for servers to produce, since it +communicates nothing additional about the error (beyond what can be inferred +from the HTTP status code itself). However, it *is* valid, and a client *could* +choose to handle it explicitly, e.g for this HTTP 418: + +.. code-block:: console + + Error: Failed to fetch https://py.example.com/... + Cause: I'm a Teapot: Server refuses to brew coffee because it is a teapot + | + |-+ hint: The server returned a problem details object, but it was empty + |-+ hint: HTTP status: 418 + + +Another possible Problem Details object has nothing except a ``type`` and/or +``instance`` URI: + +.. code-block:: json + + { + "type": "https://py.example.com/docs/auth-issues", + "instance": "https://py.example.com/ORGNAME/..." + } + +One potential presentation of these (for an HTTP 401) would be: + +.. code-block:: console + + Error: Failed to fetch https://py.example.com/... + Cause: Unauthorized: No permission -- see authorization schemes + | + |-+ hint: Recommended documentation: https://py.example.com/docs/auth-issues + |-+ hint: Further resources: https://py.example.com/ORGNAME/... + |-+ hint: HTTP status: 401 + +The most common case, however, likely involves the ``title`` and ``detail`` fields: + +.. code-block:: json + + { + "title": "Authorization failed (invalid OAuth credential)", + "detail": "The OAuth credential is well-formed, but expired" + } + +Could produce: + +.. code-block:: console + + Error: Failed to fetch https://py.example.com/... + Cause: Authorization failed (invalid OAuth credential) + | + |-+ hint: The OAuth credential is well-formed, but expired + |-+ hint: HTTP status: 401 + + +Finally, we can imagine a "maximalist" Problem Details, containing every optional +field *and* some extensions: + +.. code-block:: json + + { + "type": "https://py.example.com/docs/auth-issues", + "status": 403, + "title": "The server is currently haunted.", + "detail": "Consider hiring a priest", + "instance": "https://py.example.com/ORGNAME/...", + "secret-extension": "The backdoor password is 'peekaboo'" + } + +This could be presented as: + +.. code-block:: console + + Error: Failed to fetch https://py.example.com/... + Cause: The server is currently haunted. + | + |-+ hint: Consider hiring a priest + |-+ hint: Recommended documentation: https://py.example.com/docs/auth-issues + |-+ hint: Further resources: https://py.example.com/ORGNAME/... + |-+ hint: The server responded with 401, but the underlying error reports 403 + |-+ hint: The error contains non-standard fields; + | re-run with '--verbose' to see them + +.. _problem-details-appendix-2: + +Appendix 2: Reference Implementation +==================================== + +The following example demonstrates how a client that interacts +with an instance of the simple repository API *might* choose to +handle error messages, including graceful fallbacks when +the Problem Details response is missing, malformed, or otherwise +insufficiently detailed. + +.. code-block:: python + + @dataclass + class ProblemDetails: + # deserialized from 'type' + type_: str | None + status: int | None + title: str | None + detail: str | None + instance: str | None + + # deserialized from the rest of the object body + extensions: dict[str, object] + + @dataclass + class Error: + """ + An idealized error message type. Each error has a primary message and zero or more + "context" breadcrumbs. A client could choose to render this as a message with hints, e.g.: + + Cause: The server is currently haunted. + | + |-+ hint: Consider hiring a priest + |-+ hint: HTTP status: 418 + """ + message: str + context: list[str] = field(default_factory=list) + + def add_context(self, breadcrumb: str) -> Self: + self.context.append(breadcrumb) + return self + + def http_status_phrase(resp: Response) -> str: + """ + Try to recover a useful HTTP status phrase, + starting from the response itself (if present), + then turning the code into a standard phrase (if standard), + and finally an "unknown" fallback for non-standard HTTP responses. + """ + if phrase := resp.status_phrase: + return phrase + + try: + status = HTTPStatus(resp.status_code) + # Example: "Not Found: Nothing matches the given URI" + return f"{status.phrase}: {status.description}" + except ValueError: + return "Unknown HTTP status code" + + + def generic_error(resp: Response) -> Error: + """ + Produce a generic error for an HTTP error response. + """ + + phrase = http_status_phrase(resp) + return Error(message=f"HTTP {resp.status_code}: {phrase}") + + def parse_error(resp: Response) -> Error: + """ + Turn an HTTP error response into a useful human-readable error. + + Precondition: resp.status_code is an error status. + """ + + error = generic_error(resp) + + # If the server does not indicate a Problem Details response, + # we assume that it isn't one. + if resp.content_type != "application/problem+json": + return error.add_context("The server didn't send any additional error details") + + # If the server indicated a Problem Details response but didn't + # send a valid one, treat it as a generic error. + if not (problem := ProblemDetails.from_json(resp.text)): + return error.add_context("The server sent us an error message, but it was malformed") + + + # Now that we have a Problem Details object, we can incrementally + # refine `error`. We might end up with no refinements, of course, + # since all fields are optional. + if title := problem.title: + error.message = title + if detail := problem.detail: + error.add_context(detail) + if (status := problem.status) and status != resp.status_code: + # This can be useful to report, as a discrepancy suggests + # that a proxy or other intermediate rewrote the status. + error.add_context(f"The server responded with {resp.status_code}, but the underlying error reports {status}") + + # Similar for type and instance. + + return error + + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive.