[woodruffw]

I'm opening this as a direct living spec PR, since it's arguably just a clarification of the existing requirement, not a new one 🙂. However, if there's disagreement about that, I'm happy to take this to DPO for discussion first.

My rationale: the sdist spec already says that the tarball must be a POSIX.1-2001 pax archive. This implies a bunch of things that people may not realize, but are already fully covered by that requirement. For example, GNU-style tar files are not pax-conforming (they use a different magic and have special extended value encoding rules), and the S typeflag is GNU-specific.

There's a passable argument to be made that GNU.sparse.* is pax-conforming, since they're pax-style records. However:

1. Some versions of GNU.sparse.* are illegal to represent in pax, since they use duplicated record names;
2. GNU.sparse.* in general changes the framing logic, making it hard to correctly model the parse of a sparse member with a standard pax-conforming parser.

Finally, I think the most controversial part of this is adding a NOT RECOMMENDED for other pax-style (but not standard) vendor records. These are technically allowed in pax, just not defined by the standard themselves. However, they're also subject to a bunch of interoperability problems. For example, SCHILY.xattr is sometimes encoded as raw binary, which is not pax-conforming and causes problems for standard parsers.

(And more generally, Python source distributions have no business encoding xattrs or anything like that, from what I can tell 🙂)

---

📚 Documentation preview 📚: https://python-packaging-user-guide--2108.org.readthedocs.build/en/2108/

[woodruffw]

CCing some other folks who I think might be interested: @konstin @miketheman

[sethmlarson]

LGTM, thanks @woodruffw!

[rgommers]

Thanks @woodruffw. I agree that overall this is a useful clarification, and most of this PR looks fine as such. However, this particular sentence is much more than that:

Installers and other consumers SHOULD reject source distributions that are not strictly pax-conforming

That would be a backwards compatibility break, and it does not look like a healthy idea. I looked through all build backends I could think of, and while most explicitly produce PAX_FORMAT archives, there are at least two that have been creating GNU rather than pax archives:

• setuptools relies on the tarfile module's defaults, so pre python 3.8 that was producing non-pax sdists (there'll be a ton of those on PyPI)
• uv_build changed from GNU to pax only very recently, in 0.13 (relevant PR is astral-sh/uv#22094)

I think you can either leave out that recommendation completely, or (my preference) use something like "MAY warn" and a note on keeping backwards compatibility with sdists produced by older tools in mind before rejecting package versions.

Re vendor extensions: yes that's technically a change, but it's clearly undesirable, and I think this should hit DPO anyway to raise awareness, so I'd just use MUST NOT here and get it right now rather than leave it for later relitigation.

[rgommers]

In addition to previously non-conformant build backends, there'll also be a decent amount of manual munging around to edit sdist contents. E.g., I wrote this trim_sdist_content.py script for scipy not too long ago. That one happens to be fine, since Meson has a way (add_dist_script) to invoke it as part of the sdist building, however if you're using setuptools or another less feature-full build backend, you typically end up doing it as post-processing, and if you repack with say the GNU tar utility, you'll end up with a non-compliant archive.

[rgommers]

It'd be fine for uploaders and index servers to start rejecting non-compliant archives though, even if installers and other consuming tools need to be more permissive.

[woodruffw]

Thanks, I can loosen that; MAY seems reasonable.

(For context PyPI will likely begin rejecting these in the medium-term. It's already rejecting some other non-pax weirdness, e.g. GNU sparse encodings.)

[rgommers]

I'm opening this as a direct living spec PR, since it's arguably just a clarification of the existing requirement, not a new one 🙂. However, if there's disagreement about that, I'm happy to take this to DPO for discussion first.

(final comment for the day) Now that we have a documented smaller-changes-to-specs-without-a-PEP route (https://github.com/python/packaging-council#process-for-packaging-standards), this PR looks like an excellent candidate to exercise that. I'd be happy to help land this without much friction, as are other PPC members I'm sure.

[woodruffw]

Also pushed up a change around pax records, strengthening the NOT RECOMMENDED to MUST NOT.

[rgommers]

Thanks @woodruffw, this LGTM now!

Mind posting on DPO with a summary, and explaining that PyPI may start enforcing this so anyone doing manual sdist processing may want to check it's not accidentally losing pax compliance?

The Packaging Council didn't put an exact time window in its guidance, and I suspect we don't want to - it depends a bit on the size of the change. This one is very concise, so I suggest merging in 7 days unless any concerns get articulated.