feat(pypirsf): read dependency data from an RSF file

[jonyoder]

What

pypirsf.File — open a local RSF, index every package by canonical name in one pass, then look up any package's dependency metadata by offset.

This is what makes the standalone resolver possible: point it at an RSF you already have, no network.

Validated against the real production file

Not just synthetic fixtures. Against the live 932,861-record, 936 MB decompressed PyPI RSF:

Full scan + index	256 ms
Dictionary	4,096 names
Per-package lookup	31–703 µs (flask 66 µs; boto3 at 2,086 versions, 703 µs)

And the decoded content is unmistakably real:

• flask 3.1.3 → >=3.9, with asgiref>=3.2 ; extra == "async", blinker>=1.9.0, click>=8.1.3
• django 6.1rc1 → argon2-cffi>=23.1.0; extra == "argon2", asgiref>=3.9.1
• boto3 → botocore>=1.12.99,<1.13.0

That last point is the reason the assertions check recognizable content rather than just a nil error: a desynchronized reader of a binary format produces garbled strings, not correct requirement specifiers.

Relevant to #18651's <100ms cold / <1ms warm gate: the 256 ms is a one-time load, and per-package access is microseconds.

The design decision that matters

This reader never traverses the snapshots array. Every hazard in the format lives in there:

• three historical subfield layouts, which must be detected from the file's own schema rather than assumed
• a per-element index block whose width cannot be taken from the schema, because production files use the v1 index format and report the array as un-indexed even though the block is physically present

AdvanceTo skips the array wholesale by its own size header — which the format explicitly supports — so none of that applies.

Version sets therefore come from the deps blob's own keys, not the snapshot timeline. That isn't a shortcut: the set of versions with captured dependencies is exactly the set a resolver can safely use, since a version whose dependencies are unknown can't be resolved through anyway. The documented cost is that File reports captured versions rather than every version that ever existed.

Concurrency by construction, not by lock

Each lookup gets its own io.SectionReader and its own rsf.Reader, sharing only the immutable schema. The alternative — seeking one shared reader — requires resetting the buffered reader in lockstep with every seek, and getting that wrong reads stale bytes from the previous position instead of failing. Covered by a -race test.

Two facts learned the hard way, now recorded in the code

• The rsf struct tags drive the writer, not the reader. There is no tag-driven Unmarshal; reading is manual field navigation against the per-file schema. So record.go documents the layout and lets tests generate real files — it is not itself a reader. I added a comment saying so, since the previous PR could be read as implying otherwise.
• WriteObject reflects over a value, not a pointer. Passing &record reports unknown field type 0x16 (reflect.Pointer) rather than dereferencing.

Testing

Synthetic fixtures are written with the real writer over these same structs, so record framing, the snapshots array header, and the trailing additive fields come from the code that produces production files rather than from this test's idea of the format.

Covered: a record after the first (the dictionary lives on record 0 and is read differently, so a reader can be correct there and wrong everywhere else), an empty deps field, ErrPackageNotFound, a pre-cutover file whose schema has no deps fields reported at Open, and concurrent lookups under -race.

TestRealRSF is opt-in via PYPIRSF_TEST_FILE since the file is ~1 GB and can't be committed. Its doc comment records how to fetch one, and notes those files are licensed for Package Manager customers.

82.6% statement coverage; -race -count=1 clean; gofmt -l . silent; golangci-lint run ./... at CI's pinned v2.11.2 → 0 issues from the module root.

Next

The MetadataIndex implementation over this, then the CLI.

🤖 Generated with Claude Code