the manifest can name bytes that were not the bytes bound

[FBumann]

Prompt: This package assumes the data (files) didnt change, right?

Note

The following content was generated by AI.

Yes, and within a single bind() as well as across runs. _resolve hashes each
source with its own full read of the path; the data is read afterwards by
separate opens — _coordinates and the parameter loop each call
scan.select(...).collect(), and a scan_parquet/scan_csv LazyFrame is a
plan, not a buffer, so every collect re-opens the file.

So a source feeding three parameters is opened once for the hash and three more
times for the data, with nothing revalidating in between. The manifest's implied
claim is these bytes produced this answer; what it can actually support is
this path hashed to X at one moment, and some later read of it produced the
answer.

On static local files this is theoretical. On a network mount, a shared scratch
directory, or a path a nightly job rewrites — which is the audience — it is not.

The fix, and what it collides with

Read each source's bytes once, hash those bytes, and hand them to polars, so
the digest is of the data bound by construction rather than by assumption. That
means holding the file in memory, which runs straight into the deferred
streaming hand-off (SPEC §9): the version that keeps
lpspec's memory claim intact wants to hand over a path and never materialise.
The two decisions should be made together rather than separately, which is why
this is filed rather than fixed.

Cheaper interim if that stays parked: re-digest after reading and refuse on a
change. It cannot prevent a torn read, but it turns a silent wrong answer into
an error.

Cost is not the obstacle

Measured on this laptop, so the hash is not what makes the choice hard:

sha256 throughput	~1.9 GB/s (hashlib, 1 MiB chunks)
_digest inside a bind() of a 77 MB parquet + 8760-row parquet	34 ms of 1850 ms — 1.8% of the call

Method: cProfile over one bind(), 5M-row source feeding three parameters and
two coordinates, warm cache. The digest only rivals the data read when that read
is a narrow projection of a wide parquet — 159 ms vs 13 ms for two of nine
columns off 301 MB — and even there it is small in absolute terms.

The assumption is now stated in SPEC §2 rather than left implicit (#1).