fix: error when a path cannot be made relative to the dataset base

[LuciferYang]

remove_prefix existed twice, behaviourally identical, in dataset/cleanup.rs and dataset/files.rs, and both fell back to returning the input unchanged when Path::prefix_match did not match. That hands back an absolute path where every caller wants one relative to the dataset root. tracked_files and all_files both document their path column as relative to base_uri, so the fallback silently broke a documented contract: a consumer joining the two gets a path that resolves nowhere. Erroring is the only honest answer, because the correct root-relative path is not recoverable at that point.

This is not only tidying. The fallback could destroy data. If a misreported location happens to read as base-relative under data — for example a base whose own first segment is data, or the reported path simply starting with it — cleanup's classifier passes it through the starts_with("data") check, fails to find it in the keep-set, and hands a file the manifest still references to the delete stream. The guard removes that path entirely.

Hardening this now, rather than folding it into the work it unblocks, is deliberate. #8097 adds Dataset::referenced_files, an authoritative keep-set for external distributed cleanup, and it builds that set by inserting this helper's output directly — one entry per referenced data file, per deletion file, and per index prefix. Those inputs are join-derived too, so the fallback is no more reachable there than anywhere else, but the blast radius differs in kind: an absolute path in a keep-set means the real root-relative path is absent from it, and a driver computing the difference against a listing then deletes a file the manifest still references. A helper whose output authorizes deletion should not carry a silent fallback at all.

The two copies are now one strip_prefix returning Result, kept in files.rs because files/scan.rs is its child and shares it. Error::internal rather than invalid_input: the helper is private and no user-supplied path reaches it, so a mismatch is a broken invariant rather than bad input.

Seven of the eight production call sites propagate with ?. Four of those build their input by joining onto the base and so cannot reach the error at all; the other three pass a location the object store reported from a listing, and there dropping an entry is what causes damage — an omission from a keep-set gets a live file deleted, and an omission from a file inventory gets one deleted by whatever consumes that inventory. The eighth, CleanupTask::cleanup_file_if_not_referenced, is the one site that runs while the pass is already deleting, and there the safe answer is the opposite: it skips the object and returns Ok(None), the value that function already returns for every other shape it cannot classify. Propagating would strand a pass that has already removed files, leaving old manifests behind while some of the data they reference is gone, and it would repeat on every later run. Because a store that misreports one path misreports every path of that kind, the per-file detail stays at debug! and the pass warns once with a count, so an operator learns at the default log level that files are being left behind without getting one line per object.

One user-visible regression to flag rather than bury. On stores Lance reaches through the OpenDAL bridge (hf:// and goosefs:// always, s3:///gs:///az:// with use_opendal=true), listings return percent-encoded locations while Dataset::base holds the raw string, so for a dataset path containing a character in object_store's encode set the two never match. all_files and tracked_files go from emitting a wrong-but-present path to failing outright. The root cause is in lance-io's bridge, not in this change, and is filed separately as #8652; this change makes the existing breakage loud instead of silent. Cleanup is unaffected either way, since the absolute path already matched none of its expected prefixes.

Deliberately out of scope: two byte-wise prefix strips with the same silent-fallback shape remain in mem_wal/memtable/flush.rs and lance-table/src/format/index.rs, both unreachable today, and DataFile.path's single-segment invariant is neither documented nor validated at the boundaries that accept it. Both belong with the follow-up that moves the helper somewhere lance-table can reach it.

Test plan

Four strip_prefix cases cover the relative result, the equal-to-base result, a path under a different dataset, and a near-miss sibling (bucket/dataset2 against base bucket/dataset) that pins matching as per path segment rather than per byte. build_all_files_batch_rejects_a_path_outside_the_base constructs an ObjectMeta directly and asserts the batch fails naming the offending path, rather than emitting an absolute one.

cleanup_skips_listed_files_outside_the_dataset_base installs a store wrapper that reports a rewritten location for the data files and asserts the pass leaves them alone and still completes. The rewrite target is data/outside-base/{filename}, chosen so the test discriminates in both directions: it is outside the base, so the guard skips it, and it still reads as a collectable data path, so the removed fallback would have handed it to the delete stream. Verified by reverting each half — restoring the fallback fails the test, and replacing the skip with ? fails it too. The rewritten entry is stamped a day before the epoch so it clears any cutoff cleanup computes rather than sitting on the boundary, and two count assertions stop the test decaying into a vacuous pass.

cargo fmt --all --check, cargo clippy --all --tests --benches -- -D warnings, and RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps are clean. cargo test -p lance --lib dataset:: passes, 2232 tests. The full suite is left to CI.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The shared segment-aware prefix check restores the documented relative-path contract and, critically, makes cleanup leave an unrelatable listing untouched while reporting the condition. This is the right fail-closed containment for the demonstrated deletion path.

On OpenDAL-backed stores, percent-encoded listed locations can now make all_files and tracked_files fail until #8652 normalizes the bridge output. The author has explicitly accepted and documented that bounded regression for this PR; no further change is requested here.

[LuciferYang]

cc @wjones127 FYI

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!