fix(io): normalize OpenDAL listed paths - #8654
Conversation
|
Blocked: the current head is held by an unrelated intermittent macOS BQ test failure that this App cannot rerun. Remote head 1d7f8e3 contains the current main tip b945b3e. The mac-build (stable) Check failed vector::bq::storage::tests::test_accurate_approx_mode_reduces_binary_lut_quantization_error with accurate_error=0.07851982 and normal_error=0.06913757; this PR changes only lance-io, and the identical base SHA passed the macOS Check. I fetched and verified base ancestry, inspected the live log, ran both the failed BQ test and the OpenDAL regression successfully, and pushed the assigned branch, which was already up to date. A failed-workflow rerun was requested, but the GitHub App lacks Actions rerun permission. The smallest unblock is for a maintainer to rerun the failed mac-build (stable) job; alternatively, rerun the full Rust workflow on this head. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The shared adapter now uses the requested prefix as its compatibility boundary: it fixes raw reserved-character listings while preserving literal percent-escape paths that already round-trip. This is the right boundary because descendant spellings are otherwise ambiguous after OpenDAL decodes the backend key.
Summary
Root cause
object_store_opendalrebuilds OpenDAL listing entries withPath::from, which percent-encodes reserved characters. Lance constructs dataset bases withPath::from_url_path, so a raw reserved-character base can mismatch the returned listing. Unconditionally decoding the output is also unsafe because a deliberately double-encoded URI segment relies on the existing encoded listing spelling.The adapter now preserves any upstream location that already prefix-matches the request and decodes only mismatches. This fixes raw reserved-character bases without changing literal percent-escape paths that already worked.
Validation
cargo test -p lance-io --no-default-features --features oss object_store::opendal_store::tests::test_list_preserves_request_path_spellingcargo test -p lance-io --lib -- --skip uring::tests(202 passed)cargo check -p lance-io --all-features --testscargo fmt --all -- --checkcargo clippy --all --tests --benches -- -D warningsThe complete
lance-iolibrary test command cannot finish in this container because io_uring initialization is denied by the environment; the non-io_uring library suite passes.Fixes #8652