perf: stop listing the entire _versions directory for limited version queries

[LuQQiu]

Problem

DirectoryNamespace::list_versions_under always collects every object under the table's _versions/ prefix (read_dir_all(...).try_collect()) before sorting and applying the caller's limit. On object stores this paginates the entire directory even when the caller only wants the latest version (descending=true, limit=1) — which is exactly what get_latest_version does on every dataset resolution through the namespace commit path.

For a table with ~340k versions on S3 this is ~344 sequential ListObjectsV2 pages per resolution: ~25s of pure I/O wait (idle≈26s / busy≈1s per span), paid by every open_table/describe/merge_insert that resolves the latest version. The equivalent non-namespace path (resolve_version_from_listing) already resolves the latest V2 manifest from roughly one list page.

Fix

On lexically-ordered stores the V2 manifest naming scheme (u64::MAX - version, zero-padded) already yields newest-first listings. When the requested order matches the stream's natural order, consume the stream and stop after limit entries instead of collecting everything:

• descending + limit on an ordered store with V2 naming → stream is already in the right order → stop after limit entries (the get_latest_version hot path becomes a single list page).
• Everything else (unordered stores, mismatched order, no limit) keeps the existing collect-then-sort behavior.

Also fixes a latent edge: the first stream entry is read for naming-scheme detection, so the limit is re-enforced afterward (covers limit=0).

Measured

Against a real ~340k-version table on S3 (list_table_versions with descending=true, limit=1, same binary, 3 runs):

	elapsed	S3 LIST requests
before	~25s	344 (full pagination)
after	0.20–0.23s	1 page

Tests

• New test_list_versions_under_ordering_and_limit covers both paths (memory:// ordered → early-stop; local fs unordered → collect-then-sort) for every descending/limit combination including limit=0 and over-limit.
• cargo test -p lance-namespace-impls --lib: 299 passed, 0 failed.
• cargo clippy --no-deps -p lance-namespace-impls --all-targets -- -D warnings clean.

🤖 Generated with Claude Code

[lance-gatekeeper]

✅ Gate recommendation: approve.

The retained-staging case is fixed: ordered V2 latest-version lookup now detects its naming scheme from the first committed manifest and remains bounded after skipping retained non-manifest entries. No Gate blocker remains.

[westonpace]

What happens on non-ordered stores like S3 express? Does it just fall back to the old behavior?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[LuQQiu]

What happens on non-ordered stores like S3 express? Does it just fall back to the old behavior?

yep,,, Like S3 express, it's still the old behavior, still reading all the version pages