fix: refuse to drop a path that is not a Lance dataset

[xiaguanglei]

Summary

Dataset.drop recursively deletes whatever path it is handed via remove_dir_all, with no check that the target is a Lance dataset. The path comes from a public Python/Java API and from catalog-driven callers (e.g. Spark DROP TABLE / CREATE OR REPLACE TABLE), so a mistyped warehouse root or bucket prefix can destroy unrelated data with no recovery.

⚠️ This is not a theoretical issue — we have already encountered this risk in production. A mistyped base_uri could wipe out the entire bucket root, affecting all tables and data stored in the same location, with no recovery path.
For example, consider:

ds = lance.dataset('s3://my-bucket/lance_test')
ds.drop(base_uri='s3://my-bucket')  # typo: intended to drop only the table

This currently deletes everything under s3://my-bucket/ — all files, all directories, and all other tables or data stored in that bucket — with no way to recover.The same risk applies to S3, GCS, and other object stores.

This PR adds a fail-closed check immediately before deletion: the path must hold positive Lance evidence — a file under _versions/ that both parses as a manifest location and deserializes as a manifest (committed or detached), or a namespace declare/deregister marker (.lance-reserved / .lance-deregistered). Anything else is rejected with InvalidInput / ValueError / IllegalArgumentException. Missing or empty paths keep today's not-found / ignore_not_found behavior.

This is intentionally minimal: only Dataset.drop (Python/Java); namespace drop_table is unchanged; create semantics are unchanged; a successful drop still deletes everything under the path.

Policy

A path may be dropped when either:

1. _versions/ holds a file that parses as a manifest location (ManifestLocation::try_from / detached-version naming) and read_manifest succeeds on it, or
2. a .lance-reserved / .lance-deregistered marker is present (declared or deregistered, not yet materialized).

Nothing weaker qualifies. In particular:

• a merely nonempty _versions/ is not identity (e.g. _versions/README next to reports/q1.csv is refused);
• a path whose only Lance-looking content is an unreadable / empty / staging-named “manifest” is refused;
• a path that holds only data/ (or data/ + tree/) is refused.

Those leftovers look identical to a storage root whose only top-level prefix is data/ — a common layout on object stores — and deleting the wrong one is unrecoverable. Corrupt or uncommitted cleanup is left to an explicit storage-level delete rather than a weaker default guard. Failing closed is cheap: such leftovers do not block re-create (create only refuses a path that already has a manifest), and cleanup_old_versions removes data files no manifest references.

Unmanaged files next to a committed dataset do not block drop (matching cleanup's allowance for them). Note that the recursive delete still removes those files once validation passes.

Probes list _versions/, parse candidate names with the existing manifest naming logic, and stop at the first successful read_manifest, then fall back to marker exists checks. Dropping a real dataset therefore costs one listing plus one manifest read, and does not list an entire bucket root.

Compatibility

• Real dataset paths: same as today (delete succeeds).
• Missing / empty paths: still fall through to remove_dir_all + existing not-found / ignore_not_found handling (OSError in Python), not a new validation error.
• Non-dataset paths that exist: now raise instead of deleting.
• DirectoryNamespace::drop_table / ManifestNamespace::drop_table: untouched (catalog-derived paths; different threat model).

Boundaries / non-goals

• Still recursive: when validation passes, remove_dir_all deletes the whole tree, including unmanaged siblings (images/, notes.txt). Making drop selective (only Lance-managed prefixes) would align better with cleanup but is a separate semantic change.
• No create-side check: create still only looks for an existing manifest; it does not require an empty directory.
• No deep nesting walk: nested _versions/ under a child table name does not make a parent warehouse look like a dataset (warehouse roots are rejected).
• Corrupt / uncommitted leftovers: refused by default; use an explicit storage-level delete. This includes external-store staging names before the canonical manifest is materialized.

Test plan

• Rust: cargo test -p lance --lib validate_dataset_root_for_drop (committed + unmanaged siblings, detached manifest, markers, warehouse / home / unrelated / data-only / layout-without-manifest, nonempty _versions without a readable manifest, unreadable / empty / staging-named manifests, missing path)
• Python: uv run pytest python/tests/test_dataset.py -k drop (including rejects-paths-without-readable-manifest and create-over-uncommitted-leftovers)
• Java: ./mvnw test -Dtest='DatasetTest#testDropPath+testDropRejectsNonDatasetPath' (2 passed)
• cargo fmt --all, cargo clippy -p lance --tests -- -D warnings, uv run make lint, ./mvnw spotless:check, RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebased revision preserves the positive-identity guard: recursive drop is authorized only by a readable committed or detached manifest, or an explicit namespace marker. The new rooted local/UNC object-store handling keeps validation, manifest reads, and deletion scoped to the same store root, so the prior safety conclusion remains supported.