fix(scanner): honor batch_readahead to bound v2 scan decode concurrency

[zhangyue19921010]

Background & Motivation

Under concurrent-scan workloads like Spark, a single large worker (Executor) typically runs many Tasks in parallel (in Lance's read path, 1 Task = 1 fragment = 1 scan). The problem: each Task independently sizes its own scan concurrency from the core count of the current Executor / host.

Concretely in Lance, the decode concurrency of the v2 read path FilteredReadExec (plan name LanceRead) — the try_buffered(num_threads) — is unconditionally set to get_num_compute_intensive_cpus() = num_cpus − 2.

total decode concurrency per Executor ≈ (concurrent Tasks) × (num_cpus − 2)

Why It Was Deprecated, and the Gap It Left Behind

Scanner.batch_readahead is marked Ignored in v2 and newer format — a deliberate decision, not an oversight. In v1 it controlled two things: prefetch depth and decode concurrency. v2 replaced prefetch with a byte-budget model (io_buffer_size + fragment_readahead), so its prefetch role became meaningless and was rightly dropped.

The gap: v2 split the decode-concurrency role into FilteredReadExec's threading mode (try_buffered(num_threads)), hard-coded to get_num_compute_intensive_cpus(). The only override today is the process-wide LANCE_CPU_THREADS env var — far too coarse, since it governs every compute-intensive path at once (vector/KNN search, index building, take, update/merge-insert, …), not just scan decode concurrency. With no per-scan knob, there's no way to rein in the over-parallelization above.

What This Change Does & Its Impact

Have new_filtered_read pass Scanner.batch_readahead through as FilteredReadThreadingMode::OnePartitionMultipleThreads(batch_readahead), reattaching batch_readahead to the decode-concurrency dimension that v2 had left without a knob.

[codecov]

Codecov Report

❌ Patch coverage is 90.32258% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/scanner.rs	88.88%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!