perf: evaluate primitive predicates on encoded pages

[Xuanwo]

Primitive filters on current structural data files are evaluated only after decoding the predicate column to Arrow. For selective filters, this spends CPU and memory materializing values that only need a boolean result, and can read substantially more payload data than the result requires.

This PR evaluates supported i32/u32 comparisons directly on Constant, RLE, and bitpacked pages. The default auto mode makes its cost decision independently for each fragment from the encoding type, sampled selectivity, selected range count, and predicted read amplification. Full evaluation stops and retains the ordinary filter when it exceeds either the encoding-specific amplification threshold or 4,096 ranges. Unsupported or unfavorable cases also retain the existing fallback, and Constant outcomes only short-circuit when metadata and nullability make the result exact.

Benchmark

Measured on AWS EC2 c7i.4xlarge with a release-with-debug build and a 10M-row dataset split across 10 data files. Each variant runs the same user-level scan: project the payload columns, apply the primitive filter, and consume all output batches. Each workload used two opposite-order blocks per mode (ABBA for local filesystem and BAAB for S3), with six iterations per block. The first iteration of each block was discarded, leaving 10 warm samples per mode; the table reports their median.

Workload	Local filesystem	S3	S3 bytes read
RLE sparse equality, 12,288 rows returned	120.4 ms → 6.23 ms (19.3×)	291.4 ms → 103.0 ms (2.83×)	276 MB → 876 KB
RLE constant-value no-match	117.0 ms → 6.21 ms (18.9×)	316.9 ms → 62.4 ms (5.08×)	276 MB → 508 KB
Fragmented bitpacked equality, rejected by auto	129.3 ms → 125.7 ms	302.0 ms → 322.4 ms (+6.7%)	288.47 MB → 288.90 MB
RLE constant-value all-match, rejected by auto	117.7 ms → 116.4 ms	284.7 ms → 324.1 ms (+13.8%)	276.06 MB → 276.32 MB

The rejected S3 workloads show the bounded cost of making the safety decision per fragment: profiling adds requests even though the ordinary filter is ultimately retained. The accepted selective workloads still reduce both end-to-end latency and bytes read substantially.

The measured library/runtime contents match PR head 1140cac9d0. The private benchmark implementation commit is 00c4018a5c; benchmark runner and summarizer provenance is f089a32876. Raw results and the harness remain in the private benchmark project.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

Both prior findings are fixed: nullable constants preserve null-filter semantics, and auto planning is now per fragment with a bounded full-evaluation fallback.

The published rejected-bitpacking benchmark predates that planner change. On its ten-fragment topology, current auto profiles 327,680 rows across 20 windows instead of 32,768 across two, so the reported +8.5% S3 rejection overhead is not evidence for this head. Re-run that guardrail workload before relying on its remote-overhead figure.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!