[core] Support file index and predicate push down in DataEvolutionSplitRead

[wombatu-kun]

Purpose

Resolves the TODO in DataEvolutionSplitRead.withFilter, which was a no-op: Support File index push down (all conditions) and Predicate push down (only if no column merge).

As a result, data evolution tables never used their file indexes on read: AppendOnlyFileStoreScan evaluates the embedded index in filterByStats(ManifestEntry), but DataEvolutionFileStoreScan overrides that method and only checks row id ranges. Bloom, bitmap and bsi indexes were written for data evolution files and never read back. Format level push down did not reach the reader either, since the FormatReaderMapping.Builder was built with filters = null.

Push down is now applied wherever it can not interfere with the column merging:

• No column merge: the file index can skip the file, a BitmapIndexResult narrows the read to the matching positions, and the filters reach the format reader, mirroring RawFileSplitRead.
• Merged group: whole group skipping only. DataEvolutionFileReader zips its inner readers positionally, so dropping rows in one of them would break the column alignment. Skipping the group is sound because a logical row takes each column from exactly one file.

The existing enabledFilterPushDown flag of FormatReaderMapping.Builder already draws that line (the merge path calls build(..., false)), so passing filters into the builder needs no extra branching.

Three safety points:

• Only plain data files may skip a merged group. mergeRangesAndSort asserts areAllRangesSame(dataFiles), while a blob or vector-store file covers only a sub range and proves nothing about the other rows.
• Filters on _ROW_ID and _SEQUENCE_NUMBER are never pushed down: they are assigned from the manifest entry, and data evolution may reassign row ids, so a physical copy in the file can be stale.
• A filter is pushed to a single file only for the columns that file actually wrote. A column missing from a data evolution file is not null, its values live in another file of the same row id range, and formats such as Parquet read a column absent from the file schema as all null and would drop every row. The missing set is computed from the file writeCols, and the single file reader mappings use a dedicated cache keyed by writeCols so they can not collide with the merge path cache.

Semantics are unchanged: withFilter is a hint, and this only drops rows proven not to match.

Benchmark (throwaway, not in the diff): 200k rows in 40 row id groups, ids interleaved so min/max stats prune nothing, bloom filter, point filter. 39 of 40 groups skipped.

Scenario	Rows read	best	Speedup
Groups of one file	200,000 to 5,000	74-79 ms to 24-26 ms	~3.1x
Merged groups	200,000 to 5,000	83-90 ms to 24 ms	~3.6x

Left for a follow up: row level bitmap selection inside a merged group, which needs the bitmap translated into global row id ranges and fed through the existing rowRanges mechanism.

Tests

New DataEvolutionFileIndexTest (11 tests). Readers are created directly from the splits without executeFilter(), so an empty result proves the reader itself pruned the rows; filter values sit inside the column min/max range so the scan does not drop the split first.

Covered: a single file group skipped by a standalone .index file, a merged group skipped by an embedded index, column alignment preserved in a merged group, exact bitmap index selection, a merged group skipped after RENAME COLUMN (the filter has to be devolved by field id), a filter on a column absent from the read file returns all rows across parquet, orc and avro, a bitmap selection composed with a deletion vector, file-index.read.enabled = false, and a filter mixing _ROW_ID with a data column.

[JingsongLi]

• skipByFileIndex iterates through all files in a merge group; if any old file’s index is determined to be a mismatch, the entire group is skipped.
• However, the same field may have been overwritten by an updated file, rendering the old file effectively obsolete.
• New local regression: Old f1=a*, new f1=c*; query f1=‘c050’; expected 100 rows, actual 0 rows; consistently reproducible.

I suggest you use ChatGPT and set it to the highest review level; as far as I recall, your PR has already contained several serious errors. This is dangerous for the community.

[wombatu-kun]

Fixed in 40f9115: skipByFileIndex now evaluates each file's index only over the columns it is the newest writer of (matching the winner selection in evolutionStats and createUnionReader). Added regression test testMergedGroupKeptWhenFilterColumnOverwritten reproduces the report (old f1=a*, new f1=c*, query f1='c050' -> 100 rows).

[JingsongLi]

It is better to introduce Table tests to validate reading with filter.

[wombatu-kun]

Done d3c0e6f. Each scenario now also reads the whole plan through TableRead#executeFilter() and asserts the rows a query returns, compared against a full scan filtered in memory.