Skip to content

[core] Push down deletion vectors for merged data evolution groups - #9987

Open
lilei1128 wants to merge 1 commit into
apache:masterfrom
lilei1128:merged_group_optimze
Open

lilei1128 wants to merge 1 commit into
apache:masterfrom
lilei1128:merged_group_optimze

Conversation

@lilei1128

Copy link
Copy Markdown
Contributor

Purpose

Merged data evolution groups could use file indexes to skip an entire
group, but index evaluation did not account for deletion vectors. When
all indexed matches had been deleted, the union reader still opened
every file in the group even though no live row could match.

  • Read the group deletion vector before evaluating merged-group file
    indexes.
  • Pass the deletion vector, selected row ranges, and file offset to
    file index evaluation.
  • Intersect each winning-column file index result with the group's live
    rows before deciding whether to skip the whole group.
  • Keep blob/vector files and overwritten field indexes excluded from
    group-level pruning.

Tests

  • Add testMergedGroupFileIndexComposesWithDeletionVector to verify that:
    • a merged group is skipped when its only bitmap match is deleted;
    • deleting a neighboring row does not skip the live bitmap match.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED (triage: GO)

Implementation: FINDINGS

[P2] Preserve the cheap file-index rejection before loading the deletion vector

DataEvolutionSplitRead#createReader now calls readDeletionVector at lines 253-254 before skipByFileIndex. DeletionVector.Factory#create opens and reads the DV sidecar, so every merged row-id group with a DV now pays that storage read even when the ordinary file index would have rejected the group without opening any data/DV file. This regresses the most selective scans: a table with many merged groups can add one remote sidecar read per group that used to be skipped cheaply.

Please keep a first file-index-only rejection before loading the DV (return immediately if it is already SKIP), and only load/intersect the DV when the raw index still has candidates. The new deleted-candidate behavior and its test can remain as the second-stage evaluation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants