Skip to content

[Feature] Support deletion vectors for chain table #8573

Description

@juntaozhang

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Currently, chain tables support merge-on-read across snapshot and delta branches, but they do not support deletion vectors. For workloads that use deletion-vectors.enabled=true to speed up point deletes and updates, users cannot benefit from chain tables.

Solution

Expose DV-marked rows as -D tombstone KeyValues during chain read, so the merge function can apply cross-branch deletes correctly. The normal (non-chain) read path still uses ApplyDeletionVectorReader for branch-local filtering.

  • ChainSplit: carry deletion files from both snapshot and delta branches alongside their data files.
  • ChainGroupReadTable: collect the relevant deletion files for each data file when constructing ChainSplit.
  • ExposeDeletionKeyValueReader (new): wraps a KeyValueDataFileRecordReader and flips valueKind to RowKind.DELETE for positions marked by the deletion vector, so the merge function receives the delete record instead of silently dropping the row.
  • ChainKeyValueFileReaderFactory: use ExposeDeletionKeyValueReader when a non-empty DV is present.
  • MergeFileSplitRead#createChainReader: unwrap LookupMergeFunction.Factory so cross-branch records are merged by the inner merge function (e.g., DeduplicateMergeFunction) using the sequence field.
  • Validation: restrict chain-table DV support to the DEDUPLICATE merge engine.

Anything else?

Why only DEDUPLICATE?

DEDUPLICATE and PARTIAL_UPDATE handle -U / -D differently. In DEDUPLICATE both -U and -D mean retract/delete, so a DV tombstone maps cleanly. In PARTIAL_UPDATE, -U is a branch-local retraction that must not cross-delete snapshot records, while -D is a real delete. Because DV bitmap marks are exposed uniformly as -D tombstones during chain read, the -U/-D distinction is lost, which would cause delta -U operations to incorrectly delete matching snapshot records. Therefore DV is only enabled for DEDUPLICATE in this change.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions