Search before asking
Motivation
Currently, chain tables support merge-on-read across snapshot and delta branches, but they do not support deletion vectors. For workloads that use deletion-vectors.enabled=true to speed up point deletes and updates, users cannot benefit from chain tables.
Solution
Expose DV-marked rows as -D tombstone KeyValues during chain read, so the merge function can apply cross-branch deletes correctly. The normal (non-chain) read path still uses ApplyDeletionVectorReader for branch-local filtering.
- ChainSplit: carry deletion files from both snapshot and delta branches alongside their data files.
- ChainGroupReadTable: collect the relevant deletion files for each data file when constructing ChainSplit.
- ExposeDeletionKeyValueReader (new): wraps a
KeyValueDataFileRecordReader and flips valueKind to RowKind.DELETE for positions marked by the deletion vector, so the merge function receives the delete record instead of silently dropping the row.
- ChainKeyValueFileReaderFactory: use
ExposeDeletionKeyValueReader when a non-empty DV is present.
- MergeFileSplitRead#createChainReader: unwrap
LookupMergeFunction.Factory so cross-branch records are merged by the inner merge function (e.g., DeduplicateMergeFunction) using the sequence field.
- Validation: restrict chain-table DV support to the DEDUPLICATE merge engine.
Anything else?
Why only DEDUPLICATE?
DEDUPLICATE and PARTIAL_UPDATE handle -U / -D differently. In DEDUPLICATE both -U and -D mean retract/delete, so a DV tombstone maps cleanly. In PARTIAL_UPDATE, -U is a branch-local retraction that must not cross-delete snapshot records, while -D is a real delete. Because DV bitmap marks are exposed uniformly as -D tombstones during chain read, the -U/-D distinction is lost, which would cause delta -U operations to incorrectly delete matching snapshot records. Therefore DV is only enabled for DEDUPLICATE in this change.
Are you willing to submit a PR?
Search before asking
Motivation
Currently, chain tables support merge-on-read across snapshot and delta branches, but they do not support deletion vectors. For workloads that use
deletion-vectors.enabled=trueto speed up point deletes and updates, users cannot benefit from chain tables.Solution
Expose DV-marked rows as -D tombstone KeyValues during chain read, so the merge function can apply cross-branch deletes correctly. The normal (non-chain) read path still uses
ApplyDeletionVectorReaderfor branch-local filtering.KeyValueDataFileRecordReaderand flips valueKind toRowKind.DELETEfor positions marked by the deletion vector, so the merge function receives the delete record instead of silently dropping the row.ExposeDeletionKeyValueReaderwhen a non-empty DV is present.LookupMergeFunction.Factoryso cross-branch records are merged by the inner merge function (e.g., DeduplicateMergeFunction) using the sequence field.Anything else?
Why only DEDUPLICATE?
DEDUPLICATEandPARTIAL_UPDATEhandle-U/-Ddifferently. InDEDUPLICATEboth-Uand-Dmean retract/delete, so a DV tombstone maps cleanly. InPARTIAL_UPDATE,-Uis a branch-local retraction that must not cross-delete snapshot records, while -D is a real delete. Because DV bitmap marks are exposed uniformly as -D tombstones during chain read, the-U/-Ddistinction is lost, which would cause delta -U operations to incorrectly delete matching snapshot records. Therefore DV is only enabled forDEDUPLICATEin this change.Are you willing to submit a PR?