feat(vector-search): use data conjuncts only for the PK-vector residual and pre-filter guard#609
Open
JunRuiLee wants to merge 2 commits into
Open
feat(vector-search): use data conjuncts only for the PK-vector residual and pre-filter guard#609JunRuiLee wants to merge 2 commits into
JunRuiLee wants to merge 2 commits into
Conversation
…r the residual The primary-key vector search reused the whole with_filter predicate as the per-row residual applied after recall, even though the same predicate is already pushed into scan planning (where partition-only conjuncts prune partitions/files). Re-checking partition-only conjuncts per row is redundant. Derive the residual from the data (non-partition) conjuncts only, via split_partition_and_data_predicates: a partition-only filter now needs no residual at all (skipped), and an AND of partition + data conjuncts residual- checks only the data conjunct. Mixed conjuncts (e.g. partition OR data) stay whole in the residual and evaluate against the materialized partition column, which primary-key data files carry. Mirrors Java BatchVectorSearchBuilderImpl. Covers the whole-snapshot execute_read path (single and batch converge on the shared core).
JunRuiLee
marked this pull request as ready for review
July 25, 2026 08:40
…ncts resolve_pk_vector_search_params rejected any with_filter set on a table without deletion vectors (or with merge-on-read), keying the guard on the whole filter via filter.is_some(). But a partition-only filter needs no per-row residual — partition pruning is enforced entirely by scan planning — so requiring physical rows (deletion vectors without merge-on-read) for it is wrong: it rejects a partition-only filter that Java admits. Java splits the predicate at the builder level (BatchVectorSearchBuilderImpl .withFilter) and leaves this.filter == null for a partition-only filter, so the PrimaryKeyVectorScan guard is skipped. Mirror that: split off the data (non-partition) conjuncts and gate the guard on those, so a partition-only (or absent) filter passes while a data conjunct on a non-deletion-vector table still fails loud.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Primary-key vector search treated the whole
with_filterpredicate as if itneeded physical-row (residual) handling — both when gating the query and when
building the residual — even though the same predicate is already pushed into
scan planning (
PkVectorScan), where partition-only conjuncts prunepartitions/files. That made two things wrong for partition-referencing filters:
partition columns for the residual is wasted I/O.
with_filteron a table withoutdeletion vectors (or with merge-on-read) via
filter.is_some(), so apartition-only filter (e.g.
dt = 'a') was refused even though it needs nophysical-row read at all — a query Java admits.
Change
Split the predicate into partition and data (non-partition) conjuncts via
split_partition_and_data_predicatesand use only the DATA conjuncts whereverphysical rows are involved. Two call sites:
Residual construction — derive the residual from the data conjuncts only:
dt = 'a') now needs no residual at all — theresidual read is skipped entirely (planning already enforces the partition).
AND(partition, data)residual-checks only the data conjunct (one fewercolumn re-read per candidate file).
dt = 'a' OR id > 5) is not partition-only, so itstays whole in the residual and evaluates against the materialized partition
column (primary-key data files carry partition columns) — results unchanged.
Pre-filter guard — gate the deletion-vector requirement on the data
conjuncts, not the whole filter:
vectors — it never triggers a physical-row read.
OR) on a non-deletion-vector table stillfails loud, unchanged.
Planning still receives the whole filter, so partition pruning is unaffected.
Both sites mirror Java
BatchVectorSearchBuilderImpl.withFilter(which splits atthe builder level, leaving
filter == nullfor a partition-only filter soPrimaryKeyVectorScan's deletion-vector guard is skipped). Covers thewhole-snapshot
execute_readpath (single-query and batch converge on theshared core).
Output is unchanged; this reduces residual I/O and per-row work for
partition-referencing filters, keeps the residual's inputs to data-only
predicates, and lets a partition-only filter run on tables without deletion
vectors.
Tests
residual_uses_only_data_conjuncts_of_the_filter: partition-only → noresidual; data-only → kept;
AND(partition, data)→ only the data conjunct;mixed
OR→ stays whole.execute_read_partition_only_filter_without_deletion_vectors_passes_guard:on a partitioned PK-vector table without deletion vectors, a partition-only
filter is admitted (empty stream, no error), while data-only,
AND(partition, data), and mixedOR(partition, data)filters all still fail loud.cargo test -p paimon,cargo fmt --all --check, andcargo clippy -p paimon --all-targets -- -D warningspass.