Skip to content

feat(vector-search): use data conjuncts only for the PK-vector residual and pre-filter guard#609

Open
JunRuiLee wants to merge 2 commits into
apache:mainfrom
JunRuiLee:feat/vector-filter-partition-split
Open

feat(vector-search): use data conjuncts only for the PK-vector residual and pre-filter guard#609
JunRuiLee wants to merge 2 commits into
apache:mainfrom
JunRuiLee:feat/vector-filter-partition-split

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Purpose

Primary-key vector search treated the whole with_filter predicate as if it
needed physical-row (residual) handling — both when gating the query and when
building the residual — even though the same predicate is already pushed into
scan planning (PkVectorScan), where partition-only conjuncts prune
partitions/files. That made two things wrong for partition-referencing filters:

  1. Re-checking partition conjuncts per row is redundant, and re-reading
    partition columns for the residual is wasted I/O.
  2. The pre-filter guard rejected any with_filter on a table without
    deletion vectors (or with merge-on-read) via filter.is_some(), so a
    partition-only filter (e.g. dt = 'a') was refused even though it needs no
    physical-row read at all — a query Java admits.

Change

Split the predicate into partition and data (non-partition) conjuncts via
split_partition_and_data_predicates and use only the DATA conjuncts wherever
physical rows are involved. Two call sites:

Residual construction — derive the residual from the data conjuncts only:

  • A partition-only filter (e.g. dt = 'a') now needs no residual at all — the
    residual read is skipped entirely (planning already enforces the partition).
  • AND(partition, data) residual-checks only the data conjunct (one fewer
    column re-read per candidate file).
  • A mixed conjunct (e.g. dt = 'a' OR id > 5) is not partition-only, so it
    stays whole in the residual and evaluates against the materialized partition
    column (primary-key data files carry partition columns) — results unchanged.

Pre-filter guard — gate the deletion-vector requirement on the data
conjuncts, not the whole filter:

  • A partition-only (or absent) filter passes the guard even without deletion
    vectors — it never triggers a physical-row read.
  • A data conjunct (including a mixed OR) on a non-deletion-vector table still
    fails loud, unchanged.

Planning still receives the whole filter, so partition pruning is unaffected.
Both sites mirror Java BatchVectorSearchBuilderImpl.withFilter (which splits at
the builder level, leaving filter == null for a partition-only filter so
PrimaryKeyVectorScan's deletion-vector guard is skipped). Covers the
whole-snapshot execute_read path (single-query and batch converge on the
shared core).

Output is unchanged; this reduces residual I/O and per-row work for
partition-referencing filters, keeps the residual's inputs to data-only
predicates, and lets a partition-only filter run on tables without deletion
vectors.

Tests

  • residual_uses_only_data_conjuncts_of_the_filter: partition-only → no
    residual; data-only → kept; AND(partition, data) → only the data conjunct;
    mixed OR → stays whole.
  • execute_read_partition_only_filter_without_deletion_vectors_passes_guard:
    on a partitioned PK-vector table without deletion vectors, a partition-only
    filter is admitted (empty stream, no error), while data-only, AND(partition, data), and mixed OR(partition, data) filters all still fail loud.

cargo test -p paimon, cargo fmt --all --check, and
cargo clippy -p paimon --all-targets -- -D warnings pass.

…r the residual

The primary-key vector search reused the whole with_filter predicate as the
per-row residual applied after recall, even though the same predicate is
already pushed into scan planning (where partition-only conjuncts prune
partitions/files). Re-checking partition-only conjuncts per row is redundant.

Derive the residual from the data (non-partition) conjuncts only, via
split_partition_and_data_predicates: a partition-only filter now needs no
residual at all (skipped), and an AND of partition + data conjuncts residual-
checks only the data conjunct. Mixed conjuncts (e.g. partition OR data) stay
whole in the residual and evaluate against the materialized partition column,
which primary-key data files carry. Mirrors Java BatchVectorSearchBuilderImpl.

Covers the whole-snapshot execute_read path (single and batch converge on the
shared core).
@JunRuiLee
JunRuiLee marked this pull request as ready for review July 25, 2026 08:40
…ncts

resolve_pk_vector_search_params rejected any with_filter set on a table without
deletion vectors (or with merge-on-read), keying the guard on the whole filter
via filter.is_some(). But a partition-only filter needs no per-row residual —
partition pruning is enforced entirely by scan planning — so requiring physical
rows (deletion vectors without merge-on-read) for it is wrong: it rejects a
partition-only filter that Java admits.

Java splits the predicate at the builder level (BatchVectorSearchBuilderImpl
.withFilter) and leaves this.filter == null for a partition-only filter, so the
PrimaryKeyVectorScan guard is skipped. Mirror that: split off the data
(non-partition) conjuncts and gate the guard on those, so a partition-only (or
absent) filter passes while a data conjunct on a non-deletion-vector table still
fails loud.
@JunRuiLee JunRuiLee changed the title feat(vector-search): split with_filter into partition + data parts for the residual feat(vector-search): use data conjuncts only for the PK-vector residual and pre-filter guard Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant