[Bug] Spark vector search returns a short top-K when a residual filter is not pushed down

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master

Compute Engine

Spark (primary-key / data-evolution vector search via the vector_search TVF)

Minimal reproduce step

Run a Spark vector_search(...) with a LIMIT k, and a WHERE conjunct that SparkV2FilterConverter cannot convert to a Paimon predicate: a UDF call, a column-to-column comparison (a > b), or an unresolvable cast. For example a filter like udf(col) = 1 combined with the vector search, limit = k.

What doesn't meet your expectations?

The result can miss qualifying rows.

The vector search returns exactly k rows, the top-k by vector distance, computed over the pushed-down (convertible) predicates only. A non-convertible conjunct is not pushed into Paimon (PaimonBaseScanBuilder.pushPredicates keeps it in postScan), so Spark applies it as a FilterExec above the already-truncated top-k. Post-filtering the top-k can only drop rows, never refill the ones displaced out of it, so rows that satisfy the residual but ranked just outside the returned k are lost, and the result is short.

This is distinct from #9883, which is about a predicate that reaches Paimon but the scalar index cannot evaluate. Here the predicate never reaches Paimon at all, so the evaluator-level fix in #9883 does not cover it. The Flink vector_search procedure does not have this problem: it parses the whole where string into the Paimon predicate, so an inexpressible conjunct fails the call rather than silently becoming a residual.

Anything else?

Found while tracing the vector-search filter pushdown for #9883. Fix direction: when a vector search has a non-convertible residual, either refuse the pushdown / fail fast, or over-retrieve and re-apply the residual to refill the top-k. The lateral / dynamic-query-vector path (PushDownLateralVectorSearchFilter) has the same exposure.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[LuciferYang]

This issue and #9855 (merged today) overlap, so I want to settle on one consistent behavior before the next release.

Both concern the same situation: a vector_search / hybrid_search / full_text_search with a WHERE conjunct that Spark cannot push into Paimon (a UDF, a column-to-column comparison, id % 2 = 1, an unresolvable cast). That conjunct stays a Spark residual applied above the search, which has already truncated to the top-K. Post-filtering the top-K can only drop rows, so the result is a subset of the top-K and may be short. It is a silent recall loss: the returned rows are correct, but rows that satisfy the predicate and rank just outside the top-K are never considered.

#9855 treats this as expected for full-text search and documents it (filtered.subsetOf(unfiltered), "may be short"). #9931 (PR #9964) treats it as a bug and fails the query fast with a clear error, matching the Flink vector_search procedure, which rejects an inexpressible predicate. So two opposite defaults landed for the same case, close together, and my PR's fail-fast currently breaks #9855's full-text test.

For reference, this is the standard prefilter/postfilter split. Lance, for example, exposes an explicit prefilter flag: postfilter (its default) runs the ANN first and applies the residual afterward, so it may return fewer than k, while prefilter feeds the predicate into the search so the top-K is computed over matching rows. Prefilter is the only mode that returns the true k-nearest-that-match, but it needs the predicate to be evaluable by the engine. For a Spark-only residual that Paimon cannot express, prefilter would degrade to a full scan, so the cheap choices here really are post-filter (short) or fail.

Proposal: unify the three search TVFs under one option that selects the residual behavior, instead of letting "did the optimizer manage to translate the predicate" decide it silently:

• post-filter: the current [core][spark][docs] Support row filters in full-text search #9855 behavior, apply the residual above the top-K, may be short. [core][spark][docs] Support row filters in full-text search #9855's test keeps its subsetOf assertion under this mode.
• fail: reject the query with a clear message (Flink parity, the strict kNN contract).

Same code path for all three search types, so the only real decision is the default. My lean is post-filter as the default (it matches #9855 and the common industry default) with fail available as an opt-in for callers who want the strict contract, but I would rather align on this with you first. @zhuxiangyi @JingsongLi what do you think the default should be?

If that direction sounds right, I will rework #9964 into the unified option across all three TVFs rather than the current fail-only.

[zhuxiangyi]

@LuciferYang @JingsongLi

In #9855, keeping predicates that Spark cannot push down into Paimon as post-filters was intentional. The documentation and tests explicitly cover the possibility of returning fewer than K rows: these predicates filter the already-truncated search results, without refilling them.

For this case, I lean toward post-filter as the default, with fail available as an opt-in, consistently across the three search TVFs. This would preserve the behavior documented in #9855 while giving callers an explicit way to reject post-filter semantics.

Keeping the existing full-text test under post-filter and adding rejection coverage under fail sounds reasonable. The check should distinguish genuinely non-pushable predicates from Spark-side rechecks of predicates already pushed into Paimon.

I’m open to aligning the behavior across the search APIs, and would also like to hear Jingsong’s thoughts before settling on the default.