[spark] Add spark.paimon.search.residual-filter to select search TVF residual behavior

[LuciferYang]

Purpose

close #9931

A vector / hybrid / full-text search truncates to the top-K. A WHERE conjunct on searched-table columns that Spark cannot push into Paimon (a UDF, a column-to-column comparison, id % 2 = 1, an unresolvable cast) stays a Spark residual applied above that top-K. Post-filtering the top-K can only drop rows, so the result is a subset of the top-K and may be short: a row that satisfies the predicate but ranks just outside the returned K is never considered. This is a silent recall loss.

#9855 documented this as the accepted behavior for full-text search (filtered.subsetOf(unfiltered), may be short). This PR first landed the opposite default, failing the query fast, which broke that test, so #9931 and #9855 had two opposite defaults for the same case. After coordinating on #9931, this unifies the three search TVFs under one option that selects the residual behavior instead of letting "did the optimizer translate the predicate" decide it silently.

spark.paimon.search.residual-filter:

• post-filter (default): apply the residual above the top-K. The result is a subset of the top-K and may be shorter than K. This is the [core][spark][docs] Support row filters in full-text search #9855 behavior.
• fail: reject the query with a clear message, matching the Flink vector_search procedure and the strict kNN contract.

The same code path covers the static form (CheckUnpushableSearchFilter) and the lateral form (PushDownLateralVectorSearchFilter) of all three TVFs. Under fail, the rejection fires only for a genuinely non-pushable residual: a conjunct that references a searched-table column and that SparkV2FilterConverter cannot convert. A predicate Paimon accepted is convertible by the same converter the pushdown path uses, so a Spark-side recheck of an already-pushed predicate is never rejected.

Tests

PrimaryKeyVectorSearchTest: vector search fail mode rejects a non-convertible residual filter sets residual-filter=fail and asserts an id > threshold column-to-column residual is rejected. vector search post-filters a non-convertible residual filter by default runs the same query under the default and asserts the result is a subset of the top-K (here empty) rather than an error. vector search fail mode keeps a pushable filter asserts a convertible threshold = 100 is not rejected under fail and returns the top-K, pinning that a pushed-and-rechecked predicate never trips the check.

FullTextSearchTest's existing "predicate that cannot be pushed down is applied after the search" keeps its subsetOf assertion, now exercising the default post-filter mode.

[LuciferYang]

Marking this draft. The fail-fast approach here overlaps with #9855 (merged after this branch was cut), which intentionally applies a non-pushable residual after full-text search and documents the "may be short" result. As a result this PR currently fails #9855's full-text test.

I have raised the conflict on #9931 with a proposal to unify all three search TVFs under one option (post-filter vs fail) instead of two opposite defaults. I will rework this PR once we agree on the default there.