Skip to content

fix(fts): enforce fuzzy max_expansions globally across index partitions#7634

Draft
BubbleCal wants to merge 1 commit into
mainfrom
yang/fts-global-fuzzy-expansion
Draft

fix(fts): enforce fuzzy max_expansions globally across index partitions#7634
BubbleCal wants to merge 1 commit into
mainfrom
yang/fts-global-fuzzy-expansion

Conversation

@BubbleCal

Copy link
Copy Markdown
Contributor

Follow-up to the review discussion on #7601: fuzzy expansion previously ran
inside every partition with a per-partition max_expansions cap, and the
per-partition picks were unioned afterwards. The effective cap was
num_partitions × max_expansions, so the same query could match more terms —
and return different results — purely because of how the corpus happened to be
partitioned (e.g. after a tail-heavy build splits its leftovers).

What

  • Expansion now runs once per query, in InvertedIndex::bm25_search,
    under the query-wide max_expansions budget. For each query token the
    per-partition FST candidates merge into one lexicographically ordered set
    and the remaining budget takes a prefix of it. Each partition is only asked
    for its remaining lexicographically smallest candidates, which is lossless
    for that selection: any term among the merged lex-smallest remaining is
    also among its own partition's smallest remaining.
  • Partitions receive the final token list and no longer expand
    (load_posting_lists drops its expand_fuzzy call); params.fuzziness
    still drives the grouped fuzzy dedup/scoring semantics.
  • The BM25 scorer reuses the same expansion (bm25_scorer_for_final_tokens),
    so scorer terms and searched terms stay in lockstep and a query pays for one
    expansion instead of one for the scorer plus one per partition.
  • Fuzzy AND/phrase keep their contract — every original token position
    must retain at least one expansion — via an index-level check (previously
    implicit in each partition's own expansion).
  • InvertedPartition::expand_fuzzy stays for API compatibility,
    reimplemented on the shared per-token candidate collector; single-partition
    behavior is unchanged (same FST lexicographic order, same budget fill).

Semantics

Single-partition indexes behave exactly as before. Multi-partition indexes
previously over-expanded in proportion to their partition count; they now
honor the documented cap, and the expansion (hence the result set) is a pure
function of the segment's vocabulary, independent of partition shape.

Tests

  • test_fuzzy_expansion_cap_is_global_across_partitions: two partitions with
    disjoint variants, binding cap → exactly the 3 lexicographically smallest
    terms across both (fails on main, which returns 4).
  • test_fuzzy_results_independent_of_partition_shape: the same four docs
    built as one partition and as two return identical (row_id, score) sets
    under a binding cap (fails on main).
  • Existing fuzzy suite (grouped AND, position grouping, whole-query cap)
    unchanged and passing. The non-binding-cap regression test over the real
    tail-split builder path lives in fix(fts): split tail-partition merge by the worker memory budget #7601.

🤖 Generated with Claude Code

Fuzzy expansion ran inside every partition with a per-partition cap and
the picks were unioned afterwards, so the effective cap was
num_partitions x max_expansions and fuzzy results depended on the
partition shape. Expand once in InvertedIndex::bm25_search under the
query-wide budget: per-partition FST candidates (each truncated at the
remaining budget, lossless for lex-order selection) merge
lexicographically and the budget takes a prefix. Partitions receive the
final token list; the BM25 scorer reuses the same expansion; fuzzy
AND/phrase keep the every-position-must-survive contract via an
index-level check.
@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Jul 6, 2026
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.81768% with 13 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-index/src/scalar/inverted/index.rs 92.81% 10 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant