fix(fts): enforce fuzzy max_expansions globally across index partitions#7634
Draft
BubbleCal wants to merge 1 commit into
Draft
fix(fts): enforce fuzzy max_expansions globally across index partitions#7634BubbleCal wants to merge 1 commit into
BubbleCal wants to merge 1 commit into
Conversation
Fuzzy expansion ran inside every partition with a per-partition cap and the picks were unioned afterwards, so the effective cap was num_partitions x max_expansions and fuzzy results depended on the partition shape. Expand once in InvertedIndex::bm25_search under the query-wide budget: per-partition FST candidates (each truncated at the remaining budget, lossless for lex-order selection) merge lexicographically and the budget takes a prefix. Partitions receive the final token list; the BM25 scorer reuses the same expansion; fuzzy AND/phrase keep the every-position-must-survive contract via an index-level check.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the review discussion on #7601: fuzzy expansion previously ran
inside every partition with a per-partition
max_expansionscap, and theper-partition picks were unioned afterwards. The effective cap was
num_partitions × max_expansions, so the same query could match more terms —and return different results — purely because of how the corpus happened to be
partitioned (e.g. after a tail-heavy build splits its leftovers).
What
InvertedIndex::bm25_search,under the query-wide
max_expansionsbudget. For each query token theper-partition FST candidates merge into one lexicographically ordered set
and the remaining budget takes a prefix of it. Each partition is only asked
for its
remaininglexicographically smallest candidates, which is losslessfor that selection: any term among the merged lex-smallest
remainingisalso among its own partition's smallest
remaining.(
load_posting_listsdrops itsexpand_fuzzycall);params.fuzzinessstill drives the grouped fuzzy dedup/scoring semantics.
bm25_scorer_for_final_tokens),so scorer terms and searched terms stay in lockstep and a query pays for one
expansion instead of one for the scorer plus one per partition.
must retain at least one expansion — via an index-level check (previously
implicit in each partition's own expansion).
InvertedPartition::expand_fuzzystays for API compatibility,reimplemented on the shared per-token candidate collector; single-partition
behavior is unchanged (same FST lexicographic order, same budget fill).
Semantics
Single-partition indexes behave exactly as before. Multi-partition indexes
previously over-expanded in proportion to their partition count; they now
honor the documented cap, and the expansion (hence the result set) is a pure
function of the segment's vocabulary, independent of partition shape.
Tests
test_fuzzy_expansion_cap_is_global_across_partitions: two partitions withdisjoint variants, binding cap → exactly the 3 lexicographically smallest
terms across both (fails on main, which returns 4).
test_fuzzy_results_independent_of_partition_shape: the same four docsbuilt as one partition and as two return identical
(row_id, score)setsunder a binding cap (fails on main).
unchanged and passing. The non-binding-cap regression test over the real
tail-split builder path lives in fix(fts): split tail-partition merge by the worker memory budget #7601.
🤖 Generated with Claude Code