fix(fts): use complete English stop-word list for ICU tokenizer#7621
Open
LuQQiu wants to merge 4 commits into
Open
fix(fts): use complete English stop-word list for ICU tokenizer#7621LuQQiu wants to merge 4 commits into
LuQQiu wants to merge 4 commits into
Conversation
The ICU stop-word path (StopWordFilter::all()) sourced its English list
from the local Tantivy-style stopwords::ENGLISH constant (~33 words),
which omits extremely common pronouns and function words: you, my, your,
we, she, what, can, about, us, them, him, our.
Because these are the highest-frequency tokens in English text, they
survived stop-word removal and built pathologically large single-term
posting lists. On large corpora (100M+ rows) with positions enabled, one
such term's in-memory posting list exceeded u32::MAX bytes and panicked
the whole FTS index build with 'posting list memory size overflowed u32'.
Switch the English source to stop_words::get("en") (~198 words), matching
how the other CJK/misc languages in all_stop_words() are already sourced
from the stop-words crate. The leak is independent of stemming, so the
new tests cover both stem=false and stem=true for icu and icu/split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a regression test alongside the English one confirming common
Chinese function words/particles (我 在 有 了 是 的 和) are removed by the
ICU all() stop-word path while real content words (英语) survive. The
Chinese list is sourced from stop_words::get("zh") (complete), so this
already passed; the test locks it in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
A-index
…th too
The per-language StopWordFilter::new(English) path (used by non-ICU
tokenizers such as `simple`, the recommended tokenizer for monolingual
English) also sourced the truncated local stopwords::ENGLISH (~33 words),
so common pronouns (you/my/your/we/...) leaked there as well. Switch it to
stop_words::get("en") to match the ICU all() path fix, and extend the
regression test to cover base_tokenizer=simple.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The ICU tokenizer's stop-word removal used an incomplete English stop-word list, letting the highest-frequency English pronouns/function words through the index. On large corpora this built pathologically large single-term posting lists and panicked the FTS index build with
posting list memory size overflowed u32.StopWordFilter::all()(the path used forbase_tokenizer: icu/icu/split) sourced its English words from the local Tantivy-stylestopwords::ENGLISHconstant (~33 words):This omits extremely common words:
you, my, your, we, she, what, can, about, us, them, him, our. Since these are among the highest-frequency tokens in English text, they survived stop-word removal. Withwith_position: trueon a 100M+ row dataset, a single such term's in-memory posting list exceededu32::MAXbytes (~4 GiB) and panicked the whole build.Root cause
Every other language in
all_stop_words()(Arabic, Chinese, Japanese, Korean, …) is sourced from thestop-wordscrate viastop_words::get(...), whose lists are complete. Only English used the local, truncated constant. The Chinese list (stop_words::get("zh"), ~841 words) was already complete — Chinese stop words like了/是/的were correctly removed; only English leaked.Fix
One-line change in
all_stop_words(): source English fromstop_words::get("en")(~198 words) like the other languages. This list contains the missing pronouns/function words.Tests
test_icu_common_english_stop_words_do_not_leak— assertsyou/my/your/weare removed via the ICUall()path, foricuandicu/split, withstemboth false and true (the leak is independent of stemming — it's purely the incomplete list).test_icu_common_chinese_stop_words_do_not_leak— asserts common Chinese function words (我 在 有 了 是 的 和) are removed while content words (英语) survive.All existing tokenizer tests continue to pass (20 passed, 0 failed).
Impact
Fixes FTS index builds that panicked on large English (and mixed EN+ZH) corpora, and reduces index size by correctly dropping the highest-frequency English stop words.
🤖 Generated with Claude Code