Improve component search scoring relevance

[Mbeaulne]

Description

Improves the lexical search scoring model with three enhancements:

• Prefix match boost: Partial query terms (e.g. classif) now rank components where the term is a prefix of a token higher than components where it appears only as a mid-string substring.
• IDF-style rare token weighting: Query tokens that match fewer components are weighted more heavily than common tokens, preventing high-frequency terms from dominating scores. For example, searching train xgboost will surface components mentioning xgboost above generic train matches.
• All-query-tokens bonus: When a component matches every token in the query (across any fields), it receives an additional score bonus, ensuring more complete matches rank above partial ones.

The phrase match bonus previously applied only to the name field has been extended to all search fields using per-field bonus weights (FIELD_PHRASE_BONUS).

The tokenize function has been refactored to extract a reusable uniqueTokens helper, and a new requiredQueryTokens function produces stemmed, deduplicated tokens from the raw query without synonym expansion, used for phrase and completeness checks.

Related Issue and Pull requests

Type of Change

• Bug fix
• Improvement
• Cleanup/Refactor
• Breaking change
• Documentation update

Checklist

• I have tested this does not break current pipelines / runs functionality
• I have tested the changes on staging

Test Instructions

Three new unit tests cover the added behaviors:

1. Search classif — verify classify_rows ranks above a component with classif as a non-prefix substring.
2. Search train xgboost — verify the component with the rare token xgboost ranks first.
3. Search train model — verify the component matching both tokens across fields ranks above one matching only train.

Run the test suite with:

npx jest componentSearchIndex

Additional Comments

Token weights are computed per-query using a smoothed inverse document frequency: 1 + log((N+1) / (df+1)), where N is the index size and df is the number of entries containing the token.

[github-actions]

🎩 Preview

A preview build has been created at: 06-18-improve_component_search_scoring_relevance/afd8b04

[Mbeaulne]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Add AI model quick-select to app menu and injectable model config #2461
• Simplify component search result cards #2459
• Add context-aware component search suggestions #2458
• Add component discovery docs #2457
• Add component search analytics baseline #2456
• Limit dashboard component browse rendering #2454
• Add component search empty state suggestions #2453
• Decouple editor component search input #2452
• Polish component search explanations #2451
• Add compatible component suggestions #2450
• Add component collection search results #2449
• Add component lifecycle badges #2448
• Add component search URL state #2447
• Disable AI search when literal search finds no matches #2444
• Debounce component search input #2433
• add client-side embeddings cached in IndexedDB #2432
• Add search quality expectation tests for lexical search #2431
• Add deep AI search to rerank all components in selected sources #2430
• Add source-diverse AI rerank candidate pool #2429
• Add negative constraint parsing to lexical search #2428
• Add fuzzy/typo-tolerant matching for component name and I/O fields #2427
• Improve component search scoring relevance #2426 👈 (View in Graphite)
• Add synonym expansion to component lexical search #2425
• Normalize component search tokens for better matching #2424
• Expand component search indexing fields #2423
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[camielvs]

🤖 Code review — Improve component search scoring relevance

This is the strongest PR in the stack so far. Four well-chosen relevance signals — word-boundary prefix bonus, IDF rare-token weighting, all-query-tokens bonus, and per-field phrase bonuses — and the tests are genuinely discriminating: each one is constructed so it fails if the specific signal is removed (e.g. the alphabetical tie-break deliberately favors the wrong candidate so only the bonus can flip it). The inline comments explaining that are exactly what a reviewer wants. The per-entry fieldTokensFor cache to avoid re-splitting field text inside the hot per-token loop is a nice touch.

Findings:

• IDF down-weights common tokens but doesn't resolve the synonym/stem stacking from Normalize component search tokens for better matching #2424–Add synonym expansion to component lexical search #2425. Scoring is still Σ over tokens of fieldWeight × tokenWeight, and tokens is the stem+synonym-expanded set. So a component containing several members of one synonym group (or both the inflected and stemmed form of a word) still accumulates a contribution per surface variant — IDF only scales each, it doesn't collapse them to one concept. If you want one logical match to count once, dedupe by group/stem before the per-token loop. IDF genuinely helps (common expansions get ~1.0 weight), so this is lower-stakes than before, but the additive stacking is still there.

• Math.max(0, inverseFrequency) is dead code. documentFrequency ≤ index.length always, so (N+1)/(df+1) ≥ 1 and log(...) ≥ 0. The clamp can never fire. Harmless, but either drop it or add a comment if it's guarding against a future change.

• Per-keystroke cost is now ~2× a full index pass. buildRareTokenWeights scans the whole index once per query token, then scoreEntry scans again, and the all-tokens bonus re-runs entryMatchesToken per entry. lexicalSearch runs in the DashboardComponentsV2View render path on every query change (debounce only lands in Debounce component search input #2433), and synonym expansion multiplies the token count. Fine for hundreds of components; worth keeping in mind as libraries grow or if entryMatchesToken's substring scans get hotter. Minor: index.filter(...).length allocates an array just to count — a plain counter avoids it.

• Inflected multi-word phrase bonus still won't fire (carryover from Add synonym expansion to component lexical search #2425). The index normalization interleaves [inflected, stem] (training_testing → "... training train testing test"), so the stemmed requiredTokens join "train test" is never contiguous in the index. The common non-inflected case (train test split → train_test_split) works. Low priority.

Routing the phrase/all-token bonuses through the synonym-free, stemmed requiredTokens (rather than the expanded tokens) is the correct separation — good call.