Skip to content

feat(index)!: quantized doc-length scoring and slimmer posting blocks for FTS v3#7626

Closed
BubbleCal wants to merge 6 commits into
yang/lan2-88-fts-simd-merge-kernelsfrom
yang/lan2-88-fts-v3-quantized-norms
Closed

feat(index)!: quantized doc-length scoring and slimmer posting blocks for FTS v3#7626
BubbleCal wants to merge 6 commits into
yang/lan2-88-fts-simd-merge-kernelsfrom
yang/lan2-88-fts-v3-quantized-norms

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Stacked on #7625. Format/scoring change for V3 only — covered by the format discussion #7606 (spec updated there).

What

Two changes scoped to block_size=256 (V3); 128-block indexes are untouched on disk and keep exact scoring bit-for-bit:

  • Quantized doc-length scoring (Lucene norm semantics). BM25 doc lengths on V3 partitions are quantized to a Lucene SmallFloat-style byte code (4 mantissa bits: 0-7 exact, <= 6.25% relative error, decode = bucket floor). The byte-norm slab bakes lazily per loaded DocSet (gated at partition load by block size, threaded through LazyDocSet), quartering the doc-length working set the scoring gather pulls through the cache (200M docs: 800MB u32 -> 200MB u8). Impact bounds bake against the same quantized lengths; quantization is monotone, so the stored (freq, doc_len) pareto frontier still dominates every doc in range and WAND pruning stays lossless relative to quantized scores.
  • 256-doc blocks drop the leading block-max-score f32. V3 always carries impact skip data (feat(fts): impact skip data for posting lists #7602), which supplies a tighter per-block bound; the stored f32 was dead weight (~1.5G on the 200M benchmark index, 131G -> 130G). Block layout becomes [first_doc u32][doc num_bits u8][docs][pfor freqs]; posting_block_score_prefix_len(block_size) keys every reader/writer, and block_max_score falls back to the list-level max for prefix-less blocks.

Breaking / behavior notes

  • BREAKING: existing V3 (block_size=256) indexes must be rebuilt; V3 is unreleased, so no migration is provided. V1/V2 (128) untouched.
  • V3 BM25 scores differ from exact-length BM25 by the norm quantization — the same trade Lucene has always made. Measured on the (score-clustered, synthetic) mmlb corpus: top-k overlap vs exact scoring = 98.1% mean for phrase, 89.7% mean for 3-word AND; real corpora with more score spread shift less.
  • Merge note: perf(fts): cache the num_tokens-only DocSet on LazyDocSet #7600 adds a cached tokens-only DocSet constructor in lazy_docset.rs; whichever lands second must apply set_quantized_scoring inside that cached constructor too (the integration branch yang/fts-lucene-parity-integration shows the resolution).

Results (mmlb warm, 8 concurrent)

query #7625 this PR
AND k10 @200M 0.049s / 161 qps 0.0456s / 174 qps
AND k100 @200M 0.098s / 82 qps 0.0916s / 87 qps
phrase 2w k10 @50m 0.040s 0.039s

Verification

  • Bulk-vs-classic A/B under quantized scoring: score_diff=0 (both paths quantize identically).
  • Quantized-vs-exact recall characterized against a pre-change baseline (numbers above); every step's benchmark indexes rebuilt with the new layout.
  • Roundtrip/layout tests updated (256 block offsets, impact bound quantization); cargo test -p lance-index, clippy -D warnings, fmt --check clean.

🤖 Generated with Claude Code

Yang Cen and others added 6 commits July 3, 2026 13:10
feat(fts): impact skip data for posting lists
0ec1da3 
Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting
blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32
blocks, and drive block-max WAND pruning from them instead of build-time
scores that go stale as index stats drift:

- Bounds bake once per cached list into an Arc-shared slab (max doc weight
  per entry plus the list-wide max); per-query clones reuse it, so query
  time pays one multiply per bound instead of frontier rescans.
- Entry doc_up_tos decode once at construction.
- Lagging iterators park in the WAND tail under the data-driven global
  bound (query_weight x baked list max) instead of INFINITY.

128-doc-block indexes keep fixed-width u32 impact entries.
perf(fts): bulk MAXSCORE search path for top-k disjunctions
0781681 
Port of Lucene's MaxScoreBulkScorer, opt-in via LANCE_FTS_MAXSCORE=1: per
outer window (bounded by the essential clauses' blocks with adaptive
growth), clauses split into a non-essential prefix and essential rest by
window max score vs the running threshold. Essential clauses bulk-stream
decompressed blocks (single-essential windows stream with no accumulator);
non-essential clauses are only probed for candidates that can still beat
the threshold. Dead ranges with one live clause skip by scanning the baked
per-block bound slab. Candidate emission matches the classic path (must
beat the running threshold), so results are score-identical.

Measured on a 200M-doc warm benchmark at 24 partitions: 3-word OR match
k10 0.137s -> 0.035s, k100 0.250s -> 0.064s; hot single-term 250ms -> 3ms.
perf(fts)!: enable the bulk MAXSCORE path by default
4732787 
With right-sized partitions the bulk path reaches Lucene-parity latency
(k10 0.034-0.037s/213-235qps vs Lucene 10.4's 0.037s/216qps on the same
200M-doc corpus, queries, and warm protocol) and its results are
score-identical to the classic WAND loop. LANCE_FTS_MAXSCORE=0 opts back
into the classic loop.
@claude
perf(index): bulk fts conjunction path and per-doc position seeking
fc1af4b 
AND and phrase queries previously leapfrogged doc-at-a-time through
boxed PostingIterator::next calls (~61% of the AND profile) and phrase
checks decoded a whole 256-doc position block per candidate (~39% of
the phrase profile).

- and_bulk_search: block-max window pruning plus a k-pointer merge over
  decompressed block slices; per-candidate advance cost drops to a few
  loads. Results are identical to the classic loop (LANCE_FTS_BULK_AND=0
  opts out). Phrase queries ride the same path.
- seek_packed_doc_positions: PackedDelta full groups are self-describing
  ([num_bits u8][16*num_bits bytes]), so group offsets are recovered by
  hopping headers; decode only the 1-2 groups overlapping the candidate
  doc's delta range, with a lazily-built group index, memoized unpacked
  group, and a decoded-tail cache per block.
- check_exact_positions_bulk: allocation-free slop=0 alignment check on
  the decoded scratch slices for parked lead clauses.

Warm mmlb benchmarks, 8 concurrent queries: AND\@200M k10 0.114->0.060s,
k100 0.240->0.118s; phrase\@50m 3-word k10 0.335->0.210s, 2-word k10
0.098->0.042s. All steps verified score-identical to the classic path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude
perf(index): specialized SIMD merge kernels and freq-bound pruning fo…
2f9de34 
…r bulk fts conjunctions

Three changes to the bulk AND/phrase candidate pipeline, all
score-identical to the classic loop:

- Clause-count-specialized merge kernels (2 and 3 clauses) keep cursors
  and bounds in registers, with an AVX2 find_next_geq catch-up scan (the
  analogue of Lucene's VectorUtil.findNextGEQ) replacing the mispredict-
  heavy scalar walk. Generic kernel covers other widths.
- Two-pass batched scoring: kernels only record (doc, offsets) matches;
  doc lengths are then gathered back-to-back so their cache misses
  overlap, and the prune/score/insert pass runs in doc order with the
  classic semantics.
- A per-clause frequency-bucketed score-bound LUT lets kernels skip lead
  docs before any follower advance when even the doc-length-free bound
  cannot beat the threshold (score-first ordering, Lucene-style); the
  bound is monotone so the skipped set is a subset of the exact prune.

Warm mmlb AND @200M, 8 concurrent: k10 0.060->0.049s (161qps), k100
0.118->0.098s (81qps); phrase\@50m 3w k10 0.210->0.204s, 2w 0.042->0.040s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude
feat(index)!: quantized doc-length scoring and slimmer posting blocks…
07650a9 
… for fts v3

Two v3-only (256-doc block) changes; 128-block indexes are untouched on
disk and keep exact scoring bit-for-bit.

- BM25 doc lengths for v3 partitions are quantized to a Lucene
  SmallFloat-style byte code (4 mantissa bits, exact below 8, <= 6.25%
  relative error). The byte-norm slab bakes lazily per loaded DocSet and
  quarters the bytes the scoring gather pulls through the cache
  (200M docs: 800MB -> 200MB working set). Impact bounds bake against
  the same quantized lengths; quantization is monotone, so the stored
  (freq, doc_len) pareto frontier still dominates and bounds stay valid.
- 256-doc posting blocks drop the leading block-max-score f32: v3
  always carries impact skip data, which supplies the tight per-block
  bound (the stored one was already unused on that path). Block layout
  becomes [first_doc u32][doc num_bits u8][docs][pfor freqs]; readers
  key the prefix off the block size, and CompressedPostingList::
  block_max_score falls back to the list-level max for prefix-less
  blocks.

BREAKING: v3 (block_size=256) indexes must be rebuilt; v3 is unreleased
so no migration is provided. BM25 scores on v3 differ from exact by the
norm quantization, matching Lucene's norm semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request breaking-change labels Jul 4, 2026
@BubbleCal BubbleCal mentioned this pull request Jul 4, 2026
Closed
@BubbleCal BubbleCal force-pushed the yang/lan2-88-fts-simd-merge-kernels branch from 2f9de34 to 3c60d9a Compare July 4, 2026 18:17
@BubbleCal

BubbleCal commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Closing: per review direction, all breaking changes are consolidated into #7466. The quantized doc-length scoring + block-max-score-prefix removal moved there (commit c4b12f1 on that branch); the impact-bound quantization folded into #7602 where impact skip data lives. The rest of this stack (#7624/#7625/#7627) is rebased on the updated train and carries no breaking changes.

@BubbleCal BubbleCal closed this Jul 4, 2026
@BubbleCal BubbleCal deleted the yang/lan2-88-fts-v3-quantized-norms branch July 4, 2026 18:18
@BubbleCal BubbleCal mentioned this pull request Jul 4, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

A-index Vector index, linalg, tokenizer breaking-change enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BubbleCal