feat(fts): impact skip data for posting lists#7602
Draft
BubbleCal wants to merge 2 commits into
Draft
Conversation
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
This was referenced Jul 3, 2026
Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that go stale as index stats drift: - Bounds bake once per cached list into an Arc-shared slab (max doc weight per entry plus the list-wide max); per-query clones reuse it, so query time pays one multiply per bound instead of frontier rescans. - Entry doc_up_tos decode once at construction. - Lagging iterators park in the WAND tail under the data-driven global bound (query_weight x baked list max) instead of INFINITY. 128-doc-block indexes keep fixed-width u32 impact entries.
0ec1da3 to
96b172d
Compare
Contributor
Author
|
Rebased on the updated #7466 (which now owns all V3 breaking changes, including quantized doc-length scoring). This PR's impact bounds are baked against the same quantized lengths for the Varint (256-doc) format — quantization is monotone, so the stored (freq, doc_len) pareto frontier still dominates and the bounds stay valid upper bounds of quantized scores. No breaking change here. |
This was referenced Jul 5, 2026
…nto yang/lan2-88-impact-skip-data The base's shift/mask fix for the posting-iterator hot loops conflicts textually with this branch's posting_block_idx/offset/start helpers, which already specialize 128/256 to shifts and masks. Keep the helper form everywhere in wand.rs and drop the base's now-unused block_shift/block_mask methods; the power-of-two debug_assert stays.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7466 (configurable posting block size).
Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting
blocks, plus one level1 entry per 32 blocks, and drive block-max WAND
pruning from them instead of build-time scores that go stale as index
stats drift:
128-doc-block indexes keep fixed-width u32 entries.
per entry plus the list-wide max); per-query clones reuse the slab, so
query time pays one multiply per bound instead of frontier rescans.
bound (query_weight x baked list max) instead of INFINITY.
Impacts shrink from 3.8 to ~0.6 bits/posting with the varint encoding.
Measured vs #7466 (its base)
Per-branch-tip wheels, 200M-doc v3-256 index (24 partitions), 1000 queries
× 8 concurrent, warm. Without impacts a 256-block index prunes on the
(valid but loose) list-level max that #7466 falls back to; the impact skip
data restores block-granular bounds on the same classic query loops:
🤖 Generated with Claude Code