feat(fts)!: add configurable posting block size

[BubbleCal]

Feature

Linear: OSS-1344

What is the new feature?

FTS inverted index creation now accepts a block_size parameter for compressed posting blocks. Supported values are 128 and 256.

Why do we need this feature?

The posting block size was previously fixed at 128, which made the block-max granularity impossible to tune for different datasets and query profiles.

How does it work?

• Adds block_size to InvertedIndexParams, protobuf details, posting-list schema metadata, and cache headers.
• Uses 128 as the default for newly created indexes.
• Treats older serialized params, schema metadata, and cache entries that omit block_size as legacy 128.
• Rejects unsupported values, including 512, with a clear validation error.
• Uses Lance-owned BitPacker4x for physical 128-value posting blocks and BitPacker8x for physical 256-value posting blocks.
• Marks block_size=256 as experimental in public API docs because it may introduce breaking changes.
• Keeps position-stream packing on the legacy 128-value block format.
• Keeps downgrade compatibility tests on explicit legacy block_size=128, since older wheels cannot read current-created physical 256 FTS posting blocks.
• Threads the configured block size through FTS build, read, iterator, WAND, cache, and MemWAL flush paths.
• Exposes the parameter in Python and Java FTS index creation APIs, with docs and focused tests.

Validation

• cargo fmt --all
• cargo fmt --all --check
• git diff --check
• CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo test -p lance-index block_size -- --nocapture
• CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo clippy -p lance-index --tests -- -D warnings
• uv run make build from python/
• uv run pytest python/tests/test_scalar_index.py::test_create_scalar_index_fts_block_size from python/
• uv run ruff format --check python/tests/test_scalar_index.py python/lance/dataset.py from python/
• uv run ruff check python/tests/test_scalar_index.py python/lance/dataset.py from python/
• CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index block_size -- --nocapture
• CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index test_256_posting_block_uses_single_physical_bitpack_chunk -- --nocapture
• CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-bitpacking
• CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo clippy -p lance-bitpacking -p lance-index --tests -- -D warnings
• uv run ruff format --check python/tests/compat/test_scalar_indices.py from python/
• uv run ruff check python/tests/compat/test_scalar_indices.py from python/
• uv run pytest --run-compat -vvv -s python/tests/compat/test_scalar_indices.py::test_FtsIndex_downgrade --durations=30 from python/
• CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index test_new_training_request_defaults_missing_block_size_to_128
• CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index block_size
• uv run ruff format --check python/lance/dataset.py from python/
• uv run ruff check python/lance/dataset.py from python/

Not run locally: Java focused test / spotless check, because this machine has no Java Runtime installed (Unable to locate a Java Runtime).

---

Update: all V3 breaking changes consolidated here

Per review direction, every breaking change for the 256-doc block format now lands in this single PR (the follow-up stack #7602/#7603/#7604/#7624/#7625/#7629 carries none). On top of the configurable block size and PFOR frequency encoding, this PR now also includes:

• Quantized doc-length scoring (Lucene norm semantics), 256-doc blocks only. BM25 doc lengths are quantized to a SmallFloat-style byte code (4 mantissa bits: 0-7 exact, <= 6.25% relative error, decode = bucket floor). The byte-norm slab bakes lazily per loaded DocSet and quarters the doc-length bytes scoring pulls through the cache (200M docs: 800MB -> 200MB). 128-block indexes keep exact-length scoring bit-for-bit. Measured top-k overlap vs exact scoring on the (score-clustered, synthetic) mmlb corpus: 98.1% mean for phrase, 89.7% for 3-word AND; corpora with more score spread shift less.
• 256-doc posting blocks drop the leading block-max-score f32 (~1.5G on a 200M-doc index; 131G -> 130G). Block layout: [first_doc u32][doc num_bits u8][docs][pfor freqs]; posting_block_score_prefix_len(block_size) keys every reader/writer. The impact skip data from the stacked feat(fts): impact skip data for posting lists #7602 supplies a tighter per-block bound; until it lands, 256-block block-max pruning falls back to the (valid, looser) list-level max score.

BREAKING: 256-doc-block (v3) indexes must be rebuilt; v3 is unreleased so no migration is provided. BM25 scores on v3 differ from exact-length BM25 by the norm quantization, matching Lucene's norm semantics. The format discussion #7606 documents the final layout and scoring semantics.

Additional validation for this update: bulk-vs-classic A/B under quantized scoring is score-identical (both paths quantize identically); the full stack's warm benchmarks vs Lucene 10.4 on mmlb-200m: OR k10 0.0249s/318qps and OR k100 0.0467s/170qps (both ahead of Lucene sliced), AND k10 0.0443s (1.29x), AND k100 0.0883s (1.94x).

---

Standalone results vs main (per-branch-tip wheels)

Legacy (128) read-path parity. Threading a runtime block_size through
PostingIterator initially replaced the compile-time BLOCK_SIZE division
(a shift) with real div instructions in the doc()/next() hot loops,
measured as +11-14% on 3-word OR against the 200M legacy index
(PostingIterator::next grew from 16.5% to 23.6% of the profile). Block
sizes are validated powers of two, so the iterator now derives block indices
with trailing_zeros shifts and masks; after that fix the legacy path is at
parity with main: 3-word OR k10 0.131s (main 0.132-0.134s), k100 0.255-0.256s (main 0.256-0.257s), single-term 0.025s (main 0.027s) across 3 warm passes on the 200M legacy index, 400G cache.

block_size=256 index size (5M-doc controlled build, same wheel,
with_position=false): postings shrink 3.33 GiB → 2.62 GiB (−21%) from
PFOR frequencies + no per-block max-score prefix + half the block headers.
Index build 192s → 210s (+10%, PFOR encode cost). Top-10 overlap vs the
128 exact-length scoring on 10 3-word OR queries: 95% mean (5/10 identical
sets; the rest differ by 1-2 near-tie docs, from the quantized-norm
scoring).

Query wins for 256 land in the stacked PRs. A 256 index without impact
skip data prunes on the (valid, looser) list-level max and is slower than
128 — e.g. classic AND k10 0.547s until #7602's impacts restore
block-granular bounds (0.115s), and #7603/#7604/#7624/#7625/#7629 take the
same index to 0.025s OR k10 / 0.045s AND k10.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

❌ Patch coverage is 86.05016% with 178 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/index.rs	83.36%	55 Missing and 24 partials ⚠️
rust/lance-index/src/scalar/inverted/encoding.rs	85.62%	25 Missing and 19 partials ⚠️
rust/lance-index/src/scalar/inverted/builder.rs	81.25%	23 Missing and 4 partials ⚠️
rust/lance-index/src/scalar/inverted/tokenizer.rs	92.85%	4 Missing and 6 partials ⚠️
rust/lance/src/dataset/mem_wal/index.rs	89.70%	7 Missing ⚠️
rust/lance/src/dataset/mem_wal/index/fts.rs	83.72%	5 Missing and 2 partials ⚠️
rust/lance-index/src/scalar/inverted.rs	92.85%	0 Missing and 2 partials ⚠️
...ust/lance-index/src/scalar/inverted/cache_codec.rs	80.00%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[BubbleCal]

@claude reivew

[Xuanwo]

I think this needs a compatibility boundary before merge.

1. block_size=256 changes the persisted posting-block layout, but the stored index version still looks like the existing FTS format. Older readers will ignore the new metadata/details field and try to decode the blocks as legacy 128-doc BitPacker4x blocks. That can fail open as wrong FTS results or decode panics instead of cleanly ignoring the index. We should either bump the FTS/index version for non-legacy block sizes or reject writing 256 until older readers can be gated out.

2. Legacy segments with no block_size and newly written default-128 segments with block_size=128 are semantically identical, but the multi-segment details check compares the raw protobuf values. Mixed old/new default-128 segments can be rejected as inconsistent. The comparison should canonicalize missing block_size to 128 before comparing.

[BubbleCal]

I think this needs a compatibility boundary before merge.

1. block_size=256 changes the persisted posting-block layout, but the stored index version still looks like the existing FTS format. Older readers will ignore the new metadata/details field and try to decode the blocks as legacy 128-doc BitPacker4x blocks. That can fail open as wrong FTS results or decode panics instead of cleanly ignoring the index. We should either bump the FTS/index version for non-legacy block sizes or reject writing 256 until older readers can be gated out.
2. Legacy segments with no block_size and newly written default-128 segments with block_size=128 are semantically identical, but the multi-segment details check compares the raw protobuf values. Mixed old/new default-128 segments can be rejected as inconsistent. The comparison should canonicalize missing block_size to 128 before comparing.

bumped block_size=256 to version=3
keep 128 the default

[Xuanwo]

Thanks for pushing the format-version boundary for block_size=256; that addresses the old-reader compatibility concern.

I think there is still one blocking invariant gap in the MemWAL FTS path.

MemIndexConfig::fts_from_metadata can construct InvertedIndexParams with format_version=V3 and the legacy/default block_size=128 when IndexMetadata.index_version == 3 but index_details are missing, invalid, or carry default params. FtsMemIndex::with_params / FtsIndexConfig::with_params also accept params without calling validate_format_version().

Later, MemWAL flush calls InnerBuilder::new_with_format_version_and_block_size(format_version, block_size), whose validation is currently an expect(...), so this invalid public/metadata state becomes a panic instead of a Result error.

Could we validate the format/block-size invariant at the MemWAL config/metadata boundary and add a regression test for index_version=3 + missing/default block_size? The expected behavior should be a clear error, not a panic. A positive MemWAL roundtrip for block_size=256 would also lock in the intended path.

[BubbleCal]

Format-change discussion / PMC vote thread for the V3 (256-doc block + patched-FOR frequency) format introduced here: #7606