sparse-ngrams: report each query gram's index-folded bytes

[aneubeck]

Problem

QueryGrams emits only the compact NGram key. Callers that want to show a gram — tracing, snapshot tests, debugging output — cannot recover the text from it: grams longer than 3 bytes are hashed into the 24-bit payload, so Debug degrades to NGram('0x20baba', len=4).

Blackbird hit exactly this after switching its query-time extraction over to this crate: its iterator-tree snapshots went from

$[OR2[AND[*0=c:boring :20, c:art is g:40, 1=c:ing mi:20, ...

to

$[OR2[AND[*c:NGram('t w', len=3):40, 0=c:NGram('0x20baba', len=4):2, ...

Change

The state already holds the bytes: extract_gram builds the key from the packed content window. Align that window once and hand the consumer a &[u8] view of it. Consumers become FnMut(NGram, u32, Option<u8>, &[u8]):

arg	meaning
gram	the compact NGram key
end	position of the character just after the gram
follow	that following byte, when already fed
bytes	the gram's index-folded bytes, in reading order

bytes borrows a stack array that is live only for the duration of the call, so nothing is allocated per gram and the hot path is unchanged (the alignment shift was already being done inside from_window_masked).

NGram::from_window_masked is removed: aligning in extract_gram makes its masking redundant, since the left shift already drops the high bytes.

Compatibility

Breaking for the four QueryGrams emitters (append_char, append_byte, flush, consume_first). The indexing entrypoints (collect_sparse_grams*) are untouched — their callers already hold the content slice and can slice it themselves. Version bumped to 0.4.0.

Testing

emitted_follow_is_the_actual_next_char is extended (and renamed to emitted_follow_and_bytes_match_the_input) to assert, for every emitted gram, that the reported slice equals the corresponding input slice and that NGram::from_bytes(bytes) reproduces the emitted key. cargo test -p sparse-ngrams, cargo clippy --all-targets and cargo fmt --check are clean.

[Copilot]

Pull request overview

Adds index-folded source bytes to QueryGrams callbacks for readable diagnostics without per-gram allocation.

Changes:

• Extends callbacks with &[u8] and validates emitted bytes.
• Simplifies aligned-window NGram construction.
• Updates documentation and bumps version to 0.4.0.

Show a summary per file

File	Description
crates/sparse-ngrams/src/query.rs	Emits gram bytes and updates tests.
crates/sparse-ngrams/src/ngram.rs	Removes redundant masked-window helper.
crates/sparse-ngrams/README.md	Documents the callback API.
crates/sparse-ngrams/Cargo.toml	Bumps the breaking-release version.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Medium