[MayCXC]

Ports asg017's rescore index (asg017#276), the first of its approximate-nearest-neighbour indexes, as asg017's main has it at 04d28bd. It is rebased onto main at f6a3244, which has #7.

What it adds. A float vector column can be declared indexed by rescore(quantizer=bit) or indexed by rescore(quantizer=int8), with oversample=N from 1 to 128 (8 by default). It also keeps each vector quantized, 1 bit or 8 bits per dimension, in _rescore_chunksNN, and its float vectors by rowid in _rescore_vectorsNN. A KNN query scans the quantized vectors for the k * oversample nearest rows, then ranks those candidates by the distances of their float vectors, so the distances it returns are exact:

create virtual table v using vec0(emb float[768] indexed by rescore(quantizer=bit));
select rowid, distance from v where emb match ? and k = 10;
insert into v(v) values ('oversample=16');  -- for this connection's queries

A full scan reads a rescore column's vectors row by row through vec0_get_vector_data(), and a flat column's by chunk as main does, since only a flat column keeps its vectors in _vector_chunksNN.

The first commit adds indexed by flat(), the exhaustive search a column without the clause gets, with the scanner's (, ) and , tokens (asg017@0de765f). SQLITE_VEC_ENABLE_RESCORE=0 builds without rescore, as upstream.

What it ports. asg017@ba0db0b with the fixes 5e4c557, 82f4eb0 and 5522e86; auxiliary columns (b7fc459); the oversample=N command and its unknown vec0 command message (6e2c4c6); the int8 quantizer of 575371d, which maps [-1, 1] onto [-128, 127]; and the distance constraints of the unmerged asg017#309.

Where it differs from upstream.

• rescore_knn() takes the chunk scan's arguments and results, so vec0Filter_knn() picks between the two and the MMR rerank runs after either: mmr_lambda works on rescore columns. Its errors go through vec0_error_rc(), so an interrupted query reports interrupted. It picks each chunk's candidates with this fork's min_idx(), whose heap (Select each chunk's k nearest rows with a heap #2) needs no bitmap of taken slots, as asg017#337 does for upstream's rescore KNN.
• A distance constraint whose value is NULL, TEXT or BLOB acts as it does for a flat column.
• optimize, which upstream lacks, writes a moved row's quantized vectors to its new slot, keeps its row of _rescore_vectorsNN, whose rowid does not change, and deletes the old _rescore_chunksNN rows with the old chunks. With Port the rest of asg017#268: delete a chunk once DELETE empties it #7, a DELETE that empties a chunk deletes its _rescore_chunksNN row too.
• A point query for a rowid the table does not have returns no row. Upstream fails:

  create virtual table t using vec0(emb float[8] indexed by rescore(quantizer=bit));
  insert into t(rowid, emb) values (1, '[1,1,1,1,1,1,1,1]');
  select rowid from t where rowid = 999;
  -- asg017 main: Could not fetch vector data for 999 from rescore vectors

• _rescore_chunksNN declares rowid INTEGER PRIMARY KEY, as this fork's new tables do, and oversample=N takes only digits, where upstream's atoi() reads oversample=8x as 8.

The table names are upstream's, so files stay compatible; ARCHITECTURE.md lists them with the other backing tables. As upstream, a table with a rescore column refuses metadata and partition key columns. Distance constraints apply to the exact distances of the k * oversample candidates, so distance > x can return fewer than k rows when most candidates are nearer; the README's new section says so, since it describes cursor-based pagination just above.

Tests.

• Upstream's test-rescore.py, with Apply distance constraints in rescore KNN queries (fixes #308) asg017/sqlite-vec#309's two distance constraint tests, and test-rescore-mutations.py; the rescore cases of its test-auxiliary.py and test-rename.py; its seven rescore fuzz targets.
• test-unit.c: upstream's bit quantizer, byte size and parser cases, the parser cases run strict and lenient. Upstream's int8 cases test the min/max quantizer that 575371d replaced, so the int8 cases here test the current mapping (-1 to -128, 0 to 0, 1 to 126, clamping, NaN to 127, never decreasing), and rescore-quantize-edge.c's uniform-vector case checks that one value quantizes to one value. Its modes 1 and 2 copy the fuzz bytes out to float alignment before reading them as floats, as asg017#343 has them; upstream's read them as floats starting at the byte after the mode byte, offset 1 of the fuzzer's buffer, and UBSan reports that misaligned load on the first input of either mode.
• New: every row's slot in _rescore_chunks00 holds the quantized form of its vector through inserts, updates, deletes that empty chunks, and optimize (test_quantized_slots_follow_writes, which a corrupted slot fails); a KNN with an oversample of 1, where the quantized vectors decide the candidates, returns the same rows before and after optimize; MMR; non-numeric distance constraints; the missing-rowid point query; the fuzz target rescore-optimize.c.
• test-fullscan.py reads a rescore column beside a flat one, and runs its deleted-row and optimize-during-the-scan cases on a rescore column too. Read through the chunk handle instead, a rescore column fails all three with Could not fetch vector data.
• The last commit adds a rescore table to test_interrupt_is_reported_as_interrupt: a KNN, a full scan, a point query, an insert, an update, a delete, a delete that empties a chunk, and optimize, each interrupted at every progress handler call. A build whose rescore insert returns SQLITE_ERROR in place of vec0_error_rc()'s code fails it.

make test-loadable (559 passed, 3 skipped), make test-unit and make test pass, and so does the Python suite against an AddressSanitizer and UndefinedBehaviorSanitizer build. The build has no warnings, clang-format 21 leaves sqlite-vec.c unchanged, and each of the eight rescore fuzz targets ran 60 seconds without a finding.

git diff --histogram shows the sqlite-vec.c change as the insertion it is; the default algorithm pairs lines of vec0Filter_knn() with look-alike lines of the new code above it.

[mceachen]

Thanks for porting the rescore index. We'd like to land #2 through #7 first and then review this one on its own, since it builds on #7 and adds more code than the rest of them combined. Please leave it open; we'll start on it once #7 is on main.