[MayCXC]

refs #266

Adds Maximal Marginal Relevance reranking to vec0 KNN queries. When mmr_lambda is provided, candidates are over-fetched and then greedily re-selected to balance relevance against diversity.

create virtual table vec_items using vec0(
  embedding float[4] distance_metric=cosine
);

insert into vec_items(rowid, embedding) values
  (1, '[1,0,0,0]'),
  (2, '[0.99,0.1,0,0]'),
  (3, '[0.98,0.2,0,0]'),
  (4, '[0,1,0,0]'),
  (5, '[0,0,1,0]');

-- Standard KNN: returns 1, 2, 3 (clustered near [1,0,0,0])
select rowid, distance from vec_items
where embedding match '[1,0,0,0]' and k = 3;

/*
┌───────┬──────────┐
│ rowid │ distance │
├───────┼──────────┤
│ 1     │ 0.0      │
│ 2     │ 0.005... │
│ 3     │ 0.020... │
└───────┴──────────┘
*/

-- MMR: returns 1, then diverse picks instead of near-duplicates
select rowid, distance from vec_items
where embedding match '[1,0,0,0]' and k = 3 and mmr_lambda = 0.5;

/*
┌───────┬──────────┐
│ rowid │ distance │
├───────┼──────────┤
│ 1     │ 0.0      │
│ 5     │ 1.0      │
│ 4     │ 1.0      │
└───────┴──────────┘
*/

• mmr_lambda range [0.0, 1.0]: 1.0 = pure relevance (same as no MMR), 0.0 = pure diversity
• Over-fetches min(k * 5, 4096) candidates, greedy-selects k using MMR score
• distance column reports original distance to query vector, not MMR score
• Supports float32, int8, and bit vectors; all distance metrics (L2, cosine, L1, hamming)
• Composes with distance constraints (vec0: Support constraints on distance column for pagination/custom thresholds #165 / WIP: Support constaints on distance column in KNN queries, for pagination and range queries #166) and partition keys
• Zero impact when mmr_lambda is not provided

Tests

9 pytest + syrupy snapshot tests covering:

• Cosine diversity (baseline vs lambda=1.0, 0.5, 0.0)
• L2 distance metric
• Int8 vector element type
• Cluster monopoly breaking
• Composition with distance constraints and partition keys
• Edge cases (k=1, k=0)
• Error handling (invalid lambda range)
• Insert guard for hidden column

All 158 existing tests pass (+ 8 skipped). Verified in both aarch64 and x86_64 VMs.

[MayCXC]

Pushed a073842, which keeps each candidate's similarity to the selection as a running maximum. The rerank recomputed every remaining candidate's distance to every result already selected at each step, about 285,000 distance computations for k = 50 over 250 candidates. It now raises each candidate's maximum against the newest selection alone, about 12,500.

The picks are unchanged: 20 queries over 5,000 cosine vectors of 768 dimensions, with k from 20 to 100 and mmr_lambda from 0.3 to 0.7, return identical rowids before and after, in 410 ms against 9.8 s in total. The MMR snapshot tests pass unchanged (79 passed, 4 skipped, 205 snapshots on this branch).

This branch also conflicts with main now. I can rebase it if you'd like to take it.

[MayCXC]

Rebased onto main (v0.1.10-alpha.4), which conflicted with the branch. The mmr_lambda hidden column now follows main's command column where a table has one, so vec0_column_mmr_lambda_idx() counts it; the other four commits are unchanged, the running-maximum commit included as f8e0ff5. The Python suite passes on the rebased head (248 passed, 66 skipped, 219 snapshots). The branch merges cleanly with #327, #328, #330, #331 and #336 through #357. With #329 it conflicts on one line, the table declaration both extend (distance real hidden there, mmr_lambda hidden here), which keeps both whichever lands second.