[MayCXC]

A KNN query on a DiskANN column ignores a rowid in (...) filter and returns the unfiltered k nearest rows.

create virtual table v using vec0(emb float[8] indexed by diskann(neighbor_quantizer=binary, n_neighbors=8));
-- rows 1 to 20; :q is row 3's vector
select rowid, distance from v where emb match :q and k = 3 and rowid in (1, 2);
-- 3 (0.0), 14 (0.0), 17 (1.034); a flat column returns 1 (1.349), 2 (1.507)

vec0BestIndex() passes the filter to xFilter as an sqlite3_vtab_in() list and sets omit, so SQLite does not check it again, but vec0Filter_knn_diskann() reads only the MATCH and k entries of idxStr. A rowid filter now ranks the rows it names by their exact distances, read from _vectorsNN, which holds the buffered rows as well as the graph's, and keeps the k nearest. A rowid or id the table does not have is skipped, as SQL's in does (a flat column returns no rows at all when a text-id filter names one). The graph search and the buffer scan run only without a filter, since the graph search reaches only the rows near the query, which the filter may not name.

Tests in test-diskann.py: test_diskann_knn_rowid_in compares rowids and distances with a flat column built from the same vectors, for filters naming graph rows, buffered rows, a duplicated rowid and a missing one, on a table with and without a buffer; test_diskann_knn_id_in_text_primary_key filters a text primary key with an id the table does not have. All three fail on main and pass here. The Python suite passes (242 passed, 66 skipped), under ASan and UBSan as well, and the build's warnings are main's.

The distance half of the same gap is #316, which applies distance constraints in every backend. The two merge cleanly, and together a distance constraint applies to the k rows the filter ranked, as it does to the graph's, so rowid in (...) and distance > x can return fewer than k of the named rows where a flat column returns k. This also merges cleanly with every open PR that touches the DiskANN code, #311, #313, #342, #344, #345, #346 and #347 included.

Origin. Found in the DiskANN port to photostructure/sqlite-vec, photostructure#11 (cd53413), whose diskann_knn() takes the rowid filter and distance constraints of the rescore index's KNN; the change and its tests are written here for asg017's main. Skipping an id the table does not have is the rule mceachen's photostructure/sqlite-vec@07eb685 gives that fork's flat filter, and mceachen's photostructure/sqlite-vec@1eb6348 applies a one-value rowid = ? or rowid in (?) there before the k, which a DiskANN column here still applies after it, as a flat column does. No open or closed PR or issue here makes any of these changes.