Skip to content

Reset the cached DiskANN and IVF lookups once their row is read - #347

Open
MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:ann-reset-read-statements
Open

MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:ann-reset-read-statements

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

VACUUM fails after a read-only query on a DiskANN column, and after a point query or full scan on an IVF column.

create virtual table v using vec0(a float[8] indexed by diskann(neighbor_quantizer=binary));
-- 10 inserts, then a commit
select rowid from v where a match :q and k = 3;
vacuum;  -- cannot VACUUM - SQL statements in progress

diskann_node_read() and diskann_vector_read() reset their cached statements before each use, step them to the row they read, copy the row out, and return with the statement still on that row, so it stays in progress. fe94171 (#295) finalizes the cached statements in xSync, which a read-only query never reaches: a KNN query, a point query or a full scan of a DiskANN column leaves one in progress until the connection's next commit, and VACUUM refuses to run. IVF's rowid-map lookup does the same in ivf_get_vector_data() and ivf_delete(), so an IVF point query or full scan leaves it in progress; an IVF KNN query does not.

Each of these lookups now resets its statement as soon as its row is copied out, on every path, including the errors after the step. The IVF cell lookup in ivf_cell_find_or_create() is left as it is: it runs inside an insert, whose commit finalizes it, and #313 rewrites that function with the same reset.

Built with the IVF index, on main, VACUUM fails after each of the five reads (a DiskANN KNN query, point query and full scan; an IVF point query and full scan) and runs after rescore and flat reads; here it runs after all of them. Tests: test_vacuum_after_diskann_read in test-cache-finalize.py, parametrized over the three DiskANN reads, and test_ivf_vacuum_after_read in test-ivf.py over the two IVF reads. All five fail on main and pass here. The Python suite passes (242 passed, 68 skipped; with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, 306 passed, 4 skipped), and both builds' warnings are main's.

It merges cleanly with every open PR that touches the DiskANN or IVF code, #313, #344, #345 and #346 included.

Origin. Found in the DiskANN and IVF ports to photostructure/sqlite-vec, photostructure#11 (cd53413) and photostructure#12 (65817f4), which list it among the ports' differences from upstream; the change and its tests are written here for asg017's main. #19 was the same failure on a flat column, which 7fc8248 fixed by finalizing vec0's statements before each commit, and #295 asked for the DiskANN statements to be released without a disconnect. No open or closed PR or issue here makes this change.

diskann_node_read(), diskann_vector_read() and IVF's rowid-map lookup
reset their cached statements before each use and left them on the row
they had read. fe94171 finalizes them at the next commit, but a
read-only query has none, so after any DiskANN KNN query, point query or
full scan, or an IVF point query or full scan, the connection kept a
statement in progress and VACUUM failed with "cannot VACUUM - SQL
statements in progress". Each of these lookups now resets its statement
as soon as its row is copied out, on every path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant