Repository navigation
Conversation
ivf_is_trained() kept the trained state per connection once it had read it, and compute-centroids, set-centroid and clear-centroids set that cache as they ran. Another connection's command, or a rolled-back one, left it wrong: a connection that had queried before another trained the index scanned only the emptied unassigned cells and found no rows, and one that had trained it, then saw another connection clear it or rolled its own training back, failed every insert and query with "SQL logic error". The state is now read from _info each time, so it is always that of the transaction reading it.
The IVF commands, and the cell writes behind every insert, ignored the result of most of their statements, so one that failed part way, an interrupt included, left the command or insert to carry on and report success over a half-rebuilt index. compute-centroids and clear-centroids read every vector through ivf_load_all_vectors(), which stopped at a failed step and returned what it had read as all of them, then deleted the cells and rebuilt them from that part, and compute-centroids its _ivf_vectors too. assign-vectors deleted the unassigned cells whatever happened to the vectors it was moving out of them, and ivf_cell_insert() wrote a vector, its rowid and its map entry without checking any of them. Where a failure was reported, it was as "SQL logic error" or "No vectors". compute-centroids also wrapped its work in a SAVEPOINT that never opened, since none can while the INSERT running the command is in progress. Every statement and blob access in the commands, ivf_exec(), ivf_is_trained(), ivf_load_all_vectors() and ivf_cell_insert() now returns its error, and the command or insert returns it to SQLite, which undoes the statement's writes. The savepoint is gone.
An error the command returns is rolled back with the transaction in autocommit mode, or inside BEGIN when it is an interrupt or a memory, I/O or full-disk error. A constraint or other error inside BEGIN leaves the writes before it, since a single-row INSERT into a virtual table opens no statement journal.
ivf_delete(), ivf_get_vector_data() and the KNN scan ignored the result of most of their steps, so a delete or a query whose statement failed part way, an interrupt included, carried on and reported success over what it had not done: - ivf_get_vector_data() read a failed step of its rowid-map lookup as a missing row. The point query a DELETE runs to find its row then found none, and the delete reported success with the row still in the table and the index. - ivf_delete() read a failed lookup the same way, and cleared the slot's validity bit, decremented n_vectors and deleted the rowid-map and _ivf_vectors rows without checking any of them, so a delete could report success with the row gone from the table but still in the index. - The KNN query stopped reading centroids or cells at a failed step and ranked what it had read, returning fewer rows than it should as if they were all, or none. With oversample, a failed read of a candidate's float32 vector kept its quantized distance. Each of those statements and blob accesses now returns its error, and the delete or query returns it to SQLite. Interrupting one statement at each VM step of a delete of one of 20 rows, before this change 32 of 123 runs reported success and left the row in the table and the index (21) or in the index alone (11), and 35 of 133 with the int8 quantizer; after it none do. Of a KNN query for all 20 rows on a trained index, 52 of 167 runs reported success with no rows or 11, and 8 failed with "SQL logic error"; with the int8 quantizer and an oversample of 4, 160 more of 327 reported success with 20 rows or distances other than the uninterrupted query's. After it, every interrupted run reports "interrupted" and the run that finishes returns all 20 rows and their distances. test_ivf_delete_interrupted_at_each_step and test_ivf_knn_interrupted_at_each_step check that for every step, on an untrained and a trained index, without a quantizer and with int8, and the KNN test with an oversample of 4 too. A failed delete may still have taken effect when the interrupt comes after the row is gone, as it can for any SQLite table, so the delete test asks that the table and the index agree, and that a delete reporting success removed the row.
This was referenced Oct 23, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a step inside an IVF delete, point read or KNN query fails, an interrupt included, the statement carries on and can report success over what it has not done:
What passes over a failed step:
ivf_get_vector_data()reads a failed step of its rowid-map lookup as a missing row. The point query a DELETE runs to find its row then finds none, and the delete reports success with the row still in the table and the index.ivf_delete()reads a failed lookup the same way, and clears the slot's validity bit, decrementsn_vectorsand deletes the rowid-map and_ivf_vectorsrows without checking any of them, so a delete can report success with the row gone from the table but still in the index.oversample, a failed read of a candidate's float32 vector keeps its quantized distance, and an unprepared statement skips the rescoring altogether.Each of those statements and blob accesses now returns its error, and the delete or query returns it to SQLite.
Measured by interrupting one statement at each VM step on 20 rows (with the IVF define):
oversample=4Here every interrupted KNN run reports "interrupted". A delete's failure can still read as vec0's own message, such as "could not open validity blob for chunk 1", when the interrupted step is vec0's chunk bookkeeping rather than the index's; it does on a flat column too, and in those runs the row stays in both the table and the index. And a delete interrupted at its last steps can report "interrupted" with the row already gone, from the table and the index alike, as a plain SQLite table's DELETE does under the same sweep.
Test:
test_ivf_delete_interrupted_at_each_stepandtest_ivf_knn_interrupted_at_each_stepin test-ivf.py interrupt each step of a delete and a KNN query, on an untrained and a trained index, without a quantizer and with int8, and the KNN query with an oversample of 4 too. A delete that reports success must have taken its row out of the index, and one that fails must leave the table and the index agreeing; a KNN query must report "interrupted" or return every row and distance of the uninterrupted query. All seven fail on #354's head and pass here. The Python suite passes (with-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, 315 passed, 4 skipped; without it, 239 passed, 80 skipped), and test-ivf.py passes under ASan and UBSan. The IVF build prints the same warnings as #354's head, and without the definesqlite-vec-ivf.cis not compiled.sqlite-vec-ivf.cis outside the Makefile'sFORMAT_FILES, and the changed lines follow its style.This builds on #354, and through it #351, and carries their commits: #354 makes the IVF commands, inserts and
ivf_is_trained()return their errors, and left these paths because #347 and #350 change them. It merges cleanly with #347 and with #350 (on #349), and the three merged build and pass the suite (325 passed, 4 skipped) with no run of the sweep above reporting success over a failed step; #349'sivf_full_vector_read(), which quantized point reads go through, already returns its step's error. Like #354 it conflicts with #313, in the same places.Origin. Found in the IVF port to photostructure/sqlite-vec, photostructure#12 (65817f4), whose IVF code returns these errors, and measured here after #354 named them. The sweep interrupts each step as mceachen's
test_interrupt_is_reported_as_interruptdoes vec0's statements in photostructure/sqlite-vec@7a9ab11, which the port extends to IVF; the change and the tests are written here for asg017's main. No open or closed PR or issue here makes the same change.