Repository navigation
Conversation
This was referenced Oct 22, 2026
A vec0 full scan read each row's vector through vec0_get_vector_data(), which for a flat column looks up the row's chunk, opens the chunk's vectors blob, reads one vector and closes the blob again. A chunk's blob holds chunk_size vectors (3 MiB for 1024 float[768]), and the first read through a newly opened handle walks the blob's overflow pages up to the row's offset, so the opens dominated the scan. The scan already visits rows in (chunk_id, chunk_offset) order, and for a flat column the cursor now keeps one blob handle per vector column, moved to the next chunk with sqlite3_blob_reopen(). Each row's chunk position is looked up when its vector is read: the scan's rowid statement sorts every row when the scan starts, so its positions would miss a delete or insert the same connection runs during the scan. A write that replaces the chunk's row expires the handle, and a rollback of a transaction that changed the schema invalidates it (SQLITE_ABORT, or SQLITE_ABORT_ROLLBACK with extended result codes on); the read is then retried once on a newly opened handle. Any other failure to move the handle, such as an interrupt, fails the read. Rescore, IVF and DiskANN columns keep their vectors in tables of their own and are read row by row as before. The position lookup and that failure handling follow photostructure/sqlite-vec@f793711 by Matthew McEachen, which corrected this change after it merged there as photostructure#1. On a 10,000-row float[768] table (medians of 3 interleaved runs of 7 queries), selecting every vector takes 12.6 ms instead of 762.4 ms in a database file and 5.4 ms instead of 104.7 ms in memory, and vec_distance_cosine() against each 33.8 ms instead of 853.5 ms and 24.5 ms instead of 111.8 ms. tests/test-fullscan.py checks every row's vectors against point lookups across chunks with gaps and two vector columns, scans rescore and DiskANN columns beside a flat one, reads a chunk rewritten in the middle of a scan, fails on a row deleted during the scan instead of returning another row's vector, and fails a scan cancelled while its handle moves to the next chunk; tests/test-unit.c scans across a rollback that changed the schema, with extended result codes off and on.
MayCXC
force-pushed
the
fullscan-chunk-reads
branch
from
October 23, 2026 22:26
8a96cfa to
6262246
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
vec0full scan reads each row's vector throughvec0_get_vector_data(), which for a flat column looks up the row's chunk, opens the chunk's vectors blob, reads one vector and closes the blob again. A chunk's blob holdschunk_sizevectors (3 MiB for 1024float[768]), and the first read through a newly opened handle walks the blob's overflow pages up to the row's offset, so the opens dominate the scan.On a 10,000-row
float[768]table (medians of 3 interleaved runs of 7 queries):select sum(length(embedding)) from vselect sum(vec_distance_cosine(embedding, ?)) from vThe scan already visits rows in
(chunk_id, chunk_offset)order, and for a flat column the cursor now keeps one blob handle per vector column, moved to the next chunk withsqlite3_blob_reopen(). Each row's chunk position is looked up withvec0_get_chunk_position()when its vector is read: the scan's rowid statement sorts every row when the scan starts, so positions taken from it would miss a delete or insert the same connection runs during the scan. A write that replaces the chunk's row expires the handle, and a rollback of a transaction that changed the schema invalidates it (SQLITE_ABORT, orSQLITE_ABORT_ROLLBACKwith extended result codes on); the read is then retried once on a newly opened handle. Any other failure to move the handle, such as an interrupt raised by a progress handler, fails the read. Rescore, IVF and DiskANN columns keep their vectors in tables of their own and are read row by row as before.The position lookup and that failure handling are mceachen's: photostructure/sqlite-vec@f793711 corrected this change after it merged there as photostructure#1, for the three problems his review found. Adapted here: this tree has no
vec0_error_rc(), so a failed move fails the read withSQLITE_ERRORlike the other failures in the function; a cancellation inside vec0's own statements is reported as a vec0 error rather thaninterrupted, so his interrupt test checks that the cancelled scan fails rather than its message; and hisoptimizetest is left out, since this tree has nooptimizecommand.tests/test-fullscan.pychecks every row's vectors against point lookups across chunks with deleted rows and two vector columns, scans rescore and DiskANN columns beside a flat one (it fails if the chunk handle is used for them), reads a chunk whose shadow row is rewritten in the middle of a scan, fails on a row deleted during the scan instead of returning the vector of the row that took its slot, and fails a scan cancelled while its handle moves to the next chunk.tests/test-unit.cscans across a rollback that changed the schema, with extended result codes off and on. The deleted-row and interrupt tests fail on this PR's previous head and pass here; the Python suite passes (244 passed, 66 skipped, 204 snapshots).make test-unitdoes not build on main; on a tree with stumpylog's #305, which makes it build, the rollback test fails on the previous head and passes here.It merges cleanly with #267, #327 through #331 and #337 through #357. The same change is merged on photostructure/sqlite-vec as photostructure#1 with f793711 on top, and open on vlasky/sqlite-vec as vlasky#17.