Skip to content

Read a full scan's vectors through one blob handle per chunk - #336

Open
MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:fullscan-chunk-reads
Open

MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:fullscan-chunk-reads

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 22, 2026 •

Copy link
Copy Markdown

A vec0 full scan reads each row's vector through vec0_get_vector_data(), which for a flat column looks up the row's chunk, opens the chunk's vectors blob, reads one vector and closes the blob again. A chunk's blob holds chunk_size vectors (3 MiB for 1024 float[768]), and the first read through a newly opened handle walks the blob's overflow pages up to the row's offset, so the opens dominate the scan.

On a 10,000-row float[768] table (medians of 3 interleaved runs of 7 queries):

query main, file this change, file main, memory this change, memory
select sum(length(embedding)) from v 762.4 ms 12.6 ms 104.7 ms 5.4 ms
select sum(vec_distance_cosine(embedding, ?)) from v 853.5 ms 33.8 ms 111.8 ms 24.5 ms

The scan already visits rows in (chunk_id, chunk_offset) order, and for a flat column the cursor now keeps one blob handle per vector column, moved to the next chunk with sqlite3_blob_reopen(). Each row's chunk position is looked up with vec0_get_chunk_position() when its vector is read: the scan's rowid statement sorts every row when the scan starts, so positions taken from it would miss a delete or insert the same connection runs during the scan. A write that replaces the chunk's row expires the handle, and a rollback of a transaction that changed the schema invalidates it (SQLITE_ABORT, or SQLITE_ABORT_ROLLBACK with extended result codes on); the read is then retried once on a newly opened handle. Any other failure to move the handle, such as an interrupt raised by a progress handler, fails the read. Rescore, IVF and DiskANN columns keep their vectors in tables of their own and are read row by row as before.

The position lookup and that failure handling are mceachen's: photostructure/sqlite-vec@f793711 corrected this change after it merged there as photostructure#1, for the three problems his review found. Adapted here: this tree has no vec0_error_rc(), so a failed move fails the read with SQLITE_ERROR like the other failures in the function; a cancellation inside vec0's own statements is reported as a vec0 error rather than interrupted, so his interrupt test checks that the cancelled scan fails rather than its message; and his optimize test is left out, since this tree has no optimize command.

tests/test-fullscan.py checks every row's vectors against point lookups across chunks with deleted rows and two vector columns, scans rescore and DiskANN columns beside a flat one (it fails if the chunk handle is used for them), reads a chunk whose shadow row is rewritten in the middle of a scan, fails on a row deleted during the scan instead of returning the vector of the row that took its slot, and fails a scan cancelled while its handle moves to the next chunk. tests/test-unit.c scans across a rollback that changed the schema, with extended result codes off and on. The deleted-row and interrupt tests fail on this PR's previous head and pass here; the Python suite passes (244 passed, 66 skipped, 204 snapshots). make test-unit does not build on main; on a tree with stumpylog's #305, which makes it build, the rollback test fails on the previous head and passes here.

It merges cleanly with #267, #327 through #331 and #337 through #357. The same change is merged on photostructure/sqlite-vec as photostructure#1 with f793711 on top, and open on vlasky/sqlite-vec as vlasky#17.

A vec0 full scan read each row's vector through vec0_get_vector_data(),
which for a flat column looks up the row's chunk, opens the chunk's vectors
blob, reads one vector and closes the blob again. A chunk's blob holds
chunk_size vectors (3 MiB for 1024 float[768]), and the first read through a
newly opened handle walks the blob's overflow pages up to the row's offset,
so the opens dominated the scan.

The scan already visits rows in (chunk_id, chunk_offset) order, and for a
flat column the cursor now keeps one blob handle per vector column, moved to
the next chunk with sqlite3_blob_reopen(). Each row's chunk position is
looked up when its vector is read: the scan's rowid statement sorts every
row when the scan starts, so its positions would miss a delete or insert the
same connection runs during the scan. A write that replaces the chunk's row
expires the handle, and a rollback of a transaction that changed the schema
invalidates it (SQLITE_ABORT, or SQLITE_ABORT_ROLLBACK with extended result
codes on); the read is then retried once on a newly opened handle. Any other
failure to move the handle, such as an interrupt, fails the read. Rescore,
IVF and DiskANN columns keep their vectors in tables of their own and are
read row by row as before.

The position lookup and that failure handling follow
photostructure/sqlite-vec@f793711 by Matthew McEachen, which corrected this
change after it merged there as photostructure#1.

On a 10,000-row float[768] table (medians of 3 interleaved runs of 7
queries), selecting every vector takes 12.6 ms instead of 762.4 ms in a
database file and 5.4 ms instead of 104.7 ms in memory, and
vec_distance_cosine() against each 33.8 ms instead of 853.5 ms and 24.5 ms
instead of 111.8 ms. tests/test-fullscan.py checks every row's vectors
against point lookups across chunks with gaps and two vector columns, scans
rescore and DiskANN columns beside a flat one, reads a chunk rewritten in
the middle of a scan, fails on a row deleted during the scan instead of
returning another row's vector, and fails a scan cancelled while its handle
moves to the next chunk; tests/test-unit.c scans across a rollback that
changed the schema, with extended result codes off and on.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant