[MayCXC]

Every DELETE clears the row's validity bit through one blob handle, then vec0Update_Delete_DeleteChunkIfEmpty() opens the chunk's validity blob a second time and reads all of it to learn whether the chunk is now empty. A byte that still holds another row's bit keeps the chunk, as it does for most deletes, so vec0Update_Delete_ClearValidity() now reports whether the chunk is empty, reading the rest of the bitmap through the handle it already holds and only once the cleared byte is zero, and vec0Update_Delete_DeleteChunk() deletes the chunk when it is. This is photostructure/sqlite-vec@4a8ffe0 by Matthew McEachen, ported.

Deleting 20,000 rows of a float[8] table one statement at a time takes 48,094 instructions per DELETE instead of 52,443 under callgrind, 8.3% fewer, where two copies of main's build count the same. test_delete_keeps_chunk_while_another_validity_byte_has_rows clears either byte of a 16-row chunk's bitmap and checks that the chunk stays with the other byte's rows, then goes once both are clear. The full suite passes (144 passed, 4 skipped, 259 snapshots), also with vec0 built with AddressSanitizer and UBSan, and make test-unit passes. It merges cleanly with each of our open PRs here (#9, #10, #11, #12, #13, #14, #17, #18, #19, #20, #21, #22, #23, #24) and with #25, which fixes the bit check in the same function.

The same change is open upstream as asg017#359.

[vlasky]

Thanks for porting this; the restructure is right and the saving is real for the common case. I could not push to your branch: your repository is a fork of asg017/sqlite-vec rather than of this one, so GitHub never grants the maintainer-edit permission here even with the box ticked (the push is refused with 403). So I merged the PR with one commit of mine on top, 7b4bd2c, for one case it changed.

The DELETE learns that the chunk looks empty at step 2, but steps 5 to 7 run statements on the shadow tables, and a trigger on one of them (SQLite allows that unless SQLITE_DBCONFIG_DEFENSIVE is on) can insert into the table and refill the chunk before step 8 reclaims it:

create virtual table v using vec0(emb float[1], chunk_size=8);
insert into v(rowid, emb) values (1, '[1]');
create trigger refill after delete on v_rowids when old.rowid = 1
begin insert into v(rowid, emb) values (2, '[2]'); end;
delete from v where rowid = 1;

main keeps the chunk and row 2 is found by a KNN query. On this branch the chunk is deleted under it: v_chunks is empty, v_rowids still has rowid 2 pointing at the deleted chunk, and the KNN query returns nothing. Exotic, but a regression, and the fix keeps your saving: a chunk whose byte went to zero is read once more through a fresh handle before it goes, so the second open happens only on the rare delete that empties a byte. The test pins the trigger case. Full suite and make test-unit pass. Merged.