Repository navigation
Conversation
Every DELETE cleared the row's validity bit through one blob handle, and vec0Update_Delete_DeleteChunkIfEmpty() then opened the chunk's validity blob a second time and read all of it to learn whether the chunk was now empty. A byte that still holds another row's bit keeps the chunk, as it does for most deletes, so vec0Update_Delete_ClearValidity() now reports whether the chunk is empty, reading the rest of the bitmap through the handle it already holds and only once the cleared byte is zero, and vec0Update_Delete_DeleteChunk() deletes the chunk when it is. Ported from photostructure/sqlite-vec@4a8ffe0 by Matthew McEachen. Deleting 20,000 rows of a float[8] table one statement at a time takes 48,232 instructions per DELETE instead of 52,557 under callgrind, 8.2% fewer, where two copies of main's build count the same. Deleting 100,000 rows took 0.54 s of CPU instead of 0.64 s (medians of 11 interleaved runs, where two copies of main's build differed by 0.0%). test_delete_keeps_chunk_while_another_validity_byte_has_rows clears either byte of a 16-row chunk's bitmap and checks that the chunk stays with the other byte's rows, then goes once both are clear. The full suite passes (240 passed, 66 skipped, 204 snapshots), also with the IVF index compiled in (302 passed, 4 skipped) and with vec0 built with AddressSanitizer and UBSan.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every DELETE clears the row's validity bit through one blob handle, then
vec0Update_Delete_DeleteChunkIfEmpty(), from cb147c8 (#268), opens the chunk's validity blob a second time and reads all of it to learn whether the chunk is now empty. A byte that still holds another row's bit keeps the chunk, as it does for most deletes, sovec0Update_Delete_ClearValidity()now reports whether the chunk is empty, reading the rest of the bitmap through the handle it already holds and only once the cleared byte is zero, andvec0Update_Delete_DeleteChunk()deletes the chunk when it is. This is photostructure/sqlite-vec@4a8ffe0 by Matthew McEachen, ported.Deleting 20,000 rows of a
float[8]table one statement at a time takes 48,232 instructions per DELETE instead of 52,557 under callgrind, 8.2% fewer, where two copies of main's build count the same. Deleting 100,000 rows took 0.54 s of CPU instead of 0.64 s (medians of 11 interleaved runs, where two copies of main's build differed by 0.0%).test_delete_keeps_chunk_while_another_validity_byte_has_rowsclears either byte of a 16-row chunk's bitmap and checks that the chunk stays with the other byte's rows, then goes once both are clear. The full suite passes (240 passed, 66 skipped, 204 snapshots), also with the IVF index compiled in (302 passed, 4 skipped) and with vec0 built with AddressSanitizer and UBSan. It merges cleanly with each of our open PRs here and with #358, which fixes the bit check in the same function.The same change is open on vlasky's fork as vlasky#26.