Repository navigation
Conversation
IVF's cells, centroids and quantizers hold float32 vectors, sized by ivf_vec_size() and filled by ivf_quantize() from the inserted vector as an array of floats, but the column parser accepted `int8[N]` and `bit[N]` columns with `indexed by ivf(...)`. Every insert into one read past the end of its vector, and every distance was computed from those bytes. The parser now refuses them, as it does for rescore.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An
int8orbitvector column is accepted withindexed by ivf(...), though every part of the IVF index holds float32 vectors.ivf_insert()hands the inserted vector toivf_quantize()as an array ofdimensionsfloats, and the cells, the centroids and the quantizers are all sized and computed for float32, but the column parser never checked the element type. It now refuses an IVF index on a column that is not float32, as it already does for rescore, so the CREATE fails with "could not parse vector column".Test:
test_ivf_create_error_non_float32_columnin test-ivf.py, parametrized overint8[8]andbit[8]; both fail on main and pass here. The Python suite passes (with-DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, 303 passed, 4 skipped; without it, 239 passed, 68 skipped), under ASan and UBSan as well, and both builds' warnings are main's.It merges cleanly with every open PR that touches the IVF code or the column parser, #310, #313, #319 and #349 to #352 included.
Origin. Found in the IVF port to photostructure/sqlite-vec, photostructure#12 (65817f4), which lists it among the port's differences from upstream; the change and its test are written here for asg017's main. No open or closed PR or issue here makes the same change.