Skip to content

Refuse int8 and bit columns for IVF indexes - #353

Open
MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:ivf-float32-columns
Open

MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:ivf-float32-columns

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

An int8 or bit vector column is accepted with indexed by ivf(...), though every part of the IVF index holds float32 vectors.

create virtual table t using vec0(v int8[2048] indexed by ivf(nlist=2));
insert into t(rowid, v) values (1, vec_int8(:v));
-- ASan: heap-buffer-overflow, a READ of 8192 bytes from the 2048-byte vector in ivf_quantize(), from ivf_insert()
select rowid, distance from t where v match vec_int8(:v) and k = 1;
-- 1, NULL: a vector's distance to itself is computed from the bytes past it

ivf_insert() hands the inserted vector to ivf_quantize() as an array of dimensions floats, and the cells, the centroids and the quantizers are all sized and computed for float32, but the column parser never checked the element type. It now refuses an IVF index on a column that is not float32, as it already does for rescore, so the CREATE fails with "could not parse vector column".

Test: test_ivf_create_error_non_float32_column in test-ivf.py, parametrized over int8[8] and bit[8]; both fail on main and pass here. The Python suite passes (with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, 303 passed, 4 skipped; without it, 239 passed, 68 skipped), under ASan and UBSan as well, and both builds' warnings are main's.

It merges cleanly with every open PR that touches the IVF code or the column parser, #310, #313, #319 and #349 to #352 included.

Origin. Found in the IVF port to photostructure/sqlite-vec, photostructure#12 (65817f4), which lists it among the port's differences from upstream; the change and its test are written here for asg017's main. No open or closed PR or issue here makes the same change.

IVF's cells, centroids and quantizers hold float32 vectors, sized by
ivf_vec_size() and filled by ivf_quantize() from the inserted vector as
an array of floats, but the column parser accepted `int8[N]` and
`bit[N]` columns with `indexed by ivf(...)`. Every insert into one read
past the end of its vector, and every distance was computed from those
bytes. The parser now refuses them, as it does for rescore.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant