[MayCXC]

vec0 checks a KNN query's constraints itself and sets aConstraintUsage[].omit, which tells SQLite a constraint "is assumed to be fully handled by the virtual table and might not be checked again" (sqlite3.h). Each constraint vec0 claims therefore has to keep exactly the rows SQLite would keep for the same WHERE clause, and today the same WHERE clause keeps different rows with and without a match:

create virtual table v using vec0(a float[1], n integer, chunk_size=8);
insert into v(rowid, a, n) values (1, '[1]', 0), (2, '[2]', 5);

select rowid from v where n = 'abc';                               -- no rows
select rowid from v where a match '[0]' and k = 10 and n = 'abc';  -- 1

On main:

• n = ? bound to NULL or 'abc' keeps n = 0, n = 5.5 keeps n = 5, n < 5.5 drops n = 5, and n in (5.5, 6) keeps n = 5.
• f > 'abc' keeps every positive f, though every number is less than any text.
• b = 2 keeps the true rows.
• t in (null) keeps t = '', and t = x'78' keeps t = 'x'.
• t < ? bound to NULL crashes (a NULL pointer read) once a chunk holds a text longer than 12 bytes, and <, <=, > and >= fail with "Could not filter metadata fields" once a chunk holds a text of exactly 12 bytes. The prefix bounds SQLite derives from t LIKE 'abc%' and t GLOB 'abc*' are such comparisons, so those fail too; the LIKE and GLOB boundary snapshots recorded that error.
• Text comparisons ignore the query's collation: t = 'abc' COLLATE NOCASE misses 'ABC', and COLLATE RTRIM misses 'abc '.
• LIKE matches with SQLite's default case folding and claims the result, so under PRAGMA case_sensitive_like t LIKE '%c' keeps 'ABC'. GLOB reads a prefix pattern with a character class literally ([aA]* matches nothing), and both read past the end of a 12-byte text.
• A partition key compares as stored: p = '5' misses p = 5, s = 5 misses s = '5', and collations are ignored.
• rowid in (5.5, 'abc', null) matches rowids 5 and 0, and a point lookup rowid = ? matches rowid 0 for NULL, 'abc' and a blob, and rowid 5 for 5.5. With a text primary key, an id in id in (...) that names no row empties the whole result.
• distance > ? bound to NULL keeps every row and distance < 'abc' none, and distances compared in f32 keep the row at distance 2 for distance <= 1.999999999999, which SQLite drops.

This follows SQLite's R-Tree, the extension that checks its own constraints:

• vec0 declares its columns with their types, as rtreeInit() does: metadata columns boolean, integer, float or text, partition keys and a text primary key with theirs, rowid integer and distance real. SQLite's own checks then give a value the affinity vec0 gives it.
• Each filter gives the constraint's value its column's affinity and compares by Datatypes In SQLite, 4.1 and 4.2: NULL compares with nothing, INTEGER and REAL compare exactly, as sqlite3IntFloatCompare() does, every number sorts before any text, and every text before any blob. A rowid lookup takes the value as rtreeFilter() does: numeric affinity, then an integer or a real equal to one.
• Text compares under the constraint's collation, from sqlite3_vtab_collation(): BINARY, NOCASE and RTRIM as binCollFunc, nocaseCollatingFunc and rtrimCollFunc do, and any other collation is refused with an error. SQLite names no collation for != or IS NOT (it hands them over without their left operand), so vec0 keeps the rows unequal under BINARY and leaves those two for SQLite to check again. SQLite orders text under BINARY by the bytes of the database's encoding, so in a UTF-16 database vec0 compares the bytes SQLite stores for each value, converting its UTF-8 with SQLite's own READ_UTF8, WRITE_UTF16LE and WRITE_UTF16BE (utf.c), and orders a text longer than its 12-byte prefix by its full text, since a prefix can end inside a character. vec0 reads the encoding when it opens a table, from sqlite3_value_encoding() of a string literal on SQLite 3.41 and later, which needs no PRAGMA an authorizer might deny, and from PRAGMA encoding before that, where no row means UTF-8 only in a build with SQLITE_OMIT_UTF16. A read that fails fails the constructor with its error, and an interrupted one reports as an interrupt. This read follows photostructure/sqlite-vec@8156329 by Matthew McEachen, which corrected it after this change merged there as Port the rest of vlasky#12: collated text partition keys and UTF-16 text ranges photostructure/sqlite-vec#4.
• LIKE and GLOB match as SQLite's built-in functions do and are left for SQLite to check again, as FTS5 leaves them (fts5BestIndexMethod()), so the rows returned satisfy the connection's own like() and glob().
• A distance compares as the double SQLite receives.

The tests compare each filter with a plain table declaring the same types and with SQLite's own scan of the vtab: every operator over NULL, integers, reals, numeric and other text and blobs, texts around the 12 bytes a chunk keeps inline, each collation, LIKE and GLOB patterns (also under case_sensitive_like), in (...) lists, partition keys, rowid and text id lookups, distances a hair either side of a row's, and UTF-16 databases, where every filter and literal LIKE and GLOB pattern is compared at each k and with LIMIT. A KNN query whose k is the number of matching rows must return all of them, so a filter that keeps any other row fails. A connection whose authorizer denies PRAGMAs opens the table and keeps its text range filters, as on main, and creating a table fails when the encoding cannot be read. The tests fail on main and pass here. The full suite passes (151 passed, 4 skipped, 259 snapshots), also with vec0 built with AddressSanitizer and UBSan, and make test-unit passes.

The same change is open upstream as asg017#329.

[vlasky]

Thanks for the update, and for the UTF-16 comparator: it is the right fix (I had found that SQLite's BINARY in a UTF-16LE database is a memcmp over little-endian bytes, so 'Ā' < 'a', and the previous head returned fewer than k rows for ASCII ranges there). My differential harness (about 6,400 bound-parameter checks and 7,400 literal checks through an ASan+UBSan build, in UTF-8 and UTF-16 databases) agrees with the description: every remaining mismatch is either pre-existing or something the vtab API cannot see. I could not push to your branch: your repository is a fork of asg017/sqlite-vec rather than of this one, so GitHub never grants the maintainer-edit permission here even with the box ticked (the push is refused with 403). So the small things are one commit on top of your head, 537e17b on the branch pr-12-fix of this repository; git pull https://github.com/vlasky/sqlite-vec.git pr-12-fix into your branch brings it in, or tell me if you would rather take the changes another way. It covers:

1. Text IN values with an embedded NUL read past their copy. The entries were copied with sqlite3_mprintf("%.*s"), which stops at the first NUL, while vec0_text_compare() reads all entry.n bytes. Binding "a\0" + "x" * 62 to t IN (?, 'other') gives a heap-buffer-overflow under AddressSanitizer, and under RTRIM the trailing-space trim reads b[62] of a 3-byte allocation. main used strncmp, which stopped at the NUL, so this is new with the length-based comparator. The copy is now sqlite3_malloc plus memcpy of the full length (and a value sqlite3_value_text() cannot convert is treated as out of memory), with a test comparing BINARY, NOCASE and RTRIM against a plain table for values and stored texts containing NULs.
2. vec0BestIndex now frees the partly built idxStr on its error exits; the three new collation refusals leaked it.
3. sqlite3_value_encoding() arrived in SQLite 3.40.0 (vendor/sqlite3ext.h says so above the macro), not 3.41, so both guards and the tests now say 3.40; on 3.40 with an authorizer that denies PRAGMAs the constructor was failing for no reason.
4. A core built with SQLITE_OMIT_UTF16 may also omit sqlite3_compileoption_used(), in which case PRAGMA encoding returns no row and the probe cannot ask, so every constructor failed; with SQLITE_CORE and SQLITE_OMIT_UTF16 both defined, no row now means UTF-8 at compile time.
5. Docs: the metadata page and ARCHITECTURE.md now say which constraints SQLite re-checks after the k nearest rows are chosen (!= and IS NOT under NOCASE or RTRIM, LIKE and GLOB), that such a query returns the nearest qualifying rows but possibly fewer than k, and which cases a larger k does not recover: a replacement like()/glob() that accepts more than the built-in matching, IN (SELECT ... COLLATE x) (vec0 sees the left operand's collation), a replaced NOCASE or RTRIM collation, and a join condition against a column of another affinity (v.t = nums.n with '05' and 5 keeps the row in SQLite and not in vec0, on main too, as with R-Tree). ARCHITECTURE.md also now describes the fourth idxStr character as the collation and the distance comparison as double rather than f32.

Full suite: 152 passed, 4 skipped, 259 snapshots; make test-unit passes; the ASan run of the NUL reproduction is clean, and a static -DSQLITE_CORE -DSQLITE_OMIT_UTF16 build compiles.

Two things I left alone for your judgement: vec0_constraint_collation() still checks for SQLite < 3.22, which this file cannot run on (it uses sqlite3_str_new, 3.24), so that branch is dead; and the x IN (SELECT ... COLLATE y) case could be refused outright rather than documented, if you prefer an error to a silently narrower result.