[MayCXC]

An int8 column indexed by DiskANN reads past the vectors it quantizes, and stores its neighbors' quantized vectors wrong.

diskann_search() quantizes the query only when it is float32. For an int8 query it calls diskann_distance_quantized(), which casts the vector to const f32 * and quantizes it as floats, reading four times its bytes. Every insert searches the graph with the new vector, so every insert into an int8 column reads past it. Built with AddressSanitizer, five inserts into an int8[2048] column with neighbor_quantizer=int8:

ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 4
    #0 in diskann_quantize_vector sqlite-vec-diskann.c:160
    #1 in diskann_quantize_query sqlite-vec-diskann.c:210

The neighbors of an int8 column are stored by copying the start of their vectors (the memcpy beside each diskann_quantize_vector() call). Under the int8 quantizer that copy is the vector itself, but under the binary quantizer a neighbor's bits become its first dimensions / 8 bytes rather than its signs: in a 12-row int8[16] table with neighbor_quantizer=binary, none of the 91 stored neighbor vectors holds its row's sign bits. Under the int8 quantizer the query distances are garbage, so the search goes astray: over 10 queries on 100 rows of int8[64], with search_list_size=128 so the list can hold every row, it returns 28 of the exact 100 nearest.

This adds diskann_quantize_element_vector(), which quantizes a vector of either element type a DiskANN column accepts: a float32 vector as diskann_quantize_vector() does, and an int8 vector as itself under the int8 quantizer and as the bits of its elements above 0 under the binary quantizer, as for a float32 vector. The query and every stored neighbor go through it, and diskann_distance_quantized(), which nothing else calls, is removed.

Two tests in test-diskann.py fail on main and pass here: test_diskann_int8_column_knn_matches_exact (the exact 10 nearest rows for 10 queries) and test_diskann_int8_column_binary_neighbors_are_sign_bits (every valid neighbor slot holds its row's sign bits). The Python suite passes (241 passed, 66 skipped), test-diskann.py passes against an AddressSanitizer and UndefinedBehaviorSanitizer build, the five-insert case above runs clean under it, and the build's warnings are main's.

#311 clamps the int8 quantizer's conversion inside diskann_quantize_vector(), which this leaves alone; it merges cleanly with #311 and with every other open PR that touches the DiskANN code.

Origin. Found in the DiskANN port to photostructure/sqlite-vec, photostructure#11 (cd53413), which lists it among the port's differences from upstream; the change and its tests are written here for asg017's main. No open or closed PR or issue here makes the same change.