refactor(diskann): adapt diskann to turbo int8 pq quantizer

[richyreachy]

• delete the homegrown PQ implementation and remove unused buffers in diskann context
• train and encode through PQ quantizer, and search and compute candidate distances via the
• new quantizer distances
• store one serialized quantizer object instead of separate sections in the PQ meta segment

Build Index

DataSet	main	turbo_pq
cohere	7 min 35 sec / 18.2GB	6 min 20 sec / 6.6GB
gist	9 min 11 sec / 22.6GB	7 min 47 sec / 8.2GB

QPS

L	cohere main	cohere turbo_pq	gist main	gist turbo_pq
10	391.7	301.8	271.5	288.4
20	299.0	320.2	290.6	285.8
50	250.1	225.9	226.8	213.1
100	198.2	186.6	145.2	164.3
200	124.3	134.3	117.2	115.5
300	98.1	107.7	76.6	91.0
500	66.3	62.9	65.5	68.2

Recall@10

L	cohere main	cohere turbo_pq	gist main	gist turbo_pq
10	85.48	86.27	58.38	58.21
20	91.56	91.70	70.70	70.30
50	96.92	96.80	83.52	83.27
100	98.26	98.17	91.17	91.34
200	99.34	99.30	95.90	95.75
300	99.54	99.52	97.41	97.33
500	99.72	99.68	98.75	98.71

With the introduction of Turbo, build memory consumption shrinks to one-third while the QPS and recall keep up.