feat(index): f64 centroids support in streaming coreset k-means

[a-erofeev]

Implemented f64 centroid support for streaming coreset k-means.

Key changes:

• Generalized centroid storage and arithmetic over f32/f64.
• Preserved Float64 throughout coreset construction, reduction, hierarchical training, and refinement.
• Added schema-based dispatch and explicit f16/int8 → f32 preparation.
• Added precision, type-preservation, refinement, cosine, and resampling tests.

Benchmark report.

Before changes:

Percentages are relative to LanceStream stream=64, coreset=16, prefetch=1, refine=0 at the same vector type and k. A baseline cannot be calculated for f64 because the old streaming path fails.

k	algorithm	status	train time	time delta	RSS	RSS delta	exact loss	loss delta
1,024	LanceStream, f32 vectors, L2 distance	ok	13.07s	baseline	101.0 MiB	baseline	2,593,041.879	baseline
1,024	Lance non-stream, f32 vectors, L2 distance	ok	4.14s	-68.3%	254.6 MiB	+152.1%	2,588,575.751	-0.172%
4,096	LanceStream, f32 vectors, L2 distance	ok	38.92s	baseline	280.8 MiB	baseline	10,141,326.860	baseline
4,096	Lance non-stream, f32 vectors, L2 distance	ok	15.27s	-60.8%	733.4 MiB	+161.2%	10,109,608.843	-0.313%
1,024	LanceStream, f64 vectors, L2 distance	error: invalid Float32 downcast	—	unavailable	—	unavailable	—	unavailable
1,024	Lance non-stream, f64 vectors, L2 distance	ok	4.25s	unavailable	396.8 MiB	unavailable	2,588,382.089	unavailable
4,096	LanceStream, f64 vectors, L2 distance	error: invalid Float32 downcast	—	unavailable	—	unavailable	—	unavailable
4,096	Lance non-stream, f64 vectors, L2 distance	ok	29.07s	unavailable	1,336.6 MiB	unavailable	10,109,749.792	unavailable

After changes:

Percentages use the same per-type, per-k LanceStream baseline.

k	algorithm	status	train time	time delta	RSS	RSS delta	exact loss	loss delta
1,024	LanceStream, f32 vectors, L2 distance	ok	12.84s	baseline	119.7 MiB	baseline	2,593,060.384	baseline
1,024	Lance non-stream, f32 vectors, L2 distance	ok	4.04s	-68.5%	268.8 MiB	+124.6%	2,588,180.264	-0.188%
4,096	LanceStream, f32 vectors, L2 distance	ok	40.82s	baseline	300.1 MiB	baseline	10,141,197.028	baseline
4,096	Lance non-stream, f32 vectors, L2 distance	ok	18.21s	-55.4%	751.2 MiB	+150.3%	10,109,680.003	-0.311%
1,024	LanceStream, f64 vectors, L2 distance	ok	24.99s	baseline	173.0 MiB	baseline	2,592,598.480	baseline
1,024	Lance non-stream, f64 vectors, L2 distance	ok	4.86s	-80.5%	414.4 MiB	+139.5%	2,587,814.467	-0.185%
4,096	LanceStream, f64 vectors, L2 distance	ok	77.22s	baseline	510.8 MiB	baseline	10,143,331.479	baseline
4,096	Lance non-stream, f64 vectors, L2 distance	ok	32.13s	-58.4%	1,352.8 MiB	+164.8%	10,109,706.606	-0.331%

Direct after-versus-before comparison

These are the clean one-shot measurements above. Positive values mean the after revision is larger or slower.

vector type	k	algorithm	train time change	raw RSS change	exact loss change
f32	1,024	LanceStream	-1.8%	+18.5% (+18.7 MiB)	+0.0007%
f32	1,024	Lance non-stream	-2.5%	+5.6% (+14.2 MiB)	-0.0153%
f32	4,096	LanceStream	+4.9%	+6.9% (+19.3 MiB)	-0.0013%
f32	4,096	Lance non-stream	+19.2%	+2.4% (+17.8 MiB)	+0.0007%
f64	1,024	LanceStream	newly supported	newly supported	newly supported
f64	1,024	Lance non-stream	+14.4%	+4.4% (+17.6 MiB)	-0.0219%
f64	4,096	LanceStream	newly supported	newly supported	newly supported
f64	4,096	Lance non-stream	+10.5%	+1.2% (+16.3 MiB)	-0.0004%

Dataset and methodology:

• Dimension: 128, matching the convention in the referenced
  LanceStream benchmark report.

• Datasets:

  • 10,000,000 FixedSizeList<Float32, 128> vectors, 4.8 GiB on disk.
  • 10,000,000 FixedSizeList<Float64, 128> vectors, 9.6 GiB on disk.

• Generator: independent uniform random coordinates in [0, 1), generated in
  the dataset's native scalar type with SmallRng seed
  2026082010000128. Both datasets report exactly 10,000,000 rows and contain
  ten Lance data files.

• f32 generation took 9.52s and peaked at 169,472 KiB RSS; f64 generation took
  17.61s and peaked at 298,572 KiB RSS.

• Both algorithms use L2 distance, sample_rate=256, and max_iters=50.

• LanceStream uses streaming_sample_rate=64,
  streaming_coreset_rate=16, LANCE_STREAMING_IVF_PREFETCH_DEPTH=1, and
  streaming_refine_passes=0.

• Non-stream uses the same k, sample rate, iteration budget, metric, and
  dataset, with streaming parameters unset.

• Both revisions were built from isolated source archives with
  release-with-debug, Thin-LTO, 16 codegen units, and default Lance features
  disabled. The final binary SHA-256 values are
  aed6884e...e5688 (before) and ea15c7e2...868c6 (after).

• Train time is measured inside build_ivf_model; centroid serialization is
  outside the timer.

• RSS is GNU /usr/bin/time -v maximum resident set size from a fresh process
  for each configuration.

• Exact loss is evaluated in a separate process over the first k * 256 rows:

  sum_x min_c ||x - c||^2
  

  HNSW acceleration was explicitly disabled, so every evaluation vector was
  compared with every centroid. Lance's current L2 API returns each distance
  as f32, including for f64 vectors, and these values were summed into f64.
  Thus “exact” means exhaustive assignment, not fully f64 loss arithmetic.

• Streaming uses deterministic fixed sample ranges. Batch uses its normal
  random sampler, so one-run batch timing and loss include sampling and
  initialization variance.

• The main tables contain one measured run per configuration. The additional
  f32 k=1024 repetitions were diagnostic and are reported separately.

Host:

• CPU: Intel Core i7-9700K at 3.60 GHz, 8 physical cores / 8 threads.
• Memory: 31 GiB.
• Architecture: x86-64.
• Toolchain: Rust 1.97.0, Cargo 1.97.0.

---

According to this benchmark, RSS is not very good, but seems it is not implementation problem.
So i repeated f64-centroid benchmark with more precise approach.

Five fresh-process repetitions were run for each configuration on each revision. Each
revision used its own isolated release-with-debug binary; processes were pinned to
CPUs 0–7 with RAYON_NUM_THREADS=8, and configuration order was shuffled per
repetition. RSS is /usr/bin/time -v maximum resident set size. Values below are
mean ± sample standard deviation.

Raw measurements:

revision	vectors	algorithm	k	status	train seconds	RSS (KiB)
before	f32	stream	1024	0	10.282 ± 0.827	120,163 ± 896
before	f32	batch	1024	0	2.872 ± 0.721	271,254 ± 3,221
before	f32	stream	4096	0	35.345 ± 3.339	303,823 ± 1,083
before	f32	batch	4096	0	17.160 ± 1.740	759,851 ± 6,813
before	f64	stream	1024	101 (unsupported)	—	—
before	f64	batch	1024	0	3.776 ± 0.265	427,419 ± 5,080
before	f64	stream	4096	101 (unsupported)	—	—
before	f64	batch	4096	0	31.568 ± 1.582	1,385,742 ± 6,524
after	f32	stream	1024	0	9.469 ± 0.060	118,806 ± 1,082
after	f32	batch	1024	0	2.882 ± 0.789	274,706 ± 2,423
after	f32	stream	4096	0	35.069 ± 1.899	300,614 ± 4,235
after	f32	batch	4096	0	17.049 ± 1.253	759,155 ± 6,226
after	f64	stream	1024	0	19.323 ± 1.778	175,271 ± 1,129
after	f64	batch	1024	0	3.814 ± 0.139	422,583 ± 6,177
after	f64	stream	4096	0	68.767 ± 3.056	519,189 ± 1,734
after	f64	batch	4096	0	30.566 ± 1.052	1,380,254 ± 8,455

After vs before:

For configurations available on both revisions, percentage deltas use the revision
means. Streaming f32 changed by −7.9% time / −1.1% RSS at k=1024 and −0.8% /
−1.1% at k=4096. Batch f32 changed by +0.3% / +1.3% and −0.6% / −0.1%,
respectively. Batch f64 changed by +1.0% / −1.1% at k=1024 and −3.2% / −0.4%
at k=4096. The f64 streaming path is newly supported, so it has no before value.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The type-dispatched coreset pipeline fixes the Float64 failure at its root and preserves Float64 through hierarchical training and refinement while retaining the established Float32 path.