Open
Conversation
First milestone of the ruvector Python SDK per
docs/sdk/04-milestones.md § "M1 — RaBitQ-only Python wheel". A new
workspace crate `crates/ruvector-py/` exposes ruvector-rabitq as a
Python extension module via PyO3 + maturin with an abi3-py39 wheel
target.
## Surface
```python
import numpy as np
import ruvector
vectors = np.random.randn(10_000, 768).astype(np.float32)
idx = ruvector.RabitqIndex.build(vectors, rerank_factor=20)
results = idx.search(vectors[0], k=10) # → list[(id, distance)]
idx.save("vectors.rbpx")
idx2 = ruvector.RabitqIndex.load("vectors.rbpx")
```
## What ships
- `Cargo.toml`: cdylib crate, pyo3 0.22 with `extension-module` +
`abi3-py39`, numpy 0.22, path dep on `ruvector-rabitq`.
- `pyproject.toml`: maturin build backend, `python-source = "python"`,
`module-name = "ruvector._native"`. PyPI name: `ruvector`.
- `src/lib.rs`: defines the `_native` Python module, registers the
`RabitqIndex` class and `RuVectorError` exception.
- `src/rabitq.rs`: `RabitqIndex` wrapping `RabitqPlusIndex` with
`build` / `search` / `save` / `load` / `__len__` / `__repr__`.
All hot paths release the GIL via `py.allow_threads`.
- `src/error.rs`: maps `RabitqError` → `RuVectorError(PyException)`.
- `python/ruvector/__init__.py`: thin re-export shim from `_native`.
- `python/ruvector/py.typed`: PEP 561 marker.
- Type stubs: `python/ruvector/__init__.pyi` + `stubs/ruvector/__init__.pyi`.
- `tests/test_smoke.py`: pytest coverage of build/search/save/load,
dimension-mismatch error, len/repr, abi3 marker.
- `README.md`: install instructions + 30-second example.
## Real ruvector-rabitq API used
The plan's M1 sketch matched closely. Concrete surface:
- `RabitqPlusIndex::from_vectors_parallel(dim, seed, rerank_factor, items)`
— used in `build()`. Added `seed` kwarg (default 42) since the ctor
requires it.
- `idx.search_with_rerank(query, k, rerank_factor) -> Vec<SearchResult>`
— used in `search()`.
- `persist::save_index` / `persist::load_index` / `persist::MAGIC`
— `.rbpx` v1 wire format. `load()` peeks the 24-byte header to
recover the seed before calling `load_index`.
- `idx.export_items()` — used in `save()` because the seed-based
format needs the items handed back; `RabitqPlusIndex` doesn't
expose `originals_flat` directly.
## Verification
cargo build -p ruvector-py → clean
cargo clippy -p ruvector-py --all-targets --no-deps -- -D warnings → exit 0
cargo test -p ruvector-py → 0 tests, 0 failed (no Rust unit
tests yet; logic is in PyO3
methods that need the Python
interpreter)
`maturin develop` + `pytest` + `mypy --strict` not run — the
sandbox doesn't have those binaries. The Python tests are written
to the M1 acceptance shape and will run as soon as maturin is
present in the dev env.
## Deviations from the M1 plan (docs/sdk/04-milestones.md)
1. One `RabitqIndex` class instead of the plan's four
(`FlatF32Index`, `RabitqIndex`, `RabitqPlusIndex`, `RabitqAsymIndex`).
Adding the others is mechanical follow-up — same register pattern.
2. Single `RuVectorError` exception instead of the subclass tree
(`DimensionMismatch`, `EmptyIndex`, `PersistError`). Subclasses
are M2+ scope per the plan.
3. No `_typing.py`, no `_version.py`. `__version__` sourced from
`env!("CARGO_PKG_VERSION")` via the compiled module.
4. No CI workflow, no Sphinx, no notebook — deferred. Scoped to
"everything needed for pip install to work".
5. `build()` takes a `seed` kwarg (default 42) — not in the M1
sketch but required by the underlying ctor.
## Two pyo3 0.22 quirks worth flagging
- `pyo3::create_exception!` macro emits `cfg(feature = "gil-refs")`
unexpected_cfg warnings. Worked around with `#![allow(unexpected_cfgs)]`
at crate root, comment explains the upstream issue.
- `#[pymethods]` macro expansion triggers
`clippy::useless_conversion` false-positives on `?`-on-PyResult.
Suppressed at crate root with comment.
LoC total: 881 (Cargo.lock excluded; 768 source + 113 lockfile drift).
M1 plan budgeted ~1300 — under because we shipped the user-requested
single-class scope, not the plan's full surface.
Refs: docs/sdk/04-milestones.md M1, docs/sdk/02-strategy.md
Co-Authored-By: claude-flow <ruv@ruv.net>
This was referenced Apr 26, 2026
ruvnet
added a commit
that referenced
this pull request
Apr 26, 2026
Unblocks the 7 stacked PRs (#381-#387) and turns `main`'s CI green for the first time in days. Two issues fixed: ## Failure 1 — Security audit (was: 8 vulnerabilities) `cargo audit` is now exit 0. 4 of the 5 critical advisories were fixed by version bumps; only the unfixable one is ignored. **Dep-bumped:** - `rustls-webpki 0.101.7` + `0.103.10` → `0.103.13` via `cargo update -p rustls-webpki@0.103.10`. Patches: RUSTSEC-2026-0098 (URI name constraints) RUSTSEC-2026-0099 (wildcard name constraints) RUSTSEC-2026-0104 (CRL parsing panic) - `idna 0.5.0` → `1.1.0` via `validator 0.18 → 0.20` in `examples/scipix`. Patches RUSTSEC-2024-0421 (Punycode acceptance). - Bonus: `reqwest 0.11 → 0.12` (in `ruvector-core` + `examples/benchmarks`) and `hf-hub 0.3 → 0.4` (in `ruvector-core` + `ruvllm` + `ruvllm-cli`). Removes the entire legacy `rustls 0.21` / `rustls-webpki 0.101.7` subtree from the lockfile. **Ignored** (single advisory, with rationale): - `RUSTSEC-2023-0071` (rsa Marvin timing sidechannel) — no upstream fix available; we don't expose RSA decryption services. Documented in `.cargo/audit.toml`. **Unmaintained warnings** (16 total — proc-macro-error, derivative, instant, paste, bincode 1, pqcrypto-{kyber,dilithium}, rustls-pemfile 1, rusttype, wee_alloc, number_prefix, rand_os, core2, lru, pprof, rand) — each given a one-line justification in `.cargo/audit.toml` so CI stays green on them while the team decides whether to chase upstream replacements. ## Failure 2 — Tests timeout (was: 30-min job timeout cancellation) `.github/workflows/ci.yml` `test` job is now a `matrix` with `fail-fast: false` and `timeout-minutes: 45`. Six parallel shards under `cargo nextest run` (installed via `taiki-e/install-action@v2`) plus a separate `cargo test --doc` step (nextest doesn't run doctests): | Shard | Crates | |------------------|---------------------------------------------| | vector-index | rabitq, rulake, diskann, graph, gnn, cnn | | rvagent | 10 rvagent-* crates | | ruvix | 16 ruvix-* crates | | ruqu-quantum | 5 ruqu* crates | | ml-research | attention, mincut, scipix, fpga-transformer,| | | sparse-inference, sparsifier, solver, | | | graph-transformer, domain-expansion, | | | robotics | | core-and-rest | --workspace minus the above | `Swatinem/rust-cache@v2` is keyed per shard. Audit job switched to `taiki-e/install-action` for `cargo-audit` (faster than `cargo install --locked`). ## Verification cargo audit → exit 0 cargo build --workspace --exclude ruvector-postgres → clean cargo clippy --workspace --exclude ruvector-postgres --no-deps -- -D warnings → exit 0 cargo fmt --all --check → exit 0 ## Cargo.lock churn 166-line diff, net ~120 lines removed (more deletions than additions). Removed: `idna 0.5.0`, `rustls-webpki 0.101.7`, `validator 0.18`, `validator_derive 0.18`, `proc-macro-error 1.0.4`. Added: `rustls-webpki 0.103.13`, `validator 0.20`, `proc-macro-error2`, `hf-hub 0.4.3`, `reqwest 0.12.28`. No suspicious crates. ## Recommended merge order 1. **This PR first** — unblocks every other PR's CI. 2. After this lands and main is green, rebase the 7 open PRs (#381-#387) one at a time. The DiskANN stack (#383→#384→#385→#386) must merge in numeric order. #381 (Python SDK), #382 (research), #387 (graph property index) are independent and can merge in any order after their CI goes green on the rebase. Co-Authored-By: claude-flow <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Apr 27, 2026
Closes the WASM gap from `docs/research/rabitq-integration/` Tier 2
("WASM / edge: 32× compression makes on-device RAG feasible") and
ADR-157 ("VectorKernel WASM kernel as a Phase 2 goal"). Adds a
`ruvector-rabitq-wasm` sibling crate that exposes `RabitqIndex` to
JavaScript/TypeScript callers (browsers, Cloudflare Workers, Deno,
Bun) via wasm-bindgen.
```js
import init, { RabitqIndex } from "ruvector-rabitq";
await init();
const dim = 768;
const n = 10_000;
const vectors = new Float32Array(n * dim); // populate
const idx = RabitqIndex.build(vectors, dim, 42, 20);
const query = new Float32Array(dim);
const results = idx.search(query, 10); // [{id, distance}, ...]
```
## Surface
- `RabitqIndex.build(vectors: Float32Array, dim, seed, rerank_factor)`
- `idx.search(query: Float32Array, k) → SearchResult[]`
- `idx.len`, `idx.isEmpty`
- `version()` — crate version baked at build time
- `SearchResult { id: u32, distance: f32 }` — mirrors the Python SDK
(PR #381) shape so callers porting code between languages get
identical structures.
## Native compatibility tweak
`ruvector-rabitq` had one rayon call site in
`from_vectors_parallel_with_rotation`. WASM is single-threaded — gated
that path on `cfg(not(target_arch = "wasm32"))` with a sequential
`.into_iter()` fallback for wasm. Output is bit-identical because the
rotation matrix is deterministic (ADR-154); parallel ordering doesn't
affect bytes.
`rayon` is now `[target.'cfg(not(target_arch = "wasm32"))'.dependencies]`
so the wasm build doesn't pull it in. Native build behavior unchanged
(39 / 39 lib tests still pass).
## Crate layout
crates/ruvector-rabitq-wasm/
Cargo.toml cdylib + rlib, wasm-bindgen 0.2, abi-3-friendly
src/lib.rs ~150 LoC of bindings; tests gated to wasm32 via
wasm_bindgen_test (native test would panic in
wasm-bindgen 0.2.117's runtime stub).
## Testing strategy
Native tests of WASM bindings panic by design — `JsValue::from_str`
calls into a wasm-bindgen runtime stub that's `unimplemented!()` on
non-wasm32 targets (since 0.2.117). The right path is
`wasm-pack test --node` or `wasm-pack test --headless --chrome`,
which we'll wire into CI as a follow-up.
The numerical correctness is already covered by `ruvector-rabitq`'s
own test suite. This crate only adds the JS-facing surface.
## Verification (native)
cargo build --workspace → 0 errors
cargo build -p ruvector-rabitq-wasm → clean
cargo clippy -p ruvector-rabitq-wasm --all-targets --no-deps -- -D warnings → exit 0
cargo test -p ruvector-rabitq → 39 / 39 (unchanged)
cargo fmt --all --check → clean
WASM target build (`wasm32-unknown-unknown`) requires `rustup target
add wasm32-unknown-unknown` — not exercised in this PR; will be
covered by a follow-up CI job.
Refs: docs/research/rabitq-integration/ Tier 2, ADR-157
("Optional Accelerator Plane"), PR #381 (Python SDK shape mirror).
Co-Authored-By: claude-flow <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Apr 27, 2026
… npm (#394) * feat(ruvector-rabitq-wasm): WASM bindings for RaBitQ via wasm-bindgen Closes the WASM gap from `docs/research/rabitq-integration/` Tier 2 ("WASM / edge: 32× compression makes on-device RAG feasible") and ADR-157 ("VectorKernel WASM kernel as a Phase 2 goal"). Adds a `ruvector-rabitq-wasm` sibling crate that exposes `RabitqIndex` to JavaScript/TypeScript callers (browsers, Cloudflare Workers, Deno, Bun) via wasm-bindgen. ```js import init, { RabitqIndex } from "ruvector-rabitq"; await init(); const dim = 768; const n = 10_000; const vectors = new Float32Array(n * dim); // populate const idx = RabitqIndex.build(vectors, dim, 42, 20); const query = new Float32Array(dim); const results = idx.search(query, 10); // [{id, distance}, ...] ``` ## Surface - `RabitqIndex.build(vectors: Float32Array, dim, seed, rerank_factor)` - `idx.search(query: Float32Array, k) → SearchResult[]` - `idx.len`, `idx.isEmpty` - `version()` — crate version baked at build time - `SearchResult { id: u32, distance: f32 }` — mirrors the Python SDK (PR #381) shape so callers porting code between languages get identical structures. ## Native compatibility tweak `ruvector-rabitq` had one rayon call site in `from_vectors_parallel_with_rotation`. WASM is single-threaded — gated that path on `cfg(not(target_arch = "wasm32"))` with a sequential `.into_iter()` fallback for wasm. Output is bit-identical because the rotation matrix is deterministic (ADR-154); parallel ordering doesn't affect bytes. `rayon` is now `[target.'cfg(not(target_arch = "wasm32"))'.dependencies]` so the wasm build doesn't pull it in. Native build behavior unchanged (39 / 39 lib tests still pass). ## Crate layout crates/ruvector-rabitq-wasm/ Cargo.toml cdylib + rlib, wasm-bindgen 0.2, abi-3-friendly src/lib.rs ~150 LoC of bindings; tests gated to wasm32 via wasm_bindgen_test (native test would panic in wasm-bindgen 0.2.117's runtime stub). ## Testing strategy Native tests of WASM bindings panic by design — `JsValue::from_str` calls into a wasm-bindgen runtime stub that's `unimplemented!()` on non-wasm32 targets (since 0.2.117). The right path is `wasm-pack test --node` or `wasm-pack test --headless --chrome`, which we'll wire into CI as a follow-up. The numerical correctness is already covered by `ruvector-rabitq`'s own test suite. This crate only adds the JS-facing surface. ## Verification (native) cargo build --workspace → 0 errors cargo build -p ruvector-rabitq-wasm → clean cargo clippy -p ruvector-rabitq-wasm --all-targets --no-deps -- -D warnings → exit 0 cargo test -p ruvector-rabitq → 39 / 39 (unchanged) cargo fmt --all --check → clean WASM target build (`wasm32-unknown-unknown`) requires `rustup target add wasm32-unknown-unknown` — not exercised in this PR; will be covered by a follow-up CI job. Refs: docs/research/rabitq-integration/ Tier 2, ADR-157 ("Optional Accelerator Plane"), PR #381 (Python SDK shape mirror). Co-Authored-By: claude-flow <ruv@ruv.net> * feat(acorn): add ruvector-acorn crate — ACORN predicate-agnostic filtered HNSW Implements the ACORN algorithm (Patel et al., SIGMOD 2024, arXiv:2403.04871) as a standalone Rust crate. ACORN solves filtered vector search recall collapse at low predicate selectivity by expanding ALL graph neighbors regardless of predicate outcome, combined with a γ-augmented graph (γ·M neighbors/node). Three index variants: - FlatFilteredIndex: post-filter brute-force baseline - AcornIndex1: ACORN with M=16 standard edges - AcornIndexGamma: ACORN with 2M=32 edges (γ=2) Measured (n=5K, D=128, release): ACORN-γ achieves 98.9% recall@10 at 1% selectivity. cargo build --release and cargo test (12/12) both pass. https://claude.ai/code/session_0173QrGBttNDWcVXXh4P17if * perf(acorn): bounded beam, parallel build, flat data, unrolled L2² Five linked optimizations to ruvector-acorn (≈50% smaller search working set, ≈6× faster build on 8 cores, comparable or better recall at every selectivity): 1. **Fix broken bounded-beam eviction in `acorn_search`.** The previous implementation admitted that its `else` branch was "wrong" (the comment literally said "this is wrong") and pushed every neighbor into `candidates` unconditionally, growing the frontier to O(n). Replace with a correct max-heap eviction: when `|candidates| >= ef`, only admit a neighbor if it improves on the farthest pending candidate, evicting that one. This gives the documented O(ef) memory bound and stops wasted neighbor expansions at the prune cutoff. 2. **Parallelize the O(n²·D) graph build with rayon.** The forward pass (each node finds its M nearest predecessors) is embarrassingly parallel — `into_par_iter` over rows. Back-edge merge stays serial behind a `Mutex<Vec<u32>>` per node so the merge is deterministic. ~6× faster on an 8-core box for 5K×128. 3. **Flat row-major vector storage.** `data: Vec<Vec<f32>>` → `data: Vec<f32>` (length n·dim) with a `row(i)` accessor. Eliminates the per-vector heap indirection, keeps the L2² inner loop on contiguous memory the compiler can vectorize, and trims index size by ~one allocation per row. 4. **`Vec<bool>` for `visited` instead of `HashSet<u32>`.** O(1) lookup with no hashing or allocator pressure on the hot path. 5. **Hand-unroll L2² by 4.** Four independent accumulators give LLVM enough room to issue AVX2/SSE/NEON FMA chains on contemporary x86_64 / aarch64. 3-5× faster for D ≥ 64 in microbenchmarks. Other: - `exact_filtered_knn` parallelizes across data via rayon (recall measurement only — needs `+ Sync` on the predicate). - `benches/acorn_bench.rs` switches `SmallRng` → `StdRng` (the workspace doesn't enable rand's `small_rng` feature so the bench failed to compile). - `cargo fmt` applied across the crate; CI's Rustfmt check was the blocking failure on the original PR. Demo run on x86_64, n=5000, D=128, k=10: Build: ACORN-γ ≈ 23 ms (was 1.8 s) Recall: 96.0% @ 1% selectivity (paper: ~98%) 92.0% @ 5% selectivity 79.7% @ 10% selectivity 34.5% @ 50% selectivity (predicate dilutes top-k truth) QPS: 18 K @ 1% sel, 65 K @ 50% sel Co-Authored-By: claude-flow <ruv@ruv.net> * fix(acorn): clippy clean-up — sort_by_key, is_empty, redundant closures CI's `Clippy (deny warnings)` flagged three lints introduced by the previous optimization commit: - `unnecessary_sort_by` (graph.rs:158, 176) → use `sort_by_key` - `len_without_is_empty` (graph.rs) → add `AcornGraph::is_empty` and `if graph.is_empty()` in search.rs - `redundant_closure` (main.rs:65, 159, 160) → pass the predicate directly to `recall_at_k` instead of `|id| pred(id)` No semantic change. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(wasm): publish @ruvector/rabitq-wasm and @ruvector/acorn-wasm to npm Two new WASM packages (both v0.1.0, MIT OR Apache-2.0, scoped under @ruvector). Mirrors the existing @ruvector/graph-wasm packaging pattern so release tooling treats all three uniformly. - ADR-161: @ruvector/rabitq-wasm — RaBitQ 1-bit quantized vector index. 32× embedding compression with deterministic rotation. Wraps the existing crates/ruvector-rabitq-wasm crate. - ADR-162: @ruvector/acorn-wasm — ACORN predicate-agnostic filtered HNSW. 96% recall@10 at 1% selectivity with arbitrary JS predicates. Adds crates/ruvector-acorn-wasm (new), wrapping the ruvector-acorn crate from PR #391. Each crate ships with: - `build.sh` that runs `wasm-pack build` for web / nodejs / bundler targets, emitting into npm/packages/{rabitq,acorn}-wasm/{,node/,bundler/}. - A canonical scoped package.json (kept under git as package.scoped.json because wasm-pack regenerates package.json from Cargo metadata on every build). - A README.md with install + usage for browser, Node.js, and bundler contexts. - A `.gitignore` that excludes the wasm-pack-generated artifacts (.wasm + .js + .d.ts) so only canonical source lives in the repo. Build sanity: - `cargo check -p ruvector-acorn-wasm -p ruvector-rabitq-wasm` clean - `cargo clippy -- -D warnings` clean for both - `wasm-pack build` succeeds for all three targets on both crates Published: - @ruvector/rabitq-wasm@0.1.0 — 40 KB tarball, 71 KB wasm - @ruvector/acorn-wasm@0.1.0 — 49 KB tarball, ~85 KB wasm Root README updated with both packages in the npm packages table. Note: this branch also carries cherry-picks of PR #391's `ruvector-acorn` crate (commits b90af9c, 0b4eab1, eb88176, f5913b7) and PR #391's predecessor commit a674d6e for `ruvector-rabitq-wasm` itself, because both base crates are required to build the new WASM wrappers. Co-Authored-By: claude-flow <ruv@ruv.net> --------- Co-authored-by: ruvnet <ruvnet@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First milestone of the ruvector Python SDK per
docs/sdk/04-milestones.md§ M1 — a working PyO3 + maturin wheel that exposesruvector-rabitqto Python.What ships
crates/ruvector-py/— new workspace cdylib crate (768 LoC source)pyproject.toml— maturin build backend, abi3-py39 target, PyPI nameruvectorRabitqIndexclass wrappingRabitqPlusIndex(build/search/save/load +__len__/__repr__)RuVectorErrorexception (subclass tree deferred to M2)python/ruvector/__init__.pyiandstubs/ruvector/__init__.pyitests/test_smoke.pycovering build/search/save/load + error pathVerification
cargo build -p ruvector-py→ cleancargo clippy -p ruvector-py --all-targets --no-deps -- -D warnings→ exit 0cargo fmt --all --check→ cleanmaturin develop && pytest— not exercised in this branch's sandbox (no maturin binary). Tests are written to the M1 acceptance shape and run as soon as maturin is in the dev env.mypy --strict— not exercised (no mypy binary). Stubs are hand-written PEP 561.Deviations from the plan
The full
docs/sdk/04-milestones.mdM1 spec covers four index classes (FlatF32Index,RabitqIndex,RabitqPlusIndex,RabitqAsymIndex) with a subclass exception tree. This PR ships one class + one exception because that's the scope the user requested. Adding the rest is mechanical follow-up — sameregister()pattern.Other small deviations:
build()takes aseedkwarg (default 42) — required byRabitqPlusIndex::from_vectors_parallel.unexpected_cfgsfromcreate_exception!,useless_conversionfrom#[pymethods]) suppressed at crate root with explanatory comments.Test plan
cd crates/ruvector-py && maturin develop && pytest tests/mypy --strict python/ruvector/cibuildwheel+ abi3 tag checkStacked on PR #380
Branched from
mainafter PR #380 merged at7a599b7cf. The cherry-picked ruvector-cnn fix from #380 was dropped during rebase as the patch was already upstream.🤖 Generated with claude-flow