Skip to content

feat(csr)!: store the csr inside each lance dataset and pack edge rows - #109

Merged
prrao87 merged 9 commits into
mainfrom
feat/persist-csr
Nov 2, 2026
Merged

feat(csr)!: store the csr inside each lance dataset and pack edge rows#109
prrao87 merged 9 commits into
mainfrom
feat/persist-csr

Conversation

@prrao87

@prrao87 prrao87 commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

User benefits

  • Faster engine starts. With persist_csr=True, the CSR is computed once, stored inside each Lance relationship dataset, and loaded by later engines instead of being rebuilt in memory from every edge. On GDC SF1, engine start drops from about 1.7 s to 0.46 s. Writes made since the stored version are applied as a small delta, so the stored CSR stays useful as the data changes.
  • Small files on disk. The stored CSR is zstd-compressed: 134 MiB for SF1's 17.3 million edges, against 711 MiB uncompressed.
  • Less memory, whether or not the CSR is stored. Each edge's row address is now bit-packed in blocks of 128 values instead of taking a full 64 bits. That cuts the in-memory CSR at SF1 from 926 to 732 MiB, in both memory and lazy mode, with no change in query times.
  • Opt-in and easy to undo. Without persist_csr, nothing is written. delete_stored_csr() removes the stored files to go back to building in memory only.

Why

Every engine builds its CSR from a full scan of the relationship tables, even when nothing changed since the last run: about 1.6 s of start-up on GDC SF1, repeated by every process. The CSR also held each edge's row as a raw 64-bit address, the larger half of its arrays.

This implements Stage 2 of #72 (store the CSR and load it at start-up) and the edge-row half of #79 (store CSR data in fewer bytes). Node indexes aren't stored yet, so #72 stays open for them; #79 stays open for neighbours (see below).

What changed

Stored CSRs (persist_csr=True). An engine over Lance datasets stores each CSR it fully builds inside the relationship dataset's own directory, at _lance_csr/<version>.arrow, and later engines load it instead of building.

  • Changes since the stored version apply as a delta, under the same 25% threshold as checkout_latest(). The file records the fragments it was built from, so this keeps working after compaction and cleanup_old_versions(), which leaves the folder alone.
  • Every full build is stored, including those from checkout_latest() and optimize_csr(), and older files are deleted. A file that doesn't parse or doesn't match is rebuilt, never trusted. A dataset that can't be written builds as before.
  • delete_stored_csr() removes the stored files, for going back to in-memory builds only. The engine keeps the CSR it already holds.
  • csr_stats() reports update "loaded" for a CSR read from a file.
  • The file is an Arrow IPC file of zstd-compressed UInt64 arrays in lance-csr's own layout (documented in the internals pages); the arrays aren't meant to be read on their own.
  • PyArrow inputs, including Lance data loaded with to_table(), are never stored, so the benchmarks' memory mode doesn't use this.

Packed edge rows. Edge rows are now stored in blocks of 128 values: each block keeps its minimum and a bit width, and each value is its offset from the minimum, so any position stays directly readable. Only hops that name their relationship, and deletes applied by checkout_latest(), unpack them.

GDC SF1, both directions Before After
CSR arrays in memory 613 MiB 419 MiB
Whole in-memory CSR (arrays and key maps) 926 MiB 732 MiB
Stored files on disk 134 MiB

Tried and rejected: packing neighbours the same way roughly halved them, but every hop reads neighbours and lookup-heavy queries got slower: custom Q30 by about 22%. #79's u32 offset idea doesn't fit SF1 either, since its node keys span up to 2^45. Uncompressed files were 711 MiB at SF1; zstd brought them to 150 MiB, and storing the packed arrays to 134 MiB.

User impact. Python gains persist_csr and delete_stored_csr(), both experimental and off unless used. In Rust (marked experimental), EngineOptions and CypherEngine::with_options are new, CsrIndex::edge_rows() returns an iterator instead of a slice, and CsrIndex::from_parts is replaced by from_packed. The CSR guide has a new "Storing the CSR" section, clearer guidance on checkout_latest() against optimize_csr(), and measured memory figures.

Renamed refresh() to checkout_latest() (merged in from #110), after Lance's LanceDataset.checkout_latest(), so the engine pairs checkout_latest() with optimize_csr() the way Lance pairs checkout_latest() with optimize_indices(). refresh() still works as a deprecated alias: Python warns with DeprecationWarning and Rust marks it #[deprecated].

The root README is trimmed to the essentials. It no longer carries benchmark tables comparing engines, which go stale quickly; it links the two benchmark repositories instead, and docs/sync_benchmarks.py now updates only the docs site's benchmarks page.

Validation

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass. New tests cover loading, catching up with appends and deletes, corrupt files, optimize_csr() storing a new base, delete_stored_csr(), and packed arrays (round trips, partial reads, and refusing inconsistent parts).
  • make test (1672 passed) and make lint pass, as do the strict Zensical build, docs/check_snippets.py and typos. New Python tests cover loading after compaction plus cleanup_old_versions(), rebuilding past the threshold, read-only datasets, PyArrow input and deletion.
  • CI is pending.

Start-up, GDC SF1 read lazily (23 relationship types, 17.3M edges), fresh processes, medians:

Engine start Peak memory
main 1.65 s 1.56 to 1.60 GiB
Branch, persist_csr off 1.71 s 1.37 to 1.42 GiB
Branch, first start (build and store) 2.63 s 1.25 to 1.46 GiB
Branch, later starts (load) 0.46 s 1.06 to 1.12 GiB

python/benchmarks/csr_persist.py measures the same on a synthetic graph: at 10M edges, a load took 55 ms against 1.18 s for a build (measured with the first stage, before edge rows were packed).

Benchmarks. Release wheels from main (d7634f8) and this branch (the code of 94769f1; only comments and docs changed after), run with the CONTRIBUTING.md harness, main and branch alternating twice per mode. In lazy mode a third build loads stored CSRs: the same branch wheel with persist_csr=True set by a sitecustomize.py on PYTHONPATH, without editing the runners; a warm-up run stored the CSRs, and both timed runs loaded all 23. Every answer check passed. graph-benchmark-ldbc f1d67be, graph-benchmark-ldbc-complex 552771f. Apple M5, 24 GiB, macOS 26.6.2.

Suite, mode Branch vs main (geomean) Loading vs main Gap between main runs
IC1 to IC14, memory +2.8% 2.0%
IC1 to IC14, lazy -0.3% -1.0% 1.6%
Custom 30, memory +0.5% 0.6%
Custom 30, lazy -0.2% -0.4% 1.6%

Queries that moved beyond the gap between main runs:

  • IC12, memory, +15.6%: the bimodal query from perf(planner): walk pattern predicates on the csr and keep comprehension rows once #108, which lands in a fast or slow mode per process. In a follow-up A/B with three processes per build, main ran 17.4/18.0/20.2 ms and the branch 17.6/17.7/18.3 ms.
  • IC5, memory, +6.3%: the only one of these that reads edge rows. In a five-process A/B, the branch's median was 2.8% faster than main.
  • IC3, memory, +6.2%: in the five-process A/B, +2.9% (25.6 to 27.6 ms against 25.1 to 25.7 ms). Its hops don't read edge rows, and the committed first stage showed the same, so I can't attribute it to packing. Small, but listed for review.
  • Custom Q30, memory, +4.0% (lazy +1.6%): far below the 22% when neighbours were packed. Custom Q27, loading, +11.4%: the in-memory CSR is the same as the branch's, which was -0.9%; it was -0.8% in an earlier run.
Every query, both suites and modes (mean of two runs per build)

graph-benchmark-ldbc-complex (IC1 to IC14), memory

Query main (ms) branch (ms) Change
IC1 23.53 22.90 -2.7%
IC2 2.46 2.54 +3.5%
IC3 25.30 26.87 +6.2%
IC4 5.93 6.00 +1.2%
IC5 80.02 85.03 +6.3%
IC6 22.72 23.87 +5.1%
IC7 9.67 9.76 +0.9%
IC8 2.23 2.29 +2.6%
IC9 39.12 39.64 +1.3%
IC10 24.72 24.56 -0.7%
IC11 2.99 2.94 -1.7%
IC12 17.97 20.78 +15.6%
IC13 1.06 1.09 +2.9%
IC14 49.08 49.09 +0.0%

graph-benchmark-ldbc-complex (IC1 to IC14), lazy

Query main (ms) branch (ms) Change Loading a stored CSR (ms) Change
IC1 31.91 31.43 -1.5% 31.42 -1.5%
IC2 25.80 25.60 -0.8% 25.28 -2.0%
IC3 166.94 165.40 -0.9% 164.27 -1.6%
IC4 12.87 12.61 -2.0% 12.53 -2.6%
IC5 110.07 110.83 +0.7% 110.64 +0.5%
IC6 22.89 22.65 -1.1% 22.20 -3.0%
IC7 14.53 14.57 +0.3% 14.40 -0.9%
IC8 3.29 3.27 -0.8% 3.26 -0.9%
IC9 307.13 302.17 -1.6% 303.53 -1.2%
IC10 34.01 34.14 +0.4% 33.76 -0.7%
IC11 5.37 5.31 -1.3% 5.31 -1.3%
IC12 20.76 21.39 +3.1% 21.00 +1.2%
IC13 1.42 1.42 +0.0% 1.42 -0.5%
IC14 49.79 50.17 +0.8% 50.13 +0.7%

graph-benchmark-ldbc (30 custom queries), memory

Query main (ms) branch (ms) Change
Q1 0.65 0.61 -5.8%
Q2 0.52 0.55 +4.3%
Q3 0.64 0.65 +1.4%
Q4 0.51 0.51 +0.5%
Q5 0.77 0.77 -0.5%
Q6 0.42 0.44 +3.1%
Q7 1.02 1.03 +1.4%
Q8 0.63 0.63 +0.8%
Q9 0.80 0.81 +0.5%
Q10 0.71 0.72 +0.6%
Q11 1.56 1.53 -1.7%
Q12 0.75 0.75 +0.9%
Q13 0.79 0.82 +4.9%
Q14 0.53 0.53 +1.9%
Q15 0.56 0.56 +0.0%
Q16 0.65 0.66 +0.7%
Q17 0.91 0.91 +0.2%
Q18 0.69 0.70 +1.0%
Q19 0.92 0.91 -1.0%
Q20 1.94 1.92 -1.0%
Q21 0.77 0.75 -2.2%
Q22 0.70 0.70 -0.2%
Q23 0.73 0.75 +3.7%
Q24 0.69 0.68 -0.2%
Q25 0.79 0.79 +0.3%
Q26 0.74 0.73 -1.3%
Q27 0.85 0.85 -0.1%
Q28 1.39 1.40 +1.0%
Q29 0.69 0.68 -1.9%
Q30 40.60 42.23 +4.0%

graph-benchmark-ldbc (30 custom queries), lazy

Query main (ms) branch (ms) Change Loading a stored CSR (ms) Change
Q1 1.16 1.25 +8.1% 1.12 -2.8%
Q2 1.44 1.38 -4.5% 1.37 -4.8%
Q3 0.78 0.81 +3.1% 0.78 -0.6%
Q4 0.77 0.81 +4.9% 0.78 +1.1%
Q5 2.11 2.21 +4.8% 2.11 -0.0%
Q6 0.71 0.73 +2.8% 0.72 +0.1%
Q7 1.05 1.10 +4.3% 1.05 +0.3%
Q8 1.73 1.74 +0.9% 1.73 +0.0%
Q9 1.32 1.33 +0.8% 1.33 +0.5%
Q10 0.92 0.92 +0.2% 0.92 +0.2%
Q11 1.97 1.96 -0.4% 1.95 -1.4%
Q12 6.76 6.68 -1.2% 6.70 -0.8%
Q13 1.07 1.03 -3.4% 1.05 -2.3%
Q14 0.67 0.68 +0.9% 0.68 +0.7%
Q15 0.75 0.74 -1.6% 0.74 -0.8%
Q16 1.12 1.07 -3.8% 1.08 -3.1%
Q17 1.50 1.37 -8.4% 1.39 -7.1%
Q18 1.17 1.13 -3.5% 1.13 -3.1%
Q19 1.34 1.32 -1.4% 1.33 -1.0%
Q20 16.41 16.44 +0.2% 16.42 +0.0%
Q21 0.82 0.82 -0.8% 0.83 +1.0%
Q22 0.91 0.90 -1.5% 0.92 +0.8%
Q23 1.25 1.21 -2.6% 1.23 -1.1%
Q24 1.05 1.06 +0.6% 1.05 -0.2%
Q25 1.22 1.17 -4.3% 1.19 -2.7%
Q26 1.05 1.04 -1.0% 1.07 +1.7%
Q27 1.29 1.27 -0.9% 1.43 +11.4%
Q28 2.02 2.04 +1.1% 2.08 +3.2%
Q29 1.01 1.02 +0.7% 1.02 +0.3%
Q30 41.42 42.07 +1.6% 41.60 +0.4%

🤖 Generated with Claude Code

@prrao87
feat(csr): store the csr inside each lance dataset with persist_csr
a3aedfc 
Engines rebuild every CSR from a full scan of the relationship tables at
start-up. With persist_csr=True, each full build of a Lance relationship
dataset is written to <dataset>/_lance_csr/<version>.arrow, a zstd
compressed Arrow IPC file, and later engines load it instead of building.
Changes since the stored version apply as a delta under the usual 25%
threshold, read from the file's recorded fragments, so this survives
compaction and cleanup_old_versions(). Unusable files are rebuilt, and
failed writes fall back to an ordinary build.

delete_stored_csr() removes the stored files. csr_stats() reports
update "loaded" for a CSR read from a file. CsrIndex now keeps its arrays
as Arrow buffers, and Rust gains EngineOptions and
CypherEngine::with_options.

On GDC SF1 read lazily, engine start goes from 1.55 s to 0.58 s, the
files take 150 MiB, and query times are unchanged.
@prrao87
perf(csr): pack edge rows in blocks and store the csr packed
94769f1 
Each CSR slot held its edge row as a raw 64-bit Lance row address.
Edge rows are now packed in blocks of 128 values, each stored as its
offset from the block's minimum in as few bits as the block needs, with
every position still directly readable. On GDC SF1 that cuts the CSR's
arrays from 613 to 419 MiB, and the whole in-memory CSR from 926 to
732 MiB. Plain hops never unpack edge rows.

Neighbours stay a plain array: packing them too slowed lookup-heavy
queries (custom Q30 by about 22%) for a smaller saving.

Stored CSR files now hold the arrays as they sit in memory (format 2,
still zstd compressed), so a load only decompresses and checks them:
134 MiB at SF1, and engine start drops from 0.58 to 0.46 s.

The CSR guide now contrasts refresh() and optimize_csr() in a table,
and its memory figures are measured rather than estimated.
@prrao87
docs(csr): correct the peak memory units in the storing cost table
459c066
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Deploying lance-csr with  Cloudflare Pages  Cloudflare Pages

Latest commit: 706e181
Status: ✅  Deploy successful!
Preview URL: https://9736eba7.lance-csr.pages.dev
Branch Preview URL: https://feat-persist-csr.lance-csr.pages.dev

View logs

@prrao87 prrao87 mentioned this pull request Nov 2, 2026
Merged
@prrao87
refactor(engine): rename refresh() to checkout_latest() to match lance (
36200df 
#110)

* refactor(engine): rename refresh() to checkout_latest() to match lance

Lance moves a dataset to its newest version with checkout_latest() and
has no refresh(). CypherEngine.refresh() does the same for every dataset
the engine reads, so it is now checkout_latest(), in Rust and Python,
which also pairs it with optimize_csr() the way Lance pairs
checkout_latest() with optimize_indices().

refresh() stays as a deprecated alias: Python warns with
DeprecationWarning, and Rust marks it #[deprecated]. Docs, examples,
tests and benchmarks use the new name, and the lance_snapshot_refresh.py
example is now lance_snapshots.py.

* docs: say checkout_latest() follows lance and optimize_csr() rebuilds the csr
@prrao87
prrao87 marked this pull request as ready for review November 2, 2026 17:50
@prrao87
docs(csr): rewrite keeping the csr current for newcomers
706e181 
Explain what checkout_latest() and optimize_csr() do and why before how, drop the remaining refresh wording, and add the what-and-why-first rule to the docs style guide.
@prrao87
prrao87 merged commit 44cf6bd into main Nov 2, 2026
8 checks passed
@prrao87
prrao87 deleted the feat/persist-csr branch November 2, 2026 21:25
@prrao87 prrao87 mentioned this pull request Nov 2, 2026
Merged
prrao87 added a commit that referenced this pull request Nov 2, 2026
@prrao87
chore: bump version 0.5.1 → 0.6.0 (#111)
e2339cc 
Minor bump: #109 breaks the Python and Rust API (refresh() is renamed to checkout_latest() and CsrIndex's accessors changed), so until 1.0 it takes a minor version.
This was referenced Nov 3, 2026
Closed
Merged
Open
Open
Open
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@prrao87