Repository navigation
refactor(engine): rename refresh() to checkout_latest() to match lance - #110
Merged
Merged
Conversation
Lance moves a dataset to its newest version with checkout_latest() and has no refresh(). CypherEngine.refresh() does the same for every dataset the engine reads, so it is now checkout_latest(), in Rust and Python, which also pairs it with optimize_csr() the way Lance pairs checkout_latest() with optimize_indices(). refresh() stays as a deprecated alias: Python warns with DeprecationWarning, and Rust marks it #[deprecated]. Docs, examples, tests and benchmarks use the new name, and the lance_snapshot_refresh.py example is now lance_snapshots.py.
Deploying lance-csr with
|
| Latest commit: |
4500e39
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e69eaaef.lance-csr.pages.dev |
| Branch Preview URL: | https://refactor-checkout-latest.lance-csr.pages.dev |
prrao87
marked this pull request as ready for review
November 2, 2026 17:34
prrao87
added a commit
that referenced
this pull request
Nov 2, 2026
#109) * feat(csr): store the csr inside each lance dataset with persist_csr Engines rebuild every CSR from a full scan of the relationship tables at start-up. With persist_csr=True, each full build of a Lance relationship dataset is written to <dataset>/_lance_csr/<version>.arrow, a zstd compressed Arrow IPC file, and later engines load it instead of building. Changes since the stored version apply as a delta under the usual 25% threshold, read from the file's recorded fragments, so this survives compaction and cleanup_old_versions(). Unusable files are rebuilt, and failed writes fall back to an ordinary build. delete_stored_csr() removes the stored files. csr_stats() reports update "loaded" for a CSR read from a file. CsrIndex now keeps its arrays as Arrow buffers, and Rust gains EngineOptions and CypherEngine::with_options. On GDC SF1 read lazily, engine start goes from 1.55 s to 0.58 s, the files take 150 MiB, and query times are unchanged. * perf(csr): pack edge rows in blocks and store the csr packed Each CSR slot held its edge row as a raw 64-bit Lance row address. Edge rows are now packed in blocks of 128 values, each stored as its offset from the block's minimum in as few bits as the block needs, with every position still directly readable. On GDC SF1 that cuts the CSR's arrays from 613 to 419 MiB, and the whole in-memory CSR from 926 to 732 MiB. Plain hops never unpack edge rows. Neighbours stay a plain array: packing them too slowed lookup-heavy queries (custom Q30 by about 22%) for a smaller saving. Stored CSR files now hold the arrays as they sit in memory (format 2, still zstd compressed), so a load only decompresses and checks them: 134 MiB at SF1, and engine start drops from 0.58 to 0.46 s. The CSR guide now contrasts refresh() and optimize_csr() in a table, and its memory figures are measured rather than estimated. * docs(csr): correct the peak memory units in the storing cost table * docs: trim the readme and link the benchmark repositories * docs(csr): name the optimize_csr() section after what it does * docs(csr): lead with the two ways to keep the csr current * docs(csr): explain why the csr falls behind and what each call does * refactor(engine): rename refresh() to checkout_latest() to match lance (#110) * refactor(engine): rename refresh() to checkout_latest() to match lance Lance moves a dataset to its newest version with checkout_latest() and has no refresh(). CypherEngine.refresh() does the same for every dataset the engine reads, so it is now checkout_latest(), in Rust and Python, which also pairs it with optimize_csr() the way Lance pairs checkout_latest() with optimize_indices(). refresh() stays as a deprecated alias: Python warns with DeprecationWarning, and Rust marks it #[deprecated]. Docs, examples, tests and benchmarks use the new name, and the lance_snapshot_refresh.py example is now lance_snapshots.py. * docs: say checkout_latest() follows lance and optimize_csr() rebuilds the csr * docs(csr): rewrite keeping the csr current for newcomers Explain what checkout_latest() and optimize_csr() do and why before how, drop the remaining refresh wording, and add the what-and-why-first rule to the docs style guide.
prrao87
added a commit
that referenced
this pull request
Nov 7, 2026
* refactor!: remove the deprecated refresh()
checkout_latest() replaced refresh() two releases ago, and refresh() has
warned with DeprecationWarning since. Remove it from the Rust engine, the
Python bindings, the type stub and the Python CypherEngine, along with its
deprecation test and docs note, and have the wheel check call
checkout_latest().
* docs: describe checkout_latest() instead of refresh
Drop the remaining refresh wording now that refresh() is gone: the
internals pages and their nav entry ("Indexes and updates"), Rust doc
comments, test names and the micro-benchmarks, whose output keys become
checkout_ms, checkout_unchanged_ms and checkout_after_append_ms. Most of
the docs change was written after #110 merged and never reached main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Lance moves a dataset to its newest version with
checkout_latest()and has norefresh().CypherEngine.refresh()does the same job for every Lance dataset the engine reads, so its name made lance-csr look different from Lance where it isn't. With the new name the engine pairs with Lance the same way Lance's own calls pair up:LanceDataset.checkout_latest()CypherEngine.checkout_latest()optimize_indices()optimize_csr()This is stacked on #109 (base
feat/persist-csr), since both change the same CSR guide pages. Retarget it tomainonce #109 merges.What changed
checkout_latest()replacesrefresh()onCypherEngine, in Python and Rust, with the same behaviour.refresh()stays as a deprecated alias for a release. In Python it warns withDeprecationWarningand then callscheckout_latest(); in Rust it's marked#[deprecated].refresh()" note, and its anchor is now#checkout-latest. Thelance_snapshot_refresh.pyexample is renamedlance_snapshots.py.checkout_latest()too; "Indexes and refresh" is now "Indexes and updates". The only mention ofrefresh()left in the docs is the note about the old name. Benchmark output keys likerefresh_msstay, so old results remain comparable.User impact. Existing code keeps working. Calls to
refresh()now warn withDeprecationWarningin Python and a deprecation warning in Rust, so code that turns warnings into errors (pytest'serror::DeprecationWarning, for example) needsrefresh()changed tocheckout_latest().Validation
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warningsandcargo test --workspacepass.make test(1673 passed) andmake lintpass. A new test checks thatrefresh()warns and still moves the engine to the latest version; withDeprecationWarningtreated as an error in the test config, any leftoverrefresh()call in the suite would fail.docs/check_snippets.pyandtypospass, and the renamed example runs.🤖 Generated with Claude Code