Repository navigation
ci: key the python rust cache on the root crates' manifests - #116
Merged
Merged
Conversation
Deploying lance-csr with
|
| Latest commit: |
e873c0e
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://fdf42ff7.lance-csr.pages.dev |
| Branch Preview URL: | https://ci-python-rust-cache-key.lance-csr.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Python test job spends 270 to 320 s in "Build Python extension", while the tests themselves take about 35 s with
pytest -n auto. The build is slow because the job's rust-cache is stale and never refreshed.rust-cache builds its key from the
Cargo.tomlof each member of the workspaces it's given (versions blanked) plus the crates.io packages in theirCargo.lock. The step only listedcrates/lance-csr-python, which is excluded from the root workspace, so its key never includedcrates/lance-csr/Cargo.tomlorcrates/lance-csr-catalog/Cargo.toml, which the bindings use as path dependencies.#109 added
ipc_compressiontoarrow's features incrates/lance-csr/Cargo.toml. That added no crates.io packages, so the keyv0-rust-lance-csr-python-Linux-x64-113ee5d7-125ac86f, saved on 2026-10-30, still matched exactly. rust-cache doesn't re-save an exact match, so every run since restores a cache built with the old features and recompilesarrow, DataFusion and Lance. The Rust jobs aren't affected: their workspace is the root one, so the feature change gave them a new key.What
List both workspaces in the step:
The key now hashes the root members' manifests too, still with versions blanked, so release version bumps don't bust the cache. This job never builds the root workspace, so
./targetdoesn't exist; rust-cache's target cleanup on save and restore is wrapped in try/catch, and the cache save skips paths that don't exist.Effect
This PR's run computes a new key but can't save it (
save-ifis main only), so its build is still cold. The speed-up shows from the second main run after merge: the first main run builds cold and saves the cache, later runs restore it.Testing
113ee5d7-125ac86f.🤖 Generated with Claude Code