Skip to content

ci: key the python rust cache on the root crates' manifests - #116

Merged
prrao87 merged 1 commit into
mainfrom
ci/python-rust-cache-key
Nov 3, 2026
Merged

prrao87 merged 1 commit into
mainfrom
ci/python-rust-cache-key

Conversation

@prrao87

@prrao87 prrao87 commented Nov 3, 2026

Copy link
Copy Markdown
Member

Why

The Python test job spends 270 to 320 s in "Build Python extension", while the tests themselves take about 35 s with pytest -n auto. The build is slow because the job's rust-cache is stale and never refreshed.

rust-cache builds its key from the Cargo.toml of each member of the workspaces it's given (versions blanked) plus the crates.io packages in their Cargo.lock. The step only listed crates/lance-csr-python, which is excluded from the root workspace, so its key never included crates/lance-csr/Cargo.toml or crates/lance-csr-catalog/Cargo.toml, which the bindings use as path dependencies.

#109 added ipc_compression to arrow's features in crates/lance-csr/Cargo.toml. That added no crates.io packages, so the key v0-rust-lance-csr-python-Linux-x64-113ee5d7-125ac86f, saved on 2026-10-30, still matched exactly. rust-cache doesn't re-save an exact match, so every run since restores a cache built with the old features and recompiles arrow, DataFusion and Lance. The Rust jobs aren't affected: their workspace is the root one, so the feature change gave them a new key.

What

List both workspaces in the step:

workspaces: |
  . -> target
  crates/lance-csr-python -> target

The key now hashes the root members' manifests too, still with versions blanked, so release version bumps don't bust the cache. This job never builds the root workspace, so ./target doesn't exist; rust-cache's target cleanup on save and restore is wrapped in try/catch, and the cache save skips paths that don't exist.

Effect

This PR's run computes a new key but can't save it (save-if is main only), so its build is still cold. The speed-up shows from the second main run after merge: the first main run builds cold and saves the cache, later runs restore it.

Testing

  • The Python test run on this PR reports a new rust-cache key instead of a full match on 113ee5d7-125ac86f.

🤖 Generated with Claude Code

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying lance-csr with  Cloudflare Pages  Cloudflare Pages

Latest commit: e873c0e
Status: ✅  Deploy successful!
Preview URL: https://fdf42ff7.lance-csr.pages.dev
Branch Preview URL: https://ci-python-rust-cache-key.lance-csr.pages.dev

View logs

@prrao87
prrao87 merged commit d4b4e79 into main Nov 3, 2026
4 checks passed
@prrao87
prrao87 deleted the ci/python-rust-cache-key branch November 3, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant