Reuse graph integration build outputs in CI

[dnikolayev]

🌟 What is the purpose of this PR?

Reduce graph build overhead during integration startup by reusing a validated graph executable. This is a draft implementation; performance savings and full integration equivalence have not yet been established.

The starting baseline is 67f60d5446ed3224609161f938e1b36bc9d62f89. Application source, dependencies and feature selection are unchanged. The original graph command remains cargo build --bin hash-graph --all-features.

🔗 Related links

• Proposal: reuse graph integration build outputs
• Existing frontend caching work, excluded from this change
• Candidate fork smoke run
• Baseline fork smoke run
• Fork-only benchmark generator

🚫 Blocked by

• Successful fresh-runner integration restoration and equivalent executed test results
• Three successful paired cold and warm measurements per variant, plus changed-input rebuild evidence
• Complete Test workflow timings, critical-path analysis and aggregate job-minutes
• Maintainer approval to run upstream Test/Lint and other required checks
• A maintainer-provided Linear ticket for the required PR title check
• Maintainer review; preview authorization checks are also blocked

🔍 What does this change?

• Adds exact-key GitHub Actions caching for only the graph executable and its checksum manifest. No producer job is added.
• After pruning, the helper replaces the compile entrypoint only in the supported CI checkout. The tracked application scripts and non-opt-in builds retain their existing behavior.
• Binds the effective pruned source tree, transitive local Cargo files and generated inputs, Cargo resolution/lock/configuration, actual compiler/tool identities, explicit compiler environment, runner identity and absolute checkout path. Unsupported inputs use the normal build path.
• Validates the key, checksum, executable permissions and ELF architecture before an atomic restoration. Missing or malformed payloads rebuild. Inputs are checked again after compilation before storing an output.
• Keeps Turbo compile caching disabled and preserves migrations, service startup, healthchecks, tests, coverage, retries and timeouts. Adds a required focused helper/timing check.

Caches have no restore prefixes. GitHub's repository/ref cache boundaries remain in force; no privileged execution of fork code or upstream secret sharing is introduced. Environment values are hashed rather than written to receipts.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

• are internal and do not require a docs change

The helper includes implementation and reproduction documentation for contributors.

🕸️ Does this require a change to the Turbo Graph?

• do not affect the execution graph

The graph compile task remains uncached; only its supported CI invocation is guarded.

⚠️ Known issues

The initial input boundary is conservative: unrelated tracked edits, different pruning or runner identities can miss. Linux build and fresh-runner backend restoration have passed. Metadata, hashing and transfer costs remain part of the repeated measurements. No saving is claimed from the historical run or from focused unit tests.

The fork harness uses identical credential-free settings for both variants: no upstream Turbo/sccache access, local Turbo cache on fresh runners, and CARGO_INCREMENTAL=0. The primary baseline retains the original Cargo command and environment; the candidate pays its preparation, validation and cache costs. Earlier sealed-environment baseline runs are classified as diagnostic, not net performance evidence. Smoke scope contains only Playwright and backend integration and cannot establish complete Test workflow savings. The generator also supports the complete push-profile Test selection/gates. The first full net baseline passed on fixed source f4dc7dd3faa93fd22120ba3ce48ece73592b5324: 32m 09s to final job completion and 440m 37s aggregate job time. Playwright took 30m 22s and was the last substantive job; backend integration took 19m 01s. All 76 executed jobs passed, with one expected publish dry-run skip. This single baseline establishes no performance saving. Every source checkout is pinned so warm repetitions retain identical selection.

Workflow/job	Cache condition	Samples	Before median	After median	Seconds saved	Percent saved	Run links
Playwright	Cold / warm	0 paired	Pending	Pending	Unmeasured	Unmeasured	Initial smoke linked above
Backend integration	Cold / warm	0 paired	Pending	Pending	Unmeasured	Unmeasured	Initial smoke linked above
Other affected integration jobs	Cold / warm	0 paired	Pending	Pending	Unmeasured	Unmeasured	Pending
Complete Test workflow	Cold / warm	0 paired	Pending	Pending	Unmeasured	Unmeasured	Pending

Diagnostic results on source 12556720a1c8a455ac8922899579f95bd5807459 (one successful cold run per variant, sealed-environment baseline):

Job	Baseline	Candidate	Observed difference
Playwright	24m 56s	31m 07s	Candidate 6m 11s slower
Backend integration	21m 29s	19m 06s	Candidate 2m 23s faster

Baseline diagnostic and candidate diagnostic both executed 32 Playwright tests with 8 skips and 110 backend tests plus 48 snapshot tests with 9 skips. These single samples are inconclusive and are excluded from primary net-savings claims. Later graph Cargo invocations in these sealed diagnostics compiled no crates and spent minutes waiting for shared Cargo locks; the original fallback smoke runs had much shorter repeat waits. The sealed wrapper can affect scheduling, so those waits cannot be presented as normal-baseline compilation savings. A later implementation change removes duplicate payload hashing; it still needs repeated measurement.

The complete diagnostic baseline passed all gates: 31m 48s to final job completion (31m 49s GitHub-reported total run time) and 455.45 aggregate job-minutes. It used the sealed baseline and overlapped diagnostic runs, so it is feasibility evidence rather than a primary baseline.

The fresh-runner diagnostic has confirmed an exact Actions cache hit and executable restoration for backend integration, with the same executed counts and a 14m 18s job. Playwright also passed with 32 tests and 8 skips in 31m 37s, but its initial Actions restore missed: only the machine fingerprint changed. The run is classified as primed with 1 of 2 actual Actions hits, not a fully restored warm run. It does not establish a repeated warm comparison.

The initial candidate smoke fell back to ordinary Cargo and passed; it did not exercise reuse. The initial baseline smoke retained a Playwright extension service-worker setup timeout (31 passed, 8 skipped, 1 failed). Both attempts remain retained and excluded from performance summaries.

🐾 Next steps

Complete the fork execution proof and paired measurements; retain failed attempts and cache receipts. Report regressions and inconclusive comparisons alongside improvements. Maintainer-run validation may still be necessary to establish performance with upstream external caches. Frontend reuse is outside this PR.

🛡 What tests cover this?

• 16 colocated stdlib unit tests cover direct/transitive/generated/non-Rust inputs, dependency/tool/environment invalidation, unsupported flags/profiles/paths, exact restoration, corruption, missing outputs, permissions, failed builds and preparation fallback.
• 4 timing tests cover parallel wall time versus aggregate work, missing/failed/duplicate evidence, pagination, medians/ranges and scope separation, and current native Turbo task timing with cached/incomplete/malformed evidence rejected.
• Focused Python checks, YAML parsing and formatting checks passed locally. The updated candidate and baseline smoke integrations passed with matching counts. The complete diagnostic baseline passed; repeated net comparisons remain pending.

❓ How to test this?

1. Run python3 .github/actions/graph-build-cache/graph_build_cache_test.py and python3 .github/scripts/integration_build_timings_test.py.
2. Use the linked generator for identical pinned-source baseline/candidate runs. Keep each sample's workflow and branch unchanged for its cold/warm repetition; verify actual cache receipts instead of assuming a hit.
3. Retain each run/attempt and all job API pages, toolchain/runner details, phase events and executed test counts. Run python3 .github/scripts/integration-build-timings.py manifest.json to derive comparisons. The script and helper README document the manifest and retrieval commands.
4. Verify live migrations, readiness and integration outcomes, then repeat with a relevant input change. Measure preparation, transfer, startup, test execution, complete jobs/workflow and aggregate job-minutes separately.

📹 Demo

Not applicable; CI behavior is demonstrated by execution receipts and timing results.

[vercel]

@dnikolayev is attempting to deploy a commit to the HASH Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[semgrep-code-hashintel]

Detected subprocess function 'run' with user controlled data. A malicious actor could leverage this to perform command injection. You may consider using 'shlex.quote()'.

🥳 Fixed in commit 1082f6c 🥳

[dnikolayev]

Removed the version probes that executed environment-selected tool paths in 1082f6c. Tool overrides remain bound to the cache key by their resolved paths and SHA-256 binary hashes; only fixed Rust/Cargo commands perform version discovery. A regression checks that compiler and wrapper overrides are never executed for identity inspection and that changed wrapper bytes invalidate the identity. All 15 helper checks and 3 timing checks pass locally. Leaving this thread for reviewer resolution.