SRE-999: Make the frontend and Petrinaut docs builds cacheable - #9647
SRE-999: Make the frontend and Petrinaut docs builds cacheable#9647claude[bot] wants to merge 5 commits into
Conversation
`apps/hash-frontend#build` and `apps/petrinaut-docs#build` both set `cache: false`. Replace it with what the two builds actually produce and read, so Turborepo can hash and restore them. The frontend's `outputs` excludes `.next/cache`, webpack's own cache directory, which is the part that made the artifact large. Its `env` lists the variables `next.config.js` re-exports under `NEXT_PUBLIC_` and Next inlines into the client bundle. The docs build gains `.astro/**` alongside `dist/**`, and the variables `astro.config.mjs` folds into `__PND_DIFF_COMPARE__`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rkoy44pLnrjaxwq1qM5yhQ
|
The latest updates on your projects. Learn more about Vercel for GitHub.
3 Skipped Deployments
|
`docs/task-dependencies.json` records each task's `cache` and `env`, so declaring the environment variables left both packages' copies stale and failed the Global lint job's task-dependency check. `cache` is dropped, since the generator only writes the key when caching is off, and the sorted `env` list is added in its place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rkoy44pLnrjaxwq1qM5yhQ
PR SummaryMedium Risk Overview For the frontend, For Petrinaut docs, Reviewed by Cursor Bugbot for commit 4ad3a8c. Bugbot is set up for automated code reviews on this repo. Configure here. |
| // `next.config.js` runs dotenv-flow against the repository root. The | ||
| // tracked root dotfiles are already global dependencies; the gitignored | ||
| // `.env.local` is outside Turborepo's default inputs. | ||
| "inputs": ["$TURBO_DEFAULT$", "$TURBO_ROOT$/.env*"] |
There was a problem hiding this comment.
Doesn't this apply to every dotfile?
There was a problem hiding this comment.
Yes, and it turns out this line is redundant. $TURBO_DEFAULT$ is the git index plus untracked-but-not-ignored files, so a gitignored .env.local is outside it — but the root turbo.json globalDependencies already lists .*, and those globs are walked on disk, so every root dotfile, gitignored ones included, is in every task's global hash. Removed the line and its comment in 4ad3a8c.
Verified with a dry-run: with the line gone, adding or editing a root .env.local still changes the task hash. The petrinaut-website precedent (".env*" package-local) is different — root .* only matches root-level files, so a package's own .env* does need declaring.
Generated by Claude Code
| "VERCEL_GIT_COMMIT_REF", | ||
| "VERCEL_GIT_COMMIT_SHA", |
There was a problem hiding this comment.
This probably implies, that every now brunch/commit on Vercel invalidates the cache, no? As mentioned earlier, I don't get why we need that. For what it is needed?
There was a problem hiding this comment.
Yes: every Vercel commit is a miss for this task. The env entries are not what causes it, though. The build's inputs include the architecture bundle from @local/petrinaut-arch-docs#doc:architecture, and that bundle embeds the commit SHA in every source link (libs/@local/petrinaut-arch-docs/src/source-url.ts:22-29), so the input hash already differs per commit. Listing VERCEL_GIT_* only keeps a hit correct where one is possible; it never removes one.
Where that leaves the cache: no GitHub Actions workflow runs @apps/petrinaut-docs#build, so today it only helps local runs (and possibly same-commit Vercel redeploys). It is correct caching with a small benefit.
What the variables are for: astro.config.mjs:285-293 inlines them as __PND_DIFF_COMPARE__, which DiffBadges.astro renders as the two header chips on diff previews ("this preview" / "compared against", with PR number and short SHA). Added in #9366 (FE-1514).
Options, your call:
- Keep as is.
- Revert this package to
cache: falsewith the reason in-file, since the cross-commit benefit is nil. - Make the build hermetic: badge and bundle source links read commit identity at request time instead of build time. Larger change, touches
petrinaut-arch-docstoo.
The frontend has the same shape: NEXT_PUBLIC_VERCEL_GIT_COMMIT_REF/SHA in its env (for buildstamp.js) means every Vercel commit misses there as well; local and CI hashes are unaffected because those variables are unset.
Generated by Claude Code
Requested by Tim Diekmann · Slack thread
🌟 What is the purpose of this PR?
Before: the
buildtasks ofapps/hash-frontendandapps/petrinaut-docsboth setcache: false, so every run rebuilds them from scratch. The frontend has carried it since #4675, which turned it on as an experiment — that PR's whole description is "The output of the build command of the frontend is probably a big part of the cache. We disable the caching of that command and see how slow the CI might become." It also commented outoutputs, so the real problem is visible in the same diff:./.next/**includes.next/cache, webpack's own cache directory. That is what made the artifact big. Nobody diagnosed it; the flag stayed.After: both tasks are cacheable, and each declares what it produces and what it reads. The frontend stores
.nextwithout.next/cache, and lists the environment variablesnext.config.jsinlines into the client bundle, so a bundle built for one environment can never be restored for another. The docs build stores.astroalongsidedist, becauselint:tscreads it, and lists the variables that end up in__PND_DIFF_COMPARE__.How: this is the same shape as #9359, which fixed
apps/petrinaut-website— dropcache: false, declareenv, add.env*toinputs, and write the reason into the file.--env-mode=strict(the default) hides anything not inenvfrom the task, so declaring the variables is what makes the hash honest rather than just faster.Final values
apps/hash-frontend#buildoutputs:[".next/**", "!.next/cache/**", "next-env.d.ts"]env:ANALYZE,API_ORIGIN,ENVIRONMENT,FRONTEND_URL,GOOGLE_OAUTH_CLIENT_ID,NEXT_PUBLIC_VERCEL_GIT_COMMIT_REF,NEXT_PUBLIC_VERCEL_GIT_COMMIT_SHA,NODE_ENV,NOTIFICATION_POLL_INTERVAL,SELF_HOSTED_HASH,SENTRY_DSN,SENTRY_ENVIRONMENT,SENTRY_REPLAYS_SESSION_SAMPLE_RATE,SHOW_WORKER_COSTinputs:["$TURBO_DEFAULT$", "$TURBO_ROOT$/.env*"]apps/petrinaut-docs#buildoutputs:["dist/**", ".astro/**"]env:PETRINAUT_ARCH_DOCS_DIFF_BASE,VERCEL_GIT_COMMIT_REF,VERCEL_GIT_COMMIT_SHA,VERCEL_GIT_PULL_REQUEST_ID,VERCEL_GIT_REPO_OWNER,VERCEL_GIT_REPO_SLUGinputs:["$TURBO_DEFAULT$", "src/content/**", "public/architecture.*"]The judgement call on
apps/petrinaut-docs: caching is on — please overrule if you disagreesrc/diff-context.tsdoes two things no hash can cover, and part of the inlined constant comes from them. I decided neither makes a cache hit wrong, and droppedcache: false. The evidence:resolveDiffCompareContextreturnsnullatsrc/diff-context.ts:108-110whenmanifest.diffis undefined, before it reads env or touches git.manifest.diffis only set whenPETRINAUT_ARCH_DOCS_DIFF_BASEis set, andvercel-build.sh:25-27only sets it on a Vercel preview build. On any other build, neither the git call nor the network call runs at all.git rev-parseshell-out atsrc/diff-context.ts:36-42is a fallback, not a path. Its two call sites (:123-124and:126) reach it only whenVERCEL_GIT_COMMIT_REF/VERCEL_GIT_COMMIT_SHAare empty. A Vercel preview build always sets both. So the one kind of build that can reach the function is the one kind that never uses the git fallback.fetchtoapi.github.comat:57-60does run on every diff build, and its answer — whether the base ref has an open PR — is genuinely unhashable. ButVERCEL_GIT_COMMIT_SHAis now inenv, so on Vercel each commit hashes differently and builds fresh. A hit means the same commit rebuilt, and the only thing that can have changed is one header badge reading a branch name where a PR has since opened. That is a stale label on preview chrome, not wrong output.outputsandinputsare fixed either way, so if you want the flag back it is a one-line change on top of this. The reasoning is written intoapps/petrinaut-docs/turbo.jsonnext to theenvlist, following the convention inlibs/@local/petrinaut-arch-docs/turbo.json:5-11.🔗 Related links
cache: falsesections in turbo.json (internal)apps/petrinaut-website, and the shape this followsoutputs🔍 What does this change?
apps/hash-frontend/turbo.json: dropscache: falsefrombuild, restoresoutputswith.next/cacheexcluded, addsenvandinputs.dependsOnis unchanged.apps/petrinaut-docs/turbo.json: dropscache: falsefrombuild, adds.astro/**tooutputs, addsenvandinputs.dependsOnis unchanged.sync:bundleandlint:tsckeepcache: falseand are untouched.Each value was read off the current tree:
vercel.json:8sets"outputDirectory": "./.next", andnext.config.jssets nodistDir, nooutput: "standalone"and nooutputFileTracingRoot, so.nextis where the build lands.next-env.d.tsis inoutputsbecause.gitignore:95ignores it andapps/hash-frontend/tsconfig.jsonlists it ininclude— it is generated at the package root and read back bylint:tsc.envlist is every variable read innext.config.js(:11,:38-64) andbuildstamp.js:1-3, plusNODE_ENV, which picks which dotenv fileconfig()loads.next.config.js:14runs dotenv-flow against the repository root. The tracked root dotfiles (.env,.env.development,.env.test) are already covered byglobalDependencies'".*";.gitignore:79-80ignores.env.localand.env.*.local, hence the$TURBO_ROOT$/.env*input..astro/is inapps/petrinaut-docs/.gitignoreand is the sync directoryastro checkreads, solint:tscneeds it restored whenbuildhits.inputsare the pathsscripts/sync-bundle.mjs:36-85writes, which.gitignorelists as derived build inputs. Thesync:bundle→@local/petrinaut-arch-docs#doc:architecturechain should already cover them; the globs are belt and braces.envlist mirrors whatsrc/diff-context.ts:113-129reads, modelled onlibs/@local/petrinaut-arch-docs/turbo.json:25-36. That file is not edited.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
Every
dependsOnis left as it was. Only caching, outputs, inputs and env change.withSentryConfig(apps/hash-frontend/next.config.js:81) uploads sourcemaps duringnext build. That is a side effect, so a cache hit skips it. This PR does not address it — see next steps.🐾 Next steps
buildinto asentry:sourcemapstask. The rootturbo.jsonalready defines one; the frontend has no such script, so there is nothing for it to run yet. Until then a restored frontend build does not re-upload sourcemaps.apps/hash-frontend/vercel-build.sh:19andapps/petrinaut-docs/vercel-build.sh:39both runturbo build --env-mode=loose, where an undeclared variable reaches the task without entering its hash. That is the likely mechanical cause of the original Vercel problems, and it is blocked on SRE-999 because declaring the variables is the prerequisite. Deliberately not changed here.apps/petrinaut-docs'lint:tscandsync:bundlestill setcache: false, and are out of scope for this PR.🛡 What tests cover this?
Nothing automated covers a
turbo.jsonbeyond schema validation. Correctness here is whether the declared outputs and env match the code, which is what the "What does this change?" section above walks through file by file.❓ How to test this?
turbo build --filter='@apps/hash-frontend' --dry=jsonand confirm the task'sresolvedTaskDefinitionshows the outputs and env above, and thatcache.statusis no longer forced.cache hit. Then check.next/cacheis absent from the stored artifact.@apps/petrinaut-docs, then runlint:tscagainst the restored.astrodirectory to confirmastro checkstill has what it needs.