Skip to content

SRE-999: Make the frontend and Petrinaut docs builds cacheable - #9647

Open
claude[bot] wants to merge 5 commits into
mainfrom
td/sre-999-make-frontend-and-petrinaut-docs-builds-cacheable
Open

SRE-999: Make the frontend and Petrinaut docs builds cacheable#9647
claude[bot] wants to merge 5 commits into
mainfrom
td/sre-999-make-frontend-and-petrinaut-docs-builds-cacheable

Conversation

@claude

@claude claude Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Requested by Tim Diekmann · Slack thread

🌟 What is the purpose of this PR?

Before: the build tasks of apps/hash-frontend and apps/petrinaut-docs both set cache: false, so every run rebuilds them from scratch. The frontend has carried it since #4675, which turned it on as an experiment — that PR's whole description is "The output of the build command of the frontend is probably a big part of the cache. We disable the caching of that command and see how slow the CI might become." It also commented out outputs, so the real problem is visible in the same diff: ./.next/** includes .next/cache, webpack's own cache directory. That is what made the artifact big. Nobody diagnosed it; the flag stayed.

After: both tasks are cacheable, and each declares what it produces and what it reads. The frontend stores .next without .next/cache, and lists the environment variables next.config.js inlines into the client bundle, so a bundle built for one environment can never be restored for another. The docs build stores .astro alongside dist, because lint:tsc reads it, and lists the variables that end up in __PND_DIFF_COMPARE__.

How: this is the same shape as #9359, which fixed apps/petrinaut-website — drop cache: false, declare env, add .env* to inputs, and write the reason into the file. --env-mode=strict (the default) hides anything not in env from the task, so declaring the variables is what makes the hash honest rather than just faster.

Final values

apps/hash-frontend#build

  • outputs: [".next/**", "!.next/cache/**", "next-env.d.ts"]
  • env: ANALYZE, API_ORIGIN, ENVIRONMENT, FRONTEND_URL, GOOGLE_OAUTH_CLIENT_ID, NEXT_PUBLIC_VERCEL_GIT_COMMIT_REF, NEXT_PUBLIC_VERCEL_GIT_COMMIT_SHA, NODE_ENV, NOTIFICATION_POLL_INTERVAL, SELF_HOSTED_HASH, SENTRY_DSN, SENTRY_ENVIRONMENT, SENTRY_REPLAYS_SESSION_SAMPLE_RATE, SHOW_WORKER_COST
  • inputs: ["$TURBO_DEFAULT$", "$TURBO_ROOT$/.env*"]

apps/petrinaut-docs#build

  • outputs: ["dist/**", ".astro/**"]
  • env: PETRINAUT_ARCH_DOCS_DIFF_BASE, VERCEL_GIT_COMMIT_REF, VERCEL_GIT_COMMIT_SHA, VERCEL_GIT_PULL_REQUEST_ID, VERCEL_GIT_REPO_OWNER, VERCEL_GIT_REPO_SLUG
  • inputs: ["$TURBO_DEFAULT$", "src/content/**", "public/architecture.*"]

The judgement call on apps/petrinaut-docs: caching is on — please overrule if you disagree

src/diff-context.ts does two things no hash can cover, and part of the inlined constant comes from them. I decided neither makes a cache hit wrong, and dropped cache: false. The evidence:

  • Both are behind an early return. resolveDiffCompareContext returns null at src/diff-context.ts:108-110 when manifest.diff is undefined, before it reads env or touches git. manifest.diff is only set when PETRINAUT_ARCH_DOCS_DIFF_BASE is set, and vercel-build.sh:25-27 only sets it on a Vercel preview build. On any other build, neither the git call nor the network call runs at all.
  • The git rev-parse shell-out at src/diff-context.ts:36-42 is a fallback, not a path. Its two call sites (:123-124 and :126) reach it only when VERCEL_GIT_COMMIT_REF / VERCEL_GIT_COMMIT_SHA are empty. A Vercel preview build always sets both. So the one kind of build that can reach the function is the one kind that never uses the git fallback.
  • The fetch to api.github.com at :57-60 does run on every diff build, and its answer — whether the base ref has an open PR — is genuinely unhashable. But VERCEL_GIT_COMMIT_SHA is now in env, so on Vercel each commit hashes differently and builds fresh. A hit means the same commit rebuilt, and the only thing that can have changed is one header badge reading a branch name where a PR has since opened. That is a stale label on preview chrome, not wrong output.

outputs and inputs are fixed either way, so if you want the flag back it is a one-line change on top of this. The reasoning is written into apps/petrinaut-docs/turbo.json next to the env list, following the convention in libs/@local/petrinaut-arch-docs/turbo.json:5-11.

🔗 Related links

🔍 What does this change?

  • apps/hash-frontend/turbo.json: drops cache: false from build, restores outputs with .next/cache excluded, adds env and inputs. dependsOn is unchanged.
  • apps/petrinaut-docs/turbo.json: drops cache: false from build, adds .astro/** to outputs, adds env and inputs. dependsOn is unchanged. sync:bundle and lint:tsc keep cache: false and are untouched.

Each value was read off the current tree:

  • vercel.json:8 sets "outputDirectory": "./.next", and next.config.js sets no distDir, no output: "standalone" and no outputFileTracingRoot, so .next is where the build lands.
  • next-env.d.ts is in outputs because .gitignore:95 ignores it and apps/hash-frontend/tsconfig.json lists it in include — it is generated at the package root and read back by lint:tsc.
  • The frontend env list is every variable read in next.config.js (:11, :38-64) and buildstamp.js:1-3, plus NODE_ENV, which picks which dotenv file config() loads.
  • next.config.js:14 runs dotenv-flow against the repository root. The tracked root dotfiles (.env, .env.development, .env.test) are already covered by globalDependencies' ".*"; .gitignore:79-80 ignores .env.local and .env.*.local, hence the $TURBO_ROOT$/.env* input.
  • .astro/ is in apps/petrinaut-docs/.gitignore and is the sync directory astro check reads, so lint:tsc needs it restored when build hits.
  • The docs inputs are the paths scripts/sync-bundle.mjs:36-85 writes, which .gitignore lists as derived build inputs. The sync:bundle@local/petrinaut-arch-docs#doc:architecture chain should already cover them; the globs are belt and braces.
  • The docs env list mirrors what src/diff-context.ts:113-129 reads, modelled on libs/@local/petrinaut-arch-docs/turbo.json:25-36. That file is not edited.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

Every dependsOn is left as it was. Only caching, outputs, inputs and env change.

⚠️ Known issues

withSentryConfig (apps/hash-frontend/next.config.js:81) uploads sourcemaps during next build. That is a side effect, so a cache hit skips it. This PR does not address it — see next steps.

🐾 Next steps

  • Split the sourcemap upload out of build into a sentry:sourcemaps task. The root turbo.json already defines one; the frontend has no such script, so there is nothing for it to run yet. Until then a restored frontend build does not re-upload sourcemaps.
  • SRE-1050: apps/hash-frontend/vercel-build.sh:19 and apps/petrinaut-docs/vercel-build.sh:39 both run turbo build --env-mode=loose, where an undeclared variable reaches the task without entering its hash. That is the likely mechanical cause of the original Vercel problems, and it is blocked on SRE-999 because declaring the variables is the prerequisite. Deliberately not changed here.
  • apps/petrinaut-docs' lint:tsc and sync:bundle still set cache: false, and are out of scope for this PR.

🛡 What tests cover this?

Nothing automated covers a turbo.json beyond schema validation. Correctness here is whether the declared outputs and env match the code, which is what the "What does this change?" section above walks through file by file.

❓ How to test this?

  1. turbo build --filter='@apps/hash-frontend' --dry=json and confirm the task's resolvedTaskDefinition shows the outputs and env above, and that cache.status is no longer forced.
  2. Build the frontend twice with no change in between and confirm the second run reports cache hit. Then check .next/cache is absent from the stored artifact.
  3. Repeat for @apps/petrinaut-docs, then run lint:tsc against the restored .astro directory to confirm astro check still has what it needs.
  4. On a Vercel preview, confirm the diff badges in the docs header still name the right refs.

`apps/hash-frontend#build` and `apps/petrinaut-docs#build` both set
`cache: false`. Replace it with what the two builds actually produce and
read, so Turborepo can hash and restore them.

The frontend's `outputs` excludes `.next/cache`, webpack's own cache
directory, which is the part that made the artifact large. Its `env`
lists the variables `next.config.js` re-exports under `NEXT_PUBLIC_`
and Next inlines into the client bundle.

The docs build gains `.astro/**` alongside `dist/**`, and the variables
`astro.config.mjs` folds into `__PND_DIFF_COMPARE__`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rkoy44pLnrjaxwq1qM5yhQ
@vercel

vercel Bot commented Sep 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Sep 11, 2026 7:28am UTC
3 Skipped Deployments
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Sep 11, 2026 7:28am UTC
petrinaut Skipped Skipped Sep 11, 2026 7:28am UTC
petrinaut-docs Skipped Skipped Sep 11, 2026 7:28am UTC

Request Review

`docs/task-dependencies.json` records each task's `cache` and `env`, so
declaring the environment variables left both packages' copies stale and
failed the Global lint job's task-dependency check.

`cache` is dropped, since the generator only writes the key when caching
is off, and the sorted `env` list is added in its place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rkoy44pLnrjaxwq1qM5yhQ
@cursor

cursor Bot commented Sep 10, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Incorrect env/outputs declarations could restore a frontend bundle or docs build for the wrong environment; cache hits also skip Sentry sourcemap upload during next build (called out as a known follow-up).

Overview
Re-enables Turborepo caching for @apps/hash-frontend#build and @apps/petrinaut-docs#build, which previously forced cache: false (frontend had also dropped outputs, so huge .next/cache artifacts were the likely reason caching was turned off).

For the frontend, build now declares deployable outputs (.next minus .next/cache, plus next-env.d.ts) and an env allowlist of variables next.config.js inlines into the client bundle, so restored builds cannot cross environments.

For Petrinaut docs, build adds @local/petrinaut-arch-docs#doc:architecture to dependsOn, caches dist/** and .astro/**, lists Vercel/git env used for diff preview context, and inputs that hash the generated architecture bundle/** via dependency outputs. Matching task-dependencies.json entries are updated for both apps.

Reviewed by Cursor Bugbot for commit 4ad3a8c. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread apps/hash-frontend/turbo.json Outdated
Comment on lines +43 to +46
// `next.config.js` runs dotenv-flow against the repository root. The
// tracked root dotfiles are already global dependencies; the gitignored
// `.env.local` is outside Turborepo's default inputs.
"inputs": ["$TURBO_DEFAULT$", "$TURBO_ROOT$/.env*"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this apply to every dotfile?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and it turns out this line is redundant. $TURBO_DEFAULT$ is the git index plus untracked-but-not-ignored files, so a gitignored .env.local is outside it — but the root turbo.json globalDependencies already lists .*, and those globs are walked on disk, so every root dotfile, gitignored ones included, is in every task's global hash. Removed the line and its comment in 4ad3a8c.

Verified with a dry-run: with the line gone, adding or editing a root .env.local still changes the task hash. The petrinaut-website precedent (".env*" package-local) is different — root .* only matches root-level files, so a package's own .env* does need declaring.


Generated by Claude Code

Comment on lines +38 to +39
"VERCEL_GIT_COMMIT_REF",
"VERCEL_GIT_COMMIT_SHA",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably implies, that every now brunch/commit on Vercel invalidates the cache, no? As mentioned earlier, I don't get why we need that. For what it is needed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes: every Vercel commit is a miss for this task. The env entries are not what causes it, though. The build's inputs include the architecture bundle from @local/petrinaut-arch-docs#doc:architecture, and that bundle embeds the commit SHA in every source link (libs/@local/petrinaut-arch-docs/src/source-url.ts:22-29), so the input hash already differs per commit. Listing VERCEL_GIT_* only keeps a hit correct where one is possible; it never removes one.

Where that leaves the cache: no GitHub Actions workflow runs @apps/petrinaut-docs#build, so today it only helps local runs (and possibly same-commit Vercel redeploys). It is correct caching with a small benefit.

What the variables are for: astro.config.mjs:285-293 inlines them as __PND_DIFF_COMPARE__, which DiffBadges.astro renders as the two header chips on diff previews ("this preview" / "compared against", with PR number and short SHA). Added in #9366 (FE-1514).

Options, your call:

  1. Keep as is.
  2. Revert this package to cache: false with the reason in-file, since the cross-commit benefit is nil.
  3. Make the build hermetic: badge and bundle source links read commit identity at request time instead of build time. Larger change, touches petrinaut-arch-docs too.

The frontend has the same shape: NEXT_PUBLIC_VERCEL_GIT_COMMIT_REF/SHA in its env (for buildstamp.js) means every Vercel commit misses there as well; local and CI hashes are unaffected because those variables are unset.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash* Affects HASH (a `hash-*` app) area/apps type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

2 participants