infra(app): deploy the frontend through a smoked candidate revision

CHANGELOG.md

@@ -143,6 +143,27 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
 
 ### Changed
 
+- **The frontend deploys through a candidate revision instead of straight onto live
+  traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as
+  `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate
+  --revision-suffix=b$BUILD_ID`, smoke the candidate on its tag URL, then `update-traffic`
+  to exactly that revision (the chains are not identical — this one pushes `:latest` only
+  after the promotion, where the API still pushes it alongside the deploy). The service
+  carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =`
+  bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every
+  bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy
+  site; until now a typo in it went live unchecked and the daily bot-serving monitor was
+  the only net, a night later. The smoke probes both halves of the split (a browser UA
+  must get `<div id="root">`, Googlebot must get the prerendered page — asserted on the
+  `<link rel="canonical">`, which the SPA shell carries not at all and whose value names
+  the route) on the home page and a deep route, plus `robots.txt` and `llms.txt` from the
+  `location =` bypasses and the latter's UTF-8 charset. The candidate tag is re-asserted
+  after the probes as well as before, so a concurrent build moving it mid-smoke fails this
+  build instead of getting it promoted on someone else's evidence. `:latest` moves only
+  after this build's promotion, so the tag can no longer name an image that was never
+  rolled out, and the build timeout goes to 20 min to leave room for a cold candidate.
+  (#11207)
+
 - **The `babysit-pipeline` skill gains the backfill scheduler and the driver's per-spec
   liveness check** — `run_queue.sh <queue-dir> [slots]` keeps N `run_spec.sh` drivers
   in flight over a queue file, skips libraries already on main and confirmed gaps,

agentic/docs/project-guide.md

@@ -1027,6 +1027,42 @@ gcloud builds submit --config=app/cloudbuild.yaml --project=YOUR_PROJECT_ID
 - **`app/cloudbuild.yaml`**: Frontend build + deploy
 - **`app/Dockerfile`**: Multi-stage build (Node -> nginx)
 
+Both cloudbuild files deploy through a **candidate revision**: `gcloud run deploy
+--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, a smoke step against the
+candidate's tag URL, then `gcloud run services update-traffic --to-revisions=…=100`. A
+revision that fails its smoke therefore never takes **live traffic** — the only requests
+it ever answers are the smoke's own, sent deliberately to its tag URL. (For the app that
+holds even with two builds in flight; for the API it holds for one build at a time — see
+the candidate-tag recheck below.) The app's smoke
+covers both halves of the crawler split (browser UA -> SPA shell, Googlebot ->
+prerendered page via `@seo_proxy`) plus the `location =` bypasses for `robots.txt` and
+`llms.txt`; the API's covers `/health`, the public read paths and the fail-closed admin
+gate.
+
+The two chains differ in two places, both on the app's side:
+
+- **`:latest`.** `app/cloudbuild.yaml` pushes it only after the promotion, so the tag
+  cannot name an image that was never rolled out; `api/cloudbuild.yaml` still pushes it
+  alongside the deploy. Neither pipeline *reads* `:latest` — both deploy `:$BUILD_ID` — so
+  the tag is a convenience for humans, and with two overlapping builds the later push wins
+  regardless.
+- **The candidate-tag recheck.** `candidate` is a shared tag, so a concurrent build of the
+  same service can move it mid-smoke. `app/cloudbuild.yaml` re-asserts after its probes
+  that the tag still names this build's revision and refuses to promote otherwise; a
+  competing build only ever tags its own revision and never ours back, so matching at both
+  ends means the tag was never re-assigned during the smoke. `api/cloudbuild.yaml` checks
+  only before its probes, so two overlapping API builds can still promote a revision that
+  was not the one smoked — the same four lines would close it there.
+
+Both checks read `status.traffic`, which is the **control-plane tag assignment** and not
+proof of which revision answered a given request: because `candidate` is reused across
+builds, a probe issued while a re-assignment is still propagating can reach the previous
+revision. Closing that gap needs either a build-unique tag (tag URLs then accumulate on
+the service without bound) or a build id the served response itself carries. Neither is in
+place, and the residual is narrow: it needs the previous candidate to pass every probe as
+well, so the worst case is promoting a revision that was believed smoked, not shipping a
+page known to be broken.
+
 ## Debugging Tips
 
 ### Database Connection Issues

app/cloudbuild.yaml

@@ -1,4 +1,19 @@
 # Cloud Build configuration for anyplot Frontend
+#
+# This file follows the same candidate-rollout PATTERN as api/cloudbuild.yaml:
+# build -> push the build tag -> deploy --no-traffic --tag=candidate -> smoke
+# the candidate -> update-traffic to exactly that revision. The step chains are
+# not identical: this one pushes `:latest` only after the promotion (the API
+# still pushes it alongside the deploy) and has no final get-url step.
+#
+# Why this service deserves the same care as the API: it serves the public
+# website AND carries the whole crawler path in app/nginx.conf (the `$is_bot`
+# map, the `location =` bypasses for robots/llms/sitemap/og, the `@seo_proxy`
+# upstream). That is the very file whose breakage served every bot an HTTP 502
+# for four weeks between 2026-06-12 and 2026-07-09 — a `proxy_ssl_verify_depth`
+# default against a 4-deep chain — while humans, Plausible and CI all saw a
+# healthy site. Until now a typo in it went live unchecked and the daily
+# bot-serving monitor was the only net, a night later.
 substitutions:
   _REGION: 'europe-west4'
   _SERVICE_NAME: 'anyplot-app'
@@ -28,10 +43,18 @@ steps:
         'app/Dockerfile',
         'app',
       ]
+    id: 'build-image'
 
-  # Push the container image to Artifact Registry
+  # Only the build-specific tag goes up here — it is the one the deploy pulls.
+  # `:latest` waits for the promote step (see push-latest at the bottom): a
+  # `--all-tags` push before the rollout means `:latest` names an image that
+  # may never have served a request.
   - name: 'gcr.io/cloud-builders/docker'
-    args: ['push', '--all-tags', 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}']
+    args:
+      - 'push'
+      - 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:$BUILD_ID'
+    id: 'push-image'
+    waitFor: ['build-image']
 
   # Deploy container image to Cloud Run
   - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
@@ -69,6 +92,138 @@ steps:
       - '--cpu-throttling'
       - '--concurrency'
       - '15'
+      # Deploy WITHOUT routing traffic: the smoke step probes this revision on
+      # its tag URL first, and only the promote step below shifts traffic — so
+      # a broken nginx.conf never serves visitors or crawlers, not even between
+      # deploy and a failing smoke. The `candidate` tag moves to each new
+      # revision, and the deterministic revision suffix lets promote target
+      # EXACTLY the revision this build smoked (never a concurrent build's
+      # newer one). `b` prefix: Cloud Run requires the suffix to start with a
+      # lowercase letter, and $BUILD_ID is a UUID that usually starts with a
+      # digit.
+      - '--no-traffic'
+      - '--tag=candidate'
+      - '--revision-suffix=b$BUILD_ID'
+    id: 'deploy'
+    waitFor: ['push-image']
+
+  # Pre-traffic smoke against the CANDIDATE revision (tag URL, serving no
+  # traffic yet). Both halves of the split matter and only one of them is
+  # visible to a human: the SPA shell for a browser, and the prerendered page
+  # for a mapped crawler UA — the `$is_bot` map plus the `@seo_proxy` upstream,
+  # which is exactly the hop that can break silently.
+  - name: 'gcr.io/cloud-builders/gcloud'
+    entrypoint: 'bash'
+    args:
+      - '-ceu'
+      - |
+        # Resolve the candidate tag's URL AND assert it still points at THIS
+        # build's revision (deterministic --revision-suffix) — a concurrent
+        # build re-tagging `candidate` must fail this build, not get smoked
+        # on its behalf.
+        #
+        # What this checks is the CONTROL PLANE: `status.traffic` is the tag
+        # assignment, not proof of which revision answered a given request.
+        # `candidate` is reused across builds, so in the moment after a
+        # re-assignment the tag URL can still reach the previous revision, and
+        # a probe would then pass against the wrong image. Closing that needs
+        # either a build-unique tag (which accumulates tag URLs on the service
+        # without bound) or a build id the app serves in its own response —
+        # neither is taken here. The residual is narrow and its worst case is
+        # mild: it needs the PREVIOUS candidate to pass every probe too, so the
+        # failure is "promoted a revision we only believed we smoked", not
+        # "shipped a page we know is broken".
+        read -r URL REV <<< "$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \
+          | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['url'], c['revisionName'])")"
+        test "$$REV" = "${_SERVICE_NAME}-b$BUILD_ID" || { echo "candidate tag moved to $$REV (expected ${_SERVICE_NAME}-b$BUILD_ID)"; exit 1; }
+        echo "smoke against candidate $$URL ($$REV)"
+        # Retry EVERY probe: this service runs at min-instances 0, so the
+        # candidate is ALWAYS cold, and the crawler probes additionally wait on
+        # the first upstream call to api.anyplot.ai.
+        RETRY="--retry 5 --retry-delay 5 --retry-all-errors"
+        HUMAN="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
+        GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+        # `expect <ua> <path> <needle> <what-it-means>`: fetch to a file rather
+        # than piping into `grep -q`. `grep -q` exits at the first match and
+        # SIGPIPEs curl, so the pipe form only ever passes because `-ceu`
+        # carries no `pipefail` — a later hardening pass that adds it would red
+        # every deploy. Writing the body out also lets each failure say which
+        # probe failed and what was expected, which is the whole value of a log
+        # someone reads while a deploy is blocked.
+        expect() {
+          curl -fsS $$RETRY -A "$$1" -o body.out "$$URL$$2" \
+            || { echo "candidate did not serve $$2 to UA '$$1'"; exit 1; }
+          grep -qF "$$3" body.out || { echo "$$4 ($$2 is missing: $$3)"; exit 1; }
+          echo "OK: $$2"
+        }
+        # Humans get the SPA shell.
+        expect "$$HUMAN" "/" '<div id="root">' "a browser did not get the SPA shell"
+        # Crawlers get the prerendered page through @seo_proxy. The canonical
+        # link is the marker: the SPA shell carries none at all, and its value
+        # names the route, so one grep proves both "the bot hop ran" and "the
+        # right page came back". Probing a deep route as well as the home page
+        # covers `proxy_pass .../seo-proxy$$request_uri` — a home-only check
+        # would pass with the request URI dropped. (`$$` is Cloud Build's
+        # escape for a literal `$`; an unescaped one is read as a substitution
+        # and fails the build before a single step runs.)
+        expect "$$GOOGLEBOT" "/" '<link rel="canonical" href="https://anyplot.ai/" />' \
+          "a crawler did not get the prerendered home page"
+        expect "$$GOOGLEBOT" "/scatter-basic" '<link rel="canonical" href="https://anyplot.ai/scatter-basic" />' \
+          "a crawler did not get the prerendered page for this route"
+        # The machine files must come from the `location =` bypasses, not the
+        # proxy — and llms.txt carries UTF-8 punctuation that needs the charset.
+        expect "$$GOOGLEBOT" "/robots.txt" 'User-agent: Bytespider' \
+          "robots.txt did not come from the location = bypass"
+        expect "$$GOOGLEBOT" "/llms.txt" '# anyplot' \
+          "llms.txt did not come from the location = bypass"
+        ct=$$(curl -fsS $$RETRY -A "$$GOOGLEBOT" -o /dev/null -w '%{content_type}' "$$URL/llms.txt")
+        case "$$ct" in
+          *charset=utf-8*) ;;
+          *) echo "llms.txt served as '$$ct' (expected a utf-8 charset)"; exit 1 ;;
+        esac
+        # Re-assert the tag AFTER the probes. `candidate` is a shared tag, so a
+        # concurrent build could move it between the check above and the last
+        # probe, and this build would then have smoked someone else's revision
+        # while promoting its own. A competing build only ever tags its OWN
+        # revision and never ours back, so seeing our revision at both ends
+        # means every probe in between hit it.
+        REV_AFTER=$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \
+          | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['revisionName'])")
+        test "$$REV_AFTER" = "$$REV" || { echo "candidate tag moved to $$REV_AFTER mid-smoke (started on $$REV) — not promoting"; exit 1; }
+        echo "smoke OK"
+    id: 'smoke'
+    waitFor: ['deploy']
+
+  # Smoke passed — route 100% of traffic to EXACTLY the revision this build
+  # deployed and smoked (deterministic name via --revision-suffix), never
+  # `--to-latest`, which could promote a concurrent build's unsmoked revision.
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    entrypoint: gcloud
+    args:
+      - 'run'
+      - 'services'
+      - 'update-traffic'
+      - '${_SERVICE_NAME}'
+      - '--region'
+      - '${_REGION}'
+      - '--platform'
+      - 'managed'
+      - '--to-revisions=${_SERVICE_NAME}-b$BUILD_ID=100'
+    id: 'promote'
+    waitFor: ['smoke']
+
+  # `:latest` moves only after THIS build promoted, so it can no longer name an
+  # image that was never rolled out — which a `--all-tags` push before the
+  # deploy did. It is not a guarantee across concurrent builds: two overlapping
+  # deploys still race for the tag and the later push wins whichever revision
+  # serves. That is why the deploy step pulls `:$BUILD_ID` and never `:latest`;
+  # the tag is a convenience for humans, not an input to the rollout.
+  - name: 'gcr.io/cloud-builders/docker'
+    args:
+      - 'push'
+      - 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:latest'
+    id: 'push-latest'
+    waitFor: ['promote']
 
 # Store images in Artifact Registry
 images:
@@ -77,3 +232,7 @@ images:
 
 options:
   logging: CLOUD_LOGGING_ONLY
+
+# The chain is now deploy -> smoke (cold start + a first upstream call) ->
+# promote; the default 10 min left no room for a retrying probe.
+timeout: '1200s'