From 234b88e74c0aacb17e332bbb2e17ffe72fd25215 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:12:38 +0200 Subject: [PATCH 1/5] infra(app): deploy the frontend through a smoked candidate revision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit app/cloudbuild.yaml deployed straight onto live traffic while api/cloudbuild.yaml already had the candidate chain. This service is the one that carries the whole crawler path in app/nginx.conf — the $is_bot map, the `location =` bypasses, the @seo_proxy upstream — and that is the file whose breakage served every bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy site. Until now a typo in it went live unchecked and the daily bot-serving monitor was the only net, a night later. The chain is now build -> push $BUILD_ID -> deploy --no-traffic --tag=candidate --revision-suffix=b$BUILD_ID -> smoke the candidate's tag URL -> update-traffic to exactly that revision -> push :latest. The smoke probes both halves of the split, only one of which a human ever sees: a browser UA must get
, Googlebot must get the prerendered page. The marker is the canonical link — the SPA shell carries none at all, and its value names the route, so one grep proves both that the bot hop ran and that the right page came back. A deep route is probed alongside the home page, which covers the request URI reaching the upstream. robots.txt and llms.txt must come from the `location =` bypasses, and llms.txt must keep its utf-8 charset. Each probe fetches to a file instead of piping into `grep -q`: grep exits at the first match and SIGPIPEs curl, so the pipe form passes only because `bash -ceu` carries no pipefail. That also lets a failure name which probe failed and what it expected. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke --- CHANGELOG.md | 16 ++++ agentic/docs/project-guide.md | 10 +++ app/cloudbuild.yaml | 136 +++++++++++++++++++++++++++++++++- 3 files changed, 160 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d2675e6f22..680248ccc83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -127,6 +127,22 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **The frontend deploys through a candidate revision instead of straight onto live + traffic** — `app/cloudbuild.yaml` now mirrors `api/cloudbuild.yaml` step for step: + deploy with `--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, smoke the + candidate on its tag URL, then `update-traffic` to exactly that revision. The service + carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =` + bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every + bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy + site; until now a typo in it went live unchecked and the daily bot-serving monitor was + the only net, a night later. The smoke probes both halves of the split (a browser UA + must get `
`, Googlebot must get the prerendered page — asserted on the + ``, which the SPA shell carries not at all and whose value names + the route) on the home page and a deep route, plus `robots.txt` and `llms.txt` from the + `location =` bypasses and the latter's UTF-8 charset. `:latest` moves only after the + promotion, so the tag can no longer name an image that never served a request, and the + build timeout goes to 20 min to leave room for a cold candidate. + - **The `babysit-pipeline` skill gains the backfill scheduler and the driver's per-spec liveness check** — `run_queue.sh [slots]` keeps N `run_spec.sh` drivers in flight over a queue file, skips libraries already on main and confirmed gaps, diff --git a/agentic/docs/project-guide.md b/agentic/docs/project-guide.md index d2e7fbf7f98..556c859c628 100644 --- a/agentic/docs/project-guide.md +++ b/agentic/docs/project-guide.md @@ -1026,6 +1026,16 @@ gcloud builds submit --config=app/cloudbuild.yaml --project=YOUR_PROJECT_ID - **`app/cloudbuild.yaml`**: Frontend build + deploy - **`app/Dockerfile`**: Multi-stage build (Node -> nginx) +Both cloudbuild files deploy through a **candidate revision**: `gcloud run deploy +--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, a smoke step against the +candidate's tag URL, then `gcloud run services update-traffic --to-revisions=…=100`. A +revision that fails its smoke therefore never serves a request, and `:latest` is pushed +only after the promotion, so the tag always names an image that did serve. The app's +smoke covers both halves of the crawler split (browser UA -> SPA shell, Googlebot -> +prerendered page via `@seo_proxy`) plus the `location =` bypasses for `robots.txt` and +`llms.txt`; the API's covers `/health`, the public read paths and the fail-closed admin +gate. + ## Debugging Tips ### Database Connection Issues diff --git a/app/cloudbuild.yaml b/app/cloudbuild.yaml index 83914a107f1..b2a73d55950 100644 --- a/app/cloudbuild.yaml +++ b/app/cloudbuild.yaml @@ -1,4 +1,17 @@ # Cloud Build configuration for anyplot Frontend +# +# Since the candidate chain below, this file mirrors api/cloudbuild.yaml step +# for step: build -> push -> deploy --no-traffic -> smoke -> promote -> +# push latest. +# +# Why this service deserves the same care as the API: it serves the public +# website AND carries the whole crawler path in app/nginx.conf (the `$is_bot` +# map, the `location =` bypasses for robots/llms/sitemap/og, the `@seo_proxy` +# upstream). That is the very file whose breakage served every bot an HTTP 502 +# for four weeks between 2026-06-12 and 2026-07-09 — a `proxy_ssl_verify_depth` +# default against a 4-deep chain — while humans, Plausible and CI all saw a +# healthy site. Until now a typo in it went live unchecked and the daily +# bot-serving monitor was the only net, a night later. substitutions: _REGION: 'europe-west4' _SERVICE_NAME: 'anyplot-app' @@ -28,10 +41,18 @@ steps: 'app/Dockerfile', 'app', ] + id: 'build-image' - # Push the container image to Artifact Registry + # Only the build-specific tag goes up here — it is the one the deploy pulls. + # `:latest` waits for the promote step (see push-latest at the bottom): a + # `--all-tags` push before the rollout means `:latest` names an image that + # may never have served a request. - name: 'gcr.io/cloud-builders/docker' - args: ['push', '--all-tags', 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}'] + args: + - 'push' + - 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:$BUILD_ID' + id: 'push-image' + waitFor: ['build-image'] # Deploy container image to Cloud Run - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' @@ -69,6 +90,113 @@ steps: - '--cpu-throttling' - '--concurrency' - '15' + # Deploy WITHOUT routing traffic: the smoke step probes this revision on + # its tag URL first, and only the promote step below shifts traffic — so + # a broken nginx.conf never serves visitors or crawlers, not even between + # deploy and a failing smoke. The `candidate` tag moves to each new + # revision, and the deterministic revision suffix lets promote target + # EXACTLY the revision this build smoked (never a concurrent build's + # newer one). `b` prefix: Cloud Run requires the suffix to start with a + # lowercase letter, and $BUILD_ID is a UUID that usually starts with a + # digit. + - '--no-traffic' + - '--tag=candidate' + - '--revision-suffix=b$BUILD_ID' + id: 'deploy' + waitFor: ['push-image'] + + # Pre-traffic smoke against the CANDIDATE revision (tag URL, serving no + # traffic yet). Both halves of the split matter and only one of them is + # visible to a human: the SPA shell for a browser, and the prerendered page + # for a mapped crawler UA — the `$is_bot` map plus the `@seo_proxy` upstream, + # which is exactly the hop that can break silently. + - name: 'gcr.io/cloud-builders/gcloud' + entrypoint: 'bash' + args: + - '-ceu' + - | + # Resolve the candidate tag's URL AND assert it still points at THIS + # build's revision (deterministic --revision-suffix) — a concurrent + # build re-tagging `candidate` must fail this build, not get smoked + # on its behalf. + read -r URL REV <<< "$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ + | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['url'], c['revisionName'])")" + test "$$REV" = "${_SERVICE_NAME}-b$BUILD_ID" || { echo "candidate tag moved to $$REV (expected ${_SERVICE_NAME}-b$BUILD_ID)"; exit 1; } + echo "smoke against candidate $$URL ($$REV)" + # Retry EVERY probe: this service runs at min-instances 0, so the + # candidate is ALWAYS cold, and the crawler probes additionally wait on + # the first upstream call to api.anyplot.ai. + RETRY="--retry 5 --retry-delay 5 --retry-all-errors" + HUMAN="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" + GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" + # `expect `: fetch to a file rather + # than piping into `grep -q`. `grep -q` exits at the first match and + # SIGPIPEs curl, so the pipe form only ever passes because `-ceu` + # carries no `pipefail` — a later hardening pass that adds it would red + # every deploy. Writing the body out also lets each failure say which + # probe failed and what was expected, which is the whole value of a log + # someone reads while a deploy is blocked. + expect() { + curl -fsS $$RETRY -A "$$1" -o body.out "$$URL$$2" \ + || { echo "candidate did not serve $$2 to UA '$$1'"; exit 1; } + grep -qF "$$3" body.out || { echo "$$4 ($$2 is missing: $$3)"; exit 1; } + echo "OK: $$2" + } + # Humans get the SPA shell. + expect "$$HUMAN" "/" '
' "a browser did not get the SPA shell" + # Crawlers get the prerendered page through @seo_proxy. The canonical + # link is the marker: the SPA shell carries none at all, and its value + # names the route, so one grep proves both "the bot hop ran" and "the + # right page came back". Probing a deep route as well as the home page + # covers `proxy_pass .../seo-proxy$$request_uri` — a home-only check + # would pass with the request URI dropped. (`$$` is Cloud Build's + # escape for a literal `$`; an unescaped one is read as a substitution + # and fails the build before a single step runs.) + expect "$$GOOGLEBOT" "/" '' \ + "a crawler did not get the prerendered home page" + expect "$$GOOGLEBOT" "/scatter-basic" '' \ + "a crawler did not get the prerendered page for this route" + # The machine files must come from the `location =` bypasses, not the + # proxy — and llms.txt carries UTF-8 punctuation that needs the charset. + expect "$$GOOGLEBOT" "/robots.txt" 'User-agent: Bytespider' \ + "robots.txt did not come from the location = bypass" + expect "$$GOOGLEBOT" "/llms.txt" '# anyplot' \ + "llms.txt did not come from the location = bypass" + ct=$$(curl -fsS $$RETRY -A "$$GOOGLEBOT" -o /dev/null -w '%{content_type}' "$$URL/llms.txt") + case "$$ct" in + *charset=utf-8*) ;; + *) echo "llms.txt served as '$$ct' (expected a utf-8 charset)"; exit 1 ;; + esac + echo "smoke OK" + id: 'smoke' + waitFor: ['deploy'] + + # Smoke passed — route 100% of traffic to EXACTLY the revision this build + # deployed and smoked (deterministic name via --revision-suffix), never + # `--to-latest`, which could promote a concurrent build's unsmoked revision. + - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' + entrypoint: gcloud + args: + - 'run' + - 'services' + - 'update-traffic' + - '${_SERVICE_NAME}' + - '--region' + - '${_REGION}' + - '--platform' + - 'managed' + - '--to-revisions=${_SERVICE_NAME}-b$BUILD_ID=100' + id: 'promote' + waitFor: ['smoke'] + + # `:latest` means "the image currently serving", so it moves only after the + # promote step. + - name: 'gcr.io/cloud-builders/docker' + args: + - 'push' + - 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:latest' + id: 'push-latest' + waitFor: ['promote'] # Store images in Artifact Registry images: @@ -77,3 +205,7 @@ images: options: logging: CLOUD_LOGGING_ONLY + +# The chain is now deploy -> smoke (cold start + a first upstream call) -> +# promote; the default 10 min left no room for a retrying probe. +timeout: '1200s' From 1922cf0dceba59698797d90c5fda9572dc3b345f Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:21:37 +0200 Subject: [PATCH 2/5] infra(app): close the candidate-tag race, correct the parity claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings on #11207: - The `candidate` tag is shared, so a concurrent build could move it between the check before the probes and the last probe — this build would then have smoked another revision while promoting its own. The smoke now re-asserts the tag after the probes. A competing build only ever tags its own revision and never ours back, so seeing our revision at both ends means every probe in between hit it. - The `:latest` comment claimed more than waitFor can deliver. It orders steps within one build; two overlapping deploys still race for the tag. What it does guarantee is that the tag never names an image that was not rolled out, and neither pipeline reads it — both deploy $BUILD_ID. - "mirrors api/cloudbuild.yaml step for step" was wrong in the header comment, the changelog and the project guide: the API still pushes :latest alongside the deploy and ends with get-url. All three now say "the same candidate-rollout pattern" and name the difference. - The project guide said a failed revision "never serves a request"; the smoke's own requests go to exactly that revision. It now says live traffic. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke --- CHANGELOG.md | 14 ++++++++------ agentic/docs/project-guide.md | 12 +++++++++--- app/cloudbuild.yaml | 25 ++++++++++++++++++++----- 3 files changed, 37 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 680248ccc83..d736615e6e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -128,9 +128,11 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed - **The frontend deploys through a candidate revision instead of straight onto live - traffic** — `app/cloudbuild.yaml` now mirrors `api/cloudbuild.yaml` step for step: - deploy with `--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, smoke the - candidate on its tag URL, then `update-traffic` to exactly that revision. The service + traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as + `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate + --revision-suffix=b$BUILD_ID`, smoke the candidate on its tag URL, then `update-traffic` + to exactly that revision (the chains are not identical — this one pushes `:latest` only + after the promotion, where the API still pushes it alongside the deploy). The service carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =` bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy @@ -139,9 +141,9 @@ aggregate instead: an italic *Catalog* line at the end of the version section an must get `
`, Googlebot must get the prerendered page — asserted on the ``, which the SPA shell carries not at all and whose value names the route) on the home page and a deep route, plus `robots.txt` and `llms.txt` from the - `location =` bypasses and the latter's UTF-8 charset. `:latest` moves only after the - promotion, so the tag can no longer name an image that never served a request, and the - build timeout goes to 20 min to leave room for a cold candidate. + `location =` bypasses and the latter's UTF-8 charset. `:latest` moves only after this + build's promotion, so the tag can no longer name an image that was never rolled out, and + the build timeout goes to 20 min to leave room for a cold candidate. - **The `babysit-pipeline` skill gains the backfill scheduler and the driver's per-spec liveness check** — `run_queue.sh [slots]` keeps N `run_spec.sh` drivers diff --git a/agentic/docs/project-guide.md b/agentic/docs/project-guide.md index 556c859c628..c966df7da2d 100644 --- a/agentic/docs/project-guide.md +++ b/agentic/docs/project-guide.md @@ -1029,13 +1029,19 @@ gcloud builds submit --config=app/cloudbuild.yaml --project=YOUR_PROJECT_ID Both cloudbuild files deploy through a **candidate revision**: `gcloud run deploy --no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, a smoke step against the candidate's tag URL, then `gcloud run services update-traffic --to-revisions=…=100`. A -revision that fails its smoke therefore never serves a request, and `:latest` is pushed -only after the promotion, so the tag always names an image that did serve. The app's -smoke covers both halves of the crawler split (browser UA -> SPA shell, Googlebot -> +revision that fails its smoke therefore never takes **live traffic** — the only requests +it ever answers are the smoke's own, sent deliberately to its tag URL. The app's smoke +covers both halves of the crawler split (browser UA -> SPA shell, Googlebot -> prerendered page via `@seo_proxy`) plus the `location =` bypasses for `robots.txt` and `llms.txt`; the API's covers `/health`, the public read paths and the fail-closed admin gate. +The two chains differ in one step: `app/cloudbuild.yaml` pushes `:latest` only after the +promotion, so the tag cannot name an image that was never rolled out, while +`api/cloudbuild.yaml` still pushes it alongside the deploy. Neither pipeline reads +`:latest` — both deploy `:$BUILD_ID` — so the tag is a convenience for humans, and with +two overlapping builds the later push wins regardless. + ## Debugging Tips ### Database Connection Issues diff --git a/app/cloudbuild.yaml b/app/cloudbuild.yaml index b2a73d55950..73917f89f3a 100644 --- a/app/cloudbuild.yaml +++ b/app/cloudbuild.yaml @@ -1,8 +1,10 @@ # Cloud Build configuration for anyplot Frontend # -# Since the candidate chain below, this file mirrors api/cloudbuild.yaml step -# for step: build -> push -> deploy --no-traffic -> smoke -> promote -> -# push latest. +# This file follows the same candidate-rollout PATTERN as api/cloudbuild.yaml: +# build -> push the build tag -> deploy --no-traffic --tag=candidate -> smoke +# the candidate -> update-traffic to exactly that revision. The step chains are +# not identical: this one pushes `:latest` only after the promotion (the API +# still pushes it alongside the deploy) and has no final get-url step. # # Why this service deserves the same care as the API: it serves the public # website AND carries the whole crawler path in app/nginx.conf (the `$is_bot` @@ -167,6 +169,15 @@ steps: *charset=utf-8*) ;; *) echo "llms.txt served as '$$ct' (expected a utf-8 charset)"; exit 1 ;; esac + # Re-assert the tag AFTER the probes. `candidate` is a shared tag, so a + # concurrent build could move it between the check above and the last + # probe, and this build would then have smoked someone else's revision + # while promoting its own. A competing build only ever tags its OWN + # revision and never ours back, so seeing our revision at both ends + # means every probe in between hit it. + REV_AFTER=$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ + | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['revisionName'])") + test "$$REV_AFTER" = "$$REV" || { echo "candidate tag moved to $$REV_AFTER mid-smoke (started on $$REV) — not promoting"; exit 1; } echo "smoke OK" id: 'smoke' waitFor: ['deploy'] @@ -189,8 +200,12 @@ steps: id: 'promote' waitFor: ['smoke'] - # `:latest` means "the image currently serving", so it moves only after the - # promote step. + # `:latest` moves only after THIS build promoted, so it can no longer name an + # image that was never rolled out — which a `--all-tags` push before the + # deploy did. It is not a guarantee across concurrent builds: two overlapping + # deploys still race for the tag and the later push wins whichever revision + # serves. That is why the deploy step pulls `:$BUILD_ID` and never `:latest`; + # the tag is a convenience for humans, not an input to the rollout. - name: 'gcr.io/cloud-builders/docker' args: - 'push' From 2a37693be2f692e33e3d1bfc7d82c1a81f5842a4 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:27:50 +0200 Subject: [PATCH 3/5] docs(changelog): carry the PR reference and the tag-recheck detail Every bullet in the file cites its PR; this one was missing it. The entry also now mentions the post-probe candidate-tag re-assertion added in the review round. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke --- CHANGELOG.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d736615e6e6..e0359ebb930 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -141,9 +141,12 @@ aggregate instead: an italic *Catalog* line at the end of the version section an must get `
`, Googlebot must get the prerendered page — asserted on the ``, which the SPA shell carries not at all and whose value names the route) on the home page and a deep route, plus `robots.txt` and `llms.txt` from the - `location =` bypasses and the latter's UTF-8 charset. `:latest` moves only after this - build's promotion, so the tag can no longer name an image that was never rolled out, and - the build timeout goes to 20 min to leave room for a cold candidate. + `location =` bypasses and the latter's UTF-8 charset. The candidate tag is re-asserted + after the probes as well as before, so a concurrent build moving it mid-smoke fails this + build instead of getting it promoted on someone else's evidence. `:latest` moves only + after this build's promotion, so the tag can no longer name an image that was never + rolled out, and the build timeout goes to 20 min to leave room for a cold candidate. + (#11207) - **The `babysit-pipeline` skill gains the backfill scheduler and the driver's per-spec liveness check** — `run_queue.sh [slots]` keeps N `run_spec.sh` drivers From 53fa877d46cbb8613f49ca5c34a646567f6e240f Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:31:44 +0200 Subject: [PATCH 4/5] docs: scope the candidate guarantee to each pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on #11207: the project guide claimed the candidate guarantee for both cloudbuild files, but api/cloudbuild.yaml checks the shared `candidate` tag only before its probes. Two overlapping API builds can therefore still promote a revision that was not the one smoked. The paragraph now says the guarantee holds unconditionally for the app and one-build-at-a-time for the API, and the "the chains differ" list carries the recheck as the second difference — with the note that the same four lines would close it on the API side. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke --- agentic/docs/project-guide.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/agentic/docs/project-guide.md b/agentic/docs/project-guide.md index c966df7da2d..0efc4f61608 100644 --- a/agentic/docs/project-guide.md +++ b/agentic/docs/project-guide.md @@ -1030,17 +1030,28 @@ Both cloudbuild files deploy through a **candidate revision**: `gcloud run deplo --no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, a smoke step against the candidate's tag URL, then `gcloud run services update-traffic --to-revisions=…=100`. A revision that fails its smoke therefore never takes **live traffic** — the only requests -it ever answers are the smoke's own, sent deliberately to its tag URL. The app's smoke +it ever answers are the smoke's own, sent deliberately to its tag URL. (For the app that +holds even with two builds in flight; for the API it holds for one build at a time — see +the candidate-tag recheck below.) The app's smoke covers both halves of the crawler split (browser UA -> SPA shell, Googlebot -> prerendered page via `@seo_proxy`) plus the `location =` bypasses for `robots.txt` and `llms.txt`; the API's covers `/health`, the public read paths and the fail-closed admin gate. -The two chains differ in one step: `app/cloudbuild.yaml` pushes `:latest` only after the -promotion, so the tag cannot name an image that was never rolled out, while -`api/cloudbuild.yaml` still pushes it alongside the deploy. Neither pipeline reads -`:latest` — both deploy `:$BUILD_ID` — so the tag is a convenience for humans, and with -two overlapping builds the later push wins regardless. +The two chains differ in two places, both on the app's side: + +- **`:latest`.** `app/cloudbuild.yaml` pushes it only after the promotion, so the tag + cannot name an image that was never rolled out; `api/cloudbuild.yaml` still pushes it + alongside the deploy. Neither pipeline *reads* `:latest` — both deploy `:$BUILD_ID` — so + the tag is a convenience for humans, and with two overlapping builds the later push wins + regardless. +- **The candidate-tag recheck.** `candidate` is a shared tag, so a concurrent build of the + same service can move it mid-smoke. `app/cloudbuild.yaml` re-asserts after its probes + that the tag still names this build's revision and refuses to promote otherwise; a + competing build only ever tags its own revision and never ours back, so matching at both + ends means every probe in between hit the right revision. `api/cloudbuild.yaml` checks + only before its probes, so two overlapping API builds can still promote a revision that + was not the one smoked — the same four lines would close it there. ## Debugging Tips From 10a1c98077e44e8ec7c53a4146a410042b326251 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:38:50 +0200 Subject: [PATCH 5/5] docs: say what the candidate-tag check does and does not prove MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on #11207: both tag checks read status.traffic, which is the control-plane tag assignment, not proof of which revision answered a given request. Because `candidate` is reused across builds, a probe issued while a re-assignment is still propagating can reach the previous revision. Documented rather than closed, in the smoke step and in the project guide, with the two ways to close it named: a build-unique tag (tag URLs then accumulate on the service without bound) or a build id the served response itself carries. The residual is narrow — it needs the previous candidate to pass every probe too, so the worst case is promoting a revision believed smoked, not shipping a page known to be broken. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01PBQdMbboxo59sSThGSbfke --- agentic/docs/project-guide.md | 11 ++++++++++- app/cloudbuild.yaml | 12 ++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/agentic/docs/project-guide.md b/agentic/docs/project-guide.md index 0efc4f61608..f581ca575ce 100644 --- a/agentic/docs/project-guide.md +++ b/agentic/docs/project-guide.md @@ -1049,10 +1049,19 @@ The two chains differ in two places, both on the app's side: same service can move it mid-smoke. `app/cloudbuild.yaml` re-asserts after its probes that the tag still names this build's revision and refuses to promote otherwise; a competing build only ever tags its own revision and never ours back, so matching at both - ends means every probe in between hit the right revision. `api/cloudbuild.yaml` checks + ends means the tag was never re-assigned during the smoke. `api/cloudbuild.yaml` checks only before its probes, so two overlapping API builds can still promote a revision that was not the one smoked — the same four lines would close it there. +Both checks read `status.traffic`, which is the **control-plane tag assignment** and not +proof of which revision answered a given request: because `candidate` is reused across +builds, a probe issued while a re-assignment is still propagating can reach the previous +revision. Closing that gap needs either a build-unique tag (tag URLs then accumulate on +the service without bound) or a build id the served response itself carries. Neither is in +place, and the residual is narrow: it needs the previous candidate to pass every probe as +well, so the worst case is promoting a revision that was believed smoked, not shipping a +page known to be broken. + ## Debugging Tips ### Database Connection Issues diff --git a/app/cloudbuild.yaml b/app/cloudbuild.yaml index 73917f89f3a..81e86d956de 100644 --- a/app/cloudbuild.yaml +++ b/app/cloudbuild.yaml @@ -121,6 +121,18 @@ steps: # build's revision (deterministic --revision-suffix) — a concurrent # build re-tagging `candidate` must fail this build, not get smoked # on its behalf. + # + # What this checks is the CONTROL PLANE: `status.traffic` is the tag + # assignment, not proof of which revision answered a given request. + # `candidate` is reused across builds, so in the moment after a + # re-assignment the tag URL can still reach the previous revision, and + # a probe would then pass against the wrong image. Closing that needs + # either a build-unique tag (which accumulates tag URLs on the service + # without bound) or a build id the app serves in its own response — + # neither is taken here. The residual is narrow and its worst case is + # mild: it needs the PREVIOUS candidate to pass every probe too, so the + # failure is "promoted a revision we only believed we smoked", not + # "shipped a page we know is broken". read -r URL REV <<< "$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['url'], c['revisionName'])")" test "$$REV" = "${_SERVICE_NAME}-b$BUILD_ID" || { echo "candidate tag moved to $$REV (expected ${_SERVICE_NAME}-b$BUILD_ID)"; exit 1; }