diff --git a/CHANGELOG.md b/CHANGELOG.md index ec1b748100..117e3b240b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -143,6 +143,27 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **The frontend deploys through a candidate revision instead of straight onto live + traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as + `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate + --revision-suffix=b$BUILD_ID`, smoke the candidate on its tag URL, then `update-traffic` + to exactly that revision (the chains are not identical — this one pushes `:latest` only + after the promotion, where the API still pushes it alongside the deploy). The service + carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =` + bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every + bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy + site; until now a typo in it went live unchecked and the daily bot-serving monitor was + the only net, a night later. The smoke probes both halves of the split (a browser UA + must get `
`, Googlebot must get the prerendered page — asserted on the + ``, which the SPA shell carries not at all and whose value names + the route) on the home page and a deep route, plus `robots.txt` and `llms.txt` from the + `location =` bypasses and the latter's UTF-8 charset. The candidate tag is re-asserted + after the probes as well as before, so a concurrent build moving it mid-smoke fails this + build instead of getting it promoted on someone else's evidence. `:latest` moves only + after this build's promotion, so the tag can no longer name an image that was never + rolled out, and the build timeout goes to 20 min to leave room for a cold candidate. + (#11207) + - **The `babysit-pipeline` skill gains the backfill scheduler and the driver's per-spec liveness check** — `run_queue.sh [slots]` keeps N `run_spec.sh` drivers in flight over a queue file, skips libraries already on main and confirmed gaps, diff --git a/agentic/docs/project-guide.md b/agentic/docs/project-guide.md index 7728070b79..0384a4d725 100644 --- a/agentic/docs/project-guide.md +++ b/agentic/docs/project-guide.md @@ -1027,6 +1027,42 @@ gcloud builds submit --config=app/cloudbuild.yaml --project=YOUR_PROJECT_ID - **`app/cloudbuild.yaml`**: Frontend build + deploy - **`app/Dockerfile`**: Multi-stage build (Node -> nginx) +Both cloudbuild files deploy through a **candidate revision**: `gcloud run deploy +--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, a smoke step against the +candidate's tag URL, then `gcloud run services update-traffic --to-revisions=…=100`. A +revision that fails its smoke therefore never takes **live traffic** — the only requests +it ever answers are the smoke's own, sent deliberately to its tag URL. (For the app that +holds even with two builds in flight; for the API it holds for one build at a time — see +the candidate-tag recheck below.) The app's smoke +covers both halves of the crawler split (browser UA -> SPA shell, Googlebot -> +prerendered page via `@seo_proxy`) plus the `location =` bypasses for `robots.txt` and +`llms.txt`; the API's covers `/health`, the public read paths and the fail-closed admin +gate. + +The two chains differ in two places, both on the app's side: + +- **`:latest`.** `app/cloudbuild.yaml` pushes it only after the promotion, so the tag + cannot name an image that was never rolled out; `api/cloudbuild.yaml` still pushes it + alongside the deploy. Neither pipeline *reads* `:latest` — both deploy `:$BUILD_ID` — so + the tag is a convenience for humans, and with two overlapping builds the later push wins + regardless. +- **The candidate-tag recheck.** `candidate` is a shared tag, so a concurrent build of the + same service can move it mid-smoke. `app/cloudbuild.yaml` re-asserts after its probes + that the tag still names this build's revision and refuses to promote otherwise; a + competing build only ever tags its own revision and never ours back, so matching at both + ends means the tag was never re-assigned during the smoke. `api/cloudbuild.yaml` checks + only before its probes, so two overlapping API builds can still promote a revision that + was not the one smoked — the same four lines would close it there. + +Both checks read `status.traffic`, which is the **control-plane tag assignment** and not +proof of which revision answered a given request: because `candidate` is reused across +builds, a probe issued while a re-assignment is still propagating can reach the previous +revision. Closing that gap needs either a build-unique tag (tag URLs then accumulate on +the service without bound) or a build id the served response itself carries. Neither is in +place, and the residual is narrow: it needs the previous candidate to pass every probe as +well, so the worst case is promoting a revision that was believed smoked, not shipping a +page known to be broken. + ## Debugging Tips ### Database Connection Issues diff --git a/app/cloudbuild.yaml b/app/cloudbuild.yaml index 83914a107f..81e86d956d 100644 --- a/app/cloudbuild.yaml +++ b/app/cloudbuild.yaml @@ -1,4 +1,19 @@ # Cloud Build configuration for anyplot Frontend +# +# This file follows the same candidate-rollout PATTERN as api/cloudbuild.yaml: +# build -> push the build tag -> deploy --no-traffic --tag=candidate -> smoke +# the candidate -> update-traffic to exactly that revision. The step chains are +# not identical: this one pushes `:latest` only after the promotion (the API +# still pushes it alongside the deploy) and has no final get-url step. +# +# Why this service deserves the same care as the API: it serves the public +# website AND carries the whole crawler path in app/nginx.conf (the `$is_bot` +# map, the `location =` bypasses for robots/llms/sitemap/og, the `@seo_proxy` +# upstream). That is the very file whose breakage served every bot an HTTP 502 +# for four weeks between 2026-06-12 and 2026-07-09 — a `proxy_ssl_verify_depth` +# default against a 4-deep chain — while humans, Plausible and CI all saw a +# healthy site. Until now a typo in it went live unchecked and the daily +# bot-serving monitor was the only net, a night later. substitutions: _REGION: 'europe-west4' _SERVICE_NAME: 'anyplot-app' @@ -28,10 +43,18 @@ steps: 'app/Dockerfile', 'app', ] + id: 'build-image' - # Push the container image to Artifact Registry + # Only the build-specific tag goes up here — it is the one the deploy pulls. + # `:latest` waits for the promote step (see push-latest at the bottom): a + # `--all-tags` push before the rollout means `:latest` names an image that + # may never have served a request. - name: 'gcr.io/cloud-builders/docker' - args: ['push', '--all-tags', 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}'] + args: + - 'push' + - 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:$BUILD_ID' + id: 'push-image' + waitFor: ['build-image'] # Deploy container image to Cloud Run - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' @@ -69,6 +92,138 @@ steps: - '--cpu-throttling' - '--concurrency' - '15' + # Deploy WITHOUT routing traffic: the smoke step probes this revision on + # its tag URL first, and only the promote step below shifts traffic — so + # a broken nginx.conf never serves visitors or crawlers, not even between + # deploy and a failing smoke. The `candidate` tag moves to each new + # revision, and the deterministic revision suffix lets promote target + # EXACTLY the revision this build smoked (never a concurrent build's + # newer one). `b` prefix: Cloud Run requires the suffix to start with a + # lowercase letter, and $BUILD_ID is a UUID that usually starts with a + # digit. + - '--no-traffic' + - '--tag=candidate' + - '--revision-suffix=b$BUILD_ID' + id: 'deploy' + waitFor: ['push-image'] + + # Pre-traffic smoke against the CANDIDATE revision (tag URL, serving no + # traffic yet). Both halves of the split matter and only one of them is + # visible to a human: the SPA shell for a browser, and the prerendered page + # for a mapped crawler UA — the `$is_bot` map plus the `@seo_proxy` upstream, + # which is exactly the hop that can break silently. + - name: 'gcr.io/cloud-builders/gcloud' + entrypoint: 'bash' + args: + - '-ceu' + - | + # Resolve the candidate tag's URL AND assert it still points at THIS + # build's revision (deterministic --revision-suffix) — a concurrent + # build re-tagging `candidate` must fail this build, not get smoked + # on its behalf. + # + # What this checks is the CONTROL PLANE: `status.traffic` is the tag + # assignment, not proof of which revision answered a given request. + # `candidate` is reused across builds, so in the moment after a + # re-assignment the tag URL can still reach the previous revision, and + # a probe would then pass against the wrong image. Closing that needs + # either a build-unique tag (which accumulates tag URLs on the service + # without bound) or a build id the app serves in its own response — + # neither is taken here. The residual is narrow and its worst case is + # mild: it needs the PREVIOUS candidate to pass every probe too, so the + # failure is "promoted a revision we only believed we smoked", not + # "shipped a page we know is broken". + read -r URL REV <<< "$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ + | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['url'], c['revisionName'])")" + test "$$REV" = "${_SERVICE_NAME}-b$BUILD_ID" || { echo "candidate tag moved to $$REV (expected ${_SERVICE_NAME}-b$BUILD_ID)"; exit 1; } + echo "smoke against candidate $$URL ($$REV)" + # Retry EVERY probe: this service runs at min-instances 0, so the + # candidate is ALWAYS cold, and the crawler probes additionally wait on + # the first upstream call to api.anyplot.ai. + RETRY="--retry 5 --retry-delay 5 --retry-all-errors" + HUMAN="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" + GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" + # `expect `: fetch to a file rather + # than piping into `grep -q`. `grep -q` exits at the first match and + # SIGPIPEs curl, so the pipe form only ever passes because `-ceu` + # carries no `pipefail` — a later hardening pass that adds it would red + # every deploy. Writing the body out also lets each failure say which + # probe failed and what was expected, which is the whole value of a log + # someone reads while a deploy is blocked. + expect() { + curl -fsS $$RETRY -A "$$1" -o body.out "$$URL$$2" \ + || { echo "candidate did not serve $$2 to UA '$$1'"; exit 1; } + grep -qF "$$3" body.out || { echo "$$4 ($$2 is missing: $$3)"; exit 1; } + echo "OK: $$2" + } + # Humans get the SPA shell. + expect "$$HUMAN" "/" '
' "a browser did not get the SPA shell" + # Crawlers get the prerendered page through @seo_proxy. The canonical + # link is the marker: the SPA shell carries none at all, and its value + # names the route, so one grep proves both "the bot hop ran" and "the + # right page came back". Probing a deep route as well as the home page + # covers `proxy_pass .../seo-proxy$$request_uri` — a home-only check + # would pass with the request URI dropped. (`$$` is Cloud Build's + # escape for a literal `$`; an unescaped one is read as a substitution + # and fails the build before a single step runs.) + expect "$$GOOGLEBOT" "/" '' \ + "a crawler did not get the prerendered home page" + expect "$$GOOGLEBOT" "/scatter-basic" '' \ + "a crawler did not get the prerendered page for this route" + # The machine files must come from the `location =` bypasses, not the + # proxy — and llms.txt carries UTF-8 punctuation that needs the charset. + expect "$$GOOGLEBOT" "/robots.txt" 'User-agent: Bytespider' \ + "robots.txt did not come from the location = bypass" + expect "$$GOOGLEBOT" "/llms.txt" '# anyplot' \ + "llms.txt did not come from the location = bypass" + ct=$$(curl -fsS $$RETRY -A "$$GOOGLEBOT" -o /dev/null -w '%{content_type}' "$$URL/llms.txt") + case "$$ct" in + *charset=utf-8*) ;; + *) echo "llms.txt served as '$$ct' (expected a utf-8 charset)"; exit 1 ;; + esac + # Re-assert the tag AFTER the probes. `candidate` is a shared tag, so a + # concurrent build could move it between the check above and the last + # probe, and this build would then have smoked someone else's revision + # while promoting its own. A competing build only ever tags its OWN + # revision and never ours back, so seeing our revision at both ends + # means every probe in between hit it. + REV_AFTER=$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ + | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['revisionName'])") + test "$$REV_AFTER" = "$$REV" || { echo "candidate tag moved to $$REV_AFTER mid-smoke (started on $$REV) — not promoting"; exit 1; } + echo "smoke OK" + id: 'smoke' + waitFor: ['deploy'] + + # Smoke passed — route 100% of traffic to EXACTLY the revision this build + # deployed and smoked (deterministic name via --revision-suffix), never + # `--to-latest`, which could promote a concurrent build's unsmoked revision. + - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' + entrypoint: gcloud + args: + - 'run' + - 'services' + - 'update-traffic' + - '${_SERVICE_NAME}' + - '--region' + - '${_REGION}' + - '--platform' + - 'managed' + - '--to-revisions=${_SERVICE_NAME}-b$BUILD_ID=100' + id: 'promote' + waitFor: ['smoke'] + + # `:latest` moves only after THIS build promoted, so it can no longer name an + # image that was never rolled out — which a `--all-tags` push before the + # deploy did. It is not a guarantee across concurrent builds: two overlapping + # deploys still race for the tag and the later push wins whichever revision + # serves. That is why the deploy step pulls `:$BUILD_ID` and never `:latest`; + # the tag is a convenience for humans, not an input to the rollout. + - name: 'gcr.io/cloud-builders/docker' + args: + - 'push' + - 'europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:latest' + id: 'push-latest' + waitFor: ['promote'] # Store images in Artifact Registry images: @@ -77,3 +232,7 @@ images: options: logging: CLOUD_LOGGING_ONLY + +# The chain is now deploy -> smoke (cold start + a first upstream call) -> +# promote; the default 10 min left no room for a retrying probe. +timeout: '1200s'