From 5553071a57683c2694d9e916f230dac6a1094125 Mon Sep 17 00:00:00 2001 From: Vasyl Vdovychenko Date: Sun, 4 Oct 2026 18:24:04 -0400 Subject: [PATCH] fix(ops): deploy waits for a running SSG rebuild; gate ignores ssg-worker content state Two deploys today failed the same way: an SSG rebuild was rendering into dist/ssg-new when the deploy's frontend build emptied dist/, the 08-31 guard failed the job, ssg-worker went unhealthy, and #691's new gate then failed the next deploy on it. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/deploy.yml | 30 ++++++++++++++++++++++++++++-- CHANGELOG.md | 1 + 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index ccd37234c..5e420c1b6 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -89,6 +89,30 @@ jobs: exit 1 fi + # Never build while an SSG rebuild is rendering. The frontend build below empties + # apps/web/dist, which ssg-worker writes into (dist/ssg-new); a rebuild caught in that + # window fails by design (assertBuildSurvived — the 2026-08-31 guard) and leaves + # ssg-worker unhealthy. Rebuilds come from the nightly backup, publishing, the admin + # button and the periodic safety net, so any of them can be mid-run when a merge lands. + # Waits up to 40 min (a full rebuild is ~25), then proceeds with a warning: a wedged + # job must not block deploys forever, and the guard still protects the live tree. + # Only jobs touched in the last hour count — nothing recovers a job left 'Running' by a + # crash, and one such row would otherwise cost every deploy the full 40 minutes. + - name: Wait for any running SSG rebuild + run: | + DEADLINE=$(( $(date +%s) + 2400 )) + while :; do + ACTIVE=$(docker exec textstack_db_prod psql -U textstack_prod textstack_prod -tAc \ + "SELECT count(*) FROM ssg_rebuild_jobs WHERE status IN ('Queued','Running') AND COALESCE(started_at, created_at) > now() - interval '60 minutes'" 2>/dev/null | tr -d '[:space:]') + [ "$ACTIVE" = "0" ] && { echo "No SSG rebuild in progress"; break; } + if [ "$(date +%s)" -ge "$DEADLINE" ]; then + echo "::warning::SSG rebuild still active after 40 min — deploying anyway" + break + fi + echo "SSG rebuild in progress ($ACTIVE active) — waiting… ($(date +%T))" + sleep 30 + done + - name: Snapshot SSG dir + referenced assets before vite wipes dist # Closes the deploy-race window where vite's emptyOutDir=true # nukes dist/ssg/ AND dist/assets/* before the SSG worker can @@ -309,8 +333,10 @@ jobs: curl -sf --retry 30 --retry-delay 5 --retry-all-errors --retry-connrefused http://localhost:8080/health || exit 1 echo "API health check passed" # `up -d` returns before healthchecks run; a container already marked unhealthy - # (worker, ssg-worker, mcp-server…) would otherwise ship green. - UNHEALTHY=$(cd $PROJECT_DIR && docker compose --profile mcp -f docker-compose.yml -f docker-compose.gpu.yml ps --format '{{.Service}} {{.Health}}' | awk '$2 == "unhealthy" {print $1}') + # (worker, mcp-server…) would otherwise ship green. ssg-worker is excluded: its + # healthcheck also reports "last rebuild failed", which is content state that + # health-check.yml alarms on — not a reason to call this deploy broken. + UNHEALTHY=$(cd $PROJECT_DIR && docker compose --profile mcp -f docker-compose.yml -f docker-compose.gpu.yml ps --format '{{.Service}} {{.Health}}' | awk '$2 == "unhealthy" && $1 != "ssg-worker" {print $1}') [ -z "$UNHEALTHY" ] || { echo "::error::unhealthy services: $UNHEALTHY"; exit 1; } # A full rebuild is ~25 min of Puppeteer on the prod box and held the only runner for it. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1dbe270af..b861aa0bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ the archive; if it broke production, it belongs in `docs/incidents/`. See ## [Unreleased] +- **Ops** — a deploy waits for any SSG rebuild in progress (≤40 min, only jobs active in the last hour) before the frontend build empties `dist/`; the post-deploy health gate no longer treats ssg-worker's "last rebuild failed" as a broken deploy — infra - **Ops** — deploys no longer run a full SSG rebuild (~25 min of CPU, held the only runner): it runs nightly after the backup; `rebuild_ssg` input on a manual deploy for releases that change SEO rendering; health check alarms at 36h stale (was 72h) — infra - **Security** — hardening from the architecture review: stored files are served with a sandbox CSP + nosniff (nginx and the API), `/internal/*` is refused at nginx and its network check is one tested helper, two copyrighted test PDFs removed (image-only test now uses a generated PDF) — backend, infra - **Security** — connect keys get the same account-route scope as OAuth tokens; per-IP rate limits group IPv6 by /64; nginx rate-limit zones key on the client address — api, infra