fix(ops): deploy waits for a running SSG rebuild; gate ignores ssg-worker content state

.github/workflows/deploy.yml

@@ -89,6 +89,30 @@ jobs:
             exit 1
           fi
 
+      # Never build while an SSG rebuild is rendering. The frontend build below empties
+      # apps/web/dist, which ssg-worker writes into (dist/ssg-new); a rebuild caught in that
+      # window fails by design (assertBuildSurvived — the 2026-08-31 guard) and leaves
+      # ssg-worker unhealthy. Rebuilds come from the nightly backup, publishing, the admin
+      # button and the periodic safety net, so any of them can be mid-run when a merge lands.
+      # Waits up to 40 min (a full rebuild is ~25), then proceeds with a warning: a wedged
+      # job must not block deploys forever, and the guard still protects the live tree.
+      # Only jobs touched in the last hour count — nothing recovers a job left 'Running' by a
+      # crash, and one such row would otherwise cost every deploy the full 40 minutes.
+      - name: Wait for any running SSG rebuild
+        run: |
+          DEADLINE=$(( $(date +%s) + 2400 ))
+          while :; do
+            ACTIVE=$(docker exec textstack_db_prod psql -U textstack_prod textstack_prod -tAc \
+              "SELECT count(*) FROM ssg_rebuild_jobs WHERE status IN ('Queued','Running') AND COALESCE(started_at, created_at) > now() - interval '60 minutes'" 2>/dev/null | tr -d '[:space:]')
+            [ "$ACTIVE" = "0" ] && { echo "No SSG rebuild in progress"; break; }
+            if [ "$(date +%s)" -ge "$DEADLINE" ]; then
+              echo "::warning::SSG rebuild still active after 40 min — deploying anyway"
+              break
+            fi
+            echo "SSG rebuild in progress ($ACTIVE active) — waiting… ($(date +%T))"
+            sleep 30
+          done
+
       - name: Snapshot SSG dir + referenced assets before vite wipes dist
         # Closes the deploy-race window where vite's emptyOutDir=true
         # nukes dist/ssg/ AND dist/assets/* before the SSG worker can
@@ -309,8 +333,10 @@ jobs:
           curl -sf --retry 30 --retry-delay 5 --retry-all-errors --retry-connrefused http://localhost:8080/health || exit 1
           echo "API health check passed"
           # `up -d` returns before healthchecks run; a container already marked unhealthy
-          # (worker, ssg-worker, mcp-server…) would otherwise ship green.
-          UNHEALTHY=$(cd $PROJECT_DIR && docker compose --profile mcp -f docker-compose.yml -f docker-compose.gpu.yml ps --format '{{.Service}} {{.Health}}' | awk '$2 == "unhealthy" {print $1}')
+          # (worker, mcp-server…) would otherwise ship green. ssg-worker is excluded: its
+          # healthcheck also reports "last rebuild failed", which is content state that
+          # health-check.yml alarms on — not a reason to call this deploy broken.
+          UNHEALTHY=$(cd $PROJECT_DIR && docker compose --profile mcp -f docker-compose.yml -f docker-compose.gpu.yml ps --format '{{.Service}} {{.Health}}' | awk '$2 == "unhealthy" && $1 != "ssg-worker" {print $1}')
           [ -z "$UNHEALTHY" ] || { echo "::error::unhealthy services: $UNHEALTHY"; exit 1; }
 
       # A full rebuild is ~25 min of Puppeteer on the prod box and held the only runner for it.

CHANGELOG.md

@@ -23,6 +23,7 @@ the archive; if it broke production, it belongs in `docs/incidents/`. See
 
 ## [Unreleased]
 
+- **Ops** — a deploy waits for any SSG rebuild in progress (≤40 min, only jobs active in the last hour) before the frontend build empties `dist/`; the post-deploy health gate no longer treats ssg-worker's "last rebuild failed" as a broken deploy — infra
 - **Ops** — deploys no longer run a full SSG rebuild (~25 min of CPU, held the only runner): it runs nightly after the backup; `rebuild_ssg` input on a manual deploy for releases that change SEO rendering; health check alarms at 36h stale (was 72h) — infra
 - **Security** — hardening from the architecture review: stored files are served with a sandbox CSP + nosniff (nginx and the API), `/internal/*` is refused at nginx and its network check is one tested helper, two copyrighted test PDFs removed (image-only test now uses a generated PDF) — backend, infra
 - **Security** — connect keys get the same account-route scope as OAuth tokens; per-IP rate limits group IPv6 by /64; nginx rate-limit zones key on the client address — api, infra