From ab6b3f92d716204f2075a42acf73954659bbaef5 Mon Sep 17 00:00:00 2001
From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com>
Date: Thu, 10 Sep 2026 15:33:34 -0700
Subject: [PATCH] Fix: scale-to-zero silently broke the nightly backup
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Today's scheduled backup failed with "app sentinel-license has no
started VMs". Cause is the 2026-09-09 change that made this service
scale to zero — my own change, and I did not think through this
consequence.

`flyctl ssh console` does not wake a stopped machine. Fly auto-starts on
a connection to one of the app's SERVICES; SSH is not that path. So from
the moment the app started sleeping, every scheduled run failed at the
first step and no dump was written.

Silent in the way that matters: nothing alerts on it. The only signal is
a red X on a scheduled workflow nobody watches, and the failure mode of
a backup is that you discover it at restore time.

The job now starts the machine explicitly before SSH, then waits for the
port to bind. Safe when already running, and the machine idles back to
stopped on its own afterwards, so scale-to-zero is preserved.

Note for anyone copying this workflow: Command Center's equivalent does
NOT need this, because its web tier never sleeps. The two are otherwise
meant to stay in step — this is a deliberate divergence, not drift.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 .github/workflows/backup.yml | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/.github/workflows/backup.yml b/.github/workflows/backup.yml
index ced61eb..4ff58d6 100644
--- a/.github/workflows/backup.yml
+++ b/.github/workflows/backup.yml
@@ -36,6 +36,29 @@ jobs:
     steps:
       - uses: superfly/flyctl-actions/setup-flyctl@master
 
+        # REQUIRED since this app started scaling to zero (2026-09-09).
+        #
+        # `flyctl ssh console` does NOT wake a stopped machine. Fly
+        # auto-starts on a connection to one of the app's SERVICES; SSH is
+        # not that path, so the very next step failed outright with
+        # "app sentinel-license has no started VMs" — and it failed
+        # SILENTLY, in the sense that nothing but this workflow's own red
+        # X reported that a day's backup had not been taken.
+        #
+        # Starting explicitly is the fix. It is safe when the machine is
+        # already running (no-op), and the machine idles back to stopped
+        # on its own afterwards, so scale-to-zero is preserved.
+      - name: Wake the machine (it scales to zero)
+        run: |
+          MACHINE=$(flyctl machines list -a sentinel-license --json | jq -r '.[0].id')
+          test -n "$MACHINE" && test "$MACHINE" != "null"
+          flyctl machine start "$MACHINE" -a sentinel-license || true
+          # Give the process time to bind before SSH; boot is ~4s.
+          for i in $(seq 1 20); do
+            flyctl ssh console -a sentinel-license -C "true" >/dev/null 2>&1 && break
+            sleep 3
+          done
+
       - name: Run pg_dump on the machine
         # Runs on the machine so DATABASE_URL stays in Fly and is never
         # copied into GitHub secrets.