fix(backup): start the VM before ssh console, stop it after - #9
Draft
Sbussiso wants to merge 1 commit into
Draft
Conversation
The scheduled backup workflow fails with "app sentinel-license has no started VMs" because fly.toml scales the app to zero (auto_stop_machines=stop, min_machines_running=0). At 09:47 UTC the service has no self-hosted installs calling in, so its only VM is stopped. flyctl ssh console connects over WireGuard/SSH, not through the HTTP proxy that auto-starts machines, so it cannot reach a stopped machine. Fix: start the machine before the dump, pin all subsequent ssh/sftp commands to it with --machine, and stop it at the end with if:always() so the app returns to scale-to-zero (zero cost when idle) even if a later step fails. No app code, secrets, or infra config touched — workflow only. Fixes run #4 (34484996772).
Sbussiso
referenced
this pull request
Sep 10, 2026
* Deploy from CI — fly.toml was a file that did nothing
Deploy automation was deferred so a human could watch the first-ever
deploy of new infrastructure, with a note to automate once phase 1 was
stable. Phase 1 is stable, and leaving it deferred turned out worse than
what it avoided: fly.toml only took effect if someone remembered to run
`fly deploy` by hand.
That failed silently on 2026-09-09. A scale-to-zero change merged with
CI fully green and never reached Fly — the app kept running always-on,
and the config looked applied because the commit was on master. It was
only caught by checking the machine state directly. Config that silently
doesn't apply is more dangerous than no config, because it reads as done.
Deploy runs on push to master only, after BOTH matrix legs (sqlite and
postgres) pass — this service runs one codebase against either dialect,
so a green sqlite leg alone is not evidence a deploy is safe.
Two flags, each for a concrete reason:
--strategy immediate this app mounts sentinel_license_data, and the
default rolling strategy stands up a parallel
machine first, which errors on the volume's
single attachment slot.
--ha=false Fly otherwise provisions two machines. It did
exactly that on the manual deploy of the sibling
Sync service and the extra had to be scaled away
by hand; the volume could not serve two anyway.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Watch dependencies — this repo had nothing watching them
No Dependabot config existed here. That is the same gap that produced
the Sentinel AI agent's three production breakages in two days: mcp 2.x
renaming a symbol, Starlette removing on_startup, and a ceiling added to
stop the second one silently pinning the service to a Starlette line
carrying 7 advisories. All three surfaced on a rebuild of a repo nobody
had rebuilt since June.
`pip-audit --strict` in CI catches a known advisory against what is
already pinned. It does nothing about a lockfile quietly ageing out of
support, which is the failure that actually happened.
No auto-merge workflow, deliberately. This repo now deploys to Fly on
push to master, and GitHub does not trigger `on: push` workflows for
commits pushed with GITHUB_TOKEN — so an auto-merge would land a bump on
master WITHOUT deploying it, and nothing would report an error. That is
an observed bug in Sentinel-Command, where four frontend bumps merged
and never shipped. A human merge triggers the deploy normally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* README: this service deploys from CI and scales to zero
Two claims were false. "First deploy is manual by design — deploy
automation is deferred" stopped being true when the deploy job landed;
leaving it deferred had made fly.toml a file that did nothing, and a
scale-to-zero change merged green and never reached Fly. And nothing
documented that the service now sleeps between check-ins, or why that is
safe (4s boot inside Fly's ~8s proxy window, 10s caller timeout, and a
72-hour grace window on the Command Center side).
Also records the condition that would reverse it: a licence check moving
onto a user-blocking path turns a 4s cold start into a 4s page stall.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause — run #4 (34484996772) failed
The Scheduled DB Backup workflow failed at the first step, Run pg_dump on the machine:
fly.tomlscales this service to zero when idle (auto_stop_machines = "stop",min_machines_running = 0). With no self-hosted installs calling in yet, the only VM is stopped at 09:47 UTC when the cron fires.flyctl ssh consoleconnects over WireGuard/SSH, not the HTTP proxy that auto-starts machines — so it cannot reach a stopped machine and fails immediately.Fix
Workflow-only change to
.github/workflows/backup.yml— no app code, no secrets, no infra config:flyctl machine list -qto find the ID,flyctl machine start, thensleep 8(app boots in ~4s; Fly proxy allows ~8s — both perfly.toml).ssh console/ssh sftpcalls to that machine with--machine "$machine_id"(stored via$GITHUB_ENV).if: always()to restore scale-to-zero — so a failed run does not leave the VM running and accumulating cost (with no installs yet, nothing would trigger auto-stop).Why this is safe
DATABASE_URL,fly.toml, or secrets.scripts/backup_db.sh) is unchanged.if: always()on the stop step guarantees the app returns to zero-cost idle state regardless of outcome.Testing
FLY_API_TOKENforsentinel-licensehere). Recommend Run workflow → merge once run build(deps): bump actions/upload-artifact from 4 to 7 #5 goes green.