From 66f47a94d2144f61a777fcc7072815e6d2ea240f Mon Sep 17 00:00:00 2001 From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com> Date: Wed, 9 Sep 2026 14:19:38 -0700 Subject: [PATCH 1/3] =?UTF-8?q?Deploy=20from=20CI=20=E2=80=94=20fly.toml?= =?UTF-8?q?=20was=20a=20file=20that=20did=20nothing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deploy automation was deferred so a human could watch the first-ever deploy of new infrastructure, including this family's first Postgres instance. That rationale expired once the service went live, and leaving it deferred turned out worse than what it avoided: fly.toml only took effect if someone remembered to run `fly deploy` by hand. That failed silently on 2026-09-09. A scale-to-zero change merged with CI fully green and never reached Fly — the app kept running always-on, and the config looked applied because the commit was on master. It was only caught by checking the machine state directly. Config that silently doesn't apply is more dangerous than no config, because it reads as done. Deploy runs on push to master only, after the tests pass. --ha=false because Fly provisions TWO machines by default. It did exactly that on this service's manual deploy and the extra machine had to be scaled away by hand. No --strategy override: this app has no volume, so the default rolling strategy is fine. The sibling License service needs `immediate` because it mounts one. Needed a repo secret that did not exist — this repo had NO secrets at all, so a deploy job would have failed on its first run. FLY_API_TOKEN is now set, scoped to sentinel-sync only. Co-Authored-By: Claude Opus 5 --- .github/workflows/test.yml | 44 ++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 1c78a6d..075a21b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -1,10 +1,15 @@ -name: Test +name: Test & Deploy -# Deploy automation is deferred deliberately, same rationale as -# Sentinel-License-Service's workflow of the same name: a human should -# be watching the first-ever deploy of new infrastructure (including, -# for this service, the first-ever Postgres instance in this family). -# This workflow only covers lint + dependency audit + tests. +# Tests on every push and PR; deploys to Fly on pushes to master. +# +# Deploy automation used to be deferred here so a human could watch the +# first-ever deploy of new infrastructure. That rationale expired once +# the service was live, and leaving it deferred created a worse problem: +# fly.toml became a file that did nothing. A scale-to-zero change was +# merged with CI green on 2026-09-09 and simply never reached Fly — +# the app kept running always-on until someone noticed and deployed by +# hand. Config that silently doesn't apply is more dangerous than no +# config, so this now ships. # # Unlike License-Service's version, this one needs a real Postgres # service container — JSONB isn't SQLite-portable, and this service is @@ -63,3 +68,30 @@ jobs: - name: Run tests run: uv run pytest -v + + deploy: + name: Deploy to Fly.io + runs-on: ubuntu-latest + needs: test + # Push-only: a PR runs the tests above but never ships. + if: github.event_name == 'push' + # Serialize so two quick pushes don't race on the machine update. + concurrency: + group: deploy-sentinel-sync + cancel-in-progress: true + steps: + - uses: actions/checkout@v7 + + - uses: superfly/flyctl-actions/setup-flyctl@master + + # --ha=false: Fly provisions TWO machines by default and this + # service needs one. It did exactly that on the 2026-09-09 manual + # deploy and the extra machine had to be scaled away by hand. + # + # No --strategy override: this app has no volume, so the default + # rolling strategy works. (Command Center needs `immediate` + # because its web machine mounts a single-attachment volume.) + - name: flyctl deploy + run: flyctl deploy --remote-only --ha=false --yes + env: + FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} From dd9134d26cda71933bf75e94c11847b7170557bf Mon Sep 17 00:00:00 2001 From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com> Date: Wed, 9 Sep 2026 14:29:31 -0700 Subject: [PATCH 2/3] =?UTF-8?q?Watch=20dependencies=20=E2=80=94=20this=20r?= =?UTF-8?q?epo=20had=20nothing=20watching=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No Dependabot config existed here. That is the same gap that produced the Sentinel AI agent's three production breakages in two days: mcp 2.x renaming a symbol, Starlette removing on_startup, and a ceiling added to stop the second one silently pinning the service to a Starlette line carrying 7 advisories. All three surfaced on a rebuild of a repo nobody had rebuilt since June. `pip-audit --strict` in CI catches a known advisory against what is already pinned. It does nothing about a lockfile quietly ageing out of support, which is the failure that actually happened. No auto-merge workflow, deliberately. This repo now deploys to Fly on push to master, and GitHub does not trigger `on: push` workflows for commits pushed with GITHUB_TOKEN — so an auto-merge would land a bump on master WITHOUT deploying it, and nothing would report an error. That is an observed bug in Sentinel-Command, where four frontend bumps merged and never shipped. A human merge triggers the deploy normally. Co-Authored-By: Claude Opus 5 --- .github/dependabot.yml | 51 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..4b2f5cd --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,51 @@ +# Dependabot configuration. +# +# This repo had NO dependency watching until 2026-09-09, which is the +# same gap that produced the Sentinel AI agent's three production +# breakages in two days: mcp 2.x renaming a symbol, Starlette removing +# on_startup, and a ceiling added to stop the second one silently +# pinning the service to a Starlette line with 7 advisories. All three +# surfaced on a rebuild of a repo nobody had rebuilt since June. +# +# `pip-audit --strict` in the Test workflow catches a *known advisory* +# in what is already pinned. It does nothing about a lockfile quietly +# ageing out of support. That is what this is for. +# +# NO AUTO-MERGE WORKFLOW HERE, DELIBERATELY. +# +# This repo now deploys to Fly on push to master. GitHub does not +# trigger `on: push` workflows for commits pushed with GITHUB_TOKEN, so +# an auto-merge action would land a bump on master WITHOUT deploying it +# — master and production drift apart, and nothing reports an error. +# That is a real, observed bug in Sentinel-Command (four frontend bumps +# merged and never shipped on 2026-09-09). A human merge triggers the +# deploy normally, so these PRs are reviewed and merged by hand until +# that repo's PAT fix is proven. +# +# Extra reason to be careful here: this service is Postgres-native and +# its tests need a real Postgres container, so a driver bump that only +# misbehaves against the real engine will not show up anywhere else. +version: 2 +updates: + # Python (uv) — pyproject.toml + uv.lock live at the repo root. + - package-ecosystem: "uv" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 5 + commit-message: + prefix: "build(deps)" + labels: + - "dependencies" + + # GitHub Actions — pinned action versions age out the same way. + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 5 + labels: + - "dependencies" + - "ci" From cc33cac580675199e6c50f05bcd8399f01527289 Mon Sep 17 00:00:00 2001 From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com> Date: Wed, 9 Sep 2026 18:22:02 -0700 Subject: [PATCH 3/3] README: this service is deployed, deploys from CI, and scales to zero MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Status section said "Not yet deployed — no Postgres instance or Fly app provisioned yet". It has been live since 2026-09-07, on the shared sentinel-postgres cluster in its own role-isolated database. A README that says a running production service doesn't exist is the worst kind of stale, because it reads as authoritative. The Deploy section was likewise pre-CI, and told the reader that provisioning Postgres was still their decision to make. Adds what wasn't documented anywhere: that the service now sleeps between its 30-minute pushes, why that's safe (~3s boot inside Fly's ~8s proxy window, and a fail-soft push whose cursors only advance on confirmed success, so a missed cycle loses nothing), and why it deliberately has no dump job. Co-Authored-By: Claude Opus 5 --- README.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ffdd5c9..61e7507 100644 --- a/README.md +++ b/README.md @@ -60,8 +60,20 @@ Full procedure, including what deliberately isn't mirrored (node API keys, evide ## Deploy -Single-stage `Dockerfile` (no frontend build — this service has no UI). `fly.toml`'s `release_command` runs `alembic upgrade head` before each deploy starts serving traffic. Needs a `DATABASE_URL` Fly secret pointing at a real Postgres instance (Fly Postgres, Neon, RDS, etc.) — provisioning that instance is a separate infra/cost decision, not something this repo does for you. +Single-stage `Dockerfile` (no frontend build — this service has no UI). `fly.toml`'s `release_command` runs `alembic upgrade head` before each deploy starts serving traffic. + +**Deploys from CI.** Every push to `master` runs the tests against a real Postgres, then `flyctl deploy --ha=false`. (`--ha=false` because Fly provisions two machines by default; it did exactly that on the manual deploy and the extra had to be scaled away by hand. No `--strategy` override is needed here — unlike the sibling License service, this app has no volume.) + +Deploy automation was deferred while this was new infrastructure. That turned out worse than what it avoided: `fly.toml` became a file that did nothing, and a scale-to-zero change merged with CI fully green on 2026-09-09 without ever reaching Fly. + +**Scales to zero.** Self-hosted installs push on a 30-minute background tick, so this is idle ~95% of the time. Safe because boot is ~3s (inside the ~8s Fly's proxy waits for an auto-started machine to bind) and a failed push is fail-soft *and lossless*: `push_pending_changes` never raises, and cursors only advance on confirmed success, so a missed cycle's data simply waits for the next tick with the operator's local SQLite authoritative throughout. ## Status -Built and verified locally: full test suite (unit tests against a real Postgres, including tenant-isolation and deletion-reconciliation regressions) plus a live cross-service integration check against a real running License-Service instance (valid+sync-enabled key → 200 and the row lands correctly scoped by tenant; unknown key → 403; License-Service unreachable → 502). Not yet deployed — no Postgres instance or Fly app provisioned yet. +**Deployed and live** at `https://sentinel-sync.fly.dev`, on the shared `sentinel-postgres` cluster in its own `sentinel_sync` database, access-isolated by role from the other two. + +Verified by the full test suite (unit tests against a real Postgres, including tenant-isolation and deletion-reconciliation regressions) plus a live cross-service integration check against a running License-Service instance: valid + sync-enabled key → 200 with the row correctly scoped by tenant; unknown key → 403; License-Service unreachable → 502. + +This section previously read "Not yet deployed — no Postgres instance or Fly app provisioned yet", which stopped being true on 2026-09-07. + +**No backup dump job, deliberately.** This database holds a *mirror*; every row was pushed from an operator's local SQLite, which stays the source of truth. Losing it entirely costs one sync cycle. It is covered by the cluster-level snapshot — see `DISASTER_RECOVERY.md` in the Command Center repo.