Skip to content

LAUNCH_HANDOFF.md

Latest commit

 

History

History
533 lines (453 loc) · 24.5 KB

LAUNCH_HANDOFF.md

File metadata and controls

533 lines (453 loc) · 24.5 KB
Raw

Launch handoff — what only you can do

Audience: Sb (you, the operator). Goal: every code-side launch blocker I (Claude) could close has been closed and committed. What's left is everything that needs a credit card, a signature, hardware, or a human decision — none of which I can do for you.

This list is sequenced by order-of-operations, not by importance. Tackle the dependencies first (auth, transports) so the later items (legal, support process) have something to point at.

Last refreshed: 2026-09-09. Items marked ✅ have shipped. Items marked 🟡 are partially done. Unmarked items are open.

Five remain, and none of them is code:

# Item Blocked on
1 Clerk production keys you — swap at the last minute before launch
3 Status page vendor you — optional, recommended
6 Lawyer review of legal templates counsel — DPA and SUB_PROCESSORS still say DRAFT — NOT FOR EXECUTION
8 Pi performance benchmark hardware access
11 On-call rotation you — a process, not a change

Item 12 (day-before go/no-go) is a checklist to run, not to close.

Items 2, 4, 5, 7, 9 and 10 are closed. Four of those were closed earlier and never marked, which made this list look considerably worse than reality — the only genuine blocker is counsel.

1. Clerk production keys

State now. The dashboard is using Clerk's test keys (pk_test_*, sk_test_*). These work end-to-end for auth and billing in the sandbox, but they're scoped to the test environment — real Stripe charges don't post, the dev-mode badge shows in the UI, and the "Clerk dev keys in prod" memory note documents this is intentional for now.

What you need to do.

  1. In the Clerk dashboard, switch the application to production mode (or create a separate production app and copy the user/organization schema across — Clerk has a one-click clone for this).
  2. Configure the production Stripe account via Clerk's billing tab (Clerk handles Stripe under the hood per MEMORY.md::project_billing_stack).
  3. Update Fly secrets: 
    fly secrets set \
  CLERK_PUBLISHABLE_KEY=pk_live_... \
  CLERK_SECRET_KEY=sk_live_... \
  -a sentinel-command
  4. Verify the Clerk webhook endpoint https://app.sentinel-command.com/api/webhooks/clerk is registered in the production Clerk app and signing secret is set (CLERK_WEBHOOK_SECRET). Test by upgrading a test org and confirming the Setting(org_plan="pro") row shows up.

Verification. After the secret swap, sign out and sign back in. The dev-mode badge in the corner should disappear.

2. Notification transport — Resend email ✅ DONE (live)

2026-07-05: Resend is configured and EMAIL_ENABLED=true in production. All 15 notification-email kinds are render-covered by backend/tests/test_email_templates.py. The original operator walkthrough is retained below for reference.

State now. Email v1 + v1.1 are fully built and deployed. 12 notification kinds gated by 7 per-org per-kind toggles. Six default ON for new orgs (camera offline/recovered, CameraNode offline/recovered, AI-agent incidents, MCP key audit, CameraNode disk, member audit); motion defaults OFF with a per-camera 15-min cooldown + digest mechanism for volume control. Transport is Resend; integration lives in app/core/email.py, app/core/email_worker.py, app/core/recipients.py, app/core/email_templates.py, app/core/email_unsubscribe.py. 22 Jinja2 templates in app/templates/emails/. Webhook-driven bounce/ complaint handling at /api/webhooks/resend writes to EmailSuppression. Sub-processor disclosure already in SUB_PROCESSORS.md + DPA.md. Marketing copy already swept across SecurityPage / PricingPage / FAQ / docs Notifications.

SMS and mobile push are still explicitly out of scope per the project_notification_channels memory note. MCP-driven external alerting (wire to Twilio / PagerDuty via your own MCP agent) is the answer for those.

Operator action to activate email:

  1. Sign up at resend.com (free tier covers 3K emails/month — comfortably above realistic volume for the operator-critical kinds).
  2. Verify a sending domain — Resend gives you 4 DNS records (SPF TXT, DKIM CNAMEs ×3, optional DMARC). 15-60 min for DNS to propagate. This step is already done: sentinel-command.com is verified, with DKIM and SPF/Return-Path on send.sentinel-command.com. This previously recommended notifications.sourceboxsentry.com, which is the pre-rename brand and is not a verified sending domain — setting EMAIL_FROM_ADDRESS to it would have failed every send.
  3. Configure a webhook in Resend → endpoint https://app.sentinel-command.com/api/webhooks/resend. Copy the signing secret (starts with whsec_).
  4. Set the four Fly secrets: 
    fly secrets set \
  RESEND_API_KEY=re_... \
  RESEND_WEBHOOK_SECRET=whsec_... \
  EMAIL_FROM_ADDRESS=notifications@sentinel-command.com \
  EMAIL_ENABLED=true \
  -a sentinel-command
    (Done — sending is verified on sentinel-command.com; the code defaults already match these values so the secrets are only needed to override. notifications@ is a no-reply sender by design — support is a separate channel at support@sentinel-command.com.)
  5. Smoke test: kill a CameraNode for >90s, watch the test admin's inbox for the offline email, click the unsubscribe link, verify the toggle flipped off in /settings. Then repeat for motion: trigger a camera, confirm the first-motion email arrives, and confirm a second trigger inside the cooldown window produces a digest rather than a second email.

Code is safe to keep deployed indefinitely with EMAIL_ENABLED=false (the default). The worker still runs but the transport short-circuits with a logged "would have sent" line.

3. Status page vendor (recommended)

State now. Two health endpoints live and ready for an external monitor to poll:

  • /api/health/ready (commit 6265b32, 2026-05-04) — readiness check with DB + Clerk + disk + email-worker probes. Returns HTTP 200 when ready, 503 with detail body when any critical dependency is unhealthy. 30s cached so a swarm of pollers doesn't hammer Clerk. This is the better target for external status pages and uptime monitors because it speaks HTTP status codes rather than nesting status in the body.
  • /api/health/detailed — verbose status snapshot, always 200. Useful for dashboards that parse JSON; bad for vendors that only check status codes.

There's no public status page yet.

Options.

  • BetterStack / Better Uptime — modern, generous free tier.
  • Instatus ($20/mo for the smallest paid plan, free tier works for solo operations).
  • Statuspage.io by Atlassian (more features, pricier).
  • UptimeRobot — free tier, good for the "page me when it's actually down" minimum.

What to do.

  1. Create a status page / synthetic monitor on the chosen vendor.
  2. Point the synthetic monitor at /api/health/ready (NOT /detailed) every minute. The monitor only needs to look at HTTP status: 200 = up, 5xx = down. Body is for humans.
  3. Link the status page from /security (replace the placeholder "No public status page yet" line in the "Honest gaps" section).
  4. Subscribe customers to status updates via the vendor's subscription widget — automatic for most.

4. Domain / DNS ✅ DONE

State now. Live and serving. app.sentinel-command.com (the app and API) holds a Fly-issued cert, valid to 2026-12-02; the apex sentinel-command.com (marketing site) is valid to 2026-12-07. CORS is hard-coded for that origin (backend/app/main.py::cors_origins).

Note the split, because it has bitten the docs twice: the API lives on app., not the apex. The apex has no /api and no /docs. Anything pointing an integration, webhook or health probe at the bare domain is wrong.

The rest of this section is a procedure for changing domains later, not outstanding work.

To switch.

  1. Buy a domain (e.g. sentry.sourceboxlabs.com).
  2. Add a Fly cert via fly certs add.
  3. Update cors_origins in app/main.py to include the new domain.
  4. Update FRONTEND_URL env var on Fly.
  5. Update Clerk's allowed origins to include the new domain.
  6. Update install.sh (Linux/macOS) so it points at the new base URL — this URL gets baked into customer CameraNodes at install time, so transitioning takes weeks. The Windows MSI doesn't need a parallel update because it's a static download from GitHub Releases (the MSI's URL doesn't change with the Command Center domain).

5. Sentry production setup ✅ DONE

State now. Sentry is fully wired and verified in production. SENTRY_DSN is set in Fly secrets via the Sentry extension (fly ext sentry create -a sentinel-command provisioned a sponsored Team plan and auto-injected the DSN). SENTRY_TRACES_SAMPLE_RATE=0.1 keeps us inside the free-tier event budget. app/core/sentry.py::init_sentry() no-ops gracefully when DSN is absent (local dev), so no extra config needed there. Email alerting confirmed firing — you've received at least one Sentry alert email (OPENSENTRY-COMMAND-1).

The disk-check loop that was added in the SaaS-readiness sweep (_check_and_emit_disk_critical at 95% threshold) routes its alert via logger.error() with structured extra fields, which Sentry captures as a server-side event. This replaced an earlier (incorrect) attempt to email customer admins about the platform disk — see ADR in commit 594b86c for the multi-tenant violation rationale.

Dashboard: fly ext sentry dashboard -a sentinel-command.

6. Lawyer review of legal templates

State now. I wrote docs/legal/DPA.md and docs/legal/SUB_PROCESSORS.md as engineering-truth working drafts. Both lead with DRAFT — NOT FOR EXECUTION so nobody can sign them accidentally.

What you need to do.

  1. Find a privacy lawyer. Many SaaS-friendly firms have flat-fee "starter DPA review" packages for early-stage companies in the $1.5–4K range.
  2. Send them the markdown drafts. They will return a redlined PDF.
  3. Save the lawyer-approved PDF in your records system (NOT in this repo — the markdown stays as the engineering record).
  4. When sub-processors change, update SUB_PROCESSORS.md in master and email the billing contact (per the DPA's 14-day notice policy). The repo edit IS the public notice.

Other legal templates you may need that I haven't drafted.

  • Terms of Service (the existing /legal page has an outline; have the lawyer review it).
  • Privacy Policy (same — check /legal).
  • Acceptable Use Policy (probably worth one, given the camera context — what users cannot point cameras at).

7. Backups and disaster recovery ✅ DONE

2026-07-06 restore drill — VERIFIED; blocker found AND fixed. A Fly volume snapshot restored end-to-end into a throwaway volume: PRAGMA integrity_check = ok, all 20 tables present (see docs/runbooks/DISASTER_RECOVERY.md). The drill exposed that the live DB was /data/opensentry.db (a DATABASE_URL secret overrode the fly.toml sentinel.db env), so backup_db.sh had been failing on the missing /data/sentinel.db. Fixed same day: secret repointed to sqlite:////data/sentinel.db, app restarted onto a fresh sentinel.db (empty pre-launch DB, no data lost), leftover opensentry.db + orphaned opensentry_data volume removed, /api/health/detailed = database ok, and a manual backup run succeeded. Remaining optional: set BACKUP_ENCRYPTION_KEY — closed 2026-09-07: declined. Backups are Fly-only by decision; see DISASTER_RECOVERY.md.

2026-09-07 — migrated to Postgres; drill re-run and PASSED. The paragraph below described SQLite-on-a-volume, which is no longer how this runs. See docs/runbooks/DISASTER_RECOVERY.md for the current procedure and the new drill log entry.

State now. The hosted database is Postgres, in the sentinel_command database on the managed sentinel-postgres cluster (shared with Sync-Service and License-Service — separate databases). DATABASE_URL is a Fly secret, not a fly.toml env value, because it carries a password. The sentinel_data volume still exists but now holds only HLS segment working files and /data/backups — losing it no longer loses data. Backups are the cluster's managed snapshots (primary) plus a daily pg_dump from .github/workflows/backup.yml (portable secondary).

Self-hosted installs still run SQLite; the codebase supports both and CI tests both.

What you need to do.

  1. Verify the cluster's snapshot schedule: 
    fly volumes list -a sentinel-postgres
fly volumes snapshots list <volume_id> -a sentinel-postgres
    You should see daily snapshots going back 5+ days.
  2. Test a restore. ✅ Done 2026-09-07 — see the drill log in DISASTER_RECOVERY.md. Re-run quarterly. The procedure restores a production pg_dump into a throwaway postgres:18-alpine container, deliberately not the origin cluster, since restoring somewhere else is the scenario the portable dump exists for: 
    docker run -d --name pgdrill -e POSTGRES_PASSWORD=drill \
  -e POSTGRES_DB=drill -p 15499:5432 postgres:18-alpine
DATABASE_URL=postgresql://postgres:drill@127.0.0.1:15499/drill \
  bash backend/scripts/restore_db.sh <dump> --yes
    Sanity-check key tables have rows: Camera, CameraNode, Setting, Notification.
  3. Document the restore procedure in docs/runbooks/DISASTER_RECOVERY.md (still unwritten — wait until you've done a real restore so you can capture what actually broke vs. what worked).

Single-machine deploy caveat: because we run a single Fly machine with a single volume, "restore" means downtime. The deploy strategy is immediate (also documented in fly.toml), so a deploy already involves ~30-60s of unavailability. A restore would be similar but with the additional manual swap step. Acceptable for current scale; worth re-evaluating when usage warrants HA (LiteFS or migrating to Postgres for clusterability).

8. Pi performance benchmark

State now. The CameraNode README and the /docs site describe the node as running on "any Linux, macOS, or Windows machine, including a Raspberry Pi". I haven't validated that actually works under a realistic camera load.

What you need to do.

  1. Get a Pi 4 (or Pi 5, increasingly common). Install Sentinel CameraNode via the install script.
  2. Connect 1, 2, 4 USB cameras at 1080p / 30fps and watch:
    • CPU steady-state under load.
    • Memory steady-state.
    • Egress bandwidth to Command Center.
    • Whether motion detection completes within the segment window.
  3. Document the result somewhere — at minimum in Sentinel-CameraNode/README.md under a "Performance reference" section. If a Pi 4 only handles 2 cameras at 1080p, that's useful for users to know upfront. If it handles 8, even better.

If the Pi turns out to be too weak for the advertised use case, update the docs honestly. Better to say "Pi 5 recommended for 4+ cameras" than to lose a customer who tried it on a Pi 3.

9. GitHub repo settings ✅ DONE

State now (2026-09-09). Branch protection on master:

  • allow_force_pushes: false · allow_deletions: false · required_linear_history: true
  • Required status checks: Backend tests (sqlite), Backend tests (postgres), Frontend audit + build
  • Required PR reviews: 1 approver, dismiss_stale_reviews: true
  • enforce_admins: false — the sole admin can override via the UI. Defense against fat-finger, not against deliberate action.

This item previously read "status-check + review enforcement deferred" on the grounds that there was no PR flow and checks "only kick in during merges, so they're decorative in direct-push mode". Both premises are gone: all work now lands through PRs, and the checks gate them.

strict (require branch up to date) is deliberately off. With it on, every Dependabot PR needs a rebase whenever master moves, which stalls auto-merge for no safety gain — the checks still run against the PR head.

Two things this does NOT fix, both live:

  • Dependabot auto-merge does not deploy. GitHub doesn't fire on: push workflows for commits pushed with GITHUB_TOKEN, so an auto-merged bump lands on master without deploying, and CodeQL going green on it hides that convincingly. Needs a PAT (DEPENDABOT_PAT); deploy.yml has a workflow_dispatch trigger as the interim lever. See the notes at the top of .github/workflows/dependabot-auto-merge.yml.
  • CI does not lint the frontend — audit, test and build only, which is why several eslint errors have sat unnoticed. The backend gates on ruff.

Dependabot: now configured on all four repos (Command Center, CameraNode, License, Sync). Only Command Center has an auto-merge workflow, deliberately — the other three deploy on push, so a GITHUB_TOKEN merge there would silently skip their deploy too.

10. Customer support process ✅ DONE (inbox live)

State now (2026-07-05). Support + security mailboxes are live via ImprovMX email forwarding on sentinel-command.com (MX → mx1/mx2.improvmx.com), forwarding to the operator inbox:

  • support@sentinel-command.com — customer support. Wired into the in-app ErrorBoundary crash screen, the SUB_PROCESSORS.md sub-processor-concern channel.
  • security@sentinel-command.com — vulnerability reports. Published as the primary Contact: in /.well-known/security.txt (GitHub Security Advisories remains as the secondary channel), in SECURITY.md, and in the DPA's vulnerability-management section.

ImprovMX free tier covers this comfortably; note it handles incoming mail only — outbound transactional email still goes through Resend (item 2). Both coexist on the domain: MX points at ImprovMX, SPF/DKIM (TXT) authorize Resend, so there's no conflict.

Remaining (optional, not blocking).

  1. Define an internal first-response target (e.g. within 1 business day). Don't promise an SLA on the public site at the Free / Pro tiers (the security page already says "No formal SLA on Free or Pro").
  2. Keep /docs#faq current so customers can self-serve the common questions before they email.

11. On-call rotation

State now. You're a one-person team. The runbook (docs/runbooks/ON_CALL.md) is written as if any human can pick up a page.

What you need to do later.

  1. As soon as you have a second engineer / co-maintainer, define a PagerDuty (or alternative) rotation.
  2. Update the runbook with rotation contact info.
  3. The runbook itself doesn't change — it's already in the "scannable under pressure" shape.

12. Final go/no-go checklist (to run the day before launch)

[ ] Clerk production keys swapped (item 1)
[X] Backup restore tested (item 7) — Fly snapshot restore VERIFIED 2026-07-06;
    the opensentry.db/sentinel.db mismatch found during the drill was FIXED same
    day (DATABASE_URL secret repointed to sentinel.db, app healthy, backup job now
    succeeds). BACKUP_ENCRYPTION_KEY deliberately left unset — Fly-only
    backups is an accepted decision, not an outstanding task.
[ ] DPA + sub-processors PDF on file with lawyer signoff (item 6)
[ ] Status page live and pointed at /api/health/ready (item 3)
[X] Sentry alerts confirmed firing in production env (item 5)        — done 2026-05-03
[ ] Custom domain (if applicable) live + Clerk allows it (item 4)
[X] Branch protection enabled on master (item 9)                      — done 2026-05-04
[X] Support inbox configured and monitored (item 10)                 — done 2026-07-05 (ImprovMX: support@ + security@)
[X] Resend signup + EMAIL_ENABLED=true + smoke test (item 2)         — done
[ ] Run `cd backend && uv run pytest` — all green (450+ tests)
[ ] Run `cd frontend && npm run build && npm audit --omit=dev` — both clean
[ ] Browse the live site at 375px, 1024px, 1440px — nothing broken
[ ] Hit /api/health/detailed — overall "healthy", DB latency < 50ms,
    disk.percent_used < 80%, resend.status either "ok" or
    "unconfigured" (intentional pre-launch)

When all twelve check, ship the launch announcement.

What's shipped since the original draft (audit trail)

  • 2026-04-26 → 2026-05-01: SaaS-readiness sweep — composite indexes on McpActivityLog + MotionEvent, disk-full alarm in /api/health/detailed, motion-ingestion per-org kill switch, HLS global byte-cap eviction. Tigris/AWS dead-secret cleanup. Marketing pass 1+2 (SEO meta + benefit-first hero copy + Clerk dark theme). Docs drift fixes (Postgres→SQLite, ~15s→~60s cache buffer, MCP tool count corrections, SLA wording).
  • 2026-05-02: Verified Sentry production setup (item 5 done).
  • 2026-05-03: Email v1 — Resend transport + worker + recipient lookup + 3 new tables, create_notification email side-channel, /api/webhooks/resend, disk-check loop, templates + UI + copy sweep + DPA + sub-processor disclosure. Three review-fix commits (idempotency-key routing, rate-limit on unsubscribe, EmailLog + EmailOutbox retention).
  • 2026-05-04: Multi-tenant violation removed (disk_critical no longer routes to customers — operator-only Sentry path). Four new email kinds added (camera/node recovery, MCP key audit, CameraNode disk warning, member audit via Clerk webhook). Motion email v1.1 with per-camera cooldown + digest. CI workflow rewritten three times (Fly remote builder → depot.dev → local Buildkit on the runner) after WireGuard auth regression on Fly's side. Branch protection on master.
  • 2026-05-04 → 2026-05-05 (SaaS launch-checklist closeout, 14 commits): Operator-debugging hygiene — per-request IDs in a contextvar, ContextFilter that injects request_id+org_id into every log line, Sentry tag, X-Request-Id response header, ruff in CI with conservative ruleset. Multi-tenant rate-limit audit caught + closed 7 missing-decorator endpoints including 3 SSE streams, 4 admin DB endpoints, the incident- evidence proxy, and a custom in-memory connect-throttle for the WebSocket (slowapi only does HTTP). Full first-touch UX pass — welcome email on organization.created, Help link in authenticated nav, in-app CameraNode install widget that auto-creates a node + bakes credentials into the displayed one-liner, contextual ? tooltips on the three highest- confusion settings, member promotion-request button. Audit- log CSV export across all three audit endpoints with shared streaming helper. RFC 9116 /.well-known/security.txt with rolling 11-month Expires + full Vulnerability Disclosure Policy section on /security with CFAA safe-harbour language. GDPR Article 17 cascade gap-fix (both danger/full-reset and the organization.deleted Clerk webhook were only clearing 5–7 of 14 org-scoped tables — fixed via app/core/gdpr.py as single source of truth) + Article 20 export endpoint streaming a ZIP per table. /api/health/ready with DB + Clerk + disk + email-worker probes returning 503 on critical failure (the existing /api/health and /detailed always returned 200, useless for external uptime monitors). pip-audit in CI for backend deps; vitest wired into the frontend CI step (suite existed but wasn't running, gated nothing); 23 new frontend tests including one that caught a real interaction bug in HelpTooltip on touch devices. Closer: built OrgAuditLogPanel admin component on top of the now- paginated /api/audit-logs, completing the admin dashboard's audit surface. Backend tests 464 → 549.
  • 2026-05-05 cleanup pass: Pulled drop_orphan_tables and sanitize_existing_codecs out of the boot path (one-shot fixes that had been no-op'ing for weeks); kept as documented helpers for snapshot-restore. Deleted dead EmptyState component + tests (superseded by WelcomeHero). Renamed misleading "Recording toggled (legacy)" audit label. Replaced dead legal@sourcebox.dev contact in LegalPage.jsx with in-app self-serve (Settings → Privacy & Data) for Article 17 / 20 / CCPA + GitHub Issues for everything else until the legal@ mailbox lands.

Originally closed by Claude on 2026-04-25. Refreshed 2026-05-05 after the SaaS launch-checklist closeout sweep (req-IDs, rate-limit audit, first-touch UX, audit CSV, security.txt, GDPR delete + export, /healthz/ready, pip-audit + vitest in CI, AuditLog UI, cleanup pass). The remaining items still all require you — credit cards, signatures, hardware, human decisions.