Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ CameraNode captures and encodes video on your network, then pushes it **outbound
| **Command Center** *(this repo)* | The hosted dashboard, API, and live-video hub | [Live app ›](https://sentinel-command.com) |
| **CameraNode** | The camera daemon you install on your own hardware | [Repo ›](https://github.com/SourceBox-LLC/Sentinel-CameraNode) |
| **Home Assistant integration** | Your Sentinel cameras inside Home Assistant | [Repo ›](https://github.com/SourceBox-LLC/Sentinel-HomeAssistant) |
| **Sentinel AI agent** | Serverless agent that investigates motion & incidents | [Repo ›](https://github.com/SourceBox-LLC/SourceBox-Sentinel) |
| **Sentinel AI agent** | AI agent that investigates motion & incidents | [Docs ›](docs/SENTINEL_AGENT.md) · in this repo at `backend/app/sentinel_agent/` |

---

Expand All @@ -94,6 +94,7 @@ CameraNode captures and encodes video on your network, then pushes it **outbound
| If you want to… | Go to |
|-----------------|-------|
| **Use Sentinel** — set up cameras, recording, notifications, integrations | The in-app [Documentation](https://sentinel-command.com/docs) |
| **See how the whole system fits together** — every repo, every deployed service, the paths between them | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) |
| **Understand the code** — architecture, API, data models, configuration | [AGENTS.md](AGENTS.md) |
| **Operate it** — decision records, runbooks, legal templates | [docs/](docs/) |
| **Audit or run the source locally** for review | [AGENTS.md › Build & Run](AGENTS.md) |
Expand Down
117 changes: 117 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Sentinel system architecture

How the whole system fits together — every repository, every deployed service, and the paths between them. Start here if you're new, or if you need to know where something lives before changing it.

`README.md` is setup and reference for this repo. `AGENTS.md` is the deep architecture reference for Command Center's internals. **This file is the level above both**: the system, not the codebase.

*Verified against the running infrastructure on 2026-09-09.*

## The pieces

### Repositories

| Repo | What it is | Language |
| ---- | ---------- | -------- |
| **Sentinel-Command** (this one) | Command Center: API, dashboard, MCP server, and the Sentinel AI agent | Python + React |
| **Sentinel-CameraNode** | The binary customers install on the machine their cameras are on | Rust |
| **Sentinel-License-Service** | Validates self-hosted licence keys | Python |
| **Sentinel-Sync-Service** | Optional cloud mirror for self-hosted installs | Python |
| **Sentinel Home Assistant** | Home Assistant integration | Python |

`SourceBox-Sentinel` was the AI agent's repository until 2026-09-09. It is **archived, not deleted** — the agent's pre-move commit history exists only there, because the move was squash-merged.

### Deployed services (Fly.io)

Four apps. Command Center runs **two process groups from one image** — Fly gives one image per app, and process groups differ only by command.

| App | Process | Memory | Runs | Purpose |
| --- | ------- | ------ | ---- | ------- |
| `sentinel-command` | `app` | 1 GB | always | API, SPA, MCP server, in-memory video cache, 8 background loops |
| `sentinel-command` | `agent` | 512 MB | always | Sentinel AI agent |
| `sentinel-license` | `app` | 256 MB | **scales to zero** | Licence validation |
| `sentinel-sync` | `app` | 256 MB | **scales to zero** | One-way mirror receiver |
| `sentinel-postgres` | `app` | 512 MB | always | One cluster, three databases |

**Why two of these scale to zero and two don't.** Fly's proxy waits ~8s for an auto-started machine to bind its port. License (~4s) and Sync (~3s) clear that comfortably, and a missed call is free for both — a failed licence check-in falls into a 72-hour grace window, a failed sync push just retries next cycle. The agent needs ~10s and cannot make it, so it stays warm; see `SENTINEL_AGENT.md`. Command Center's web tier serves live video and never sleeps.

## The signal path — camera to browser

This is the part most often assumed to work differently. There is **no object storage anywhere in the live path** — no S3, no Tigris, no presigned URLs. Segments live in Command Center's process memory and are evicted as they age out.

1. **CameraNode** cuts HLS segments with FFmpeg — 1-second `.ts` files by default.
2. **CameraNode pushes** them: `POST /api/cameras/{id}/push-segment` with the raw body and an `X-Node-API-Key` header. The node dials *out*, so no inbound port opens on the customer's network.
3. **Command Center** stores bytes in `_segment_cache[camera_id][filename]`, evicting oldest past `SEGMENT_CACHE_MAX_PER_CAMERA`.
4. **CameraNode pushes the playlist** separately: `POST /api/cameras/{id}/playlist`.
5. **Command Center rewrites** the playlist's segment filenames to relative `segment/<file>` proxy URLs, so the browser learns nothing about the node's own addressing.
6. **Browser** plays it as ordinary HLS. A camera whose heartbeat gap exceeds 90s flips to `offline` via the sweep loop.

The memory ceiling matters: `SEGMENT_CACHE_MAX_TOTAL_BYTES` is 384 MiB of the web machine's 1 GiB. Raise one without the other and the kernel OOM-killer takes every org's streams down at once, well before the cache's own eviction can help.

## The AI agent

Command Center owns the queue; the agent is a worker draining it. That single decision is what lets the same code run hosted or on someone else's hardware.

- **Push** (hosted default) — CC fires an HMAC-signed wakeup at `http://sentinel-command.flycast:8080/wakeup`, internal over 6PN.
- **Poll** — the agent asks CC for pending runs on an interval. No inbound connectivity, so it works behind NAT. Same shape CameraNode uses.

A run: claim via `POST /runs/{id}/start` → investigate through MCP tools → report via `POST /runs/{id}/complete` with `incident`, `no_action`, or `error`. Bounded at every layer, with a 270s wall clock under Fly's 300s `kill_timeout`, and a CC-side reaper for runs that strand.

The model is a config string (`LLM_MODEL`, via LiteLLM). **Changing it on the hosted deployment moves customer camera imagery to a different processor** — see `legal/SUB_PROCESSORS.md` before you do.

Full detail: [SENTINEL_AGENT.md](SENTINEL_AGENT.md).

## Who can talk to it

There is no single "API key". Every class of caller has its own credential, scope, and revocation path.

| Credential | Used by | Scope |
| ---------- | ------- | ----- |
| Clerk JWT | Browser users (hosted) | Per-user, per-org, role-aware |
| Local auth | Browser users (self-hosted) | Single admin, `AUTH_PROVIDER=local` |
| Node API key | CameraNodes | One node; hashed at rest, rotatable |
| MCP API key | Claude and other MCP clients | Org + `readonly` or full; daily cap |
| Integration key | Home Assistant | Org-wide camera read |
| `osa_` agent key | Sentinel AI agent | Per-org, issued in the dashboard |

The MCP surface is **23 tools, 16 read / 7 write**. A `readonly` key is intersected with the read set in middleware, so scope is enforced before a tool runs rather than inside each one.

## Data

**Hosted — Postgres.** Three databases on the single `sentinel-postgres` cluster: `sentinel_command`, `sentinel_license`, `sentinel_sync`. Isolation is by *role*, not by cluster — each service's role owns exactly one database, `CONNECT` is revoked from `PUBLIC`, and none is a superuser. One cluster rather than three was a deliberate cost decision; roles supply the isolation separate clusters would have charged for.

**Self-hosted — SQLite.** The same codebase. This is why the backend suite is parametrised across **both dialects** and a failure in either blocks the deploy.

Schema changes run through a `sync_schema()` sweep on every boot rather than Alembic — see [ADR 0001](adr/0001-sync-schema-vs-alembic.md).

Backups: nightly `pg_dump` for `sentinel_command` and `sentinel_license`, with restores rehearsed rather than assumed. `sentinel_sync` deliberately has none — it holds a mirror whose source of truth is the operator's local SQLite. See [DISASTER_RECOVERY.md](runbooks/DISASTER_RECOVERY.md).

## Two ways to run it

| | Hosted | Self-hosted |
| --- | ------ | ----------- |
| Auth | Clerk | Local admin |
| Database | Postgres | SQLite |
| Cameras, recording, motion, MCP | Plan-limited | **Free and unrestricted** |
| Sentinel AI | Plan-limited | Unlocked by licence key |
| Agent | `agent` process group | Bundled; `python -m app.sentinel_agent` |

Only Sentinel AI is gated for self-hosters, because it is the one feature with an ongoing per-run cost. Everything else ships unlocked.

Plans are enforced on five axes: camera cap, **viewer-hours** (the real tier axis — see [ADR 0002](adr/0002-viewer-hour-billing.md)), SSE caps, MCP daily cap, and log retention.

## How code ships

`.github/workflows/deploy.yml` builds one image and `flyctl deploy` reconciles both of Command Center's process groups. License and Sync each deploy from their own repo's `Test & Deploy` workflow.

**Path filtering is asymmetric, and that is load-bearing.** `push` is filtered (docs and Markdown only); `pull_request` is **never** filtered. `master` requires three status checks, and GitHub reports *no status at all* for a workflow a path filter skipped — so a filtered PR trigger would hang every PR that missed it, presenting as a stuck check rather than a config error.

**Known gap:** GitHub does not trigger `on: push` workflows for commits pushed with `GITHUB_TOKEN`, so an auto-merged Dependabot PR lands on `master` **without deploying**. CodeQL still goes green on those commits, which hides it convincingly. Closing it properly needs a PAT; meanwhile `deploy.yml` carries a `workflow_dispatch` trigger. This is why no repo here uses an auto-merge workflow except Command Center.

## Where to go next

- [SENTINEL_AGENT.md](SENTINEL_AGENT.md) — the AI agent in depth
- [runbooks/ON_CALL.md](runbooks/ON_CALL.md) — "the app is broken"
- [runbooks/DISASTER_RECOVERY.md](runbooks/DISASTER_RECOVERY.md) — "the data is gone"
- [LAUNCH_HANDOFF.md](LAUNCH_HANDOFF.md) — what's left before paying customers
- [adr/](adr/) — why non-obvious decisions were made
- `../AGENTS.md` — Command Center's internals, in detail
7 changes: 5 additions & 2 deletions docs/LAUNCH_HANDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,11 @@ the answer for those.
above realistic volume for the operator-critical kinds).
2. Verify a sending domain — Resend gives you 4 DNS records (SPF TXT,
DKIM CNAMEs ×3, optional DMARC). 15-60 min for DNS to propagate.
Recommended subdomain: `notifications.sourceboxsentry.com` (keeps
marketing-email reputation isolated from transactional).
**This step is already done:** `sentinel-command.com` is verified,
with DKIM and SPF/Return-Path on `send.sentinel-command.com`. This
previously recommended `notifications.sourceboxsentry.com`, which is
the pre-rename brand and is *not* a verified sending domain — setting
`EMAIL_FROM_ADDRESS` to it would have failed every send.
3. Configure a webhook in Resend → endpoint
`https://sentinel-command.com/api/webhooks/resend`. Copy the
signing secret (starts with `whsec_`).
Expand Down
12 changes: 10 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,19 @@
# Command Center docs

Supplementary documentation for Sentinel Command Center — the SaaS we operate at https://sourceboxsentry.com. The top-level `README.md` is the engineer-facing setup + reference for anyone reading or running the source locally (for audit or fixes); `AGENTS.md` is the developer / LLM-facing architecture reference. End users sign up at the live app; they don't deploy Command Center themselves. The docs in this tree cover the things that don't fit cleanly in either of those two files — operator launch checklist, ADRs, runbooks, and legal templates.
Supplementary documentation for Sentinel Command Center — the SaaS we operate at <https://sentinel-command.com>. The top-level `README.md` is the engineer-facing setup + reference for anyone reading or running the source locally (for audit or fixes); `AGENTS.md` is the developer / LLM-facing architecture reference. End users sign up at the live app; they don't deploy Command Center themselves. The docs in this tree cover the things that don't fit cleanly in either of those two files — operator launch checklist, ADRs, runbooks, and legal templates.

## [LAUNCH_HANDOFF.md](LAUNCH_HANDOFF.md) — what you need to do before paying customers

Twelve user-only items (Clerk prod keys, backup restore test, lawyer signoff, status page vendor, etc.) that every code-side launch blocker has been closed against. Start here if you're driving toward launch.

## [ARCHITECTURE.md](ARCHITECTURE.md) — how the whole system fits together

Start here if you're new. Every repository, every deployed service, and the paths between them — the signal path from camera to browser, the six credential types, where data lives, and how code ships. `README.md` is this repo's setup; `AGENTS.md` is Command Center's internals; ARCHITECTURE is the level above both.

## [SENTINEL_AGENT.md](SENTINEL_AGENT.md) — the AI agent

How the Sentinel AI agent works, how to run one yourself, and every environment variable it reads. It lives in this repo at `backend/app/sentinel_agent/` and deploys as the `agent` process group of the `sentinel-command` Fly app — not, as older references may suggest, a separate repository or app.

## Architecture Decision Records (`docs/adr/`)

One decision per file, numbered in order. ADRs capture the *why* behind a non-obvious choice so future maintainers don't re-litigate it. Format follows Michael Nygard's template (Context / Decision / Consequences).
Expand Down Expand Up @@ -45,6 +53,6 @@ Working drafts of customer-facing legal documents. Each is marked `DRAFT — NOT
## Writing new docs

- **ADR** — when you make a decision that was hard to make, or that someone else will almost certainly re-argue. Write it *while the tradeoffs are fresh*, not six months later.
- **Runbook** — when you catch yourself pasting the same sequence of commands into more than one support thread. Cheap to write, saves time forever. (None yet — add `docs/runbooks/` if/when one shows up.)
- **Runbook** — when you catch yourself pasting the same sequence of commands into more than one support thread. Cheap to write, saves time forever. Two exist (`ON_CALL.md`, `DISASTER_RECOVERY.md`); add to those before starting a third file.
- **Legal templates** — `docs/legal/` is for drafts that capture engineering truth; the lawyer-reviewed binding version lives elsewhere (a signed PDF in your records system). Update the draft *whenever* the underlying processing changes (new sub-processor, new data category, new retention window) so the lawyer review stays small.
- **README / AGENTS** — these two are the primary docs and get updated in-place with every feature. Don't fork them into `docs/`.
Loading