From c0eae03594ae2294f7460177552444a4ee987fb0 Mon Sep 17 00:00:00 2001 From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com> Date: Wed, 9 Sep 2026 18:19:10 -0700 Subject: [PATCH 1/2] Docs: correct what today's architecture changes made false MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A day of infrastructure work left the documentation describing a system that no longer exists. Accuracy first — a confidently wrong doc costs more than a missing one, because nobody goes looking for a second source. docs/SENTINEL_AGENT.md — rewritten, not patched. It was still titled after the archived SourceBox-Sentinel repo, described the agent as a separate Fly app with its own secrets and deploy token, told readers to `pip install -r requirements.txt`, documented an Ollama client that no longer exists, and warned against poll mode because it "defeats auto_stop_machines" on a machine that is now deliberately always warm. It also contained a half-finished edit of mine sitting inside a code block. Preserved what was still true and good — the reliability layers, the multi-tenancy model, the limitations — and corrected the rest. docs/legal/SUB_PROCESSORS.md — this one has consequences. The Ollama Cloud entry is still factually right about who processes data (verified against the running configuration: ollama_chat/qwen3.5:cloud), but it described the agent as "a separate SourceBox-operated service" and, more importantly, predates the provider becoming a config value. LiteLLM means one environment variable now decides who receives customer camera imagery. That is a sub-processor change under the DPA — 14 days' notice, not a config tweak — so the warning is recorded both at the point of use and in the changelog, where someone editing LLM_MODEL will meet it. docs/runbooks/DISASTER_RECOVERY.md — justified Sync having no dump job because it "runs two machines" and a volume would cost redundancy. It runs one machine now and scales to zero, so the stated reason is false. The conclusion survives on stronger grounds: sentinel_sync holds a mirror whose source of truth is the operator's local SQLite, so losing it entirely costs one sync cycle. Rewritten to say that instead. docs/LAUNCH_HANDOFF.md — told the operator to verify `notifications.sourceboxsentry.com` as the Resend sending domain. That is the pre-rename brand and is not verified; AGENTS.md already records that sentinel-command.com is, with DKIM and SPF on send.sentinel-command.com. Following the old step at launch would have failed every send. README.md — the agent row linked to the archived repo. docs/README.md — pointed at sourceboxsentry.com, never indexed SENTINEL_AGENT.md, and told readers no runbooks existed in the same file that lists two of them. All 26 relative links verified to resolve. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- docs/LAUNCH_HANDOFF.md | 7 +- docs/README.md | 8 +- docs/SENTINEL_AGENT.md | 266 +++++++++++++---------------- docs/legal/SUB_PROCESSORS.md | 40 ++++- docs/runbooks/DISASTER_RECOVERY.md | 22 ++- 6 files changed, 189 insertions(+), 156 deletions(-) diff --git a/README.md b/README.md index a660393..97534bc 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,7 @@ CameraNode captures and encodes video on your network, then pushes it **outbound | **Command Center** *(this repo)* | The hosted dashboard, API, and live-video hub | [Live app ›](https://sentinel-command.com) | | **CameraNode** | The camera daemon you install on your own hardware | [Repo ›](https://github.com/SourceBox-LLC/Sentinel-CameraNode) | | **Home Assistant integration** | Your Sentinel cameras inside Home Assistant | [Repo ›](https://github.com/SourceBox-LLC/Sentinel-HomeAssistant) | -| **Sentinel AI agent** | Serverless agent that investigates motion & incidents | [Repo ›](https://github.com/SourceBox-LLC/SourceBox-Sentinel) | +| **Sentinel AI agent** | AI agent that investigates motion & incidents | [Docs ›](docs/SENTINEL_AGENT.md) · in this repo at `backend/app/sentinel_agent/` | --- diff --git a/docs/LAUNCH_HANDOFF.md b/docs/LAUNCH_HANDOFF.md index 236c52c..a44cd78 100644 --- a/docs/LAUNCH_HANDOFF.md +++ b/docs/LAUNCH_HANDOFF.md @@ -90,8 +90,11 @@ the answer for those. above realistic volume for the operator-critical kinds). 2. Verify a sending domain — Resend gives you 4 DNS records (SPF TXT, DKIM CNAMEs ×3, optional DMARC). 15-60 min for DNS to propagate. - Recommended subdomain: `notifications.sourceboxsentry.com` (keeps - marketing-email reputation isolated from transactional). + **This step is already done:** `sentinel-command.com` is verified, + with DKIM and SPF/Return-Path on `send.sentinel-command.com`. This + previously recommended `notifications.sourceboxsentry.com`, which is + the pre-rename brand and is *not* a verified sending domain — setting + `EMAIL_FROM_ADDRESS` to it would have failed every send. 3. Configure a webhook in Resend → endpoint `https://sentinel-command.com/api/webhooks/resend`. Copy the signing secret (starts with `whsec_`). diff --git a/docs/README.md b/docs/README.md index 50e9e06..672e015 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,11 +1,15 @@ # Command Center docs -Supplementary documentation for Sentinel Command Center — the SaaS we operate at https://sourceboxsentry.com. The top-level `README.md` is the engineer-facing setup + reference for anyone reading or running the source locally (for audit or fixes); `AGENTS.md` is the developer / LLM-facing architecture reference. End users sign up at the live app; they don't deploy Command Center themselves. The docs in this tree cover the things that don't fit cleanly in either of those two files — operator launch checklist, ADRs, runbooks, and legal templates. +Supplementary documentation for Sentinel Command Center — the SaaS we operate at . The top-level `README.md` is the engineer-facing setup + reference for anyone reading or running the source locally (for audit or fixes); `AGENTS.md` is the developer / LLM-facing architecture reference. End users sign up at the live app; they don't deploy Command Center themselves. The docs in this tree cover the things that don't fit cleanly in either of those two files — operator launch checklist, ADRs, runbooks, and legal templates. ## [LAUNCH_HANDOFF.md](LAUNCH_HANDOFF.md) — what you need to do before paying customers Twelve user-only items (Clerk prod keys, backup restore test, lawyer signoff, status page vendor, etc.) that every code-side launch blocker has been closed against. Start here if you're driving toward launch. +## [SENTINEL_AGENT.md](SENTINEL_AGENT.md) — the AI agent + +How the Sentinel AI agent works, how to run one yourself, and every environment variable it reads. It lives in this repo at `backend/app/sentinel_agent/` and deploys as the `agent` process group of the `sentinel-command` Fly app — not, as older references may suggest, a separate repository or app. + ## Architecture Decision Records (`docs/adr/`) One decision per file, numbered in order. ADRs capture the *why* behind a non-obvious choice so future maintainers don't re-litigate it. Format follows Michael Nygard's template (Context / Decision / Consequences). @@ -45,6 +49,6 @@ Working drafts of customer-facing legal documents. Each is marked `DRAFT — NOT ## Writing new docs - **ADR** — when you make a decision that was hard to make, or that someone else will almost certainly re-argue. Write it *while the tradeoffs are fresh*, not six months later. -- **Runbook** — when you catch yourself pasting the same sequence of commands into more than one support thread. Cheap to write, saves time forever. (None yet — add `docs/runbooks/` if/when one shows up.) +- **Runbook** — when you catch yourself pasting the same sequence of commands into more than one support thread. Cheap to write, saves time forever. Two exist (`ON_CALL.md`, `DISASTER_RECOVERY.md`); add to those before starting a third file. - **Legal templates** — `docs/legal/` is for drafts that capture engineering truth; the lawyer-reviewed binding version lives elsewhere (a signed PDF in your records system). Update the draft *whenever* the underlying processing changes (new sub-processor, new data category, new retention window) so the lawyer review stays small. - **README / AGENTS** — these two are the primary docs and get updated in-place with every feature. Don't fork them into `docs/`. diff --git a/docs/SENTINEL_AGENT.md b/docs/SENTINEL_AGENT.md index 3cfba62..1f74e95 100644 --- a/docs/SENTINEL_AGENT.md +++ b/docs/SENTINEL_AGENT.md @@ -1,226 +1,206 @@ -# SourceBox Sentinel +# Sentinel AI agent -Serverless AI security agent for [Sentinel Command Center](https://github.com/SourceBox-LLC/Sentinel-Command). Sleeps when idle; wakes on Command Center webhook; processes pending agent runs across every org; goes back to sleep. Vision-capable, time-bounded at every layer, multi-tenant via signed override header. +The AI security agent that investigates motion events and files incidents. It lives in this repo at `backend/app/sentinel_agent/` and runs as the **`agent` process group** of the `sentinel-command` Fly app — its own machine, the same image, one deploy. + +It used to be a separate repository (`SourceBox-Sentinel`, archived 2026-09-09) deploying to a separate Fly app (`sourcebox-sentinel`, destroyed the same day). If you find a reference to either, it is stale. ``` -Command Center SourceBox Sentinel (Fly.io, auto-stop) -───────────────── ────────────────────────────────────── -notification fires ┌─ POST /wakeup (HMAC-signed) - → pending sentinel_run │ ↓ verify HMAC - → fire-and-forget webhook ────┘ ↓ fetch pending runs from CC - for each run: - POST /api/sentinel/runs/{id}/start - run agent loop (LLM ↔ MCP tools) - POST /api/sentinel/runs/{id}/complete - return 200 — machine stays warm (see fly.toml) +Command Center (process "app") Sentinel AI agent (process "agent") +────────────────────────────── ──────────────────────────────────── +notification fires ┌─ POST /wakeup (HMAC-signed) + → pending sentinel_run │ ↓ verify HMAC + → fire-and-forget webhook ──────────┘ ↓ fetch pending runs from CC + over .flycast (6PN, internal) for each run: + POST /api/sentinel/runs/{id}/start + run agent loop (LLM ↔ MCP tools) + POST /api/sentinel/runs/{id}/complete + return 200 ``` ## Architecture -- **LLM**: Ollama Cloud (default `qwen3.5:cloud` — vision + tools, 256K context) via the official `ollama` Python client. Vision-capable model required: the agent calls `view_camera` which returns a JPEG. -- **Tools**: Command Center's MCP server (23 tools — list/view/watch cameras, create/finalize incidents, attach evidence, etc.) over streamable HTTP transport. -- **Server**: Bare Starlette + uvicorn. No auth on its own state — every accepted request is HMAC-verified against the shared `SENTINEL_AGENT_KEY`. -- **Master of pending work**: Command Center's `sentinel_runs` table. The agent does not persist state; every wakeup re-fetches what's pending. +- **LLM**: any provider, via [LiteLLM](https://docs.litellm.ai/). `LLM_MODEL` is the only setting that changes — `ollama_chat/qwen3.5:cloud` today, `anthropic/claude-sonnet-5` for a bring-your-own-key deployment, or an OpenAI-compatible endpoint for a custom model. **Must support tool calling *and* image input**: the agent fans out tool calls and feeds camera frames back in, so a text-only model does not degrade, it fails every run. +- **Tools**: Command Center's MCP server (23 tools — list/view/watch cameras, create/finalize incidents, attach evidence) over streamable HTTP. +- **Server**: Starlette + uvicorn. No auth on its own state — every accepted request is HMAC-verified against the shared `SENTINEL_AGENT_KEY`. +- **Master of pending work**: Command Center's `sentinel_runs` table. The agent persists nothing; every wakeup re-fetches what's pending. -## Push or poll +### Why a separate process group, not a thread -The agent finds work two ways, selected by `AGENT_MODE`. +A run holds base64 camera frames for up to 270s, and the web machine's segment cache is already budgeted 384 MiB of its 1 GiB (`SEGMENT_CACHE_MAX_TOTAL_BYTES` in `fly.toml`). Sharing one machine is how the OOM killer takes every org's live streams down at once. Being a separate *app* was never what bought that isolation — a separate process group is. -**`push` (default)** — Command Center calls `POST /wakeup` and the agent -drains. Requires CC to reach the agent *inbound*, so the agent has to be -publicly addressable. This is how the SourceBox-hosted agent runs, and -it is what makes Fly's auto-stop pay off: idle costs nothing. +### Why the machine stays warm -**`poll`** — the agent asks CC for pending runs every -`POLL_INTERVAL_SECONDS`. **No inbound connectivity required**, so it runs -fine behind NAT on a home or office network. This is the mode for an -agent you host yourself, and it is the same shape CameraNode already -uses to talk to Command Center: outbound only. +`min_machines_running = 1`, deliberately, even though this worker's shape screams scale-to-zero. -Both modes share one drain implementation and one concurrency guard, so -they cannot diverge; `/wakeup` stays mounted in poll mode as a way to -force an immediate drain. +Fly's proxy waits ~8s for an auto-started machine to bind its port, and this process needs ~10s (Python + the MCP SDK + Sentry + a deferred LiteLLM import). An auto-started machine was declared unreachable and Command Center's wakeup came back `RemoteDisconnected` — on **every** wakeup, since it had scaled to zero. It was ~7s before LiteLLM, i.e. always marginal. -> **Don't set `poll` on a Fly deployment.** Polling keeps the machine -> awake, which defeats `auto_stop_machines` and bills you for idle time. -> Poll is for deployments you run yourself. +~$2/month removes cold starts from the wakeup path instead of racing them. See the comment on `[[services]]` in `fly.toml`. -This is what makes mixed setups work — a self-hosted agent against the -cloud Command Center, or an entirely local stack. +## Push or poll -## Running this agent yourself +Selected by `AGENT_MODE`. -Generate a key in Command Center under **MCP → Sentinel Agent Keys**. It -starts with `osa_` and is scoped to your organization alone: an agent -using it can never see another customer's cameras. +**`push` (default)** — Command Center calls `POST /wakeup` and the agent drains. Inside the Fly app this is an internal call over `.flycast`, so nothing is publicly addressable. -Then set two things on the agent: +**`poll`** — the agent asks CC for pending runs every `POLL_INTERVAL_SECONDS`. **No inbound connectivity required**, so it runs fine behind NAT on a home or office network. This is the mode for an agent you host yourself, and it's the same outbound-only shape CameraNode already uses. -``` +Both modes share one drain implementation and one concurrency guard, so they cannot diverge; `/wakeup` stays mounted in poll mode as a way to force an immediate drain. + +## Running the agent yourself + +You do **not** need a separate install — a self-hosted Command Center already contains the agent. Running it separately is only for putting it on different hardware. + +Generate a key in Command Center under **MCP → Sentinel Agent Keys**. It starts with `osa_` and is scoped to your organization alone. + +```bash AGENT_MODE=poll SENTINEL_AGENT_KEY=osa_... +LLM_API_KEY=... # or OLLAMA_API_KEY for the default provider +python -m app.sentinel_agent ``` -That's it — `OPENSENTRY_MCP_AGENT_KEY` defaults to `SENTINEL_AGENT_KEY` -when unset, because the issued key authenticates both the run queue and -the camera tools. +`OPENSENTRY_MCP_AGENT_KEY` defaults to `SENTINEL_AGENT_KEY` when unset, because the issued key authenticates both the run queue and the camera tools. -> Do **not** use the shared `SENTINEL_AGENT_KEY` from the first-party -> deployment. It is a multi-tenant secret that can act as *any* org via -> `X-Agent-Org-Override`, and must never be handed to a customer. The -> `osa_` keys exist precisely so it doesn't have to be. +> Do **not** use the shared `SENTINEL_AGENT_KEY` from the first-party deployment. It is a multi-tenant secret that can act as *any* org via `X-Agent-Org-Override`, and must never be handed to a customer. The `osa_` keys exist precisely so it doesn't have to be. ## Plan tiers -Sentinel is gated end-to-end (UI, dispatcher, agent MCP auth) on the org's plan. Caps reset on the 1st of each calendar month in UTC. +Gated end-to-end (UI, dispatcher, agent MCP auth) on the org's plan. Caps reset on the 1st of each calendar month, UTC. -| Plan | Monthly runs | Note | -|-----------|--------------|-------------------------------------------| -| Free | 0 | Sentinel locked; UI shows upgrade banner. | -| Pro | 100 | ~3 / day — casual home use. | -| Pro Plus | 500 | ~16 / day — commercial-shaped use. | +| Plan | Monthly runs | Note | +| -------- | ------------ | ----------------------------------------- | +| Free | 0 | Sentinel locked; UI shows upgrade banner. | +| Pro | 100 | ~3 / day — casual home use. | +| Pro Plus | 500 | ~16 / day — commercial-shaped use. | +| Self-hosted | unlimited | Unlocked by a license key — see the License Service. | -When an org hits the cap, dispatch pauses for the rest of the month. There's no overage billing; the rest of SourceBox Sentry (recordings, motion notifications, dashboard, MCP) keeps working as normal. +At the cap, dispatch pauses for the rest of the month. No overage billing; recordings, motion notifications, dashboard and MCP keep working. ## Reliability layers -A single run is bounded at every layer to prevent runaway loops or hung calls. All bounds are env-tunable via the matching `*_TIMEOUT_SECONDS` / `MAX_*` settings. +A run is bounded at every layer. All bounds are env-tunable. -- **Per-LLM-call timeout** — 120s default. `asyncio.wait_for` around the Ollama chat call. Hung calls return clean errors instead of holding the machine alive. -- **Per-MCP-tool timeout** — 60s default. Stuck tools surface to the LLM as an error result so the model can retry or pivot. -- **Iteration cap** — 10 tool-call rounds per run (`MAX_AGENT_ITERATIONS`). Hits the cap → outcome is `error` with "investigation incomplete". -- **Wall-clock cap** — 270s per wakeup (kept under Fly's 300s `kill_timeout` so the cleanup runs before SIGKILL). The `process_with_timeout` wrapper catches `asyncio.TimeoutError`, identifies the in-flight run via a mutable `in_flight` handle, and best-effort POSTs `/api/sentinel/runs/{id}/complete` with `outcome=error` so the run lands terminal on Command Center instead of stranding in `running` state. -- **CC-side stranded-run reaper** — runs stuck in `running` for more than 20 minutes are marked `error` automatically by Command Center. Catches the rare case where the agent process crashes before its own cleanup wrapper fires. +- **Per-LLM-call timeout** — 120s. `asyncio.wait_for` around the LiteLLM call. Wrapped despite LiteLLM taking its own `timeout` because that one bounds the HTTP request, not the whole call: a provider that accepts the connection then stalls mid-stream would otherwise hold the machine until the wall clock fires. +- **Per-MCP-tool timeout** — 60s. Stuck tools surface to the LLM as an error result so the model can retry or pivot. +- **Iteration cap** — 10 tool-call rounds per run (`MAX_AGENT_ITERATIONS`). Hitting it → outcome `error`, "investigation incomplete". +- **Wall-clock cap** — 270s per wakeup, under Fly's 300s `kill_timeout` so cleanup runs before SIGKILL. `process_with_timeout` catches the timeout, identifies the in-flight run, and best-effort POSTs `complete` with `outcome=error` so the run lands terminal instead of stranding in `running`. +- **CC-side stranded-run reaper** — runs stuck in `running` for >20 minutes are marked `error` by Command Center. Catches the case where the agent crashes before its own cleanup fires. ## Endpoints -| Method | Path | Description | -|--------|------------|----------------------------------------------------------------------------------------| -| `GET` | `/health` | Liveness probe (keys-configured-or-not flags). No auth. | -| `POST` | `/wakeup` | Webhook receiver. Drains pending runs. HMAC-verified. | -| `POST` | `/` | DEV-ONLY drain trigger. Disabled in production via `WEBHOOK_VERIFY_SIGNATURE=true`. | +| Method | Path | Description | +| ------ | --------- | ------------------------------------------------------------------ | +| `GET` | `/health` | Liveness probe. No auth. | +| `POST` | `/wakeup` | Webhook receiver. Drains pending runs. HMAC-verified. | +| `POST` | `/` | DEV-ONLY drain trigger. Disabled in prod via `WEBHOOK_VERIFY_SIGNATURE=true`. | -## Setup +## Running it locally -### Local +The agent resolves against `backend/`'s dependency set — there is no separate project to install. ```bash -cp .env.example .env -# Fill in OLLAMA_API_KEY, SENTINEL_AGENT_KEY, OPENSENTRY_MCP_AGENT_KEY -pip install -r requirements.txt -WEBHOOK_VERIFY_SIGNATURE=false uvicorn app.main:app --reload +cd backend +uv sync --extra dev + +OLLAMA_API_KEY=... \ +SENTINEL_AGENT_KEY=dev-secret \ +AGENT_MODE=poll \ +OPENSENTRY_API_BASE=http://localhost:8000 \ +WEBHOOK_VERIFY_SIGNATURE=false \ + uv run python -m app.sentinel_agent ``` -In another terminal: +With verification off you can force a drain: ```bash -# Trigger a drain (skips HMAC because verify is off): curl -X POST http://localhost:8080/ ``` -### Fly.io (production) - -Two shared secrets, both have to match Command Center: - -```bash -# Run-lifecycle callback secret (X-Sentinel-Agent-Key + HMAC): -fly secrets set SENTINEL_AGENT_KEY=$(python -c "import secrets; print(secrets.token_hex(32))") - -# Multi-tenant MCP bearer (per-call org via X-Agent-Org-Override): -fly secrets set OPENSENTRY_MCP_AGENT_KEY=$(python -c "import secrets; print(secrets.token_hex(32))") - -fly secrets set OLLAMA_API_KEY=... -fly deploy -``` - -Confirm: - -```bash -fly ssh console -a sentinel-command -s --process-group agent -C "curl -s localhost:8080/health" -``` +## Deployment -### CI/CD (GitHub Actions) +There is nothing agent-specific to deploy. `.github/workflows/deploy.yml` builds one image and `flyctl deploy` reconciles both process groups; `[processes]` in `fly.toml` gives each its command. -Pushes to `master` auto-deploy via `.github/workflows/deploy.yml`. To -enable on a fresh checkout: +Secrets live on the `sentinel-command` app and are shared by both groups: ```bash -# Generate a deploy-scoped Fly token for this app: -# Deploys come from Command Center's own workflow now — no separate token. - -# Add it to the GitHub repo as a secret named FLY_API_TOKEN -# (Settings → Secrets and variables → Actions → New repository secret) +fly secrets set OLLAMA_API_KEY=... -a sentinel-command +fly secrets set SENTINEL_AGENT_KEY=$(python -c "import secrets; print(secrets.token_hex(32))") -a sentinel-command +fly secrets set SENTINEL_AGENT_MCP_KEY=$(python -c "import secrets; print(secrets.token_hex(32))") -a sentinel-command +fly secrets set SENTINEL_AGENT_WEBHOOK_URL=http://sentinel-command.flycast:8080/wakeup -a sentinel-command ``` -After that, every push to `master` runs a syntax check then -`flyctl deploy --remote-only`. Manual re-runs available via the -Actions tab → "Deploy to Fly.io" → "Run workflow". +`SENTINEL_AGENT_WEBHOOK_URL` points at the app's own `agent` process group over 6PN. That requires a private IPv6 (`fly ips allocate-v6 --private`) — without it `.flycast` does not resolve and every wakeup fails to connect. -## Wiring with Command Center +The agent reads Command Center's `SENTINEL_AGENT_MCP_KEY` directly (via an `AliasChoices` on `opensentry_mcp_agent_key`), so the first-party deployment needs no duplicate copy of it under the agent's own name. -On the Command Center side, set the matching pair of secrets: +Confirm: ```bash -fly secrets set SENTINEL_AGENT_KEY= -fly secrets set SENTINEL_AGENT_MCP_KEY= -fly secrets set SENTINEL_AGENT_WEBHOOK_URL=http://sentinel-command.flycast:8080/wakeup -a sentinel-command -fly deploy +fly ssh console -a sentinel-command -s --process-group agent -C "curl -s localhost:8080/health" ``` -When a notification fires (motion / incident_created) AND the org has Sentinel configured AND the dispatch gate clears (camera in scope, schedule allows, under cap), Command Center inserts a pending `sentinel_runs` row and POSTs the wakeup webhook fire-and-forget. Sentinel wakes, drains all pending runs across all orgs, sleeps. - ## Configuration -| Variable | Default | Purpose | -|--------------------------------|---------------------------------------------|--------------------------------------------------------| -| `AGENT_MODE` | `push` | `push` = wait for CC's `/wakeup` webhook. `poll` = ask CC for work on an interval (works behind NAT). | -| `POLL_INTERVAL_SECONDS` | `30` | Poll mode only. Seconds between checks for pending runs. | -| `OLLAMA_API_KEY` | required | Ollama Cloud API key. | -| `OLLAMA_HOST` | `https://ollama.com` | Ollama API endpoint. | -| `OLLAMA_MODEL` | `qwen3.5:cloud` | Model id. Must be vision-capable (view_camera returns JPEGs). | -| `MAX_AGENT_ITERATIONS` | `10` | Max tool-call loops per run. | -| `MAX_TOKENS` | `2048` | Max output tokens per LLM response. | -| `LLM_CALL_TIMEOUT_SECONDS` | `120` | Per-LLM-call timeout — wraps the Ollama chat call in `asyncio.wait_for` so hung connections return clean errors instead of holding the machine alive. | -| `MCP_TOOL_TIMEOUT_SECONDS` | `60` | Per-MCP-tool-call timeout. Stuck tools surface to the LLM as error results so the model can retry or pivot. | -| `OPENSENTRY_API_BASE` | `https://sentinel-command.fly.dev` | Command Center base URL. `OPENSENTRY_MCP_URL` is derived from it. | -| `SENTINEL_AGENT_KEY` | required | Run-lifecycle callback secret. Match CC. | -| `OPENSENTRY_MCP_AGENT_KEY` | falls back to `SENTINEL_AGENT_KEY` | Multi-tenant MCP bearer. Set explicitly ONLY for the first-party deployment, where it differs. Self-hosted: leave unset. | -| `OPENSENTRY_MCP_URL` | derived from `OPENSENTRY_API_BASE` + `/mcp` | Override only if MCP lives elsewhere. | -| `WEBHOOK_VERIFY_SIGNATURE` | `true` | Hard-disable HMAC for local dev. Always on in prod. | -| `SENTRY_DSN` | unset | Error tracking. No-op when unset. With `min_machines_running=0` the agent sleeps between wakeups, so a run that errors every time is otherwise invisible. | -| `SENTRY_ENVIRONMENT` | `production` | Tags events so a local run doesn't pollute prod. | -| `SENTRY_TRACES_SAMPLE_RATE` | `0.1` | Performance sampling. | +| Variable | Default | Purpose | +| -------- | ------- | ------- | +| `AGENT_MODE` | `push` | `push` = wait for `/wakeup`. `poll` = ask CC on an interval (works behind NAT). | +| `POLL_INTERVAL_SECONDS` | `30` | Poll mode only. | +| `LLM_MODEL` | falls back to `ollama_chat/{OLLAMA_MODEL}` | LiteLLM model string. The only setting needed to change provider. Must support tool calling + image input. | +| `LLM_API_KEY` | falls back to `OLLAMA_API_KEY` | Provider credential. | +| `LLM_API_BASE` | falls back to `OLLAMA_HOST` for the Ollama path | Override only for self-hosted or proxied endpoints. | +| `OLLAMA_API_KEY` | — | Default provider's key. Required unless `LLM_API_KEY` is set; boot fails with neither. | +| `OLLAMA_HOST` | `https://ollama.com` | Default provider endpoint. | +| `OLLAMA_MODEL` | `qwen3.5:cloud` | Model id used to build `LLM_MODEL` when it is unset. | +| `MAX_AGENT_ITERATIONS` | `10` | Max tool-call loops per run. | +| `MAX_TOKENS` | `2048` | Max output tokens per LLM response. | +| `LLM_CALL_TIMEOUT_SECONDS` | `120` | Per-LLM-call timeout. | +| `MCP_TOOL_TIMEOUT_SECONDS` | `60` | Per-MCP-tool-call timeout. | +| `OPENSENTRY_API_BASE` | `https://sentinel-command.fly.dev` | Command Center base URL. `OPENSENTRY_MCP_URL` is derived from it. | +| `SENTINEL_AGENT_KEY` | required | Run-lifecycle callback secret. Must match CC. | +| `OPENSENTRY_MCP_AGENT_KEY` | `SENTINEL_AGENT_MCP_KEY`, then `SENTINEL_AGENT_KEY` | Multi-tenant MCP bearer. Self-hosted: leave unset. | +| `OPENSENTRY_MCP_URL` | derived from `OPENSENTRY_API_BASE` + `/mcp` | Override only if MCP lives elsewhere. | +| `WEBHOOK_VERIFY_SIGNATURE` | `true` | Hard-disable HMAC for local dev. Always on in prod. | +| `SENTRY_DSN` | unset | Error tracking. No-op when unset. | +| `SENTRY_ENVIRONMENT` | `production` | Keeps local runs out of prod. | +| `SENTRY_TRACES_SAMPLE_RATE` | `0.1` | Performance sampling. | ## Project structure ``` -app/ +backend/app/sentinel_agent/ + __main__.py Entry point: python -m app.sentinel_agent main.py Starlette app, /wakeup HMAC handler, /health, dev / processor.py Drain loop: list pending → start → run → complete agent.py LLM ↔ MCP tool conversation for one run prompts.py Trigger-specific system prompts (motion / manual / …) - llm.py Ollama AsyncClient wrapper + llm.py LiteLLM wrapper + ALL provider-shaped message translation mcp_client.py MCP client (streamable HTTP), tool dispatch sentinel_client.py HTTP client back to Command Center config.py Pydantic Settings from env ``` +`llm.py` owns every provider-shaped detail on purpose. Ollama and OpenAI-shaped APIs disagree structurally, not cosmetically — tool results key off `tool_call_id` rather than `tool_name`, arguments arrive as a JSON *string* rather than a dict, and images cannot ride on a `tool` message at all (they need a `user` message with a `data:` URI). The agent loop builds messages only through `llm.py`'s helpers, so changing providers again touches one file. `backend/tests/test_agent_llm_provider.py` pins that translation, because every failure mode in it is silent: a wrongly-shaped message doesn't raise, it just means the model never saw the camera frame. + ## Multi-tenancy model -ONE deployed agent serves every org. Per-call org scoping happens at the MCP layer via the `X-Agent-Org-Override` header — the processor sets it to each run's `org_id` before connecting MCP, and the server uses that as the authoritative scope (rejecting if the org isn't on Pro / Pro Plus or has Sentinel disabled). +ONE deployed agent serves every org. Per-call scoping happens at the MCP layer via `X-Agent-Org-Override` — the processor sets it to each run's `org_id` before connecting, and the server treats that as authoritative (rejecting if the org isn't entitled or has Sentinel disabled). Per-run isolation comes from how the loop is built, not from per-org credentials: -- Fresh `MCPClientManager` per run (built and torn down inside `_process_one_run`) -- Fresh `messages` array per run (Agent.run is stateless) -- Per-run override header from `run.org_id` (set when CC dispatched the run) +- Fresh `MCPClientManager` per run +- Fresh `messages` array per run (`Agent.run` is stateless) +- Per-run override header from `run.org_id` - Activity log writes one row per tool call, attributed to the override org -Trust model depends on which key the agent holds. The **first-party** deployment uses two shared secrets (`SENTINEL_AGENT_KEY` + `OPENSENTRY_MCP_AGENT_KEY`) and has cross-org access — compromise = act-as-any-org. A **self-hosted** agent holds an `osa_` key scoped to one org by its Command Center database row, so a compromise is bounded to that org and is revocable from the UI. Mitigations: Fly secrets only, audit log per call, per-org rate limiting on the MCP server side, rotate by changing the secrets on both sides. +Trust depends on which key the agent holds. The **first-party** deployment uses two shared secrets and has cross-org access — compromise means act-as-any-org. A **self-hosted** agent holds an `osa_` key scoped to one org by its database row, so compromise is bounded and revocable from the UI. Mitigations: Fly secrets only, audit log per call, per-org MCP rate limiting, rotate by changing both sides. + +## Contract with Command Center + +`sentinel_client.py` hand-builds the JSON that CC's `RunCompleteBody` validates. `backend/tests/test_agent_contract.py` pins the two together — worth having because the failure is silent: pydantic ignores unknown keys, so renaming a field on the CC side would make every run record zero tool calls with no 422, no exception, and no log line. ## Limitations -- **No retry of failed webhook delivery**: if Command Center's wakeup POST fails (Sentinel cold-starting too slowly, network blip), the run sits pending. The next trigger that fires another wakeup also drains the prior pending. No dead-letter queue yet — but stranded runs (those that *did* claim but never completed) are reaped automatically; only the dispatch-side webhook miss is still uncaught. -- **No cron sweeps**: `trigger_type=scheduled` is mapped in the prompt set but Command Center doesn't yet emit them. -- **Sequential per wakeup**: multiple pending runs are processed one after another inside a single wakeup. Async fan-out is a future optimization. -- **Concurrent wakeups can claim the same run**: two `/wakeup` deliveries to the same machine can both call `/start` on the same row. Command Center's `start` endpoint is idempotent so it's not a data integrity issue, just wasted tool calls + a small risk of duplicate incidents. An `asyncio.Lock` around `process_with_timeout` is the planned fix when concurrent wakeups become real. +- **No retry of failed webhook delivery.** If the wakeup POST fails, the run sits pending until the next trigger fires another wakeup, which also drains the prior one. Much less likely now the agent machine stays warm, but there is still no dead-letter queue. Stranded runs (claimed but never completed) *are* reaped. +- **No cron sweeps.** `trigger_type=scheduled` is mapped in the prompt set but CC doesn't emit them yet. +- **Sequential per wakeup.** Multiple pending runs process one after another inside a single wakeup. +- **Concurrent wakeups can claim the same run.** Two deliveries can both call `/start` on the same row. CC's `start` is idempotent so it isn't a data-integrity issue — just wasted tool calls and a small duplicate-incident risk. An `asyncio.Lock` around `process_with_timeout` is the fix when concurrent wakeups become real. diff --git a/docs/legal/SUB_PROCESSORS.md b/docs/legal/SUB_PROCESSORS.md index 9cdeb94..7321081 100644 --- a/docs/legal/SUB_PROCESSORS.md +++ b/docs/legal/SUB_PROCESSORS.md @@ -147,9 +147,26 @@ terminate the affected Service per Section 4.4 of the - **Notes:** Engaged **only** when an organization runs the Sentinel agent (a paid-tier feature with a per-organization on/off toggle in Settings). Organizations that never enable or trigger Sentinel send - no imagery to Ollama. The agent runs as a separate SourceBox-operated - service; an AGPL-3.0 fork that does not deploy the agent does not - engage Ollama as a sub-processor. + no imagery to Ollama. The agent is a process group of the + `sentinel-command` app rather than a separate deployment, but it is + still the only component that sends imagery anywhere; an AGPL-3.0 fork + that never runs it does not engage Ollama as a sub-processor. +- **⚠ This entry names the provider CURRENTLY configured, and that is + now a setting rather than a hardcoded client.** Since 2026-09-09 the + agent reaches its model through LiteLLM, so a single environment + variable (`LLM_MODEL`) selects the inference provider. The first-party + deployment is verified to resolve to `ollama_chat/qwen3.5:cloud` as of + this revision. + + **Changing `LLM_MODEL` on the first-party deployment changes who + processes customer camera imagery, and therefore changes this list.** + It is a sub-processor change under the DPA, requiring an update here + and advance notice to customers — not a config tweak. Anyone editing + that variable in production should read this paragraph first. + + Self-hosted operators who set their own `LLM_MODEL` are engaging that + provider as *their* sub-processor, not ours; see "If you fork and run + your own copy" below. --- @@ -205,6 +222,23 @@ diff this file in the repository for the full record. mirror by design. Also corrected a long-standing factual error in the DPA, which described Command Center's own storage as Postgres when it has always been SQLite on a Fly volume. + - **2026-09-09** — The Sentinel agent moved from a separate + SourceBox-operated service into a process group of the + `sentinel-command` app, and its LLM client was replaced with + LiteLLM. **No new sub-processor and no change to what is sent, to + whom, or when** — the first-party deployment still resolves to + Ollama Cloud (`ollama_chat/qwen3.5:cloud`, verified against the + running configuration), and the agent remains the only component + that transmits camera imagery. + + Recorded because it changes how a *future* change would happen: the + inference provider is now selected by an environment variable + (`LLM_MODEL`) rather than by a hardcoded client, so switching it in + production would silently move customer camera imagery to a + different processor. That is a sub-processor change under the DPA + and requires updating this list and notifying customers in advance. + The Ollama Cloud entry above carries the same warning at the point + of use. --- diff --git a/docs/runbooks/DISASTER_RECOVERY.md b/docs/runbooks/DISASTER_RECOVERY.md index af6c3cc..0ddc22d 100644 --- a/docs/runbooks/DISASTER_RECOVERY.md +++ b/docs/runbooks/DISASTER_RECOVERY.md @@ -293,11 +293,23 @@ runbook used to say nothing about them. | `sentinel_sync` | ❌ none — snapshots only | — | snapshot restore (below) | **Why Sync has no dump job, deliberately.** The other two write their -dump to a Fly volume. `sentinel-sync` has no volume, and it runs **two -machines** — Fly volumes are single-attach, so giving it one would pin -it to a single machine. That trades away real redundancy to gain a -second copy of data the cluster snapshot already holds. Not worth it; -don't "fix" this in a later pass without re-reading this paragraph. +dump to a Fly volume. `sentinel-sync` has none, and giving it one would +pin it to a single machine, because Fly volumes are single-attach. + +The original reasoning here was that Sync ran **two machines**, so a +volume would cost real redundancy. That is no longer the fact pattern — +it was scaled to one machine on 2026-09-09 and now scales to zero +between its 30-minute pushes, so there is no redundancy left to trade +away. The conclusion survives the reason changing, on stronger grounds: + +`sentinel_sync` holds a **mirror**, not a source of truth. Every row in +it was pushed from a self-hosted operator's local SQLite, which remains +authoritative, and `push_pending_changes` only advances its cursors on +confirmed success. Losing this database entirely costs one sync cycle; +the operators re-push. A dump job would be a second copy of data the +cluster snapshot already holds, of data that is itself already a copy. + +Don't "fix" this in a later pass without re-reading this paragraph. **Restoring `sentinel_sync` therefore means a snapshot restore**, which is cluster-level and brings back all three databases at once: From 6874e0c34e960e456a39b7a821c48337ac2d4012 Mon Sep 17 00:00:00 2001 From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com> Date: Wed, 9 Sep 2026 18:20:43 -0700 Subject: [PATCH 2/2] Docs: add the system-level architecture reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There was no document describing how the whole thing fits together. README covers this repo, AGENTS.md covers Command Center's internals, and nothing covered the level above both — which meant the answer to "what talks to what" lived only in the heads of people who built it. docs/ARCHITECTURE.md is that level: every repository, every deployed service, and the paths between them. Written from the running infrastructure rather than from memory, and it makes explicit several things that were previously folklore: - the live video path touches NO object storage, and the segment cache's 384 MiB ceiling is coupled to the machine's 1 GiB — raise one without the other and the OOM killer takes every org's streams - two services scale to zero and two don't, and the reason is a measurable one (Fly's proxy gives an auto-started machine ~8s to bind; License needs ~4s, the agent ~10s) - isolation between the three databases is by ROLE, not by cluster - CI path filtering is asymmetric on purpose, and inverting it would hang every PR on a check that never reports - the GITHUB_TOKEN deploy gap, which is why only one repo here has an auto-merge workflow Indexed from both README.md and docs/README.md as the recommended starting point. Co-Authored-By: Claude Opus 5 --- README.md | 1 + docs/ARCHITECTURE.md | 117 +++++++++++++++++++++++++++++++++++++++++++ docs/README.md | 4 ++ 3 files changed, 122 insertions(+) create mode 100644 docs/ARCHITECTURE.md diff --git a/README.md b/README.md index 97534bc..b300afd 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,7 @@ CameraNode captures and encodes video on your network, then pushes it **outbound | If you want to… | Go to | |-----------------|-------| | **Use Sentinel** — set up cameras, recording, notifications, integrations | The in-app [Documentation](https://sentinel-command.com/docs) | +| **See how the whole system fits together** — every repo, every deployed service, the paths between them | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | | **Understand the code** — architecture, API, data models, configuration | [AGENTS.md](AGENTS.md) | | **Operate it** — decision records, runbooks, legal templates | [docs/](docs/) | | **Audit or run the source locally** for review | [AGENTS.md › Build & Run](AGENTS.md) | diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..6cfc1cb --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,117 @@ +# Sentinel system architecture + +How the whole system fits together — every repository, every deployed service, and the paths between them. Start here if you're new, or if you need to know where something lives before changing it. + +`README.md` is setup and reference for this repo. `AGENTS.md` is the deep architecture reference for Command Center's internals. **This file is the level above both**: the system, not the codebase. + +*Verified against the running infrastructure on 2026-09-09.* + +## The pieces + +### Repositories + +| Repo | What it is | Language | +| ---- | ---------- | -------- | +| **Sentinel-Command** (this one) | Command Center: API, dashboard, MCP server, and the Sentinel AI agent | Python + React | +| **Sentinel-CameraNode** | The binary customers install on the machine their cameras are on | Rust | +| **Sentinel-License-Service** | Validates self-hosted licence keys | Python | +| **Sentinel-Sync-Service** | Optional cloud mirror for self-hosted installs | Python | +| **Sentinel Home Assistant** | Home Assistant integration | Python | + +`SourceBox-Sentinel` was the AI agent's repository until 2026-09-09. It is **archived, not deleted** — the agent's pre-move commit history exists only there, because the move was squash-merged. + +### Deployed services (Fly.io) + +Four apps. Command Center runs **two process groups from one image** — Fly gives one image per app, and process groups differ only by command. + +| App | Process | Memory | Runs | Purpose | +| --- | ------- | ------ | ---- | ------- | +| `sentinel-command` | `app` | 1 GB | always | API, SPA, MCP server, in-memory video cache, 8 background loops | +| `sentinel-command` | `agent` | 512 MB | always | Sentinel AI agent | +| `sentinel-license` | `app` | 256 MB | **scales to zero** | Licence validation | +| `sentinel-sync` | `app` | 256 MB | **scales to zero** | One-way mirror receiver | +| `sentinel-postgres` | `app` | 512 MB | always | One cluster, three databases | + +**Why two of these scale to zero and two don't.** Fly's proxy waits ~8s for an auto-started machine to bind its port. License (~4s) and Sync (~3s) clear that comfortably, and a missed call is free for both — a failed licence check-in falls into a 72-hour grace window, a failed sync push just retries next cycle. The agent needs ~10s and cannot make it, so it stays warm; see `SENTINEL_AGENT.md`. Command Center's web tier serves live video and never sleeps. + +## The signal path — camera to browser + +This is the part most often assumed to work differently. There is **no object storage anywhere in the live path** — no S3, no Tigris, no presigned URLs. Segments live in Command Center's process memory and are evicted as they age out. + +1. **CameraNode** cuts HLS segments with FFmpeg — 1-second `.ts` files by default. +2. **CameraNode pushes** them: `POST /api/cameras/{id}/push-segment` with the raw body and an `X-Node-API-Key` header. The node dials *out*, so no inbound port opens on the customer's network. +3. **Command Center** stores bytes in `_segment_cache[camera_id][filename]`, evicting oldest past `SEGMENT_CACHE_MAX_PER_CAMERA`. +4. **CameraNode pushes the playlist** separately: `POST /api/cameras/{id}/playlist`. +5. **Command Center rewrites** the playlist's segment filenames to relative `segment/` proxy URLs, so the browser learns nothing about the node's own addressing. +6. **Browser** plays it as ordinary HLS. A camera whose heartbeat gap exceeds 90s flips to `offline` via the sweep loop. + +The memory ceiling matters: `SEGMENT_CACHE_MAX_TOTAL_BYTES` is 384 MiB of the web machine's 1 GiB. Raise one without the other and the kernel OOM-killer takes every org's streams down at once, well before the cache's own eviction can help. + +## The AI agent + +Command Center owns the queue; the agent is a worker draining it. That single decision is what lets the same code run hosted or on someone else's hardware. + +- **Push** (hosted default) — CC fires an HMAC-signed wakeup at `http://sentinel-command.flycast:8080/wakeup`, internal over 6PN. +- **Poll** — the agent asks CC for pending runs on an interval. No inbound connectivity, so it works behind NAT. Same shape CameraNode uses. + +A run: claim via `POST /runs/{id}/start` → investigate through MCP tools → report via `POST /runs/{id}/complete` with `incident`, `no_action`, or `error`. Bounded at every layer, with a 270s wall clock under Fly's 300s `kill_timeout`, and a CC-side reaper for runs that strand. + +The model is a config string (`LLM_MODEL`, via LiteLLM). **Changing it on the hosted deployment moves customer camera imagery to a different processor** — see `legal/SUB_PROCESSORS.md` before you do. + +Full detail: [SENTINEL_AGENT.md](SENTINEL_AGENT.md). + +## Who can talk to it + +There is no single "API key". Every class of caller has its own credential, scope, and revocation path. + +| Credential | Used by | Scope | +| ---------- | ------- | ----- | +| Clerk JWT | Browser users (hosted) | Per-user, per-org, role-aware | +| Local auth | Browser users (self-hosted) | Single admin, `AUTH_PROVIDER=local` | +| Node API key | CameraNodes | One node; hashed at rest, rotatable | +| MCP API key | Claude and other MCP clients | Org + `readonly` or full; daily cap | +| Integration key | Home Assistant | Org-wide camera read | +| `osa_` agent key | Sentinel AI agent | Per-org, issued in the dashboard | + +The MCP surface is **23 tools, 16 read / 7 write**. A `readonly` key is intersected with the read set in middleware, so scope is enforced before a tool runs rather than inside each one. + +## Data + +**Hosted — Postgres.** Three databases on the single `sentinel-postgres` cluster: `sentinel_command`, `sentinel_license`, `sentinel_sync`. Isolation is by *role*, not by cluster — each service's role owns exactly one database, `CONNECT` is revoked from `PUBLIC`, and none is a superuser. One cluster rather than three was a deliberate cost decision; roles supply the isolation separate clusters would have charged for. + +**Self-hosted — SQLite.** The same codebase. This is why the backend suite is parametrised across **both dialects** and a failure in either blocks the deploy. + +Schema changes run through a `sync_schema()` sweep on every boot rather than Alembic — see [ADR 0001](adr/0001-sync-schema-vs-alembic.md). + +Backups: nightly `pg_dump` for `sentinel_command` and `sentinel_license`, with restores rehearsed rather than assumed. `sentinel_sync` deliberately has none — it holds a mirror whose source of truth is the operator's local SQLite. See [DISASTER_RECOVERY.md](runbooks/DISASTER_RECOVERY.md). + +## Two ways to run it + +| | Hosted | Self-hosted | +| --- | ------ | ----------- | +| Auth | Clerk | Local admin | +| Database | Postgres | SQLite | +| Cameras, recording, motion, MCP | Plan-limited | **Free and unrestricted** | +| Sentinel AI | Plan-limited | Unlocked by licence key | +| Agent | `agent` process group | Bundled; `python -m app.sentinel_agent` | + +Only Sentinel AI is gated for self-hosters, because it is the one feature with an ongoing per-run cost. Everything else ships unlocked. + +Plans are enforced on five axes: camera cap, **viewer-hours** (the real tier axis — see [ADR 0002](adr/0002-viewer-hour-billing.md)), SSE caps, MCP daily cap, and log retention. + +## How code ships + +`.github/workflows/deploy.yml` builds one image and `flyctl deploy` reconciles both of Command Center's process groups. License and Sync each deploy from their own repo's `Test & Deploy` workflow. + +**Path filtering is asymmetric, and that is load-bearing.** `push` is filtered (docs and Markdown only); `pull_request` is **never** filtered. `master` requires three status checks, and GitHub reports *no status at all* for a workflow a path filter skipped — so a filtered PR trigger would hang every PR that missed it, presenting as a stuck check rather than a config error. + +**Known gap:** GitHub does not trigger `on: push` workflows for commits pushed with `GITHUB_TOKEN`, so an auto-merged Dependabot PR lands on `master` **without deploying**. CodeQL still goes green on those commits, which hides it convincingly. Closing it properly needs a PAT; meanwhile `deploy.yml` carries a `workflow_dispatch` trigger. This is why no repo here uses an auto-merge workflow except Command Center. + +## Where to go next + +- [SENTINEL_AGENT.md](SENTINEL_AGENT.md) — the AI agent in depth +- [runbooks/ON_CALL.md](runbooks/ON_CALL.md) — "the app is broken" +- [runbooks/DISASTER_RECOVERY.md](runbooks/DISASTER_RECOVERY.md) — "the data is gone" +- [LAUNCH_HANDOFF.md](LAUNCH_HANDOFF.md) — what's left before paying customers +- [adr/](adr/) — why non-obvious decisions were made +- `../AGENTS.md` — Command Center's internals, in detail diff --git a/docs/README.md b/docs/README.md index 672e015..e5a593c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,6 +6,10 @@ Supplementary documentation for Sentinel Command Center — the SaaS we operate Twelve user-only items (Clerk prod keys, backup restore test, lawyer signoff, status page vendor, etc.) that every code-side launch blocker has been closed against. Start here if you're driving toward launch. +## [ARCHITECTURE.md](ARCHITECTURE.md) — how the whole system fits together + +Start here if you're new. Every repository, every deployed service, and the paths between them — the signal path from camera to browser, the six credential types, where data lives, and how code ships. `README.md` is this repo's setup; `AGENTS.md` is Command Center's internals; ARCHITECTURE is the level above both. + ## [SENTINEL_AGENT.md](SENTINEL_AGENT.md) — the AI agent How the Sentinel AI agent works, how to run one yourself, and every environment variable it reads. It lives in this repo at `backend/app/sentinel_agent/` and deploys as the `agent` process group of the `sentinel-command` Fly app — not, as older references may suggest, a separate repository or app.