diff --git a/AGENTS.md b/AGENTS.md index 6540493..06c17ea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,23 @@ Sentinel Command Center — cloud dashboard for managing and viewing security ca > **Brand-history note for grep-discoverability:** the product has carried three names — `OpenSentry` (early), `SourceBox Sentry` (mid), and `Sentinel by SourceBox` (current, from May 2026 onward). The `Sentinel AI` name is reserved specifically for the AI-agent feature. Both GitHub repos were renamed in May 2026: Command Center `OpenSentry-Command` → `Sentinel-Command`, and CameraNode `opensentry-cloud-node` → `Sentinel-CameraNode` (note the deliberate "CameraNode" — the repo name now describes the artifact more literally, while the binary, install paths, and product UI keep saying "CameraNode"). GitHub auto-redirects the old URLs, so any hardcoded reference in a release artifact / cached doc / external bookmark continues to resolve. Identifiers preserved verbatim across the entire rebrand (do **not** rename these without a migration plan): the binary name `sourcebox-sentry-cameranode`, the env-var prefix `SOURCEBOX_SENTRY_*`, the Windows install path `C:\ProgramData\SourceBoxSentry\`, the AES key-derivation domain string `opensentry-cameranode-machine-id-v2` (see CameraNode `database.rs::KEY_DOMAIN_V2`), and the production hostname `sentinel-command.com` (tied to the Fly app, decoupled from the repo rename). +## Contents + +Long reference — jump rather than scroll. + +| | | +| --- | --- | +| [Repository layout](#repository-layout--one-app-two-process-groups) — one app, two process groups | [Authentication](#authentication) — six credential types | +| [Build & Run](#build--run) | [Data Models](#data-models) | +| [Configuration](#configuration) | [API Routes](#api-routes) | +| [Project Structure](#project-structure) | [MCP Server](#mcp-server) — tools, scope middleware | +| [Architecture](#architecture) — request flow, video pipeline | [Plan Enforcement](#plan-enforcement) | +| [CORS](#cors) · [Rate Limiting](#rate-limiting) | [Background Loops](#background-loops) | +| [Webhook Handling](#webhook-handling) | [Key Patterns](#key-patterns) | +| [Setup Scripts](#setup-scripts) · [Key Dependencies](#key-dependencies) | [Development Notes](#development-notes) | + +Wider than this file: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) covers how Command Center relates to the other services. The AI agent has its own reference at [docs/SENTINEL_AGENT.md](docs/SENTINEL_AGENT.md). + ## Repository layout — one app, two process groups Command Center and the Sentinel AI agent ship from **one repo, one image, one deploy**. They run as two Fly *process groups* on separate machines, not as two apps. @@ -24,9 +41,7 @@ Four rules follow, and breaking any of them breaks a deploy: 3. **`[processes]` overrides the Dockerfile `CMD`.** The `app` command in `fly.toml` must stay in sync with that `CMD`. 4. **CI path filtering is asymmetric.** `push` is filtered (docs and Markdown only); `pull_request` is **never** filtered. `master` requires `Backend tests (sqlite)`, `Backend tests (postgres)` and `Frontend audit + build`, and GitHub reports *no status at all* for a workflow a path filter skipped — so a filtered PR trigger would hang every PR that missed it, presenting as a stuck check rather than a config error. -The agent machine is kept **warm** (`min_machines_running = 1`) rather than scaled to zero. Fly's proxy waits only ~8s for an auto-started machine to bind its port, and this process needs ~10s (Python + the MCP SDK + Sentry + a deferred LiteLLM import) — so an auto-started machine was declared unreachable and the wakeup came back `RemoteDisconnected`. It was ~7s before LiteLLM, i.e. always marginal. ~$2/month buys the problem away; see the comment on `[[services]]` in `fly.toml`. - -The agent is a separate **process group** rather than a thread in the web app because a run holds base64 frames for up to 270s, and the segment cache is already budgeted 384 MiB of the web machine's 1 GiB. Sharing one machine is how the OOM killer takes every org's streams down at once. Being a separate *app* was never what bought that isolation. +The agent runs as a separate **process group** — its own machine, kept warm rather than scaled to zero. Both choices are deliberate and both have non-obvious reasons: memory contention with the segment cache, and a boot time that loses a race with Fly's proxy. Neither is restated here; see [docs/SENTINEL_AGENT.md](docs/SENTINEL_AGENT.md) for the agent's side and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md#deployed-services-flyio) for how it compares to the services that *do* sleep. Self-hosting still works the same way: `python -m app.sentinel_agent` runs standalone with `AGENT_MODE=poll` and a per-org `osa_` key, needing no inbound connectivity. Agent docs are in `docs/SENTINEL_AGENT.md`. The code came from the `SourceBox-Sentinel` repo (archived 2026-09-09). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 6cfc1cb..cc87939 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -32,7 +32,17 @@ Four apps. Command Center runs **two process groups from one image** — Fly giv | `sentinel-sync` | `app` | 256 MB | **scales to zero** | One-way mirror receiver | | `sentinel-postgres` | `app` | 512 MB | always | One cluster, three databases | -**Why two of these scale to zero and two don't.** Fly's proxy waits ~8s for an auto-started machine to bind its port. License (~4s) and Sync (~3s) clear that comfortably, and a missed call is free for both — a failed licence check-in falls into a 72-hour grace window, a failed sync push just retries next cycle. The agent needs ~10s and cannot make it, so it stays warm; see `SENTINEL_AGENT.md`. Command Center's web tier serves live video and never sleeps. +**Why two of these scale to zero and two don't.** This is a cross-service comparison, so it lives here rather than in any one service's docs. + +Fly's proxy waits **~8s** for an auto-started machine to bind its port. That single number decides it: + +| Service | Boot | Verdict | +| ------- | ---- | ------- | +| `sentinel-sync` | ~3s | clears it | +| `sentinel-license` | ~4s | clears it | +| `sentinel-command` / `agent` | ~10s | **misses it** — stays warm | + +Boot time alone isn't sufficient; a missed call also has to be cheap. It is for both sleepers — a failed licence check-in falls into a grace window measured in days, and a failed sync push simply retries next cycle with the operator's local database still authoritative. Command Center's web tier serves live video and never sleeps. ## The signal path — camera to browser @@ -43,9 +53,9 @@ This is the part most often assumed to work differently. There is **no object st 3. **Command Center** stores bytes in `_segment_cache[camera_id][filename]`, evicting oldest past `SEGMENT_CACHE_MAX_PER_CAMERA`. 4. **CameraNode pushes the playlist** separately: `POST /api/cameras/{id}/playlist`. 5. **Command Center rewrites** the playlist's segment filenames to relative `segment/` proxy URLs, so the browser learns nothing about the node's own addressing. -6. **Browser** plays it as ordinary HLS. A camera whose heartbeat gap exceeds 90s flips to `offline` via the sweep loop. +6. **Browser** plays it as ordinary HLS. A camera that stops heartbeating flips to `offline` via the sweep loop. -The memory ceiling matters: `SEGMENT_CACHE_MAX_TOTAL_BYTES` is 384 MiB of the web machine's 1 GiB. Raise one without the other and the kernel OOM-killer takes every org's streams down at once, well before the cache's own eviction can help. +The cache is bounded, and its ceiling is **coupled to the machine's memory** — raise one without the other and the kernel OOM-killer takes every org's streams down at once, well before the cache's own eviction can help. Both numbers, and why they move together, are in the `[env]` comment in `fly.toml`. ## The AI agent @@ -54,7 +64,7 @@ Command Center owns the queue; the agent is a worker draining it. That single de - **Push** (hosted default) — CC fires an HMAC-signed wakeup at `http://sentinel-command.flycast:8080/wakeup`, internal over 6PN. - **Poll** — the agent asks CC for pending runs on an interval. No inbound connectivity, so it works behind NAT. Same shape CameraNode uses. -A run: claim via `POST /runs/{id}/start` → investigate through MCP tools → report via `POST /runs/{id}/complete` with `incident`, `no_action`, or `error`. Bounded at every layer, with a 270s wall clock under Fly's 300s `kill_timeout`, and a CC-side reaper for runs that strand. +A run: claim via `POST /runs/{id}/start` → investigate through MCP tools → report via `POST /runs/{id}/complete` with `incident`, `no_action`, or `error`. Bounded at every layer — per-call, per-tool, iteration count, and wall clock — with a CC-side reaper for runs that strand anyway. The model is a config string (`LLM_MODEL`, via LiteLLM). **Changing it on the hosted deployment moves customer camera imagery to a different processor** — see `legal/SUB_PROCESSORS.md` before you do. @@ -73,7 +83,7 @@ There is no single "API key". Every class of caller has its own credential, scop | Integration key | Home Assistant | Org-wide camera read | | `osa_` agent key | Sentinel AI agent | Per-org, issued in the dashboard | -The MCP surface is **23 tools, 16 read / 7 write**. A `readonly` key is intersected with the read set in middleware, so scope is enforced before a tool runs rather than inside each one. +A `readonly` MCP key is intersected with the read-tool set **in middleware**, so scope is enforced before a tool runs rather than inside each one. Tool inventory and the read/write split: `../AGENTS.md` → MCP Server. ## Data diff --git a/docs/README.md b/docs/README.md index e5a593c..e414b14 100644 --- a/docs/README.md +++ b/docs/README.md @@ -50,6 +50,17 @@ Working drafts of customer-facing legal documents. Each is marked `DRAFT — NOT - [DPA.md](legal/DPA.md) — Data Processing Agreement template, including SCC parameter annexes for EEA / UK transfers. - [SUB_PROCESSORS.md](legal/SUB_PROCESSORS.md) — public sub-processor list with notice policy. +## One fact, one home + +The failure mode for a doc set this size isn't missing information, it's the same number written in three places and updated in one. On 2026-09-09 the reasoning behind the agent's isolation existed in three files in three phrasings, and "23 tools" appeared five ways. + +The rule: + +- **Reference docs state a value once.** Whichever doc owns the subject owns the number. `AGENTS.md` owns Command Center's internals; `SENTINEL_AGENT.md` owns the agent's; `ARCHITECTURE.md` owns facts that only make sense *across* services (like the Fly proxy's bind budget, which is why two services scale to zero and two don't). Everything else links. +- **ARCHITECTURE.md carries structure, not values.** Relationships change rarely; numbers drift constantly. If you're about to add a figure there, check whether the doc that owns the subject should carry it instead. +- **Operational docs may inline a value** where stopping to look it up would make them unusable. `LAUNCH_HANDOFF.md` saying "kill a CameraNode for >90s" is correct; it's an instruction, not a specification. +- **Prefer pointing at code.** A value with a good comment beside it (`fly.toml`'s `[env]` block, `plans.py`) is more durable than the same value copied into prose, because the next person to change it is already looking at it. + ## Writing new docs - **ADR** — when you make a decision that was hard to make, or that someone else will almost certainly re-argue. Write it *while the tradeoffs are fresh*, not six months later. diff --git a/docs/SENTINEL_AGENT.md b/docs/SENTINEL_AGENT.md index 1f74e95..8fccbe3 100644 --- a/docs/SENTINEL_AGENT.md +++ b/docs/SENTINEL_AGENT.md @@ -20,21 +20,23 @@ notification fires ┌─ POST /wakeup (HMAC-signed) ## Architecture - **LLM**: any provider, via [LiteLLM](https://docs.litellm.ai/). `LLM_MODEL` is the only setting that changes — `ollama_chat/qwen3.5:cloud` today, `anthropic/claude-sonnet-5` for a bring-your-own-key deployment, or an OpenAI-compatible endpoint for a custom model. **Must support tool calling *and* image input**: the agent fans out tool calls and feeds camera frames back in, so a text-only model does not degrade, it fails every run. -- **Tools**: Command Center's MCP server (23 tools — list/view/watch cameras, create/finalize incidents, attach evidence) over streamable HTTP. +- **Tools**: Command Center's MCP server — cameras, incidents, evidence — over streamable HTTP. Inventory in `../AGENTS.md` → MCP Server. - **Server**: Starlette + uvicorn. No auth on its own state — every accepted request is HMAC-verified against the shared `SENTINEL_AGENT_KEY`. - **Master of pending work**: Command Center's `sentinel_runs` table. The agent persists nothing; every wakeup re-fetches what's pending. ### Why a separate process group, not a thread -A run holds base64 camera frames for up to 270s, and the web machine's segment cache is already budgeted 384 MiB of its 1 GiB (`SEGMENT_CACHE_MAX_TOTAL_BYTES` in `fly.toml`). Sharing one machine is how the OOM killer takes every org's live streams down at once. Being a separate *app* was never what bought that isolation — a separate process group is. +A run holds base64 camera frames for the length of its wall-clock budget, and the web machine's segment cache is already budgeted most of that machine's memory (see the `[env]` comment in `fly.toml`). Sharing one machine is how the OOM killer takes every org's live streams down at once. + +Being a separate *app* was never what bought that isolation — a separate process group is. ### Why the machine stays warm `min_machines_running = 1`, deliberately, even though this worker's shape screams scale-to-zero. -Fly's proxy waits ~8s for an auto-started machine to bind its port, and this process needs ~10s (Python + the MCP SDK + Sentry + a deferred LiteLLM import). An auto-started machine was declared unreachable and Command Center's wakeup came back `RemoteDisconnected` — on **every** wakeup, since it had scaled to zero. It was ~7s before LiteLLM, i.e. always marginal. +This process takes ~10s to bind its port (Python + the MCP SDK + Sentry + a deferred LiteLLM import), which is longer than Fly's proxy waits for a machine it auto-started. Every wakeup against a stopped machine came back `RemoteDisconnected`. It was ~7s before LiteLLM — always marginal, and the migration only exposed it. -~$2/month removes cold starts from the wakeup path instead of racing them. See the comment on `[[services]]` in `fly.toml`. +Keeping one small machine warm costs about $2/month and removes cold starts from the wakeup path instead of racing them. The proxy's actual budget, and how the sibling services compare against it, are in [ARCHITECTURE.md](ARCHITECTURE.md#deployed-services-flyio); the deployment reasoning is in the `[[services]]` comment in `fly.toml`. ## Push or poll diff --git a/docs/runbooks/DISASTER_RECOVERY.md b/docs/runbooks/DISASTER_RECOVERY.md index 0ddc22d..620c9e6 100644 --- a/docs/runbooks/DISASTER_RECOVERY.md +++ b/docs/runbooks/DISASTER_RECOVERY.md @@ -13,6 +13,20 @@ the **`sentinel_command` database on the managed `sentinel-postgres` Postgres cluster**. Recovery is **restore from a backup**, so the backup must exist and the restore must have been rehearsed. +## Start here — which situation is this? + +| Situation | Go to | +| --------- | ----- | +| The hosted database is lost, corrupted, or wrong | [Restore procedure](#restore-procedure) | +| A machine or volume is gone | [Restore procedure](#restore-procedure) | +| A **self-hosted** customer lost their local database | [Restoring from the cloud mirror](#self-hosted-installs-restoring-from-the-cloud-mirror) | +| You need to know whether a backup even exists | [Backups: how they're produced](#backups-how-theyre-produced) | +| Nothing is broken — you're preparing | [The one thing to do before launch](#the-one-thing-to-do-before-launch) · [Rehearsal drill](#rehearsal-drill-do-this-before-launch-then-quarterly) | +| The service is broken but the **data is fine** | [ON_CALL.md](ON_CALL.md) — not this file | + +**Before you restore anything:** a restore is destructive and a wrong one compounds the damage. Read the whole [Restore procedure](#restore-procedure) section before running its first command. + + > 🔀 **Migrated to Postgres (2026-09-07).** Until this date the hosted > database was a single SQLite file on the `sentinel_data` Fly volume, > and this runbook was written around that. What changed: diff --git a/docs/runbooks/ON_CALL.md b/docs/runbooks/ON_CALL.md index a4d7c7c..d505246 100644 --- a/docs/runbooks/ON_CALL.md +++ b/docs/runbooks/ON_CALL.md @@ -15,6 +15,26 @@ checks**, **Likely causes**, **Fix paths**, **When to escalate**. --- +## Start here — what are you seeing? + +You know the symptom, not the scenario letter. Find the row, jump to the section. + +| What you're seeing | Go to | +| ------------------ | ----- | +| A Sentry alert fired | [A — Sentry alert](#scenario-a-sentry-alert-fired) | +| One customer: "all my cameras are offline" | [B — cameras offline](#scenario-b-customer-reports-all-my-cameras-are-offline) | +| One customer: "the stream won't play" | [C — stream won't play](#scenario-c-customer-reports-stream-wont-play) | +| The app is slow, timing out, or the DB is unhappy | [D — database](#scenario-d-database-is-slow-unresponsive-or-out-of-disk) | +| **Several unrelated** customers at once | [E — multi-customer](#scenario-e-multiple-unrelated-customers-reporting-issues-at-once) | +| Signs of a breach or access you can't account for | [F — suspected breach](#scenario-f-suspected-data-breach-or-unauthorized-access) | +| A customer asked to be deleted | [G — deletion request](#scenario-g-customer-requests-deletion-gdpr--ccpa-right-to-erase) | +| Alerts aren't arriving in inboxes | [H — email](#scenario-h-email-isnt-sending-resend-transport-failures) | +| A deploy went red | [I — CI deploy failing](#scenario-i-ci-deploy-is-failing) | +| About to push and want to be careful | [J — pre-deploy check](#scenario-j-pre-deploy-sanity-check-before-pushing-master) | +| **Data is missing, corrupted, or gone** | **[DISASTER_RECOVERY.md](DISASTER_RECOVERY.md)** — not this file | + +That last row matters: this runbook is for "the service is broken." If the *data* is gone, you are in the wrong document and the procedures here won't help. + ## Quick reference | Tool / link | Why |