Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,23 @@ Sentinel Command Center — cloud dashboard for managing and viewing security ca

> **Brand-history note for grep-discoverability:** the product has carried three names — `OpenSentry` (early), `SourceBox Sentry` (mid), and `Sentinel by SourceBox` (current, from May 2026 onward). The `Sentinel AI` name is reserved specifically for the AI-agent feature. Both GitHub repos were renamed in May 2026: Command Center `OpenSentry-Command` → `Sentinel-Command`, and CameraNode `opensentry-cloud-node` → `Sentinel-CameraNode` (note the deliberate "CameraNode" — the repo name now describes the artifact more literally, while the binary, install paths, and product UI keep saying "CameraNode"). GitHub auto-redirects the old URLs, so any hardcoded reference in a release artifact / cached doc / external bookmark continues to resolve. Identifiers preserved verbatim across the entire rebrand (do **not** rename these without a migration plan): the binary name `sourcebox-sentry-cameranode`, the env-var prefix `SOURCEBOX_SENTRY_*`, the Windows install path `C:\ProgramData\SourceBoxSentry\`, the AES key-derivation domain string `opensentry-cameranode-machine-id-v2` (see CameraNode `database.rs::KEY_DOMAIN_V2`), and the production hostname `sentinel-command.com` (tied to the Fly app, decoupled from the repo rename).

## Contents

Long reference — jump rather than scroll.

| | |
| --- | --- |
| [Repository layout](#repository-layout--one-app-two-process-groups) — one app, two process groups | [Authentication](#authentication) — six credential types |
| [Build & Run](#build--run) | [Data Models](#data-models) |
| [Configuration](#configuration) | [API Routes](#api-routes) |
| [Project Structure](#project-structure) | [MCP Server](#mcp-server) — tools, scope middleware |
| [Architecture](#architecture) — request flow, video pipeline | [Plan Enforcement](#plan-enforcement) |
| [CORS](#cors) · [Rate Limiting](#rate-limiting) | [Background Loops](#background-loops) |
| [Webhook Handling](#webhook-handling) | [Key Patterns](#key-patterns) |
| [Setup Scripts](#setup-scripts) · [Key Dependencies](#key-dependencies) | [Development Notes](#development-notes) |

Wider than this file: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) covers how Command Center relates to the other services. The AI agent has its own reference at [docs/SENTINEL_AGENT.md](docs/SENTINEL_AGENT.md).

## Repository layout — one app, two process groups

Command Center and the Sentinel AI agent ship from **one repo, one image, one deploy**. They run as two Fly *process groups* on separate machines, not as two apps.
Expand All @@ -24,9 +41,7 @@ Four rules follow, and breaking any of them breaks a deploy:
3. **`[processes]` overrides the Dockerfile `CMD`.** The `app` command in `fly.toml` must stay in sync with that `CMD`.
4. **CI path filtering is asymmetric.** `push` is filtered (docs and Markdown only); `pull_request` is **never** filtered. `master` requires `Backend tests (sqlite)`, `Backend tests (postgres)` and `Frontend audit + build`, and GitHub reports *no status at all* for a workflow a path filter skipped — so a filtered PR trigger would hang every PR that missed it, presenting as a stuck check rather than a config error.

The agent machine is kept **warm** (`min_machines_running = 1`) rather than scaled to zero. Fly's proxy waits only ~8s for an auto-started machine to bind its port, and this process needs ~10s (Python + the MCP SDK + Sentry + a deferred LiteLLM import) — so an auto-started machine was declared unreachable and the wakeup came back `RemoteDisconnected`. It was ~7s before LiteLLM, i.e. always marginal. ~$2/month buys the problem away; see the comment on `[[services]]` in `fly.toml`.

The agent is a separate **process group** rather than a thread in the web app because a run holds base64 frames for up to 270s, and the segment cache is already budgeted 384 MiB of the web machine's 1 GiB. Sharing one machine is how the OOM killer takes every org's streams down at once. Being a separate *app* was never what bought that isolation.
The agent runs as a separate **process group** — its own machine, kept warm rather than scaled to zero. Both choices are deliberate and both have non-obvious reasons: memory contention with the segment cache, and a boot time that loses a race with Fly's proxy. Neither is restated here; see [docs/SENTINEL_AGENT.md](docs/SENTINEL_AGENT.md) for the agent's side and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md#deployed-services-flyio) for how it compares to the services that *do* sleep.

Self-hosting still works the same way: `python -m app.sentinel_agent` runs standalone with `AGENT_MODE=poll` and a per-org `osa_` key, needing no inbound connectivity. Agent docs are in `docs/SENTINEL_AGENT.md`. The code came from the `SourceBox-Sentinel` repo (archived 2026-09-09).

Expand Down
20 changes: 15 additions & 5 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,17 @@ Four apps. Command Center runs **two process groups from one image** — Fly giv
| `sentinel-sync` | `app` | 256 MB | **scales to zero** | One-way mirror receiver |
| `sentinel-postgres` | `app` | 512 MB | always | One cluster, three databases |

**Why two of these scale to zero and two don't.** Fly's proxy waits ~8s for an auto-started machine to bind its port. License (~4s) and Sync (~3s) clear that comfortably, and a missed call is free for both — a failed licence check-in falls into a 72-hour grace window, a failed sync push just retries next cycle. The agent needs ~10s and cannot make it, so it stays warm; see `SENTINEL_AGENT.md`. Command Center's web tier serves live video and never sleeps.
**Why two of these scale to zero and two don't.** This is a cross-service comparison, so it lives here rather than in any one service's docs.

Fly's proxy waits **~8s** for an auto-started machine to bind its port. That single number decides it:

| Service | Boot | Verdict |
| ------- | ---- | ------- |
| `sentinel-sync` | ~3s | clears it |
| `sentinel-license` | ~4s | clears it |
| `sentinel-command` / `agent` | ~10s | **misses it** — stays warm |

Boot time alone isn't sufficient; a missed call also has to be cheap. It is for both sleepers — a failed licence check-in falls into a grace window measured in days, and a failed sync push simply retries next cycle with the operator's local database still authoritative. Command Center's web tier serves live video and never sleeps.

## The signal path — camera to browser

Expand All @@ -43,9 +53,9 @@ This is the part most often assumed to work differently. There is **no object st
3. **Command Center** stores bytes in `_segment_cache[camera_id][filename]`, evicting oldest past `SEGMENT_CACHE_MAX_PER_CAMERA`.
4. **CameraNode pushes the playlist** separately: `POST /api/cameras/{id}/playlist`.
5. **Command Center rewrites** the playlist's segment filenames to relative `segment/<file>` proxy URLs, so the browser learns nothing about the node's own addressing.
6. **Browser** plays it as ordinary HLS. A camera whose heartbeat gap exceeds 90s flips to `offline` via the sweep loop.
6. **Browser** plays it as ordinary HLS. A camera that stops heartbeating flips to `offline` via the sweep loop.

The memory ceiling matters: `SEGMENT_CACHE_MAX_TOTAL_BYTES` is 384 MiB of the web machine's 1 GiB. Raise one without the other and the kernel OOM-killer takes every org's streams down at once, well before the cache's own eviction can help.
The cache is bounded, and its ceiling is **coupled to the machine's memory** — raise one without the other and the kernel OOM-killer takes every org's streams down at once, well before the cache's own eviction can help. Both numbers, and why they move together, are in the `[env]` comment in `fly.toml`.

## The AI agent

Expand All @@ -54,7 +64,7 @@ Command Center owns the queue; the agent is a worker draining it. That single de
- **Push** (hosted default) — CC fires an HMAC-signed wakeup at `http://sentinel-command.flycast:8080/wakeup`, internal over 6PN.
- **Poll** — the agent asks CC for pending runs on an interval. No inbound connectivity, so it works behind NAT. Same shape CameraNode uses.

A run: claim via `POST /runs/{id}/start` → investigate through MCP tools → report via `POST /runs/{id}/complete` with `incident`, `no_action`, or `error`. Bounded at every layer, with a 270s wall clock under Fly's 300s `kill_timeout`, and a CC-side reaper for runs that strand.
A run: claim via `POST /runs/{id}/start` → investigate through MCP tools → report via `POST /runs/{id}/complete` with `incident`, `no_action`, or `error`. Bounded at every layer — per-call, per-tool, iteration count, and wall clock — with a CC-side reaper for runs that strand anyway.

The model is a config string (`LLM_MODEL`, via LiteLLM). **Changing it on the hosted deployment moves customer camera imagery to a different processor** — see `legal/SUB_PROCESSORS.md` before you do.

Expand All @@ -73,7 +83,7 @@ There is no single "API key". Every class of caller has its own credential, scop
| Integration key | Home Assistant | Org-wide camera read |
| `osa_` agent key | Sentinel AI agent | Per-org, issued in the dashboard |

The MCP surface is **23 tools, 16 read / 7 write**. A `readonly` key is intersected with the read set in middleware, so scope is enforced before a tool runs rather than inside each one.
A `readonly` MCP key is intersected with the read-tool set **in middleware**, so scope is enforced before a tool runs rather than inside each one. Tool inventory and the read/write split: `../AGENTS.md` → MCP Server.

## Data

Expand Down
11 changes: 11 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,17 @@ Working drafts of customer-facing legal documents. Each is marked `DRAFT — NOT
- [DPA.md](legal/DPA.md) — Data Processing Agreement template, including SCC parameter annexes for EEA / UK transfers.
- [SUB_PROCESSORS.md](legal/SUB_PROCESSORS.md) — public sub-processor list with notice policy.

## One fact, one home

The failure mode for a doc set this size isn't missing information, it's the same number written in three places and updated in one. On 2026-09-09 the reasoning behind the agent's isolation existed in three files in three phrasings, and "23 tools" appeared five ways.

The rule:

- **Reference docs state a value once.** Whichever doc owns the subject owns the number. `AGENTS.md` owns Command Center's internals; `SENTINEL_AGENT.md` owns the agent's; `ARCHITECTURE.md` owns facts that only make sense *across* services (like the Fly proxy's bind budget, which is why two services scale to zero and two don't). Everything else links.
- **ARCHITECTURE.md carries structure, not values.** Relationships change rarely; numbers drift constantly. If you're about to add a figure there, check whether the doc that owns the subject should carry it instead.
- **Operational docs may inline a value** where stopping to look it up would make them unusable. `LAUNCH_HANDOFF.md` saying "kill a CameraNode for >90s" is correct; it's an instruction, not a specification.
- **Prefer pointing at code.** A value with a good comment beside it (`fly.toml`'s `[env]` block, `plans.py`) is more durable than the same value copied into prose, because the next person to change it is already looking at it.

## Writing new docs

- **ADR** — when you make a decision that was hard to make, or that someone else will almost certainly re-argue. Write it *while the tradeoffs are fresh*, not six months later.
Expand Down
10 changes: 6 additions & 4 deletions docs/SENTINEL_AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,23 @@ notification fires ┌─ POST /wakeup (HMAC-signed)
## Architecture

- **LLM**: any provider, via [LiteLLM](https://docs.litellm.ai/). `LLM_MODEL` is the only setting that changes — `ollama_chat/qwen3.5:cloud` today, `anthropic/claude-sonnet-5` for a bring-your-own-key deployment, or an OpenAI-compatible endpoint for a custom model. **Must support tool calling *and* image input**: the agent fans out tool calls and feeds camera frames back in, so a text-only model does not degrade, it fails every run.
- **Tools**: Command Center's MCP server (23 tools — list/view/watch cameras, create/finalize incidents, attach evidence) over streamable HTTP.
- **Tools**: Command Center's MCP server cameras, incidents, evidenceover streamable HTTP. Inventory in `../AGENTS.md` → MCP Server.
- **Server**: Starlette + uvicorn. No auth on its own state — every accepted request is HMAC-verified against the shared `SENTINEL_AGENT_KEY`.
- **Master of pending work**: Command Center's `sentinel_runs` table. The agent persists nothing; every wakeup re-fetches what's pending.

### Why a separate process group, not a thread

A run holds base64 camera frames for up to 270s, and the web machine's segment cache is already budgeted 384 MiB of its 1 GiB (`SEGMENT_CACHE_MAX_TOTAL_BYTES` in `fly.toml`). Sharing one machine is how the OOM killer takes every org's live streams down at once. Being a separate *app* was never what bought that isolation — a separate process group is.
A run holds base64 camera frames for the length of its wall-clock budget, and the web machine's segment cache is already budgeted most of that machine's memory (see the `[env]` comment in `fly.toml`). Sharing one machine is how the OOM killer takes every org's live streams down at once.

Being a separate *app* was never what bought that isolation — a separate process group is.

### Why the machine stays warm

`min_machines_running = 1`, deliberately, even though this worker's shape screams scale-to-zero.

Fly's proxy waits ~8s for an auto-started machine to bind its port, and this process needs ~10s (Python + the MCP SDK + Sentry + a deferred LiteLLM import). An auto-started machine was declared unreachable and Command Center's wakeup came back `RemoteDisconnected` — on **every** wakeup, since it had scaled to zero. It was ~7s before LiteLLM, i.e. always marginal.
This process takes ~10s to bind its port (Python + the MCP SDK + Sentry + a deferred LiteLLM import), which is longer than Fly's proxy waits for a machine it auto-started. Every wakeup against a stopped machine came back `RemoteDisconnected`. It was ~7s before LiteLLM always marginal, and the migration only exposed it.

~$2/month removes cold starts from the wakeup path instead of racing them. See the comment on `[[services]]` in `fly.toml`.
Keeping one small machine warm costs about $2/month and removes cold starts from the wakeup path instead of racing them. The proxy's actual budget, and how the sibling services compare against it, are in [ARCHITECTURE.md](ARCHITECTURE.md#deployed-services-flyio); the deployment reasoning is in the `[[services]]` comment in `fly.toml`.

## Push or poll

Expand Down
14 changes: 14 additions & 0 deletions docs/runbooks/DISASTER_RECOVERY.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ the **`sentinel_command` database on the managed `sentinel-postgres`
Postgres cluster**. Recovery is **restore from a backup**, so the backup
must exist and the restore must have been rehearsed.

## Start here — which situation is this?

| Situation | Go to |
| --------- | ----- |
| The hosted database is lost, corrupted, or wrong | [Restore procedure](#restore-procedure) |
| A machine or volume is gone | [Restore procedure](#restore-procedure) |
| A **self-hosted** customer lost their local database | [Restoring from the cloud mirror](#self-hosted-installs-restoring-from-the-cloud-mirror) |
| You need to know whether a backup even exists | [Backups: how they're produced](#backups-how-theyre-produced) |
| Nothing is broken — you're preparing | [The one thing to do before launch](#the-one-thing-to-do-before-launch) · [Rehearsal drill](#rehearsal-drill-do-this-before-launch-then-quarterly) |
| The service is broken but the **data is fine** | [ON_CALL.md](ON_CALL.md) — not this file |

**Before you restore anything:** a restore is destructive and a wrong one compounds the damage. Read the whole [Restore procedure](#restore-procedure) section before running its first command.


> 🔀 **Migrated to Postgres (2026-09-07).** Until this date the hosted
> database was a single SQLite file on the `sentinel_data` Fly volume,
> and this runbook was written around that. What changed:
Expand Down
20 changes: 20 additions & 0 deletions docs/runbooks/ON_CALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,26 @@ checks**, **Likely causes**, **Fix paths**, **When to escalate**.

---

## Start here — what are you seeing?

You know the symptom, not the scenario letter. Find the row, jump to the section.

| What you're seeing | Go to |
| ------------------ | ----- |
| A Sentry alert fired | [A — Sentry alert](#scenario-a-sentry-alert-fired) |
| One customer: "all my cameras are offline" | [B — cameras offline](#scenario-b-customer-reports-all-my-cameras-are-offline) |
| One customer: "the stream won't play" | [C — stream won't play](#scenario-c-customer-reports-stream-wont-play) |
| The app is slow, timing out, or the DB is unhappy | [D — database](#scenario-d-database-is-slow-unresponsive-or-out-of-disk) |
| **Several unrelated** customers at once | [E — multi-customer](#scenario-e-multiple-unrelated-customers-reporting-issues-at-once) |
| Signs of a breach or access you can't account for | [F — suspected breach](#scenario-f-suspected-data-breach-or-unauthorized-access) |
| A customer asked to be deleted | [G — deletion request](#scenario-g-customer-requests-deletion-gdpr--ccpa-right-to-erase) |
| Alerts aren't arriving in inboxes | [H — email](#scenario-h-email-isnt-sending-resend-transport-failures) |
| A deploy went red | [I — CI deploy failing](#scenario-i-ci-deploy-is-failing) |
| About to push and want to be careful | [J — pre-deploy check](#scenario-j-pre-deploy-sanity-check-before-pushing-master) |
| **Data is missing, corrupted, or gone** | **[DISASTER_RECOVERY.md](DISASTER_RECOVERY.md)** — not this file |

That last row matters: this runbook is for "the service is broken." If the *data* is gone, you are in the wrong document and the procedures here won't help.

## Quick reference

| Tool / link | Why |
Expand Down