Run the Sentinel AI agent as a process group of sentinel-command - #274
Merged
Conversation
Finishes what "combine the litellm sentinel ai into the command center" asked for. The previous change moved the agent's code into this repo but left it deploying as a separate Fly app, which is why sourcebox-sentinel kept looking like leftovers — it wasn't leftovers, it was the live service, because the integration only went half way. The claim that blocked this before was wrong. I said the two could not share a dependency set: Command Center resolves mcp 1.28.1, the agent 2.2.0, different majors. That compared resolved lockfile versions instead of declared constraints. The agent declares mcp>=1.6.0,<3 — which admits 1.28.1 — and every other difference (starlette, pydantic-settings, sentry-sdk) was likewise a resolution artifact, not a conflict. One merged environment resolves in 126 packages with `ollama` as the only genuinely new one. That mattered because Fly gives one image per app; process groups differ only by command. Without a single resolvable dependency set none of this was possible. Verified rather than assumed, since downgrading the agent's mcp was the real risk: 1.28.1 exports both the 1.x and 2.x symbol names, the existing import shim binds cleanly, and the agent's own MCPClientManager opened a live streamable-HTTP session against Command Center's /mcp and discovered all 23 tools. The agent also runs from its new module path in poll mode — drains /pending, serves /health 200, rejects an unsigned /wakeup with 401. The agent stays on its OWN MACHINE, just not its own app. A run holds base64 frames for up to 270s and the segment cache is already budgeted 384 MiB of the web machine's 1 GiB; sharing one machine is how the OOM killer takes every org's streams down at once. Being a separate *app* was never what bought that isolation — a separate process group is. Two things in fly.toml would each break the deploy silently if missed: [[mounts]] is now scoped to processes = ["app"], or the agent machine fights the web machine for the volume's single attachment slot; and [processes] overrides the Dockerfile CMD, so the app command must track it. The deploy step also had to change. It ran `machine update` on `machines list | .[0].id` — its own comment flagged that as needing a loop once we ran more than one machine. Two groups makes ".[0]" a coin flip, and `machine update` cannot CREATE the agent machine for a newly-declared group at all, so the group would simply never come up. Now `flyctl deploy --image` reconciles fly.toml against reality while keeping our build-and-push and its layer cache. Also drops "Agent checks" from master's required checks — its workflow no longer exists, and a required check that never reports hangs every PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finishes the integration. One repo, one image, one deploy — and
sourcebox-sentinelgoes away.The claim that blocked this was wrong
I previously said the two couldn't share a dependency set: Command Center resolves
mcp1.28.1, the agent 2.2.0, different majors. That compared resolved lockfile versions instead of declared constraints. The agent declaresmcp>=1.6.0,<3, which admits 1.28.1. Every other difference was the same kind of artifact:mcp>=1.6.0,<3starlette>=1.3.1,<2pydantic-settings>=2.0.0sentry-sdk>=2.0.0One merged environment resolves in 126 packages, with
ollamathe only genuinely new one. That mattered because Fly gives one image per app — process groups differ only by command — so without this the whole approach was impossible.Shape
The agent keeps its own machine, just not its own app. A run holds base64 frames for up to 270s and the segment cache already claims 384 MiB of the web machine's 1 GiB — sharing one machine is how the OOM killer takes every org's streams down. A separate app was never what bought that isolation.
Verified, not assumed
Downgrading the agent's
mcpwas the real risk, so it got tested directly:mcp1.28.1 exports both the 1.x and 2.x symbol names; the existing import shim binds cleanlyMCPClientManageropened a live streamable-HTTP session against Command Center's/mcpand discovered all 23 tools/pendingin poll mode,/health200, unsigned/wakeup→ 401ruffclean across the whole backend (the agent's code had 8 pre-existing violations that CC's config catches — fixed, and the version shim confirmed intact afterwards)Two things that would have broken the deploy silently
[[mounts]]is now scoped toprocesses = ["app"]. Unscoped it applies to every group, and the agent machine fails to boot fighting for the volume's single attachment slot.machine updateonmachines list | .[0].id— its own comment flagged that as needing a loop once we ran more than one machine. Two groups makes.[0]a coin flip, andmachine updatecannot create the agent machine for a newly-declared group, so it would never have come up. Nowflyctl deploy --imagereconcilesfly.tomlagainst reality while keeping our build-and-push and its layer cache.Also drops
Agent checksfrom master's required checks — its workflow is gone, and a required check that never reports hangs every PR.After merge
Cutover is ordered deliberately: verify the agent group is serving → move the five secrets → repoint
SENTINEL_AGENT_WEBHOOK_URLto.flycast→ confirm a real run end-to-end → only then destroysourcebox-sentinel. Repointing early means motion events fire webhooks at a dead host and runs sit pending until the reaper errors them.LiteLLM is Phase 2, deliberately separate — bundling a structural move with a behavioural change means one revert can't tell you which broke.
🤖 Generated with Claude Code