Make the model a config string: LiteLLM replaces the Ollama SDK - #277
Merged
Conversation
The agent hardcoded an Ollama client, so "which model runs Sentinel AI"
was a code change. It is now LLM_MODEL — `ollama_chat/qwen3.5:cloud`
today, `anthropic/claude-sonnet-5` for a bring-your-own-key self-hoster,
and eventually a SourceBox model behind an OpenAI-compatible endpoint.
That last one is the point: this is the seam a custom model plugs into.
LiteLLM replaces the ollama SDK outright rather than sitting beside it,
because it speaks Ollama natively via `ollama_chat/`. One provider path,
not two.
Existing deployments need NO env changes. LLM_* wins if set; otherwise
the OLLAMA_* trio is mapped onto it. Verified both directions, including
that an override does not leak the Ollama host to a hosted provider.
OLLAMA_API_KEY stops being a required field, so a new validator refuses
to boot with no credential at all — that check used to be a side effect
of the field being mandatory, and losing it would have turned a config
mistake into a run that fetches work, calls the model and errors.
The message translation is the real work, and I under-sold it earlier as
"42 lines". Ollama and OpenAI-shaped APIs disagree structurally:
- tool results key off tool_call_id, not tool_name
- tool arguments arrive as a JSON string, not a dict
- images cannot ride on a `tool` message at all — they need a `user`
message whose content is a parts list with a `data:` URI
So llm.py now owns every provider-shaped detail and the loop builds
messages only through its helpers. Swapping providers again touches one
file.
16 tests cover that translation because every failure mode in it is
QUIET: a wrongly-shaped message does not raise, it just means the model
never saw the camera frame or cannot match a result to the call it made.
The run completes, reports no_action, and looks fine.
Cost, accepted deliberately: +24 packages and ~160 MB, boto3 and
tokenizers included as hard requirements of litellm. The web process
group carries that too, since Fly gives one image per app. The
alternative is two images, which is the separate-app arrangement this
repo just finished removing. `import litellm` also costs ~2s, paid on
every cold start — measured against the wakeup path after deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2.
LLM_MODELnow selects the model —ollama_chat/qwen3.5:cloudtoday,anthropic/claude-sonnet-5for a bring-your-own-key self-hoster, and eventually a SourceBox model behind an OpenAI-compatible endpoint. That last one is the point: this is the seam a custom model plugs into.LiteLLM replaces the ollama SDK rather than sitting beside it — it speaks Ollama natively via
ollama_chat/, so one provider path instead of two.Existing deployments need no env changes
LLM_*wins if set; otherwise theOLLAMA_*trio is mapped onto it. Verified both ways, including that an override doesn't leak the Ollama host to a hosted provider.OLLAMA_API_KEYstops being a required field, so there's a new validator that refuses to boot with no credential. That guard used to be a side effect of the field being mandatory — losing it would have turned a config mistake into a run that fetches work, calls the model, and errors.The translation was the real work
I under-sold this earlier as "42 lines, a small swap." Ollama and OpenAI-shaped APIs disagree structurally:
tool_nametool_call_idfrom the callimages: [b64]on any messagedata:URI parts on a user message — atoolmessage can't carry onellm.pynow owns every provider-shaped detail; the loop builds messages only through its helpers. Swapping providers again touches one file.16 tests cover the translation, because every failure mode in it is quiet: a wrongly-shaped message doesn't raise, it just means the model never saw the camera frame, or can't match a result to the call it made. The run completes, reports
no_action, and looks fine.Cost, accepted deliberately
+24 packages / ~160 MB (
boto3andtokenizersare hard requirements of litellm, not optional extras). The web process group carries that too, since Fly gives one image per app — the alternative is two images, which is the separate-app arrangement we just finished removing.import litellmalso costs ~2s, paid on every cold start against Command Center's 5s webhook timeout. Measuring that against the real wakeup path after deploy; the webhook is fire-and-forget with a re-fire loop, so it degrades rather than drops, but if it's tight I'll make the import lazy.864 tests pass (848 + 16), ruff clean.
🤖 Generated with Claude Code