Import LiteLLM on first use — at module scope it broke the wakeup - #278
Merged
Conversation
Importing litellm at module scope made the agent unreachable on cold start. Measured in production immediately after the previous deploy: 20:23:07 Preparing to run: python -m app.sentinel_agent 20:23:15 proxy: gave up after 15 attempts (8.48s) 20:23:23 Uvicorn running on 0.0.0.0:8080 The machine itself started in 928ms. The Python process then took 16s to bind, because litellm pulls boto3, tokenizers, openai and ~20 more, and a cold shared-cpu-1x is far slower at it than a warm laptop — I measured 2.08s locally and took that as the cost, which understated it by roughly a factor of five. Fly's proxy waits ~8.5s for the port, so it declared the machine it had just auto-started unreachable and Command Center's webhook came back RemoteDisconnected. The agent had scaled to zero, so this hit every wakeup, not just the first. Deferring the import drops agent boot from ~2.1s to 0.33s locally and takes litellm off the critical path entirely: the server binds and answers /wakeup immediately, and the ~2s is paid inside the drain, where it sits next to a 270s budget and an LLM round-trip. Worth noting the failure would have been survivable but expensive — the webhook is fire-and-forget with a re-fire loop, so runs would have been picked up eventually while every wakeup paid a cold start and timed out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caught by measuring the cold-start path in production right after #277 deployed, rather than assuming it was fine.
The machine started in 928ms. The Python process then took 16s to bind, because litellm pulls boto3, tokenizers, openai and ~20 more and a cold shared-cpu-1x is much slower at it than a warm laptop. I measured 2.08s locally and reported that as the cost — it understated the real thing by roughly 5×.
Fly's proxy waits ~8.5s for the port, so it declared the machine it had just auto-started unreachable. Since the agent scales to zero, this hit every wakeup.
Deferring the import takes it off the critical path: agent boot goes ~2.1s → 0.33s locally, the server binds and answers
/wakeupimmediately, and litellm's ~2s is paid inside the drain where it sits beside a 270s budget and an LLM round-trip.The failure would have been survivable but expensive — the webhook is fire-and-forget with a re-fire loop, so runs get picked up eventually while every wakeup pays a cold start and times out.
864 tests pass, ruff clean.
🤖 Generated with Claude Code