From 4d324bcc18b74508a22c94e5b8df2ee0ae01a612 Mon Sep 17 00:00:00 2001 From: S'Bussiso Dube <80188685+Sbussiso@users.noreply.github.com> Date: Wed, 9 Sep 2026 13:25:06 -0700 Subject: [PATCH] =?UTF-8?q?Import=20LiteLLM=20on=20first=20use=20=E2=80=94?= =?UTF-8?q?=20at=20module=20scope=20it=20broke=20the=20wakeup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Importing litellm at module scope made the agent unreachable on cold start. Measured in production immediately after the previous deploy: 20:23:07 Preparing to run: python -m app.sentinel_agent 20:23:15 proxy: gave up after 15 attempts (8.48s) 20:23:23 Uvicorn running on 0.0.0.0:8080 The machine itself started in 928ms. The Python process then took 16s to bind, because litellm pulls boto3, tokenizers, openai and ~20 more, and a cold shared-cpu-1x is far slower at it than a warm laptop — I measured 2.08s locally and took that as the cost, which understated it by roughly a factor of five. Fly's proxy waits ~8.5s for the port, so it declared the machine it had just auto-started unreachable and Command Center's webhook came back RemoteDisconnected. The agent had scaled to zero, so this hit every wakeup, not just the first. Deferring the import drops agent boot from ~2.1s to 0.33s locally and takes litellm off the critical path entirely: the server binds and answers /wakeup immediately, and the ~2s is paid inside the drain, where it sits next to a 270s budget and an LLM round-trip. Worth noting the failure would have been survivable but expensive — the webhook is fire-and-forget with a re-fire loop, so runs would have been picked up eventually while every wakeup paid a cold start and timed out. Co-Authored-By: Claude Opus 5 --- backend/app/sentinel_agent/llm.py | 35 +++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 6 deletions(-) diff --git a/backend/app/sentinel_agent/llm.py b/backend/app/sentinel_agent/llm.py index 39f6967..46b692b 100644 --- a/backend/app/sentinel_agent/llm.py +++ b/backend/app/sentinel_agent/llm.py @@ -28,15 +28,38 @@ import logging from typing import Any -import litellm - from app.sentinel_agent.config import Settings logger = logging.getLogger(__name__) -# LiteLLM chats to stdout about provider quirks on import and on first -# call; the agent's logs are read during incidents and this is noise. -litellm.suppress_debug_info = True +_litellm = None + + +def _get_litellm(): + """Import LiteLLM on first use, not at module import. + + This is not micro-optimisation — importing it at module scope broke + the wakeup path outright. LiteLLM pulls boto3, tokenizers, openai and + ~20 more; on a cold shared-cpu-1x that pushed the agent from ~7s to + ~16s between process start and uvicorn listening. Fly's proxy gives + up waiting for the port after ~8.5s, so the machine it had just + auto-started was declared unreachable and Command Center's webhook + came back RemoteDisconnected. Observed exactly that on 2026-09-09 + before this change. + + Deferring it means the server binds immediately and answers /wakeup, + and the import is paid inside the drain instead — where it competes + with a 270 s budget and an LLM round-trip, and is noise. + """ + global _litellm + if _litellm is None: + import litellm + + # LiteLLM chats to stdout about provider quirks; the agent's logs + # are read during incidents and this is noise. + litellm.suppress_debug_info = True + _litellm = litellm + return _litellm # Marker left on a message whose frames were pruned, so the pruning pass # is idempotent and a re-prune doesn't stack notices. @@ -79,7 +102,7 @@ async def chat( kwargs["api_base"] = self.api_base response = await asyncio.wait_for( - litellm.acompletion(**kwargs), + _get_litellm().acompletion(**kwargs), timeout=self.timeout_seconds, ) return response.choices[0].message