[BUG] LiteLLM: Requested token count exceeds the model's maximum context length

[robinchrist]

Problem (one or two sentences)

Certain models don't work via our LiteLLM Proxy due to Requested token count exceeds the model's maximum context length error.

Context (who is affected and when)

I see this when using RooCode via our LiteLLM Proxy

Reproduction steps

1. Configure models via LiteLLM
2. Use them

Expected result

The task should be executed

Actual result

Task is not executed, error is generated

Variations tried (optional)

No response

App Version

v3.53.0

API Provider (optional)

LiteLLM

Model Used (optional)

No response

Roo Code Task Links (optional)

No response

Relevant logs or errors (optional)

LiteLLM streaming error: 400
litellm.BadRequestError:
Together_aiException - Requested token count exceeds the model's maximum context length of 512000 tokens.
You requested a total of 521014 tokens: 9014 tokens from the input messages and 512000 tokens for the completion.
Please reduce the number of tokens in the input messages or the completion to fit within the limit.
No fallback model group found for original model_group=deepseek-ai/DeepSeek-V4-Pro. 
Fallbacks=[{'MiniMaxAI/MiniMax-M2.5': ['MiniMaxAI/MiniMax-M2.7']}]. Received Model Group=deepseek-ai/DeepSeek-V4-Pro

[roomote-v0]

I've analyzed this issue and here's my proposed implementation plan:

Root Cause

There are two contributing problems in the LiteLLM integration:

1. Fetcher fallback to max_tokens (src/api/providers/fetchers/litellm.ts, line 44): When the LiteLLM /v1/model/info endpoint does not provide max_output_tokens, the code falls back to max_tokens. In LiteLLM's API, max_tokens represents the total token budget (equivalent to context window), not the max output tokens. For DeepSeek-V4-Pro with a 512K context, this means maxTokens gets set to 512000.

2. Missing output token cap in provider (src/api/providers/lite-llm.ts, line 187): The LiteLLM provider reads info.maxTokens directly and passes it as max_tokens in the API request. Unlike other providers (e.g., base-openai-compatible-provider.ts), it does not route through getModelMaxOutputTokens(), which clamps output tokens to 20% of the context window. This means the raw 512000 value gets sent as the completion token limit, leaving no room for input tokens.

Proposed Fix

1. src/api/providers/lite-llm.ts (line ~187): Replace info.maxTokens with a call to getModelMaxOutputTokens({ modelId, model: info, settings: this.options }). This applies the centralized 20% cap, matching how other providers handle it.

2. src/api/providers/fetchers/litellm.ts (line 44): Remove the max_tokens fallback, since it represents total tokens (not output tokens) and is misleading. Change the fallback chain to max_output_tokens || 8192, so models without explicit output token limits get a safe default rather than their full context window.

3. Add/update tests to cover these changes.

This approach aligns the LiteLLM provider with the established pattern used by other providers and prevents max_tokens from exceeding the model's available token budget.

Please mention @roomote in a comment saying that you approve this plan, and I'll proceed with the implementation.