[SDK] Compaction does not trigger for BYOK Anthropic — model token budget not resolved

[KennyTurtle]

Summary

Two related problems with infinite sessions compaction and Anthropic models:

1. BYOK providers: Compaction never triggers — the CLI can't resolve max_context_window_tokens for BYOK providers, so compaction is silently skipped.
2. Catalog models: Token budget is hardcoded to 128K for all models (test/harness/replayingCapiProxy.ts:1088), incorrect for Anthropic Claude models which have 200K context windows.

Repro (confirmed 2026-04-13)

SDK v0.2.2, model: claude-sonnet-4-20250514 via BYOK (CAPI relay)

Thresholds: 5% background, 30% blocking
Expected trigger: ~6,400 tokens (5% of 128K) or ~10,000 (5% of 200K)

  Message 1:  input=62       Message 6:  input=6,354
  Message 2:  input=1,047    Message 7:  input=7,817
  Message 3:  input=2,198    Message 8:  input=9,361
  Message 4:  input=3,516    Message 9:  input=10,870
  Message 5:  input=4,847    Message 10: input=12,390

  Total input: 58,462 tokens
  Compaction events: 0  ← NEVER triggered

No session.compaction_start or session.compaction_complete events received, even at 12K+ input tokens per call (far exceeding any 5% threshold).

SDK code references

Where the budget is hardcoded (CLI layer)

File	Line	What
test/harness/replayingCapiProxy.ts	1088	max_context_window_tokens: 128000 for ALL models

SDK type flow (passes through, no provider logic)

File	Line	What
python/copilot/generated/rpc.py	182	max_context_window_tokens: float
python/copilot/client.py	332	`max_context_window_tokens: int
go/types.go	823	MaxContextWindowTokens int
nodejs/src/types.ts	1577	max_context_window_tokens: number

Compaction threshold config

File	Line	What
go/types.go	463	BackgroundCompactionThreshold *float64
python/copilot/session.py	785	background_compaction_threshold field

Impact on Dracarys

• Long Anthropic sessions will eventually hit 200K context limit and stall — compaction should prevent this
• For catalog-routed models, compaction wastes ~58K tokens of usable context (triggers at 80% of 128K instead of 80% of 200K)
• Our PR Expose cwd/Repo information of a session #413 (infinite sessions support) is blocked on this for Anthropic providers

Suggested fix

1. BYOK: Accept max_context_window_tokens on ProviderConfig. Use provider-type defaults:
   • anthropic → 200,000
   • openai → 128,000
2. Catalog: Populate from actual model metadata (branch mackinnonbuck/byok-provider-token-limits partially addresses this)

[github-actions]

Investigation findings

Label applied: bug — the core claim is valid and confirmed.

What I looked at

File	Finding
python/copilot/session.py:803–815	ProviderConfig has no maxPromptTokens or token limit fields
python/copilot/client.py:2108–2136	_convert_provider_to_wire_format only serializes type, baseUrl, apiKey, wireApi, bearerToken, azure
nodejs/src/types.ts:1363–1400	Same gap: ProviderConfig has no token limit fields
test/harness/replayingCapiProxy.ts:1078–1091	createGetModelsResponse hardcodes max_context_window_tokens: 128000 for all models

Root cause

BYOK compaction never triggers because the SDK's ProviderConfig does not expose the token-limit fields that the Copilot CLI runtime uses to determine when to compact. Specifically, the runtime uses maxPromptTokens (not maxContextWindowTokens) to decide the compaction threshold — when the prompt approaches this limit, compaction fires. Since there's no catalog entry for a BYOK provider, the runtime has no way to learn the model's prompt token budget, and compaction is silently skipped.

One small correction to the issue description: the controlling field is maxPromptTokens (used by the runtime to trigger compaction before a request), not max_context_window_tokens (which is an internal runtime fallback not intended as a public SDK field). Both are missing from ProviderConfig today, but maxPromptTokens is the one that gates compaction.

Test harness 128K hardcode: The comment in createGetModelsResponse acknowledges this is an approximation. It would cause compaction to fire earlier than ideal for 200K models, but would not suppress it entirely. This is a secondary concern behind the BYOK bug.

Status

Branch mackinnonbuck/byok-provider-token-limits is already addressing this fix — it adds maxOutputTokens and maxPromptTokens to ProviderConfig across all four SDK languages (and explicitly does not expose maxContextWindowTokens as a public field). That branch should unblock the repro described here once merged.

Generated by Bug Handler for issue #1072 · ● 789.8K · ◷