Skip to content

perf(kiro): cache provider-scoped model catalog to avoid per-account fetches#145

Open
tonybase wants to merge 1 commit into
kaitranntt:mainfrom
tonybase:perf/kiro-models-cache
Open

perf(kiro): cache provider-scoped model catalog to avoid per-account fetches#145
tonybase wants to merge 1 commit into
kaitranntt:mainfrom
tonybase:perf/kiro-models-cache

Conversation

@tonybase

Copy link
Copy Markdown

Summary

When managing many Kiro accounts, model registration called ListAvailableModels once per account on every registration cycle — startup, config reload, and the 3-hour catalog refresh. With N accounts this amplified into N upstream requests per cycle (the kiro: fetched N models from API log line fires per account).

The Kiro model catalog is provider-scoped — identical for every account; profileArn only routes billing. So this PR caches the catalog once and shares it across all Kiro auths.

What changed

  • Add a small TTL cache (kiroModelsCache) on Service for the Kiro base (pre-agentic) model list.
  • fetchKiroModels now: cache lookup → single upstream ListAvailableModels (fetchKiroBaseModelsFromAPI) → per-read agentic variant generation.
  • Coalescing is done by holding the mutex across the fetch: a concurrent burst of accounts collapses into one upstream call (latecomers block, then read the fresh cache). The TTL (30 min) collapses repeat fetches across registration cycles.
  • Agentic variants are derived per read, not cached, because they depend on the kiro-system-prompt-inject-enable runtime flag rather than the account.
  • Failed and empty fetches are not cached, so a later caller can retry (and still falls back to the static registry.GetKiroModels()).

Result

N Kiro accounts trigger at most one ListAvailableModels call per TTL, instead of one per account per registration cycle.

Notes / scope

  • Scoped to Kiro only. The same per-account fetch amplification exists for cursor, github-copilot, kilo, qoder, and antigravity — if this approach is acceptable, it can be extended in a follow-up.
  • The TTL is a constant (30 min). Happy to make it configurable if maintainers prefer.
  • The shared cached slice is only consumed by non-mutating helpers (generateKiroAgenticVariants, applyExcludedModels, applyModelPrefixes all build new slices / clone before edits), so sharing is safe.

Testing

  • New service_kiro_models_cache_test.go covering: concurrent coalescing (50 callers → 1 fetch), TTL reuse + expiry refresh, failures not cached, empty results not cached. All pass under -race.
  • gofmt -w ., go build ./cmd/server, and go test ./sdk/cliproxy/... all pass.

🤖 Generated with Claude Code

…fetches

Kiro model registration called ListAvailableModels once per account on every
registration cycle (startup, config reload, 3h catalog refresh). With a fleet
of accounts this amplified into N upstream requests each cycle.

The Kiro model catalog is provider-scoped (identical for every account;
profileArn only routes billing), so cache the base (pre-agentic) catalog in a
small TTL cache shared across all Kiro auths. The mutex is held across the
fetch so a concurrent burst of accounts collapses into a single upstream call;
the TTL collapses repeat fetches across registration cycles. Agentic variants
are derived per read since they depend on a runtime flag, not the account.
Failed and empty fetches are not cached so a later caller can retry.

Result: N accounts trigger at most one ListAvailableModels call per TTL
instead of one per account per cycle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant