fix(kosong): fail fast on quota-exhausted 429 instead of retrying#1857
fix(kosong): fail fast on quota-exhausted 429 instead of retrying#1857vinlee19 wants to merge 3 commits into
Conversation
A 429 caused by an exhausted account quota or insufficient balance (Moonshot error.type "exceeded_current_quota_error", OpenAI "insufficient_quota") can never succeed on retry, yet it was classified as APIProviderRateLimitError and silently retried for the whole budget (10 attempts, ~3 minutes of backoff) with no UI feedback — the session appeared frozen on every request. Introduce APIProviderQuotaExhaustedError, minted in normalizeAPIStatusError from the structured body error.type/error.code forwarded by convertOpenAIError, with billing-anchored message patterns as a fallback for gateways that flatten the body to text. The new class is excluded from isRetryableGenerateError (fail fast, even when a retry-after header is present) and from isProviderRateLimitError (no swarm requeue/suspend). toKimiErrorPayload and translateProviderError map it to provider.api_error (retryable: false) instead of provider.rate_limit, and classifyApiError reports it as quota_exhausted in telemetry. agent-core-v2 mirrors the same fix. Transient rate-limit 429s keep the existing retry, backoff, and Retry-After behavior (verified end-to-end against a mock provider: quota body fails after attempt 1/10; rate-limit body still walks the full 10-attempt ladder). Behavior changes to note: quota-failed swarm subagents now fail instead of suspending indefinitely as "Rate limited...", and quota errors cross the wire as provider.api_error rather than provider.rate_limit.
🦋 Changeset detectedLatest commit: 7ceabcc The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f585cbb4f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Responses response.failed / error SSE events carry no HTTP status and were minted by errorFromOpenAIResponsesEvent as either a rate-limit error (rate_limit_exceeded / embedded status_code=429) or a base ChatProviderError — and the base class falls into the retryable unclassified-failure fallback, so an insufficient_quota event still burned the whole retry budget on the openai_responses path. Route the event code and message through the same quota-exhausted check before the rate-limit branch, in kosong and the agent-core-v2 mirror. Covers all three entry paths (error events, response.failed, nested gateway frames) since they share the single converter.
…rule agent-core-v2 comments live solely in the top-of-file block, never beside functions or statements; the kosong twins keep the full rationale.
|
Pushed two follow-up commits after a local review pass:
|
|
Hi @wbxl2000 , PTAL 🙏 — this one bit us for real: our quota ran out and every prompt turned into what looked like a ~3-minute freeze. I attached the Node inspector to the Fixes: #1857 (fail fast on quota-exhausted 429) and #1861 (show retry progress in |
Related Issue
Part of #1860 — this PR fixes the engine-side fail-fast half. Companion UI PR: #1861 (live retry progress in the TUI), which closes the issue together with this one.
Problem
When an account's quota/balance is exhausted, Moonshot returns HTTP 429 with a structured body:
normalizeAPIStatusErrorclassifies every 429 as a retryableAPIProviderRateLimitError— the structurederror.type/error.codethat the OpenAI SDK parses onto itsAPIErroris dropped at theconvertOpenAIErrorseam. A quota-exhausted 429 can never succeed on retry, yet it burns the full retry budget: 10 attempts with 0.5→32s exponential backoff, ~3 minutes per turn, with no feedback in the interactive TUI. Users read it as a frozen app.Captured session log (kimi-code 0.26.0, provider
type = "kimi"againstapi.moonshot.cn, one turn ≈ 189s):The same account produced two different message wordings for the same
error.type("You exceeded your current token quota: … please check your account balance" earlier, the "suspended due to insufficient balance" variant later) — so message text alone is not a reliable discriminator, but the structured type is.What changed
APIProviderQuotaExhaustedError(extendsAPIStatusError; deliberately not a subclass ofAPIProviderRateLimitError), minted innormalizeAPIStatusError.convertOpenAIErrorforwards the SDK-parsed bodyerror.type/error.codevia a new trailing optional options parameter onnormalizeAPIStatusError(all other callers unchanged). Recognized values:exceeded_current_quota_error(Moonshot),insufficient_quota(OpenAI). Billing-anchored message patterns (both observed Moonshot wordings, OpenAI wording, arrears) are the fallback for gateways that flatten the body to text — deliberately no bare/quota/or/balance/patterns, so transient "token quota per minute" throttles keep retrying.isRetryableGenerateError(fails fast even when aretry-afterheader is present) and fromisProviderRateLimitError(no swarm requeue/suspend).toKimiErrorPayload(v1) andtranslateProviderError(v2) map it to the existingprovider.api_errorcode (retryable: false) instead ofprovider.rate_limit(retryable: true) — no new protocol code needed; the provider's actionable "please recharge" message is preserved on the payload.classifyApiErrorreportsquota_exhaustedin telemetry.agent-core-v2mirrors the same classification fix (llmProtocol/errors.ts,providers/openai-common.ts,protocol/errors.ts).loop_controlsection (en/zh) now states retries only apply to transient failures.Behavior notes for reviewers:
provider.api_error(retryable: false) rather thanprovider.rate_limit.rate_limit_reached_errorbody withretry-after: 1still walks the full 10-attempt ladder honoring the header. (The OpenAI SDK's own internalmaxRetries=2still applies beneath one logical attempt — pre-existing behavior, untouched.)Checklist
gen-changesetsskill, or this PR needs no changeset. (changeset added: patch for@moonshot-ai/kimi-codeand@moonshot-ai/kimi-code-sdk)gen-docsskill, or this PR needs no doc update. (en/zhconfig-files.mdupdated)