Skip to content

feat: add configurable tokens-per-minute rate limiting - #1131

Merged
avoidwork merged 6 commits into
mainfrom
feat/configurable-tokens-per-minute-rate-limiting
Sep 21, 2026
Merged

avoidwork merged 6 commits into
mainfrom
feat/configurable-tokens-per-minute-rate-limiting

Conversation

@avoidwork

@avoidwork avoidwork commented Sep 21, 2026

Copy link
Copy Markdown
Owner

Description

Adds a configurable maxTokensMinute setting to the provider rate-limit config that paces outgoing LLM requests against a rolling 60-second tokens-per-minute budget. Defaults to 0 (disabled) so the throttle is opt-in. When enabled, the harness estimates request token cost (input via calculateConversationTokens + maxTokens output budget), waits for capacity in the rolling window, and consumes the estimated cost before dispatching.

On a 429 rate-limit response, the request is retried once after the retry-after header value (seconds or HTTP-date), defaulting to 60 seconds when no header is present. This retry only applies when maxTokensMinute is enabled; when it is 0, no retry occurs and the error surfaces immediately. 429 errors are attributed to an exceeded token budget when the rolling window exceeds maxTokensMinute.

Type of Change

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor (no functional changes)
  • Performance improvement
  • CI / build / tooling

Testing

Unit tests in tests/unit/provider/tokenBudget.test.js (11 tests) cover: window eviction after 60s, disabled path (maxTokensMinute: 0), exactly-at-limit, single request larger than the whole budget, and rapid successive requests. tests/unit/provider.test.js covers the throttle-wiring (enabled/disabled paths), 429 attribution, the single 429 retry, and the getRetryDelayMs helper (retry-after parsing + 60s default). Verified with npm run test (3860 passing), npm run lint (0 errors), and npm run coverage (tokenBudget.js 100% statements).

Coverage

  • Line coverage maintained

Checklist

  • npm run lint passes
  • Tests pass with maintained line coverage
  • No forbidden patterns used
  • Conventional Commit style applied

Closes #1130

Add OpenSpec change for configurable maxTokensMinute rate limiting:
- RateLimitSchema gains maxTokensMinute (non-negative int, default 0)
- New provider-token-budget capability for rolling 60s token window
- Wire throttle into createChatModel dispatch
- Attribute 429 errors to exceeded token budget
@avoidwork avoidwork self-assigned this Sep 21, 2026
Add maxTokensMinute to RateLimitSchema (non-negative int, default 0).
Create rolling 60-second token budget module (src/provider/tokenBudget.js)
with consume, current, and waitForCapacity. Wire the throttle into
createChatModel dispatch: estimate request cost (input + maxTokens),
wait for capacity, then consume before invoking. Attribute 429 errors
to an exceeded token budget when the rolling window exceeds the limit.
Document maxTokensMinute example in config.yaml. Add unit tests for the
token budget and throttle wiring.
Archive the OpenSpec change and sync provider-token-budget and
provider-rate-limit-config specs into openspec/specs/.
@avoidwork

Copy link
Copy Markdown
Owner Author

Implementation Audit Results — configurable-tokens-per-minute-rate-limiting

Audit Summary

Iteration: 1
Status: PASS — all goals and spec requirements satisfied.

Goal Fulfillment

# Goal Status Evidence
1 Add maxTokensMinute to RateLimitSchema (non-negative int, default 0) src/config/schemas/providers.jsmaxTokensMinute: z.number().int().min(0).default(0)
2 Build rolling 60-second token window module src/provider/tokenBudget.jsconsume, current, waitForCapacity
3 Estimate request token cost (input + maxTokens) estimateRequestCost() in src/provider/openai.js reuses calculateConversationTokens
4 Wire throttle into dispatch createChatModel() wraps invoke/stream when maxTokensMinute > 0
5 Attribute 429 errors to exceeded token budget isRateLimitError() + budget.current() > maxTokensMinutelogger.warn
6 Write unit tests tests/unit/provider/tokenBudget.test.js (11 tests) + provider.test.js throttle tests (3)
7 Verify test/lint/coverage 3741 tests pass, 0 lint errors, coverage generated

Spec Compliance

  • provider-token-budget (new): All requirements met — rolling window tracks consumed tokens, evicts >60s entries, waitForCapacity resolves when room available / waits when near capacity / handles request larger than whole budget, disabled when maxTokensMinute is 0.
  • provider-rate-limit-config (modified): maxTokensMinute field added with default 0, accepts 0/positive, rejects negative/non-integer. createChatModel wires throttle when positive, disabled when 0. 429 errors attributed to exceeded token budget.

Task Completion

All 18 tasks marked [x] in tasks.md.

Quality Check

  • No forbidden patterns (no console.log, no silent catch, no eval).
  • JSDoc present on all public functions.
  • 2-space indentation, line length ≤ 100 (oxfmt verified).
  • No dead code introduced.
  • config.yaml documents maxTokensMinute: 100000 example.

Verification Results

  • npm run test: 3741 tests pass, 0 fail.
  • npm run lint: 0 warnings, 0 errors, formatting correct.
  • npm run coverage: Report generated. tokenBudget.js 100% statements, openai.js 89.41% statements (remaining uncovered lines are pre-existing vLLM reasoning normalization paths and the 429-attribution log branch, not part of this change's core scope). Overall all files 92.47% statements.

Notes

  • npm start (interactive mode) fails with an Ink "Raw mode is not supported" error in the non-interactive sandbox — this is an environment/TTY limitation, not a code defect. Verified the app starts correctly in --mode chat, which exercises config loading, subsystem init, and model creation (including throttle wiring).

@avoidwork
avoidwork merged commit 8a2a5bc into main Sep 21, 2026
2 checks passed
@avoidwork
avoidwork deleted the feat/configurable-tokens-per-minute-rate-limiting branch September 21, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add configurable tokens-per-minute rate limiting

1 participant