Skip to content

feat: add TokenPoolAuthenticator for multi-token rotation in declarative CDK#1057

Draft
Aaron ("AJ") Steers (aaronsteers) wants to merge 4 commits into
mainfrom
devin/1750005100-token-pool-authenticator
Draft

feat: add TokenPoolAuthenticator for multi-token rotation in declarative CDK#1057
Aaron ("AJ") Steers (aaronsteers) wants to merge 4 commits into
mainfrom
devin/1750005100-token-pool-authenticator

Conversation

@aaronsteers

Copy link
Copy Markdown
Member

Summary

Adds TokenPoolAuthenticator — a new declarative YAML component that accepts multiple API tokens and rotates through them to distribute rate-limit consumption. This generalizes source-github's bespoke MultipleTokenAuthenticatorWithRateLimiter (~273 lines of custom Python) into a reusable declarative component any connector can use via YAML.

Architecture: Two-layer composition:

  • TokenPoolAuthenticator — composite DeclarativeAuthenticator that parses a delimited token string, delegates to a pluggable rotation strategy, and implements auth_header/token properties.
  • Rotation strategies — RoundRobinRotation (simple cycling) and RateLimitAwareRotation (tracks per-token quota from response headers, rotates when exhausted, sleeps with heartbeat when all tokens are depleted).

Response feedback hook: HttpRequester.send_request() now calls authenticator.on_http_response(response) after each response, allowing RateLimitAwareRotation to update per-token state from rate-limit headers (mirrors how HTTPAPIBudget.update_from_response() already works).

Example YAML usage (GitHub-style):

authenticator:
  type: TokenPoolAuthenticator
  tokens: "{{ config['credentials']['personal_access_token'] }}"
  token_separator: ","
  auth_method: "token"
  rotation_strategy:
    type: RateLimitAwareRotation
    ratelimit_remaining_header: "x-ratelimit-remaining"
    ratelimit_reset_header: "x-ratelimit-reset"
    max_wait_seconds: 7200

Requested by Aaron ("AJ") Steers (@aaronsteers).

Link to Devin session: https://app.devin.ai/sessions/a66b7515fdb64847942183c847c96b04

…ive CDK

Introduces TokenPoolAuthenticator - a new declarative YAML component that
accepts multiple API tokens and rotates through them automatically to
distribute rate-limit consumption across credentials.

Components added:
- TokenPoolAuthenticator: composite authenticator accepting multiple tokens
- RoundRobinRotation: cycles through tokens sequentially
- RateLimitAwareRotation: tracks per-token quota from response headers,
  rotates when exhausted, sleeps when all tokens are depleted
- on_http_response hook in HttpRequester to feed rate-limit state back

Modeled after source-github's MultipleTokenAuthenticatorWithRateLimiter
to enable the same pattern for all declarative connectors via YAML config.

Co-Authored-By: AJ Steers <aj@airbyte.io>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1750005100-token-pool-authenticator#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1750005100-token-pool-authenticator

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

Co-Authored-By: AJ Steers <aj@airbyte.io>
Comment thread airbyte_cdk/sources/declarative/auth/token_rotation_strategies.py Fixed
Comment thread airbyte_cdk/sources/declarative/auth/token_rotation_strategies.py Fixed
Comment thread airbyte_cdk/sources/declarative/auth/token_rotation_strategies.py Fixed
devin-ai-integration Bot and others added 2 commits June 22, 2026 16:32
…tedError in abstract method

Co-Authored-By: AJ Steers <aj@airbyte.io>
@github-actions

Copy link
Copy Markdown

PyTest Results (Fast)

4 130 tests  +11   4 119 ✅ +11   8m 30s ⏱️ +14s
    1 suites ± 0      11 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 6c6aa0c. ± Comparison against base commit 977f051.

@github-actions

Copy link
Copy Markdown

PyTest Results (Full)

4 133 tests  +11   4 121 ✅ +11   12m 2s ⏱️ +26s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 6c6aa0c. ± Comparison against base commit 977f051.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants