Repository navigation
feat(adapters): per-experiment endpoint routing - #45
Merged
Merged
Conversation
An adapter's endpoint overrides are per sweep: every codex arm reads the same OPENAI_BASE_URL, so two codex arms could not reach two endpoints in one run. A benchmark that drives OpenAI's model and third-party models through a gateway's unified API needs exactly that, a different base URL and token than the provider-native route, per experiment, without the adapter growing a vendor table. `agent.endpoint` names the host variables an experiment's key, base URL and headers come from; unset fields keep the vendor defaults. Naming variables rather than values keeps every existing guarantee: the key variable must be secret-named so it is redacted, the base URL variable must not be so its value stays identity and joins the egress allowlist, and the key probe follows the experiment's variables so a bad gateway token still fails before any container. Codex renders the names into its custom provider; Claude Code only reads the vendor names, so a shell shim re-exports custom ones under them in the container, and the default argv is untouched.
 [Medium impact] The PR changes provider routing for configured Codex and Claude Code experiments. Fix Claude’s bearer-auth probe before merging so valid gateway tokens do not stop the run. Findings
Fix with agent prompt### Issue 1
src/adapters/claude.ts:112-114
If a gateway requires bearer auth, Claude’s `keyProbe` rejects a valid custom key unless the user also puts an `Authorization` header in `headersEnv`. The probe sends the key only as `x-api-key`, while the new wrapper also sets `ANTHROPIC_AUTH_TOKEN`. A 401/403 from the probe then stops the whole run before Claude starts.
Make the probe send the same bearer credential as the custom-key wrapper, while keeping vendor-default behavior unchanged.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryAdds per-experiment endpoint routing through
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Gateways differ on which credential header they read. A provider-native Anthropic route takes `x-api-key`, which Claude Code sends from ANTHROPIC_API_KEY; a gateway's unified API takes `Authorization: Bearer`, which Claude Code only sends from ANTHROPIC_AUTH_TOKEN. The shim that re-exports a custom key variable now sets both, so one experiment works against either route without a second knob. The vendor route still sees only the vendor variable.
Three gaps the review found in per-experiment endpoint routing: The manifest recorded only adapter, provider, model, reasoning and CLI version, and comparison groups diff those fields, so two arms that differed only in wire or in the variables they read compared as one configuration. The declared endpoint is now an agent identity component and part of the group diff; an experiment without one records nothing new and keeps its prior identity. Key probes were deduplicated by adapter name, so of two codex arms on different gateways only the first token was checked and a rejected second token started containers. Probes now deduplicate by adapter plus the host variables it reads. A declared name could be another role's vendor variable — a key read from ANTHROPIC_CUSTOM_HEADERS and the headers from ANTHROPIC_API_KEY — which the Claude shim would read and then overwrite. Such names are refused, as are two roles resolving to one name.
Comment on lines
+112
to
114
| ...parseHeaderLines(env[this.endpoint.headersEnv]), | ||
| "x-api-key": env[this.endpoint.apiKeyEnv] ?? "", | ||
| "anthropic-version": "2023-06-01", |
There was a problem hiding this comment.
If a gateway requires bearer auth, Claude’s keyProbe rejects a valid custom key unless the user also puts an Authorization header in headersEnv. The probe sends the key only as x-api-key, while the new wrapper also sets ANTHROPIC_AUTH_TOKEN. A 401/403 from the probe then stops the whole run before Claude starts.
Make the probe send the same bearer credential as the custom-key wrapper, while keeping vendor-default behavior unchanged.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/adapters/claude.ts
Line: 112-114
Comment:
**Valid gateway tokens get rejected**
If a gateway requires bearer auth, Claude’s `keyProbe` rejects a valid custom key unless the user also puts an `Authorization` header in `headersEnv`. The probe sends the key only as `x-api-key`, while the new wrapper also sets `ANTHROPIC_AUTH_TOKEN`. A 401/403 from the probe then stops the whole run before Claude starts.
Make the probe send the same bearer credential as the custom-key wrapper, while keeping vendor-default behavior unchanged.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
codexarm reads the sameOPENAI_BASE_URL, so twocodexarms could not reach two endpoints in one run. The benchmark now drives OpenAI's model and third-party models (DeepSeek, Kimi, Gemini) through Cloudflare AI Gateway's unified API, which is a different base URL and token than the provider-native route, so the route has to be per experiment.agent.endpointonAgentSpecnames the host variables an experiment's key, base URL and headers come from (apiKeyEnv,baseUrlEnv,headersEnv), pluswireApi(responses|chat) for codex's custom provider. Unset fields keep the vendor defaults, so existing experiments are unchanged.defineExperimentrefuses anything else, includingwireApion a non-codex adapter.env_key,wire_api) and the login shim pipes the named key variable. Claude Code only reads the vendor names, so a shell shim re-exports custom ones under them inside the container; with vendor names the argv is byte-identical to before.agent.providernow reaches the adapter identity instead of being hard-coded to the vendor, so acodex-drivendeepseek/…arm is recorded as what it is.