Skip to content

feat(adapters): per-experiment endpoint routing #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gjtorikian merged 4 commits into from
Nov 10, 2026
Merged

feat(adapters): per-experiment endpoint routing #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3f2008e
feat(adapters): per-experiment endpoint routing
gjtorikian Nov 10, 2026
268f760
feat(claude): send a custom key as bearer too
gjtorikian Nov 10, 2026
ec9157f
fix(endpoint): identity, probes and roles follow the route
gjtorikian Nov 10, 2026
2b9e99c
Merge branch 'main' into feat/agent-endpoint-overrides
gjtorikian Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,50 @@ export OPENAI_CUSTOM_HEADERS='cf-aig-metadata: {"app":"my-evals"}'
Leave every override unset and both agents run vendor-direct against
api.anthropic.com and api.openai.com — nothing is rendered, nothing changes.

#### Per-experiment routing (`agent.endpoint`)

The variables above are per adapter: every codex arm in a run reads the
same `OPENAI_BASE_URL`. To route ONE experiment elsewhere — a second
gateway, or a gateway's unified OpenAI-compatible API serving a third-party
model — the experiment names the host variables that carry its key, base
URL and headers, and the host sets them next to the vendor ones:

```ts
export default defineExperiment({
id: "acme-gateway-mcp",
comparisonGroup: "acme-gateway",
treatment: "mcp",
agent: {
adapter: "codex",
provider: "acme", // recorded as-is in the manifest and report
model: "acme/coder-9", // whatever the endpoint expects in `model`
reasoning: "high",
endpoint: {
apiKeyEnv: "GATEWAY_API_TOKEN", // secret-named: redacted like a key
baseUrlEnv: "GATEWAY_BASE_URL", // not secret-named: identity + egress
headersEnv: "GATEWAY_HEADERS", // secret-named; `Name: Value` lines
wireApi: "chat", // codex only: chat completions
},
},
runtime: ...,
});
```

Unset fields keep the vendor defaults. The names must be POSIX env var
names; `defineExperiment` refuses a key or headers variable that is not
secret-named, a base URL variable that is, and `wireApi` on anything but
codex. The forwarded values travel exactly as before: the key as a provider
key, the base URL into the pair environment and the egress allowlist, the
headers redacted from artifacts. Codex renders them into its custom model
provider (`env_key`, `base_url`, `http_headers`, `wire_api`); Claude Code
only reads the vendor names, so the adapter re-exports custom ones under
them inside the container before exec — a custom key as both
`ANTHROPIC_API_KEY` (`x-api-key`) and `ANTHROPIC_AUTH_TOKEN`
(`Authorization: Bearer`), since a provider-native Anthropic route reads
the former and a gateway's unified API the latter. The preflight key probe
follows the experiment's variables, so a bad gateway token still fails
before any container starts.

## How it works

Every attempt is one point in a 2-D plan:
Expand Down
86 changes: 74 additions & 12 deletions src/adapters/claude.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
* across experiments.
*/

import { type ResolvedEndpoint, resolveEndpoint } from "../experiments/endpoint.ts";
import type { AgentEndpointSpec } from "../experiments/types.ts";
import type { McpServerConfig } from "../isolation/mcp.ts";
import { endpointUrl, parseHeaderLines } from "../secrets.ts";
import { parseClaudeStream } from "./parse.ts";
Expand All @@ -21,6 +23,18 @@ import type { AttemptMetrics, ContainerAgentAdapter, ContainerAttemptContext } f
/** Container path of the MCP config Claude Code auto-discovers. */
export const CLAUDE_MCP_CONFIG_PATH = "/workspace/.mcp.json";

/**
* The vendor's env names: what every claude experiment gets unless its
* `agent.endpoint` names something else (experiments/endpoint.ts). The wire
* is fixed — Claude Code speaks the Messages API and nothing else.
*/
export const CLAUDE_DEFAULT_ENDPOINT: ResolvedEndpoint = {
apiKeyEnv: "ANTHROPIC_API_KEY",
baseUrlEnv: "ANTHROPIC_BASE_URL",
headersEnv: "ANTHROPIC_CUSTOM_HEADERS",
wireApi: "responses",
};

/**
* Project-scope `.mcp.json` rendered from the runtime's MCP servers
* (streamable HTTP). The configs arrive credential-free — authed servers
Expand All @@ -38,17 +52,26 @@ export class ClaudeCodeAdapter implements ContainerAgentAdapter {
readonly name: string;
readonly identity;
readonly versionCommand = ["claude", "--version"] as const;
readonly requiredHostEnv = ["ANTHROPIC_API_KEY"] as const;
/**
* Endpoint overrides, forwarded only when set. ANTHROPIC_BASE_URL routes
* The env names this experiment's traffic is routed by: the vendor's
* (ANTHROPIC_API_KEY / ANTHROPIC_BASE_URL / ANTHROPIC_CUSTOM_HEADERS)
* unless `agent.endpoint` named others. Claude Code only ever reads the
* vendor names, so custom names are re-exported under them by a shell shim
* in `command` — inside the container, from env the harness already placed.
*/
readonly endpoint: ResolvedEndpoint;
/** The key variable: required on the host, forwarded into the container. */
readonly requiredHostEnv: readonly string[];
/**
* Endpoint overrides, forwarded only when set. The base URL variable routes
* the CLI (and the key probe) through an Anthropic-compatible gateway —
* e.g. Cloudflare AI Gateway's `/anthropic` endpoint, where the gateway
* token doubles as ANTHROPIC_API_KEY; ANTHROPIC_CUSTOM_HEADERS (`Name:
* Value` lines) carries whatever else that gateway wants, such as
* token doubles as the API key; the headers variable (`Name: Value`
* lines) carries whatever else that gateway wants, such as
* `cf-aig-authorization` or `cf-aig-metadata`. Claude Code reads both
* natively, so no flag changes.
* natively under the vendor names, so no flag changes.
*/
readonly optionalHostEnv = ["ANTHROPIC_BASE_URL", "ANTHROPIC_CUSTOM_HEADERS"] as const;
readonly optionalHostEnv: readonly string[];
/**
* Egress-proxy allowlist (harness-default layer). Claude Code documents
* corporate-proxy support via HTTPS_PROXY/HTTP_PROXY; api.anthropic.com is
Expand All @@ -59,10 +82,16 @@ export class ClaudeCodeAdapter implements ContainerAgentAdapter {
*/
readonly egressHosts = ["api.anthropic.com", "statsig.anthropic.com", "*.sentry.io"] as const;

constructor(name = "claude", config: { model?: string; reasoning?: string } = {}) {
constructor(
name = "claude",
config: { model?: string; reasoning?: string; provider?: string; endpoint?: AgentEndpointSpec } = {},
) {
this.name = name;
this.endpoint = resolveEndpoint(name, CLAUDE_DEFAULT_ENDPOINT, config.endpoint);
this.requiredHostEnv = [this.endpoint.apiKeyEnv];
this.optionalHostEnv = [this.endpoint.baseUrlEnv, this.endpoint.headersEnv];
this.identity = {
provider: "anthropic",
provider: config.provider ?? "anthropic",
model: config.model ?? "UNPINNED",
reasoning: config.reasoning ?? "UNPINNED",
cliVersion: "unknown",
Expand All @@ -78,15 +107,43 @@ export class ClaudeCodeAdapter implements ContainerAgentAdapter {

keyProbe(env: Record<string, string>): { url: string; headers: Record<string, string> } {
return {
url: endpointUrl(env["ANTHROPIC_BASE_URL"], "https://api.anthropic.com", "v1/models"),
url: endpointUrl(env[this.endpoint.baseUrlEnv], "https://api.anthropic.com", "v1/models"),
headers: {
...parseHeaderLines(env["ANTHROPIC_CUSTOM_HEADERS"]),
"x-api-key": env["ANTHROPIC_API_KEY"] ?? "",
...parseHeaderLines(env[this.endpoint.headersEnv]),
"x-api-key": env[this.endpoint.apiKeyEnv] ?? "",
"anthropic-version": "2023-06-01",
Comment on lines +112 to 114

@greptile-apps greptile-apps Bot Nov 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Valid gateway tokens get rejected

If a gateway requires bearer auth, Claude’s keyProbe rejects a valid custom key unless the user also puts an Authorization header in headersEnv. The probe sends the key only as x-api-key, while the new wrapper also sets ANTHROPIC_AUTH_TOKEN. A 401/403 from the probe then stops the whole run before Claude starts.

Make the probe send the same bearer credential as the custom-key wrapper, while keeping vendor-default behavior unchanged.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: src/adapters/claude.ts
Line: 112-114

Comment:
**Valid gateway tokens get rejected**

If a gateway requires bearer auth, Claude’s `keyProbe` rejects a valid custom key unless the user also puts an `Authorization` header in `headersEnv`. The probe sends the key only as `x-api-key`, while the new wrapper also sets `ANTHROPIC_AUTH_TOKEN`. A 401/403 from the probe then stops the whole run before Claude starts.

Make the probe send the same bearer credential as the custom-key wrapper, while keeping vendor-default behavior unchanged.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

},
};
}

/**
* `export VENDOR="$CUSTOM"` lines for every endpoint variable whose name
* differs from the vendor's — empty when nothing differs, so the default
* invocation stays the bare `claude` argv. Optional variables are only
* re-exported when set: an empty ANTHROPIC_BASE_URL would be a base URL.
*
* A custom key variable is presented both ways Claude Code can send a
* credential — `x-api-key` (ANTHROPIC_API_KEY) and `Authorization:
* Bearer` (ANTHROPIC_AUTH_TOKEN) — because gateways differ on which one
* they read: a provider-native Anthropic route takes the key header, a
* unified API takes the bearer. The vendor's own route only ever sees
* the vendor variable, so this changes nothing for it.
*/
private endpointExports(): string[] {
const lines: string[] = [];
const { apiKeyEnv, baseUrlEnv, headersEnv } = this.endpoint;
if (apiKeyEnv !== CLAUDE_DEFAULT_ENDPOINT.apiKeyEnv) {
lines.push(`export ANTHROPIC_API_KEY="$${apiKeyEnv}"`, `export ANTHROPIC_AUTH_TOKEN="$${apiKeyEnv}"`);
}
if (baseUrlEnv !== CLAUDE_DEFAULT_ENDPOINT.baseUrlEnv) {
lines.push(`[ -n "\${${baseUrlEnv}:-}" ] && export ANTHROPIC_BASE_URL="$${baseUrlEnv}"`);
}
if (headersEnv !== CLAUDE_DEFAULT_ENDPOINT.headersEnv) {
lines.push(`[ -n "\${${headersEnv}:-}" ] && export ANTHROPIC_CUSTOM_HEADERS="$${headersEnv}"`);
}
return lines;
}

setupFiles(ctx: ContainerAttemptContext): Record<string, string> {
if (ctx.mcpServers === undefined) return {};
return { [CLAUDE_MCP_CONFIG_PATH]: renderMcpJson(ctx.mcpServers) };
Expand Down Expand Up @@ -114,7 +171,12 @@ export class ClaudeCodeAdapter implements ContainerAgentAdapter {
args.push("--mcp-config", CLAUDE_MCP_CONFIG_PATH);
}
if (ctx.webPolicy !== "native-web-allowed") args.push("--disallowedTools", "WebSearch,WebFetch");
return args;
const exports = this.endpointExports();
if (exports.length === 0) return args;
// Custom endpoint names: re-export them under the names Claude Code reads,
// then exec the same argv. Names only reach the shell — resolveEndpoint
// admits POSIX env names and nothing else; values stay in the environment.
return ["sh", "-c", `${exports.join("; ")}; exec "$@"`, "sh", ...args];
}

parseStream(raw: string): AttemptMetrics {
Expand Down
100 changes: 74 additions & 26 deletions src/adapters/codex.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
* talks to api.openai.com — vendor-direct stays the default.
*/

import { type ResolvedEndpoint, type WireApi, resolveEndpoint } from "../experiments/endpoint.ts";
import type { AgentEndpointSpec } from "../experiments/types.ts";
import type { McpServerConfig } from "../isolation/mcp.ts";
import { endpointUrl, parseHeaderLines } from "../secrets.ts";
import { parseCodexStream } from "./parse.ts";
Expand All @@ -40,22 +42,40 @@ export const CODEX_GATEWAY_PROVIDER = "quickstudy-gateway";
const OPENAI_DEFAULT_BASE_URL = "https://api.openai.com/v1";

/**
* A custom model provider pointing at `baseUrl` (the Responses API, which
* codex requires), carrying `headers` on every request, selected as the
* default. Top-level keys must precede any table, so this block always
* renders first in the file. JSON string escapes are valid TOML basic-string
* escapes, which keeps header values such as JSON metadata intact.
* The vendor's env names and wire: what every codex experiment gets unless
* its `agent.endpoint` names something else (experiments/endpoint.ts).
*/
function renderCodexProviderToml(baseUrl: string, headers: Readonly<Record<string, string>>): string {
export const CODEX_DEFAULT_ENDPOINT: ResolvedEndpoint = {
apiKeyEnv: "OPENAI_API_KEY",
baseUrlEnv: "OPENAI_BASE_URL",
headersEnv: "OPENAI_CUSTOM_HEADERS",
wireApi: "responses",
};

/**
* A custom model provider pointing at `baseUrl`, speaking `wireApi` (the
* Responses API codex prefers, or chat completions for endpoints that only
* offer that), reading its key from `envKey`, carrying `headers` on every
* request, selected as the default. Top-level keys must precede any table,
* so this block always renders first in the file. JSON string escapes are
* valid TOML basic-string escapes, which keeps header values such as JSON
* metadata intact.
*/
function renderCodexProviderToml(
baseUrl: string,
headers: Readonly<Record<string, string>>,
envKey: string,
wireApi: WireApi,
): string {
const entries = Object.entries(headers);
return [
`model_provider = ${JSON.stringify(CODEX_GATEWAY_PROVIDER)}`,
"",
`[model_providers.${CODEX_GATEWAY_PROVIDER}]`,
'name = "OpenAI (quickstudy endpoint overrides)"',
`base_url = ${JSON.stringify(baseUrl)}`,
'env_key = "OPENAI_API_KEY"',
'wire_api = "responses"',
`env_key = ${JSON.stringify(envKey)}`,
`wire_api = ${JSON.stringify(wireApi)}`,
...(entries.length > 0
? [`http_headers = { ${entries.map(([name, value]) => `${JSON.stringify(name)} = ${JSON.stringify(value)}`).join(", ")} }`]
: []),
Expand All @@ -81,17 +101,25 @@ export class CodexAdapter implements ContainerAgentAdapter {
readonly name: string;
readonly identity;
readonly versionCommand = ["codex", "--version"] as const;
readonly requiredHostEnv = ["OPENAI_API_KEY"] as const;
/**
* Endpoint overrides, forwarded only when set: OPENAI_BASE_URL (vendor
* default https://api.openai.com/v1) — e.g. Cloudflare AI Gateway's
* `/openai` endpoint, where the gateway token doubles as OPENAI_API_KEY —
* and OPENAI_CUSTOM_HEADERS (`Name: Value` lines) for whatever else the
* The env names this experiment's traffic is routed by: the vendor's
* (OPENAI_API_KEY / OPENAI_BASE_URL / OPENAI_CUSTOM_HEADERS) unless
* `agent.endpoint` named others — the way two codex arms in one run reach
* two gateways, or one gateway's unified API serving a third-party model.
*/
readonly endpoint: ResolvedEndpoint;
/** The key variable: required on the host, forwarded into the container. */
readonly requiredHostEnv: readonly string[];
/**
* Endpoint overrides, forwarded only when set: the base URL variable
* (vendor default https://api.openai.com/v1) — e.g. Cloudflare AI Gateway's
* `/openai` endpoint, where the gateway token doubles as the API key —
* and the headers variable (`Name: Value` lines) for whatever else the
* endpoint wants on each request, such as `cf-aig-metadata`. Both are
* rendered into config.toml by setupFiles (see the module comment); the
* key probe follows them.
*/
readonly optionalHostEnv = ["OPENAI_BASE_URL", "OPENAI_CUSTOM_HEADERS"] as const;
readonly optionalHostEnv: readonly string[];
/**
* Egress-proxy allowlist (harness-default layer). Codex is reqwest-based;
* reqwest honors HTTP_PROXY/HTTPS_PROXY env by default (system proxies) —
Expand All @@ -103,10 +131,16 @@ export class CodexAdapter implements ContainerAgentAdapter {
*/
readonly egressHosts = ["api.openai.com", "auth.openai.com", "chatgpt.com", "*.chatgpt.com"] as const;

constructor(name = "codex", config: { model?: string; reasoning?: string } = {}) {
constructor(
name = "codex",
config: { model?: string; reasoning?: string; provider?: string; endpoint?: AgentEndpointSpec } = {},
) {
this.name = name;
this.endpoint = resolveEndpoint(name, CODEX_DEFAULT_ENDPOINT, config.endpoint);
this.requiredHostEnv = [this.endpoint.apiKeyEnv];
this.optionalHostEnv = [this.endpoint.baseUrlEnv, this.endpoint.headersEnv];
this.identity = {
provider: "openai",
provider: config.provider ?? "openai",
model: config.model ?? "UNPINNED",
reasoning: config.reasoning ?? "UNPINNED",
cliVersion: "unknown",
Expand All @@ -122,18 +156,30 @@ export class CodexAdapter implements ContainerAgentAdapter {

keyProbe(env: Record<string, string>): { url: string; headers: Record<string, string> } {
return {
url: endpointUrl(env["OPENAI_BASE_URL"], OPENAI_DEFAULT_BASE_URL, "models"),
headers: { ...parseHeaderLines(env["OPENAI_CUSTOM_HEADERS"]), Authorization: `Bearer ${env["OPENAI_API_KEY"] ?? ""}` },
url: endpointUrl(env[this.endpoint.baseUrlEnv], OPENAI_DEFAULT_BASE_URL, "models"),
headers: {
...parseHeaderLines(env[this.endpoint.headersEnv]),
Authorization: `Bearer ${env[this.endpoint.apiKeyEnv] ?? ""}`,
},
};
}

setupFiles(ctx: ContainerAttemptContext): Record<string, string> {
const sections: string[] = [];
const baseUrl = ctx.overrides?.["OPENAI_BASE_URL"];
const headers = parseHeaderLines(ctx.overrides?.["OPENAI_CUSTOM_HEADERS"]);
// Either override needs the custom provider; headers alone keep the vendor endpoint.
if (baseUrl !== undefined || Object.keys(headers).length > 0) {
sections.push(renderCodexProviderToml(baseUrl ?? OPENAI_DEFAULT_BASE_URL, headers));
const baseUrl = ctx.overrides?.[this.endpoint.baseUrlEnv];
const headers = parseHeaderLines(ctx.overrides?.[this.endpoint.headersEnv]);
// Either override needs the custom provider (headers alone keep the
// vendor endpoint); so does a non-vendor key variable or wire, which the
// built-in provider cannot express.
const custom =
baseUrl !== undefined ||
Object.keys(headers).length > 0 ||
this.endpoint.apiKeyEnv !== CODEX_DEFAULT_ENDPOINT.apiKeyEnv ||
this.endpoint.wireApi !== CODEX_DEFAULT_ENDPOINT.wireApi;
if (custom) {
sections.push(
renderCodexProviderToml(baseUrl ?? OPENAI_DEFAULT_BASE_URL, headers, this.endpoint.apiKeyEnv, this.endpoint.wireApi),
);
}
if (ctx.mcpServers !== undefined) sections.push(renderCodexMcpToml(ctx.mcpServers));
if (sections.length === 0) return {};
Expand All @@ -145,9 +191,11 @@ export class CodexAdapter implements ContainerAgentAdapter {
"sh",
"-c",
// codex 0.144+ ignores the OPENAI_API_KEY env var; mint auth.json from
// it in-container. Login chatter is silenced so stdout stays pure JSONL
// for parseStream; the key reaches login via stdin, never argv.
'printenv OPENAI_API_KEY | codex login --with-api-key >/dev/null 2>&1 && exec "$@"',
// the key variable in-container. Login chatter is silenced so stdout
// stays pure JSONL for parseStream; the key reaches login via stdin,
// never argv. The variable NAME is interpolated — resolveEndpoint
// admits only POSIX env names, so nothing else can reach the shell.
`printenv ${this.endpoint.apiKeyEnv} | codex login --with-api-key >/dev/null 2>&1 && exec "$@"`,
"sh",
"codex",
"exec",
Expand Down
Loading
Loading