From cb70f8b9ee2a252219cfcccaa5d01e5328cfefa9 Mon Sep 17 00:00:00 2001 From: Janghoon Lee <44862514+savagemanage@users.noreply.github.com> Date: Tue, 22 Sep 2026 13:11:39 +0000 Subject: [PATCH] docs: correct three recommendations that research overturned Both harness documents told the next reader to do things that are wrong. Fixed with the evidence, rather than quietly reworded, so the reversals are auditable. 1. dynamicTools is no longer "the better fit" LOCAL-HARNESS-BACKENDS.md called Codex app-server's dynamicTools the better path for host-owned tools. Withdrawn. OpenAI's stability warning got BROADER in three months: the 2026-06-24 snapshot carried no production warning and scoped "experimental" to the WebSocket transport, while today's page covers the app-server command itself. app-server has no row at all in the Feature Maturity table. dynamicTools is double-experimental - gated behind capabilities.experimentalApi AND absent from the generated ThreadStartParams bindings, so the field must be hand-written against no type - and it has already changed wire shape once, leaving a LegacyDynamicToolSpec compat path where exposeToContext became the INVERTED deferLoading. There is no protocol version, no breaking-change log, and an open regression in the -c MCP path under app-server mode (openai/codex#39537). The counter-signal is recorded rather than buried: Zed's ACP adapter moved ONTO app-server and OpenAI's own IDE surfaces run on it, which is why this is "revisit in a quarter" and not "avoid". MCP cannot give us a tool living in our own process memory; when Office needs that, budget for re-porting dynamicTools at least once. 2. Driving the unmodified binary is not a safe harbour The doc said it was "the documented exception". Too confident. Anthropic began blocking third-party harnesses from Claude subscription billing on 2026-04-04 and the restriction is described as extending to all of them; OpenClaw is researching a --method cli path precisely because direct OAuth use has been blocked since then, and the same issue warns that spawning claude through the Agent SDK IS the pattern being restricted. Anthropic's June article still says subscription funds claude -p and the metering change is paused - the two together are a trajectory, not a guarantee. Consequences, both load-bearing: BYOK with the user's own key becomes the FIRST deliverable rather than the fallback, and Codex comes before Claude Code for business reasons rather than technical ones. 3. The /v1/providers routes I proposed already exist PROVIDER-AUTH.md proposed four new routes to expose the engine's login flow. They are already there as the server.integration group: GET /api/integration returns Integration.Info with a methods union (OAuthMethod | KeyMethod | EnvMethod) and a connections array, and connect/key, connect/oauth, the attempt status/complete/cancel routes and DELETE /api/credential complete it. Building /v1/providers would have been a duplicate surface over one credential store with two code paths to keep in agreement. A mapping table now records which existing route replaces each proposed one. So the engine side of BYOK is DONE and the real work is product-side - making an app use those routes instead of its own store. Office, Design, the extension, Query and Recall keep five stores that never read each other, which is the whole reason a user logs in again in every app; Cowork already does it right and is named as the pattern. Also folded in, from research rather than guesswork: no off-the-shelf wrapper is reusable (the only mature tools mapping is stale Rust, the popular one says in its README that function calling is unsupported, the third is GPL-3.0, and the 52.8k-star one never spawns the CLI - it stores the OAuth token and calls the vendor backend, which is exactly what April blocked). Two specifics ARE worth copying: pooled long-lived CLI processes, and launching with --tools "" --setting-sources "" --system-prompt , which strips ~28k tokens of the harness's own agent prompt and is the concrete mechanism behind the chat tier. And claude -p's stream-json schema is now mapped, so the old "schema unverified" note is replaced by the real field locations, plus the warning that its credential precedence puts the OAuth login LAST - ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, apiKeyHelper, CLAUDE_CODE_OAUTH_TOKEN and ANTHROPIC_PROFILE in the environment all silently take over billing, and --bare never reads the OAuth login at all. Same class of bug as the Codex env trap, opposite direction. --- docs/LOCAL-HARNESS-BACKENDS.md | 161 ++++++++++++++++++++++++++++----- docs/PROVIDER-AUTH.md | 45 +++++---- 2 files changed, 167 insertions(+), 39 deletions(-) diff --git a/docs/LOCAL-HARNESS-BACKENDS.md b/docs/LOCAL-HARNESS-BACKENDS.md index 0ffdaec883..4b8efde0f3 100644 --- a/docs/LOCAL-HARNESS-BACKENDS.md +++ b/docs/LOCAL-HARNESS-BACKENDS.md @@ -93,14 +93,96 @@ chat tier: - **No tool protocol** — query, recall, reblend. Chat tier is the whole story. - **Host-owned tools** — office, browser, canvas. Chat tier works today. Giving the - harness their tools needs MCP servers (`~/.codex/config.toml`, or per-invocation - `--config mcp_servers.…`) or Codex app-server's `dynamicTools`, which lets the - tool stay in the host process. `dynamicTools` is the better fit and is labelled - experimental by OpenAI, so it is not a foundation to build on yet. + harness their tools means a **stdio MCP server** (`~/.codex/config.toml`, a + project-scoped `.codex/config.toml`, or per-invocation + `--config mcp_servers.…`; Claude Code takes `--mcp-config` plus + `--strict-mcp-config`). Codex app-server's `dynamicTools` would let the tool stay + inside the host process instead, and an earlier draft of this document called that + "the better fit" — that recommendation is **withdrawn** on evidence: + + - OpenAI's stability warning got *broader*, not narrower. The + [2026-06-24 snapshot](http://web.archive.org/web/20260624142043/https://developers.openai.com/codex/app-server) + carried no production warning at all and scoped "experimental" to the WebSocket + transport; today's page says "The app-server command and WebSocket transport are + experimental and aren't supported for production workloads." + - app-server has **no row at all** in OpenAI's + [Feature Maturity](https://developers.openai.com/codex/feature-maturity) table. + - `dynamicTools` is double-experimental: gated behind + `capabilities.experimentalApi`, and **absent from the generated + `ThreadStartParams` bindings**, so the field you must send is one you hand-write + against no type. It has already changed wire shape once — `LegacyDynamicToolSpec` + exists as the compat scar, with `exposeToContext` replaced by the *inverted* + `deferLoading`. + - No protocol version, no breaking-change log, and an open regression in the `-c` + MCP path under app-server mode + ([openai/codex#39537](https://github.com/openai/codex/issues/39537)). + + The counter-signal is real and is why this is "revisit in a quarter" rather than + "avoid": Zed's ACP adapter *moved onto* app-server + ([agentclientprotocol/codex-acp](https://github.com/agentclientprotocol/codex-acp)), + and OpenAI's own IDE surfaces run on it. The one thing MCP cannot give us is a tool + whose implementation lives in our own process memory. When Office genuinely needs + that, budget for re-porting `dynamicTools` at least once. - **Engine-owned tools** — cowork, cad, extension, design. Under a harness backend the engine's own tools are not in the loop at all. That is the agent tier's problem to solve. +## The vendor risk is asymmetric, and it sets the order + +An earlier draft of this document said driving the unmodified binary is "the +documented exception". That was too confident, and the correction matters enough to +change sequencing. + +Anthropic began blocking third-party harnesses from Claude **subscription** billing +on **2026-04-04**, and the restriction is described as being extended to *all* +third-party harnesses +([claude-mem#1826](https://github.com/thedotmack/claude-mem/issues/1826), which links +the press coverage). OpenClaw is researching a `--method cli` path precisely because +direct OAuth token use has been blocked since then — so CLI-driving is the surviving +workaround, **not a safe harbour**. The same issue warns that spawning a `claude` +subprocess through the Agent SDK "is exactly the 'third-party harness' pattern +Anthropic is restricting". Anthropic's own June support article still says +subscription limits fund `claude -p` and third-party apps and that the metering +change is paused; treat the two together as a trajectory, not a guarantee. + +OpenAI points the other way: it publishes the integration surface as a platform, and +LiteLLM openly ships a `chatgpt/` provider for subscription access. + +Two consequences, both load-bearing: + +1. **BYOK with the user's own API key is the first deliverable, not the fallback.** + It is unaffected by any of the above, it is the path Anthropic and Google both + name as the supported one for a third-party tool, and Office's policy is already + open for it. +2. **Codex before Claude Code**, for business reasons rather than technical ones. + +## Nothing off the shelf is reusable + +Surveyed 2026-09-22. No existing project is production-credible for our requirement +— drive the local CLI *and* pass caller-supplied tools through: + +- [claude-code-api-rs](https://github.com/ZhangHanDong/claude-code-api-rs) (177★, + MIT) is the only mature native `tools` → `tool_calls` mapping, and it is Rust, + five months without a commit, and ships with + `use_interactive_sessions = false # Disabled by default due to stability issues`. +- [claude-code-openai-wrapper](https://github.com/RichardAtCT/claude-code-openai-wrapper) + (622★) states in its own README that function calling is not supported. +- [codingworkflow/claude-code-api](https://github.com/codingworkflow/claude-code-api) + (331★) is **GPL-3.0** — viral, so unusable in a shipped product. +- [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) (52.8k★) looks like the + obvious answer and is not: it never spawns the CLI. It performs the CLI's OAuth + flow itself, stores the token, and calls the vendor backend — the exact + architecture Anthropic blocked in April. +- LiteLLM, OpenRouter, Portkey, Helicone: no provider that drives a local Claude Code + or Codex CLI. + +Two specifics are worth copying rather than the repos: the **pooled long-lived CLI +process** model and the native tools mapping from claude-code-api-rs, and the trick of +launching with `--tools "" --setting-sources "" --system-prompt `, which +strips roughly 28k tokens of the harness's own agent prompt and built-in tools out of +every request. That second one is the concrete mechanism behind the chat tier — it is +what turns an agent harness into something that behaves like a completion endpoint. + ## What the products have to do Almost nothing, which is the point. @@ -114,33 +196,66 @@ into "unavailable" strands the user, because the remedy differs and we are not allowed to offer the vendor's login ourselves. The remedy we may show is "run `codex login`" or "run `claude`". +Both states are readable by ASKING the runtime, never by reading its credential +store: `claude auth status` exits 0 when signed in and 1 when not (its JSON field +names are undocumented, so the exit code is the contract), and Codex app-server +exposes `account/read`. A `claude -p` run also reports `apiKeySource` in its +`system`/`init` event. + Credentials need no work anywhere. The harness holds its own; the engine holds none for it. Cowork already demonstrates the pattern for the BYOK case — it stores no provider key and treats the engine's `auth.json` as the single source of truth. ## Sequencing -1. **Land `/v1/chat/completions`.** It is the receiving route for everything above - and it is not merged: the handler exists on `feature/v1-chat-completions`, and - `test:httpapi` fails without an `httpapi-exercise` scenario. Also needs the - `chatCompletions` capability flag, the SSE OpenAPI patch, and a route test. - Nothing here can ship before it. -2. **Build the shim with both backends.** One local chat-completions server; two +1. **BYOK with the user's own API key.** Moved to the front — see the vendor-risk + section. The engine side is **already built** — `GET /api/integration` returns + each integration's `methods` (OAuth / Key / Env) and its `connections`, and + `connect/key`, `connect/oauth`, the attempt routes and `DELETE /api/credential` + complete the set. An earlier draft of `docs/PROVIDER-AUTH.md` proposed a parallel + `/v1/providers` surface; that is withdrawn, because it would duplicate these over + the same store. The remaining work is product-side: make an app use them instead + of its own key store. Cowork already does (`apps/server/src/redrob-auth.ts`). +2. **Use the user's OWN binary.** The Codex SDK pins `@openai/codex` exactly and + resolves the executable from its own bundled platform packages unless + `codexPathOverride` is passed — so the default behaviour runs a SECOND copy we + downloaded, not the one the user signed in to. That re-creates the duplicate-engine + problem this whole effort exists to remove. Detect the user's install and pass the + path explicitly. Claude Code is proprietary with no redistribution grant, so it is + the only option there anyway, which makes both runtimes the same shape. +3. **Build the shim with both backends.** One local chat-completions server; two normalizers behind it. The prototype's split — event folding separated from the subprocess — is what makes the second runtime a second normalizer rather than a second architecture. Reuse it rather than re-deriving it. -3. **Register through the local-provider path**, and surface the backends in - `/v1/providers` with their three states so a product can render settings - without hardcoding a list. -4. **Verify against real binaries.** Neither runtime is installed on the build +4. **Give the harness our tools through a stdio MCP server.** Not app-server. Know + the defaults before wiring: Codex's `startup_timeout_sec` is 10, `tool_timeout_sec` + is 60, and `required = true` makes `codex exec` exit with an error rather than + silently running without our server. +5. **Register through the local-provider path**, and report the harness backends + through `GET /api/integration` alongside every other integration, with their + three states, so a product renders settings from one list rather than hardcoding + a vendor set that goes stale. +6. **Verify against real binaries.** Neither runtime is installed on the build host, so the live path — a real subscription actually paying for a turn — is unproven until someone runs it on a machine with `codex` and `claude` signed in. -5. **Then the agent tier**, for cowork, code and cad, over ACP. - -Two things to carry forward rather than discover later. `claude -p`'s -`stream-json` event schema has not been checked against the real binary — only the -flags are confirmed — so step 2 starts by reading it, not by assuming it mirrors -Codex's JSONL. And Anthropic has announced, then paused, a change that moves -third-party subscription usage onto a capped monthly credit; it currently still -draws from the subscription, but the trajectory is known, so the Claude backend -should surface usage state rather than assume it is free. +7. **Then the agent tier**, for cowork, code and cad, over ACP. + +One thing to carry forward rather than discover later: Anthropic has announced, then +paused, a change moving third-party subscription usage onto a capped monthly credit, +so the Claude backend should surface usage state rather than assume it is free. Note +also that `claude -p`'s credential precedence puts the OAuth login **last** — +`ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `apiKeyHelper`, `CLAUDE_CODE_OAUTH_TOKEN` +and `ANTHROPIC_PROFILE` in the environment all silently take over billing, and +`--bare` never reads the OAuth login at all. That is the same class of bug as the +Codex env trap, in the opposite direction: the shim must build the child environment +from an allow-list for both runtimes. + +`claude -p`'s `stream-json` schema IS now mapped, so step 3 does not start by +guessing: `system`/`init` carries `session_id`, `tools`, `mcp_servers` and +`apiKeySource`; assistant text and `tool_use` blocks are in +`assistant.message.content`; and the final `result` message carries +`subtype` (`success` or `error_*`), `is_error`, `total_cost_usd` and `modelUsage`. +Read cost from `modelUsage`, not `usage` — `usage` covers the main loop only and +undercounts subagents. Pin non-interactive behaviour with +`--permission-mode dontAsk --permission-prompts none`, and read-only with +`--tools "Read,Glob,Grep" --disallowedTools "Edit" "Write" "NotebookEdit" "Bash" "mcp__*"`. diff --git a/docs/PROVIDER-AUTH.md b/docs/PROVIDER-AUTH.md index e478ac6413..a0429839b2 100644 --- a/docs/PROVIDER-AUTH.md +++ b/docs/PROVIDER-AUTH.md @@ -123,22 +123,35 @@ key cannot leak one, log one, or sync one. `redrob providers login` already implements both shapes — a generic OAuth flow with `authorize()` plus `auto` and `code` callbacks, and an API-key path -(`packages/redrob/src/cli/cmd/providers.ts`). What is missing is not the flow but -its exposure: a product cannot drive it today. - -Add, on the v2 surface beside the completions route: - -- `GET /v1/providers` — what this engine can use, each entry declaring which auth - methods it accepts (`oauth`, `api-key`) and whether a credential is present. - This is what lets an app render a settings page without hardcoding a vendor - list that goes stale. -- `POST /v1/providers/:id/login` — starts a flow. For OAuth it returns the - authorization URL and an opaque attempt id; for an API key it accepts the key. -- `POST /v1/providers/:id/login/:attempt` — completes an OAuth attempt with the - authorization code, or reports that the loopback callback already completed it. -- `DELETE /v1/providers/:id/credential` — disconnect. - -Three rules on those routes: +(`packages/redrob/src/cli/cmd/providers.ts`). + +**An earlier draft of this section proposed four new `/v1/providers` routes to expose +it. That was wrong: the routes already exist.** They are the `server.integration` +group on the v2 surface, and they already carry everything a settings page needs. +Building the `/v1/providers` set would have been a duplicate surface over the same +credential store, with two code paths to keep in agreement. + +What exists, and what it replaces from that proposal: + +| proposed | already exists | +| --- | --- | +| `GET /v1/providers` | `GET /api/integration` — returns `Integration.Info`, whose `methods` array is a union of `OAuthMethod`, `KeyMethod` and `EnvMethod`, plus `connections` for what is already connected | +| `POST /v1/providers/:id/login` (key) | `POST /api/integration/:integrationID/connect/key` | +| `POST /v1/providers/:id/login` (oauth) | `POST /api/integration/:integrationID/connect/oauth` | +| `POST /v1/providers/:id/login/:attempt` | `POST /api/integration/attempt/:attemptID/complete`, with `GET /api/integration/attempt/:attemptID` for status and `DELETE` to cancel | +| `DELETE /v1/providers/:id/credential` | `DELETE /api/credential/:credentialID` | + +So the engine-side work for BYOK is **done**, and the real work is product-side: +making an app use these routes instead of its own store. Office keeps keys in +`userData/ai-settings.json`, Design in the macOS keychain, the extension in +`chrome.storage.local`, Query and Recall in separate keyring services — five stores +that never read each other, which is the whole reason a user logs in again in every +app. Cowork already does it the right way: it stores no provider key and posts the +key once to the engine (`apps/server/src/redrob-auth.ts`), treating the engine's +`auth.json` as the single source of truth. That is the pattern to copy. + +Three rules still hold for those routes, and they are worth restating because they +are what makes the single store safe: 1. **The key never comes back out.** A response says a credential is present and names it; it never returns the secret. A product that cannot read the key