Skip to content

docs: correct three recommendations that research overturned #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
its-janghoon wants to merge 1 commit into from
Closed

docs: correct three recommendations that research overturned #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 138 additions & 23 deletions docs/LOCAL-HARNESS-BACKENDS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,14 +93,96 @@ chat tier:

- **No tool protocol** — query, recall, reblend. Chat tier is the whole story.
- **Host-owned tools** — office, browser, canvas. Chat tier works today. Giving the
harness their tools needs MCP servers (`~/.codex/config.toml`, or per-invocation
`--config mcp_servers.…`) or Codex app-server's `dynamicTools`, which lets the
tool stay in the host process. `dynamicTools` is the better fit and is labelled
experimental by OpenAI, so it is not a foundation to build on yet.
harness their tools means a **stdio MCP server** (`~/.codex/config.toml`, a
project-scoped `.codex/config.toml`, or per-invocation
`--config mcp_servers.…`; Claude Code takes `--mcp-config` plus
`--strict-mcp-config`). Codex app-server's `dynamicTools` would let the tool stay
inside the host process instead, and an earlier draft of this document called that
"the better fit" — that recommendation is **withdrawn** on evidence:

- OpenAI's stability warning got *broader*, not narrower. The
[2026-06-24 snapshot](http://web.archive.org/web/20260624142043/https://developers.openai.com/codex/app-server)
carried no production warning at all and scoped "experimental" to the WebSocket
transport; today's page says "The app-server command and WebSocket transport are
experimental and aren't supported for production workloads."
- app-server has **no row at all** in OpenAI's
[Feature Maturity](https://developers.openai.com/codex/feature-maturity) table.
- `dynamicTools` is double-experimental: gated behind
`capabilities.experimentalApi`, and **absent from the generated
`ThreadStartParams` bindings**, so the field you must send is one you hand-write
against no type. It has already changed wire shape once — `LegacyDynamicToolSpec`
exists as the compat scar, with `exposeToContext` replaced by the *inverted*
`deferLoading`.
- No protocol version, no breaking-change log, and an open regression in the `-c`
MCP path under app-server mode
([openai/codex#39537](https://github.com/openai/codex/issues/39537)).

The counter-signal is real and is why this is "revisit in a quarter" rather than
"avoid": Zed's ACP adapter *moved onto* app-server
([agentclientprotocol/codex-acp](https://github.com/agentclientprotocol/codex-acp)),
and OpenAI's own IDE surfaces run on it. The one thing MCP cannot give us is a tool
whose implementation lives in our own process memory. When Office genuinely needs
that, budget for re-porting `dynamicTools` at least once.
- **Engine-owned tools** — cowork, cad, extension, design. Under a harness backend
the engine's own tools are not in the loop at all. That is the agent tier's
problem to solve.

## The vendor risk is asymmetric, and it sets the order

An earlier draft of this document said driving the unmodified binary is "the
documented exception". That was too confident, and the correction matters enough to
change sequencing.

Anthropic began blocking third-party harnesses from Claude **subscription** billing
on **2026-04-04**, and the restriction is described as being extended to *all*
third-party harnesses
([claude-mem#1826](https://github.com/thedotmack/claude-mem/issues/1826), which links
the press coverage). OpenClaw is researching a `--method cli` path precisely because
direct OAuth token use has been blocked since then — so CLI-driving is the surviving
workaround, **not a safe harbour**. The same issue warns that spawning a `claude`
subprocess through the Agent SDK "is exactly the 'third-party harness' pattern
Anthropic is restricting". Anthropic's own June support article still says
subscription limits fund `claude -p` and third-party apps and that the metering
change is paused; treat the two together as a trajectory, not a guarantee.

OpenAI points the other way: it publishes the integration surface as a platform, and
LiteLLM openly ships a `chatgpt/` provider for subscription access.

Two consequences, both load-bearing:

1. **BYOK with the user's own API key is the first deliverable, not the fallback.**
It is unaffected by any of the above, it is the path Anthropic and Google both
name as the supported one for a third-party tool, and Office's policy is already
open for it.
2. **Codex before Claude Code**, for business reasons rather than technical ones.

## Nothing off the shelf is reusable

Surveyed 2026-09-22. No existing project is production-credible for our requirement
— drive the local CLI *and* pass caller-supplied tools through:

- [claude-code-api-rs](https://github.com/ZhangHanDong/claude-code-api-rs) (177★,
MIT) is the only mature native `tools` → `tool_calls` mapping, and it is Rust,
five months without a commit, and ships with
`use_interactive_sessions = false # Disabled by default due to stability issues`.
- [claude-code-openai-wrapper](https://github.com/RichardAtCT/claude-code-openai-wrapper)
(622★) states in its own README that function calling is not supported.
- [codingworkflow/claude-code-api](https://github.com/codingworkflow/claude-code-api)
(331★) is **GPL-3.0** — viral, so unusable in a shipped product.
- [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) (52.8k★) looks like the
obvious answer and is not: it never spawns the CLI. It performs the CLI's OAuth
flow itself, stores the token, and calls the vendor backend — the exact
architecture Anthropic blocked in April.
- LiteLLM, OpenRouter, Portkey, Helicone: no provider that drives a local Claude Code
or Codex CLI.

Two specifics are worth copying rather than the repos: the **pooled long-lived CLI
process** model and the native tools mapping from claude-code-api-rs, and the trick of
launching with `--tools "" --setting-sources "" --system-prompt <the caller's>`, which
strips roughly 28k tokens of the harness's own agent prompt and built-in tools out of
every request. That second one is the concrete mechanism behind the chat tier — it is
what turns an agent harness into something that behaves like a completion endpoint.

## What the products have to do

Almost nothing, which is the point.
Expand All @@ -114,33 +196,66 @@ into "unavailable" strands the user, because the remedy differs and we are not
allowed to offer the vendor's login ourselves. The remedy we may show is "run
`codex login`" or "run `claude`".

Both states are readable by ASKING the runtime, never by reading its credential
store: `claude auth status` exits 0 when signed in and 1 when not (its JSON field
names are undocumented, so the exit code is the contract), and Codex app-server
exposes `account/read`. A `claude -p` run also reports `apiKeySource` in its
`system`/`init` event.

Credentials need no work anywhere. The harness holds its own; the engine holds none
for it. Cowork already demonstrates the pattern for the BYOK case — it stores no
provider key and treats the engine's `auth.json` as the single source of truth.

## Sequencing

1. **Land `/v1/chat/completions`.** It is the receiving route for everything above
and it is not merged: the handler exists on `feature/v1-chat-completions`, and
`test:httpapi` fails without an `httpapi-exercise` scenario. Also needs the
`chatCompletions` capability flag, the SSE OpenAPI patch, and a route test.
Nothing here can ship before it.
2. **Build the shim with both backends.** One local chat-completions server; two
1. **BYOK with the user's own API key.** Moved to the front — see the vendor-risk
section. The engine side is **already built** — `GET /api/integration` returns
each integration's `methods` (OAuth / Key / Env) and its `connections`, and
`connect/key`, `connect/oauth`, the attempt routes and `DELETE /api/credential`
complete the set. An earlier draft of `docs/PROVIDER-AUTH.md` proposed a parallel
`/v1/providers` surface; that is withdrawn, because it would duplicate these over
the same store. The remaining work is product-side: make an app use them instead
of its own key store. Cowork already does (`apps/server/src/redrob-auth.ts`).
2. **Use the user's OWN binary.** The Codex SDK pins `@openai/codex` exactly and
resolves the executable from its own bundled platform packages unless
`codexPathOverride` is passed — so the default behaviour runs a SECOND copy we
downloaded, not the one the user signed in to. That re-creates the duplicate-engine
problem this whole effort exists to remove. Detect the user's install and pass the
path explicitly. Claude Code is proprietary with no redistribution grant, so it is
the only option there anyway, which makes both runtimes the same shape.
3. **Build the shim with both backends.** One local chat-completions server; two
normalizers behind it. The prototype's split — event folding separated from the
subprocess — is what makes the second runtime a second normalizer rather than a
second architecture. Reuse it rather than re-deriving it.
3. **Register through the local-provider path**, and surface the backends in
`/v1/providers` with their three states so a product can render settings
without hardcoding a list.
4. **Verify against real binaries.** Neither runtime is installed on the build
4. **Give the harness our tools through a stdio MCP server.** Not app-server. Know
the defaults before wiring: Codex's `startup_timeout_sec` is 10, `tool_timeout_sec`
is 60, and `required = true` makes `codex exec` exit with an error rather than
silently running without our server.
5. **Register through the local-provider path**, and report the harness backends
through `GET /api/integration` alongside every other integration, with their
three states, so a product renders settings from one list rather than hardcoding
a vendor set that goes stale.
6. **Verify against real binaries.** Neither runtime is installed on the build
host, so the live path — a real subscription actually paying for a turn — is
unproven until someone runs it on a machine with `codex` and `claude` signed in.
5. **Then the agent tier**, for cowork, code and cad, over ACP.

Two things to carry forward rather than discover later. `claude -p`'s
`stream-json` event schema has not been checked against the real binary — only the
flags are confirmed — so step 2 starts by reading it, not by assuming it mirrors
Codex's JSONL. And Anthropic has announced, then paused, a change that moves
third-party subscription usage onto a capped monthly credit; it currently still
draws from the subscription, but the trajectory is known, so the Claude backend
should surface usage state rather than assume it is free.
7. **Then the agent tier**, for cowork, code and cad, over ACP.

One thing to carry forward rather than discover later: Anthropic has announced, then
paused, a change moving third-party subscription usage onto a capped monthly credit,
so the Claude backend should surface usage state rather than assume it is free. Note
also that `claude -p`'s credential precedence puts the OAuth login **last** —
`ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `apiKeyHelper`, `CLAUDE_CODE_OAUTH_TOKEN`
and `ANTHROPIC_PROFILE` in the environment all silently take over billing, and
`--bare` never reads the OAuth login at all. That is the same class of bug as the
Codex env trap, in the opposite direction: the shim must build the child environment
from an allow-list for both runtimes.

`claude -p`'s `stream-json` schema IS now mapped, so step 3 does not start by
guessing: `system`/`init` carries `session_id`, `tools`, `mcp_servers` and
`apiKeySource`; assistant text and `tool_use` blocks are in
`assistant.message.content`; and the final `result` message carries
`subtype` (`success` or `error_*`), `is_error`, `total_cost_usd` and `modelUsage`.
Read cost from `modelUsage`, not `usage` — `usage` covers the main loop only and
undercounts subagents. Pin non-interactive behaviour with
`--permission-mode dontAsk --permission-prompts none`, and read-only with
`--tools "Read,Glob,Grep" --disallowedTools "Edit" "Write" "NotebookEdit" "Bash" "mcp__*"`.
45 changes: 29 additions & 16 deletions docs/PROVIDER-AUTH.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,22 +123,35 @@ key cannot leak one, log one, or sync one.

`redrob providers login` already implements both shapes — a generic OAuth flow
with `authorize()` plus `auto` and `code` callbacks, and an API-key path
(`packages/redrob/src/cli/cmd/providers.ts`). What is missing is not the flow but
its exposure: a product cannot drive it today.

Add, on the v2 surface beside the completions route:

- `GET /v1/providers` — what this engine can use, each entry declaring which auth
methods it accepts (`oauth`, `api-key`) and whether a credential is present.
This is what lets an app render a settings page without hardcoding a vendor
list that goes stale.
- `POST /v1/providers/:id/login` — starts a flow. For OAuth it returns the
authorization URL and an opaque attempt id; for an API key it accepts the key.
- `POST /v1/providers/:id/login/:attempt` — completes an OAuth attempt with the
authorization code, or reports that the loopback callback already completed it.
- `DELETE /v1/providers/:id/credential` — disconnect.

Three rules on those routes:
(`packages/redrob/src/cli/cmd/providers.ts`).

**An earlier draft of this section proposed four new `/v1/providers` routes to expose
it. That was wrong: the routes already exist.** They are the `server.integration`
group on the v2 surface, and they already carry everything a settings page needs.
Building the `/v1/providers` set would have been a duplicate surface over the same
credential store, with two code paths to keep in agreement.

What exists, and what it replaces from that proposal:

| proposed | already exists |
| --- | --- |
| `GET /v1/providers` | `GET /api/integration` — returns `Integration.Info`, whose `methods` array is a union of `OAuthMethod`, `KeyMethod` and `EnvMethod`, plus `connections` for what is already connected |
| `POST /v1/providers/:id/login` (key) | `POST /api/integration/:integrationID/connect/key` |
| `POST /v1/providers/:id/login` (oauth) | `POST /api/integration/:integrationID/connect/oauth` |
| `POST /v1/providers/:id/login/:attempt` | `POST /api/integration/attempt/:attemptID/complete`, with `GET /api/integration/attempt/:attemptID` for status and `DELETE` to cancel |
| `DELETE /v1/providers/:id/credential` | `DELETE /api/credential/:credentialID` |

So the engine-side work for BYOK is **done**, and the real work is product-side:
making an app use these routes instead of its own store. Office keeps keys in
`userData/ai-settings.json`, Design in the macOS keychain, the extension in
`chrome.storage.local`, Query and Recall in separate keyring services — five stores
that never read each other, which is the whole reason a user logs in again in every
app. Cowork already does it the right way: it stores no provider key and posts the
key once to the engine (`apps/server/src/redrob-auth.ts`), treating the engine's
`auth.json` as the single source of truth. That is the pattern to copy.

Three rules still hold for those routes, and they are worth restating because they
are what makes the single store safe:

1. **The key never comes back out.** A response says a credential is present and
names it; it never returns the secret. A product that cannot read the key
Expand Down
Loading