feat(server): OpenAI-compatible /v1/chat/completions on the engine - #58
Merged
Merged
Conversation
First half of the route from docs/LOCAL-ENGINE-API.md: the declared contract and
the translation layer. No handler yet, so nothing is served — the endpoint is not
reachable until the following commit wires it.
The group goes on the v2 protocol surface rather than the instance httpapi:
Authorization is declared once at that HttpApi level so a new group gets it
instead of having to remember .middleware(Authorization), it has the typed SSE
helper (HttpApiSchema.StreamSse), and it is the surface with a contract
changelog. Paths there are literal, so /v1/... sits alongside /api/... without a
prefix mechanism.
Tool ownership is decided by the request and nothing else: `tools` present means
the caller executes them and the turn ends with finish_reason tool_calls;
`tools` absent means a plain completion. One rule rather than three configurable
ones, and it is what Office and browser already assume — their tools mutate an
open document and a live tab, which the engine cannot do on their behalf.
Errors are one class per status sharing the OpenAI envelope. Collapsing them
would report a rejected key as a bad request, and the status is what a client's
transport sees first. 401 carries code `engine_not_authenticated`, which is the
only condition a client may turn into a sign-in prompt.
The conversion layer is pure — no Effect, no services — because it is where the
silent failures live: a mistranslation does not fail the request, it just gives
the model something other than what the caller wrote. What it refuses is the
interesting part. Malformed tool_call arguments and an unsupported content part
are rejected rather than forwarded as {} or dropped, since an image quietly
discarded looks to a user like the model ignored it. A tool message with no
tool_call_id is refused because the provider pairs results to calls by that id
and there is nothing sensible to guess.
Two asymmetries the tests pin down: the wire carries tool arguments as a JSON
string while the engine wants them parsed, so both directions convert; and
"error"/"unknown" finish reasons have no OpenAI equivalent and must not be
reported as "stop", which would tell the caller a failed turn ended cleanly.
22 tests pass. protocol and server typecheck clean.
The handler for the contract in the previous commit. The route is now reachable. It touches no Session, Agent or tool runtime: it resolves a provider the way the agent does (catalog entry plus the active integration credential), makes one model call, and translates. Everything stateful stays in the session API. handleRaw rather than handle, because the reply is JSON or text/event-stream depending on `stream`, and only the raw form chooses its own response. The cost is that the payload is not decoded for you, so the declared schema is applied in the handler rather than the body being trusted. Streaming is where the care went. Tool calls are emitted from the completed `tool-call` event, not from `tool-input-delta`: the engine parses arguments and the wire wants a JSON string, so re-emitting fragments would mean re-serializing a half-parsed value. And `provider-error` arrives as a stream ELEMENT rather than a failure — left unhandled, an upstream 500 would end the stream with a clean [DONE] and the caller would read a truncated answer as a complete one, so it is matched explicitly and sent as an error frame. Once headers are out, a frame is the only way to say the turn broke. Error mapping keeps one value load-bearing: 401 with code engine_not_authenticated, covering both a missing credential and a rejected one — from the caller's side those are one problem and neither is fixed by retrying. What is NOT remapped is a content-policy classification: the executor checks the body before the status, so an upstream 401 mentioning safety arrives as ContentPolicy, and reclassifying it here would report a policy refusal as a login problem. A caller-supplied base URL is deliberately not accepted. The config layer only admits a new openai-compatible provider at a local address (isLocalEndpoint); honouring an arbitrary URL here would bypass that and forward the engine's own credential to whatever host the caller named. A local model is selected by its `provider/model` id instead. Four APIs were guessed wrong first and corrected against the source, not worked around: this Effect version has no Effect.either, no Effect.catchAll and no Stream.mapConcat, LLMResponse carries finishReason rather than a finish object, and catalog.model.get returns an Effect on the Service (only the Draft form is synchronous). tool_choice now travels as the engine's own shape instead of a bare string. A bare string is ambiguous — the normalizer reads auto/none/required as modes and anything else as a tool name — so a tool actually named `auto` would silently become a mode. A test pins that case. 23 tests pass; protocol and server typecheck clean. Still to come: the capabilities flag, the httpapi-exercise scenario (test:httpapi runs --fail-on-missing, so CI will fail until it exists), the SSE OpenAPI patch in public.ts, and a route test.
The design of record for how a user connects their own model access, so one setup serves code, cowork, office and whatever comes after. The product ask was 'let people use the Claude or ChatGPT they already pay for'. Half of that is not available, and the split is contractual rather than technical, so the document leads with the survey and links every claim. Not available: a sign-in button in our own apps that spends a consumer subscription. Anthropic's compliance page forbids third parties offering Claude.ai login or routing through Free/Pro/Max credentials, and bans accounts for spoofing the Claude Code harness - our own upstream removed Claude support on 2026-02-19 under legal request. Google's Gemini CLI FAQ says piggybacking its OAuth may terminate the account, and there is a documented mass-ban thread. OpenAI allows it in practice, but only by embedding OpenAI's own Codex runtime, and no clause grants it, so it is a revocable product decision. Available, and it covers most of the intent: BYOK everywhere (Anthropic and Google both name a user API key as THE supported third-party path), plus three real sign-in buttons - GitHub Copilot through the Copilot SDK, which spends the user's own Copilot subscription with GitHub's blessing and a documented desktop device flow; OpenRouter OAuth PKCE, which reaches Claude and GPT models behind an account login and is the closest legitimate thing to what was asked for; and Azure OpenAI via Entra device code. The design itself adds little and removes a lot. Credentials stay in this engine's existing auth.json, and products stop having stores of their own - today Office, Design, the extension, Query and Recall keep five stores that never read each other, which is the whole reason a user logs in again in every app. Products never hold a key: /v1/chat/completions already carries the engine's credential outward and takes none from the caller, so adding a provider is one change here rather than a release in every app, and an app that holds no key cannot leak one. Four routes are proposed to expose the login flow that already exists in , with three rules: the key never comes back out, the OAuth code stays in the engine, and no caller-supplied base URL - that last one because honouring it would forward this engine's credential to whatever host the caller named. Also records a compliance check to keep: this repository contains no Anthropic OAuth client id and no claude.ai endpoint, and the only hardcoded vendor OAuth is xAI's.
Codex and Claude Code as an option in all ten products from one implementation in this engine, instead of ten adapters in ten repositories. Corrects where the first adapter was put. It was written inside redrob-cowork behind that app's engine-spawn hook; it works and it found real bugs, but it gives Cowork a Codex option and the other nine nothing. It belongs here, because every product already reaches a model through this engine and the ones that still call the Console directly are what /v1/chat/completions was built to bring in. The mechanism turned out to already exist. config/plugin/local-provider.ts admits a provider when the package is exactly the trusted @ai-sdk/openai-compatible and the URL is on this machine - a door opened for Ollama, LM Studio, llama.cpp and vLLM whose only shared trait is speaking OpenAI-compatible chat-completions locally. A small local shim in front of codex exec or claude -p meets the same bar, so this needs no new provider-loading machinery and does not widen the arbitrary-npm-package refusal by a millimetre. This recommends a shim where docs/CODEX-RUNTIME.md argues against one, and the doc says why rather than pretending to be consistent: a Cowork shim must emulate the OpenCode session API, which is large, ours and still changing, so falling subtly behind produces bugs that look like model bugs. A shim here emulates chat-completions, which is small, published, frozen and not ours. Different surface, different answer. What cannot be papered over: these are agent harnesses, not completion endpoints, so support has two tiers. The chat tier - prompt in, text out - covers every product's ordinary use and needs no per-product code. The agent tier, where the runtime owns a real loop against a workspace, cannot go through chat-completions at all, because that contract says tools present means the caller owns them while here the runtime owns the loop. Cowork, code and cad need it; it must not block the chat tier. One safety point that is easy to miss: a harness with a writable sandbox will read and edit the user's filesystem, and a user asking Office to reword a sentence has not consented to an agent walking their disk. The shim pins read-only, approvals never, network off, unless an agent-tier caller asked for more. Sequencing puts /v1/chat/completions first because nothing here ships before it, and records two things to carry rather than rediscover: claude -p's stream-json schema is unverified against the real binary, and Anthropic has announced then paused a change capping third-party subscription usage, so the Claude backend should surface usage state rather than assume it is free.
The route had a handler and a conversion layer but nothing that made CI accept it.
This adds the coverage scenario, the capability flag, and route tests - and the
tests immediately found a real bug in the handler.
The bug: a body that does not decode against ChatCompletionRequest came back as
502 api_error. schemaBodyJson fails with HttpServerError | Schema.SchemaError, and
neither was recognised, so both fell through to the catch-all that reports an
upstream failure. A caller who sent a malformed payload was told our upstream had
failed, which sends them debugging the wrong system. Now classified 400
invalid_request_error. Matched on _tag rather than instanceof, because
Schema.SchemaError and the HTTP request errors are separate hierarchies and an
instanceof chain over both stops matching silently when either is re-exported
through a different module instance.
The coverage gate was mandatory and I nearly talked myself out of it. Reading
routing.ts, coverage derives from routeKeys(OpenApi.fromApi(modules.PublicApi)) --
the instance httpapi -- while this route is registered on the v2 protocol surface,
so I concluded it was out of scope. Running it said otherwise: PublicApi merges the
v2 groups, the exerciser reported MISS POST /v1/chat/completions, and
--fail-on-missing failed the run. The scenario is real and the reading was wrong.
That scenario deliberately exercises the 400 path. A real completion needs a
provider credential this harness has none of, so asserting 200 would make the gate
depend on the machine it runs on. An unknown model id is refused before any
provider resolves, which is deterministic and still covers what most likely
breaks: that the body decodes, and that the failure arrives in the OpenAI error
envelope rather than Effect's default shape. Verified in effect mode against a
real server, not just in coverage mode.
The capability flag is chatCompletions: { version: 1, callerTools: true, stream:
true } on /experimental/capabilities, with a matching struct in the group schema.
Declared as a struct rather than a version number so a caller negotiates a
capability instead of pinning an engine build - an app needing caller-owned tools
checks callerTools, and an engine without the route omits the key entirely. version
is the contract's revision, not the engine's.
Five route tests drive the real app in-process through httpapi-layer. The envelope
assertions are the point: a caller written against OpenAI reads error.message and
error.type, so the tests also assert Effect's { name, data } shape does NOT leak.
One test pins the v1 contract by sending every promised field including the
caller-owned tools path, and checks the failure is about the model rather than a
decode, so dropping a field from the schema fails it.
Two things deliberately NOT in this change.
The SDK codegen is not regenerated. bun script/generate.ts also runs a formatter
across the repo (61 files, mostly markdown table alignment) and its openapi.json
output contains routes that are not mine - /api/variant/paraphrase among them -
which means the generated SDK was already stale before this branch. Regenerating
here would sweep another session's unregenerated surface into this PR. It needs its
own commit. This repo has no CI check for stale codegen, which is why it drifted.
The SSE response is still undocumented in OpenAPI. The group declares only
ChatCompletionResponse, so the streaming path has the same gap /event has and the
same fix site in public.ts. Doing it properly means defining a ChatCompletionChunk
schema in the protocol, which is more than closing a gate, so it is recorded rather
than half-done.
Coverage 211 pass / 0 fail / 0 missing. Route tests 5/5. Wire tests 23/23.
Typecheck clean on both packages.
…nate CI failed six jobs with: Promise error must have a literal discriminator: server.chat.chat.completions httpapi-codegen requires every declared endpoint error to carry a _tag or name STRING LITERAL to branch on (declaredErrorFields). The four error classes shared one OpenAI envelope and had nothing to discriminate by, so generation threw and every job that generates the client died with it. Removed the error declarations rather than adding a discriminator, for two independent reasons: The handler never used them. It is handleRaw - one request answers with JSON and another with text/event-stream - so it writes every response itself, failures included, through errorResponse(). The declared classes were decoration that had never been on the path. And adding _tag to satisfy the generator would make the generated spec and SDK claim a field the wire does not carry, because OpenAI's envelope has no such key. A spec that lies about the shape is worse than one that is silent about it. So the statuses are documented in the endpoint description instead - 400, 401 with code engine_not_authenticated as the only sign-in trigger, 429, 502 - and the envelope survives as an exported ChatCompletionError type so the handler and any caller build against one shape. docs/LOCAL-ENGINE-API.md already specifies the wire contract. Note this was caught by a real CI gate I had earlier described as absent. The generated client IS checked: packages/client check:generated regenerates and fails on any diff. That is a different check from the SDK staleness I noted in the PR body, and the generated client output is committed here for it. Coverage 211/0/0. Route + wire tests 28/28. protocol, server and client typecheck clean.
CI failed with: Method 'POST' already declared for route '/v1/chat/completions' taking down every test that stands up both the engine's HttpApi and the fake provider (httpapi-sdk prompt streaming, project-skills prompt context). TestLLMServer piped Layer.provide(HttpRouter.layer), which layer memoization resolves to the SAME HttpRouter instance the engine's HttpApi is built on, so the fake and the app registered into one router. That was invisible while no engine route shared a path with the fake. Adding POST /v1/chat/completions to the engine collided with llm-server.ts:702, which serves that exact path. Fixed with Layer.fresh so the fake gets its own router instance. The fake's path is deliberately NOT renamed to dodge the clash. It exists to look like a real OpenAI-compatible endpoint, and /v1/chat/completions is where a real one lives; moving it to a made-up path would stop the fake exercising what the tests are actually about. The collision was a wiring accident, not a naming one, so the wiring is what changed. Worth noting the failure mode: the fake had been sharing the app's router all along, which means any future engine route matching a test double's path would have broken the same way. This makes that class of collision impossible rather than resolving one instance of it. All 130 tests across the 12 files that use TestLLMServer pass (1 pre-existing skip). httpapi-sdk 18/18.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An OpenAI-compatible receiving route on the engine, so every Redrob product reaches a model through one contract instead of each shipping its own transport, credential store and duplicate engine copy.
Design of record:
docs/LOCAL-ENGINE-API.md(merged in #44). This branch adds the contract, the conversion layer, the handler, and the CI gates.The one rule
toolspresent → the caller owns them: the engine emitstool_callsand ends withfinish_reason: "tool_calls".toolsabsent → the engine runs its own tools. They never mix in v1. That split is what lets a host-owned-tools product (Office, browser, canvas) and an engine-owned-tools product (cowork, cad) share one route.What is here
packages/protocol/src/groups/chat-completion.ts— the contract, on the v2 surface soAuthorizationis inherited at the HttpApi level rather than re-declared per group (5 of 21 instance groups forget it), and so the typed SSE helper is available.packages/server/src/handlers/chat-completion-wire.ts— pure conversion, 23 tests.packages/server/src/handlers/chat-completion.ts— the handler.chatCompletionscapability flag, 5 route tests.A bug the route tests found
A body that does not decode against
ChatCompletionRequestcame back as 502api_error.schemaBodyJsonfails withHttpServerError | Schema.SchemaErrorand neither was recognised, so both fell into the catch-all that reports an upstream failure — telling a caller who sent a malformed payload that our upstream had failed, which sends them debugging the wrong system. Now 400invalid_request_error, matched on_tagrather thaninstanceofbecause those are separate hierarchies and aninstanceofchain stops matching silently when either is re-exported through a different module instance.The coverage gate, and a reading I got wrong
Coverage derives from
routeKeys(OpenApi.fromApi(modules.PublicApi))— the instance httpapi — while this route registers on the v2 protocol surface, so I concluded the scenario was out of scope. Running it said otherwise:PublicApimerges the v2 groups, the exerciser reportedMISS POST /v1/chat/completions, and--fail-on-missingfailed the run.The scenario deliberately exercises the 400 path. A real completion needs a provider credential this harness has none of, so asserting 200 would make the gate depend on the machine it runs on. An unknown model id is refused before any provider resolves — deterministic, and still covers what most likely breaks: that the body decodes, and that failures arrive in the OpenAI envelope. Verified in
--mode effectagainst a real server, not only in coverage mode.Capability, not a version pin
chatCompletions: { version: 1, callerTools: true, stream: true }on/experimental/capabilities. A struct rather than a version number so a caller negotiates a capability instead of pinning an engine build: an app needing caller-owned tools checkscallerTools, and an engine without the route omits the key.versionis the contract's revision, not the engine's.Deliberately not in this PR
bun script/generate.tsalso formats the whole repo (61 files, mostly markdown table alignment), and itsopenapi.jsonoutput contains routes that are not mine —/api/variant/paraphraseamong them — so the generated SDK was already stale before this branch. Regenerating here would sweep another session's unregenerated surface into this PR. Needs its own commit. There is no CI check for stale codegen, which is why it drifted.ChatCompletionResponse, so the streaming path has the same gap/eventhas and the same fix site inpublic.ts. Doing it properly means defining aChatCompletionChunkschema in the protocol — more than closing a gate, so it is recorded rather than half-done.Verification
Coverage 211 pass / 0 fail / 0 missing. Route tests 5/5. Wire tests 23/23. Typecheck clean on
packages/serverandpackages/redrob.Docs riding along
docs/PROVIDER-AUTH.md— what each vendor actually permits a third-party app, with the clause for each claim.docs/LOCAL-HARNESS-BACKENDS.md— how Codex and Claude Code become an option in all ten products from one place, via the existinglocal-providerpath, and why that splits into a chat tier (all products) and an agent tier (cowork, code, cad) that chat-completions cannot express.