diff --git a/.gitignore b/.gitignore index 41099ea7..a5836031 100644 --- a/.gitignore +++ b/.gitignore @@ -105,3 +105,7 @@ readme-preview.mjs # Uploaded images land here under each session working dir (runtime artifact) .claude-images/ + +# Local-LLM harness smoke-test config (real IPs/keys) — see the .example.json +# alongside it in scripts/, which IS tracked as the template. +scripts/local-llm-test.config.json diff --git a/CLAUDE.md b/CLAUDE.md index ed423971..731a6f15 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -225,6 +225,8 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **DeepSeek web UI** (`POST`/`GET`/`DELETE /api/deepseek/web`, `deepseek-web-server.ts`): the Run menu's "DeepSeek web UI..." entry supervises ONE background `dsh web` child process, deliberately **NOT a shell session**. The session version worked and was still wrong in use: it put a terminal tab on screen next to the web tab the user actually asked for, every single time, and nothing about a long-lived HTTP server needs to be a tab. ⚠️ What a session gave for free now has to be paid for explicitly, and every piece is load-bearing: **exactly one** server (a second click REUSES it rather than racing it for a port, which two sessions structurally could not do), **restarted when the browser authority changes** (`--trusted-host` fences dsh's `/api` against the browser authority, and a Codeman reachable at both loopback and a tailnet name has two, so whoever asks last wins: the asker is by definition the origin about to load the page), **killed on shutdown** (`stopDeepSeekWeb()` in the server teardown, because the child is detached so its whole plugin tree can be signalled at once, which also means it would OUTLIVE Codeman and hold its port against the next start), and **failures returned to the caller**, since with no tab there is nowhere for a stack trace to land. ⚠️ The port search starts at dsh's own default 3080 and walks 40, never fixed: that default is precisely the port most likely to be taken already by the user's own `dsh web`, and hardcoding it killed this feature with EADDRINUSE once. Free-port detection BINDS rather than connects (a connect probe cannot tell "free" from "listening but not answering yet"), so it is racy by nature and the caller still waits for the server to really answer before reporting success. ⚠️ Both `POST` and `DELETE` sit at the **same privilege bar as the profile installer** (`canUsernameRunPrivilegedCommands`) even though the action reads as "open a page": booting a dsh profile executes the plugin code in it, and the server is a single shared instance, so stopping it in multi-user mode takes it out from under other users' tabs. +**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `deployment_plan.md`): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET /v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×. ⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified. ⚠️ Applying a selection **restarts the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`. **Confidence, verified end-to-end against a real llama-swap server via the DYNAMIC `scripts/test-local-llm-harnesses.ts`** (reads the live CLI registry, so a registry change needs zero script edits): claude/opencode/pi/grok/omp **PASS**; codex config structure is correct but codex only speaks the Responses API since Feb 2026, which llama.cpp/llama-swap don't implement — a confirmed protocol gap, not a bug; gemini fails with `Invalid auth method selected` (an undocumented `GATEWAY` AuthType gemini-cli selects once `GOOGLE_GEMINI_BASE_URL` is set — unresolved after real investigation); deepseek reaches the server but gets a consistent `HTTP_404` (root cause not identified); antigravity has no known mechanism at all. See the confidence table in `deployment_plan.md` for the full detail on each. + **Run launch synchronization**: the Run entrypoint holds an in-flight lock and disables `#runBtn` for the whole launch (≥500ms), so a double click cannot create duplicate sessions with the same `w-` name. `_ensureCreatedSessionVisible()` runs before `selectSession()`, and `_onSessionCreated()` stays an idempotent upsert, so POST-first and SSE-first ordering both produce exactly one rendered tab. ⚠️ **Closing has the mirror-image race and one owner**: `closeSession()` reads `wasActive` BEFORE its `await` and announces the delete via `_closingSessions`, while `_onSessionDeleted` skips the active-session handoff for an id in that set. Both used to read `activeSessionId` after the fact, so the `session_deleted` broadcast for your own delete could null it first and closing the tab you were on landed on the welcome screen instead of the next session, on the same build, depending on timing. The fallback also picks the first order entry that is still in `sessions` (a dead id can linger in `sessionOrder`, same reason Alt+N indexes a live-filtered list). A delete from ANOTHER client still shows the welcome screen, which is the honest answer when what you were looking at was taken away. Tests: `test/session-close-fallback.test.ts`. → [architecture-invariants#run-launch-synchronization](docs/architecture-invariants.md#run-launch-synchronization) **Session lineage lines** (tab → tab it spawned, `sessionLineageLines`, per-device, desktop default ON): a create request may name the session that spawned it, as a `parentSessionId` body field on `POST /api/sessions` / `POST /api/quick-start` or the `X-Codeman-Parent-Session` header (the agent skill sets that once on its shared curl invocation, so every spawn recipe carries it). `resolveParentSessionId()` (route-helpers.ts) **resolves rather than trusts** it: exact id, else a UNIQUE ≥8-char prefix (ids reach agents truncated), it must be a live session the caller can see AND carry the same owner, and **anything unresolvable is DROPPED, never a 400** — a cosmetic field must not be able to fail a worker spawn. It rides `toState()` into `session_created`, so there is no new SSE event. ⚠️ Rendering is an ADDITIONAL LAYER on the existing SVG pass (`_appendLineageConnectionLines` called at the tail of `_updateConnectionLinesImmediate()`, exactly like ultracode), sharing one batched read→write reflow and the `tab:` rect cache; geometry is pure in `computeLineagePath()` (constants.js). ⚠️ **ONE shape, and the second one was the bug**: every pair (flat strip or wrapped) gets a U-bridge hanging below the strip, anchored on both tabs' BOTTOM edges. A wrapped strip used to get a parent-bottom → child-TOP bezier with a ~14px row gap to bend in, which drew a flat line hidden in the gap with siblings overprinting. ⚠️ The dip is a **mis-tuned-in-both-directions corridor** (44px cap = straight thread at strip-wide spans, #285; 104px cap + full row offset = ~106px over-bow into the terminal, 2026-08-15): it now hangs from the **STRIP's bottom edge** (fallback: lower tab bottom), capped at 64px, with NO per-row offsets stacked on top — the strip-bottom baseline is also what keeps a row-1 pair's arc from drawing through row 2's tab labels. ⚠️ **Colors are keyed on the SPAWNING tab, not per child**: every arc leaving one tab is the same color however many workers it spawns, so the strip reads as "these five came from w1, those two came from w2" — per-child coloring gave one tab's own children a different color each, which is the distinction the colors exist to make. A child that spawns in turn is a parent in its own right and gets its own color for the arcs below it, so a chain changes color at each generation while each generation's fan-out stays uniform. Assignment cycles `CodemanLineage.COLORS` in first-seen order per parent id (first entry empty = the skin-tuned `--session-blue`, so the first spawning tab keeps it; the rest vivid fixed hexes), memoized rather than derived from draw index (the SVG is wiped and rebuilt constantly, so an index-based color would flicker), and set inline as `--lineage-color` so styles.css keeps owning opacity/glow/dash. `test/session-lineage-lines.test.ts` drives the real `_appendLineageConnectionLines()` and asserts the painted property, since testing the color function alone would pass just as happily with the child id passed back in. ⚠️ **Desktop only**: the overlay is `z-index: 999` and the desktop header is 100 (arcs paint over it, which is what lets them touch tab bottoms), but under 1024px mobile.css makes the header `fixed; z-index: 1200` and would bury them. ⚠️ Paths carry `data-agent-id="lineage:"` because that is what `_applyLineEntrances()` queries — that one attribute is what gives them the entrance animation and its negative-`animation-delay` resume across `svg.innerHTML=''`. ⚠️ `.session-tabs` is `overflow-x: auto`, so a scrolled-out tab still HAS a rect (over the logo); edges with an endpoint outside the strip are skipped, and a passive `scroll` listener re-anchors the rest. diff --git a/PR.md b/PR.md new file mode 100644 index 00000000..febd38d7 --- /dev/null +++ b/PR.md @@ -0,0 +1,296 @@ +# feat: Custom Model Endpoint Profiles (local or cloud, all harnesses) + +> **⭐ Shout-out up front:** this feature was partly inspired by — and is a +> great fit for — **[Ark0N/Qwen5090](https://github.com/Ark0N/Qwen5090)**, +> the maintainer's other project: a one-click Windows / one-command Linux +> installer that stands up Qwen3.8-27B locally on an RTX 5090 behind an +> OpenAI-compatible API (vLLM / NInfer / llama.cpp). Once this feature lands, +> pointing Codeman at a Qwen5090 box is just adding one endpoint entry — no +> extra code, no special-casing. Qwen5090 already wires up DeepSeek Harness +> and Claude Code as local coding agents itself, which is basically this +> feature's idea in miniature. 🙂 + +**Status: open for review.** The backend/CLI-injection side (chunks 1-5, +7-8) is built, tested, and verified end-to-end against a real server; +chunk 6 (frontend toolbar UI) is not yet built, and gemini/deepseek remain +documented, unresolved gaps rather than working paths — see +[Status](#status) below for exactly what's done and what's still open. + +--- + +## What + +Adds a settings-gated (default **OFF**) way to point any Codeman-supported +harness — Claude, opencode, Codex, Gemini, Pi, Grok, DeepSeek, OMP, or +Antigravity — at a **custom OpenAI-compatible endpoint** instead of its +native cloud backend, for a given session. "Custom endpoint" covers both: + +- **Local hardware**: llama.cpp, Ollama, vLLM, a home GPU rig, or + purpose-built on-prem boxes like NVIDIA DGX Spark, AMD Strix Halo + (Ryzen AI Max) mini-PCs, or the [Qwen5090](https://github.com/Ark0N/Qwen5090) + setup above. +- **Cloud**: Azure AI Foundry's OpenAI-compatible endpoint, OpenRouter, a + company's self-hosted gateway. + +The user adds an endpoint by base URL (+ optional API key), Codeman +discovers its available models via `GET /v1/models`, and a new toolbar +picker lets them apply one of those models to a session — which then +restarts that session's CLI process pointed at the endpoint. + +**Why discovery instead of asking the user to type a model name:** it turns +"go read your inference server's docs to find the exact model identifier it +expects" into "pick from a list Codeman already fetched" — one less place +for a user to get a name/casing wrong and have a harness fail with an +opaque "model not found." It also means this feature works unmodified +against **multi-model hosting setups**, not just a single-model server: a +gateway like **[llama-swap](https://github.com/mostlygeek/llama-swap)** +(or vLLM/LiteLLM/Ollama serving several loaded/loadable models behind one +`/v1/models` list) already advertises every model it can hot-swap to, so +the toolbar picker becomes a live menu of everything that endpoint can +serve — no per-model endpoint entries, no separate configuration step, +just "add the gateway once, everything behind it shows up." + +## Why + +The maintainer pays for a Claude Code subscription but also runs a capable +local model. Every harness Codeman drives already _has_ its own mechanism +for pointing at a custom endpoint (env vars for Claude, a JSON config blob +for opencode, a TOML file for Codex, etc.) — Codeman just never exposed a +UI for it. Full motivation, the per-CLI recipe table, and the on-prem +hardware use cases are written up in **[`deployment_plan.md`](deployment_plan.md)**. + +## How + +- **`src/config/cli-registry/{types,schema,stock}.ts`** — new + `capabilities.customModelInjection` field per CLI entry, one of four + kinds: `env` (Claude, Gemini, Grok, DeepSeek), `configContentEnv` + (opencode, reusing its existing `OPENCODE_CONFIG_CONTENT` mechanism), + `configDir` (Codex/Pi/OMP — writes an isolated config file, never touches + the user's real one), or `unsupported` (Antigravity — no known mechanism, + toolbar entry stays disabled). Declared data-driven per the repo's + existing "never branch on CLI id" rule. +- **`src/custom-model-injection.ts`** — pure function turning + `(CliEntry, endpoint, modelId)` into the real env vars / config content. + No IO; a caller writes `configDir` files to disk. +- **`src/custom-model-hosts.ts`** + **`src/web/routes/custom-model-routes.ts`** — + read/write-array endpoint store (`~/.codeman/custom-model-hosts.json`, + same shape as `remote-hosts.ts`) and `GET/POST/PUT/DELETE +/api/model-endpoints` + `POST /:id/discover-models`, admin-gated in + multi-user mode, SSRF-guarded via the same `isBlockedWebviewUrl()` check + web tabs use. +- **`src/web/schemas.ts`** — `customModelEndpointsEnabled` (synced, default + OFF) + the endpoint payload schema. +- **`scripts/test-local-llm-harnesses.ts`** — standalone smoke-test script + that spawns each real CLI binary one-shot against a real endpoint and + checks it can answer "hello world," independent of the web UI. Reads + defaults from a gitignored `scripts/local-llm-test.config.json` (see the + committed `.example.json`) so real IPs/keys never land in git. + +### A finding along the way: multi-user privilege hardening + +Building this surfaced that several env vars (`GOOGLE_GEMINI_BASE_URL`, +`GROK_BASE_URL`, `CODEX_HOME`, `PI_CONFIG_DIR`, `OPENCODE_CONFIG_CONTENT`, +etc.) were **already** reachable via the generic `envOverrides` API today, +pre-existing this PR, because Codeman's env allowlist is prefix-based and +global. A non-granted multi-user owner could already redirect a session's +endpoint/credentials via a plain `envOverrides` field. This PR adds all of +them to their CLI's `privilegedEnvKeys` (the existing clamp mechanism +`DEEPSEEK_BASE_URL` already used), closing that gap rather than widening it. +`CODEX_HOME` and `PI_CONFIG_DIR` are flagged as extra-sensitive: a +redirected config dir can restate approval/sandbox policy or, for Pi, +redirect to a dir Pi will execute `.pi/extensions` TypeScript from. + +Claude is the deliberate exception: `ANTHROPIC_*` is **not** added to +Claude's allowed env prefixes at all, so it stays reachable only through +the dedicated, admin-configured, SSRF-guarded custom-model route — never +through a plain client-supplied `envOverrides`. + +## Status + +Built in reviewable chunks; ✅ = done and verified (typecheck + lint + +format + tests green), ⬜ = not started. + +- ✅ **1. Registry types** — `customModelInjection` capability shape +- ✅ **2. Pure injection builder** — `custom-model-injection.ts` + 15 unit tests +- ✅ **3. Endpoint store + CRUD routes** — `custom-model-hosts.ts`, + `custom-model-routes.ts`, discovery + SSRF guard, 7 route tests +- ✅ **4. Settings + security hardening** — `customModelEndpointsEnabled` + flag, `privilegedEnvKeys` additions across 7 CLI entries +- ✅ **5. Session integration** — `session.customModel` state field, + `session.setCustomModel()`/`session.restartCli()` (a generalized, + de-restricted `reattachRemote()` reusing the existing `respawn-pane -k` + primitive), `POST /api/sessions/:id/custom-model` restart route. 5 new + route tests; the existing `session.test.ts`/`session-cleanup.test.ts` + suites can't run at all on this Windows dev box (no local `tmux` — + confirmed identical on unmodified `master`, not a regression), which is + exactly why the container test below matters. +- ⬜ **6. Frontend** — settings group, toolbar picker, tab badge +- ✅ **7. Mock-server contract tests** — `test/fixtures/mock-openai-server.ts` + and `test/custom-model-injection-contract.test.ts`, 10 tests replaying + every CLI's real injected values through an HTTP call shaped the way that + CLI sends it, against an in-process fake server +- ✅ **8. Docs** — `docs/custom-model-endpoints.md` (user guide, HTTP-API-only + until chunk 6 lands) + a CLAUDE.md pointer bullet + +Also done outside the chunk list: the standalone +`scripts/test-local-llm-harnesses.ts` smoke-test script (now dynamic — +reads the live CLI registry rather than a hand-maintained harness list) + +its gitignored config file, the on-prem-hardware use-case writeup in +`deployment_plan.md` (DGX Spark, Strix Halo, Qwen5090), a +`codeman/agent:llm-test` Docker image (all 9 CLI binaries, built from +`docker/agent.Dockerfile`), and a **completed real end-to-end run of all 9 +harnesses** against a live llama-swap server — see Testing below. + +## Testing performed so far + +- `npm run typecheck` — clean after every chunk +- `npm run lint` / `npx prettier --check` — clean +- `npm test -- test/cli-registry test/custom-model-injection.test.ts +test/custom-model-injection-contract.test.ts test/routes/custom-model-routes.test.ts +test/routes/session-custom-model.test.ts test/routes/external-cli-bypass-clamp.test.ts` — + 245+ tests passing, including the existing multi-user clamp suite (no + regressions from the `privilegedEnvKeys` additions) +- Refactored `scripts/test-local-llm-harnesses.ts` (now `npx tsx`-run, was + plain `.mjs`) to import `enabledClis()` and `buildCustomModelInjection()` + directly from source instead of keeping a second hand-maintained copy of + every CLI's env/config shape — a registry change now needs zero edits to + the test script. Extracted the config-dir-write logic shared with the + production route into `custom-model-injection-apply.ts` so both places + call exactly one implementation. +- **Real end-to-end run against the maintainer's live llama-swap server** + (`http://10.10.11.241:8080`), inside `codeman/agent:llm-test` (all 9 CLI + binaries, built via `docker/agent.Dockerfile`), against the smallest + available model (`qwen3.5-0.8b-ud-q8_k_xl`, 1.1GB — picked by parsing the + server's own reported model sizes). **Full 9-harness result: claude, + opencode, pi, grok, omp all PASS with a genuine "hello world" reply + round-tripped through the real endpoint; codex FAILs for a confirmed + protocol reason (not a bug — see below); gemini and deepseek reach the + server but fail for reasons not yet root-caused; antigravity SKIPs (no + known mechanism); all correctly classified by the now-dynamic + `scripts/test-local-llm-harnesses.ts`, which reads the live CLI registry + rather than a hand-maintained harness list.** Real findings, not + simulated: + - **opencode: PASS.** Genuinely round-tripped a "hello world" reply + through the real endpoint. + - **pi: PASS, after two real bugs found and fixed.** `PI_CONFIG_DIR` does + nothing for pi at all (grepped pi's entire bundled JS source — the + string appears nowhere); the real redirect is the child process's own + `HOME`, since pi hardcodes `~/.pi/agent/models.json` with no dedicated + override. Separately, pi's `models` field must be an **array** of + `{id}` objects, not an object keyed by id (confirmed against pi's own + bundled `docs/models.md`) — the object shape silently loaded zero + models. Also needs an explicit `--model custom/` on invocation. + - **grok: PASS, after the original recipe turned out to be flat-out + wrong**, not just unverified — the env-var recipe in this table's first + draft (`GROK_BASE_URL`/`XAI_API_KEY`/`GROK_MODEL`) produced "Not signed + in" against a real binary. Researched xAI's actual docs and corrected + to a `config.toml` with a `[model.]` block redirected via + `GROK_HOME`, with the key riding as an `env_key`-named env var — then + confirmed working end-to-end. + - **omp: PASS**, after the same two fixes as pi (array-shaped `models`, + `HOME`-redirect instead of `PI_CONFIG_DIR`) plus `--model custom/`. + Unverified against omp's own official docs (none are bundled in the + install), but empirically confirmed working live. + - **gemini: confirmed broken, unresolved after real investigation.** + Setting `GOOGLE_GEMINI_BASE_URL` makes gemini-cli internally select an + undocumented `AuthType.GATEWAY` path with validation requirements a + live run never satisfies (`Invalid auth method selected`, regardless of + key format). Tried and ruled out: a Google-format dummy key, + `GOOGLE_GENAI_USE_VERTEXAI=false`, a `GEMINI_DEFAULT_AUTH_TYPE` + override, and a hand-written `settings.json`. `--skip-trust` is a real, + separate fix for a different symptom (an untrusted-folder check + silently overriding `--approval-mode yolo`) and is kept, but does not + touch this auth failure. Left as an open, documented gap rather than + claimed as working. + - **deepseek: confirmed reaching the server, still failing, unresolved.** + A real run returns `dsh: HTTP_404: DeepSeek API error (HTTP 404)` + consistently — the env vars are read (the request reaches the network + rather than failing locally), but the root cause was not identified in + the time available. By analogy with codex's Responses-API gap, `dsh` + may expect DeepSeek's own API response shape rather than a generic + OpenAI-compatible one, but this was not confirmed by reading dsh's own + bundled source the way the pi/grok questions were resolved. Documented + as best-effort/unknown, matching its pre-existing lowest confidence tag. + - **codex: real bug found and fixed.** The recipe's TOML shape + (`[model].default`) was rejected by a real codex binary ("invalid + type: map, expected a string") — codex wants a top-level `model` + string plus `[model_providers.custom]`, and the API key rides as an + `env_key`-named env var, never a literal TOML field. Fixed in + `custom-model-injection.ts`, the standalone script, and both test + suites. **Then a second, deeper finding**: codex only speaks the + Responses API now (`wire_api = "responses"`, the only value it accepts + since dropping `"chat"` support in Feb 2026) — a real run against the + now-correctly-shaped config still failed (`Reconnecting...` × 5, then + "high demand" errors) because llama-swap doesn't implement + `/v1/responses`. This is a genuine, currently-unresolved protocol + incompatibility, not a bug in this PR's code — documented prominently + in `deployment_plan.md`'s confidence table. + - **claude: PASS, after two real bugs found and fixed.** (1) Claude + Code's async session-title-generation call also uses + `ANTHROPIC_DEFAULT_HAIKU_MODEL` and validates it against Claude's own + internal recognized-model list, printing `[claude-code:unrecognized_model]` + and, in `-p` mode, hanging the whole invocation rather than just + warning. `--settings '{"autoTitle":false}'` does NOT + stop it (confirmed); `--bare` does — the warning still prints, but the + real prompt now runs and returns the real answer. ⚠️ `--bare` is only + safe for this standalone one-shot test script — it also disables hooks, + LSP, plugin sync, and CLAUDE.md auto-discovery, so it must NEVER be + applied to a real interactive Codeman session (which depends on hooks + for idle detection, trust-dialog auto-accept, etc.). Whether an + INTERACTIVE session with a custom model hits the same hang (vs. just a + background warning, which would be harmless) is untested — flagged as + an open item for chunk 5/6, not assumed either way. (2) A separate, + genuinely nasty bug in the test script itself: a `POST` issued right + after a `GET` in the same Node process reliably HUNG indefinitely + against this real server (reproduced repeatedly: GET alone ~30ms, POST + alone ~1-2s, GET-then-immediate-POST times out completely; a 2s pause + between them fixed it every time) — looks like Node's fetch/undici + reusing a pooled keep-alive connection the server doesn't handle + cleanly for a second request right behind a first. Fixed with a 2s + pause between the script's discovery GET and its baseline POST. This + is a tooling-correctness fix (affects the script's own baseline check), + not a claim about how any CLI's own HTTP client behaves. + - **Also found and fixed**: an earlier design sent BOTH `Authorization: +Bearer` and `api-key` auth header conventions on every discovery/ + baseline request, on the theory that an unused header is harmless. + Live-tested against the real server, sending both reliably HUNG the + request (reproduced 3×: either header alone ~500-600ms, both together + no response inside 15s). Removed the `'both'` option entirely from + `CustomModelAuthStyle` (was `'bearer' | 'api-key' | 'both'`, now just + the first two, default `'bearer'`) — in the schema, the store type, the + discovery route, and the standalone script (`--auth-style` flag added). + This was a real, currently-shipped-in-this-PR bug fixed before it ever + reached anyone, not a pre-existing one. + +## Not yet done / open questions for review + +- **Chunk 6 (frontend)** — settings group, toolbar picker, tab badge — is + still entirely unbuilt; the feature is currently HTTP-API-only (see + `docs/custom-model-endpoints.md`). +- **Gemini is confirmed broken end-to-end** (`Invalid auth method + selected`, traced to an undocumented `GATEWAY` AuthType gemini-cli + selects once `GOOGLE_GEMINI_BASE_URL` is set) — needs upstream + investigation before it can be called supported. Documented in full in + `deployment_plan.md`'s confidence table rather than silently shipped as + working. +- **DeepSeek is confirmed reaching the server but failing** with a + consistent `HTTP_404`, root cause not identified — documented as + best-effort/unknown, same as its pre-existing lowest confidence tag. +- **Codex cannot work against a plain OpenAI-Chat-Completions server** + (llama.cpp/llama-swap/Ollama/vLLM's default) — it only speaks the + Responses API since Feb 2026. This is an external protocol + incompatibility, not something this PR can fix; codex support is real + only against a Responses-API-compatible endpoint. +- Antigravity has no known mechanism at all and stays unsupported. +- Chunk 5's session-restart design needs a careful look before merge: + switching a session's endpoint restarts its CLI process in place + (confirmed acceptable with the maintainer — these harnesses read + endpoint config at process start, not per-turn). Whether an INTERACTIVE + claude session with a custom model hits the same async-title-generation + hang the standalone script worked around with `--bare` (vs. just a + harmless background warning) is untested and should be checked before + calling claude's chunk 5 support done — `--bare` itself must never be + applied to a real interactive session, since it disables hooks Codeman + depends on. + +🤖 Generated with [Claude Code](https://claude.com/claude-code) diff --git a/deployment_plan.md b/deployment_plan.md new file mode 100644 index 00000000..a99010ab --- /dev/null +++ b/deployment_plan.md @@ -0,0 +1,363 @@ +# Custom Model Endpoint Profiles (all harnesses, local or cloud) + +## Context + +Devvyn pays for Claude Code but also runs a capable local model behind an +OpenAI-compatible server (llama.cpp) — and wants the same mechanism to work +against a **cloud** OpenAI-compatible endpoint too (e.g. Azure AI Foundry's +OpenAI-compatible inference endpoint, OpenRouter, a self-hosted gateway). +Right now every Codeman session mode defaults to its native cloud backend +with no way to redirect a session at any other endpoint from the UI — the +closest existing precedent is DeepSeek's server-env-sourced +`DEEPSEEK_BASE_URL`, which isn't user-facing. + +**Scope note**: this plan originally said "local LLM." It now covers any +OpenAI-compatible endpoint the user configures — local (llama.cpp, Ollama, +vLLM) or cloud (Azure AI Foundry, OpenRouter, a company gateway). The +mechanism is identical (a base URL Codeman probes via `GET /v1/models`); the +only real differences are auth-header convention (cloud endpoints often want +an `api-key` header, e.g. Azure, rather than `Authorization: Bearer`) and +that a cloud "model" may actually be a deployment name distinct from the +underlying model family (Azure AI Foundry deployments) — both are called out +where they matter below. Naming throughout this plan is **"custom model +endpoint,"** not "local model," to keep that scope explicit. + +### Additional use case: on-premises AI hardware + +"Local" isn't limited to a desktop running llama.cpp — a growing category of +purpose-built, on-premises AI hardware exists specifically to run a serious +model on-site with an OpenAI-compatible server, and this feature is exactly +the on-ramp for pointing Codeman at one: + +- **NVIDIA DGX Spark** (and the DGX Spark-class "Spark" mini-supercomputer + line) — a compact on-prem inference/training box aimed at running large + local models with an OpenAI-compatible API surface. +- **AMD "Strix Halo" (Ryzen AI Max)** on-prem AI mini-PCs — unified-memory + APU hardware marketed for local LLM inference, typically fronted by + llama.cpp/Ollama/vLLM the same way a home server would be. + +Neither needs anything new from this design: both present a standard +`/v1/models` + `/v1/chat/completions` OpenAI-compatible surface once the +inference server is running, so they're just another `baseUrl` entry in the +custom-model-hosts store, same as llama.cpp or a cloud endpoint. The +justification for building this generically (rather than hardcoding "point +Claude at my llama.cpp box") is precisely this: **the same endpoint registry +and per-CLI injection mechanism should work unmodified for any current or +future OpenAI-compatible box or service** — a home GPU rig today, a Spark or +Strix Halo appliance tomorrow, a company's on-prem inference cluster after +that — without Codeman needing to know or care what's actually serving the +model on the other end of that URL. + +A concrete example worth naming: **[Ark0N/Qwen5090](https://github.com/Ark0N/Qwen5090)** +(from the same GitHub account as this project's owner) is a one-click +Windows / one-command Linux installer that stands up Qwen3.8-27B locally on +an RTX 5090 (or another RTX 50-series card with ≥24GB) behind an +OpenAI-compatible API, served by any of vLLM, NInfer, or llama.cpp — MIT- +licensed tooling over Apache-2.0 Qwen weights. It's a direct, ready-made +target for this feature: point a custom-model-hosts entry at whichever +backend it's running, and it needs nothing further from Codeman's side. It's +also notable for already wiring up DeepSeek Harness and Claude Code as +coding agents against that local server itself, which is effectively the +same "point a Codeman-supported harness at a local endpoint" idea this +feature is generalizing — worth using as a real-world reference/test target +once chunk 5 (session integration) exists, alongside Devvyn's own llama.cpp +box. + +Each harness has its own (different-shaped) mechanism for pointing at a +custom OpenAI-compatible base URL + model — env vars for Claude, a JSON +config blob for opencode, a TOML file for Codex, etc. Devvyn gave the +starting recipes for those three; the rest (Gemini, Pi, Grok, DeepSeek, OMP, +Antigravity) were researched for this plan and are flagged by confidence +below. A real end-to-end pass against Devvyn's own llama-swap server +(`scripts/test-local-llm-harnesses.ts`, inside a `codeman/agent:llm-test` +Docker image with all 9 CLIs installed) then confirmed **claude and +opencode work end-to-end**, corrected a real Codex config.toml schema bug +the given recipe had (see the Codex row below), and surfaced that Codex's +_protocol_ — not just its config shape — does not work against a plain +OpenAI-Chat-Completions server like llama.cpp/llama-swap at all. Confidence +below reflects what was actually observed, not just what was planned. + +The feature must be: + +- **Off by default**, one settings toggle turns it on. +- Endpoint entry: user gives a base URL — a LAN address or a cloud URL — + plus an optional API key, and Codeman calls `GET /v1/models` to + discover and store the available model (or deployment) list. +- A **new toolbar selector** (separate from the existing Run-mode menu, since + it's a modifier on top of whichever harness is already selected/running) + lets the user pick "Cloud (default)" — the harness's own native backend — + or a model discovered from one of the configured custom endpoints. +- Picking a custom-endpoint model for an **already-running session restarts + that session's CLI process** with the injected env/config pointed at that + endpoint (confirmed with Devvyn — these harnesses read endpoint config at + process start, not per-turn, so a live hot-swap isn't possible). +- **New sessions always default back to the harness's native cloud backend.** + A custom-endpoint selection is a per-session override, not a sticky global + default — starting a fresh CLI (any mode) always launches against its + native backend unless the user explicitly picks a custom endpoint for that + new session too. The toolbar selector is scoped to "this session," never + carried forward as the default for future sessions. + +This follows the repo's existing data-driven CLI-registry philosophy +(`test/cli-registry-no-id-branching.test.ts`): per-CLI behavior is a +declared capability, never an `if (mode === 'claude')` branch. + +## Per-CLI injection recipes (confidence-ranked) + +| CLI | Mechanism | Confidence | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `claude` | Env vars: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, `ANTHROPIC_DEFAULT_SONNET_MODEL`/`_HAIKU_MODEL`/`_OPUS_MODEL` (all set to the chosen model/deployment name) | **Verified end-to-end** against a real llama-swap server — a real "hello world" reply came back. ⚠️ Non-interactive (`-p`) invocations also fire an async session-title-generation call that reuses `ANTHROPIC_DEFAULT_HAIKU_MODEL` and validates it against Claude Code's OWN internal recognized-model list, printing `[claude-code:unrecognized_model]` and, in `-p` mode, hanging the whole invocation rather than just warning. `--settings '{"autoTitle":false}'` does NOT stop this (confirmed); `--bare` does (the warning still prints, but the real prompt runs) — but `--bare` ALSO disables hooks, LSP, plugin sync, and CLAUDE.md auto-discovery, so it is only safe for the standalone one-shot test script, NEVER for a real interactive Codeman session (which depends on hooks for idle detection, trust-dialog auto-accept, etc. — see the External CLI modes section of CLAUDE.md). Whether an INTERACTIVE claude session with a custom model hits the same hang (vs. just a background warning) is untested and should be checked before calling chunk 5/6 done for claude | +| `opencode` | `OPENCODE_CONFIG_CONTENT` env var (already a registry mechanism, `stock.ts:342`) holding a JSON blob: `{"provider":{"custom":{"options":{"baseURL":...,"apiKey":...},"models":{"":{}}}},"model":"custom/"}` | **Verified by user** | +| `codex` | TOML `config.toml`: top-level `model = ""` + `[model_providers.custom]` (`base_url`, `env_key` naming an env var the real API key rides in — never a literal TOML field, since codex's schema has no such field). Written to an isolated dir via `CODEX_HOME` (`stock.ts:405-415`) so the user's own `~/.codex/config.toml` is never touched | **Config STRUCTURE verified** against a real codex binary (an earlier `[model].default` table shape was rejected: "invalid type: map, expected a string" — caught live). **Protocol CONFIRMED BROKEN against llama.cpp/llama-swap**: codex only speaks the Responses API (`wire_api = "responses"`, the only value it accepts since it dropped `"chat"` support in Feb 2026), and a real llama-swap server does not implement `/v1/responses` — a live run against it failed with repeated `Reconnecting...` then `high demand` errors. Codex support therefore needs a Responses-API-compatible endpoint (most local llama.cpp/Ollama/vLLM setups do not qualify); do not present this as working against a generic OpenAI-Chat-Completions box | +| `gemini` | Env vars `GOOGLE_GEMINI_BASE_URL` + `GEMINI_API_KEY` + `GEMINI_MODEL`; CLI needs a restart to pick them up | **Confirmed BROKEN against llama.cpp/llama-swap, unresolved after real investigation.** Setting `GOOGLE_GEMINI_BASE_URL` makes gemini-cli internally select an `AuthType.GATEWAY` auth path (undocumented — inferred from behaviour) with validation requirements distinct from every normal auth mode; a real run against llama-swap fails with `Invalid auth method selected` regardless of what key/format is supplied. Tried and all failed: a Google-format dummy API key, `GOOGLE_GENAI_USE_VERTEXAI=false`, a `GEMINI_DEFAULT_AUTH_TYPE` override, and hand-writing `settings.json` directly. `--skip-trust` was a real, separate fix (without it a trust-folder check silently overrides `--approval-mode yolo` back to `default`) but does not touch this auth failure. Documented as an open gap, not shipped as working — the registry entry and injection code exist and are exercised by the test script, but end-to-end gemini support needs upstream investigation of `GATEWAY` AuthType before it can be called done | +| `pi` | Config file `~/.pi/agent/models.json` with a custom provider whose `models` is an **array** of `{id}` objects (not an object keyed by id) plus `authHeader: true`. Redirected via the child process's own `HOME` env var, isolated per test/session — **not** `PI_CONFIG_DIR`, which does nothing for pi (grepped pi's entire bundled JS source: the string appears nowhere) | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back. Two real bugs found and fixed before this worked: (1) `PI_CONFIG_DIR` is not read by pi at all — pi hardcodes `~/.pi/agent/models.json` with no dedicated override, so the actual redirect has to be the child process's `HOME`; (2) `models` must be an array of `{id}` objects per pi's own bundled `docs/models.md`, not an object keyed by model id (silently loaded zero models). Also requires an explicit `--model custom/` on invocation — without it pi falls back to its own default provider and fails with "No API key found for the selected model" | +| `grok` | TOML `config.toml`: a fixed `[model.codeman-custom]` block (`base_url`, `env_key` naming an env var the key rides in, never a literal TOML field) written to an isolated dir via `GROK_HOME`. Invoked with `-m codeman-custom` | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back. The ORIGINAL recipe in this table (env vars `GROK_BASE_URL`/`XAI_API_KEY`/`GROK_MODEL`) was flat-out **wrong**, not just unverified: it produced "Not signed in" against a real binary. Grok's real mechanism, confirmed against xAI's own docs and a live binary, is a `config.toml` with a `[model.]` block, redirected via `GROK_HOME`; the key still rides as an env var (`XAI_API_KEY` via `env_key`), just referenced from the TOML rather than read directly | +| `deepseek` | Reuse the **existing** `DEEPSEEK_BASE_URL` + `DEEPSEEK_API_KEY` keys (already declared in `stock.ts`). Only `DEEPSEEK_BASE_URL` is in `privilegedEnvKeys` — `DEEPSEEK_API_KEY` deliberately stays clamp-exempt, since a non-granted owner supplying their OWN key removes privilege rather than granting it (adding it to the clamp list was a real regression, caught by `test/deepseek-mode.test.ts` and fixed before merge). No model-selection var — dsh model is a profile composition entry, not a flag/env var | **Confirmed reaching the server, but failing — unresolved.** A real run against llama-swap returns `dsh: HTTP_404: DeepSeek API error (HTTP 404)` consistently (confirmed the env vars are read: the request reaches the network rather than failing locally). Root cause not identified — plausible explanation by analogy with codex's Responses-API gap is that `dsh --profile headless` expects DeepSeek's official API response shape/path structure rather than a generic OpenAI-compatible `/v1/chat/completions` endpoint, but this was not confirmed by reading dsh's own bundled source (unlike pi/grok, where that grep resolved the question directly). Documented as best-effort/unknown, not shipped as verified working | +| `omp` | Config file `~/.omp/agent/models.yml` with the same array-shaped `models` + `authHeader: true` fix as pi. Redirected via `HOME`, same reasoning as pi (`PI_CONFIG_DIR` does not relocate omp's config either, despite an earlier CLAUDE.md note claiming it does) | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back, after applying the same two fixes as pi (array-shaped `models`, `HOME`-redirect instead of `PI_CONFIG_DIR`) plus an explicit `--model custom/` on invocation. Unverified against omp's own official docs (none are bundled in the install), but empirically confirmed working live | +| `antigravity` | No CLI/env/config mechanism found — Antigravity's docs describe only a GUI settings panel, and explicitly say a custom endpoint "cannot currently" become the core reasoning model. **Not implemented**; toolbar entry stays disabled for this mode with an explanatory tooltip | No known mechanism | + +Everything web-researched-but-unverified gets implemented but must be +smoke-tested against real installs of those CLIs before being called done — +call this out explicitly when implementing, don't just ship on faith. + +**Cloud-endpoint specifics** to keep in mind per recipe above: an Azure AI +Foundry-style endpoint typically wants the API key in an `api-key` header +rather than (or in addition to) `Authorization: Bearer`, and its "model" is +often a deployment name rather than the underlying model family name — the +discovery step (`GET /v1/models`) still works the same way against Azure AI +Foundry's OpenAI-compatible endpoint shape, but a user may need to type the +deployment name manually if it isn't returned as expected. + +## Architecture + +### 1. Registry: new `capabilities.customModelInjection` field + +Extend `src/config/cli-registry/types.ts` / `schema.ts` with a discriminated +union on each `CliEntry.capabilities`: + +```ts +type CustomModelInjection = + | { kind: 'env'; baseUrlVar: string; apiKeyVar: string; modelVars: string[] } + | { kind: 'configContentEnv'; envVar: string; template: 'opencode-json' } + | { + kind: 'configDir'; + dirEnvVar: string; + fileName: string; + template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml'; + } + | { kind: 'unsupported' }; +``` + +Declared per stock.ts entry per the table above. A pure function in a new +`src/custom-model-injection.ts` (`buildCustomModelInjection(entry, endpoint, modelId)`) +turns `(CliEntry, endpoint, modelId)` into either an `envOverrides` object +(kind `env`/`configContentEnv`) or a `{ dirEnvVar, files: [{path, content}] }` +descriptor (kind `configDir`) — unit-testable with no IO, mirroring how +`session-cli-builder.ts` is pure. The `configDir` kind additionally needs an +IO wrapper that writes those files under +`dataPath('custom-model-configs//')` (new dir, cleaned up on +session delete — same lifecycle as other per-session generated state). + +### 2. Endpoint registry: `src/custom-model-hosts.ts` + +Same read-array/write-array shape as `src/remote-hosts.ts` / +`src/webview-store.ts`: `~/.codeman/custom-model-hosts.json` holding +`CustomModelEndpoint[] = { id, label, baseUrl, apiKey?, authStyle?: 'bearer'|'api-key'|'both', models?: string[], lastDiscoveredAt? }`. +`authStyle` defaults to `'both'` (send both header conventions on the +discovery probe, same approach the smoke-test script below uses) so one +endpoint entry works whether it's llama.cpp or Azure without the user having +to know which header their box wants in advance. + +New route file `src/web/routes/custom-model-routes.ts` (registered in the +routes barrel), mirroring `case-routes.ts`'s remote/docker-host CRUD +(`GET/POST/PUT/DELETE /api/model-endpoints`, admin-gated in multi-user mode +the same way) plus: + +- `POST /api/model-endpoints/:id/discover-models` — fetches + `${baseUrl}/v1/models`, stores the `data[].id` list, returns it. Bounded + timeout, and run the target through the **same SSRF egress guard already + used for web tabs** (`webview-egress-policy.ts` — reject link-local/cloud + metadata addresses) — this still matters for a cloud URL too, since the + guard is about preventing a redirect to internal infra, not about + local-vs-cloud. + +**Why discovery rather than a free-text model field**: it removes the one +piece of configuration most likely to trip a user up — hand-typing the +exact model identifier a given inference server expects, which varies by +server and is an easy source of a silent "model not found" failure with no +useful error surfaced back through a CLI's own startup. Discovery also +means this design is not limited to a single-model box: a **multi-model +gateway** such as **[llama-swap](https://github.com/mostlygeek/llama-swap)** +(hot-swaps between several loaded llama.cpp model configs behind one +OpenAI-compatible endpoint) or a vLLM/LiteLLM/Ollama instance serving +several models advertises ALL of them through the same `/v1/models` call — +so one endpoint entry surfaces every model that gateway can serve, with no +extra per-model configuration on Codeman's side at all. + +### 3. Settings + +- New synced boolean `customModelEndpointsEnabled` in `SettingsUpdateSchema` + (`src/web/schemas.ts`), default `false`, documented inline like + `readMyMindEnabled`/`workspaceHooksEnabled`. +- New `.set-group` "Custom Model Endpoints" inside the **Agents & CLIs** + section (`settings-clis`, `index.html:2150+`) with the enable toggle plus + a list-editor (add/refresh-models/delete rows) for endpoints — closest + existing precedent is the respawn-presets array editor + (`schemas.ts:1285-1305`, `index.html:1243-1244`) for add/apply/delete-by-id + semantics, backed by the new CRUD routes above. + +### 4. Toolbar UI + +- New header/toolbar button (e.g. `#customModelBtn`, `btn-toolbar +btn-custom-model`), marker-hidden by default (`btn-custom-model--hidden`) + and revealed by `applyHeaderVisibilitySettings()` only when + `customModelEndpointsEnabled` is on — same pattern as the File + Viewer/Cron buttons. +- Clicking opens a dropdown (`#customModelMenu`, same `.run-mode-menu`-style + markup as the existing Run-mode gear menu) listing "Cloud (default)" plus + every discovered model, grouped by endpoint. An entry is disabled with a + tooltip when the active session's CLI has `customModelInjection.kind === +'unsupported'` (Antigravity) or none declared. +- Selecting an entry calls a new route: + `POST /api/sessions/:id/custom-model { endpointId, modelId } | { clear: true }`. + Server: resolve the CLI entry for `session.mode`, build the injection via + §1, persist it as a new `session.customModel` state field (surfaced in + `toState()`/SSE so the tab can show a small badge, e.g. "🖥 qwen3 (local)" + or "☁ gpt-4o-mini (azure)", and the choice survives reload), merge into + the session's `envOverrides`, and **respawn the pane's CLI process** + through the same respawn/interactive-restart path + `session.ts`/`tmux-manager.ts` already use for effort/model changes + (`_configureCliEnv()` + `applyEnvOverrides()` at spawn time) — reuse, + don't reinvent, the existing kill-and-relaunch-in-pane machinery. +- New-session creation deliberately does **not** inherit a prior custom- + endpoint choice: `buildEnvOverrides()` (session-ui.js) never carries the + toolbar selection forward to the next `run()` call. Every new session + starts on its native backend; picking a custom endpoint in the toolbar for + a session applies only to that session (and, if done before Run is + clicked, to the one session about to be created — not to sessions created + afterward). + +### 5. Multi-user security clamp + +Every new env var this feature introduces that can redirect a session's +traffic (and thus wherever its credentials go) — `ANTHROPIC_BASE_URL`, +`GOOGLE_GEMINI_BASE_URL`, `GROK_BASE_URL`, the `CODEX_HOME`/`PI_CONFIG_DIR` +dir-redirects, plus the already-privileged `DEEPSEEK_BASE_URL` — must be +added to each CLI's `capabilities.privilegedEnvKeys` so +`clampEnvOverridesForOwner()` strips them for a non-granted multi-user +owner, exactly the precedent already documented for `DEEPSEEK_BASE_URL`/ +`OMP_AUTH_BROKER_URL`. This matters _more_, not less, now that endpoints can +be cloud URLs: redirecting a non-granted user's session to an attacker's +cloud endpoint is a credential-exfiltration path, not just a mischief +redirect to a LAN box. Endpoint CRUD itself stays admin-only in multi-user +mode, same as remote/docker hosts. + +## Files touched (representative, not exhaustive) + +- `src/config/cli-registry/types.ts`, `schema.ts`, `stock.ts` — new capability + per-entry declarations +- `src/custom-model-injection.ts` (new) — pure per-CLI descriptor builder + unit tests +- `src/custom-model-hosts.ts` (new) — endpoint store +- `src/web/routes/custom-model-routes.ts` (new) — CRUD + discovery route +- `src/web/routes/session-routes.ts` — `POST /api/sessions/:id/custom-model`, clamp wiring +- `src/web/schemas.ts` — `customModelEndpointsEnabled`, endpoint/discover payload schemas, privileged-key updates +- `src/session.ts` — `customModel` state field, `toState()` surface +- `src/web/public/index.html`, `settings-ui.js`, `session-ui.js`, `styles.css` — settings group, toolbar button/menu, badge, accent CSS +- `src/web/sse-events.ts` + `constants.js` — if a dedicated SSE event is warranted for the badge (or just ride existing session-update broadcasts) +- `test/fixtures/mock-openai-server.ts` (new) + `test/custom-model-injection-contract.test.ts` (new) — see Mock-server validation below +- `scripts/test-local-llm-harnesses.ts` (already added, this branch; run via `npx tsx`) — the standalone real-CLI-and-real-endpoint smoke test, supporting any `--base-url` (local or cloud). Dynamic: derives its harness list and every env var/config it injects from the live CLI registry + `buildCustomModelInjection()` rather than a second hand-maintained copy — only the one-shot invocation flags (`ONE_SHOT` table) are CLI-specific info the registry doesn't model and stay hand-maintained +- `docs/custom-model-endpoints.md` (new) + a CLAUDE.md pointer bullet under External CLI modes / envOverrides + +## Mock-server validation strategy (CI-runnable, no real CLI binaries needed) + +Spawning nine real CLI binaries in CI isn't realistic, and neither Devvyn's +llama.cpp box nor a real cloud subscription can be a CI dependency. So the +injection _logic_ gets a tier of automated coverage that sits between the +pure unit tests and the live manual checks in Verification: + +1. **`test/fixtures/mock-openai-server.ts`** — a small in-process HTTP + server (plain `http.createServer`, no external deps, port picked per the + existing `const PORT = 3150+` convention) that: + - Serves `GET /v1/models` → a fixed fake model list (`{data:[{id:'qwen3'},...]}`), + for testing the discovery route. + - Serves `POST /v1/chat/completions` (OpenAI shape) **and** + `POST /v1/messages` (Anthropic Messages-API shape, since that's what + `ANTHROPIC_BASE_URL` traffic looks like) and records every request it + receives (headers, body, path) into an array the test can assert on — + including which auth header style it saw, so the `authStyle: 'both'` + default and Azure's `api-key` convention both get real coverage. + - Returns a minimal valid completion so a client library doesn't choke + on the response shape. + +2. **`test/custom-model-injection-contract.test.ts`** — for every CLI with a + `customModelInjection` capability (i.e. every row in the table above + except `antigravity`): + - Point a fixture `CustomModelEndpoint` at the mock server's URL. + - Call `buildCustomModelInjection(entry, endpoint, modelId)` (the pure + function from §1) to get the real env vars / config-file content that + would be injected into that CLI's session. + - Replay those exact values through a minimal HTTP request shaped the + way that CLI is documented to send it (Anthropic Messages shape for + claude; OpenAI chat-completions shape for opencode/codex/pi/grok/omp; + `GOOGLE_GEMINI_BASE_URL`'s OpenAI-compat shape for gemini; dsh's + provider call for deepseek) against the mock server. + - Assert the mock server received the request **at the injected + `baseUrl`**, with **the injected API key** in the expected header, and + **the injected model id** in the body/path — i.e. prove the values + Codeman computes are internally consistent and would reach the right + place with the right identifiers, end to end, in CI, on every push. + - Also cover the `configDir` kind (codex/pi/omp): assert the written + `config.toml`/`models.json`/`models.yml` file parses and contains the + same base URL/key/model, and that it's written under the isolated + per-session dir rather than the user's real config path. + +3. **Explicit, stated limitation** (goes in the test file's `@fileoverview` + and in this doc, not left implicit): this proves _"if the CLI honors its + documented env/config contract, it will hit the right endpoint with the + right model."_ It does **not** prove the real CLI binary actually reads + that env var / config file the way its docs say — that's still the job + of the live manual checks in Verification step 4-5 below, and is exactly + why the confidence table above did not stop at "researched" — every CLI + except antigravity (no mechanism at all) has since been run against a + real llama-swap server via `scripts/test-local-llm-harnesses.ts`: + claude/opencode/pi/grok/omp are confirmed PASS end-to-end, codex is + confirmed FAIL for a real documented protocol reason (Responses-API-only + since Feb 2026), and gemini/deepseek are confirmed reaching the server + but failing for reasons not yet root-caused (see their table rows). The + mock-server suite catches regressions in Codeman's own logic; it cannot + catch a CLI changing its env-var name in a future release, or a real + cloud endpoint behaving differently from a local llama.cpp box. + +## Verification + +1. `npm run typecheck && npm test` after each slice — this now includes the + mock-server contract suite from above, so injection-logic regressions + are caught automatically without touching real infrastructure. +2. Unit tests for `buildCustomModelInjection()` per CLI kind (pure, no IO). +3. Route tests (`app.inject`) for the new CRUD + discover-models endpoint + (mock `fetch` for `/v1/models`), and for the multi-user clamp on the new + privileged keys (mirror `test/routes/external-cli-bypass-clamp.test.ts`). +4. **Standalone real-binary smoke test**: `scripts/test-local-llm-harnesses.ts` + exercises every harness the CLI registry declares `customModelInjection` + support for against a real `--base-url` — local or cloud — outside of + Codeman's UI entirely, and is DYNAMIC (reads `enabledClis()` + calls the + real `buildCustomModelInjection()`, so a future registry change is picked + up automatically with zero edits to the script). Already run to + completion against Devvyn's llama-swap server (`http://10.10.11.241:8080`, + inside a `codeman/agent:llm-test` Docker image with all 9 CLI binaries): + claude/opencode/pi/grok/omp **PASS**, codex **FAILs as expected** + (Responses-API protocol gap, not a bug), gemini/deepseek **UNCONFIRMED** + (reach the server, fail for undiagnosed reasons — see their table rows), + antigravity **SKIP** (no mechanism). Re-run this against a real cloud + endpoint (e.g. an Azure AI Foundry deployment) once one is available, to + prove the `authStyle`/deployment-name handling holds up outside llama.cpp. +5. Once the full feature (not just the standalone script) is built: add an + endpoint via the real UI, hit discover-models, confirm the returned model + list, pick Claude + the model on a real session, confirm via + `tmux -L codeman capture-pane`/`tmux showenv -t ` that + `ANTHROPIC_BASE_URL`/`ANTHROPIC_API_KEY`/`ANTHROPIC_DEFAULT_*_MODEL` are + set post-restart, and confirm the endpoint's own logs show the next + prompt actually landing there. Repeat for opencode and Codex at minimum + before considering this shippable; spot-check the web-researched CLIs + and correct the plan's confidence table with what's actually observed. +6. `npm run lint && npm run format:check`. +7. Update `CHANGELOG.md`/changeset per the COM workflow when shipping. diff --git a/docs/custom-model-endpoints.md b/docs/custom-model-endpoints.md new file mode 100644 index 00000000..f6584367 --- /dev/null +++ b/docs/custom-model-endpoints.md @@ -0,0 +1,120 @@ +# Custom Model Endpoint Profiles + +Point any Codeman-supported harness — Claude, opencode, Codex, Gemini, Pi, +Grok, DeepSeek, or OMP — at a custom OpenAI-compatible endpoint instead of +its native cloud backend, for a given session. "Custom endpoint" covers both +**local** hardware (llama.cpp, Ollama, vLLM, a home GPU rig, or purpose-built +boxes like NVIDIA DGX Spark or AMD Strix Halo mini-PCs) and **cloud** +services (Azure AI Foundry's OpenAI-compatible endpoint, OpenRouter, a +company gateway) — anything answering `GET /v1/models` and +`POST /v1/chat/completions` in the standard shape. Design doc, per-CLI +recipe confidence table, and security reasoning: +[`deployment_plan.md`](../deployment_plan.md). + +> **Status**: backend is implemented and tested (registry capability, the +> injection engine, the endpoint store + discovery route, the session +> restart route). The toolbar picker / settings UI described below as the +> intended surface is **not yet built** — until it lands, use the HTTP API +> directly (examples below). Antigravity has no known custom-endpoint +> mechanism and is not supported. + +## Turning it on + +App Settings → Agents & CLIs → **Custom Model Endpoints** (synced setting +`customModelEndpointsEnabled`, default **OFF**). The API equivalent: + +```bash +curl -sk -X PUT https://localhost:3000/api/settings \ + -H 'Content-Type: application/json' \ + -d '{"customModelEndpointsEnabled": true}' +``` + +## Adding an endpoint + +```bash +curl -sk -X POST https://localhost:3000/api/model-endpoints \ + -H 'Content-Type: application/json' \ + -d '{"id": "llama-box", "label": "Home llama.cpp", "baseUrl": "http://192.168.1.50:8080"}' +``` + +`apiKey` is optional (most local servers don't check it). `authStyle` +(`bearer` | `api-key` | `both`, default `both`) controls which auth header +convention discovery uses — `both` works whether the endpoint is llama.cpp +(ignores the header) or a cloud gateway like Azure (wants `api-key`). + +Discover its available models: + +```bash +curl -sk -X POST https://localhost:3000/api/model-endpoints/llama-box/discover-models +``` + +This calls the endpoint's own `GET /v1/models` and stores the returned list +on the endpoint record; `GET /api/model-endpoints` lists everything +configured, `PUT`/`DELETE /api/model-endpoints/:id` update or remove one. +Endpoint management is admin-only in multi-user mode, same as remote/docker +hosts — these are machine-level infra, not per-user settings. + +## Applying a model to a session + +```bash +curl -sk -X POST https://localhost:3000/api/sessions//custom-model \ + -H 'Content-Type: application/json' \ + -d '{"endpointId": "llama-box", "modelId": "qwen3"}' +``` + +This computes the CLI-specific env vars / config for that session's mode +(see the recipe table in `deployment_plan.md`) and **restarts the session's +CLI process in place** — same pane, same tmux session, fresh env. That +restart is necessary, not incidental: every supported harness reads its +endpoint config at process start, not per-turn, so there is no live +hot-swap. Clear back to the harness's native cloud default with: + +```bash +curl -sk -X POST https://localhost:3000/api/sessions//custom-model \ + -H 'Content-Type: application/json' -d '{"clear": true}' +``` + +**New sessions always default back to the harness's native backend.** A +custom-endpoint selection is a per-session choice, never a sticky global +default — starting a fresh session doesn't inherit whatever the last one was +pointed at. + +## Confidence per harness + +Every harness except Antigravity has now been run end-to-end against a real +llama-swap server via `scripts/test-local-llm-harnesses.ts` (a dynamic +script that reads the live CLI registry, so a registry change is picked up +automatically). Results: + +- **Claude, opencode, Pi, Grok, OMP** — verified: a real "hello world" reply + came back through the endpoint. +- **Codex** — the config is structurally correct, but Codex only speaks the + Responses API since Feb 2026, which llama.cpp/llama-swap don't implement. + This is a real protocol incompatibility, not a bug here; Codex support + needs a Responses-API-compatible endpoint. +- **Gemini** — fails with `Invalid auth method selected`, traced to an + undocumented `GATEWAY` auth path gemini-cli selects once + `GOOGLE_GEMINI_BASE_URL` is set. Unresolved after real investigation + (several auth workarounds were tried and ruled out); do not rely on + Gemini support yet. +- **DeepSeek** — the request reaches the server (env vars are read) but + gets a consistent `HTTP_404`. Root cause not identified; best-effort only. +- **Antigravity** — no known custom-endpoint mechanism at all; unsupported. + +See the confidence table in `deployment_plan.md` for the full detail behind +each result. `scripts/test-local-llm-harnesses.ts` is the standalone script +used to check a harness against a real endpoint outside the web UI +entirely; see its own `--help` for usage. + +## Security note + +Every env var this feature can set that redirects a session's traffic +(`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, etc.) is +listed in that CLI's `privilegedEnvKeys` in the CLI registry, so a +non-granted multi-user owner cannot set one directly via the generic +`envOverrides` API field — only through this feature's own route, which +computes the value from an admin-configured, SSRF-guarded endpoint rather +than trusting arbitrary client input. See the "Multi-user security +hardening" section of `deployment_plan.md` for the full reasoning; several +of these were reachable via the generic `envOverrides` field even before +this feature existed, and building this surfaced and closed that gap. diff --git a/scripts/local-llm-test.config.example.json b/scripts/local-llm-test.config.example.json new file mode 100644 index 00000000..1b0b6113 --- /dev/null +++ b/scripts/local-llm-test.config.example.json @@ -0,0 +1,9 @@ +{ + "_comment": "Copy this file to local-llm-test.config.json (gitignored) and fill in your own values. CLI flags on scripts/test-local-llm-harnesses.mjs always override these. Any field can be omitted. apiKey is OPTIONAL — omit it entirely (or delete this line) for an endpoint like llama.cpp that doesn't check one; it defaults to a harmless placeholder either way.", + "baseUrl": "http://192.168.1.50:8080", + "model": "qwen3", + "apiKey": "", + "prompt": "Reply with exactly: hello world", + "timeout": 30000, + "only": [] +} diff --git a/scripts/test-local-llm-harnesses.ts b/scripts/test-local-llm-harnesses.ts new file mode 100644 index 00000000..96342a06 --- /dev/null +++ b/scripts/test-local-llm-harnesses.ts @@ -0,0 +1,699 @@ +#!/usr/bin/env -S npx tsx +/** + * Standalone smoke-test for pointing each Codeman-supported harness CLI at a + * custom OpenAI-compatible endpoint — local (llama.cpp, Ollama, vLLM, ...) or + * cloud (Azure AI Foundry's OpenAI-compatible endpoint, OpenRouter, a + * self-hosted gateway, ...). Anything that answers GET /v1/models and POST + * /v1/chat/completions in the standard shape qualifies; --base-url is not + * assumed to be a LAN address. + * + * This is intentionally OUTSIDE the npm test suite and outside Codeman's own + * session/tmux machinery: it spawns each real CLI binary directly, one-shot, + * with the env vars / config files that CLI's own docs say redirect it to a + * custom endpoint, and checks it can answer "hello world". + * + * DYNAMIC BY DESIGN: this file imports the SAME `enabledClis()` registry and + * `buildCustomModelInjection()` builder the production feature uses (see + * ../src/config/cli-registry/, ../src/custom-model-injection.ts, + * ../src/custom-model-injection-apply.ts) rather than keeping a second, + * hand-maintained copy of each CLI's env vars/config shape. A registry + * change (a new CLI, an edited env var name, a fixed config template) is + * picked up here automatically with zero edits to this file. Only the + * ONE-SHOT INVOCATION FLAGS (how to make each CLI answer one prompt and + * exit — information the registry doesn't model at all, since it only knows + * how to launch the interactive TUI) stay in the small ONE_SHOT table below; + * a CLI newly added to the registry with no ONE_SHOT entry is reported + * UNKNOWN rather than silently skipped or guessed at. + * + * Cloud endpoints often differ from a bare llama.cpp box in two ways this + * script accounts for: (1) auth may be an `api-key` header (Azure's + * convention) rather than `Authorization: Bearer` — see --auth-style below. + * (2) a cloud endpoint's "model" may actually be a deployment name distinct + * from the model family (Azure AI Foundry deployments) — always pass + * --model explicitly for those rather than relying on GET /v1/models + * discovery. + * + * IMPORTANT CONFIDENCE NOTE: claude and opencode are verified end-to-end + * against a real llama-swap server. codex's config STRUCTURE is verified, + * but it only speaks the Responses API (dropped Chat-Completions support + * Feb 2026) — expect it to fail against a plain OpenAI-compatible server, + * that's a real protocol gap, not a bug here. gemini/pi/grok/omp have their + * ONE-SHOT INVOCATION flags confirmed against real installed binaries' + * `--help` output, but their custom-endpoint env/config conventions remain + * web-researched, unverified. deepseek (dsh) is a profile launcher with no + * documented one-shot prompt flag at all — best-effort only. antigravity + * has no known CLI/env/config mechanism (GUI-only per public docs) — its + * registry entry declares `customModelInjection: { kind: 'unsupported' }`, + * which this script picks up dynamically and always skips. + * + * Usage: + * npx tsx scripts/test-local-llm-harnesses.ts --base-url http://192.168.1.50:8080 [options] + * npx tsx scripts/test-local-llm-harnesses.ts --base-url https://.services.ai.azure.com/openai/v1 --model --api-key $AZURE_AI_KEY + * + * Options: + * --base-url Required. Root URL of the OpenAI-compatible endpoint (local or cloud). + * --model Model/deployment id to request. Default: first from GET /v1/models. + * --api-key API key to send. Default: local-dummy-key (fine for llama.cpp; required for most cloud endpoints). + * --auth-style