diff --git a/.changeset/cli-catalog-consumers.md b/.changeset/cli-catalog-consumers.md new file mode 100644 index 000000000..a32197aef --- /dev/null +++ b/.changeset/cli-catalog-consumers.md @@ -0,0 +1,48 @@ +--- +'aicodeman': minor +--- + +`install.sh` and the Docker agent image now read the shipped CLI catalogue instead of +hand-maintaining their own lists. + +Adding a CLI to `src/config/cli-registry/stock.ts` and running +`npm run generate:cli-catalog` wires it into the installer's detection, its install menu and +its closing reminder, and into the agent image's npm layer. Previously each of those was a +separate hand-written list that had to be kept in step and was not: upstream `b6d0f1fa` is +"wire OMP into install.sh's CLI detection (it had none)", where a user with only `omp` +installed was told no AI CLI was found and offered Claude Code, and the section comment above +that code named six of the nine CLIs. + +The generator emits two committed artifacts, because neither consumer can import TypeScript: +`config/clis.stock.json` for the Docker build, and a marked block inside `install.sh` itself, +which runs via `curl | bash` before any checkout exists. The embedded copy is the FULL +catalogue: an earlier attempt fetched it and fell back to a hardcoded two-CLI list, degrading +silently on an empty response, and there is no degraded mode to fall into now — nor a network +fetch at all, since a `curl | bash` from master already carries a catalogue exactly as fresh as +the script itself. + +**Trust model is unchanged and now mechanical.** The server still never executes an entry's +install command. `install.sh` executes only commands embedded in itself — same file, same TLS +fetch, same commit as the `curl | bash` line that fetched it — and nothing pulled from the +network at install time is ever run, because nothing is fetched at install time at all. + +**The agent image respects `enabled`.** The generated catalogue carries that flag, so a CLI +shipping disabled is no longer baked into every image. It reads the stock catalogue rather than +the merged registry, so a user's `~/.codeman/clis.json` cannot change what is inside an image +tagged `codeman/agent:base`. + +User-visible changes, all in the installer: + +- The install menu is built from the catalogue, so it offers every enabled CLI with an install command that can drive a pane on its own — eight today, rather than the previous fixed two. Gemini had a command in the registry and appeared in no list in the script at all. DeepSeek is the one enabled CLI with a registry command that is deliberately NOT offered: `npm install -g @deepseek-ai/dsh` installs only the launcher, which ships no profile that can drive a terminal on its own, so choosing it used to leave the user with an AI CLI the installer considered "found" but that could not actually run anything. It still gets a hint pointing at its docs. +- Its entries use the registry's labels ("Claude" rather than "Claude Code"), the same trade already made for `codeman doctor` rows. A suffix map would just be the hand-maintained list again. +- On a `wget`-only host, only the menu entries that actually need `curl` are held back (still shown as copy-paste hints); the `npm install -g` entries, which never needed it, are unaffected. Rewriting `curl` to `wget` inside a string about to be executed is the wrong instinct either way. +- `CODEMAN_NONINTERACTIVE=1` still defaults to Claude Code, unchanged. + +`install.sh` remains bash 3.2 compatible (macOS ships it): parallel indexed arrays with +offset/length windows instead of delimiters, no associative arrays, namerefs, `mapfile` or +here-strings. CI now runs `bash -n`, executes the script inside a real `bash:3.2` container — +which is what catches expanding an empty array under `set -u`, a runtime abort `bash -n` cannot +see — and checks the generated artifacts are in sync. + +`docker/server.Dockerfile` is deliberately untouched; its narrower CLI list is now asserted as +a declared omission list so the divergence is visible rather than accidental. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d5cac857e..cdc637830 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -37,6 +37,35 @@ jobs: - name: Format check run: npm run format:check + # install.sh reaches users through `curl | bash` with nothing between it and + # them, and until now nothing in this repo checked it at all: no shellcheck, + # no bats, and the vitest gate is Node-only. + - name: install.sh syntax + run: bash -n install.sh + + # macOS ships bash 3.2 and this runner has bash 5, so the constructs that + # actually break a Mac install are invisible here without a container. This + # step is what catches them — in particular expanding an EMPTY array under + # `set -u`, which bash 3.2 treats as an unbound variable and `bash -n` + # cannot see because it is a runtime error, not a syntax one. + - name: install.sh runs on bash 3.2 (macOS's version) + run: | + set -euo pipefail + docker run --rm -v "$PWD":/w -w /w bash:3.2 bash -n /w/install.sh + docker run --rm -v "$PWD":/w -w /w -e CODEMAN_INSTALL_SH_LIB=1 bash:3.2 bash -c ' + set -euo pipefail + . /w/install.sh + detect_all_clis + # `shell` declares no binaries, so its offset/length window is length 0. + # Iterating it is the empty-array case; reaching here means it did not abort. + echo "bash $BASH_VERSION: ${#CLI_IDS[@]} CLIs, $CLI_FOUND_COUNT found" + cli_catalog_names >/dev/null + cli_catalog_print_install_hints >/dev/null + ' + + - name: CLI catalogue artifacts are in sync with stock.ts + run: npm run generate:cli-catalog -- --check + - name: Server boot smoke test run: | set -u diff --git a/CLAUDE.md b/CLAUDE.md index ed423971a..624cb876f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -105,6 +105,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph | Test coverage | `npm run test:coverage` | | Dead-code sweep | `npm run knip` (config in `config/knip.json`, passed via `--config`) | | Rebuild gesture overlay | `npm run build:gesture` (esbuild `packages/gesture-control/src/codeman/entry.ts` → `src/web/public/gesture/gesture-codeman.js`; commit the result) | +| Regenerate the CLI catalogue | `npm run generate:cli-catalog` (`--check` to fail on drift). Rewrites `config/clis.stock.json` **and** the marked block in `install.sh` from `stock.ts`. ⚠ **Commit both.** They are what `install.sh` and the Docker agent image read, since neither can import TypeScript; `test/cli-catalog-sync.test.ts` and a CI `--check` step fail if either goes stale. See `docs/cli-registry.md` | | Build the docker agent image | `node scripts/build-agent-image.mjs --no-cache` (builds `codeman/agent:base` from `docker/agent.Dockerfile`; prerequisite for Docker cases; `--engine`/`--image`). ⚠ **Always `--no-cache`** — a plain rebuild re-uses the cached `npm install -g` layer and silently keeps the CLIs frozen at their original versions, which once shipped a BROKEN codex while reporting success. See `docs/docker-cases.md` | | Gesture playground | `npm run dev` **in** `packages/gesture-control/` (standalone vite demo, fake tabs) | | Check public-asset formatting | `npm run check:public-assets` (prettier-checks `src/web/public/**` text assets; `scripts/check-public-assets.mjs`) | @@ -449,6 +450,6 @@ Two constraints worth knowing before you touch them: the env-derived PTY buffer ## Scripts & Tunnel -**`install.sh`** (repo root, 104KB) is the public entry point: `curl -fsSL | bash` installs Node/tmux if missing, clones to `~/.codeman/app`, builds, and offers a systemd/launchd service. The network-access prompt is 3-way: **Tailscale** (loopback bind + guided `tailscale serve --bg ` HTTPS setup: install/login/operator/tailnet-HTTPS-toggle, then curl-verified end-to-end), **LAN** (0.0.0.0 + password prompt), or **local-only**; it preserves the existing binding on re-runs via `read_existing_binding()`. Tailscale state is detected dynamically from `tailscale serve status --json` (no marker files); the installer must NEVER `tailscale serve reset` or touch serve mappings other than 443→Codeman's port (users have unrelated serve config). `install.sh update`, `install.sh uninstall`, and `install.sh tailscale` (retrofit Tailscale access onto an existing install) also exist; `CODEMAN_NONINTERACTIVE=1` approves system changes for automation, `CODEMAN_TAILSCALE=1` presets the Tailscale choice (never installs Tailscale non-interactively). +**`install.sh`** (repo root, ~112KB) is the public entry point: `curl -fsSL | bash` installs Node/tmux if missing, clones to `~/.codeman/app`, builds, and offers a systemd/launchd service. The network-access prompt is 3-way: **Tailscale** (loopback bind + guided `tailscale serve --bg ` HTTPS setup: install/login/operator/tailnet-HTTPS-toggle, then curl-verified end-to-end), **LAN** (0.0.0.0 + password prompt), or **local-only**; it preserves the existing binding on re-runs via `read_existing_binding()`. Tailscale state is detected dynamically from `tailscale serve status --json` (no marker files); the installer must NEVER `tailscale serve reset` or touch serve mappings other than 443→Codeman's port (users have unrelated serve config). `install.sh update`, `install.sh uninstall`, and `install.sh tailscale` (retrofit Tailscale access onto an existing install) also exist; `CODEMAN_NONINTERACTIVE=1` approves system changes for automation, `CODEMAN_TAILSCALE=1` presets the Tailscale choice (never installs Tailscale non-interactively). Its CLI knowledge is a GENERATED block (`npm run generate:cli-catalog`, markers in the file), not a hand-written list: detection, the install menu and the closing reminder all read it, which is what stops the class of bug upstream `b6d0f1fa` fixed by hand (a user with only omp installed being told no AI CLI was found). ⚠️ It must stay **bash 3.2** clean — macOS ships it and the documented install is `curl | bash` under `set -euo pipefail`, so `declare -A`, `mapfile`, namerefs, `${x,,}` and here-strings are all fatal there; CI runs `bash -n` plus a real `bash:3.2` container, since expanding an EMPTY array under `set -u` is a runtime abort `bash -n` cannot see. ⚠️ It executes ONLY commands from the embedded block (`CLI_INSTALL_CMD_TRUSTED`) — there is no network fetch of the catalogue at install time to worry about at all. Other key scripts: `scripts/tmux-manager.sh` (safe tmux mgmt), `scripts/tunnel.sh [quick|named] start|stop|status|url` (quick = random trycloudflare URL, default; `named setup|enable` = fixed-hostname tunnel via `scripts/codeman-tunnel-named.service`; bare `start|stop|url` still means quick), `scripts/run-beta.sh` (isolated beta instance), `scripts/build-agent-image.mjs` (docker base image), `scripts/self-update.sh` (detached updater). Production services: `scripts/codeman-web.service`, `scripts/codeman-tunnel.service`. **Always set `CODEMAN_PASSWORD`** before exposing via tunnel. diff --git a/config/clis.stock.json b/config/clis.stock.json new file mode 100644 index 000000000..899822173 --- /dev/null +++ b/config/clis.stock.json @@ -0,0 +1,281 @@ +[ + { + "id": "claude", + "label": "Claude", + "shortBadge": "CC", + "enabled": true, + "order": 0, + "kind": "agent", + "discovery": { + "binaries": [ + "claude" + ], + "searchDirs": [ + "~/.local/bin", + "~/.claude/local", + "/usr/local/bin", + "~/.npm-global/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "curl -fsSL https://claude.ai/install.sh | bash", + "darwin": "curl -fsSL https://claude.ai/install.sh | bash", + "wsl": "curl -fsSL https://claude.ai/install.sh | bash" + }, + "npmPackage": "@anthropic-ai/claude-code", + "docsUrl": "https://docs.claude.com/claude-code" + } + } + }, + { + "id": "shell", + "label": "Shell", + "shortBadge": "SH", + "enabled": true, + "order": 1, + "kind": "shell", + "discovery": { + "binaries": [], + "searchDirs": [], + "install": { + "command": {} + } + } + }, + { + "id": "opencode", + "label": "OpenCode", + "shortBadge": "OC", + "enabled": true, + "order": 10, + "kind": "agent", + "discovery": { + "binaries": [ + "opencode" + ], + "searchDirs": [ + "~/.opencode/bin", + "~/.local/bin", + "/usr/local/bin", + "~/go/bin", + "~/.bun/bin", + "~/.npm-global/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "curl -fsSL https://opencode.ai/install | bash", + "darwin": "curl -fsSL https://opencode.ai/install | bash" + }, + "npmPackage": "opencode-ai", + "docsUrl": "https://opencode.ai/docs" + } + } + }, + { + "id": "codex", + "label": "Codex", + "shortBadge": "CX", + "enabled": true, + "order": 20, + "kind": "agent", + "discovery": { + "binaries": [ + "codex" + ], + "searchDirs": [ + "~/.codex/bin", + "~/.local/bin", + "/usr/local/bin", + "~/.bun/bin", + "~/.npm-global/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "npm install -g @openai/codex", + "darwin": "npm install -g @openai/codex" + }, + "npmPackage": "@openai/codex", + "docsUrl": "https://developers.openai.com/codex/cli" + } + } + }, + { + "id": "gemini", + "label": "Gemini", + "shortBadge": "GM", + "enabled": true, + "order": 30, + "kind": "agent", + "discovery": { + "binaries": [ + "gemini" + ], + "searchDirs": [ + "~/.gemini/bin", + "~/.local/bin", + "/usr/local/bin", + "~/.bun/bin", + "~/.npm-global/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "npm install -g @google/gemini-cli", + "darwin": "npm install -g @google/gemini-cli" + }, + "npmPackage": "@google/gemini-cli", + "docsUrl": "https://github.com/google-gemini/gemini-cli" + } + } + }, + { + "id": "antigravity", + "label": "Antigravity", + "shortBadge": "AG", + "enabled": true, + "order": 40, + "kind": "agent", + "discovery": { + "binaries": [ + "agy" + ], + "searchDirs": [ + "~/.local/bin", + "~/.antigravity/bin", + "/usr/local/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "curl -fsSL https://antigravity.google/cli/install.sh | bash", + "darwin": "curl -fsSL https://antigravity.google/cli/install.sh | bash" + }, + "docsUrl": "https://antigravity.google/cli" + } + } + }, + { + "id": "pi", + "label": "Pi", + "shortBadge": "PI", + "enabled": true, + "order": 50, + "kind": "agent", + "discovery": { + "binaries": [ + "pi" + ], + "searchDirs": [ + "~/.local/bin", + "/usr/local/bin", + "~/.bun/bin", + "~/.npm-global/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "npm install -g --ignore-scripts @earendil-works/pi-coding-agent", + "darwin": "npm install -g --ignore-scripts @earendil-works/pi-coding-agent" + }, + "npmPackage": "@earendil-works/pi-coding-agent", + "docsUrl": "https://pi.dev", + "agentImageLayer": { + "kind": "dedicated", + "reason": "installed with --ignore-scripts in its own layer, so the flag cannot leak to the shared block" + } + } + } + }, + { + "id": "grok", + "label": "Grok", + "shortBadge": "GK", + "enabled": true, + "order": 70, + "kind": "agent", + "discovery": { + "binaries": [ + "grok" + ], + "searchDirs": [ + "~/.grok/bin", + "~/.local/bin", + "/usr/local/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "curl -fsSL https://x.ai/cli/install.sh | bash", + "darwin": "curl -fsSL https://x.ai/cli/install.sh | bash" + }, + "docsUrl": "https://github.com/xai-org/grok-build" + } + } + }, + { + "id": "deepseek", + "label": "DeepSeek", + "shortBadge": "DS", + "enabled": true, + "order": 80, + "kind": "agent", + "discovery": { + "binaries": [ + "dsh" + ], + "searchDirs": [ + "~/.local/bin", + "/usr/local/bin", + "~/.npm-global/bin", + "~/bin" + ], + "identity": { + "arg": "--help", + "regex": "DeepSeek\\s+Harness" + }, + "install": { + "command": { + "linux": "npm install -g @deepseek-ai/dsh", + "darwin": "npm install -g @deepseek-ai/dsh" + }, + "npmPackage": "@deepseek-ai/dsh", + "docsUrl": "https://github.com/deepseek-ai/deepseek-harness", + "agentImageLayer": { + "kind": "dedicated", + "reason": "needs pnpm alongside it (dsh plugin, issue #352) and a dsh-tui profile install" + } + } + } + }, + { + "id": "omp", + "label": "OMP", + "shortBadge": "OM", + "enabled": true, + "order": 90, + "kind": "agent", + "discovery": { + "binaries": [ + "omp" + ], + "searchDirs": [ + "~/.local/bin", + "~/.omp/bin", + "/usr/local/bin", + "~/.bun/bin", + "~/.npm-global/bin", + "~/bin" + ], + "install": { + "command": { + "linux": "curl -fsSL https://omp.sh/install | sh", + "darwin": "brew install can1357/tap/omp" + }, + "docsUrl": "https://omp.sh" + } + } + } +] diff --git a/docker/agent.Dockerfile b/docker/agent.Dockerfile index 26885c50b..1c7899e8f 100644 --- a/docker/agent.Dockerfile +++ b/docker/agent.Dockerfile @@ -26,13 +26,25 @@ RUN apt-get update \ openssh-client \ && rm -rf /var/lib/apt/lists/* -# The npm-published agent CLIs. Pinning is left to the rebuild cadence (see -# docs/docker-cases-plan.md, user-decision 2). -RUN npm install -g \ - @anthropic-ai/claude-code \ - @openai/codex \ - @google/gemini-cli \ - opencode-ai \ +# The npm-published agent CLIs, supplied by scripts/build-agent-image.mjs from +# config/clis.stock.json so a new stock CLI needs no edit here. The default is +# today's literal list, so a bare `docker build` still produces the same image. +# +# ⚠️ Expanded UNQUOTED on purpose: word splitting is what turns the list into +# several arguments. Every token is validated against +# ^[@A-Za-z0-9][@A-Za-z0-9/._-]*$ on the producing side +# (scripts/lib/cli-catalog.mjs) precisely because of that. +# +# ⚠️ Filtered on each entry's `enabled` flag, so a CLI that ships disabled is +# never baked into every image. +# +# Pinning is left to the rebuild cadence (see docs/docker-cases-plan.md, +# user-decision 2). +# ⚠️ The default is in REGISTRY order, byte-identical to what the generator emits. +# A different order is a different RUN string, which is a different layer hash and +# so a needless cache miss between a bare `docker build` and a scripted one. +ARG CLI_NPM_PACKAGES="@anthropic-ai/claude-code opencode-ai @openai/codex @google/gemini-cli" +RUN npm install -g ${CLI_NPM_PACKAGES} \ && npm cache clean --force # Antigravity (`agy`) is NOT on npm — Google ships a standalone binary through its @@ -46,7 +58,8 @@ RUN curl -fsSL https://antigravity.google/cli/install.sh | bash -s -- --dir /usr # Pi (pi.dev). Upstream documents --ignore-scripts (pi needs no lifecycle scripts); # kept out of the shared npm block above so the flag cannot silently change how the -# other four CLIs install. +# rest of that block's CLIs install — a fixed count would go stale here since +# CLI_NPM_PACKAGES (above) is now a generated, dynamic list rather than a hand-kept one. RUN npm install -g --ignore-scripts @earendil-works/pi-coding-agent \ && npm cache clean --force \ && pi --version diff --git a/docs/cli-registry.md b/docs/cli-registry.md index c84664efc..d60d73192 100644 --- a/docs/cli-registry.md +++ b/docs/cli-registry.md @@ -118,6 +118,58 @@ Treat those values as **transcribed, not authoritative** — nothing enforces th Everything else in the interface is live, including `overlays.remote` / `overlays.docker`, which back `defaultRemoteCommandForMode()` and `defaultDockerCommandForMode()` directly. Those two used to be hardcoded `Record<…CommandMode, string>` tables duplicating the registry with nothing keeping the two in step; `test/location-overlay-commands.test.ts` pins every resulting command as a literal string. +## Consumers outside the server + +Two things need the catalogue but cannot import TypeScript, so `npm run generate:cli-catalog` +(`scripts/generate-cli-catalog.mts`) emits two artifacts from `stock.ts`. Both are committed, +and `test/cli-catalog-sync.test.ts` fails if either drifts from a fresh generation. + +| Artifact | Consumer | Why it exists | +| ------------------------------------ | ---------------------------------- | ---------------------------------------------------------------------------------- | +| `config/clis.stock.json` | `scripts/lib/cli-catalog.mjs` (Docker build args), tests | A `.mjs` cannot import the registry. | +| a marked block inside `install.sh` | the installer itself | It runs via `curl \| bash` before any checkout exists, so it can read neither. | + +Only `id`, `label`, `shortBadge`, `enabled`, `order`, `kind` and `discovery` are exported. +`launch`, `env`, `capabilities` and `overlays` are spawn-time concerns the server alone +interprets, and a test asserts they never leak into the artifact — a second reading of the +launch model in a consumer that cannot be tested against a real spawn is exactly what this +registry exists to prevent. + +The install.sh copy is **embedded, not fetched**, and is the FULL catalogue. An earlier design +fetched it and fell back to a hardcoded two-CLI list, which degraded silently on an empty +response; there is no degraded mode to fall into now, and no network fetch either — a `curl | +bash` from master already carries a catalogue exactly as fresh as the script itself, so there is +nothing a refresh would buy that isn't already true. An earlier draft added an opt-in refresh +with a `TRUSTED`/`DISPLAY` array split to keep it from ever writing the executed command; it was +dropped before merge rather than shipped half-verified — the split's only actual write was the +label, `DISPLAY` never diverged from `TRUSTED` in practice, and the added surface (a second +array, a fetch path, three failure shapes to warn on) bought nothing the embedded copy didn't +already have. + +### The install-command trust boundary + +Three rules, and the middle one is why the embed matters: + +1. **The server never executes an entry's `install.command`.** Unchanged, and still enforced by nothing executing it: the field is display text (`CliDiscovery.install.command`). +2. **`install.sh` executes only commands embedded in itself.** Those arrive in the same file, over the same TLS fetch, in the same commit as the `curl \| bash` line that fetched the script — identical trust to the hardcoded vendor one-liners it replaces. +3. **Nothing fetched at install time is ever executed.** There is no second code path that fetches anything after the script itself has been fetched. + +That is mechanical rather than a promise. `CLI_INSTALL_CMD_TRUSTED` is written only from the +generated block and is the only array the installer ever runs or displays — there is no second +array a refresh could rewrite, because there is no refresh. `test/install-sh-invariants.test.ts` +asserts as much: the embedded commands are exactly the registry's, and nothing in `install.sh` +`eval`s. + +### bash 3.2 + +macOS ships bash 3.2 and the documented install is `curl -fsSL | bash` under +`set -euo pipefail`, so a bash-4 construct is not a warning there — it kills the install. The +generated block therefore uses parallel indexed arrays with **offset/length windows** into one +flat array instead of delimiters (a `$HOME` containing a space needs no `IFS` handling, and an +entry with nothing to contribute gets length 0 and is never iterated). CI runs `bash -n` and +executes the script inside a real `bash:3.2` container, because the empty-window case is a +runtime `set -u` abort that `bash -n` cannot see. + ## Resolve at call time, never at import Anything reading the registry must resolve it when it is asked, not when its module is first imported. `sessionModeSchema()`, `allowedEnvPrefixes()`, `dependencyRegistry()` and each resolver's `searchDirs` thunk all re-read the catalog per call. @@ -127,8 +179,10 @@ A module-level const freezes at first import, and the failure is asymmetric: a C ## Adding a CLI 1. Add a `CliEntry` to `stock.ts`. -2. Add a golden spawn-command pin to `test/cli-registry-spawn-golden.test.ts`, a row to `test/cli-capability-predicates.test.ts`, and its remote/docker commands to `test/location-overlay-commands.test.ts`. -3. That is usually all. If you find yourself wanting to add an `if` somewhere, the guard test will tell you — and the answer is a capability field, or a named profile if it genuinely needs to run code. +2. Run `npm run generate:cli-catalog` and commit **both** artifacts (`config/clis.stock.json` and `install.sh`). The installer's detection, its install menu, its reminder text and the Docker agent image all follow from that one step — this is what makes upstream `b6d0f1fa` ("wire OMP into install.sh's CLI detection, it had none") impossible rather than merely fixed. +3. Add a golden spawn-command pin to `test/cli-registry-spawn-golden.test.ts`, a row to `test/cli-capability-predicates.test.ts`, its remote/docker commands to `test/location-overlay-commands.test.ts`, and its search paths to `test/install-sh-detection-parity.test.ts`. +4. Only if it cannot install with a plain `npm install -g `: give it a layer in `docker/agent.Dockerfile` and a reason in `AGENT_IMAGE_SPECIAL_CASES` (`scripts/lib/cli-catalog.mjs`). The coverage test requires both, so an exclusion cannot quietly become an omission. +5. That is usually all. If you find yourself wanting to add an `if` somewhere, the guard test will tell you — and the answer is a capability field, or a named profile if it genuinely needs to run code. ## See also diff --git a/docs/docker-cases.md b/docs/docker-cases.md index 738d5a3de..f930c1269 100644 --- a/docs/docker-cases.md +++ b/docs/docker-cases.md @@ -21,6 +21,40 @@ The image is **secret-free**: credentials are delivered at runtime (bind mounts node scripts/build-agent-image.mjs --no-cache ``` +### Which CLIs the image contains + +The npm-published CLIs come from `ARG CLI_NPM_PACKAGES`, which `scripts/build-agent-image.mjs` +fills from `config/clis.stock.json` (generated from `src/config/cli-registry/stock.ts`). Adding +a stock CLI that installs with a plain `npm install -g` needs no Dockerfile edit. The ARG +defaults to the same list in the same order, so a bare `docker build` produces a byte-identical +layer — a different order would be a different `RUN` string and so a needless cache miss. + +⚠️ It reads the **stock** catalogue, never the merged registry. A user's `~/.codeman/clis.json` +must not change what is inside an image tagged `codeman/agent:base`, or two machines holding +that tag hold different images and every cache decision downstream is a lie. Each entry's +`enabled` flag IS honoured, so a CLI that ships disabled is never baked in. + +Five CLIs keep hand-written layers, because the registry cannot express what makes them +special (as a REGISTRY field now — `discovery.install.agentImageLayer` in `stock.ts` — rather +than an id-keyed table duplicated between the two producers of the image's build args): + +| CLI | Why it is not in the shared npm layer | +| ------------- | ------------------------------------------------------------------------------------- | +| `pi` | Installs with `--ignore-scripts`, kept in its own layer so the flag cannot leak to the others. | +| `deepseek` | Needs `pnpm` alongside it (`dsh plugin`, issue #352) plus a `dsh-tui` profile install. | +| `antigravity` | Not on npm — Google ships a standalone binary (~190MB, the largest layer). | +| `grok`, `omp` | Not on npm — standalone vendor installers. | + +`test/docker-agent-image-coverage.test.ts` requires every special case to carry a written +reason AND still be present in the Dockerfile, so an exclusion cannot silently become an +omission — which is the same failure upstream `b6d0f1fa` hit in `install.sh`. + +Two things build this image: `scripts/build-agent-image.mjs` (a human) and +`ensureAgentBaseImage()` in `src/docker-hosts.ts` (the app, on the first Docker case). They +assemble the argv independently, because a `.mjs` cannot import TypeScript, so +`test/agent-image-build-args-parity.test.ts` pins them together. Without it, an image built by +hand and one built by the app could hold different CLIs under the same tag. + A zero exit code only proves the layers ran, not that the toolchain works. Verify by actually executing each CLI in the image, and check the build log for `Using cache` lines: ```bash diff --git a/install.sh b/install.sh index 70d3cffd8..97467eff5 100755 --- a/install.sh +++ b/install.sh @@ -76,89 +76,35 @@ TS_NEED_ROOT="0" # explicit caller override so contributors can still fetch the browser if needed. export PUPPETEER_SKIP_DOWNLOAD="${PUPPETEER_SKIP_DOWNLOAD:-1}" -# Claude CLI search paths (from src/utils/claude-cli-resolver.ts) -CLAUDE_SEARCH_PATHS=( - "$HOME/.local/bin/claude" - "$HOME/.claude/local/claude" - "/usr/local/bin/claude" - "$HOME/.npm-global/bin/claude" - "$HOME/bin/claude" -) - -# OpenCode CLI search paths (from src/utils/opencode-cli-resolver.ts) -OPENCODE_SEARCH_PATHS=( - "$HOME/.opencode/bin/opencode" - "$HOME/.local/bin/opencode" - "/usr/local/bin/opencode" - "$HOME/go/bin/opencode" - "$HOME/.bun/bin/opencode" - "$HOME/.npm-global/bin/opencode" - "$HOME/bin/opencode" -) - -# Codex CLI search paths (from src/utils/codex-cli-resolver.ts) -CODEX_SEARCH_PATHS=( - "$HOME/.codex/bin/codex" - "$HOME/.local/bin/codex" - "/usr/local/bin/codex" - "$HOME/.bun/bin/codex" - "$HOME/.npm-global/bin/codex" - "$HOME/bin/codex" -) - -# Gemini CLI search paths (from src/utils/gemini-cli-resolver.ts) -GEMINI_SEARCH_PATHS=( - "$HOME/.gemini/bin/gemini" - "$HOME/.local/bin/gemini" - "/usr/local/bin/gemini" - "$HOME/.bun/bin/gemini" - "$HOME/.npm-global/bin/gemini" - "$HOME/bin/gemini" -) - -# Pi CLI search paths (from src/utils/pi-cli-resolver.ts) -PI_SEARCH_PATHS=( - "$HOME/.local/bin/pi" - "/usr/local/bin/pi" - "$HOME/.bun/bin/pi" - "$HOME/.npm-global/bin/pi" - "$HOME/bin/pi" -) - -# DeepSeek Harness search paths (from src/utils/deepseek-cli-resolver.ts) -DSH_SEARCH_PATHS=( - "$HOME/.local/bin/dsh" - "/usr/local/bin/dsh" - "$HOME/.npm-global/bin/dsh" - "$HOME/bin/dsh" -) - -# Grok CLI search paths (from src/utils/grok-cli-resolver.ts) -GROK_SEARCH_PATHS=( - "$HOME/.grok/bin/grok" - "$HOME/.local/bin/grok" - "/usr/local/bin/grok" - "$HOME/bin/grok" -) - -# Antigravity CLI search paths (from src/utils/antigravity-cli-resolver.ts) -ANTIGRAVITY_SEARCH_PATHS=( - "$HOME/.local/bin/agy" - "$HOME/.antigravity/bin/agy" - "/usr/local/bin/agy" - "$HOME/bin/agy" -) - -# OMP CLI search paths (from src/utils/omp-cli-resolver.ts's OMP_SEARCH_DIRS — -# ~/.local/bin leads, omp.sh's installer target; ~/.omp/bin is a fallback only) -OMP_SEARCH_PATHS=( - "$HOME/.local/bin/omp" - "$HOME/.omp/bin/omp" - "/usr/local/bin/omp" - "$HOME/.bun/bin/omp" - "$HOME/.npm-global/bin/omp" - "$HOME/bin/omp" -) + +# >>> BEGIN GENERATED CLI CATALOGUE +# Generated from src/config/cli-registry/stock.ts by scripts/generate-cli-catalog.mts. +# Do not edit by hand: run `npm run generate:cli-catalog` and commit the result. +# +# Parallel indexed arrays, bash 3.2 safe (no associative arrays, no nameref, no mapfile). +# The variable-length lists use OFFSET/LENGTH windows into one flat array rather than a +# delimiter, so a $HOME containing a space needs no IFS handling and an entry with nothing +# to contribute (shell has no binaries) gets length 0 and is simply never iterated. +# +# ⚠️ TRUST BOUNDARY: CLI_CMD_LINUX/CLI_CMD_DARWIN are the ONLY source of a command this +# script will ever execute, and they arrive embedded in this file — same TLS fetch, same +# commit as the script itself. Nothing fetched at install time is ever executed; there is +# no network refresh of these arrays. See cli_catalog_select_platform below. +CLI_IDS=('claude' 'shell' 'opencode' 'codex' 'gemini' 'antigravity' 'pi' 'grok' 'deepseek' 'omp') +CLI_LABELS=('Claude' 'Shell' 'OpenCode' 'Codex' 'Gemini' 'Antigravity' 'Pi' 'Grok' 'DeepSeek' 'OMP') +CLI_ENABLED=(1 1 1 1 1 1 1 1 1 1) +CLI_KIND=('agent' 'shell' 'agent' 'agent' 'agent' 'agent' 'agent' 'agent' 'agent' 'agent') +CLI_NPM=('@anthropic-ai/claude-code' '' 'opencode-ai' '@openai/codex' '@google/gemini-cli' '' '@earendil-works/pi-coding-agent' '' '@deepseek-ai/dsh' '') +CLI_DOCS=('https://docs.claude.com/claude-code' '' 'https://opencode.ai/docs' 'https://developers.openai.com/codex/cli' 'https://github.com/google-gemini/gemini-cli' 'https://antigravity.google/cli' 'https://pi.dev' 'https://github.com/xai-org/grok-build' 'https://github.com/deepseek-ai/deepseek-harness' 'https://omp.sh') +CLI_CMD_LINUX=('curl -fsSL https://claude.ai/install.sh | bash' '' 'curl -fsSL https://opencode.ai/install | bash' 'npm install -g @openai/codex' 'npm install -g @google/gemini-cli' 'curl -fsSL https://antigravity.google/cli/install.sh | bash' 'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' 'curl -fsSL https://x.ai/cli/install.sh | bash' '' 'curl -fsSL https://omp.sh/install | sh') +CLI_CMD_DARWIN=('curl -fsSL https://claude.ai/install.sh | bash' '' 'curl -fsSL https://opencode.ai/install | bash' 'npm install -g @openai/codex' 'npm install -g @google/gemini-cli' 'curl -fsSL https://antigravity.google/cli/install.sh | bash' 'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' 'curl -fsSL https://x.ai/cli/install.sh | bash' '' 'brew install can1357/tap/omp') +CLI_ALL_BINS=('claude' 'opencode' 'codex' 'gemini' 'agy' 'pi' 'grok' 'dsh' 'omp') +CLI_BIN_OFF=(0 1 1 2 3 4 5 6 7 8) +CLI_BIN_LEN=(1 0 1 1 1 1 1 1 1 1) +CLI_ALL_PATHS=("$HOME/.local/bin/claude" "$HOME/.claude/local/claude" "/usr/local/bin/claude" "$HOME/.npm-global/bin/claude" "$HOME/bin/claude" "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode" "/usr/local/bin/opencode" "$HOME/go/bin/opencode" "$HOME/.bun/bin/opencode" "$HOME/.npm-global/bin/opencode" "$HOME/bin/opencode" "$HOME/.codex/bin/codex" "$HOME/.local/bin/codex" "/usr/local/bin/codex" "$HOME/.bun/bin/codex" "$HOME/.npm-global/bin/codex" "$HOME/bin/codex" "$HOME/.gemini/bin/gemini" "$HOME/.local/bin/gemini" "/usr/local/bin/gemini" "$HOME/.bun/bin/gemini" "$HOME/.npm-global/bin/gemini" "$HOME/bin/gemini" "$HOME/.local/bin/agy" "$HOME/.antigravity/bin/agy" "/usr/local/bin/agy" "$HOME/bin/agy" "$HOME/.local/bin/pi" "/usr/local/bin/pi" "$HOME/.bun/bin/pi" "$HOME/.npm-global/bin/pi" "$HOME/bin/pi" "$HOME/.grok/bin/grok" "$HOME/.local/bin/grok" "/usr/local/bin/grok" "$HOME/bin/grok" "$HOME/.local/bin/dsh" "/usr/local/bin/dsh" "$HOME/.npm-global/bin/dsh" "$HOME/bin/dsh" "$HOME/.local/bin/omp" "$HOME/.omp/bin/omp" "/usr/local/bin/omp" "$HOME/.bun/bin/omp" "$HOME/.npm-global/bin/omp" "$HOME/bin/omp") +CLI_PATH_OFF=(0 5 5 12 18 24 28 33 37 41) +CLI_PATH_LEN=(5 0 7 6 6 4 5 4 4 6) +# <<< END GENERATED CLI CATALOGUE # ============================================================================ # Color Output @@ -448,193 +394,35 @@ check_build_tools() { [[ -z "$(missing_build_tools)" ]] } -check_claude() { - # Check PATH first - if command -v claude &>/dev/null; then - return 0 - fi - - # Check known install locations - for path in "${CLAUDE_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 - fi - done - - return 1 -} - -get_claude_path() { - if command -v claude &>/dev/null; then - command -v claude - return - fi - - for path in "${CLAUDE_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} - -check_opencode() { - if command -v opencode &>/dev/null; then - return 0 - fi - - for path in "${OPENCODE_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 - fi - done - - return 1 -} - -get_opencode_path() { - if command -v opencode &>/dev/null; then - command -v opencode - return - fi - - for path in "${OPENCODE_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} - -check_codex() { - if command -v codex &>/dev/null; then - return 0 - fi - - for path in "${CODEX_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 - fi - done - - return 1 -} - -get_codex_path() { - if command -v codex &>/dev/null; then - command -v codex - return - fi - - for path in "${CODEX_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} - -check_gemini() { - if command -v gemini &>/dev/null; then - return 0 - fi - - for path in "${GEMINI_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 - fi - done - - return 1 -} - -get_gemini_path() { - if command -v gemini &>/dev/null; then - command -v gemini - return - fi - - for path in "${GEMINI_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} - -check_antigravity() { - if command -v agy &>/dev/null; then - return 0 - fi - - for path in "${ANTIGRAVITY_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 - fi - done - - return 1 -} - -get_antigravity_path() { - if command -v agy &>/dev/null; then - command -v agy - return - fi - - for path in "${ANTIGRAVITY_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} - -# `pi` is a short, generic name (Raspberry Pi tooling, personal scripts), so the -# server-side resolver additionally probes `pi --version`. Detection here only feeds -# the "you have no AI CLI" hint, so a plain executable test is enough. -check_pi() { - if command -v pi &>/dev/null; then - return 0 - fi - - for path in "${PI_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then +# ============================================================================ +# CLI Detection (generic, driven by the generated catalogue above) +# ============================================================================ +# +# One implementation for every CLI, replacing nine near-identical +# check_/get__path pairs plus their nine search-path arrays. Those had +# to be extended by hand for each new CLI, and once were not: upstream b6d0f1fa +# is "wire OMP into install.sh's CLI detection (it had none)", where a user with +# only omp installed was told no AI CLI was found and offered Claude Code. +# Adding an entry to stock.ts now wires detection, the install menu and the +# closing reminder in one step. +# +# Probe order per CLI is UNCHANGED and pinned by +# test/install-sh-detection-parity.test.ts: the process PATH first (each declared +# binary name in turn), then each known install path, dir-major. + +# Index of "$1" in CLI_IDS -> CLI_IDX, returning 1 with CLI_IDX=-1 when unknown. +# A global rather than an echo because this runs inside loops, and a subshell per +# lookup is a fork per CLI per call site. +CLI_IDX=-1 +_cli_index() { + local want="$1" i + CLI_IDX=-1 + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + if [[ "${CLI_IDS[$i]}" == "$want" ]]; then + CLI_IDX=$i return 0 fi done - - return 1 -} - -get_pi_path() { - if command -v pi &>/dev/null; then - command -v pi - return - fi - - for path in "${PI_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} - -# `grok` has known squatters too (the unrelated @vibe-kit/grok-cli), so the -# server-side resolver additionally probes `grok --version`. Detection here only -# feeds the "you have no AI CLI" hint, so a plain executable test is enough. -check_grok() { - if command -v grok &>/dev/null; then - return 0 - fi - - for path in "${GROK_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 - fi - done - return 1 } @@ -650,89 +438,177 @@ check_grok() { dsh_banner_probe() { local runner=() if command -v timeout &>/dev/null; then runner=(timeout 5); fi - "${runner[@]}" "$1" --help /dev/null | grep -qi "DeepSeek Harness" -} - -# Resolved ONCE and memoized: the probe executes a possibly-foreign binary, and -# the check/get/reminder call sites together used to re-run the whole scan many -# times per install. -DSH_RESOLVE_DONE="" -DSH_RESOLVED_PATH="" -resolve_dsh() { - [[ -n "$DSH_RESOLVE_DONE" ]] && return 0 - DSH_RESOLVE_DONE=1 - local candidate path - if command -v dsh &>/dev/null; then - candidate="$(command -v dsh)" - if dsh_banner_probe "$candidate"; then - DSH_RESOLVED_PATH="$candidate" - return 0 + # ⚠️ bash 3.2 (stock macOS): expanding an EMPTY array under `set -u` is an unbound-variable + # error, not a no-op — `${runner[@]}` alone abort­ed this whole probe with "runner[@]: + # unbound variable" whenever `timeout` was absent (i.e. exactly the host this comment is + # about). `${runner[@]+"${runner[@]}"}` expands to nothing when the array is empty and to + # the quoted elements otherwise, which is safe under `set -u` in both bash 3.2 and 4+. + ${runner[@]+"${runner[@]}"} "$1" --help /dev/null | grep -qi "DeepSeek Harness" +} + +# Is "$2" really the CLI "$1" claims to be? +# +# Every CLI but DeepSeek is accepted on being executable, exactly as before. +# DeepSeek stays a hand-written special case ON PURPOSE: the registry expresses +# its identity check as `discovery.identity.regex`, a JavaScript regex, and +# translating that into a `grep` pattern at install time is a transformation +# nobody should be performing on a security-adjacent check. The parity test pins +# that the registry still demands "DeepSeek Harness", so an upstream banner +# change fails a test instead of silently mis-detecting here. +_cli_candidate_ok() { + case "$1" in + deepseek) dsh_banner_probe "$2" ;; + *) return 0 ;; + esac +} + +# Resolve every CLI in ONE pass, memoized. +# +# CLI_FOUND_PATH is parallel to CLI_IDS ('' when not found). CLI_FOUND_COUNT +# counts only ENABLED entries that have a binary to look for, which is what the +# "no AI CLI found" gate asks about — `shell` has no binary and must never make +# that gate think an agent is installed. +# +# Memoizing the whole scan generalises the old resolve_dsh memo: the three call +# sites together used to re-run every probe, and for dsh that meant executing a +# possibly-foreign binary repeatedly. +CLI_DETECT_DONE="" +CLI_FOUND_PATH=() +CLI_FOUND_COUNT=0 +detect_all_clis() { + [[ -n "$CLI_DETECT_DONE" ]] && return 0 + CLI_DETECT_DONE=1 + + local i j found bin path bin_end path_end + CLI_FOUND_COUNT=0 + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + found="" + + # 1. The process PATH, each declared binary name in turn. + bin_end=$((${CLI_BIN_OFF[$i]} + ${CLI_BIN_LEN[$i]})) + for ((j = ${CLI_BIN_OFF[$i]}; j < bin_end; j++)); do + bin="${CLI_ALL_BINS[$j]}" + if command -v "$bin" &>/dev/null; then + path="$(command -v "$bin")" + if _cli_candidate_ok "${CLI_IDS[$i]}" "$path"; then + found="$path" + break + fi + fi + done + + # 2. The known install locations, dir-major. Note this still runs when a + # PATH hit was REJECTED above — that is how a Debian `dsh` on PATH + # does not hide a real harness in ~/.local/bin. + if [[ -z "$found" ]]; then + path_end=$((${CLI_PATH_OFF[$i]} + ${CLI_PATH_LEN[$i]})) + for ((j = ${CLI_PATH_OFF[$i]}; j < path_end; j++)); do + path="${CLI_ALL_PATHS[$j]}" + if [[ -x "$path" ]] && _cli_candidate_ok "${CLI_IDS[$i]}" "$path"; then + found="$path" + break + fi + done fi - fi - for path in "${DSH_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]] && dsh_banner_probe "$path"; then - DSH_RESOLVED_PATH="$path" - return 0 + CLI_FOUND_PATH[$i]="$found" + if [[ -n "$found" ]] && [[ "${CLI_ENABLED[$i]}" == "1" ]] && [[ "${CLI_BIN_LEN[$i]}" -gt 0 ]]; then + CLI_FOUND_COUNT=$((CLI_FOUND_COUNT + 1)) fi done return 0 } -check_dsh() { - resolve_dsh - [[ -n "$DSH_RESOLVED_PATH" ]] +# Is this CLI installed? Unknown id is "no", never an error. +check_cli() { + detect_all_clis + _cli_index "$1" || return 1 + [[ -n "${CLI_FOUND_PATH[$CLI_IDX]}" ]] } -get_dsh_path() { - resolve_dsh - echo "$DSH_RESOLVED_PATH" +# Where it was found, or nothing. +get_cli_path() { + detect_all_clis + _cli_index "$1" || return 1 + printf '%s\n' "${CLI_FOUND_PATH[$CLI_IDX]}" } -get_grok_path() { - if command -v grok &>/dev/null; then - command -v grok - return - fi +# ---------------------------------------------------------------------------- +# Catalogue helpers +# ---------------------------------------------------------------------------- - for path in "${GROK_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return +# Pick this platform's install commands out of the generated per-platform arrays. +# +# ⚠️ THE TRUST BOUNDARY LIVES HERE, and it is mechanical rather than a promise: +# CLI_INSTALL_CMD_TRUSTED is written ONLY from CLI_CMD_LINUX/CLI_CMD_DARWIN, i.e. +# only from the block generated into this file, and it is the sole array the +# installer ever executes or displays — there is no second copy a network +# refresh could rewrite. A command that runs therefore arrived in the same +# file, over the same TLS fetch, in the same commit as the `curl | bash` line +# that fetched this script. That is identical trust to the hardcoded vendor +# one-liners this replaces, and it is why nothing fetched at install time is +# ever executed. The server keeps its own, stricter rule unchanged: it never +# executes an entry's install command at all (see CliDiscovery.install.command +# in src/config/cli-registry/types.ts). +CLI_INSTALL_CMD_TRUSTED=() +CLI_PLATFORM_DONE="" +cli_catalog_select_platform() { + [[ -n "$CLI_PLATFORM_DONE" ]] && return 0 + CLI_PLATFORM_DONE=1 + # detect_os ONCE, not per entry: it forks a subshell, and on an unsupported + # platform it also prints. Inside the loop that was ten forks and ten copies of + # the same error, because a `die` inside $( ) can only exit the subshell. + local i platform + platform="$(detect_os)" + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + if [[ "$platform" == "macos" ]]; then + CLI_INSTALL_CMD_TRUSTED[$i]="${CLI_CMD_DARWIN[$i]}" + else + CLI_INSTALL_CMD_TRUSTED[$i]="${CLI_CMD_LINUX[$i]}" fi done } -# `omp` is a short name too, so like grok/pi the server-side resolver -# additionally probes `omp --version`. Detection here only feeds the -# "you have no AI CLI" hint, so a plain executable test is enough. -check_omp() { - if command -v omp &>/dev/null; then - return 0 - fi - - for path in "${OMP_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - return 0 +# "Claude, OpenCode, Codex, ..." — the enabled, detectable CLIs, for prose. +cli_catalog_names() { + local i out="" + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + [[ "${CLI_ENABLED[$i]}" == "1" ]] || continue + [[ "${CLI_BIN_LEN[$i]}" -gt 0 ]] || continue + out="${out:+$out, }${CLI_LABELS[$i]}" + done + printf '%s' "$out" +} + +# The "install one yourself" hints: every enabled CLI that is not installed, +# showing the trusted install command. An entry with no install command gets +# its docs URL instead of being silently omitted, which is what used to +# happen to Gemini — it had a command in the registry and appeared in no list +# in this script. DeepSeek is the one entry that deliberately HAS a command in +# the registry but an empty one here: installing the launcher alone leaves +# nothing that can drive a pane, so the generator withholds the command for +# any launcherProfile entry (see installCommandFor in generate-cli-catalog.mts) +# and this hint falls through to the docs URL instead. +cli_catalog_print_install_hints() { + detect_all_clis + local i + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + [[ "${CLI_ENABLED[$i]}" == "1" ]] || continue + [[ "${CLI_BIN_LEN[$i]}" -gt 0 ]] || continue + [[ -z "${CLI_FOUND_PATH[$i]}" ]] || continue + if [[ -n "${CLI_INSTALL_CMD_TRUSTED[$i]}" ]]; then + echo -e " ${CYAN}${CLI_INSTALL_CMD_TRUSTED[$i]}${NC} # ${CLI_LABELS[$i]}" + elif [[ -n "${CLI_DOCS[$i]}" ]]; then + echo -e " ${CLI_LABELS[$i]}: see ${CYAN}${CLI_DOCS[$i]}${NC}" fi done - - return 1 } -get_omp_path() { - if command -v omp &>/dev/null; then - command -v omp - return - fi +# Resolved at load, not lazily: every element of CLI_INSTALL_CMD_TRUSTED has to +# exist before anything indexes it, or `set -u` aborts on an unset array element +# the first time a hint is printed. +cli_catalog_select_platform - for path in "${OMP_SEARCH_PATHS[@]}"; do - if [[ -x "$path" ]]; then - echo "$path" - return - fi - done -} check_cloudflared() { # Check ~/.local/bin first (matches tunnel-manager.ts resolution order) @@ -2368,118 +2244,128 @@ main() { fi fi - # AI CLI (Codeman drives one of: Claude Code, OpenCode, Codex, Gemini, Antigravity, Pi) - local has_claude=false - local has_opencode=false - local has_codex=false - local has_gemini=false - local has_antigravity=false - local has_pi=false - local has_grok=false - local has_dsh=false - local has_omp=false - + # AI CLI. Codeman drives one of the CLIs in the generated catalogue above; + # this used to be a hand-written list here, in the gate below, and in the + # closing reminder — three places that had to agree and did not (the comment + # itself named six of the nine). info "Checking AI CLI tools..." - if check_claude; then - has_claude=true - success "Claude Code found at $(get_claude_path)" - fi - if check_opencode; then - has_opencode=true - success "OpenCode found at $(get_opencode_path)" - fi - if check_codex; then - has_codex=true - success "Codex found at $(get_codex_path)" - fi - if check_gemini; then - has_gemini=true - success "Gemini CLI found at $(get_gemini_path)" - fi - if check_antigravity; then - has_antigravity=true - success "Antigravity CLI found at $(get_antigravity_path)" - fi - if check_pi; then - has_pi=true - success "Pi CLI found at $(get_pi_path)" - fi - if check_grok; then - has_grok=true - success "Grok CLI found at $(get_grok_path)" - fi - if check_dsh; then - has_dsh=true - success "DeepSeek Harness found at $(get_dsh_path)" - fi - if check_omp; then - has_omp=true - success "OMP CLI found at $(get_omp_path)" - fi + detect_all_clis + local i + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + [[ "${CLI_ENABLED[$i]}" == "1" ]] || continue + [[ "${CLI_BIN_LEN[$i]}" -gt 0 ]] || continue + if [[ -n "${CLI_FOUND_PATH[$i]}" ]]; then + success "${CLI_LABELS[$i]} found at ${CLI_FOUND_PATH[$i]}" + fi + done - if [[ "$has_claude" == "false" && "$has_opencode" == "false" && "$has_codex" == "false" && "$has_gemini" == "false" && "$has_antigravity" == "false" && "$has_pi" == "false" && "$has_grok" == "false" && "$has_dsh" == "false" && "$has_omp" == "false" ]]; then + if [[ "$CLI_FOUND_COUNT" -eq 0 ]]; then echo "" - warn "No AI CLI found. Codeman needs at least one: Claude Code, OpenCode, Codex, Antigravity, Gemini, Pi, Grok, DeepSeek Harness, or OMP." + warn "No AI CLI found. Codeman needs at least one: $(cli_catalog_names)." headless_guard "install an AI CLI (curl | bash from its vendor)" echo "" - echo -e " ${BOLD}Which AI CLI would you like to install?${NC}" - echo -e " ${CYAN}1)${NC} Claude Code (Anthropic)" - echo -e " ${CYAN}2)${NC} OpenCode (open-source)" - echo -e " ${CYAN}3)${NC} Both" - echo -e " ${CYAN}4)${NC} Skip (I'll install one myself, e.g. Codex, Antigravity, Gemini, Pi, Grok, DeepSeek Harness or OMP)" - echo "" - local cli_choice="" - if [[ "$NONINTERACTIVE" == "1" ]] || ! has_tty; then - # Explicit automation opt-in: default to Claude Code - cli_choice="1" - info "CODEMAN_NONINTERACTIVE=1: defaulting to Claude Code" + # The menu is built from the catalogue: every enabled CLI that is not + # installed and ships an install command we can run. It used to be a + # fixed four-option prompt offering Claude Code and OpenCode only, so the + # other seven were unreachable even though the registry knows how to + # install five of them. + # + # ⚠️ TRUST BOUNDARY: the command executed comes from CLI_INSTALL_CMD_TRUSTED, + # the only array the generated block above writes and the only one the + # installer ever runs or displays — see cli_catalog_select_platform. + # + # ⚠️ The registry's install commands are a MIX: some call `curl` directly + # (vendor one-liners), others are `npm install -g …`, which never needed + # curl at all. A wget-only host used to lose the WHOLE menu over this, + # including every npm entry — the two literals this replaced went through + # download_to_stdout and so honoured `wget`, and CODEMAN_NONINTERACTIVE=1 + # silently stopped defaulting to Claude Code as documented. Filter per + # entry instead: only a command that actually starts with `curl ` is + # curl-dependent, so only THOSE are held back on a wget-only host. + # Rewriting curl to wget inside a string about to be executed is the + # wrong instinct either way — the ones we can't run, we show as a hint. + local -a offer_idx=() + local curl_only_skipped=0 + for ((i = 0; i < ${#CLI_IDS[@]}; i++)); do + [[ "${CLI_ENABLED[$i]}" == "1" ]] || continue + [[ "${CLI_BIN_LEN[$i]}" -gt 0 ]] || continue + [[ -z "${CLI_FOUND_PATH[$i]}" ]] || continue + [[ -n "${CLI_INSTALL_CMD_TRUSTED[$i]}" ]] || continue + if [[ "${DOWNLOADER:-}" != "curl" ]] && [[ "${CLI_INSTALL_CMD_TRUSTED[$i]}" == curl\ * ]]; then + curl_only_skipped=$((curl_only_skipped + 1)) + continue + fi + offer_idx[${#offer_idx[@]}]=$i + done + + if [[ "$curl_only_skipped" -gt 0 ]]; then + warn "curl is not available, so $curl_only_skipped install command(s) that need it were left out of the menu below (still shown as hints if you skip)." + fi + + if [[ ${#offer_idx[@]} -eq 0 ]]; then + warn "No AI CLI can be installed automatically here. Codeman will run, but sessions need a CLI to drive." + cli_catalog_print_install_hints else - while true; do - echo -en "${CYAN}Choose [1/2/3/4]:${NC} " >&2 - read_reply cli_choice || { cli_choice="1"; break; } - case "$cli_choice" in - 1|2|3|4) break ;; - *) echo "Please enter 1, 2, 3, or 4." >&2 ;; - esac + echo -e " ${BOLD}Which AI CLI would you like to install?${NC}" + local n=0 idx + for idx in "${offer_idx[@]}"; do + n=$((n + 1)) + echo -e " ${CYAN}${n})${NC} ${CLI_LABELS[$idx]}" done - fi + echo -e " ${CYAN}s)${NC} Skip (I'll install one myself)" + echo "" - if [[ "$cli_choice" == "1" ]] || [[ "$cli_choice" == "3" ]]; then - info "Installing Claude Code CLI..." - download_to_stdout https://claude.ai/install.sh | bash - hash -r 2>/dev/null || true - if check_claude; then - has_claude=true - success "Claude Code installed at $(get_claude_path)" + local cli_choice="" + if [[ "$NONINTERACTIVE" == "1" ]] || ! has_tty; then + # Explicit automation opt-in: default to the first offered entry, + # which is registry order, which is Claude Code (order 0) — the + # same default this prompt has always taken non-interactively. + cli_choice="1" + info "CODEMAN_NONINTERACTIVE=1: defaulting to ${CLI_LABELS[${offer_idx[0]}]}" else - warn "Claude Code installation failed." + while true; do + echo -en "${CYAN}Choose [1-${n}, or s to skip]:${NC} " >&2 + read_reply cli_choice || { cli_choice="1"; break; } + case "$cli_choice" in + s|S) break ;; + ''|*[!0-9]*) echo "Please enter a number between 1 and ${n}, or s." >&2 ;; + *) + if [[ "$cli_choice" -ge 1 ]] && [[ "$cli_choice" -le "$n" ]]; then + break + fi + echo "Please enter a number between 1 and ${n}, or s." >&2 + ;; + esac + done fi - fi - if [[ "$cli_choice" == "2" ]] || [[ "$cli_choice" == "3" ]]; then - info "Installing OpenCode CLI..." - download_to_stdout https://opencode.ai/install | bash - hash -r 2>/dev/null || true - if check_opencode; then - has_opencode=true - success "OpenCode installed at $(get_opencode_path)" + if [[ "$cli_choice" == "s" ]] || [[ "$cli_choice" == "S" ]]; then + warn "Skipping AI CLI install. Codeman will run, but sessions need a CLI to drive." + cli_catalog_print_install_hints else - warn "OpenCode installation failed." + idx="${offer_idx[$((cli_choice - 1))]}" + info "Installing ${CLI_LABELS[$idx]}..." + # /dev/null || true + CLI_DETECT_DONE="" + detect_all_clis + if [[ -n "${CLI_FOUND_PATH[$idx]}" ]]; then + success "${CLI_LABELS[$idx]} installed at ${CLI_FOUND_PATH[$idx]}" + else + warn "${CLI_LABELS[$idx]} installation failed." + fi fi - fi - if [[ "$cli_choice" == "4" ]]; then - warn "Skipping AI CLI install. Codeman will run, but sessions need a CLI to drive." - info "Install one later, e.g.: npm install -g @openai/codex (Codex)" - info " or: curl -fsSL https://antigravity.google/cli/install.sh | bash (Antigravity)" - info " or: npm install -g --ignore-scripts @earendil-works/pi-coding-agent (Pi)" - info " or: curl -fsSL https://x.ai/cli/install.sh | bash (Grok)" - elif [[ "$has_claude" == "false" ]] && [[ "$has_opencode" == "false" ]]; then - die "The selected AI CLI failed to install. Install one manually and re-run the installer." + if [[ "$CLI_FOUND_COUNT" -eq 0 ]]; then + die "The selected AI CLI failed to install. Install one manually and re-run the installer." + fi fi fi + # cloudflared (optional — for remote/mobile access via Cloudflare Tunnel) info "Checking cloudflared (optional, for remote access)..." if check_cloudflared; then @@ -2775,19 +2661,10 @@ main() { echo -e " https://github.com/Ark0N/Codeman" echo "" - if ! check_claude && ! check_opencode && ! check_codex && ! check_gemini && ! check_antigravity && ! check_pi && ! check_grok && ! check_dsh && ! check_omp; then + detect_all_clis + if [[ "$CLI_FOUND_COUNT" -eq 0 ]]; then echo -e " ${YELLOW}${BOLD}Reminder:${NC} Install at least one AI CLI to start using Codeman:" - echo -e " ${CYAN}curl -fsSL https://claude.ai/install.sh | bash${NC} # Claude Code" - echo -e " ${CYAN}curl -fsSL https://opencode.ai/install | bash${NC} # OpenCode" - echo -e " ${CYAN}npm install -g @openai/codex${NC} # Codex" - echo -e " ${CYAN}curl -fsSL https://antigravity.google/cli/install.sh | bash${NC} # Antigravity" - echo -e " ${CYAN}npm install -g --ignore-scripts @earendil-works/pi-coding-agent${NC} # Pi" - echo -e " ${CYAN}curl -fsSL https://x.ai/cli/install.sh | bash${NC} # Grok" - echo -e " ${CYAN}curl -fsSL https://omp.sh/install | sh${NC} # OMP" - echo "" - echo -e " DeepSeek Harness has no vendor one-liner — install it from within Codeman" - echo -e " once the server is up (Run dropdown → Install DeepSeek Profile, or see" - echo -e " docs/deepseek-integration.md)." + cli_catalog_print_install_hints fi # Security notice — last informational block so it stays visible (when not @@ -2980,6 +2857,11 @@ uninstall() { echo "" } +# Sourcing guard: let the test harness load this file for its pure helpers +# without running an install. bash 3.2 cannot be exercised any other way from +# CI — see .github/workflows/ci.yml and test/install-sh-invariants.test.ts. +if [[ -n "${CODEMAN_INSTALL_SH_LIB:-}" ]]; then return 0 2>/dev/null || exit 0; fi + # Wrap in main to prevent partial execution on curl | bash case "${1:-}" in update) update ;; diff --git a/package.json b/package.json index a5b957155..bff93d6e6 100644 --- a/package.json +++ b/package.json @@ -13,6 +13,7 @@ "postinstall": "node scripts/postinstall.js", "build": "node scripts/build.mjs", "build:gesture": "node scripts/build-gesture-bundle.mjs", + "generate:cli-catalog": "tsx scripts/generate-cli-catalog.mts", "start": "NODE_COMPILE_CACHE=${HOME}/.codeman/compile-cache node dist/index.js", "dev": "tsx src/index.ts web", "web": "node dist/index.js web", diff --git a/scripts/build-agent-image.mjs b/scripts/build-agent-image.mjs index 00602d24d..271b24651 100644 --- a/scripts/build-agent-image.mjs +++ b/scripts/build-agent-image.mjs @@ -12,6 +12,7 @@ import { spawn, spawnSync } from 'node:child_process'; import { fileURLToPath } from 'node:url'; import { dirname, join } from 'node:path'; +import { agentImageBuildArgPairs, readCatalog } from './lib/cli-catalog.mjs'; const __dirname = dirname(fileURLToPath(import.meta.url)); const REPO_ROOT = join(__dirname, '..'); @@ -58,6 +59,13 @@ if (args.help) { const engine = resolveEngine(args.engine); const buildArgs = ['build', '-f', DOCKERFILE, '-t', args.image]; if (args.noCache) buildArgs.push('--no-cache'); +// The CLI list comes from the generated catalogue rather than the Dockerfile, so adding a +// stock CLI needs no edit in either. `src/docker-hosts.ts` assembles the same argv for the +// in-app auto-build; test/agent-image-build-args-parity.test.ts pins the two together, since +// two independent producers of one command line is exactly how they drift. +for (const [name, value] of agentImageBuildArgPairs(readCatalog())) { + buildArgs.push('--build-arg', `${name}=${value}`); +} buildArgs.push(REPO_ROOT); console.log(`[build-agent-image] ${engine} ${buildArgs.join(' ')}`); diff --git a/scripts/generate-cli-catalog.mts b/scripts/generate-cli-catalog.mts new file mode 100644 index 000000000..ce9cd7e43 --- /dev/null +++ b/scripts/generate-cli-catalog.mts @@ -0,0 +1,264 @@ +/** + * Regenerates the two CLI-catalogue artifacts from `src/config/cli-registry/stock.ts`, + * which stays the single source of truth. + * + * npm run generate:cli-catalog # rewrite both artifacts + * npm run generate:cli-catalog -- --check # exit 1 on drift, write nothing + * + * The artifacts exist because two consumers cannot import TypeScript: + * + * - `config/clis.stock.json` — read by `scripts/lib/cli-catalog.mjs` (a `.mjs` that feeds + * the Docker build args) and by the tests. + * - a generated block inside `install.sh` — the installer runs via `curl | bash` BEFORE any + * checkout exists, so it can read neither the registry nor the JSON. Its copy is embedded. + * + * ⚠️ The embedded copy is the FULL catalogue, deliberately. An earlier design fetched the + * JSON at install time and fell back to a hardcoded two-CLI list, which degraded silently on + * an empty response. There is no degraded mode to fall into now. + * + * ⚠️ Only fields the two consumers actually need are exported. `launch`, `env`, `capabilities` + * and `overlays` are spawn-time concerns the server alone interprets, and exporting them would + * invite a second implementation of the launch model outside the process that owns it. + * + * `test/cli-catalog-sync.test.ts` pins both artifacts against a fresh generation. + */ +import { readFileSync, writeFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { resolve } from 'node:path'; +import { STOCK_CLIS } from '../src/config/cli-registry/stock.js'; +import type { CliEntry } from '../src/config/cli-registry/types.js'; + +const JSON_PATH = fileURLToPath(new URL('../config/clis.stock.json', import.meta.url)); +const INSTALL_SH_PATH = fileURLToPath(new URL('../install.sh', import.meta.url)); + +const BEGIN_MARKER = '# >>> BEGIN GENERATED CLI CATALOGUE'; +const END_MARKER = '# <<< END GENERATED CLI CATALOGUE'; + +/** Platforms install.sh can be running on. `wsl`/`win32` resolve through the linux arm. */ +type InstallPlatform = 'linux' | 'darwin'; + +// --------------------------------------------------------------------------- +// config/clis.stock.json +// --------------------------------------------------------------------------- + +interface CatalogEntry { + id: string; + label: string; + shortBadge: string; + enabled: boolean; + order: number; + kind: string; + discovery: { + binaries: string[]; + searchDirs: string[]; + identity?: { arg: string; regex: string }; + install: { + command: Record; + npmPackage?: string; + docsUrl?: string; + agentImageLayer?: { kind: 'dedicated'; reason: string }; + }; + }; +} + +function toCatalogEntry(entry: CliEntry): CatalogEntry { + const { binaries, searchDirs, identity, install } = entry.discovery; + return { + id: entry.id as string, + label: entry.label, + shortBadge: entry.shortBadge, + // ⚠️ The field the previous attempt omitted, which is how a disabled CLI's npm package + // still got baked into every agent image. Every consumer filters on it. + enabled: entry.enabled, + order: entry.order, + kind: entry.kind, + discovery: { + binaries: [...binaries], + searchDirs: [...searchDirs], + ...(identity ? { identity: { arg: identity.arg, regex: identity.regex } } : {}), + install: { + command: { ...install.command } as Record, + ...(install.npmPackage ? { npmPackage: install.npmPackage } : {}), + ...(install.docsUrl ? { docsUrl: install.docsUrl } : {}), + ...(install.agentImageLayer ? { agentImageLayer: { ...install.agentImageLayer } } : {}), + }, + }, + }; +} + +export function renderCatalogJson(entries: CliEntry[] = STOCK_CLIS): string { + return `${JSON.stringify(entries.map(toCatalogEntry), null, 2)}\n`; +} + +// --------------------------------------------------------------------------- +// The install.sh block +// --------------------------------------------------------------------------- + +/** Single-quote a value for bash, escaping any embedded single quote. */ +function shQuote(value: string): string { + return `'${value.replace(/'/g, `'\\''`)}'`; +} + +/** + * A search dir as install.sh spells it. `~` becomes `$HOME` inside DOUBLE quotes so the shell + * expands it at load time, exactly as the hand-written arrays did; everything else is + * absolute and needs no expansion. + */ +function shPath(dir: string, binary: string): string { + const expanded = dir.startsWith('~/') ? `$HOME/${dir.slice(2)}` : dir; + return `"${expanded}/${binary}"`; +} + +/** + * The install command to run on `platform`, mirroring `resolveInstallCommandForPlatform()`: + * the exact platform, else linux, else whatever is declared. Resolved HERE, at generation + * time, so that fallback logic stays in tested TypeScript instead of being reimplemented in + * bash against an array the script would have to index by platform anyway. + * + * ⚠️ EMPTY for a `launcherProfile` entry (DeepSeek today), deliberately: `npm install -g + * @deepseek-ai/dsh` installs the LAUNCHER, not something that can drive a pane on its own — it + * ships only the `web`/`headless` profiles, neither of which is a terminal TUI. Emitting the + * command made the installer offer DeepSeek as a normal menu choice: picking it printed + * "DeepSeek installed at ...", counted as a found AI CLI, and left the user with a `dsh` that + * cannot actually run anything, with no mention of the Run dropdown's profile installer that + * fixes that. An empty command here means install.sh's menu-building loop (which requires a + * non-empty CLI_INSTALL_CMD_TRUSTED entry) skips it and the hint printer falls through to the + * docs URL instead — see cli_catalog_print_install_hints in install.sh. + */ +function installCommandFor(entry: CliEntry, platform: InstallPlatform): string { + if (entry.discovery.launcherProfile) return ''; + const { command } = entry.discovery.install; + return command[platform] ?? command.linux ?? Object.values(command)[0] ?? ''; +} + +export function renderInstallShBlock(entries: CliEntry[] = STOCK_CLIS): string { + const ids: string[] = []; + const labels: string[] = []; + const enabled: string[] = []; + const kinds: string[] = []; + const npm: string[] = []; + const docs: string[] = []; + const cmdLinux: string[] = []; + const cmdDarwin: string[] = []; + const allBins: string[] = []; + const binOff: number[] = []; + const binLen: number[] = []; + const allPaths: string[] = []; + const pathOff: number[] = []; + const pathLen: number[] = []; + + for (const entry of entries) { + ids.push(shQuote(entry.id as string)); + labels.push(shQuote(entry.label)); + enabled.push(entry.enabled ? '1' : '0'); + kinds.push(shQuote(entry.kind)); + npm.push(shQuote(entry.discovery.install.npmPackage ?? '')); + docs.push(shQuote(entry.discovery.install.docsUrl ?? '')); + cmdLinux.push(shQuote(installCommandFor(entry, 'linux'))); + cmdDarwin.push(shQuote(installCommandFor(entry, 'darwin'))); + + const { binaries, searchDirs } = entry.discovery; + binOff.push(allBins.length); + binLen.push(binaries.length); + for (const bin of binaries) allBins.push(shQuote(bin)); + + // Dir-major, matching the probe order the hand-written arrays used and + // `test/install-sh-detection-parity.test.ts` pins. + pathOff.push(allPaths.length); + let count = 0; + for (const dir of searchDirs) { + for (const bin of binaries) { + allPaths.push(shPath(dir, bin)); + count++; + } + } + pathLen.push(count); + } + + const arr = (name: string, values: Array): string => + values.length === 0 ? `${name}=()` : `${name}=(${values.join(' ')})`; + + return [ + BEGIN_MARKER, + '# Generated from src/config/cli-registry/stock.ts by scripts/generate-cli-catalog.mts.', + '# Do not edit by hand: run `npm run generate:cli-catalog` and commit the result.', + '#', + '# Parallel indexed arrays, bash 3.2 safe (no associative arrays, no nameref, no mapfile).', + '# The variable-length lists use OFFSET/LENGTH windows into one flat array rather than a', + '# delimiter, so a $HOME containing a space needs no IFS handling and an entry with nothing', + '# to contribute (shell has no binaries) gets length 0 and is simply never iterated.', + '#', + '# ⚠️ TRUST BOUNDARY: CLI_CMD_LINUX/CLI_CMD_DARWIN are the ONLY source of a command this', + '# script will ever execute, and they arrive embedded in this file — same TLS fetch, same', + '# commit as the script itself. Nothing fetched at install time is ever executed; there is', + '# no network refresh of these arrays. See cli_catalog_select_platform below.', + arr('CLI_IDS', ids), + arr('CLI_LABELS', labels), + arr('CLI_ENABLED', enabled), + arr('CLI_KIND', kinds), + arr('CLI_NPM', npm), + arr('CLI_DOCS', docs), + arr('CLI_CMD_LINUX', cmdLinux), + arr('CLI_CMD_DARWIN', cmdDarwin), + arr('CLI_ALL_BINS', allBins), + arr('CLI_BIN_OFF', binOff), + arr('CLI_BIN_LEN', binLen), + arr('CLI_ALL_PATHS', allPaths), + arr('CLI_PATH_OFF', pathOff), + arr('CLI_PATH_LEN', pathLen), + END_MARKER, + ].join('\n'); +} + +/** Replace the marked block in `source`, or throw if the markers are missing/malformed. */ +export function spliceInstallShBlock(source: string, block: string): string { + const begin = source.indexOf(BEGIN_MARKER); + const end = source.indexOf(END_MARKER); + if (begin === -1 || end === -1) { + throw new Error( + `install.sh is missing the generated-catalogue markers (${BEGIN_MARKER} / ${END_MARKER}). ` + + 'Add them once by hand; the generator only rewrites between them.' + ); + } + if (end < begin) throw new Error('install.sh has the catalogue markers in the wrong order.'); + return source.slice(0, begin) + block + source.slice(end + END_MARKER.length); +} + +// --------------------------------------------------------------------------- +// main +// --------------------------------------------------------------------------- + +/** + * ⚠️ Guarded so the module can be IMPORTED for its pure renderers without running. + * `test/cli-catalog-sync.test.ts` imports them, and an unguarded main would have that test + * rewrite the very artifacts it is supposed to be checking — passing always, guarding never. + */ +function isMainModule(): boolean { + const invoked = process.argv[1]; + if (!invoked) return false; + return fileURLToPath(import.meta.url) === resolve(invoked); +} + +function main(): void { + const check = process.argv.includes('--check'); + const wantJson = renderCatalogJson(); + const wantInstallSh = spliceInstallShBlock(readFileSync(INSTALL_SH_PATH, 'utf-8'), renderInstallShBlock()); + + if (check) { + const drift: string[] = []; + if (readFileSync(JSON_PATH, 'utf-8') !== wantJson) drift.push('config/clis.stock.json'); + if (readFileSync(INSTALL_SH_PATH, 'utf-8') !== wantInstallSh) drift.push('install.sh'); + if (drift.length > 0) { + console.error(`Out of date with stock.ts: ${drift.join(', ')}`); + console.error('Run `npm run generate:cli-catalog` and commit the result.'); + process.exit(1); + } + console.log('CLI catalogue artifacts are in sync with stock.ts.'); + } else { + writeFileSync(JSON_PATH, wantJson, 'utf-8'); + writeFileSync(INSTALL_SH_PATH, wantInstallSh, 'utf-8'); + console.log(`Wrote config/clis.stock.json and install.sh's catalogue block (${STOCK_CLIS.length} entries).`); + } +} + +if (isMainModule()) main(); diff --git a/scripts/lib/cli-catalog.mjs b/scripts/lib/cli-catalog.mjs new file mode 100644 index 000000000..6d2b8d6ab --- /dev/null +++ b/scripts/lib/cli-catalog.mjs @@ -0,0 +1,66 @@ +/** + * @fileoverview Reads the generated CLI catalogue for the Docker build. + * + * `scripts/build-agent-image.mjs` is a `.mjs` and cannot import the TypeScript registry, so it + * reads `config/clis.stock.json` (generated by `scripts/generate-cli-catalog.mts`) instead. + * The pure half lives here so `src/docker-hosts.ts`'s programmatic mirror of the same build + * command can be pinned against it by a test — those two produce the docker argv independently + * and must not drift. + */ +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; + +const CATALOG_PATH = fileURLToPath(new URL('../../config/clis.stock.json', import.meta.url)); + +/** + * npm package names the AGENT image installs in its shared `npm install -g` layer. + * + * PURE: takes the parsed catalogue, returns a sorted-by-registry-order list. + * + * ⚠️ Filters on `enabled`. That is the field the earlier attempt's export omitted, which is + * how a CLI that ships disabled still had its package baked into every image. + * + * ⚠️ An entry carrying `discovery.install.agentImageLayer` is excluded here and installed by + * its own hand-written Dockerfile layer instead, because the registry cannot express what + * makes it special — a flag, a companion package, or not being on npm at all. This used to be + * an id-keyed table duplicated between this file and `src/docker-hosts.ts` (exactly the shape + * `test/cli-registry-no-id-branching.test.ts` exists to forbid inside `src/`, which is why it + * was a blind spot rather than a pass — that test scans `src/` only). It is data now: both + * producers filter on the SAME field from the SAME catalogue entry, `reason` is required by + * `schema.ts`, and `test/docker-agent-image-coverage.test.ts` requires every one of them to + * still be present in the Dockerfile, so an exclusion cannot quietly become an omission. + */ +/** Tokens allowed in an npm package name reaching a Dockerfile build arg unquoted. */ +const SAFE_PACKAGE = /^[@A-Za-z0-9][@A-Za-z0-9/._-]*$/; + +export function agentImageNpmPackages(catalog) { + const packages = []; + for (const entry of catalog) { + if (!entry.enabled) continue; + if (entry.discovery?.install?.agentImageLayer) continue; + const pkg = entry.discovery?.install?.npmPackage; + if (!pkg) continue; // antigravity/grok/omp ship standalone installers, not npm + if (!SAFE_PACKAGE.test(pkg)) { + // The value is interpolated into a Dockerfile ARG that is expanded UNQUOTED (word + // splitting is how the list becomes several arguments), so a token with whitespace or + // shell metacharacters would change what the RUN line means. + // ⚠️ This exact regex is duplicated in `agentImageNpmPackages()` in + // `src/docker-hosts.ts` (that file cannot import this one — it is the TypeScript side of + // the same two-producers split this whole module exists for). Keep both literal patterns + // identical; `test/agent-image-build-args-parity.test.ts` pins that they are. + throw new Error(`Refusing unsafe npm package name for "${entry.id}": ${JSON.stringify(pkg)}`); + } + packages.push(pkg); + } + return packages; +} + +/** The `--build-arg` pairs the agent image takes. PURE. */ +export function agentImageBuildArgPairs(catalog) { + return [['CLI_NPM_PACKAGES', agentImageNpmPackages(catalog).join(' ')]]; +} + +/** Read the committed catalogue. IO. */ +export function readCatalog(path = CATALOG_PATH) { + return JSON.parse(readFileSync(path, 'utf-8')); +} diff --git a/src/config/cli-registry/schema.ts b/src/config/cli-registry/schema.ts index 10f6d18c4..a6ed5524d 100644 --- a/src/config/cli-registry/schema.ts +++ b/src/config/cli-registry/schema.ts @@ -187,6 +187,13 @@ const discoverySchema = z .strict(), npmPackage: z.string().max(200).optional(), docsUrl: z.url().optional(), + // Requires a `reason` on purpose — see the field's own doc comment in types.ts. A + // dedicated agent-image layer with no stated reason is a silent id-keyed special case + // rebuilding itself inside the data this change moved it out of. + agentImageLayer: z + .object({ kind: z.literal('dedicated'), reason: z.string().min(1).max(300) }) + .strict() + .optional(), }) .strict(), }) diff --git a/src/config/cli-registry/stock.ts b/src/config/cli-registry/stock.ts index e75428dae..29ce9065f 100644 --- a/src/config/cli-registry/stock.ts +++ b/src/config/cli-registry/stock.ts @@ -621,6 +621,10 @@ const PI: CliEntry = { }, npmPackage: '@earendil-works/pi-coding-agent', docsUrl: 'https://pi.dev', + agentImageLayer: { + kind: 'dedicated', + reason: 'installed with --ignore-scripts in its own layer, so the flag cannot leak to the shared block', + }, }, }, launch: { @@ -835,6 +839,10 @@ const DEEPSEEK: CliEntry = { }, npmPackage: '@deepseek-ai/dsh', docsUrl: 'https://github.com/deepseek-ai/deepseek-harness', + agentImageLayer: { + kind: 'dedicated', + reason: 'needs pnpm alongside it (dsh plugin, issue #352) and a dsh-tui profile install', + }, }, }, launch: { diff --git a/src/config/cli-registry/types.ts b/src/config/cli-registry/types.ts index 174993a99..0b5729867 100644 --- a/src/config/cli-registry/types.ts +++ b/src/config/cli-registry/types.ts @@ -230,6 +230,22 @@ export interface CliDiscovery { /** Package name for an npm-installable CLI. Display/tooling metadata only. */ npmPackage?: string; docsUrl?: string; + /** + * Present when the agent Docker image (`docker/agent.Dockerfile`) cannot install this + * CLI in the shared `npm install -g` layer with the rest and needs its own hand-written + * layer instead — a flag that would leak into the shared install (pi's `--ignore-scripts`), + * a companion package (deepseek's `pnpm`), or not being on npm at all (antigravity, grok, + * omp ship standalone installers). `reason` is REQUIRED, not decorative: it is what + * `test/docker-agent-image-coverage.test.ts` prints when a layer for this id goes missing + * from the Dockerfile, and it is what keeps this a data field rather than the id-keyed + * table it replaced (`AGENT_IMAGE_SPECIAL_CASE_IDS` in `docker-hosts.ts`, + * `AGENT_IMAGE_SPECIAL_CASES` in `scripts/lib/cli-catalog.mjs` — two copies kept in step by + * hand, outside stock.ts, which is exactly what this registry exists to prevent). + * `agentImageNpmPackages()` (docker-hosts.ts) and its `.mjs` mirror both filter on its + * presence rather than an id, so the shared npm layer and the special-case layers can never + * silently disagree about which CLI belongs in which. + */ + agentImageLayer?: { kind: 'dedicated'; reason: string }; }; } diff --git a/src/docker-hosts.ts b/src/docker-hosts.ts index 2cfa65024..d83ef866c 100644 --- a/src/docker-hosts.ts +++ b/src/docker-hosts.ts @@ -25,6 +25,7 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import fs from 'node:fs/promises'; import { dirname, isAbsolute, join, relative, resolve } from 'node:path'; import { enabledCliIds, getCli } from './config/cli-registry/registry.js'; +import { STOCK_CLIS } from './config/cli-registry/stock.js'; import { fileURLToPath } from 'node:url'; import { homedir } from 'node:os'; import { createHash } from 'node:crypto'; @@ -488,8 +489,68 @@ export function buildDockerCreateArgs(ctx: DockerCreateContext): string[] { * scripts/build-agent-image.mjs): `build -f -t [--no-cache] * `. Kept pure + unit-testable; the caller prepends the engine binary. */ -export function agentImageBuildArgs(dockerfile: string, image: string, contextDir: string, noCache = false): string[] { - return ['build', '-f', dockerfile, '-t', image, ...(noCache ? ['--no-cache'] : []), contextDir]; +export function agentImageBuildArgs( + dockerfile: string, + image: string, + contextDir: string, + noCache = false, + buildArgs: Array<[string, string]> = [] +): string[] { + return [ + 'build', + '-f', + dockerfile, + '-t', + image, + ...(noCache ? ['--no-cache'] : []), + ...buildArgs.flatMap(([name, value]) => ['--build-arg', `${name}=${value}`]), + contextDir, + ]; +} + +/** Tokens allowed in an npm package name reaching a Dockerfile build arg unquoted. */ +const SAFE_PACKAGE = /^[@A-Za-z0-9][@A-Za-z0-9/._-]*$/; + +/** + * npm packages the agent image installs in its shared layer, from the STOCK catalogue. + * + * ⚠️ Stock, deliberately, NOT the merged registry. A user's `~/.codeman/clis.json` must not + * change what lands inside an image tagged `codeman/agent:base`, or two machines holding that + * same tag hold different images and every cache-hit decision downstream is a lie. + * + * ⚠️ An entry carrying `discovery.install.agentImageLayer` is excluded here — see that field's + * doc comment in `types.ts` for why some CLIs need their own hand-written Dockerfile layer + * instead of the shared one, and `test/docker-agent-image-coverage.test.ts` for the guard that + * an exclusion here still lands in the Dockerfile somewhere. + * + * ⚠️ This mirrors `agentImageNpmPackages()` in `scripts/lib/cli-catalog.mjs`, which the CLI + * build path uses because a `.mjs` cannot import TypeScript. Two producers of one command + * line drift; `test/agent-image-build-args-parity.test.ts` is what stops them — including the + * SAFE_PACKAGE regex below, which is duplicated (not imported) in that file for the same + * reason and must stay byte-identical to it. + */ +export function agentImageNpmPackages(): string[] { + const packages: string[] = []; + for (const entry of STOCK_CLIS) { + if (!entry.enabled || entry.discovery.install.agentImageLayer) continue; + const pkg = entry.discovery.install.npmPackage; + if (!pkg) continue; + // The value is interpolated into a Dockerfile ARG expanded UNQUOTED (word splitting is + // how the list becomes several arguments), so a token with whitespace or shell + // metacharacters would change what the RUN line means. The source is `stock.ts`, so the + // practical risk is nil, but this is the in-app auto-build path and the only one of the + // two producers where that had gone unchecked. + if (!SAFE_PACKAGE.test(pkg)) { + throw new Error(`Refusing unsafe npm package name for "${String(entry.id)}": ${JSON.stringify(pkg)}`); + } + packages.push(pkg); + } + return packages; +} + +/** The `--build-arg` pairs the agent image takes. */ +export function agentImageBuildArgPairs(): Array<[string, string]> { + return [['CLI_NPM_PACKAGES', agentImageNpmPackages().join(' ')]]; } // ========== Credential mount resolution (IO) ========== @@ -1020,7 +1081,7 @@ function buildAgentImage( const argv = dockerEngineArgv(docker); const args = [ ...argv.slice(1), - ...agentImageBuildArgs(resolved.dockerfile, image, resolved.contextDir, opts.noCache), + ...agentImageBuildArgs(resolved.dockerfile, image, resolved.contextDir, opts.noCache, agentImageBuildArgPairs()), ]; return new Promise((resolve) => { // async spawn (NEVER spawnSync) so a multi-minute build never wedges the event loop. diff --git a/src/web/public/index.html b/src/web/public/index.html index 3de02b409..e3734285e 100644 --- a/src/web/public/index.html +++ b/src/web/public/index.html @@ -2894,7 +2894,7 @@

Docker

- Build it once with node scripts/build-agent-image.mjs. Contains node + claude/codex/gemini/opencode/agy/pi/grok/dsh + tmux. + Build it once with node scripts/build-agent-image.mjs. Contains node + claude/opencode/codex/gemini/agy/pi/grok/dsh/omp + tmux.
diff --git a/test/agent-image-build-args-parity.test.ts b/test/agent-image-build-args-parity.test.ts new file mode 100644 index 000000000..89c0cdd19 --- /dev/null +++ b/test/agent-image-build-args-parity.test.ts @@ -0,0 +1,98 @@ +/** + * @fileoverview The two producers of the agent-image `docker build` command line must agree. + * + * There are two, and there have to be: `scripts/build-agent-image.mjs` is what a human runs + * and is a `.mjs`, so it cannot import the TypeScript registry and reads the generated + * `config/clis.stock.json` instead; `src/docker-hosts.ts` builds the same command for the + * in-app auto-build on the first Docker case, from `STOCK_CLIS` directly. + * + * Two independent producers of one command line is exactly the shape that drifts, and the + * failure would be quiet and confusing: an image built by hand and an image built by the app + * would hold different CLIs under the SAME `codeman/agent:base` tag, so which CLIs a container + * has would depend on who built it. + * + * Port: none (pure). + */ + +import { describe, expect, it } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { + agentImageBuildArgPairs as mjsPairs, + agentImageNpmPackages as mjsPackages, +} from '../scripts/lib/cli-catalog.mjs'; +import { + agentImageBuildArgPairs as tsPairs, + agentImageBuildArgs, + agentImageNpmPackages as tsPackages, +} from '../src/docker-hosts.js'; + +const CATALOG = JSON.parse(readFileSync(fileURLToPath(new URL('../config/clis.stock.json', import.meta.url)), 'utf-8')); + +describe('agent-image build args: the .mjs and the TS mirror agree', () => { + it('resolve the same npm package list, in the same order', () => { + // Order matters as well as membership: a different order is a different RUN string, hence + // a different layer hash, hence a cache miss between the two build paths. + expect(tsPackages()).toEqual(mjsPackages(CATALOG)); + }); + + it('produce the same --build-arg pairs', () => { + expect(tsPairs()).toEqual(mjsPairs(CATALOG)); + }); + + it('render the same argv', () => { + // What the .mjs assembles by hand around its pairs, spelled out here so a change to + // either side's argv SHAPE (not just its values) fails too. + const pairs = tsPairs(); + const expected = [ + 'build', + '-f', + '/repo/docker/agent.Dockerfile', + '-t', + 'codeman/agent:base', + '--no-cache', + ...pairs.flatMap(([name, value]) => ['--build-arg', `${name}=${value}`]), + '/repo', + ]; + expect(agentImageBuildArgs('/repo/docker/agent.Dockerfile', 'codeman/agent:base', '/repo', true, pairs)).toEqual( + expected + ); + }); + + it('keeps --build-arg out of the argv when nothing is passed', () => { + // The parameter defaults to empty, so an existing caller that has not been updated still + // produces exactly the command it produced before. + expect(agentImageBuildArgs('/d', 'i', '/c')).toEqual(['build', '-f', '/d', '-t', 'i', '/c']); + }); + + it('resolves a non-empty list (anti-vacuity)', () => { + // Two empty lists compare equal very happily. + expect(tsPackages().length).toBeGreaterThan(3); + expect(tsPairs()[0][1].length).toBeGreaterThan(20); + }); + + it('matches the Dockerfile ARG default, so a bare `docker build` is cache-identical', () => { + const dockerfile = readFileSync(fileURLToPath(new URL('../docker/agent.Dockerfile', import.meta.url)), 'utf-8'); + const declared = /^ARG CLI_NPM_PACKAGES="([^"]*)"$/m.exec(dockerfile)?.[1]; + expect(declared, 'the Dockerfile no longer declares CLI_NPM_PACKAGES').toBeDefined(); + expect(declared).toBe(tsPackages().join(' ')); + }); + + it('validates an unsafe package name with the SAME regex on both sides', () => { + // Equal OUTPUT on today's catalogue (asserted above) does not prove equal VALIDATION — a + // looser regex on one side would only show up the day someone ships a hostile package name. + // The regex is duplicated rather than shared (the .mjs side cannot import the .ts side, the + // whole reason this file exists), so pin the literal PATTERN text is identical between the + // two source files rather than trusting the comment that says so. + const tsSource = readFileSync(fileURLToPath(new URL('../src/docker-hosts.ts', import.meta.url)), 'utf-8'); + const mjsSource = readFileSync(fileURLToPath(new URL('../scripts/lib/cli-catalog.mjs', import.meta.url)), 'utf-8'); + const extract = (source: string, file: string): string => { + // Non-greedy to `/;` deliberately: the pattern itself contains a `/` (inside the + // character class), so a naive `[^/]+` stops at the wrong slash. + const m = /const SAFE_PACKAGE = (\/.+?\/);/.exec(source); + expect(m, `could not find the SAFE_PACKAGE regex literal in ${file}`).toBeDefined(); + return m![1]; + }; + expect(extract(tsSource, 'docker-hosts.ts')).toBe(extract(mjsSource, 'cli-catalog.mjs')); + }); +}); diff --git a/test/cli-catalog-sync.test.ts b/test/cli-catalog-sync.test.ts new file mode 100644 index 000000000..fcb932973 --- /dev/null +++ b/test/cli-catalog-sync.test.ts @@ -0,0 +1,80 @@ +/** + * @fileoverview Pins the two generated CLI-catalogue artifacts against a fresh generation. + * + * `config/clis.stock.json` and the marked block inside `install.sh` are both derived from + * `src/config/cli-registry/stock.ts`. Generated files that are committed rot the moment + * someone edits the source and forgets the generator, and the failure is silent in the worst + * possible way: the installer keeps detecting the OLD set of CLIs while the server offers the + * new one. Same class as the drift this whole change exists to remove, just moved one level + * out. + * + * ⚠️ The renderers are imported from the generator, which means the generator's `main()` must + * stay behind its `isMainModule()` guard. Without it, importing this module would rewrite the + * artifacts as a side effect of checking them — the test would pass unconditionally and + * guard nothing. + * + * Port: none (pure, over two files and the registry). + */ + +import { describe, expect, it } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { renderCatalogJson, renderInstallShBlock, spliceInstallShBlock } from '../scripts/generate-cli-catalog.mts'; +import { STOCK_CLIS } from '../src/config/cli-registry/stock.js'; + +const REGENERATE = 'Run `npm run generate:cli-catalog` and commit the result.'; + +const jsonPath = fileURLToPath(new URL('../config/clis.stock.json', import.meta.url)); +const installShPath = fileURLToPath(new URL('../install.sh', import.meta.url)); + +describe('generated CLI catalogue artifacts', () => { + it('config/clis.stock.json matches a fresh generation', () => { + expect(readFileSync(jsonPath, 'utf-8'), `config/clis.stock.json is stale. ${REGENERATE}`).toBe(renderCatalogJson()); + }); + + it("install.sh's generated block matches a fresh generation", () => { + const current = readFileSync(installShPath, 'utf-8'); + expect(current, `install.sh's catalogue block is stale. ${REGENERATE}`).toBe( + spliceInstallShBlock(current, renderInstallShBlock()) + ); + }); + + it('exports every stock CLI, carrying the enabled flag', () => { + const exported = JSON.parse(readFileSync(jsonPath, 'utf-8')) as Array<{ id: string; enabled: boolean }>; + expect(exported.map((e) => e.id)).toEqual(STOCK_CLIS.map((e) => e.id as string)); + // The field the previous attempt omitted, which let a disabled CLI's npm package be baked + // into every agent image. Its PRESENCE is the contract; its value is whatever stock says. + for (const entry of exported) { + expect(typeof entry.enabled, `${entry.id} has no enabled flag`).toBe('boolean'); + } + }); + + it('exports no spawn-time fields', () => { + // launch/env/capabilities/overlays are the server's alone. Exporting them would invite a + // second reading of the launch model in a consumer that cannot be tested against a spawn. + const raw = readFileSync(jsonPath, 'utf-8'); + for (const forbidden of ['"launch"', '"env"', '"capabilities"', '"overlays"']) { + expect(raw.includes(forbidden), `${forbidden} leaked into the exported catalogue`).toBe(false); + } + }); + + it('splices only between the markers (anti-clobber)', () => { + // The generator rewrites a window, not the file. If the splice ever widened, it would eat + // hand-written installer code on the next run and nothing else here would notice. + const current = readFileSync(installShPath, 'utf-8'); + const spliced = spliceInstallShBlock( + current, + '# >>> BEGIN GENERATED CLI CATALOGUE\n# <<< END GENERATED CLI CATALOGUE' + ); + expect(spliced.startsWith(current.slice(0, current.indexOf('# >>> BEGIN GENERATED CLI CATALOGUE')))).toBe(true); + expect( + spliced.endsWith( + current.slice(current.indexOf('# <<< END GENERATED CLI CATALOGUE') + '# <<< END GENERATED CLI CATALOGUE'.length) + ) + ).toBe(true); + }); + + it('refuses a file with no markers rather than appending', () => { + expect(() => spliceInstallShBlock('#!/usr/bin/env bash\necho hi\n', 'block')).toThrow(/markers/); + }); +}); diff --git a/test/docker-agent-image-coverage.test.ts b/test/docker-agent-image-coverage.test.ts new file mode 100644 index 000000000..a6b8b0cab --- /dev/null +++ b/test/docker-agent-image-coverage.test.ts @@ -0,0 +1,161 @@ +/** + * @fileoverview Every shipped CLI reaches the Docker agent image, and no unshipped one does. + * + * The image's npm layer is now a build arg fed from the generated catalogue, but four CLIs + * still install through hand-written layers because the registry cannot describe what makes + * them special — a flag, a companion package, or not being on npm at all. That mix is fine; + * what is not fine is a CLI landing in `stock.ts` and reaching NEITHER, which is upstream + * `b6d0f1fa` (omp shipped with no installer wiring) in the image instead of the installer. + * + * So this asserts total coverage rather than checking the arg alone, and requires every + * special case to carry a written reason. + * + * Port: none (pure, over two Dockerfiles, the catalogue and the registry). + */ + +import { describe, expect, it } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { agentImageNpmPackages } from '../scripts/lib/cli-catalog.mjs'; +import { STOCK_CLIS } from '../src/config/cli-registry/stock.js'; + +const read = (rel: string): string => readFileSync(fileURLToPath(new URL(`../${rel}`, import.meta.url)), 'utf-8'); + +const AGENT_DOCKERFILE = read('docker/agent.Dockerfile'); +const SERVER_DOCKERFILE = read('docker/server.Dockerfile'); +const INDEX_HTML = read('src/web/public/index.html'); +const CATALOG = JSON.parse(read('config/clis.stock.json')) as Array<{ + id: string; + enabled: boolean; + discovery: { + binaries: string[]; + install: { npmPackage?: string; agentImageLayer?: { kind: 'dedicated'; reason: string } }; + }; +}>; + +const enabledAgents = CATALOG.filter((e) => e.enabled && e.discovery.binaries.length > 0); + +/** + * A layer's PROOF it installed the right thing, not merely a substring anywhere in the file. + * Every dedicated layer in agent.Dockerfile ends by running ` --version`, so anchoring + * on that (rather than `Dockerfile.includes(binary)`) survives a layer being deleted while its + * COMMENT — which also names the binary — is left behind. That gap is why this replaced the + * looser check. + */ +const hasVersionProof = (binary: string): boolean => AGENT_DOCKERFILE.includes(`${binary} --version`); + +describe('docker agent image covers the catalogue', () => { + it('installs every enabled npm CLI, via the build arg or a documented dedicated layer', () => { + const inBuildArg = new Set(agentImageNpmPackages(CATALOG)); + const missing: string[] = []; + for (const entry of enabledAgents) { + const pkg = entry.discovery.install.npmPackage; + if (!pkg) continue; // standalone installer, checked below + if (inBuildArg.has(pkg)) continue; + if (entry.discovery.install.agentImageLayer) continue; + missing.push(`${entry.id} (${pkg})`); + } + expect( + missing, + `npm CLI reaches neither the build arg nor a dedicated layer:\n ${missing.join('\n ')}\n` + + 'Add it to the arg (it is automatic) or give it a Dockerfile layer AND an agentImageLayer.reason in stock.ts.' + ).toEqual([]); + }); + + it('gives every dedicated-layer entry a reason and a real, provable layer', () => { + for (const entry of CATALOG) { + const layer = entry.discovery.install.agentImageLayer; + if (!layer) continue; + expect(layer.reason.length, `${entry.id} has an empty agentImageLayer.reason`).toBeGreaterThan(20); + const binary = entry.discovery.binaries[0]; + // Excluded from the shared arg, so it MUST appear in a hand-written layer that actually + // ran the binary, or it is simply not installed at all — an exclusion silently becoming + // an omission. + expect( + hasVersionProof(binary), + `${entry.id} is excluded from the arg but has no "${binary} --version" proof line in the Dockerfile` + ).toBe(true); + } + }); + + it('installs every enabled non-npm CLI in its own layer', () => { + for (const entry of enabledAgents) { + if (entry.discovery.install.npmPackage) continue; + const binary = entry.discovery.binaries[0]; + expect( + hasVersionProof(binary), + `${entry.id} ships no npm package and no Dockerfile layer proves it ran "${binary} --version"` + ).toBe(true); + } + }); + + it('bakes in nothing from a DISABLED entry', () => { + // The maintainer's finding: the earlier export carried no `enabled` field, so a CLI that + // ships disabled still had its package installed into every image. + for (const entry of CATALOG) { + if (entry.enabled) continue; + const pkg = entry.discovery.install.npmPackage; + if (!pkg) continue; + expect(AGENT_DOCKERFILE.includes(pkg), `disabled ${entry.id} is still baked into the image`).toBe(false); + } + }); + + it('excludes a disabled entry from the build arg (unit, since none ships disabled today)', () => { + // Every stock entry is enabled right now, so the assertion above passes vacuously. Feed + // the pure helper a fabricated disabled entry so the fix is genuinely covered TODAY + // rather than the first time someone ships one. + const fabricated = [ + ...CATALOG, + { id: 'ghost', enabled: false, discovery: { binaries: ['ghost'], install: { npmPackage: '@ghost/cli' } } }, + ]; + expect(agentImageNpmPackages(fabricated)).not.toContain('@ghost/cli'); + const enabledTwin = fabricated.map((e) => (e.id === 'ghost' ? { ...e, enabled: true } : e)); + expect(agentImageNpmPackages(enabledTwin)).toContain('@ghost/cli'); + }); + + it('refuses an npm package name that would not survive unquoted expansion', () => { + // The Dockerfile expands ${CLI_NPM_PACKAGES} unquoted so word splitting makes the list. + // A token with a space or a metacharacter would therefore change what the RUN line means. + const hostile = [ + { id: 'x', enabled: true, discovery: { binaries: ['x'], install: { npmPackage: 'a && rm -rf /' } } }, + ]; + expect(() => agentImageNpmPackages(hostile)).toThrow(/unsafe npm package name/i); + }); +}); + +describe('docker server image divergence is declared, not accidental', () => { + // server.Dockerfile deliberately ships a NARROWER list than the agent image, and is left + // untouched by this change because two other open PRs already modify it. Asserting the + // omissions here makes the divergence reviewable without editing the file: if someone adds + // a CLI there, or the intent changes, this fails and the list has to be restated. + const SERVER_INTENTIONAL_OMISSIONS = new Set(['antigravity', 'pi', 'grok', 'deepseek', 'omp']); + + it('installs exactly the CLIs it declares, and no more', () => { + for (const entry of enabledAgents) { + const pkg = entry.discovery.install.npmPackage; + if (!pkg) continue; + const present = SERVER_DOCKERFILE.includes(pkg); + if (SERVER_INTENTIONAL_OMISSIONS.has(entry.id)) { + expect(present, `${entry.id} is listed as an intentional omission but IS in server.Dockerfile`).toBe(false); + } else { + expect(present, `${entry.id} is missing from server.Dockerfile and not declared as omitted`).toBe(true); + } + } + }); +}); + +describe('the in-app agent-image hint stays accurate', () => { + it('names every enabled CLI binary the image contains', () => { + // index.html tells the user what the image holds. It was stale (it omitted omp), which is + // the same drift one layer out: prose describing a list nobody re-checks. + const hint = INDEX_HTML.split('\n').find((l) => l.includes('build-agent-image.mjs')); + expect(hint, 'the agent-image hint disappeared from index.html').toBeDefined(); + for (const entry of enabledAgents) { + expect(hint, `the hint does not mention ${entry.discovery.binaries[0]}`).toContain(entry.discovery.binaries[0]); + } + }); + + it('is checked against the registry, not a copy of itself (anti-vacuity)', () => { + expect(STOCK_CLIS.filter((e) => e.enabled && e.discovery.binaries.length > 0).length).toBeGreaterThan(5); + }); +}); diff --git a/test/install-sh-detection-parity.test.ts b/test/install-sh-detection-parity.test.ts new file mode 100644 index 000000000..5d393ce85 --- /dev/null +++ b/test/install-sh-detection-parity.test.ts @@ -0,0 +1,181 @@ +/** + * @fileoverview Pins `install.sh`'s CLI detection paths BEFORE they are generated. + * + * PR B replaces nine hand-written `*_SEARCH_PATHS` arrays in `install.sh` with one block + * generated from `STOCK_CLIS`. The arrays are NOT uniform — claude alone has + * `~/.claude/local`, opencode alone has `~/go/bin`, opencode/codex/gemini/pi/omp have + * `~/.bun/bin` while dsh/grok/agy do not, and omp's `~/.omp/bin` sits SECOND rather than + * first — so "generate them from the registry" is a claim that has to be proved, not + * assumed. If the generated list silently narrows, a user with that CLI installed stops + * being detected and is told no AI CLI was found: exactly the bug upstream `b6d0f1fa` fixed + * for omp by hand. + * + * This file is deliberately written FIRST, against the hand-written arrays, and kept + * afterwards as a regression pin. It asserts a three-way identity: + * + * 1. the literals below === what `install.sh` actually contains today + * 2. the literals below === `searchDirs x binaries` from the registry + * + * Together those mean the generator can only produce what is already shipping. (1) fails if + * `install.sh` drifts from the pin; (2) fails if a registry entry's `searchDirs` drifts from + * the installer — which, once the block is generated, is the same statement. + * + * ⚠️ The literals are the SOURCE OF TRUTH here and were transcribed from `install.sh` at + * `72fd231d`. Do not "fix" a failure by re-copying the current file into them; that turns + * the pin into a mirror and it stops guarding anything. Work out which side moved. + * + * Port: none (pure, over one source file and the registry). + */ + +import { describe, expect, it } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { STOCK_CLIS } from '../src/config/cli-registry/stock.js'; + +const INSTALL_SH = readFileSync(fileURLToPath(new URL('../install.sh', import.meta.url)), 'utf-8'); + +/** + * The nine arrays exactly as `install.sh` declares them, in declaration order, with the + * shell-variable form (`$HOME/...`) they carry there rather than the registry's `~/...`. + * + * Keyed by the array's own prefix, which is NOT always the registry id: DeepSeek's entry is + * `deepseek` but its binary and array are `DSH`, and antigravity's binary is `agy`. + */ +const LITERAL_SEARCH_PATHS: Record = { + CLAUDE: [ + '$HOME/.local/bin/claude', + '$HOME/.claude/local/claude', + '/usr/local/bin/claude', + '$HOME/.npm-global/bin/claude', + '$HOME/bin/claude', + ], + OPENCODE: [ + '$HOME/.opencode/bin/opencode', + '$HOME/.local/bin/opencode', + '/usr/local/bin/opencode', + '$HOME/go/bin/opencode', + '$HOME/.bun/bin/opencode', + '$HOME/.npm-global/bin/opencode', + '$HOME/bin/opencode', + ], + CODEX: [ + '$HOME/.codex/bin/codex', + '$HOME/.local/bin/codex', + '/usr/local/bin/codex', + '$HOME/.bun/bin/codex', + '$HOME/.npm-global/bin/codex', + '$HOME/bin/codex', + ], + GEMINI: [ + '$HOME/.gemini/bin/gemini', + '$HOME/.local/bin/gemini', + '/usr/local/bin/gemini', + '$HOME/.bun/bin/gemini', + '$HOME/.npm-global/bin/gemini', + '$HOME/bin/gemini', + ], + PI: ['$HOME/.local/bin/pi', '/usr/local/bin/pi', '$HOME/.bun/bin/pi', '$HOME/.npm-global/bin/pi', '$HOME/bin/pi'], + DSH: ['$HOME/.local/bin/dsh', '/usr/local/bin/dsh', '$HOME/.npm-global/bin/dsh', '$HOME/bin/dsh'], + GROK: ['$HOME/.grok/bin/grok', '$HOME/.local/bin/grok', '/usr/local/bin/grok', '$HOME/bin/grok'], + ANTIGRAVITY: ['$HOME/.local/bin/agy', '$HOME/.antigravity/bin/agy', '/usr/local/bin/agy', '$HOME/bin/agy'], + OMP: [ + '$HOME/.local/bin/omp', + '$HOME/.omp/bin/omp', + '/usr/local/bin/omp', + '$HOME/.bun/bin/omp', + '$HOME/.npm-global/bin/omp', + '$HOME/bin/omp', + ], +}; + +/** Array prefix in `install.sh` -> registry id, for the two that differ. */ +const ARRAY_PREFIX_TO_CLI_ID: Record = { + CLAUDE: 'claude', + OPENCODE: 'opencode', + CODEX: 'codex', + GEMINI: 'gemini', + PI: 'pi', + DSH: 'deepseek', + GROK: 'grok', + ANTIGRAVITY: 'antigravity', + OMP: 'omp', +}; + +/** + * The per-CLI search paths install.sh will actually probe, read back out of the GENERATED + * block: `CLI_ALL_PATHS` sliced by each id's `CLI_PATH_OFF`/`CLI_PATH_LEN` window. + * + * This parser replaced one that read the nine hand-written `*_SEARCH_PATHS` arrays, which + * this change deletes. The literals below did NOT move: they are still the same strings + * transcribed from those arrays, so the pin still measures the generated block against what + * shipped before it existed, which is the only comparison worth making. + */ +function parseInstallShSearchPaths(source: string): Record { + const readArray = (name: string): string[] => { + const m = new RegExp(`^${name}=\\((.*)\\)$`, 'm').exec(source); + if (!m) throw new Error(`install.sh has no ${name}= array`); + // Tokens are double-quoted (paths, which carry $HOME), single-quoted (ids, labels) or + // bare (the numeric offset/length windows). + return [...m[1].matchAll(/"([^"]*)"|'([^']*)'|(\S+)/g)].map((t) => t[1] ?? t[2] ?? t[3]); + }; + const ids = readArray('CLI_IDS'); + const paths = readArray('CLI_ALL_PATHS'); + const offs = readArray('CLI_PATH_OFF').map(Number); + const lens = readArray('CLI_PATH_LEN').map(Number); + const out: Record = {}; + ids.forEach((id, i) => { + const prefix = Object.entries(ARRAY_PREFIX_TO_CLI_ID).find(([, cliId]) => cliId === id)?.[0]; + if (prefix) out[prefix] = paths.slice(offs[i], offs[i] + lens[i]); + }); + return out; +} + +/** + * What the generated block must contain for one entry: `searchDirs x binaries`, in that + * nesting order, with `~` rewritten to `$HOME` the way the generator will emit it. + * + * The dir-major order matters and is not arbitrary — it is the order the resolvers probe in, + * so a binary-major flattening would still contain every path while checking them in the + * wrong sequence, and the first hit would change on a machine with two installs. + */ +function registrySearchPaths(cliId: string): string[] { + const entry = STOCK_CLIS.find((e) => (e.id as string) === cliId); + if (!entry) throw new Error(`no stock entry ${cliId}`); + return entry.discovery.searchDirs.flatMap((dir) => + entry.discovery.binaries.map((bin) => `${dir.startsWith('~/') ? `$HOME/${dir.slice(2)}` : dir}/${bin}`) + ); +} + +describe('install.sh CLI detection parity', () => { + const parsed = parseInstallShSearchPaths(INSTALL_SH); + + it('finds every generated search-path window (anti-vacuity)', () => { + // If the parse returns nothing, every it.each below passes by comparing [] to []. + expect(Object.keys(parsed).sort()).toEqual(Object.keys(LITERAL_SEARCH_PATHS).sort()); + for (const [name, paths] of Object.entries(parsed)) { + expect(paths.length, `${name} window parsed empty`).toBeGreaterThan(0); + } + }); + + it.each(Object.keys(LITERAL_SEARCH_PATHS))('%s search paths match the pinned literals', (prefix) => { + expect(parsed[prefix]).toEqual(LITERAL_SEARCH_PATHS[prefix]); + }); + + it.each(Object.entries(ARRAY_PREFIX_TO_CLI_ID))( + '%s search paths are reproduced by registry entry "%s"', + (prefix, cliId) => { + // The claim the generator rests on: the registry already knows every path the + // installer probes, in the same order. A failure here means the generated block would + // detect a different set than the hand-written one it replaces. + expect(registrySearchPaths(cliId)).toEqual(LITERAL_SEARCH_PATHS[prefix]); + } + ); + + it('covers every stock CLI that has a binary to find', () => { + // `shell` declares no binaries, so it has nothing to detect and no array. Everything + // else must be pinned above, or a new CLI could land with no installer coverage — which + // is the omp bug (upstream b6d0f1fa) restated as a test. + const detectable = STOCK_CLIS.filter((e) => e.discovery.binaries.length > 0).map((e) => e.id as string); + expect(detectable.sort()).toEqual(Object.values(ARRAY_PREFIX_TO_CLI_ID).sort()); + }); +}); diff --git a/test/install-sh-invariants.test.ts b/test/install-sh-invariants.test.ts new file mode 100644 index 000000000..70a70915f --- /dev/null +++ b/test/install-sh-invariants.test.ts @@ -0,0 +1,166 @@ +/** + * @fileoverview Static guards over `install.sh`, the one file in this repo nothing else checks. + * + * There is no shellcheck, no bats, and CI is Node-only, so a bash mistake here reaches users + * through `curl | bash` with nothing in between. The CI workflow now runs `bash -n` and a real + * `bash:3.2` container (see `.github/workflows/ci.yml`), which catches syntax and the + * `set -u` classes; this file catches the things that are perfectly valid bash and still wrong + * for THIS script. + * + * Port: none (pure, over one source file). + */ + +import { describe, expect, it } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; + +const SOURCE = readFileSync(fileURLToPath(new URL('../install.sh', import.meta.url)), 'utf-8'); + +/** Lines with the leading `#` comments removed, so prose quoting a banned form is not a hit. */ +const CODE_LINES = SOURCE.split('\n').filter((line) => !/^\s*#/.test(line)); +const CODE = CODE_LINES.join('\n'); + +describe('install.sh stays bash 3.2 compatible', () => { + // macOS ships bash 3.2 (the last GPLv2 release) and the documented install is + // `curl -fsSL | bash`, so a bash-4 construct is not a warning on a Mac, it is a + // syntax error that kills the install mid-run. + it.each([ + ['associative arrays (`declare -A`)', /\b(?:declare|local|typeset)\s+-[A-Za-z]*A/], + ['case-conversion expansion (`${x,,}` / `${x^^}`)', /\$\{[A-Za-z_][A-Za-z0-9_]*(?:\[[^\]]*\])?[,^]{1,2}\}/], + ['`mapfile` / `readarray`', /\b(?:mapfile|readarray)\b/], + ['namerefs (`declare -n`)', /\b(?:declare|local|typeset)\s+-[A-Za-z]*n\b/], + ['here-strings (`<<<`)', /<< { + const offenders = CODE_LINES.filter((line) => pattern.test(line)); + expect(offenders, `bash 4+ construct found:\n ${offenders.join('\n ')}`).toEqual([]); + }); +}); + +describe('install.sh generated-catalogue block', () => { + it('has exactly one matched marker pair', () => { + expect(SOURCE.split('# >>> BEGIN GENERATED CLI CATALOGUE').length - 1).toBe(1); + expect(SOURCE.split('# <<< END GENERATED CLI CATALOGUE').length - 1).toBe(1); + expect(SOURCE.indexOf('# >>> BEGIN GENERATED CLI CATALOGUE')).toBeLessThan( + SOURCE.indexOf('# <<< END GENERATED CLI CATALOGUE') + ); + }); + + it('declares every array the detection code indexes', () => { + for (const name of [ + 'CLI_IDS', + 'CLI_LABELS', + 'CLI_ENABLED', + 'CLI_KIND', + 'CLI_NPM', + 'CLI_DOCS', + 'CLI_CMD_LINUX', + 'CLI_CMD_DARWIN', + 'CLI_ALL_BINS', + 'CLI_BIN_OFF', + 'CLI_BIN_LEN', + 'CLI_ALL_PATHS', + 'CLI_PATH_OFF', + 'CLI_PATH_LEN', + ]) { + expect(new RegExp(`^${name}=\\(`, 'm').test(SOURCE), `${name} is not declared`).toBe(true); + } + }); + + it('keeps no hand-written per-CLI detection behind', () => { + // The nine `*_SEARCH_PATHS` arrays and eighteen `check_`/`get__path` pairs are + // what this change removes. One left behind would be a second source of truth that the + // generator does not update — the exact shape of upstream b6d0f1fa. + expect(CODE.match(/_SEARCH_PATHS=\(/g) ?? []).toEqual([]); + + // Keyed on the catalogue's OWN ids and binaries rather than an allowlist of the helpers + // that may exist. `check_tmux` and `check_cloudflared` are legitimate and unrelated; a + // `check_claude` or `get_omp_path` is the thing being removed. Deriving the ban from the + // catalogue means a CLI added later is covered with no edit here. + const names = new Set(); + for (const arrayName of ['CLI_IDS', 'CLI_ALL_BINS']) { + const m = new RegExp(`^${arrayName}=\\((.*)\\)$`, 'm').exec(SOURCE); + for (const token of m?.[1].match(/'([^']*)'/g) ?? []) names.add(token.replace(/'/g, '')); + } + expect(names.size, 'could not read the catalogue ids/binaries').toBeGreaterThan(5); + + const perCliFunctions = [...names] + .flatMap((name) => [`check_${name}()`, `get_${name}_path()`]) + .filter((fn) => new RegExp(`^${fn.replace(/[()]/g, '\\$&')}`, 'm').test(CODE)); + expect(perCliFunctions, `hand-written per-CLI detection still present:\n ${perCliFunctions.join('\n ')}`).toEqual( + [] + ); + }); +}); + +describe('install.sh trust boundary', () => { + // A command the installer EXECUTES must have arrived embedded in this file, over the same + // TLS fetch and in the same commit as the script itself — there is no second, network-derived + // copy of these commands anywhere in the script (an earlier draft that added one, and split + // a TRUSTED/DISPLAY pair to keep the fetched copy display-only, was dropped before merge: + // see docs/cli-registry.md). These three assertions are what is left to guard now that the + // fetch path itself does not exist: everything the installer runs or shows still comes only + // from the generated block, and nothing in the file eval()s. + it('writes CLI_INSTALL_CMD_TRUSTED only from the generated per-platform arrays', () => { + const writes = CODE_LINES.filter((line) => /CLI_INSTALL_CMD_TRUSTED\s*\[[^\]]*\]\s*=/.test(line)); + expect(writes.length, 'expected exactly the two platform assignments').toBe(2); + for (const line of writes) { + expect(line, `TRUSTED written from something other than the generated block:\n ${line}`).toMatch( + /=\s*"\$\{CLI_CMD_(?:LINUX|DARWIN)\[\$i\]\}"/ + ); + } + }); + + it('fetches no CLI catalogue over the network at install time', () => { + // The exact shape of the earlier, dropped design: a URL built from the repo/branch this + // script came from, an opt-in env var to enable it, and a `download()` call feeding + // straight into the trusted arrays. None of that exists in this file any more; this pins + // the absence so it cannot quietly come back without a reviewer noticing. + for (const needle of [ + 'cli_catalog_refresh', + 'cli_catalog_default_url', + 'CODEMAN_CLI_CATALOGUE_URL', + 'CODEMAN_REFRESH_CLI_CATALOGUE', + 'CLI_INSTALL_CMD_DISPLAY', + ]) { + expect(SOURCE.includes(needle), `${needle} should not exist — the catalogue refresh was dropped`).toBe(false); + } + }); + + it('never eval()s anything', () => { + // install.sh has two long-standing, legitimate evals (`eval "$(brew shellenv)"`, Homebrew's + // documented idiom, and one inside a node -e that reads `tailscale serve status`), both of + // which operate on output this script itself produced, never on fetched content. With no + // network-derived catalogue left to eval, the word should not appear at all outside those. + const offenders = CODE_LINES.filter( + (line) => /\beval\b/.test(line) && !/eval "\$\(.*shellenv\)"/.test(line) && !line.includes('eval(process.argv') + ); + expect(offenders, `unexpected eval:\n ${offenders.join('\n ')}`).toEqual([]); + }); + + it("redirects stdin for every command it executes on the user's behalf", () => { + // Under `curl | bash` the script IS stdin, so a child that reads stdin eats the rest of + // it. Every spawn of an untrusted-length vendor command must carry ` /\bbash -c "\$\{CLI_INSTALL_CMD_TRUSTED/.test(line)); + expect(spawns.length, 'expected the single install-menu spawn').toBe(1); + for (const line of spawns) { + expect(line, `install spawn without { + it('can be sourced without installing anything', () => { + // The bash 3.2 CI step sources this file to exercise detect_all_clis. Without the guard + // the dispatch `case` at the tail would run a real install inside the container. + expect(SOURCE).toMatch( + /if \[\[ -n "\$\{CODEMAN_INSTALL_SH_LIB:-\}" \]\]; then return 0 2>\/dev\/null \|\| exit 0; fi/ + ); + const guardAt = SOURCE.indexOf('CODEMAN_INSTALL_SH_LIB'); + const dispatchAt = SOURCE.indexOf('case "${1:-}" in'); + expect(guardAt, 'the sourcing guard must precede the dispatch case').toBeLessThan(dispatchAt); + }); + + it('still sets the strict flags it has always run under', () => { + expect(SOURCE).toMatch(/^set -euo pipefail$/m); + }); +});