Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -603,10 +603,14 @@ jobs:
# opencode-live. Every pull request that does not
# touch the scope below, and a push to main.
# - `probes`: adds the minimal keyed probes (keyed-probes.e2e.ts), one
# request of a few tokens per plane × protocol × URL shape, plus six
# recorded variants of each plane's default completions cell (request
# id, project, reasoning effort; KEYED_VARIANTS) and every 429's
# Retry-After. A pull request
# request of a few tokens per plane × protocol × URL shape, plus the
# recorded variants of each plane's default completions cell
# (KEYED_VARIANTS: the lessons' request-id, project and
# reasoning-effort rows, and the console study's raw-session,
# session-only and harness-User-Agent rows, three of them on a free
# model sent with no key) and every keyed rate-limit 429's
# Retry-After (the keyless free rows and the free tier's usage cap
# left out). A pull request
# that touches src/host/**; src/utils/protocols.ts, which decides the
# legacy URLs the probes send and the protocol each probe model speaks;
# src/utils/plane-model.ts, which decides the family the plugin
Expand Down
9 changes: 5 additions & 4 deletions AGENTS.md

Large diffs are not rendered by default.

7 changes: 4 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ The e2e files, one job each:

- `opencode-live.e2e.ts` — the only place the plugin meets the real API. It catches what a stub cannot: **the vendor changing the payload**. The meter parses `/zen/go/v1/usage` and the picker consumes an enriched `/models` listing, so both shapes are asserted against the live service. The unkeyed cases assert reachability (never a 404) and that our header set does not change the gateway's verdict on a request — compared against the same call without it, rather than hardcoding a status the vendor may tighten.
- `keyless-matrix.e2e.ts` — the keyless 401-vs-404 matrix: two planes × four protocols × three URL shapes — legacy (`/zen(/go)/v1`), the root (`/inference(/go)/v1`, the provider-level base the catalog declared until 2026-10-16) and modern (`/inference(/go)/{openai,anthropic,google}/…`) — one PAID model per cell from the live catalog, never a free one, which the gateway serves keyless with a real completion. It **sends no key and reads none**, so the `e2e` job runs it on every pull request, and it retries a cell once on a 429, a 5xx or a transport error before reading it. A 404 means the path is gone, a 401/403 that it exists, a 410, a 400 "Model is unavailable" or a refusal of type `ModelDeprecated` at any status that the model is. It asserts that all twenty-four cells have a model and were sent, that every legacy and modern answer is readable (a root answer is recorded whatever it is: nothing dispatches there), and that no cell the plugin dispatches to today with the default endpoint (`auto`, an alias of `modern`; `TODAY_DISPATCH`: all four protocols on the modern family, per plane) is gone. A redirect is never followed (`redirect: "manual"`): it reads as `redirected`, unreadable. Beside the matrix, a keyless free-model probe sends a free completions model to `/inference/openai/v1/chat/completions` and asserts it is served or free-tier-gated; a retired free model (`model-gone`, a `ModelDeprecated` refusal included) or a busy upstream (a 503, a 529, or a 5xx naming an overload) moves on to the next candidate (`runFreeProbe`, which records the deciding candidate's own model and status), and a rate limit (a 429, or a 429 and then no answer), a busy upstream no candidate got past, or the free tier's usage cap (`429 FreeUsageLimitError`, its own verdict `free-usage-limited`, never retried and never read as served) skips it. The System One family's three shapes on the Zen plane (`/zen/v1/systemone`, `/inference/v1/systemone`, `/inference/systemone/v1/systemone`) are probed keyless with no model and recorded, never asserted: the plugin serves no classifier. `endpoint: legacy` is recorded, not guarded: the cells it dispatches to (`LEGACY_DISPATCH`: completions, Responses and Messages on both planes, and Zen's Google; Go's Google stays on `/inference/go/google/v1beta`, `GO_GOOGLE_BASE_URL`) must answer readably, like every legacy cell, but their path-gone is recorded and asserted nowhere live (Go's Google, whose cell under `endpoint: legacy` is the modern one, is guarded as a default cell), and `test/endpoint-family.test.ts` pins their mapping offline. Both probe suites rewrite their record after every cell, so a job killed at its time limit keeps what it measured. The cell table, the sets and the classifiers are a pure module (`test/support/keyless-matrix.ts`), unit-tested offline in `test/keyless-matrix.test.ts`, which also derives `TODAY_DISPATCH` from the URLs each host cell's rows measured (every cell reaches the same cells: the plugin builds every base from its own endpoint table).
- `keyed-probes.e2e.ts` — the keyed half: the same twenty-four cells, each sent with the plane's own key in the header its protocol reads, the plugin's identity headers (`PROBE_IDENTITY`: the turn's user agent, `x-opencode-client`, `x-opencode-project`, an in-band session id under both session headers — the Go plane refuses a keyed request without one; the per-request metric id `x-opencode-request` is left out on purpose) and a body of a few output tokens, on the `probes` and `full` tiers. Per plane it decides from the `TODAY_DISPATCH` cells (`planeOutcome`), each read beside the legacy and root cells of the same protocol, which carry the same model and key: it fails when one answers path-gone, wrong-protocol or auth-refused, a retired model another shape accepted, or nothing readable — `other`, a redirect, a refused session, `transport-error` or `no-model`. Only a wrong-protocol, path-gone, auth-refused or model-gone that the cell `endpoint: legacy` dispatches to — a different one, so never Go's Google — accepted is flip evidence; a refusal only the root or Go Google's `/zen/go` cell accepted is red but not evidence (no flip reaches it), a key refused on every shape that read it names the CI secret instead, and a protocol or a path the flip's cell measured nothing about asks for a re-run. The account's answer counts as acceptance beside a refused key or a gone path, never beside a refused protocol or model. It skips when none of them was served (the account's answer, a rate limit, a busy upstream, a retired model, a parsed but refused body), since that measured nothing. Every other cell is recorded. Like the keyless matrix, it skips while a mirror base URL is set. A second case per plane records the keyed variants of the default completions cell — a minted `x-opencode-request`, `x-opencode-project` as a folder name and as a 40-hex digest beside the identity's `global`, and a paid completions model that reasons with no `reasoning_effort` and with `"none"` — and every 429's `Retry-After`; they assert nothing (the plan's lessons L4, L5, L9 and L10, for P6 and P-R3). Headers and keys are never printed, and a body only through a redacting excerpt. The cells, tier, classifier and plane outcome are `test/support/keyed-probes.ts`, unit-tested in `test/keyed-probes.test.ts`, which also runs every keyed cell through the offline gateway mock's judge.
- `keyed-probes.e2e.ts` — the keyed half: the same twenty-four cells, each sent with the plane's own key in the header its protocol reads, the plugin's identity headers (`PROBE_IDENTITY`: the turn's user agent, `x-opencode-client`, `x-opencode-project`, an in-band session id under both session headers — the Go plane refuses a keyed request without one; the per-request metric id `x-opencode-request` is left out on purpose) and a body of a few output tokens, on the `probes` and `full` tiers. Per plane it decides from the `TODAY_DISPATCH` cells (`planeOutcome`), each read beside the legacy and root cells of the same protocol, which carry the same model and key: it fails when one answers path-gone, wrong-protocol or auth-refused, a retired model another shape accepted, or nothing readable — `other`, a redirect, a refused session, `transport-error` or `no-model`. Only a wrong-protocol, path-gone, auth-refused or model-gone that the cell `endpoint: legacy` dispatches to — a different one, so never Go's Google — accepted is flip evidence; a refusal only the root or Go Google's `/zen/go` cell accepted is red but not evidence (no flip reaches it), a key refused on every shape that read it names the CI secret instead, and a protocol or a path the flip's cell measured nothing about asks for a re-run. The account's answer counts as acceptance beside a refused key or a gone path, never beside a refused protocol or model. It skips when none of them was served (the account's answer, a rate limit, a busy upstream, a retired model, a parsed but refused body), since that measured nothing. Every other cell is recorded. Like the keyless matrix, it skips while a mirror base URL is set. A second case per plane records, and asserts nothing about: the lessons' variants of the default completions cell (a minted `x-opencode-request`; `x-opencode-project` as a folder name and as a 40-hex digest beside the identity's `global`; a paid completions model that reasons, with no `reasoning_effort` and with `"none"`), for L4, L5 and L10; the console study's rows (the host's raw `session-<uuid>` session id, OC9; the session-only header set, OC8; the harness's User-Agent, OC6; and a free completions model sent keyless inside the keyed tier with the free turn's body, as the identity, with a raw session id and with the harness's User-Agent), which feed an owner's OC6 decision, the session derivation's OC9 and P6's OC8; and every keyed rate-limit 429's `Retry-After` (`retryAfterOf`; the keyless free rows and the free tier's usage cap are left out, `docs/redirect-planes.md` §13), for L9. Headers and keys are never printed, and a body only through a redacting excerpt. The cells, tier, classifier and plane outcome are `test/support/keyed-probes.ts`, unit-tested in `test/keyed-probes.test.ts`, which also runs every keyed cell through the offline gateway mock's judge.
- `protocol-routing.e2e.ts` — asks the gateway which endpoint serves each model's wire format, at the URL the plane adapter sends it to (the modern family by default, Go's Google included): keyless for the free classes, keyed (and skipped without a secret, or below the `full` tier) for paid ones. The keyed record of every URL shape per protocol, the legacy family included, is `keyed-probes.e2e.ts`. Its last block sends paid models through the plane adapter on a tier-A host: each protocol walks up to three candidates (`walkPerProtocol`, `test/support/live-walk.ts`, unit-tested in `test/live-walk.test.ts`), moving past an entitlement answer or a withdrawal that a raw probe of the same id confirms (`test/support/live-probe.ts`), and a protocol no candidate reached skips the case by name rather than passing.
- `dispatch-levels.e2e.ts` — a keyed live block, on the `full` tier, that sends one toggle model per plane and protocol through the plane adapter on a tier-A host with the picker's "On" effort, walking up to three toggle candidates per protocol the same way, and skips without the plane's key, below the `full` tier, or, naming it, when a protocol had no entitled, still-served candidate. Its offline half — every thinking level the picker offers is one pi's own rule accepts, swept over the whole shipped catalog — needs no network and no port, so it is `test/dispatch-levels.test.ts` in the unit run.
- `host-binary.e2e.ts` — the published host binary; see [The real-host harness](#the-real-host-harness). It skips unless `DSH_HOST_E2E=1`.
Expand All @@ -64,7 +64,8 @@ The e2e files, one job each:
- `plane-dispatch.mock.ts` — every characterized model on each plane, through the real `LlmRuntime` and the plugin's `PlaneAdapter` on a tier-A host.
- `protocol-compat.mock.ts` — plane × protocol × endpoint family, driven through pi's own protocol implementations.
- `patched-fetch-headers.mock.ts` — a local `node:http` listener, so the _outgoing_ header set crosses a real socket. The live gateway cannot echo a request back; everywhere else in the suite the injected headers are asserted against a captured `fetch`.
- `refusal-classification.mock.ts` — the refusal classifier's verdicts on the gateway's bytes over a real `node:http` socket.
- `refusal-classification.mock.ts` — the refusal classifier's verdicts on the gateway's bytes over a real `node:http` socket, on pi-ai's own route.
- `plane-refusals.mock.ts` — the gateway's five typed refusals on a plane turn, through the real `LlmRuntime` and the `PlaneAdapter` on a tier-A host: each finishes as `FREE_TIER` or `MODEL_UNAVAILABLE` with the gateway's words, and its status except on Google-protocol models, whose failure text carries none; a 401 and a plain 429 keep the host's reading. Its request count guards against a duplicate dispatch only: retrying belongs to `dsh-llm-retry`, which tier A does not mount, so whether a refusal is resent is judged by its code being outside the plane's `retryableCodes`.
- `host-drift-cli.mock.ts` — the nightly host-drift report's CLI against a loopback registry, as CI runs it; the rules it prints are `test/host-drift.test.ts`'s, on the registry views in `test/support/host-drift-fixture.ts`. The bump proposal's CLI (`scripts/propose-host-bump.ts`) runs the same way: a data plan, a cut inside the release-age window, a refusing registry, the `--out` files CI's push step reads (with no tracked file touched without `--write`), and, on a scratch copy through `--root`, a `--write` whose pnpm step fails putting every file back. Its rules are `test/host-bump.test.ts`'s.
- `cli-version.mock.ts` — `scripts/check-vendor-version.ts` against a loopback registry through `--registry`: the dist-tags request, a pinned `latest`, a bump, a generation change (`latest` left the pinned major) and a refusing registry.

Expand Down Expand Up @@ -157,7 +158,7 @@ The web profile wires this checkout with `link:`, so builds are picked up like t
- **Raise them when you genuinely add coverage** — a PR that lifts statements by a few points should lift the threshold with it, or the next contributor inherits a gate nobody re-measured.
- **Never set one above the current measurement.** That does not make the suite better; it makes every subsequent PR fail until someone deletes tests.

`src/index.ts` is excluded from the report — it is a pure re-export barrel, and the coverage tools attribute an untaken re-export line to whichever file re-exports it, so leaving it in reports a hole nobody can fill while hiding real ones. Load-bearing modules carry their own per-file floors, so a well-covered average cannot hide one that stopped being exercised: `host/patch-method.ts` and the four pure plane modules (`utils/protocols.ts`, `utils/model-metadata.ts`, `utils/plane-model.ts`, `utils/plane-profile.ts`) at 100%, because each branch is a reload case, a protocol, a precedence or one of pi-ai's checks; `host/route-claim.ts`, `host/plane-provider.ts`, `host/describe-graft.ts` and `host/host-modules.ts` just below what they measure; and the three modules that install the plugin's effects on the host, `lifecycle.ts`, `stream-hook.ts` and `fetch-patch.ts`, just below what they measure too, with the branches no composition reaches named beside the floor. `test/architecture.test.ts` fails the build on a floor whose file no longer exists, which would otherwise be a floor on nothing.
`src/index.ts` is excluded from the report — it is a pure re-export barrel, and the coverage tools attribute an untaken re-export line to whichever file re-exports it, so leaving it in reports a hole nobody can fill while hiding real ones. Load-bearing modules carry their own per-file floors, so a well-covered average cannot hide one that stopped being exercised: `host/patch-method.ts` and the four pure plane modules (`utils/protocols.ts`, `utils/model-metadata.ts`, `utils/plane-model.ts`, `utils/plane-profile.ts`) and `utils/gateway-refusal.ts` at 100%, because each branch is a reload case, a protocol, a precedence, one of pi-ai's checks, a refusal type or a text shape; `host/route-claim.ts`, `host/plane-provider.ts`, `host/describe-graft.ts` and `host/host-modules.ts` just below what they measure; and the three modules that install the plugin's effects on the host, `lifecycle.ts`, `stream-hook.ts` and `fetch-patch.ts`, just below what they measure too, with the branches no composition reaches named beside the floor. `test/architecture.test.ts` fails the build on a floor whose file no longer exists, which would otherwise be a floor on nothing.

## The freshness checks in CI

Expand Down
Loading
Loading