From d884c8a177ed8fd2855be3cfdfceff1aec6859ee Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Fri, 14 Aug 2026 20:26:00 +0000 Subject: [PATCH 1/6] Collapse the packages into @onkernel/loop Merge @onkernel/cua-ai, @onkernel/cua-agent, and @onkernel/cua-pi-extension into one package. The tree follows the framework-neutral boundary rather than the old package boundary: src/core is the neutral core exported as ".", src/pi is the pi binding exported as "./pi", and src/pi-extension is registered through the package's own pi.extensions field. Rename Cua* to Loop*, or to a plain domain name where the concept is computer use rather than this product, and move tool identities from cua.*.v1 to kloop.*.v1. Model-facing tool names are unchanged. Drop the pi-ai re-export from the package root, replace the two release workflows with one, and update the docs and the release skill. --- .agents/skills/release/SKILL.md | 177 ++-- .github/workflows/ci.yml | 57 +- .github/workflows/release-cua-ai.yml | 77 -- ...release-cua-agent.yml => release-loop.yml} | 48 +- .gitignore | 1 + README.md | 83 +- docs/agent-tool-configuration-spec.md | 202 ++-- docs/architecture.md | 133 +-- docs/npm-releases.md | 47 +- package-lock.json | 69 +- package.json | 12 +- packages/agent/CHANGELOG.md | 299 ------ packages/agent/README.md | 289 ------ packages/agent/examples/agent-openai-smoke.ts | 38 - packages/agent/src/index.ts | 43 - packages/agent/tsconfig.build.json | 24 - packages/agent/vitest.config.ts | 16 - packages/ai/CHANGELOG.md | 433 --------- packages/ai/README.md | 288 ------ packages/ai/package.json | 54 -- packages/ai/src/actions/index.ts | 47 - packages/ai/src/api-keys.ts | 69 -- packages/ai/src/index.ts | 29 - packages/ai/test/api-keys.test.ts | 65 -- packages/ai/tsconfig.build.json | 12 - packages/ai/tsconfig.json | 3 - packages/ai/tsdown.config.ts | 11 - packages/ai/vitest.integration.config.ts | 13 - packages/loop/CHANGELOG.md | 885 ++++++++++++++++++ packages/loop/README.md | 541 +++++++++++ .../{ai => loop}/docs/supported-models.md | 32 +- packages/loop/examples/agent-openai-smoke.ts | 46 + .../examples/agent-provider-matrix.ts | 25 +- .../examples/anthropic-native-smoke.ts | 34 +- .../examples/harness-openai-smoke.ts | 33 +- .../examples/harness-provider-matrix.ts | 28 +- packages/{ai => loop}/examples/quickstart.ts | 27 +- packages/{ai => loop}/examples/screenshot.png | Bin .../examples/shared/logging.ts | 2 +- .../examples/shared/scenarios.ts | 0 .../{agent => loop}/examples/shared/tools.ts | 35 +- packages/{agent => loop}/package.json | 47 +- .../scripts/native-action-probe.ts | 0 .../src => loop/src/core}/actions/browser.ts | 100 +- .../src => loop/src/core}/actions/computer.ts | 96 +- packages/loop/src/core/actions/index.ts | 47 + .../src/core}/browser-result-format.ts | 0 packages/{ai/src => loop/src/core}/menu.ts | 56 +- .../{agent/src => loop/src/core}/resources.ts | 23 +- .../{ai/src => loop/src/core}/tool-catalog.ts | 188 ++-- .../src => loop/src/core}/tool-manager.ts | 55 +- .../{ai/src/cua.ts => loop/src/core/tools.ts} | 252 ++--- .../src/core}/translator/browser-act.ts | 20 +- .../browser-document-reconciliation.ts | 0 .../translator/browser-frame-collection.ts | 0 .../core}/translator/browser-observation.ts | 0 .../core}/translator/browser-ref-lifecycle.ts | 2 +- .../src/core}/translator/browser-wait.ts | 10 +- .../src/core}/translator/browser.ts | 76 +- .../src => loop/src/core}/translator/cdp.ts | 0 .../src => loop/src/core}/translator/keys.ts | 0 .../src/core}/translator/translator.ts | 84 +- .../src => loop/src/core}/translator/types.ts | 0 packages/loop/src/core/url.ts | 7 + packages/loop/src/index.ts | 34 + .../src/pi-extension}/browser-runtime.ts | 26 +- .../src => loop/src/pi-extension}/index.ts | 76 +- .../src => loop/src/pi-extension}/render.ts | 0 .../src/pi-extension}/selection.ts | 56 +- .../src => loop/src/pi-extension}/state.ts | 6 +- packages/loop/src/pi/api-keys.ts | 69 ++ packages/{agent/src => loop/src/pi}/attach.ts | 115 ++- packages/loop/src/pi/index.ts | 35 + packages/{ai/src => loop/src/pi}/models.ts | 116 +-- .../src => loop/src/pi}/provider-retry.ts | 8 +- packages/{ai/src => loop/src/pi}/providers.ts | 74 +- .../providers/anthropic/adaptive-thinking.ts | 4 +- .../providers/anthropic/browser-fallback.ts | 26 +- .../pi}/providers/anthropic/capabilities.ts | 0 .../src/pi}/providers/anthropic/native.ts | 8 +- .../src => loop/src/pi}/providers/common.ts | 18 +- .../src/pi}/providers/google/provider.ts | 20 +- .../src/pi}/providers/openai/provider.ts | 28 +- .../test/anthropic-browser-fallback.test.ts | 28 +- .../test/anthropic-native.integration.test.ts | 17 +- .../test/anthropic-payload.test.ts | 16 +- packages/loop/test/api-keys.test.ts | 65 ++ .../test/attach-session.test.ts | 76 +- packages/{agent => loop}/test/attach.test.ts | 66 +- .../test/browser-act-fail-fast.test.ts | 2 +- .../test/browser-cross-process.live.test.ts | 18 +- .../test/browser-frame-collection.test.ts | 4 +- .../test/browser-ref-lifecycle.test.ts | 2 +- .../test/browser-runtime.test.ts | 20 +- .../{agent => loop}/test/browser-wait.test.ts | 21 +- packages/{agent => loop}/test/cdp.test.ts | 2 +- .../{agent => loop}/test/e2e.live.test.ts | 30 +- .../test/example-provider-matrix.test.ts | 10 +- .../test/extension.test.ts | 30 +- .../{ai => loop}/test/google-provider.test.ts | 22 +- .../test/harness-context.test.ts | 36 +- packages/{agent => loop}/test/keys.test.ts | 2 +- packages/{ai => loop}/test/menu.test.ts | 35 +- packages/{ai => loop}/test/models.test.ts | 156 +-- .../test/openai-adapter-routing.test.ts | 36 +- .../test/openai-native-provider.test.ts | 32 +- .../test/pi-modes.test.ts | 6 +- .../test/provider-retry.test.ts | 2 +- .../test/provider-stream.test.ts | 28 +- packages/{ai => loop}/test/providers.test.ts | 30 +- .../test/published-declarations.test.ts | 45 +- .../test/published-package.test.ts | 7 +- .../{agent => loop}/test/resources.test.ts | 45 +- .../test/selection.test.ts | 40 +- .../{ai => loop}/test/tool-catalog.test.ts | 199 ++-- .../{agent => loop}/test/tool-manager.test.ts | 51 +- .../test/translator-browser.test.ts | 198 ++-- .../{agent => loop}/test/translator.test.ts | 10 +- packages/loop/tsconfig.build.json | 20 + packages/{agent => loop}/tsconfig.json | 0 packages/{agent => loop}/tsdown.config.ts | 2 +- packages/{ai => loop}/vitest.config.ts | 13 + packages/loop/vitest.integration.config.ts | 21 + packages/pi-extension/CHANGELOG.md | 48 - packages/pi-extension/README.md | 115 --- packages/pi-extension/package.json | 58 -- packages/pi-extension/tsconfig.build.json | 23 - packages/pi-extension/tsconfig.json | 3 - packages/pi-extension/vitest.config.ts | 15 - tsconfig.json | 6 +- 130 files changed, 3713 insertions(+), 4107 deletions(-) delete mode 100644 .github/workflows/release-cua-ai.yml rename .github/workflows/{release-cua-agent.yml => release-loop.yml} (53%) delete mode 100644 packages/agent/CHANGELOG.md delete mode 100644 packages/agent/README.md delete mode 100644 packages/agent/examples/agent-openai-smoke.ts delete mode 100644 packages/agent/src/index.ts delete mode 100644 packages/agent/tsconfig.build.json delete mode 100644 packages/agent/vitest.config.ts delete mode 100644 packages/ai/CHANGELOG.md delete mode 100644 packages/ai/README.md delete mode 100644 packages/ai/package.json delete mode 100644 packages/ai/src/actions/index.ts delete mode 100644 packages/ai/src/api-keys.ts delete mode 100644 packages/ai/src/index.ts delete mode 100644 packages/ai/test/api-keys.test.ts delete mode 100644 packages/ai/tsconfig.build.json delete mode 100644 packages/ai/tsconfig.json delete mode 100644 packages/ai/tsdown.config.ts delete mode 100644 packages/ai/vitest.integration.config.ts create mode 100644 packages/loop/CHANGELOG.md create mode 100644 packages/loop/README.md rename packages/{ai => loop}/docs/supported-models.md (74%) create mode 100644 packages/loop/examples/agent-openai-smoke.ts rename packages/{agent => loop}/examples/agent-provider-matrix.ts (60%) rename packages/{agent => loop}/examples/anthropic-native-smoke.ts (59%) rename packages/{agent => loop}/examples/harness-openai-smoke.ts (59%) rename packages/{agent => loop}/examples/harness-provider-matrix.ts (69%) rename packages/{ai => loop}/examples/quickstart.ts (63%) rename packages/{ai => loop}/examples/screenshot.png (100%) rename packages/{agent => loop}/examples/shared/logging.ts (94%) rename packages/{agent => loop}/examples/shared/scenarios.ts (100%) rename packages/{agent => loop}/examples/shared/tools.ts (51%) rename packages/{agent => loop}/package.json (51%) rename packages/{ai => loop}/scripts/native-action-probe.ts (100%) rename packages/{ai/src => loop/src/core}/actions/browser.ts (89%) rename packages/{ai/src => loop/src/core}/actions/computer.ts (75%) create mode 100644 packages/loop/src/core/actions/index.ts rename packages/{agent/src => loop/src/core}/browser-result-format.ts (100%) rename packages/{ai/src => loop/src/core}/menu.ts (67%) rename packages/{agent/src => loop/src/core}/resources.ts (92%) rename packages/{ai/src => loop/src/core}/tool-catalog.ts (82%) rename packages/{agent/src => loop/src/core}/tool-manager.ts (61%) rename packages/{ai/src/cua.ts => loop/src/core/tools.ts} (81%) rename packages/{agent/src => loop/src/core}/translator/browser-act.ts (93%) rename packages/{agent/src => loop/src/core}/translator/browser-document-reconciliation.ts (100%) rename packages/{agent/src => loop/src/core}/translator/browser-frame-collection.ts (100%) rename packages/{agent/src => loop/src/core}/translator/browser-observation.ts (100%) rename packages/{agent/src => loop/src/core}/translator/browser-ref-lifecycle.ts (99%) rename packages/{agent/src => loop/src/core}/translator/browser-wait.ts (97%) rename packages/{agent/src => loop/src/core}/translator/browser.ts (96%) rename packages/{agent/src => loop/src/core}/translator/cdp.ts (100%) rename packages/{agent/src => loop/src/core}/translator/keys.ts (100%) rename packages/{agent/src => loop/src/core}/translator/translator.ts (85%) rename packages/{agent/src => loop/src/core}/translator/types.ts (100%) create mode 100644 packages/loop/src/core/url.ts create mode 100644 packages/loop/src/index.ts rename packages/{pi-extension/src => loop/src/pi-extension}/browser-runtime.ts (79%) rename packages/{pi-extension/src => loop/src/pi-extension}/index.ts (84%) rename packages/{pi-extension/src => loop/src/pi-extension}/render.ts (100%) rename packages/{pi-extension/src => loop/src/pi-extension}/selection.ts (73%) rename packages/{pi-extension/src => loop/src/pi-extension}/state.ts (87%) create mode 100644 packages/loop/src/pi/api-keys.ts rename packages/{agent/src => loop/src/pi}/attach.ts (80%) create mode 100644 packages/loop/src/pi/index.ts rename packages/{ai/src => loop/src/pi}/models.ts (73%) rename packages/{agent/src => loop/src/pi}/provider-retry.ts (96%) rename packages/{ai/src => loop/src/pi}/providers.ts (53%) rename packages/{ai/src => loop/src/pi}/providers/anthropic/adaptive-thinking.ts (91%) rename packages/{ai/src => loop/src/pi}/providers/anthropic/browser-fallback.ts (90%) rename packages/{ai/src => loop/src/pi}/providers/anthropic/capabilities.ts (100%) rename packages/{ai/src => loop/src/pi}/providers/anthropic/native.ts (96%) rename packages/{ai/src => loop/src/pi}/providers/common.ts (79%) rename packages/{ai/src => loop/src/pi}/providers/google/provider.ts (94%) rename packages/{ai/src => loop/src/pi}/providers/openai/provider.ts (95%) rename packages/{ai => loop}/test/anthropic-browser-fallback.test.ts (85%) rename packages/{ai => loop}/test/anthropic-native.integration.test.ts (75%) rename packages/{ai => loop}/test/anthropic-payload.test.ts (68%) create mode 100644 packages/loop/test/api-keys.test.ts rename packages/{agent => loop}/test/attach-session.test.ts (83%) rename packages/{agent => loop}/test/attach.test.ts (80%) rename packages/{agent => loop}/test/browser-act-fail-fast.test.ts (96%) rename packages/{agent => loop}/test/browser-cross-process.live.test.ts (87%) rename packages/{agent => loop}/test/browser-frame-collection.test.ts (93%) rename packages/{agent => loop}/test/browser-ref-lifecycle.test.ts (98%) rename packages/{pi-extension => loop}/test/browser-runtime.test.ts (75%) rename packages/{agent => loop}/test/browser-wait.test.ts (96%) rename packages/{agent => loop}/test/cdp.test.ts (97%) rename packages/{agent => loop}/test/e2e.live.test.ts (95%) rename packages/{agent => loop}/test/example-provider-matrix.test.ts (87%) rename packages/{pi-extension => loop}/test/extension.test.ts (94%) rename packages/{ai => loop}/test/google-provider.test.ts (91%) rename packages/{agent => loop}/test/harness-context.test.ts (91%) rename packages/{agent => loop}/test/keys.test.ts (96%) rename packages/{ai => loop}/test/menu.test.ts (71%) rename packages/{ai => loop}/test/models.test.ts (54%) rename packages/{ai => loop}/test/openai-adapter-routing.test.ts (85%) rename packages/{ai => loop}/test/openai-native-provider.test.ts (89%) rename packages/{pi-extension => loop}/test/pi-modes.test.ts (96%) rename packages/{agent => loop}/test/provider-retry.test.ts (99%) rename packages/{pi-extension => loop}/test/provider-stream.test.ts (75%) rename packages/{ai => loop}/test/providers.test.ts (68%) rename packages/{agent => loop}/test/published-declarations.test.ts (78%) rename packages/{pi-extension => loop}/test/published-package.test.ts (67%) rename packages/{agent => loop}/test/resources.test.ts (80%) rename packages/{pi-extension => loop}/test/selection.test.ts (80%) rename packages/{ai => loop}/test/tool-catalog.test.ts (68%) rename packages/{agent => loop}/test/tool-manager.test.ts (69%) rename packages/{agent => loop}/test/translator-browser.test.ts (96%) rename packages/{agent => loop}/test/translator.test.ts (95%) create mode 100644 packages/loop/tsconfig.build.json rename packages/{agent => loop}/tsconfig.json (100%) rename packages/{agent => loop}/tsdown.config.ts (82%) rename packages/{ai => loop}/vitest.config.ts (50%) create mode 100644 packages/loop/vitest.integration.config.ts delete mode 100644 packages/pi-extension/CHANGELOG.md delete mode 100644 packages/pi-extension/README.md delete mode 100644 packages/pi-extension/package.json delete mode 100644 packages/pi-extension/tsconfig.build.json delete mode 100644 packages/pi-extension/tsconfig.json delete mode 100644 packages/pi-extension/vitest.config.ts diff --git a/.agents/skills/release/SKILL.md b/.agents/skills/release/SKILL.md index 4ead80ca..e729f080 100644 --- a/.agents/skills/release/SKILL.md +++ b/.agents/skills/release/SKILL.md @@ -1,17 +1,15 @@ --- name: release -description: Prepare and publish @onkernel/cua-ai, @onkernel/cua-agent, and @onkernel/cua-pi-extension npm releases from kernel/cua. Use when checking release readiness, choosing package versions, writing package changelogs, committing release metadata to main, pushing package-prefixed tags, or monitoring release workflows. +description: Prepare and publish the @onkernel/loop npm release from kernel/cua. Use when checking release readiness, choosing a package version, writing the package changelog, committing release metadata to main, pushing package-prefixed tags, or monitoring the release workflow. --- # Release -Use this workflow to release `@onkernel/cua-ai`, `@onkernel/cua-agent`, and -`@onkernel/cua-pi-extension`. The packages do not need to release in lockstep. +Use this workflow to release `@onkernel/loop`. -`@onkernel/cua-pi-extension` has no release workflow yet, and that is -deliberate: it merges into the renamed single package, and a first publish under -a new name is manual regardless, because npm binds a trusted publisher to a -(repository, workflow filename) pair and a brand-new package name has none. +The first publish under the `@onkernel/loop` name is manual, because npm binds a +trusted publisher to a (repository, workflow filename) pair and a brand-new +package name has none. See `docs/npm-releases.md`. If a release run hits an unexpected bump, unclear decision, missing command, or avoidable manual step, update this skill as part of the release cleanup. Keep @@ -22,11 +20,8 @@ error-prone. | package | directory | tag prefix | workflow | | --- | --- | --- | --- | -| `@onkernel/cua-ai` | `packages/ai` | `cua-ai/v` | `release-cua-ai.yml` | -| `@onkernel/cua-agent` | `packages/agent` | `cua-agent/v` | `release-cua-agent.yml` | -| `@onkernel/cua-pi-extension` | `packages/pi-extension` | — | none yet (manual) | - -When both change, release in dependency order: `cua-ai`, then `cua-agent`. +| `@onkernel/loop` | `packages/loop` | `loop/v` | `release-loop.yml` | +| `@onkernel/ptywright` | `packages/ptywright` | — | not published | ## Quick Start @@ -42,60 +37,46 @@ git fetch --tags origin ```bash git status --short -npm view @onkernel/cua-ai versions --json -npm view @onkernel/cua-agent versions --json -test -f .github/workflows/release-cua-ai.yml -test -f .github/workflows/release-cua-agent.yml +npm view @onkernel/loop versions --json +test -f .github/workflows/release-loop.yml ``` -3. For each package, find the previous release tag: +3. Find the previous release tag: ```bash -git tag --list "cua-ai/v*" --sort=-v:refname | head -1 -git tag --list "cua-agent/v*" --sort=-v:refname | head -1 +git tag --list "loop/v*" --sort=-v:refname | head -1 ``` -If no tag exists, treat the next release as the package's current -`package.json` version unless npm already has that version. +If no tag exists, treat the next release as the current `package.json` version +unless npm already has that version. -4. Inspect package-specific changes since the last tag: +4. Inspect changes since the last tag: ```bash -git log --oneline ..HEAD -- packages/ai package.json package-lock.json tsconfig.base.json -git diff --name-status ..HEAD -- packages/ai package.json package-lock.json tsconfig.base.json - -git log --oneline ..HEAD -- packages/agent packages/ai package.json package-lock.json tsconfig.base.json -git diff --name-status ..HEAD -- packages/agent packages/ai package.json package-lock.json tsconfig.base.json +git log --oneline ..HEAD -- packages/loop package.json package-lock.json tsconfig.base.json +git diff --name-status ..HEAD -- packages/loop package.json package-lock.json tsconfig.base.json ``` -For a dependent package, include upstream package changes only when they affect -its published dependency version or runtime behavior. - ## Version Choice -Choose a version per package from source changes, existing npm versions, and the -previous tag: +Choose a version from source changes, existing npm versions, and the previous +tag: -- No package-relevant changes: do not release that package. -- Bug fixes, docs that affect package consumers, dependency metadata, or small - behavior fixes: patch. +- No consumer-relevant changes: do not release. +- Bug fixes, docs that affect consumers, dependency metadata, or small behavior + fixes: patch. - New exported APIs, new model/provider support, new examples intended for consumers, or materially expanded behavior: minor. -- Breaking API or behavior changes: major. While packages are `0.x`, use a - minor bump for breaking changes unless the package is intentionally moving to - `1.0.0`. +- Breaking API or behavior changes: major. While the package is `0.x`, use a + minor bump for breaking changes unless it is intentionally moving to `1.0.0`. -The candidate version must be greater than both the last tag for that package -and every version returned by `npm view versions --json`. +The candidate version must be greater than both the last tag and every version +returned by `npm view @onkernel/loop versions --json`. ## Changelog -Changes land under a `## Unreleased` heading as they merge, so every changelog -has at most one unreleased section: - -- `packages/ai/CHANGELOG.md` -- `packages/agent/CHANGELOG.md` -- `packages/pi-extension/CHANGELOG.md` +Changes land under a `## Unreleased` heading in `packages/loop/CHANGELOG.md` as +they merge, so the changelog has at most one unreleased section. Releasing renames that heading in place — do not add a second top entry: @@ -108,33 +89,25 @@ merges, so it can carry entries that contradict each other or describe a state that never shipped: an API added and then removed, or a note that a provider "keeps" a behavior when a later entry deletes that provider. Consumers upgrade from the previous release, not through the intermediate steps, so collapse -those into the net change and drop what nobody can observe. Cross-package -"update `@onkernel/cua-ai` to X" notes belong here too — the version is not -known until this step. +those into the net change and drop what nobody can observe. Write customer-facing changes. Do not dump commit subjects, internal issue -names, Slack context, or vague entries like "misc improvements." Group details -only when it improves readability. If the release is only metadata or docs, -say that plainly. +names, or vague entries like "misc improvements." Group details only when it +improves readability. If the release is only metadata or docs, say that plainly. Merges between releases add to `## Unreleased`, creating it directly under `# Changelog` when it is absent. Never invent a version heading for a merge: -package versions are chosen at release time from the accumulated changes, and a +the version is chosen at release time from the accumulated changes, and a per-merge heading claims a release that never happened. ## Edit Release Metadata -Set versions explicitly: +Set the version explicitly: ```bash -npm pkg set version= --workspace @onkernel/cua-ai -npm pkg set version= --workspace @onkernel/cua-agent +npm pkg set version= --workspace @onkernel/loop ``` -Ensure exact internal dependencies point at the versions that will be published -first: agent to AI. Edit the package manifests directly if `npm pkg set` is -awkward for scoped dependency keys. - Refresh the lockfile: ```bash @@ -143,84 +116,60 @@ npm install --package-lock-only ## Validate -Run the checks for each package being released: - ```bash npm ci -npm run build --workspace @onkernel/cua-ai -npm test --workspace @onkernel/cua-ai -npm pack --workspace @onkernel/cua-ai --dry-run +npm run build --workspace @onkernel/loop +npm run typecheck +npm test --workspace @onkernel/loop +npm pack --workspace @onkernel/loop --dry-run ``` -For `@onkernel/cua-agent`: - -```bash -npm run build --workspace @onkernel/cua-ai -npm run build --workspace @onkernel/cua-agent -npm test --workspace @onkernel/cua-agent -npm pack --workspace @onkernel/cua-agent --dry-run -``` +Build before testing: the pi print/RPC test loads the extension the way pi does, +through the package's own entry points. Run the full unit suite — do not pass +individual test files. Integration tests run separately (`npm run +test:integration --workspace @onkernel/loop`), and live e2e tests skip unless +`LOOP_E2E_LIVE=1` is set. -Run the full unit suites — do not pass individual test files. `cua-ai` -excludes integration/live tests by default (`npm run test:integration ---workspace @onkernel/cua-ai` runs them separately), and the `cua-agent` live -e2e tests skip unless `CUA_E2E_LIVE=1` is set. +Confirm the packed tarball ships `dist` and the extension source pi loads +through jiti, since `pi.extensions` points at `./src/pi-extension/index.ts`. -Do not push release tags if build, tests, or pack dry-runs fail. +Do not push a release tag if build, tests, or the pack dry-run fail. ## Commit To Main Direct commits to `main` are acceptable for release metadata. Keep the commit -limited to package versions, changelogs, and `package-lock.json`. +limited to the package version, changelog, and `package-lock.json`. ```bash git status --short -git add package-lock.json packages/ai/package.json packages/ai/CHANGELOG.md packages/agent/package.json packages/agent/CHANGELOG.md -git commit -m "Release CUA packages" +git add package-lock.json packages/loop/package.json packages/loop/CHANGELOG.md +git commit -m "Release @onkernel/loop v" git push origin main ``` -Use a package-specific commit message if releasing only one package, for -example `Release CUA AI v0.2.0`. - ## Tag And Push -After the release commit is on `main`, create annotated package tags at that -commit: - -```bash -git tag -a cua-ai/v -m "@onkernel/cua-ai v" -git push origin cua-ai/v -``` - -For the agent package: +After the release commit is on `main`, create an annotated tag at that commit: ```bash -git tag -a cua-agent/v -m "@onkernel/cua-agent v" -git push origin cua-agent/v +git tag -a loop/v -m "@onkernel/loop v" +git push origin loop/v ``` -Push and verify the AI tag before the agent tag. - ## Monitor -Find and watch the workflow run triggered by each tag: +Find and watch the workflow run triggered by the tag: ```bash -gh run list --workflow release-cua-ai.yml --json databaseId,status,conclusion,headBranch,displayTitle,url --limit 10 -gh run watch --exit-status - -gh run list --workflow release-cua-agent.yml --json databaseId,status,conclusion,headBranch,displayTitle,url --limit 10 +gh run list --workflow release-loop.yml --json databaseId,status,conclusion,headBranch,displayTitle,url --limit 10 gh run watch --exit-status ``` -After a workflow succeeds, verify npm: +After the workflow succeeds, verify npm: ```bash -npm view @onkernel/cua-ai@ version -npm dist-tag ls @onkernel/cua-ai -npm view @onkernel/cua-agent@ version -npm dist-tag ls @onkernel/cua-agent +npm view @onkernel/loop@ version +npm dist-tag ls @onkernel/loop ``` Then verify the published artifact actually imports — `npm view` only proves @@ -229,13 +178,11 @@ the version exists, not that the tarball is loadable: ```bash cd "$(mktemp -d)" npm init -y -npm install @onkernel/cua-ai@ -node --input-type=module -e "import('@onkernel/cua-ai').then((m) => { if (typeof m.getCuaModel !== 'function') process.exit(1); })" +npm install @onkernel/loop@ +node --input-type=module -e "import('@onkernel/loop').then((m) => { if (typeof m.compileLoopToolCatalog !== 'function') process.exit(1); })" +node --input-type=module -e "import('@onkernel/loop/pi').then((m) => { if (typeof m.attach !== 'function') process.exit(1); })" ``` -For `@onkernel/cua-agent`, install `@onkernel/cua-agent@` the same -way and check `typeof m.attach === "function"`. - -If a workflow fails after a tag is pushed, do not reuse the same package -version unless npm did not publish it. Fix forward with a new commit and a new -patch version when a package version has reached npm. +If the workflow fails after a tag is pushed, do not reuse the same version +unless npm did not publish it. Fix forward with a new commit and a new patch +version when a version has reached npm. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 19c82e9a..cad52ec1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,47 +20,21 @@ jobs: node-version: 22 cache: npm - run: npm ci - - run: npm run build --workspace @onkernel/cua-ai + # The pi print/RPC test loads the extension the way pi does, through the + # package's own entry points, so dist has to exist before the unit run. + - run: npm run build --workspace @onkernel/loop - name: Unit tests - run: npm test --workspace @onkernel/cua-ai + run: npm test --workspace @onkernel/loop - name: Pack tarball - run: npm pack --workspace @onkernel/cua-ai --pack-destination "$RUNNER_TEMP" + run: npm pack --workspace @onkernel/loop --pack-destination "$RUNNER_TEMP" - name: ESM import smoke test run: | mkdir -p "$RUNNER_TEMP/esm-smoke" cd "$RUNNER_TEMP/esm-smoke" npm init -y - npm install "$RUNNER_TEMP"/onkernel-cua-ai-*.tgz - node --input-type=module -e "import('@onkernel/cua-ai').then((m) => { if (typeof m.getCuaModel !== 'function') process.exit(1); })" - - agent-unit: - runs-on: ubuntu-latest - timeout-minutes: 15 - steps: - - uses: actions/checkout@v5 - - uses: actions/setup-node@v5 - with: - node-version: 22 - cache: npm - - run: npm ci - - run: npm run build --workspace @onkernel/cua-ai - - name: Agent unit tests - run: npm test --workspace @onkernel/cua-agent -- --exclude "**/*.live.test.ts" - - pi-extension-unit: - runs-on: ubuntu-latest - timeout-minutes: 15 - steps: - - uses: actions/checkout@v5 - - uses: actions/setup-node@v5 - with: - node-version: 22 - cache: npm - - run: npm ci - - run: npm run build --workspace @onkernel/cua-ai - - run: npm run build --workspace @onkernel/cua-agent - - name: Pi extension unit tests - run: npm test --workspace @onkernel/cua-pi-extension + npm install "$RUNNER_TEMP"/onkernel-loop-*.tgz + node --input-type=module -e "import('@onkernel/loop').then((m) => { if (typeof m.compileLoopToolCatalog !== 'function') process.exit(1); })" + node --input-type=module -e "import('@onkernel/loop/pi').then((m) => { if (typeof m.attach !== 'function') process.exit(1); })" typecheck-and-ptywright: runs-on: ubuntu-latest @@ -97,8 +71,6 @@ jobs: packages/ptywright/.cache packages/ptywright/native/build key: ptywright-${{ runner.os }}-${{ hashFiles('packages/ptywright/GHOSTTY_UPSTREAM', 'packages/ptywright/native/**', 'packages/ptywright/scripts/**') }} - - run: npm run build --workspace @onkernel/cua-ai - - run: npm run build --workspace @onkernel/cua-agent # The only job that typechecks the whole project graph rather than one # package, and the only one that builds and tests ptywright. - name: Typecheck workspace @@ -121,7 +93,6 @@ jobs: node-version: 22 cache: npm - run: npm ci - - run: npm run build --workspace @onkernel/cua-ai - name: Integration tests env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} @@ -130,9 +101,9 @@ jobs: META_API_KEY: ${{ secrets.META_API_KEY }} XAI_API_KEY: ${{ secrets.XAI_API_KEY }} MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} - run: npm run test:integration --workspace @onkernel/cua-ai + run: npm run test:integration --workspace @onkernel/loop - agent-e2e: + live-e2e: runs-on: ubuntu-latest timeout-minutes: 45 # Only run on the main repo (not forks) so secrets are available. @@ -145,11 +116,9 @@ jobs: node-version: 22 cache: npm - run: npm ci - - run: npm run build --workspace @onkernel/cua-ai - - run: npm run build --workspace @onkernel/cua-agent - - name: Agent live smoke tests (all providers) + - name: Live smoke tests (all providers) env: - CUA_E2E_LIVE: "1" + LOOP_E2E_LIVE: "1" KERNEL_API_KEY: ${{ secrets.KERNEL_API_KEY }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} @@ -157,4 +126,4 @@ jobs: META_API_KEY: ${{ secrets.META_API_KEY }} XAI_API_KEY: ${{ secrets.XAI_API_KEY }} MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} - run: npm test --workspace @onkernel/cua-agent -- test/e2e.live.test.ts + run: npm run test:integration --workspace @onkernel/loop -- test/e2e.live.test.ts diff --git a/.github/workflows/release-cua-ai.yml b/.github/workflows/release-cua-ai.yml deleted file mode 100644 index 8180489b..00000000 --- a/.github/workflows/release-cua-ai.yml +++ /dev/null @@ -1,77 +0,0 @@ -name: Release CUA AI - -on: - push: - tags: - - "cua-ai/v*" - -permissions: - contents: read - id-token: write - -concurrency: - group: release-cua-ai-${{ github.ref_name }} - cancel-in-progress: false - -jobs: - publish: - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@v5 - with: - fetch-depth: 0 - - - name: Verify tag is on main - run: | - git fetch origin main:refs/remotes/origin/main - git merge-base --is-ancestor "$GITHUB_SHA" origin/main - - - uses: actions/setup-node@v5 - with: - node-version: 24 - registry-url: https://registry.npmjs.org - - - name: Ensure npm supports trusted publishing - run: npm install -g npm@^11.5.1 - - - run: npm ci - - - name: Verify package version matches tag - run: | - node --input-type=module <<'EOF' - import { readFileSync } from "node:fs"; - - const tag = process.env.GITHUB_REF_NAME; - const prefix = "cua-ai/v"; - if (!tag?.startsWith(prefix)) { - throw new Error(`Expected tag to start with ${prefix}, got ${tag}`); - } - - const tagVersion = tag.slice(prefix.length); - const pkg = JSON.parse(readFileSync("packages/ai/package.json", "utf8")); - if (pkg.version !== tagVersion) { - throw new Error(`Tag version ${tagVersion} does not match ${pkg.name} package.json version ${pkg.version}`); - } - - console.log(`${pkg.name}@${pkg.version}`); - EOF - - - run: npm run build --workspace @onkernel/cua-ai - - - name: Unit tests - run: npm test --workspace @onkernel/cua-ai - - - name: Pack tarball - run: npm pack --workspace @onkernel/cua-ai --pack-destination "$RUNNER_TEMP" - - - name: ESM import smoke test - run: | - mkdir -p "$RUNNER_TEMP/esm-smoke" - cd "$RUNNER_TEMP/esm-smoke" - npm init -y - npm install "$RUNNER_TEMP"/onkernel-cua-ai-*.tgz - node --input-type=module -e "import('@onkernel/cua-ai').then((m) => { if (typeof m.getCuaModel !== 'function') process.exit(1); })" - - - name: Publish to npm - run: npm publish --workspace @onkernel/cua-ai --access public diff --git a/.github/workflows/release-cua-agent.yml b/.github/workflows/release-loop.yml similarity index 53% rename from .github/workflows/release-cua-agent.yml rename to .github/workflows/release-loop.yml index 15f47bac..65effe6d 100644 --- a/.github/workflows/release-cua-agent.yml +++ b/.github/workflows/release-loop.yml @@ -1,16 +1,16 @@ -name: Release CUA Agent +name: Release Loop on: push: tags: - - "cua-agent/v*" + - "loop/v*" permissions: contents: read id-token: write concurrency: - group: release-cua-agent-${{ github.ref_name }} + group: release-loop-${{ github.ref_name }} cancel-in-progress: false jobs: @@ -39,60 +39,54 @@ jobs: - name: Verify package version matches tag run: | - node --input-type=module <<'EOF' + node --input-type=module <<'NODE' import { readFileSync } from "node:fs"; const tag = process.env.GITHUB_REF_NAME; - const prefix = "cua-agent/v"; + const prefix = "loop/v"; if (!tag?.startsWith(prefix)) { throw new Error(`Expected tag to start with ${prefix}, got ${tag}`); } const tagVersion = tag.slice(prefix.length); - const pkg = JSON.parse(readFileSync("packages/agent/package.json", "utf8")); + const pkg = JSON.parse(readFileSync("packages/loop/package.json", "utf8")); if (pkg.version !== tagVersion) { throw new Error(`Tag version ${tagVersion} does not match ${pkg.name} package.json version ${pkg.version}`); } console.log(`${pkg.name}@${pkg.version}`); - EOF + NODE - - run: npm run build --workspace @onkernel/cua-ai - - run: npm run build --workspace @onkernel/cua-agent - - run: npm run typecheck --workspace @onkernel/cua-agent + - run: npm run build --workspace @onkernel/loop + - run: npm run typecheck --workspace @onkernel/loop - name: Unit tests - run: npm test --workspace @onkernel/cua-agent -- --exclude "**/*.live.test.ts" + run: npm test --workspace @onkernel/loop - - name: Verify dependency package is published - run: npm view @onkernel/cua-ai@$(node -p 'require("./packages/agent/package.json").dependencies["@onkernel/cua-ai"]') version - - - name: Pack tarballs - run: | - mkdir -p /tmp/pack - npm pack --workspace @onkernel/cua-ai --pack-destination /tmp/pack - npm pack --workspace @onkernel/cua-agent --pack-destination /tmp/pack + - name: Pack tarball + run: npm pack --workspace @onkernel/loop --pack-destination "$RUNNER_TEMP" - name: ESM import smoke test run: | SMOKE_DIR=$(mktemp -d) cd "$SMOKE_DIR" npm init -y > /dev/null - npm install /tmp/pack/*.tgz - cat > smoke.mjs <<'EOF' - import { cua, CuaAgent, CuaAgentHarness, formatBrowserActResult, NodeExecutionEnv } from "@onkernel/cua-agent"; + npm install "$RUNNER_TEMP"/onkernel-loop-*.tgz + cat > smoke.mjs <<'NODE' + import { compileLoopToolCatalog, formatBrowserActResult, loop } from "@onkernel/loop"; + import { attach, getLoopModel, NodeExecutionEnv } from "@onkernel/loop/pi"; - for (const [name, value] of Object.entries({ CuaAgent, CuaAgentHarness, formatBrowserActResult, NodeExecutionEnv })) { + for (const [name, value] of Object.entries({ compileLoopToolCatalog, formatBrowserActResult, attach, getLoopModel, NodeExecutionEnv })) { if (typeof value !== "function") { throw new Error(`expected ${name} to be a function, got ${typeof value}`); } } - if (typeof cua !== "object" || cua === null) { - throw new Error(`expected cua to be an object, got ${typeof cua}`); + if (typeof loop !== "object" || loop === null) { + throw new Error(`expected loop to be an object, got ${typeof loop}`); } console.log("ESM import smoke OK"); - EOF + NODE node smoke.mjs - name: Publish to npm - run: npm publish --workspace @onkernel/cua-agent --access public + run: npm publish --workspace @onkernel/loop --access public diff --git a/.gitignore b/.gitignore index 7fb0578f..fb65c792 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ dist/ *.tsbuildinfo packages/*/dist/ packages/*/dist-tsc/ +packages/*/dist-published/ packages/*/*.tsbuildinfo packages/*/native/build/ packages/ptywright/.cache/ diff --git a/README.md b/README.md index 91476f43..efe1e9a5 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# cua +# loop Browser tools for your agent, built on [pi](https://github.com/earendil-works/pi). @@ -6,13 +6,13 @@ Point any model at a [Kernel cloud browser](https://kernel.sh/): pick the tools, get plain agent objects back, and run whatever loop you already have. ```ts -import { attach } from "@onkernel/cua-agent"; -import { cua } from "@onkernel/cua-ai"; +import { loop } from "@onkernel/loop"; +import { attach } from "@onkernel/loop/pi"; const kb = attach({ client, browser }); const { model, agentTools, models } = kb.compile({ model: "anthropic:claude-opus-5", - tools: cua.toolsets.browser(), + tools: loop.toolsets.browser(), }); ``` @@ -20,8 +20,8 @@ Already in pi? Install the extension instead and keep pi's session, UI, and model selection: ```bash -pi install npm:@onkernel/cua-pi-extension -pi -p --cua-tools browser,browser-act "open example.com and report the heading" +pi install npm:@onkernel/loop +pi -p --browser-tools browser,browser-act "open example.com and report the heading" ``` --- @@ -40,7 +40,7 @@ All of them expect you to: it had the intended effect. 4. Know which of those protocols the model you picked actually accepts. -This repo does all of that and stops there. `@onkernel/cua-ai` represents the +This repo does all of that and stops there. `@onkernel/loop` represents the provider differences as an explicit, identity-keyed tool catalog; you choose the exact tools, and provider transforms compose only the declarations and request fields those identities require. It does not supply an agent class, a session @@ -52,32 +52,30 @@ format, or a front-end — your framework already has those. ``` packages/ -├── ai/ # @onkernel/cua-ai - model catalog, tool schemas, provider adapters -├── agent/ # @onkernel/cua-agent - Kernel-browser tool execution -├── pi-extension/ # @onkernel/cua-pi-extension - the same tools inside pi's own session -└── ptywright/ # @onkernel/ptywright - development-only PTY/TUI test infrastructure +├── loop/ # @onkernel/loop - tools, catalog compilation, pi bindings, pi extension +└── ptywright/ # @onkernel/ptywright - development-only PTY/TUI test infrastructure ``` -| Package | What it ships | +| Entry point | What it ships | | --- | --- | -| [`@onkernel/cua-ai`](packages/ai) | Model catalog, tool factories/toolsets, per-model compatibility checks, provider adapters. | -| [`@onkernel/cua-agent`](packages/agent) | `attach()`: binds a Kernel browser and compiles a (model, tools) pair into plain pi objects. | -| [`@onkernel/cua-pi-extension`](packages/pi-extension) | A pi extension contributing those tools to pi's own agent session. | +| `@onkernel/loop` | Canonical actions, tool factories/toolsets, catalog compilation, the tool menu, and Kernel-browser execution. | +| `@onkernel/loop/pi` | `attach()`: binds a Kernel browser and compiles a (model, tools) pair into plain pi objects, plus model resolution and provider adapters. | +| `pi.extensions` | A pi extension contributing those tools to pi's own agent session. | | [`@onkernel/ptywright`](packages/ptywright) | Development-only PTY/TUI test infrastructure. | ```mermaid flowchart LR - ai[("@onkernel/cua-ai")] - agent[("@onkernel/cua-agent")] - ext[("@onkernel/cua-pi-extension")] - pi[("pi-agent-core / pi-ai / pi-coding-agent")] - sdk[("@onkernel/sdk")] - ai --> agent - agent --> ext - ai --> ext - pi --> agent + core["@onkernel/loop"] + pibind["@onkernel/loop/pi"] + ext["pi extension"] + pi["pi-agent-core / pi-ai / pi-coding-agent"] + sdk["@onkernel/sdk"] + core --> pibind + core --> ext + pibind --> ext + pi --> pibind pi --> ext - sdk --> agent + sdk --> core sdk --> ext ``` @@ -86,12 +84,12 @@ flowchart LR ## Building an agent `attach()` binds the browser once; `compile()` turns a (model, tools) pair into -plain pi objects. Nothing here is a CUA type you have to learn: +plain pi objects. Nothing here is a Loop type you have to learn: ```ts import Kernel from "@onkernel/sdk"; -import { cua } from "@onkernel/cua-ai"; -import { Agent, attach } from "@onkernel/cua-agent"; +import { loop } from "@onkernel/loop"; +import { Agent, attach } from "@onkernel/loop/pi"; const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); const browser = await client.browsers.create({ stealth: true }); @@ -99,7 +97,7 @@ const kb = attach({ client, browser }); const { model, agentTools, models } = kb.compile({ model: "anthropic:claude-opus-5", - tools: [...cua.toolsets.browser(), cua.tools.browser.act()], + tools: [...loop.toolsets.browser(), loop.tools.browser.act()], }); const agent = new Agent({ @@ -118,7 +116,7 @@ try { The compiled `model` carries the transport its tools derive: selecting a provider-native browser or computer surface can change `model.api`, so the pair has to reach the agent together. See -[`packages/agent/README.md`](packages/agent/README.md) for the harness variant, +[`packages/loop/README.md`](packages/loop/README.md) for the harness variant, swapping tools on a running session, and tool contexts. ## Choosing tools for a model @@ -126,9 +124,10 @@ swapping tools on a running session, and tool contexts. Not every model accepts every tool. Ask, rather than guess: ```ts -import { cuaToolMenu, getCuaModel } from "@onkernel/cua-ai"; +import { loopToolMenu } from "@onkernel/loop"; +import { getLoopModel } from "@onkernel/loop/pi"; -for (const entry of cuaToolMenu(getCuaModel("openai:gpt-5.6-sol"))) { +for (const entry of loopToolMenu(getLoopModel("openai:gpt-5.6-sol"))) { console.log(entry.label, entry.available ? "ok" : `unavailable: ${entry.unavailableReason}`); } ``` @@ -143,11 +142,10 @@ per-tool verdict. ## How it works -1. **Model layer** — `@onkernel/cua-ai` opens pi-ai's whole model catalog, with - stable tool identities, explicit tool factories/toolsets, per-model - compatibility checks, and provider declarations/headers/payload transforms. - Compilation is declaration-only: it never sees an executor. -2. **Execution layer** — `@onkernel/cua-agent` materializes the caller's exact +1. **Model layer** — `@onkernel/loop/pi` opens pi-ai's whole model catalog and + composes provider declarations, headers, and payload transforms around it. + Catalog compilation is declaration-only: it never sees an executor. +2. **Execution layer** — `@onkernel/loop` materializes the caller's exact catalog over one shared resource pool and executes canonical actions through Kernel's computer API or a raw-CDP browser executor. 3. **Transport** — the compiled catalog derives `model.api` from the selected @@ -165,13 +163,14 @@ See [`docs/architecture.md`](docs/architecture.md) for the full end-to-end flow. ```bash npm ci npm run typecheck -npm test --workspace @onkernel/cua-ai -npm test --workspace @onkernel/cua-agent -npm test --workspace @onkernel/cua-pi-extension +npm run build --workspace @onkernel/loop +npm test --workspace @onkernel/loop ``` -`cua-agent`'s live end-to-end tests skip unless `CUA_E2E_LIVE=1` is set, and -`cua-ai` runs integration tests separately via `npm run test:integration`. +Build before testing: the pi print/RPC test loads the extension the way pi does, +through the package's own entry points. Live end-to-end tests skip unless +`LOOP_E2E_LIVE=1` is set, and integration tests run separately via +`npm run test:integration`. --- diff --git a/docs/agent-tool-configuration-spec.md b/docs/agent-tool-configuration-spec.md index 53d96bc7..27ff0f86 100644 --- a/docs/agent-tool-configuration-spec.md +++ b/docs/agent-tool-configuration-spec.md @@ -7,7 +7,7 @@ rule it establishes still holds; what changed is where the array lives. `CuaAgen and its in-tool `executionMode: "sequential"` guard described below no longer exist. Retained as the record of why the tool array is explicit and required. -**Scope:** `@onkernel/cua-agent` and the tool-building surface in `@onkernel/cua-ai` +**Scope:** `@onkernel/loop` (written when this code lived in two packages) **Compatibility:** Not a goal; these packages are alpha and may make breaking API changes. ## Summary @@ -16,15 +16,15 @@ Retained as the record of why the tool array is explicit and required. The array may contain: -- CUA-authored tools, such as browser snapshots, browser action plans, browser waits, batches, and `playwright_execute` +- Loop-authored tools, such as browser snapshots, browser action plans, browser waits, batches, and `playwright_execute` - provider-defined native browser or computer tools and predefined toolsets - ordinary caller-provided `AgentTool` objects -Provider namespaces expose only surfaces justified by linked first-party documentation. CUA-authored capabilities remain separate under `cua.tools` and `cua.toolsets`. A caller can combine either category with custom application tools while seeing exactly what the model receives. +Provider namespaces expose only surfaces justified by linked first-party documentation. Loop-authored capabilities remain separate under `loop.tools` and `loop.toolsets`. A caller can combine either category with custom application tools while seeing exactly what the model receives. -The former `extraTools`, `mode`, `nativeTool`, and `playwright` constructor options are removed. `activeToolNames`, `setActiveTools()`, `setMode()`, `getMode()`, `computer_use_extra`, and CUA-generated default system prompts are also removed. No global or derived mode replaces them. +The former `extraTools`, `mode`, `nativeTool`, and `playwright` constructor options are removed. `activeToolNames`, `setActiveTools()`, `setMode()`, `getMode()`, `computer_use_extra`, and Loop-generated default system prompts are also removed. No global or derived mode replaces them. -Each CUA tool specification must contain enough information to build, expose, execute, and describe that tool independently. Convenience toolsets may return arrays of tool specifications, but they must not establish hidden runtime state or add undeclared tools. +Each Loop tool specification must contain enough information to build, expose, execute, and describe that tool independently. Convenience toolsets may return arrays of tool specifications, but they must not establish hidden runtime state or add undeclared tools. ## Motivation @@ -59,7 +59,7 @@ The public distinction should instead be simple: For example, Anthropic's native `browser` is one tool with multiple actions. `browser_act` is one tool containing a sequence of step actions. `browser_snapshot` is a single-purpose tool whose input does not need an action discriminator. -`CuaAction` may remain an internal normalized execution representation, but agent constructors should not expose it as their tool-selection API. +`ComputerUseAction` may remain an internal normalized execution representation, but agent constructors should not expose it as their tool-selection API. This terminology must also be used consistently in the architecture document, package READMEs, API documentation, and user-facing examples. Users should not need to understand the internal action IR to configure model-facing tools. @@ -67,8 +67,8 @@ This terminology must also be used consistently in the architecture document, pa 1. Make the exact model-facing tool catalog obvious at the constructor call site. 2. Support minimal and empty configurations without hidden additions. -3. Distinguish first-party provider surfaces from CUA-authored capabilities. -4. Allow provider-native, CUA-authored, Playwright, and caller tools to compose in one list. +3. Distinguish first-party provider surfaces from Loop-authored capabilities. +4. Allow provider-native, Loop-authored, Playwright, and caller tools to compose in one list. 5. Let every tool own the runtime policy required to execute it correctly. 6. Support cache-aware mid-conversation tool additions, removals, and replacements. 7. Validate tool/model and tool/tool incompatibilities directly and early. @@ -82,43 +82,43 @@ This terminology must also be used consistently in the architecture document, pa - Exposing the internal canonical action IR as constructor configuration - Automatically adding prerequisite, navigation, screenshot, or fallback tools - Silently replacing incompatible tools when the model changes -- Preserving CUA's current default system prompts +- Preserving Loop's current default system prompts ## Public namespace Tool factories and toolsets should be exported through one discoverable namespace rather than as a collection of global functions. -CUA-authored capabilities live under `cua.tools` and `cua.toolsets`: +Loop-authored capabilities live under `loop.tools` and `loop.toolsets`: ```ts -cua.tools.browser.snapshot() -cua.tools.browser.act() -cua.tools.browser.waitFor() -cua.tools.browser.batch(...) -cua.tools.computer.batch(...) -cua.tools.playwright() - -cua.toolsets.browser() -cua.toolsets.computer() -cua.toolsets.mixed() +loop.tools.browser.snapshot() +loop.tools.browser.act() +loop.tools.browser.waitFor() +loop.tools.browser.batch(...) +loop.tools.computer.batch(...) +loop.tools.playwright() + +loop.toolsets.browser() +loop.toolsets.computer() +loop.toolsets.mixed() ``` Provider-defined surfaces live under provider namespaces and carry their first-party source: ```ts -cua.providers.anthropic.source -cua.providers.anthropic.tools.browser(...) -cua.providers.anthropic.tools.computer(...) +loop.providers.anthropic.source +loop.providers.anthropic.tools.browser(...) +loop.providers.anthropic.tools.computer(...) -cua.providers.google.source -cua.providers.google.toolsets.browser() +loop.providers.google.source +loop.providers.google.toolsets.browser() ``` The distinction is deliberate: -- `cua.providers.` contains only documented native declarations or predefined toolsets. Each namespace exposes its first-party `source` or versioned `sources`, and every returned spec carries the applicable URL. -- `cua.tools` contains additional tools CUA designed, such as snapshots, semantic waits, browser action plans, browser batches, and Playwright execution. -- `cua.toolsets` contains CUA-curated combinations of CUA-authored tools. +- `loop.providers.` contains only documented native declarations or predefined toolsets. Each namespace exposes its first-party `source` or versioned `sources`, and every returned spec carries the applicable URL. +- `loop.tools` contains additional tools Loop designed, such as snapshots, semantic waits, browser action plans, browser batches, and Playwright execution. +- `loop.toolsets` contains Loop-curated combinations of Loop-authored tools. The exact property names may be refined, but the final exports must remain namespaced, autocomplete-friendly, and free of a large flat list of package-level tool factory functions. @@ -132,8 +132,8 @@ const agent = new CuaAgent({ client, initialState: { model: "anthropic:claude-opus-5" }, tools: [ - cua.tools.browser.snapshot(), - cua.tools.browser.act(), + loop.tools.browser.snapshot(), + loop.tools.browser.act(), customerLookupTool, ], }); @@ -146,7 +146,7 @@ const harness = new CuaAgentHarness({ session, model: "openai:gpt-5.6-sol", tools: [ - cua.tools.playwright(), + loop.tools.playwright(), ], }); ``` @@ -154,21 +154,21 @@ const harness = new CuaAgentHarness({ Conceptually: ```ts -// Defined and exported by @onkernel/cua-agent. -type CuaAgentTool = CuaToolSpec | AgentTool; +// Defined and exported by @onkernel/loop. +type LoopAgentTool = LoopToolSpec | AgentTool; interface CuaAgentOptions { // Existing non-tool options omitted. - tools: CuaAgentTool[]; + tools: LoopAgentTool[]; } interface CuaAgentHarnessOptions { // Existing non-tool options omitted. - tools: CuaAgentTool[]; + tools: LoopAgentTool[]; } ``` -A `CuaToolSpec` is declarative because `@onkernel/cua-agent` must materialize it against the Kernel browser, SDK client, selected model, and provider transport. An `AgentTool` is already executable and can be installed directly; cua-agent projects it into a fresh declaration-only object before cua-ai compiles the catalog, so cua-ai never sees executors. +A `LoopToolSpec` is declarative because `@onkernel/loop` must materialize it against the Kernel browser, SDK client, selected model, and provider transport. An `AgentTool` is already executable and can be installed directly; the tool manager projects it into a fresh declaration-only object before the catalog is compiled, so compilation never sees executors. There is one current tool list, not separate installed and active lists. `setTools()` changes that list for subsequent provider requests. @@ -178,7 +178,7 @@ There is one current tool list, not separate installed and active lists. `setToo ```ts tools: [ - cua.providers.anthropic.tools.browser({ + loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true, }), @@ -186,13 +186,13 @@ tools: [ ] ``` -The model receives exactly the native browser tool and `customer_lookup`. CUA must not add canonical browser tools, navigation helpers, screenshots, batches, or Playwright. +The model receives exactly the native browser tool and `customer_lookup`. Loop must not add canonical browser tools, navigation helpers, screenshots, batches, or Playwright. ### Playwright only ```ts tools: [ - cua.tools.playwright(), + loop.tools.playwright(), ] ``` @@ -202,18 +202,18 @@ The model receives exactly `playwright_execute`. ```ts tools: [ - cua.tools.browser.act(), + loop.tools.browser.act(), ] ``` -The model receives exactly `browser_act`. CUA may warn that ref-based steps require refs from another source, but it must not silently add a snapshot tool. +The model receives exactly `browser_act`. Loop may warn that ref-based steps require refs from another source, but it must not silently add a snapshot tool. A practical minimal ref-based plan configuration is explicit: ```ts tools: [ - cua.tools.browser.snapshot(), - cua.tools.browser.act(), + loop.tools.browser.snapshot(), + loop.tools.browser.act(), ] ``` @@ -221,22 +221,22 @@ tools: [ ```ts tools: [ - cua.tools.browser.snapshot(), - cua.tools.browser.find(), - cua.tools.browser.text(), - cua.tools.browser.act(), - cua.tools.browser.waitFor(), - cua.tools.browser.navigate(), + loop.tools.browser.snapshot(), + loop.tools.browser.find(), + loop.tools.browser.text(), + loop.tools.browser.act(), + loop.tools.browser.waitFor(), + loop.tools.browser.navigate(), ] ``` -### Native and CUA-authored tools together +### Native and Loop-authored tools together ```ts tools: [ - cua.providers.anthropic.tools.computer({ version: "20260701" }), - cua.tools.browser.snapshot(), - cua.tools.browser.act(), + loop.providers.anthropic.tools.computer({ version: "20260701" }), + loop.tools.browser.snapshot(), + loop.tools.browser.act(), ] ``` @@ -255,22 +255,22 @@ No tool is added implicitly. Convenience helpers provide ordinary arrays: ```ts -tools: cua.providers.google.toolsets.browser() +tools: loop.providers.google.toolsets.browser() ``` ```ts -tools: cua.toolsets.browser() +tools: loop.toolsets.browser() ``` ```ts -tools: cua.toolsets.mixed() +tools: loop.toolsets.mixed() ``` Callers can inspect and compose them: ```ts tools: [ - ...cua.toolsets.browser(), + ...loop.toolsets.browser(), customerLookupTool, ] ``` @@ -283,20 +283,20 @@ them explicitly: ```ts tools: [ - ...cua.toolsets.browser(), - cua.tools.browser.act(), + ...loop.toolsets.browser(), + loop.tools.browser.act(), ] ``` -The CLI uses this explicit composition for its structured CUA-browser catalogs. +The CLI uses this explicit composition for its structured Loop-browser catalogs. -Provider toolsets must expose the first-party source they mirror and must not silently include CUA-authored additions. +Provider toolsets must expose the first-party source they mirror and must not silently include Loop-authored additions. ## No global or derived mode The runtime must not derive `computer`, `browser`, or `hybrid` state from the selected tools. Those labels are too coarse to govern execution safely. -Instead, each `CuaToolSpec` supplies the policy needed for that tool to do its work: +Instead, each `LoopToolSpec` supplies the policy needed for that tool to do its work: - stable tool identity and preferred model-facing name - description and schema or native declaration @@ -336,7 +336,7 @@ The implemented naming policy is: A toolset factory should not need hidden global state. The central composer sees all expanded tool specs and applies the collision policy. A toolset may expose explicit naming or namespace options, but automatic context-sensitive aliasing must not make the resulting catalog unpredictable. -Catalog tests cover provider-native tools composed with CUA browser and caller tools, and verify first-party sources for every provider surface. +Catalog tests cover provider-native tools composed with Loop browser and caller tools, and verify first-party sources for every provider surface. ## Tools and actions @@ -367,7 +367,7 @@ Current action-bearing tools include: - `browser_batch`, which would accept ordered browser-plane actions - `browser_act`, whose `steps` are dependent browser actions with optional semantic expectations -Some single-purpose tools do not need an explicit action argument. Internally converting their call into a `CuaAction` does not make the public callable surface an action. +Some single-purpose tools do not need an explicit action argument. Internally converting their call into a `ComputerUseAction` does not make the public callable surface an action. ## Batch tools @@ -375,22 +375,22 @@ Batch tools need first-class treatment in this design rather than inheriting an ### Computer batch -`computer_batch` is a CUA-authored tool over computer-plane actions. Its factory should let the caller control the allowed action schema: +`computer_batch` is a Loop-authored tool over computer-plane actions. Its factory should let the caller control the allowed action schema: ```ts -cua.tools.computer.batch({ +loop.tools.computer.batch({ actions: ["click", "type", "keypress", "screenshot"], }) ``` -A CUA toolset may choose and document a default batch configuration, but constructing the batch tool directly must make its allowed actions visible. The batch must not gain actions merely because unrelated individual tools are present. +A Loop toolset may choose and document a default batch configuration, but constructing the batch tool directly must make its allowed actions visible. The batch must not gain actions merely because unrelated individual tools are present. ### Browser batch -CUA offers a browser-plane equivalent that does not dispatch OS computer-use input: +Loop offers a browser-plane equivalent that does not dispatch OS computer-use input: ```ts -cua.tools.browser.batch({ +loop.tools.browser.batch({ actions: ["snapshot", "click", "fill", "wait_for", "text"], }) ``` @@ -433,7 +433,7 @@ Both agent classes must support changing the exact tool list between model reque ```ts await harness.setTools([ ...harness.getTools(), - cua.tools.browser.act(), + loop.tools.browser.act(), ]); ``` @@ -450,9 +450,9 @@ The implementation should build on pi's dynamic tool-loading semantics: Purely additive changes must preserve the stable provider prompt/schema prefix when the provider supports native deferred loading. Existing tools must not be renamed or reordered merely because another tool was added. -Tool descriptions should carry the instructions needed by lazily added tools. CUA should not modify the system prompt when the tool list changes, because doing so can invalidate the provider cache even when deferred tool schemas are supported. +Tool descriptions should carry the instructions needed by lazily added tools. Loop should not modify the system prompt when the tool list changes, because doing so can invalidate the provider cache even when deferred tool schemas are supported. -`setTools()` must be coherent with CUA's materialized executors, payload transforms, headers, and shared resources. It must not update only pi's visible list while leaving an independent CUA runtime stale. +`setTools()` must be coherent with Loop's materialized executors, payload transforms, headers, and shared resources. It must not update only pi's visible list while leaving an independent Loop runtime stale. ## Model changes @@ -462,7 +462,7 @@ The requested tool list remains caller-owned when a model changes. await harness.setModel("openai:gpt-5.6-sol"); ``` -CUA revalidates the same tool specifications against the new model. It must not silently replace, add, remove, or rename tools. +Loop revalidates the same tool specifications against the new model. It must not silently replace, add, remove, or rename tools. An incompatible native tool produces a direct error: @@ -470,26 +470,26 @@ An incompatible native tool produces a direct error: anthropic browser_20260701 requires an Anthropic model; selected openai:gpt-5.6-sol ``` -Caller-provided generic tools and compatible CUA-authored tools remain installed. +Caller-provided generic tools and compatible Loop-authored tools remain installed. ## One current tool list `tools` defines the current catalog exposed to the model. There is no separate constructor-level installed catalog and active subset. -The following CUA-facing configuration should be removed: +The following Loop-facing configuration should be removed: ```ts activeToolNames setActiveTools() ``` -Callers use `setTools()` for additions, removals, and replacements. CUA may use pi's registration and activation machinery internally to implement deferred loading, but that distinction must not become a second public source of truth in `CuaAgent` or `CuaAgentHarness`. +Callers use `setTools()` for additions, removals, and replacements. Loop may use pi's registration and activation machinery internally to implement deferred loading, but that distinction must not become a second public source of truth in `CuaAgent` or `CuaAgentHarness`. The CLI's interactive `/tools` menu is an application-level consumer of exactly this contract, not a second mechanism. It holds the list it composed for the active model as the baseline, and applies a user-selected **subset** of that baseline through one `setTools()` call. It never adds a tool the application did not compose, so it cannot introduce an unsupported tool. Because tool identities are provider-specific, a `/model` change rebuilds the baseline from the new model's defaults and discards the previous selection with an explicit notice — the alternative, re-applying a selection by key across providers, is the silent replacement forbidden under Non-goals. ## System instructions and descriptions -CUA should get out of the business of generating default system prompts. +Loop should get out of the business of generating default system prompts. The model should learn what is available from the exact tool names, descriptions, and schemas it receives. Correctness-critical prerequisites belong in tool descriptions and schemas. @@ -497,7 +497,7 @@ For example, `browser_act` must explain that ref-based steps require current ref Selecting a provider-native tool or predefined toolset must not silently install the provider's example system prompt. The caller owns the system prompt. -CUA tool specifications should not contribute `promptSnippet`, `promptGuidelines`, or active-tool-specific system-prompt fragments by default. This keeps tool additions cache-friendly and makes `tools: []` genuinely free of CUA interaction instructions. +Loop tool specifications should not contribute `promptSnippet`, `promptGuidelines`, or active-tool-specific system-prompt fragments by default. This keeps tool additions cache-friendly and makes `tools: []` genuinely free of Loop interaction instructions. If a correctness requirement cannot be expressed in a tool description or schema, that is a design issue to resolve explicitly before adding system-prompt generation back into scope. @@ -526,11 +526,11 @@ Native adapters compose by selected identity: OpenAI replaces only its native co No replacement navigation helper is added automatically or under a new hidden name. A caller who needs navigation chooses an explicit capability, such as: - a provider-native browser tool -- `cua.tools.browser.navigate()` -- `cua.tools.playwright()` +- `loop.tools.browser.navigate()` +- `loop.tools.playwright()` - a caller-provided navigation tool -An OS-computer-only toolset may still navigate through ordinary keyboard input. CUA should not silently append a separate escape-hatch tool. +An OS-computer-only toolset may still navigate through ordinary keyboard input. Loop should not silently append a separate escape-hatch tool. ## Error behavior @@ -539,7 +539,7 @@ Construction, `setTools()`, or model switching should fail with errors that name Examples: ```text -tool name "browser_act" is requested by both cua.browser.act and custom.plan +tool name "browser_act" is requested by both kloop.browser.act and custom.plan ``` ```text @@ -554,7 +554,7 @@ tools "provider..native.computer" and "provider..native.browser" require c provider google does not accept the schema used by "browser_act" ``` -CUA must not silently drop tools, substitute a different selected toolset, append tools, or rename an existing tool after a dynamic addition. A selected native tool may declare an equivalent function-transport fallback under the same identity, name, schema, and executor for credentials that cannot access the native provider feature; this does not change the caller's tool catalog. +Loop must not silently drop tools, substitute a different selected toolset, append tools, or rename an existing tool after a dynamic addition. A selected native tool may declare an equivalent function-transport fallback under the same identity, name, schema, and executor for credentials that cannot access the native provider feature; this does not change the caller's tool catalog. ## Removal of current API @@ -568,7 +568,7 @@ playwright activeToolNames ``` -The following methods are removed from the CUA-facing API: +The following methods are removed from the Loop-facing API: ```ts setMode() @@ -576,18 +576,18 @@ getMode() setActiveTools() ``` -`computer_use_extra` and CUA-generated default system prompts are removed with them. +`computer_use_extra` and Loop-generated default system prompts are removed with them. Their replacements are direct tool-list entries: | Current option | Replacement | | --- | --- | | `extraTools: [tool]` | include `tool` in `tools` | -| `mode: "computer"` | `tools: cua.toolsets.computer()` or an explicit provider-native list | -| `mode: "browser"` | `tools: cua.toolsets.browser()` or an explicit list | -| `mode: "hybrid"` | compose the desired provider and CUA tools explicitly | -| `nativeTool: spec` | `tools: [cua.providers.anthropic.tools.browser(spec)]` | -| `playwright: true` | `tools: [cua.tools.playwright()]` | +| `mode: "computer"` | `tools: loop.toolsets.computer()` or an explicit provider-native list | +| `mode: "browser"` | `tools: loop.toolsets.browser()` or an explicit list | +| `mode: "hybrid"` | compose the desired provider and Loop tools explicitly | +| `nativeTool: spec` | `tools: [loop.providers.anthropic.tools.browser(spec)]` | +| `playwright: true` | `tools: [loop.tools.playwright()]` | | `activeToolNames` | pass the exact current list and change it with `setTools()` | ## Documentation requirements @@ -597,9 +597,9 @@ The implementation updates: - `docs/architecture.md` with the tool-spec composition and provider-adapter ownership boundaries - package READMEs with exact constructor examples and no legacy mode terminology - API documentation with the definitions of tool, action, and toolset -- user-facing examples for native-only, provider-native plus CUA, Playwright-only, browser-act-only, empty, batch, and dynamic-loading configurations +- user-facing examples for native-only, provider-native plus Loop, Playwright-only, browser-act-only, empty, batch, and dynamic-loading configurations -Every provider tool surface must expose the first-party source it mirrors. CUA-authored additions must be described as CUA capabilities rather than provider defaults. +Every provider tool surface must expose the first-party source it mirrors. Loop-authored additions must be described as Loop capabilities rather than provider defaults. ## Implemented design resolutions @@ -609,27 +609,27 @@ Every provider tool surface must expose the first-party source it mirrors. CUA-a 4. **Batch overlap:** batches are mechanical; `browser_act` remains semantic; browser batches share ref state without a workflow DSL. 5. **Dynamic loading:** `setTools()` uses pi 0.83.0 additive markers only for final, cache-preserving in-tool additions; other changes are eager. 6. **Shared resources:** one resource pool survives tool/model changes and owns the translator and lazy CDP executor. -7. **Provider exports:** the native OpenAI, Anthropic, and Google surfaces are namespaced, cite first-party sources, and are tested against their declared contracts. Meta, xAI, and Moonshot use CUA-authored browser tools; the CLI explicitly appends `browser_act` to the Meta and xAI catalogs. Moonshot is excluded: its API accepts the complex `browser_wait_for` schema but rejects a request carrying `browser_act`'s much larger one, so the catalog gates oversized schemas separately from merely-complex ones. +7. **Provider exports:** the native OpenAI, Anthropic, and Google surfaces are namespaced, cite first-party sources, and are tested against their declared contracts. Meta, xAI, and Moonshot use Loop-authored browser tools; the CLI explicitly appends `browser_act` to the Meta and xAI catalogs. Moonshot is excluded: its API accepts the complex `browser_wait_for` schema but rejects a request carrying `browser_act`'s much larger one, so the catalog gates oversized schemas separately from merely-complex ones. ## Decisions recorded - `tools: []` is valid. -- CUA does not generate a default system prompt. +- Loop does not generate a default system prompt. - `computer_use_extra` is removed with no implicit replacement. -- There is one current public tool list; no CUA-facing `activeToolNames` layer. -- First-party provider-native tools are namespaced separately from CUA-authored tools. +- There is one current public tool list; no Loop-facing `activeToolNames` layer. +- First-party provider-native tools are namespaced separately from Loop-authored tools. - Tool factories and toolsets are discoverable under a namespace, not exported as many global functions. -- `browser_act` remains outside `cua.toolsets.browser()`; applications may opt - into it explicitly, and the CLI does so for structured CUA-browser catalogs. +- `browser_act` remains outside `loop.toolsets.browser()`; applications may opt + into it explicitly, and the CLI does so for structured Loop-browser catalogs. - Naming, payload-transform composition, result formatting, and batch overlap must be resolved before code is written. ## Acceptance criteria - Both constructors have one required tool-selection source of truth and accept `tools: []`. - The current tool-related constructor options, active-tool option, and mode methods are removed. -- `computer_use_extra` and CUA-generated default system prompts are removed. -- CUA-authored and first-party provider-native tools are exposed through distinct, discoverable namespaces. -- Exact native-browser-only, provider-native-plus-CUA, Playwright-only, browser-act-only, and empty configurations are tested. +- `computer_use_extra` and Loop-generated default system prompts are removed. +- Loop-authored and first-party provider-native tools are exposed through distinct, discoverable namespaces. +- Exact native-browser-only, provider-native-plus-Loop, Playwright-only, browser-act-only, and empty configurations are tested. - No undeclared helper tool is installed. - `computer_batch` exposes explicit action control, and a browser batch design is resolved and tested. - Mid-conversation additive tool loading uses provider-native deferred loading where supported and preserves the prompt cache. diff --git a/docs/architecture.md b/docs/architecture.md index 0df6d2ff..4fc2604e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ # Architecture -This document explains how `cua` is wired together for contributors and +This document explains how `loop` is wired together for contributors and integrators. ## Product principles @@ -10,74 +10,81 @@ provider payload quirks, coordinate conversion, action execution, and action feedback. They do **not** choose an agent's tools or system prompt. Callers own both explicitly and may use pi's orchestration primitives directly. -## Package boundaries - -- `@onkernel/cua-ai` owns the model catalog, stable tool identities, tool - factories/toolsets, provider declarations, compatibility validation, headers, - payload transforms, and incoming native-call normalization. Catalog - compilation is declaration-only and deterministic; the package has no - `AgentTool` or materialization types and no `pi-agent-core` dependency. -- `@onkernel/cua-agent` is provider-neutral runtime glue around - `pi-agent-core`. It defines `CuaAgentTool`, materializes catalog specs - exactly once per shared resource pool against a Kernel browser, owns shared - execution resources, and applies catalog plans supplied as data. -- `@onkernel/cua-pi-extension` contributes these tools to a pi session that pi - itself owns. It is the one consumer that uses neither `attach()` nor the - harness: pi owns the model collection and the agent loop, so the extension - takes the two pieces that are not pi-shaped — the catalog compiler and - `CuaExecutionResources` — and applies headers and payload transforms through - pi's own `before_provider_headers` and `before_provider_request` hooks. +## Module boundaries + +`@onkernel/loop` is one package with two entry points and three source trees: + +- `.` (`src/core/`) is the framework-neutral core: canonical actions, the tool + declarations namespace, the catalog compiler, the tool menu, the tool manager, + and Kernel-browser execution (translator, CDP executor, execution resources). + Catalog compilation is declaration-only and deterministic. Its coupling to pi + is type-level — `Api`, `Model`, `Tool`, `AgentTool` — except for the model + resolution and provider modules it still reaches into, which the next split + moves behind an interface. +- `./pi` (`src/pi/`) is the pi binding: `attach()`/`compile()`, model + resolution, transport derivation, the provider adapters, provider retry, and + header composition. +- `src/pi-extension/` contributes these tools to a pi session that pi itself + owns. It is the one consumer that uses neither `attach()` nor the harness: pi + owns the model collection and the agent loop, so the extension takes the two + pieces that are not pi-shaped — the catalog compiler and + `LoopExecutionResources` — and applies headers and payload transforms through + pi's own `before_provider_headers` and `before_provider_request` hooks. It + imports the rest of the package by name (`@onkernel/loop`, + `@onkernel/loop/pi`) rather than by relative path, because pi loads the + extension as TypeScript through jiti and jiti's pi-ai alias cannot follow the + deep `@earendil-works/pi-ai/api/*` imports the provider adapters make. - `@onkernel/ptywright` is development-only PTY/TUI test infrastructure. It has no in-repo consumer since the CLI was retired; its own tests are what exercise it. -The invariant is that `packages/agent/src` contains no provider-name branches. -Adding provider behavior means adding data and transforms in `cua-ai`, not a -conditional in `cua-agent`. +The invariant is that execution contains no provider-name branches. Adding +provider behavior means adding data and transforms under `src/pi/providers/`, +not a conditional in the translator. ```mermaid flowchart LR - ai["@onkernel/cua-ai"] - agent["@onkernel/cua-agent"] - ext["@onkernel/cua-pi-extension"] + core["@onkernel/loop (src/core)"] + pibind["@onkernel/loop/pi (src/pi)"] + ext["src/pi-extension"] pi["pi-agent-core / pi-ai / pi-coding-agent"] sdk["@onkernel/sdk"] - ai --> agent - agent --> ext - ai --> ext - pi --> agent + core --> pibind + core --> ext + pibind --> ext + pi --> pibind pi --> ext - sdk --> agent + sdk --> core sdk --> ext ``` ## Explicit tool catalog -`cua-ai` exposes one frozen namespace: +The core exposes one frozen namespace: ```ts -import { cua } from "@onkernel/cua-ai"; +import { loop } from "@onkernel/loop"; const tools = [ - cua.tools.browser.snapshot(), - cua.tools.browser.click(), - cua.tools.computer.screenshot(), + loop.tools.browser.snapshot(), + loop.tools.browser.click(), + loop.tools.computer.screenshot(), ]; ``` The main groups are: -- `cua.tools.browser.*`: CDP/page tools, using element refs and viewport pixels. -- `cua.tools.computer.*`: Kernel OS input/read tools, using pixel coordinates by +- `loop.tools.browser.*`: CDP/page tools, using element refs and viewport pixels. +- `loop.tools.computer.*`: Kernel OS input/read tools, using pixel coordinates by default. -- `cua.tools.playwright()`: a Playwright code execution tool. -- `cua.toolsets.browser()`, `computer()`, and `mixed()`: ordinary convenience - arrays of CUA-authored tools. -- `cua.providers.*`: only provider-native tools and predefined toolsets backed +- `loop.tools.playwright()`: a Playwright code execution tool. +- `loop.toolsets.browser()`, `computer()`, and `mixed()`: ordinary convenience + arrays of Loop-authored tools. +- `loop.providers.*`: only provider-native tools and predefined toolsets backed by linked first-party documentation. Each provider namespace exposes its `source` (or versioned `sources`), and every returned spec carries that URL. -Each CUA-owned tool has a stable identity independent of its caller-visible +Each Loop-owned tool has a stable identity independent of its caller-visible name. Compilation preserves requested order and derives provider-safe names, schema fingerprints, coordinate contracts, loading eligibility, headers, payload transforms, and native input mappings. Duplicate identities, @@ -108,7 +115,7 @@ combinations fail without partial mutation. ## Shared execution resources -A single `CuaExecutionResources` pool is created per agent/harness and survives +A single `LoopExecutionResources` pool is created per agent/harness and survives catalog and model changes. It owns: - the Kernel client and browser handle; @@ -123,11 +130,11 @@ per spec object. ## Action planes and result feedback -Canonical actions live under `packages/ai/src/actions/`: +Canonical actions live under `packages/loop/src/core/actions/`: - **Computer actions** use Kernel's `browsers.computer` API and OS screenshot coordinates. -- **Browser actions** use `packages/agent/src/translator/browser.ts` over the +- **Browser actions** use `packages/loop/src/core/translator/browser.ts` over the browser's raw CDP websocket. Element refs are snapshot-scoped and stale refs fail with a request to snapshot again. @@ -160,13 +167,13 @@ catalog: - Anthropic native browser/computer declarations replace only their own placeholders and merge required beta headers with caller headers. - OpenAI streams through pi's builtin Responses transport and its automatic - prompt caching by default; a CUA-owned adapter handles OpenAI's native + prompt caching by default; a Loop-owned adapter handles OpenAI's native computer tool and tool-search namespace round-trips. - Anthropic's native browser tool falls back to an equivalent function-tool declaration when the active credential cannot access `browser_20260701`; the selected tool identity, name, schema, and executor remain unchanged. - Google's current predefined browser toolset serializes one `computer_use` - declaration plus exact exclusions through the CUA-owned Interactions API + declaration plus exact exclusions through the Loop-owned Interactions API adapter. Excluded calls fail with a named catalog error instead of reaching generic tool dispatch. - Meta, xAI, and Moonshot disable parallel tool calls when the selected catalog @@ -176,27 +183,27 @@ catalog: The transport a model streams through is a function of **(model, selected tools)**, derived at catalog compilation — never stamped on the model ahead of -time and never branched on a provider name. A `CuaProviderBinding` may declare -`requiresApi`: the api id its provider-native tool needs. `compileCuaToolCatalog` +time and never branched on a provider name. A `LoopProviderBinding` may declare +`requiresApi`: the api id its provider-native tool needs. `compileLoopToolCatalog` reads `requiresApi` off the selected bindings after normalizing the requested catalog and returns a `catalog.model` carrying that api; selecting tools whose bindings require different transports fails to compile with a named catalog error. A model resolved with no such tool selected keeps its ordinary registry api. -This is why an OpenAI model selected with only CUA browser tools streams +This is why an OpenAI model selected with only Loop browser tools streams through pi's builtin `openai-responses` transport, but the same model selected -with `cua.providers.openai.tools.computer()` compiles to the CUA-owned -`openai-cua-computer` api — and symmetrically for Google's -`google-cua-interactions` Interactions API versus pi's builtin Google +with `loop.providers.openai.tools.computer()` compiles to the Loop-owned +`openai-computer-use` api — and symmetrically for Google's +`google-interactions` Interactions API versus pi's builtin Google transport. ### The tool menu -`cuaToolMenu(model, selected)` in `packages/ai/src/menu.ts` returns every tool -CUA can offer for a model, each marked available or not. It decides availability +`loopToolMenu(model, selected)` in `packages/loop/src/core/menu.ts` returns every tool +Loop can offer for a model, each marked available or not. It decides availability by compiling the candidate catalog rather than by restating the compiler's -rules, so the menu cannot drift from what `compileCuaToolCatalog` accepts: an +rules, so the menu cannot drift from what `compileLoopToolCatalog` accepts: an entry is available exactly when selecting it compiles. Compilation is pure and declaration-only, so probing it per entry is cheap and side-effect-free. @@ -220,13 +227,13 @@ serialization, provider fields, then the caller's `onPayload` hook. ## Extension composition -`packages/pi-extension/src/index.ts` is the composition root for pi sessions. pi +`packages/loop/src/pi-extension/index.ts` is the composition root for pi sessions. pi owns the agent loop, session, UI, and model selection; the extension contributes only what Kernel owns: 1. registers every selectable tool as a pi tool, and keeps the model-facing names identical to what the library produces; -2. resolves a selection from `--cua-tools` (or a persisted command selection), +2. resolves a selection from `--browser-tools` (or a persisted command selection), and validates it by compiling for the active model, so an incompatible tool deactivates with the compiler's own reason instead of failing at request time; 3. re-validates on `model_select` and `before_agent_start`, restoring a @@ -257,7 +264,7 @@ user prompt -> caller onPayload -> provider stream -> incoming native/function call normalization - -> shared CuaExecutionResources + -> shared LoopExecutionResources -> Kernel computer API or raw-CDP BrowserExecutor -> policy-specific action result -> transcript + TUI/stdout/JSONL @@ -265,14 +272,14 @@ user prompt ## Validation and test ownership -- `packages/ai/test/tool-catalog.test.ts`: identities, collisions, provider +- `packages/loop/test/tool-catalog.test.ts`: identities, collisions, provider composition, compatibility, declarations, and coordinate contracts. -- `packages/agent/test/resources.test.ts`: action feedback and batch boundaries. -- `packages/agent/test/attach.test.ts` and `attach-session.test.ts`: compiled +- `packages/loop/test/resources.test.ts`: action feedback and batch boundaries. +- `packages/loop/test/attach.test.ts` and `attach-session.test.ts`: compiled pairs, applying one to a running harness, and the behaviors `activate()` installs. -- `packages/agent/test/translator-browser.test.ts`: browser behavior and ref +- `packages/loop/test/translator-browser.test.ts`: browser behavior and ref lifecycle. -- `packages/pi-extension/test/`: selection and availability, provider stream +- `packages/loop/test/`: selection and availability, provider stream ownership, browser lifecycle, and an end-to-end run against real `pi` in print and RPC modes. diff --git a/docs/npm-releases.md b/docs/npm-releases.md index 52e0ab5b..562761c1 100644 --- a/docs/npm-releases.md +++ b/docs/npm-releases.md @@ -1,64 +1,45 @@ # npm releases -`@onkernel/cua-ai` and `@onkernel/cua-agent` publish from package-specific tags: +`@onkernel/loop` publishes from a package tag: -- `cua-ai/v0.1.0` runs `.github/workflows/release-cua-ai.yml` -- `cua-agent/v0.1.0` runs `.github/workflows/release-cua-agent.yml` +- `loop/v0.11.0` runs `.github/workflows/release-loop.yml` -`@onkernel/cua-pi-extension` has no workflow yet. It merges into the renamed -single package, and a first publish under a new name is manual either way — see -below. - -The tag version must match the target package's `package.json` version, and the -tagged commit must be contained in `main`. +The tag version must match `packages/loop/package.json`, and the tagged commit +must be contained in `main`. ## Trusted publishing setup -Configure each package on npm with a GitHub Actions trusted publisher: +Configure the package on npm with a GitHub Actions trusted publisher: | package | organization | repository | workflow filename | environment | | --- | --- | --- | --- | --- | -| `@onkernel/cua-ai` | `kernel` | `cua` | `release-cua-ai.yml` | leave blank | -| `@onkernel/cua-agent` | `kernel` | `cua` | `release-cua-agent.yml` | leave blank | - +| `@onkernel/loop` | `kernel` | `cua` | `release-loop.yml` | leave blank | The same configuration can be created from the npm CLI: ```sh npm install -g npm@^11.17.0 -npm trust github @onkernel/cua-ai --repo kernel/cua --file release-cua-ai.yml --allow-publish -npm trust github @onkernel/cua-agent --repo kernel/cua --file release-cua-agent.yml --allow-publish +npm trust github @onkernel/loop --repo kernel/cua --file release-loop.yml --allow-publish ``` -npm requires packages to exist before a trusted publisher can be configured. If -the package has not been published yet, either publish the first version manually -and use trusted publishing for later versions, or publish a bootstrap version -first, configure trusted publishing, then release `0.1.0` from tags. +npm requires packages to exist before a trusted publisher can be configured, so +`@onkernel/loop`'s first version is published manually — see below. -## Releasing 0.1.0 - -Publish `@onkernel/cua-ai` first because `@onkernel/cua-agent` depends on it: +## Releasing from a tag ```sh git checkout main git pull --ff-only -git tag cua-ai/v0.1.0 -git push origin cua-ai/v0.1.0 -``` - -After `@onkernel/cua-ai@0.1.0` is available on npm: - -```sh -git tag cua-agent/v0.1.0 -git push origin cua-agent/v0.1.0 +git tag loop/v0.11.0 +git push origin loop/v0.11.0 ``` ## First publish of a new package name npm requires a package to exist before a trusted publisher can be configured for it, so the first release of any new name is a manual publish from a local -checkout. This applies to `@onkernel/cua-pi-extension` today, and will apply to -the renamed single package. +checkout. This applies to `@onkernel/loop`, whose predecessors published under +the retired `@onkernel/cua-ai` and `@onkernel/cua-agent` names. Two related constraints, because a trusted publisher is bound to a *(repository, workflow filename)* pair: diff --git a/package-lock.json b/package-lock.json index 4a7f3b02..9d4a9b83 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8,10 +8,8 @@ "name": "cua", "version": "0.1.0", "workspaces": [ - "packages/ai", - "packages/agent", - "packages/ptywright", - "packages/pi-extension" + "packages/loop", + "packages/ptywright" ], "devDependencies": { "@types/node": "22.18.4", @@ -2244,16 +2242,8 @@ } } }, - "node_modules/@onkernel/cua-agent": { - "resolved": "packages/agent", - "link": true - }, - "node_modules/@onkernel/cua-ai": { - "resolved": "packages/ai", - "link": true - }, - "node_modules/@onkernel/cua-pi-extension": { - "resolved": "packages/pi-extension", + "node_modules/@onkernel/loop": { + "resolved": "packages/loop", "link": true }, "node_modules/@onkernel/ptywright": { @@ -3997,63 +3987,32 @@ "zod": "^3.25.28 || ^4" } }, - "packages/agent": { - "name": "@onkernel/cua-agent", - "version": "0.10.0", + "packages/loop": { + "name": "@onkernel/loop", + "version": "0.11.0", "license": "MIT", "dependencies": { "@earendil-works/pi-agent-core": "0.83.0", "@earendil-works/pi-ai": "0.83.0", - "@onkernel/cua-ai": "0.10.0", "@onkernel/sdk": "0.49.0", + "openai": "^6.26.0", "sharp": "^0.35.3" }, "devDependencies": { - "tsdown": "^0.22.2", - "vitest": "^3.2.4" - }, - "engines": { - "node": ">=22.19.0" - } - }, - "packages/ai": { - "name": "@onkernel/cua-ai", - "version": "0.10.0", - "license": "MIT", - "dependencies": { - "@earendil-works/pi-ai": "0.83.0", - "openai": "^6.26.0" - }, - "devDependencies": { - "tsdown": "^0.22.2", - "vitest": "^3.2.4" - }, - "engines": { - "node": ">=22.19.0" - } - }, - "packages/pi-extension": { - "name": "@onkernel/cua-pi-extension", - "version": "0.10.0", - "license": "MIT", - "dependencies": { - "@onkernel/cua-agent": "0.10.0", - "@onkernel/cua-ai": "0.10.0", - "@onkernel/sdk": "0.49.0" - }, - "devDependencies": { - "@earendil-works/pi-agent-core": "0.83.0", - "@earendil-works/pi-ai": "0.83.0", "@earendil-works/pi-coding-agent": "0.83.0", + "tsdown": "^0.22.2", "vitest": "^3.2.4" }, "engines": { "node": ">=22.19.0" }, "peerDependencies": { - "@earendil-works/pi-agent-core": "*", - "@earendil-works/pi-ai": "*", "@earendil-works/pi-coding-agent": "*" + }, + "peerDependenciesMeta": { + "@earendil-works/pi-coding-agent": { + "optional": true + } } }, "packages/ptywright": { diff --git a/package.json b/package.json index a650d141..7cdb1e71 100644 --- a/package.json +++ b/package.json @@ -1,19 +1,17 @@ { "name": "cua", "version": "0.1.0", - "description": "Kernel-cloud-browser computer-use: CLI and SDKs", + "description": "Kernel-cloud-browser computer-use: tools, catalog compilation, and pi bindings", "type": "module", "private": true, "workspaces": [ - "packages/ai", - "packages/agent", - "packages/ptywright", - "packages/pi-extension" + "packages/loop", + "packages/ptywright" ], "scripts": { - "build": "npm run build --workspace @onkernel/cua-ai && npm run build --workspace @onkernel/cua-agent && tsc -b && npm run build:native --workspace @onkernel/ptywright --if-present", + "build": "npm run build --workspace @onkernel/loop && tsc -b && npm run build:native --workspace @onkernel/ptywright --if-present", "dev": "tsc -b --watch", - "typecheck": "npm run build --workspace @onkernel/cua-ai && npm run build --workspace @onkernel/cua-agent && tsc -b", + "typecheck": "tsc -b", "clean": "tsc -b --clean && npm run clean:native --workspace @onkernel/ptywright --if-present" }, "engines": { diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md deleted file mode 100644 index ad16c599..00000000 --- a/packages/agent/CHANGELOG.md +++ /dev/null @@ -1,299 +0,0 @@ -# Changelog - -## Unreleased - -- `browser_act` no longer spends a plan's whole deadline waiting for the effect of - a step whose action failed. An unresolvable ref throws immediately, but the - step's `expect` was still awaited afterwards, so a model that invented a ref - burned a full global timeout per attempt instead of being told to snapshot - first. A step whose action never dispatched now skips its own expectation and - stops the plan. Uncertain *delivery* still waits, because a lost acknowledgement - may mean the input landed and the expectation is how that is discovered. - -Breaking: `CuaAgent` and `CuaAgentHarness` are removed. cua-agent hands back -plain pi objects; the caller constructs the agent. - -- Add `attach({ browser, client })`, returning a handle that compiles - (model, tools) pairs into plain pi objects: `model` carrying the transport its - tools derive, `tools` / `agentTools` materialized against the handle's browser - pool, `models` adding provider retry, required headers, the catalog's payload - transforms and the tool-result image bound, `activate(harness)` for the - behaviors that are pi event handlers rather than constructor options, and - `apply(harness)` to swap a running harness onto a new pair. The handle owns - what actually persists — the Kernel client and browser, the translator, the - raw-CDP executor, ref and frame state — so a spec materializes once across - repeat compiles. -- `getTools()`, `setTools()`, `setModel()`, and `setModelAndTools()` are gone - with the classes. A change compiles a new pair and applies it, so the current - selection belongs to the caller and there is no second copy of it to drift. - `compile()` throws before anything reaches pi, and `apply()` restores the - previous pair if pi rejects the new one, so the atomicity those methods - provided is preserved. `apply()` sets the model only when the derived - transport actually moved, so a tools-only change records no model change. -- `CuaToolManager` is now immutable: one compiled pair per instance, with - `prepareTools`/`prepareModel`/`prepareModelAndTools`/`commit`/`getTools` and - the async-local execution scope removed. Removing the execution scope also - removes cache-preserving deferred tool addition: a tool that added tools mid - execution used to have those names recorded on its result as - `addedToolNames`, letting pi extend an OpenAI request without invalidating the - prompt-cache prefix. Nothing produced them outside that mutation path. The - transport still consumes `addedToolNames` on a transcript that carries them. -- The tool-result image bound, payload transforms, and required headers now - follow whichever pair is active rather than the one a harness was built with. - pi fixes `models` at construction while those are per-catalog, so one - collection per handle is what makes a swap possible at all. -- A model ref absent from a supplied `Models` collection falls back to the - registry, and an id the registry lacks is synthesized. -- The model streamed for a Google model depends on which tools it was compiled - with: selecting Google's native browser toolset compiles to the CUA-owned - Interactions API, while a Google model selected with only CDP browser tools - streams through pi's builtin Google transport. -- `responseThreading` (`CuaAttachOptions`) no longer affects OpenAI models: - OpenAI streams through pi-ai's builtin Responses transport and its automatic - prompt caching regardless of this flag. The option still governs Google's - `previous_response_id`-style continuation. -- Exempt OpenAI's native computer tool from the tool-result image replay limit. - Its `computer_call_output` items must each carry a screenshot, and stateless - replay no longer leaves them in provider-stored state. - -Breaking: Tzafon and Yutori support is removed. - -- Compiling a Tzafon or Yutori model ref now fails to resolve the model, and - `cua.providers.tzafon` / `cua.providers.yutori` no longer exist. -- Remove `CuaExecutionResources.viewport`. It only fed the removed catalog - viewport option; the same value is still on `resources.browser.viewport`. - -## 0.10.0 - 2026-08-04 - -Breaking: upgrade `@earendil-works/pi-agent-core` and `@earendil-works/pi-ai` -to 0.83.0 and adopt pi's context-first harness API. - -- `CuaAgentHarness` and `CuaAgentHarnessOptions` now take the tool context as - their first type parameter — `CuaAgentHarness` — mirroring pi's `AgentHarness` generic order. The - supplied `toolContext` is forwarded to pi untouched, and every executable - harness tool receives the exact object on each call. -- Executable harness tools are pi `AgentHarnessTool`s via the new - `CuaHarnessTool` union (a CUA spec or an `AgentHarnessTool`). - `CuaAgent` stays on the ordinary pi `AgentTool` (`CuaAgentTool`); the two - tool APIs are no longer conflated. -- Remove `CuaAgentHarnessOptions.env` and `CuaAgentHarness.env`. Execution - environments now travel through the tool context (for example - `toolContext: { env: new NodeExecutionEnv({ cwd }) }` for pi's - read/bash/edit/write tools). No alias is preserved. -- Remove `CuaSystemPromptCallback`; `systemPrompt` is pi's - `AgentHarnessSystemPrompt` through the harness options. -- Keep `streamFn` optional on `CuaAgentOptions` (CUA supplies its default - stream) even though pi 0.83.0 makes `AgentOptions.streamFn` required. -- Published declarations target pi's TypeBox 1.3 as-is; a downstream compile - test with `skipLibCheck: false` guards the packaged types. -- Preserve explicit Tzafon screenshot results in model context even when they - fall outside `toolResultImageReplayLimit`, because its native continuation - protocol requires those images. Other tool-result images remain bounded. - -## 0.9.0 - 2026-08-03 - -- Add OpenRouter Kimi K3 support through `@onkernel/cua-ai` 0.9.0, - including the browser-primitives-only example catalog used by the provider - matrix. -- Resolve `CuaAgentHarness` string model references against its supplied - `Models` collection during construction and `setModel()`, while preserving - the curated CUA model gate and fallback support for CUA model overrides. -- Keep `CuaAgent` aligned with pi's low-level `Agent`: callers can pass a - concrete OpenRouter model and inject `models.streamSimple` without adding a - `Models` dependency to the agent API. - -## 0.8.0 - 2026-07-31 - -Breaking: `CuaAgent` and `CuaAgentHarness` now require one exact `tools` list and -use composition instead of inheriting from pi's `Agent`/`AgentHarness`. - -- Add `getTools()` and atomic `setTools()`. Model changes recompile and - revalidate the full requested catalog. Empty catalogs are valid; - no tools or system-prompt text are inferred or appended. Catalog changes from - inside a tool require sequential execution, including model changes. -- Remove `mode`, `nativeTool`, `extraTools`, `playwright`, `setMode()` / - `getMode()`, and implicit `computer_use_extra` behavior. -- Add one shared `CuaExecutionResources` pool per agent/harness. Catalog and - model changes preserve the canonical translator, lazy raw-CDP browser - executor, refs, tabs, screenshots, and Playwright capability. -- Define and export `CuaAgentTool` here (moved out of cua-ai, which now - compiles declaration-only catalogs). cua-agent owns all `AgentTool` - materialization — each CUA spec is materialized exactly once per shared - execution-resource pool — and owns implementation identity for - cache-preserving deferred-tool decisions: a reused `execute` function keeps - its identity across wrappers, a new `execute` or freshly created spec object - is a conservative replacement, and the same objects stay stable across model - recompilation. -- Integrate pi 0.80.10 dynamic tool loading. Eligible additions made from inside - a running tool emit `addedToolNames`; outside-tool additions and all - provider-native changes are eager. Schema/executor replacements are treated - as real changes, not name-only no-ops. -- Refactor atomic tools to operation-specific argument objects while preserving - the existing `browser_act` schema. Export `formatBrowserActResult()` so direct - application surfaces can render the same bounded plan feedback as agents. -- Add mechanical `computer_batch` and `browser_batch` execution. Computer writes - coalesce across write-only runs and flush around reads; browser actions run - sequentially against shared ref state. Failure details include the failed - action index, completed reads, and skipped count. -- Return screenshots only for explicit screenshot or zoom actions. Ordinary - writes return status text, semantic tools return structured feedback, and - failed batches replace images from earlier explicit screenshot steps with - textual markers. -- Native multi-action turns stop after the first failed tool call. Every - remaining call in that assistant turn receives the configured error result - instead of executing against stale browser state. -- Update shared examples to use the same browser-oriented provider catalogs as - the CLI: explicit `browser_act` plans where the provider accepts the schema, - browser primitives alone for Moonshot, and Anthropic native-browser selection - with model fallback. -- Security: require `sharp` `^0.35.3` (was `^0.34.5`) to pick up the libvips - fixes for GHSA-f88m-g3jw-g9cj (CVE-2026-33327, CVE-2026-33328, CVE-2026-35590, - CVE-2026-35591). `sharp` decodes cloud-browser screenshots inside the - translator's `zoom()`, so this is the one advisory in this release that - touched attacker-influenced bytes. The APIs this package uses are unchanged by - sharp 0.35, and no source changes were needed. Two packaging notes for - installers: sharp 0.35 no longer ships an `install` lifecycle script, and it - no longer falls back to building from source — installing with - `--omit=optional`, or on a platform with no prebuilt `@img/sharp-*` binary, - now fails at import instead of silently compiling. sharp 0.35 requires Node - `>=20.9.0`, well below this package's floor. -- Declare `engines.node` `>=22.19.0`. This is not a new requirement: every - `@earendil-works/pi-*` dependency already declares the same floor, so it was - previously enforced only transitively and never stated on this package. - -## 0.7.0 - 2026-07-17 - -- `CuaAgent` and `CuaAgentHarness` support Moonshot Kimi K3 - (`moonshotai:kimi-k3`) via `@onkernel/cua-ai` 0.7.0, resolving auth from - `MOONSHOT_API_KEY`. Kimi's fractional coordinates are scaled to viewport - pixels by the existing translator. -- Bumped `@earendil-works/pi-ai` and `@earendil-works/pi-agent-core` to - 0.80.10; the wrapped `Models` collections forward the new - `checkAuth`/`getAvailable`/`login`/`logout` methods. - -## 0.6.0 - 2026-07-10 - -Adds explicit request-recovery and context-management policies while keeping -provider retries and exact-empty recovery disabled by default. - -- `retry` adds opt-in transient provider-request retries to `CuaAgent` and - `CuaAgentHarness`, with configurable attempt and backoff limits. Failed - partial streams are buffered and discarded before a clean retry is exposed. -- `toolResultImageReplayLimit` limits each model request to the newest four - tool-result images by default. It operates on a request-time projection and - leaves agent state and persisted sessions unchanged. Harness context hooks - settle before this limit is applied at the `Models` boundary. -- `responseThreading` replaces the process-wide environment switch with a - constructor option for OpenAI and Tzafon `previous_response_id` chaining. -- `emptyResponseRecovery` optionally follows a successful exact-empty response - with a bounded, caller-supplied pi `followUp()` message. Omitting it preserves - pi's normal completion behavior. -- Updated `@onkernel/cua-ai` to 0.6.0. - -## 0.5.0 - 2026-07-09 - -Adds the browser action plane and runtime mode switching. Breaking: the -`computerUseExtra` option is removed — the `computer_use_extra` navigation -helper is always registered. - -- New `BrowserExecutor`: drives the browser plane over CDP. Accessibility - snapshots with element refs (`[e12]`), node states - (checked/expanded/disabled/value/…), and cursor:pointer clickable hints - for elements with no interactive ARIA role; iframe and OOPIF stitching - with per-frame session-aware refs; StaticText dedupe and wrapper - collapsing; an unchanged-snapshot short-circuit; lexical `find`, `fill`, - CDP navigation and tab management; and a JavaScript dialog guard. Refs invalidate on real - navigations (`Page.frameNavigated`), self-heal via (role, name, nth) when - the page changes but the element is still unambiguous, and the ref table is - bounded (per-target cap, generation sweeps). `exportRefState()` / - `importRefState()` persist refs across processes against the same browser. -- `CuaAgent` and `CuaAgentHarness` accept `mode` (`"computer"` | `"browser"` - | `"hybrid"`) and `nativeTool`, and support runtime plane switching via - `setMode()` / `getMode()`. Mode switches preserve the requested activation - state of surviving tools and keep the translator — CDP connection, tabs, - and element refs — alive; the translator is only rebuilt when a model - switch changes the provider's coordinate system or screenshot transform. - Both switches roll back cleanly on failure. -- Post-action grounding captures and the navigation helper are mode-aware: - browser mode grounds on the viewport and routes navigation through CDP - (browser and hybrid modes both route `computer_use_extra` navigation over - the browser plane so refs invalidate correctly). -- Updated `@onkernel/cua-ai` to 0.5.0. - -## 0.4.0 - 2026-07-07 - -Breaking: follows pi-agent-core 0.80's `Models`-based harness. - -- `CuaAgentHarness` accepts an optional `models` (a pi `Models` collection) - and defaults to `cuaModels()` from `@onkernel/cua-ai`. The - `getApiKeyAndHeaders` option is gone — pi-agent-core 0.80 resolves auth - through provider auth on the collection; pass a custom `models` to override - resolution (e.g. in tests). -- `CuaAgent`'s default stream path is `cuaModels().streamSimple` instead of - pi-ai's removed global `streamSimple`. Custom `streamFn` options work - unchanged. -- Updated `@earendil-works/pi-agent-core` and `@earendil-works/pi-ai` to - 0.80.3 and `@onkernel/cua-ai` to 0.4.0. - -## 0.3.5 - 2026-06-24 - -- Update the `@onkernel/cua-ai` dependency to 0.3.2, adding computer-use - support for the `gemini-3.5-flash` Google model. - -## 0.3.4 - 2026-06-23 - -- Add an opt-in `playwright` option to `CuaAgent` and `CuaAgentHarness` that - exposes a `playwright_execute` tool, running Playwright/TypeScript against - the live browser session via the Kernel SDK. Results, stdout, and stderr - come back as tool content; SDK-reported failures surface as content rather - than throwing. Adds the `PlaywrightDetails` export. - -## 0.3.3 - 2026-06-12 - -- The action translator now consumes the canonical `CuaAction` union with an - exhaustive switch. Malformed action shapes fail loudly instead of silently - coercing (previously e.g. a click at 0,0); the documented mouse-button - coercion to `"left"` is unchanged. -- `prepareNextTurn` no longer rebuilds the turn context on every turn: it - keeps stock pi behavior until a user hook returns an update or a mid-run - model assignment requires a refresh. -- One translator instance per runtime is shared between the executor tools - and the provider screenshot capability. -- The `CuaAgentHarness` README quickstart showcases session-backed turns and - mid-session model switching; `computerUseExtra` is documented with its - rationale. -- Update the `@onkernel/cua-ai` dependency to 0.3.0. - -## 0.3.2 - 2026-06-11 - -- Update the `@onkernel/cua-ai` dependency to 0.2.2. - -## 0.3.1 - 2026-06-11 - -- Update the `@onkernel/cua-ai` dependency to 0.2.1. - -## 0.3.0 - 2026-06-10 - -- Replaces the vendored pi-agent-core snapshot with the released `@earendil-works/pi-agent-core@0.79.1` dependency. The full pi surface is still re-exported, but it now tracks the published package instead of a frozen fork. -- BREAKING: `harness.agent` is removed. It only existed in the vendored pre-release snapshot and never shipped in any pi-agent-core release; use `getModel()`, `getTools()`, and `getActiveTools()` instead. -- BREAKING: `steer()`, `followUp()`, `nextTurn()`, and `setStreamOptions()` on the harness now return promises and must be awaited. -- BREAKING: the harness `model_select` and `thinking_level_select` events are renamed `model_update` and `thinking_level_update`, and the `steeringMode`/`followUpMode` property accessors became `getSteeringMode()`/`setSteeringMode()`/`getFollowUpMode()`/`setFollowUpMode()` methods. -- BREAKING: `ExecutionEnv` is now `Result`-based. Custom env implementations return `Result` values instead of throwing. -- BREAKING: requires Node.js >= 22.19.0. -- `NodeExecutionEnv` now comes from `@earendil-works/pi-agent-core`'s `/node` subpath; importing it from `@onkernel/cua-agent` keeps working. -- Tool execution follows pi's throw-on-failure contract: failed browser actions throw an error labeled with the action instead of also encoding the failure into tool result content and details. -- Moves the yutori screenshot payload append into `@onkernel/cua-ai`'s payload middleware. -- Built ESM output uses explicit `.js` relative import specifiers so `dist` resolves under plain Node.js. - -## 0.2.0 - 2026-05-13 - -- Adds `CuaAgentHarness`, a provider-aware harness API with session-backed turns, resource and prompt helpers, active tool selection, and model switching. -- Keeps CUA runtime defaults in sync when changing models so provider-specific tools, prompts, and payload middleware update together. -- Improves browser keyboard shortcut translation for Kernel computer actions. - -## 0.1.0 - -- Class-first CUA runtime: `CuaAgent` and `CuaHarness` on top of pi-agent-core. -- Provider-neutral browser tool executors for canonical CUA tool names, backed by Kernel browser actions. -- Includes examples plus unit and live e2e coverage for common provider/model combinations. diff --git a/packages/agent/README.md b/packages/agent/README.md deleted file mode 100644 index 8f94c088..00000000 --- a/packages/agent/README.md +++ /dev/null @@ -1,289 +0,0 @@ -# `@onkernel/cua-agent` - -Kernel-browser execution for explicit [`@onkernel/cua-ai`](../ai) tool catalogs, -built on `@earendil-works/pi-agent-core`. - -## Install - -```bash -npm install @onkernel/cua-agent @onkernel/cua-ai @onkernel/sdk -``` - -Requires Node 22.19 or newer, `KERNEL_API_KEY`, and the selected model provider's -API key. - -## `attach()` - -`attach()` binds a Kernel browser to CUA's execution resources and returns a -handle. `compile()` turns a (model, tools) pair into plain pi objects; you -construct whatever pi agent you want with them. There is no CUA agent class. - -```ts -import Kernel from "@onkernel/sdk"; -import { cua } from "@onkernel/cua-ai"; -import { Agent, attach } from "@onkernel/cua-agent"; - -const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); -const browser = await client.browsers.create({ stealth: true }); -const kb = attach({ client, browser }); - -const { model, agentTools, models } = kb.compile({ - model: "anthropic:claude-opus-5", - tools: cua.toolsets.browser(), -}); - -const agent = new Agent({ - streamFn: (selected, context, options) => models.streamSimple(selected, context, options), - initialState: { - model, - tools: [...agentTools], - systemPrompt: "Inspect and interact with the page using the requested tools.", - }, -}); - -try { - await agent.prompt("Open example.com and report the heading."); -} finally { - await kb.dispose(); - await client.browsers.deleteByID(browser.session_id); -} -``` - -The compiled `model` carries the transport its tools derive: selecting a -provider-native browser or computer surface can change `model.api`, so the pair -has to reach pi together. - -## With pi's `AgentHarness` - -Use pi's harness for session-backed transcripts, skills, prompt templates, -compaction, steering, and follow-ups. `activate()` registers the behaviors CUA -owns that are pi event handlers rather than constructor options, and points the -handle's `models` at this catalog: - -```ts -import { AgentHarness, attach, InMemorySessionRepo } from "@onkernel/cua-agent"; -import { cua } from "@onkernel/cua-ai"; - -const session = await new InMemorySessionRepo().create(); -const kb = attach({ client, browser }); -const compiled = kb.compile({ model: "openai:gpt-5.6-sol", tools: cua.toolsets.browser() }); - -const harness = new AgentHarness({ - session, - model: compiled.model, - models: compiled.models, - tools: [...compiled.tools], - activeToolNames: compiled.tools.map((tool) => tool.name), - systemPrompt: "Use the supplied browser tools.", -}); -compiled.activate(harness); - -await harness.prompt("Find the pricing page."); -``` - -To change the model or the tool list on a running harness, compile the new pair -and apply it: - -```ts -await kb.compile({ model: "google:gemini-3.6-flash", tools: cua.providers.google.toolsets.browser() }).apply(harness); -``` - -`apply()` moves the model and tools together, sets the model only when the -derived transport actually moved, and restores the previous pair if pi rejects -the new one. - -The package re-exports pi-agent-core session, skill, prompt-template, compaction, -and execution-environment primitives used with the harness. - -### Tool context - -Executable harness tools are pi `AgentHarnessTool`s: `execute` receives the -harness's tool context as its last argument. Supply it once as `toolContext` -and pi delivers the exact object (or the result of a zero-argument provider) -to every tool call: - -```ts -import { - AgentHarness, - attach, - NodeExecutionEnv, - createBashTool, - createReadTool, - type ExecutionToolContext, -} from "@onkernel/cua-agent"; - -const compiled = kb.compile({ - model: "openai:gpt-5.6-sol", - tools: [createReadTool(), createBashTool(), ...cua.toolsets.browser()], -}); -const harness = new AgentHarness({ - session, - model: compiled.model, - models: compiled.models, - tools: [...compiled.tools], - activeToolNames: compiled.tools.map((tool) => tool.name), - toolContext: { env: new NodeExecutionEnv({ cwd: process.cwd() }) }, - systemPrompt: "Use the supplied tools.", -}); -compiled.activate(harness); -``` - -Compile for the same context the harness delivers, so a later swap stays -type-compatible. CUA specs and plain pi `AgentTool`s are accepted too — they -simply ignore the context. `compiled.agentTools` is the context-free view for -the low-level `Agent`. - -## Choosing tools - -```ts -import { cua } from "@onkernel/cua-ai"; - -const browser = cua.toolsets.browser(); -const computer = cua.toolsets.computer(); -const mixed = cua.toolsets.mixed(); -// Use a normalized contract when the model emits screen-relative coordinates: -// the schema advertises 0–1000 and execution scales them to viewport pixels. -const normalized = cua.toolsets.computer({ - coordinates: cua.coordinates.normalized([0, 1000]), -}); -const playwright = cua.tools.playwright(); -``` - -Tool factories accept fixed caller-visible names (and toolsets accept a -namespace) without changing stable identity: - -```ts -const tools = [ - cua.tools.browser.snapshot({ name: "page_snapshot" }), - cua.tools.browser.click({ name: "page_click" }), -]; -``` - -### Provider-native tools - -Provider-native declarations compose with ordinary function tools: - -```ts -const tools = [ - cua.providers.anthropic.tools.computer({ - version: "20260701", - enableZoom: true, - }), - cua.tools.browser.snapshot(), -]; -``` - -Other provider groups include OpenAI native computer and Google's current -predefined browser toolset. Every provider surface exposes linked first-party -documentation. xAI uses CUA browser primitives plus -`cua.tools.browser.act()` in the provider-matrix examples. Moonshot uses browser -primitives alone because its API rejects `browser_act`'s larger schema. -Compilation rejects incompatible tool/model combinations before a request. -Anthropic's native browser tool uses an equivalent function-tool transport when -the active credential cannot access `browser_20260701`. - -## Dynamic catalogs - -Changing the model or the tool list compiles a new pair; nothing mutates in -place: - -```ts -const next = kb.compile({ model: nextModel, tools: nextTools }); -await next.apply(harness); -``` - -Duplicate identities, caller-visible name collisions, provider-normalized name -collisions, and incompatible model/tool combinations fail in `compile()`, before -anything reaches pi. `apply()` then moves the model and tools together and -restores the previous pair if pi rejects the new one. - -The handle holds the current selection only in the sense that `models` serves -whichever pair was last activated; the *selection* itself belongs to the caller, -which is what removes the class of bug where a compiled pair and the live one -disagree. - -One shared execution-resource pool survives all catalog/model changes, so -browser refs, tabs, connections, and translator state are not reset. - -## Mechanical batches - -Batch factories require an explicit non-empty allowlist: - -```ts -const tools = [ - cua.tools.computer.batch({ actions: ["click", "keypress", "screenshot"] }), - cua.tools.browser.batch({ actions: ["snapshot", "click", "wait_for", "text"] }), -]; -``` - -Batch inputs are bounded primitive action arrays—not a workflow language. -Computer writes coalesce until a read boundary; browser actions run -sequentially over one shared raw-CDP executor. Results preserve read order. -Failure stops at the first failed action and reports its index plus skipped -count. - -## Action feedback - -Tools return only requested feedback: - -- write actions return concise status text; -- read actions return their requested text or structured data; -- explicit screenshot and zoom actions return images; -- `browser_act` returns causal outcomes and a bounded successor diff; -- failed batches replace images from earlier explicit screenshot steps with - textual markers. - -`toolResultImageReplayLimit` controls how many recent tool-result images remain -in model context (`4` by default, or `false` to disable projection). OpenAI -native computer results are exempt because its protocol requires each -`computer_call_output` to carry a screenshot. - -## Custom tools - -Ordinary pi `AgentTool`s can appear anywhere in the exact list: - -```ts -import { Type } from "@earendil-works/pi-ai"; - -const lookup = { - name: "customer_lookup", - label: "Customer lookup", - description: "Look up a customer by id.", - parameters: Type.Object({ id: Type.String() }), - async execute(_id, { id }) { - return { content: [{ type: "text", text: await lookupCustomer(id) }], details: {} }; - }, -}; - -kb.compile({ model, tools: [lookup, ...cua.toolsets.browser()] }); -``` - -Caller tools receive identity `caller.` through cua-ai's canonical -`callerToolIdentity()` helper and participate in the same collision and -fingerprint rules. `CuaAgentTool` is defined and exported by this package: -cua-ai compiles declaration-only catalogs and never sees executors, while -cua-agent projects caller `AgentTool`s into fresh declarations, joins compiled -entries back by identity, and materializes each CUA spec exactly once per shared -execution-resource pool, so repeat compiles hand pi a stable implementation. - -## Events and state - -The agent is pi's, so its lifecycle, events, and session APIs are pi's too. -What CUA adds on top is `activate()`: failed tool results are marked, a turn's -remaining calls are blocked after one fails, and an empty successful response -can be followed up. It returns a release. - -## Development - -```bash -npm run typecheck --workspace @onkernel/cua-agent -npm test --workspace @onkernel/cua-agent -npm run build --workspace @onkernel/cua-agent -``` - -See [`examples/`](examples) for direct-agent, harness, provider-matrix, and -Anthropic-native smoke tests. - -## License - -MIT. diff --git a/packages/agent/examples/agent-openai-smoke.ts b/packages/agent/examples/agent-openai-smoke.ts deleted file mode 100644 index 4725e4f4..00000000 --- a/packages/agent/examples/agent-openai-smoke.ts +++ /dev/null @@ -1,38 +0,0 @@ -import Kernel from "@onkernel/sdk"; -import { cua, requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai"; -import { CuaAgent } from "../src/index"; -import { logAgentEvent, logAssistant } from "./shared/logging"; -import { SCENARIOS } from "./shared/scenarios"; - -const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.6-sol"; - -async function main(): Promise { - const kernelApiKey = process.env.KERNEL_API_KEY; - if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); - requireCuaEnvApiKeyForModel(modelRef); - const client = new Kernel({ apiKey: kernelApiKey }); - const browser = await client.browsers.create({ stealth: true }); - - try { - const agent = new CuaAgent({ - browser, - client, - // Prefer structured browser refs and semantic reads for the OpenAI smoke, - // and opt into verified dependent plans without changing the base toolset. - tools: [...cua.toolsets.browser(), cua.tools.browser.act()], - initialState: { model: modelRef, systemPrompt: "Use the provided computer and browser tools to interact with the page." }, - }); - - agent.subscribe(logAgentEvent); - - const scenario = SCENARIOS[0]!; - console.log(`running scenario: ${scenario.name} model=${modelRef}`); - await agent.prompt(scenario.prompt); - const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant"); - logAssistant(assistant?.role === "assistant" ? assistant : undefined); - } finally { - await client.browsers.deleteByID(browser.session_id); - } -} - -void main(); diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts deleted file mode 100644 index aa78a66e..00000000 --- a/packages/agent/src/index.ts +++ /dev/null @@ -1,43 +0,0 @@ -export * from "@earendil-works/pi-agent-core"; -export { NodeExecutionEnv } from "@earendil-works/pi-agent-core/node"; -export { cua } from "@onkernel/cua-ai"; -export type { CuaToolSpec } from "@onkernel/cua-ai"; -export type { CuaAgentTool, CuaHarnessTool } from "./tool-manager"; - -export type { KernelBrowser } from "./translator/translator"; -export { InternalComputerTranslator } from "./translator/translator"; -export { CuaExecutionResources } from "./resources"; -export { formatBrowserActResult } from "./browser-result-format"; -export { CdpConnection } from "./translator/cdp"; -export { BrowserExecutor } from "./translator/browser"; -export type { BrowserFindCandidate } from "./translator/browser"; -export type { BrowserRefState } from "./translator/browser-ref-lifecycle"; -export type { - BatchExecutionResult, - BatchReadResult, - BrowserActExpectationEvidence, - BrowserActExpectationStatus, - BrowserActObservedSuccessor, - BrowserActOutcome, - BrowserActResult, - BrowserActStepResult, - BrowserActStopReason, - BrowserActSuccessor, - BrowserActUnavailableSuccessor, - BrowserExpectationEvidence, - BrowserExpectationState, - BrowserObservationDiff, - BrowserObservationDiffEntry, - BrowserWaitForResult, - BrowserWaitReason, -} from "./translator/types"; -export { attach } from "./attach"; -export type { - CuaAttachOptions, - CuaBrowserHandle, - CuaCompiled, - CuaEmptyResponseRecoveryOptions, - CuaModelInput, - ToolResultImageReplayLimit, -} from "./attach"; -export type { CuaRetryOptions } from "./provider-retry"; diff --git a/packages/agent/tsconfig.build.json b/packages/agent/tsconfig.build.json deleted file mode 100644 index a7344f9a..00000000 --- a/packages/agent/tsconfig.build.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "extends": "../../tsconfig.base.json", - "compilerOptions": { - "outDir": "./dist-tsc", - "rootDir": "./src", - "emitDeclarationOnly": true, - "sourceMap": false, - "declarationMap": false - }, - "include": [ - "src/**/*.ts" - ], - "exclude": [ - "node_modules", - "dist", - "**/*.d.ts", - "src/**/*.d.ts" - ], - "references": [ - { - "path": "../ai" - } - ] -} diff --git a/packages/agent/vitest.config.ts b/packages/agent/vitest.config.ts deleted file mode 100644 index ab19cb94..00000000 --- a/packages/agent/vitest.config.ts +++ /dev/null @@ -1,16 +0,0 @@ -import { defineConfig } from "vitest/config"; - -export default defineConfig({ - server: { - host: "127.0.0.1", - }, - test: { - globals: true, - environment: "node", - testTimeout: 30000, - // pi-ai ships real ESM and imports "openai" itself; both need to run - // through Vitest's module graph (not Node's native loader) for - // vi.mock("openai") to intercept requests pi's builtin transport makes. - server: { deps: { inline: [/@earendil-works\/pi-ai/, /^openai$/] } }, - }, -}); diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md deleted file mode 100644 index 7a7045c7..00000000 --- a/packages/ai/CHANGELOG.md +++ /dev/null @@ -1,433 +0,0 @@ -# Changelog - -## Unreleased - -- Fix OpenAI's native computer transport rejecting every request after a - screenshot-less action. A `computer_call_output` whose result carried no image - put the failure text in an `error` key, which the Responses API refuses outright - (`400 Unknown parameter: 'input[N].output.error'`), so one failed action poisoned - the rest of the conversation. The output now always carries a valid - `computer_screenshot`, and the failure text follows as a user message so the - model still learns what happened. A 1x1 placeholder is not enough — the - Responses API rejects it even though the vision endpoint accepts one. -- Anthropic's native browser and native computer tools can no longer be selected - together. Anthropic answers 400 because the browser tool addresses a viewport - coordinate frame and the computer tool a display frame; the catalog now refuses - the pair at compile time instead of on the wire. -- Google no longer carries a schema quirk. The Gemini API rejects the JSON Schema - keywords `const` and `additionalProperties` outright rather than ignoring them, - so a payload transform rewrites both for Google — `const: x` becomes a - single-value `enum`, which means the same thing. Gemini now accepts every - function tool CUA offers, including `browser_act` and `browser_wait_for`, which - the removed quirk had marked unavailable. - -- Add `cuaToolMenu(model, selected)`: every tool CUA can offer for a model, each - marked available or not with the compiler's own reason when it is not. It - decides availability by compiling the candidate catalog rather than restating - the compiler's rules, so the menu cannot drift from what - `compileCuaToolCatalog` accepts. Availability is relative to the current - selection, because two providers' native surfaces cannot coexist and a native - surface pins the transport. - -Breaking: the model allowlist is removed. - -- `listCuaModels()` returns pi-ai's whole catalog — 37 providers, ~1,150 models — - instead of a curated subset, and each entry now carries `nativeSurfaces` and - `vision` so callers can render what a model can do. -- `getCuaModel(ref)` resolves any model pi-ai carries, and synthesizes one for - an id the registry has not caught up with, using the sibling that shares the - longest id prefix and preferring the latest such sibling. Providers migrate - transports mid-generation, so a new id follows its nearest, newest relative. - Only an unqualified ref or a provider pi-ai does not carry is refused. -- `CUA_MODEL_ANNOTATIONS`, `CUA_PROVIDERS`, `isCuaProvider`, and the - `CuaProvider` union are gone; `CuaProvider` is now a provider id string and - `cuaProviders()` returns what pi-ai carries. `providerForModel` no longer - throws. -- Two tables replace the allowlist, neither of which decides whether a model may - run: `CUA_NATIVE_SURFACES` (which models have a provider-native computer or - browser tool, with first-party sources) and `CUA_MODEL_QUIRKS` (request-shape - limits, each carrying the documented limit or observed failure that justifies - it). `cuaModelCapabilities` reads the quirk table and defaults to permissive; - `cuaNativeSurfaces(model)` and `cuaModelQuirks(model)` are exported for menus - and diagnostics. - -Breaking: a model's transport is derived from the tools selected with it, rather -than stamped on the model. - -- `compileCuaToolCatalog` derives the compiled model's `api` from the selected - tools' provider bindings: a `CuaProviderBinding` may declare `requiresApi`, - and the returned `catalog.model` carries that transport. Selecting tools whose - bindings require different transports fails to compile with a named catalog - error. This makes transport a function of `(model, selected tools)` instead of - `(model)` alone. -- `getCuaModel("google:...")` no longer forces `google-cua-interactions`. A - Google model resolved without Google's native browser toolset selected now - keeps pi-ai's builtin `google-generative-ai` transport; selecting - `cua.providers.google.toolsets.browser()` still compiles to - `google-cua-interactions` as before. -- Add `OPENAI_CUA_COMPUTER_API` (`"openai-cua-computer"`). A model compiled with - `cua.providers.openai.tools.computer()` selected carries this api, and the - OpenAI provider wrapper dispatches to the CUA adapter on `model.api` alone. - The one remaining request-shape check, `requiresCuaOpenAINamespaceAdapter`, - covers only what cannot be derived from the model: a transcript carrying a - deferred tool-search addition or a replayed function-call namespace, neither - of which pi-ai's builtin transport round-trips. -- Remove `routeCuaApi`. Every provider now takes its transport from pi-ai's - registry or from the selected tools' `requiresApi`, so model resolution - returns pi-ai's data unmodified. Its last remaining branch patched grok-4.5's - thinking-level map, price tiers, and compat flags onto pi-ai's registry entry; - live runs on grok-4.5 at the default, `off`, and `xhigh` thinking levels - behave identically without it. The only lost detail is a >200k-token price - tier that pi's registry does not carry, which affects `usage.cost` reporting - for long requests and nothing else. - -Breaking: OpenAI models no longer carry a CUA-owned api id. - -- OpenAI models resolve to pi-ai's builtin `"openai-responses"` api instead of - the removed `openai-cua-responses`, and stream through pi's builtin Responses - transport (`store: false`, automatic prompt-cache-key matching) by default. - `OPENAI_CUA_RESPONSES_API` and the OpenAI adapter's `previous_response_id` - threading are removed; `previous_response_id` and `store: true` no longer - appear on any OpenAI request. -- OpenAI's native computer adapter now sends the same `prompt_cache_key`, - `prompt_cache_retention`, `prompt_cache_options`, and session-affinity headers - as the function-tool path. It previously relied on stored response state for - context reuse and sent no cache key of its own. -- Remove the xAI Responses fork. It existed only to thread - `previous_response_id` and to set `parallel_tool_calls: false`, which the tool - catalog already emits for the provider. `xai-cua-responses` is gone and Grok - streams through pi's builtin xAI provider; `XAI_CUA_RESPONSES_API`, - `streamXaiResponses`, and `streamSimpleXaiResponses` are no longer exported. -- Google keeps its continuation protocol: it threads a provider-specific field - with no builtin equivalent, and the shared helpers in `providers/common.ts` - are unchanged for it. - -Breaking: the Tzafon, Yutori, and Meta providers are removed. - -- Remove the `tzafon` and `yutori` providers: their model annotations and - overrides, `cua.providers.tzafon`, `cua.providers.yutori`, - `TZAFON_API_KEY`/`YUTORI_API_KEY`, and the exported `TZAFON_RESPONSES_API`, - `YUTORI_CHAT_COMPLETIONS_API`, `streamTzafonResponses`, - `streamSimpleTzafonResponses`, `streamYutori`, and `streamSimpleYutori` stream - functions. `createCuaModels()` no longer registers either provider, and refs - like `tzafon:tzafon.northstar-cua-fast` or `yutori:n1.5-latest` now fail to - resolve. Drops the `@tzafon/lightcone` dependency. -- Remove the `meta` provider. pi-ai ships no `meta` provider, so cua hand-wrote - a model entry and pointed pi's own Responses transport at `api.meta.ai`. - `META_API_KEY`, `META_RESPONSES_API`, `streamMetaResponses`, and - `streamSimpleMetaResponses` go with it. Meta was the last user of the - model-override mechanism, so `CUA_MODEL_OVERRIDES` and `cuaOverrideModels()` - are gone too, and `getCuaModel` no longer has a "supported but not registered" - fallback. Muse Spark remains available through pi's OpenRouter catalog as - `openrouter:meta/muse-spark-1.1`, annotated with explicit capabilities because - OpenRouter's provider-level defaults are conservative. -- `CuaProviderBinding` loses its `tzafon-native` and `yutori-native` variants, - and `CuaIncomingToolPlan` loses `tzafonComputerName` and `yutoriNames`. The - Yutori-only rule rejecting a partial n1 native action set is gone with them; - every surviving native toolset can be selected in part. -- `CompileCuaToolCatalogOptions.viewport` is removed. It existed only to fill - Tzafon's `display_width`/`display_height` declaration defaults, and no - surviving declaration reads it. - -## 0.10.0 - 2026-08-04 - -Breaking: upgrade `@earendil-works/pi-ai` to 0.83.0. - -- Remove the local `claude-opus-5`, `gemini-3.6-flash`, and - `gemini-3.5-flash-lite` model overrides: pi-ai 0.83.0's registry now carries - all three with the same metadata. Overrides remain only for the CUA-only - providers pi-ai does not ship (Meta, Tzafon, Yutori). -- Kimi K3 reasoning effort follows pi-ai's catalog metadata with no CUA - override: `low`/`high`/`max` map through `thinkingLevelMap` (the rest clamp - away), and requests carry `reasoning_effort` on Moonshot or OpenRouter's - nested `reasoning.effort`. -- Disable Tzafon native non-screenshot action loops: its Responses API requires - every `computer_call_output` to carry an image, while CUA returns screenshots - only when explicitly requested. Unsupported native actions now fail before - browser execution instead of entering a text-only loop that the API rejects. - -## 0.9.0 - 2026-08-03 - -- Add Kimi K3 through OpenRouter as `openrouter:moonshotai/kimi-k3`, - authenticated with `OPENROUTER_API_KEY`, while retaining direct Moonshot access - through `moonshotai:kimi-k3`. -- Share Kimi K3's CUA capability metadata across both transports: nested browser - schemas are accepted, the larger `browser_act` schema is rejected, and - state-mutating function tools disable parallel calls. -- Resolve schema and tool-call compatibility from concrete model capabilities - instead of provider-wide allowlists. Existing provider-family defaults are - preserved, and provider-native tools remain restricted to their native - transports. - -## 0.8.0 - 2026-07-31 - -Breaking: agent tools are now one explicit, identity-keyed catalog. The mode, -implicit-tool, and runtime-spec APIs are removed. - -- Add the frozen `cua` namespace with atomic browser/computer/Playwright tools, - `browser`/`computer`/`mixed` convenience toolsets, mechanical batch tools, - coordinate contracts, and provider-native tools/toolsets. -- Add `compileCuaToolCatalog()` with stable identities; exact requested-order - preservation; schema/catalog fingerprints; exact and - provider-normalized collision checks; model compatibility checks; inspectable - declarations; dynamic-loading eligibility; generated header composition; - ordered payload transforms; and incoming native-call plans. -- Catalog compilation is declaration-only and deterministic: it accepts CUA - specs and sanitized caller `Tool` declarations plus a viewport, returns - pi-ai `Tool` declarations (`catalog.toolDeclarations`) and provider plans, - and never constructs executable tools or retains the requested inputs. - `callerToolIdentity()` is the single canonical identity scheme for caller - tools, shared with cua-agent and cua-cli. The package no longer depends on - `@earendil-works/pi-agent-core`; materialization and implementation identity - live in `@onkernel/cua-agent`. -- Remove `resolveCuaRuntimeSpec`, `CuaRuntimeSpec`, `CuaMode`, mode inference, - legacy native-tool switches, implicit navigation tools, and provider-owned - default prompt selection from the public API. -- Provider-native declarations now compose by selected identity with ordinary - functions. Add fixed-version Anthropic computer/browser factories, OpenAI - native computer composition, Tzafon viewport-aware declaration replacement, - Google's current predefined browser toolset, and identity-scoped Yutori - native selection. Every provider surface exposes its first-party source. -- Add deterministic provider composition: generated model preparation, tool - serialization, provider fields, then caller payload hooks. - Header requirements merge without overwriting unrelated caller headers. -- Add a Google Interactions API adapter plus current `computer_use` browser - action names and `[0, 999]` coordinates. Exact-subset declarations exclude - every unselected current action, and excluded incoming calls fail with a - named catalog error. Support the documented `gemini-3.6-flash`, - `gemini-3.5-flash`, and `gemini-3.5-flash-lite` models. -- Add `anthropic:claude-opus-5` with its 1M-token context window, 128k output - limit, adaptive thinking levels, and July 2026 native-tool compatibility. - Native `browser_20260701` transparently retries through an equivalent - function-tool declaration when the active credential lacks beta access. -- Add the verified `openai:gpt-5.6-sol` model. -- Expose Google's predefined actions through `browser()` only. Remove legacy - Google actions and the Meta/xAI/Moonshot coordinate toolsets; those - custom-function providers use the standard CUA browser toolset. -- Validate large function schemas separately from ordinary nested schemas. - Moonshot retains CUA browser primitives and `browser_wait_for`, but catalog - compilation now rejects `browser_act`, whose larger schema its API refuses. -- Native tool execution metadata carries stop-on-first-failure policy without - introducing provider branches in cua-agent. -- Declare `engines.node` `>=22.19.0`. This is not a new requirement: every - `@earendil-works/pi-*` dependency already declares the same floor, so it was - previously enforced only transitively and never stated on this package. - -## 0.7.0 - 2026-07-17 - -- Added Moonshot Kimi K3 computer-use support: `moonshotai:kimi-k3` - (`moonshot:` accepted as a ref alias), authenticated with - `MOONSHOT_API_KEY`. Kimi streams through the OpenAI-compatible chat - completions transport with ordinary function tools. -- New `moonshot` provider namespace following the standard conventions - (`computerTools`, `coordinateSystem`, `buildMoonshotSystemPrompt`, - `providerModule`, …). Kimi's coordinate contract is normalized 0–1 - width/height fractions, matching the model's native visual grounding; - payload middleware disables parallel tool calls. -- Bumped `@earendil-works/pi-ai` to 0.80.10 (carries the Kimi K3 registry - entry). Grok 4.5's registry api id is now `openai-responses`; CUA routing - behavior is unchanged. - -## 0.6.0 - 2026-07-10 - -Breaking: response threading is now configured per request instead of through -`CUA_DISABLE_RESPONSE_THREADING`. - -- `CuaSimpleStreamOptions` now includes `disableResponseThreading`, allowing - OpenAI and Tzafon Responses API calls to send the complete current context - instead of continuing through `previous_response_id`. -- Removed the process-wide `CUA_DISABLE_RESPONSE_THREADING` environment - variable. `@onkernel/cua-agent` users can set `responseThreading: false` on - `CuaAgent` or `CuaAgentHarness`; lower-level callers can pass - `disableResponseThreading: true` in stream options. - -## 0.5.0 - 2026-07-09 - -Introduces action planes (modes) and Anthropic native computer-use tools. - -- New `mode` option (`"computer"` | `"browser"` | `"hybrid"`, exported as - `CuaMode`) on `resolveCuaRuntimeSpec`, `computerTools`, and the executor - builders selects which action plane(s) the model sees. `computer` is the - pre-modes default and stays byte-compatible. `browser` exposes the new - browser-plane canonical actions; `hybrid` exposes both planes deduplicated - to one tool per capability, with browser actions restricted to element refs - so the OS screenshot is the single coordinate frame. -- New browser-plane canonical actions (`CUA_BROWSER_ACTION_TYPES`): - `browser_snapshot`, `browser_find`, `browser_text`, `browser_click`, - `browser_fill`, `browser_scroll_to`, `browser_navigate`, - `browser_list_tabs`, `browser_new_tab`, `browser_screenshot`, - `browser_evaluate`, and friends, with per-mode tool naming - (`cuaToolNameForAction`), descriptions, schemas, and system prompts. -- New `nativeTool` option drives Anthropic models through their native - computer-use declarations: `computer_20260701` (computer mode, with - `enable_zoom`) behind `anthropic-beta: computer-use-2026-07-01`, and - `browser_20260701` (browser mode) behind - `anthropic-beta: browser-use-2026-07-01`. -- JavaScript execution is on by default: `browser_evaluate` is part of the - default browser/hybrid action sets, and native `browser_20260701` - declarations default `enable_javascript_exec` to true (an explicit value on - the spec wins). Opt out by passing an explicit `actions` list or native - tool spec. -- New `zoom` computer action (cropped display inspection). - -## 0.4.0 - 2026-07-07 - -Breaking: adopts pi-ai 0.80's instance-based `Models` API and drops the -global api-registry surface. - -- Requests now stream through a pi `Models` collection. New exports: - `createCuaModels(options?)` builds a collection with pi's builtin providers - plus CUA's adjustments (OpenAI routed through `openai-cua-responses`, - Google accepting `GOOGLE_API_KEY` or `GEMINI_API_KEY`, Tzafon and Yutori - registered); `cuaModels()` returns the shared default collection. -- Removed `registerCuaProviders()` and the import-time registration side - effect. Build a collection with `createCuaModels()` instead; nothing global - is mutated. -- The pi-ai re-export now follows pi-ai 0.80: the free functions `complete`, - `stream`, `completeSimple`, `streamSimple`, `getModel`, `getModels`, and - the api-registry (`registerApiProvider`, `getApiProvider`, - `resetApiProviders`, …) are gone. Call the equivalent methods on - `cuaModels()`. -- API keys resolve from the documented env-var convention through provider - auth when streaming via the collection; explicit `apiKey` stream options - still take precedence. -- Removed the `claude-sonnet-5` and `gpt-5.5` model overrides (pi-ai 0.80's - registry carries both) and the `gpt-5.5-2026-04-23` dated-snapshot - override. Dated snapshot refs no longer resolve — use the family id - (`openai:gpt-5.5`). -- Updated `@earendil-works/pi-ai` to 0.80.3. - -## 0.3.4 - 2026-06-30 - -- Adapt newer Anthropic models to the adaptive thinking payload format, including `claude-sonnet-5`, `claude-opus-4-8`, and `claude-opus-4-7`. - -## 0.3.3 - 2026-06-30 - -- Add computer-use support for the `claude-sonnet-5` Anthropic model. - -## 0.3.2 - 2026-06-24 - -- Add computer-use support for the `gemini-3.5-flash` Google model. - -## 0.3.1 - 2026-06-23 - -- Add the `playwright_execute` tool definition: `CuaPlaywrightSchema`, - `CUA_PLAYWRIGHT_TOOL_NAME`, `CUA_PLAYWRIGHT_TOOL_DESCRIPTION`, - `createCuaPlaywrightToolDefinition()`, and the `CuaPlaywrightInput` type. - -## 0.3.0 - 2026-06-12 - -- Add `CuaSimpleStreamOptions`: pi-ai `SimpleStreamOptions` plus the - `keepToolNames` extension the Yutori/Tzafon stream adapters consume, so - callers can pass it through `streamSimple` without a cast. - -## 0.2.2 - 2026-06-11 - -- Add computer-use support for `gpt-5.4-mini`, `gemini-3.1-flash-lite`, `tzafon.northstar-cua-fast-1.6`, and `tzafon.northstar-cua-fast-1.7-experiment`. -- Drop `gemini-3-pro-preview`, which Google has retired (the API now returns 404 for it). - -## 0.2.1 - 2026-06-11 - -- Add computer-use support for the `claude-fable-5` Anthropic model. - -## 0.2.0 - 2026-06-10 - -### Fixed - -- The published package is now importable under plain Node ESM. 0.1.0 shipped - extensionless relative imports in `dist/`, so `import "@onkernel/cua-ai"` - failed outside bundlers; `dist/` is now bundled with tsdown. -- The shipped `examples/quickstart.ts` imports `@onkernel/cua-ai` instead of a - `../src` path that does not exist in the tarball, checks `stopReason` so - provider errors are no longer silent, resolves its API key via - `requireCuaEnvApiKeyForModel`, and switches providers with the `CUA_MODEL` - env var. -- `docs/` (the supported-models list the README links to) is now included in - the npm tarball. -- A malformed Yutori tool call now degrades to an empty-arguments call instead - of failing the entire response, matching the existing Tzafon hardening. - -### Breaking changes - -- Provider namespaces follow one convention. Every namespace now exports - `computerTools({ actions? })` / `computerToolExecutors({ actions? })`, - `createActionSchema`, `coordinateSystem()`, `providerModule`, - `_CUA_ACTION_TYPES`, `_COMPUTER_INSTRUCTIONS`, a - `Action` type, and `ComputerToolsOptions`. This replaces 0.1.0's - `createComputerToolDefinitions(options)` / - `CreateComputerToolDefinitionsOptions`, the per-namespace - `COMPUTER_TOOL_COORDINATES` constants, `TZAFON_ACTION_TYPES` / - `YUTORI_ACTION_TYPES`, and the `OPENAI_BATCH_INSTRUCTIONS` / - `GEMINI_INSTRUCTIONS_RAW` / `TZAFON_INSTRUCTIONS_RAW` / - `YUTORI_INSTRUCTIONS_RAW` prompt constants. -- `CUA_BATCH_TOOL_NAME` is now `"computer_batch"` (was - `"batch_computer_actions"`), matching the batch tool Anthropic ships by - default. `anthropic.ANTHROPIC_BATCH_TOOL_NAME` carries the same new value; - the other per-namespace batch aliases (`TZAFON_BATCH_TOOL_NAME`, - `YUTORI_BATCH_TOOL_NAME`, `*_BATCH_DESCRIPTION`, `*BatchSchema`, - `*BatchInput`) were removed — use `CUA_BATCH_TOOL_NAME`, - `CUA_BATCH_TOOL_DESCRIPTION`, `CuaBatchSchema`, and `CuaBatchInput`. -- Anthropic tools are now the 13 canonical browser actions Anthropic supports - (no `back`/`forward`/`url`) plus a `computer_batch` batch tool by default; - pass `excludeBatch: true` to omit it. Unsupported `actions` entries throw. - `anthropic.ANTHROPIC_CUA_ACTION_TYPES` reflects the supported subset rather - than aliasing the full canonical list. -- Yutori models now use Yutori's documented native `tool_set` request field. - `streamYutori` strips canonical action tools from the outbound payload - (preserve specific tools via the `keepToolNames` stream option), selects the - n1.5 core tool set where applicable, and normalizes native tool calls back - to canonical names. `yutori.providerModule.toolDefinitions()` is `[]`; - `yutori.computerTools()` builds local mirrors for executor lookup, validates - `{ actions }` against the supported subset, and throws on unsupported - actions. `yutoriBuiltinToolsOnPayload` was replaced by - `yutoriNativeToolSetOnPayload`. The Yutori runtime spec also carries a - screenshot policy (append a 1280x800 webp screenshot to the latest message). -- Family model annotations now match only the family root plus numeric - revision or dated-snapshot suffixes (`claude-opus-4-7`, - `gpt-5.5-2026-04-23`). Named sibling variants such as `gpt-5.4-mini` are no - longer listed by `listCuaModels()` or accepted by `getCuaModel()` without - their own annotation. -- `google:gemini-2.5-computer-use-preview-10-2025` was removed from the - catalog: it rejects the standard function declarations this package sends - and requires Google's native `tools.computer_use` wrapper. Use - `google:gemini-3-flash-preview` or `google:gemini-3-pro-preview`. -- `streamTzafonResponses` no longer accepts a `maxOutputTokens` option — use - the standard `maxTokens` stream option. - -### Added - -- `CuaProviderModule` contract plus a `providerModule` export per namespace, - and a richer `CuaRuntimeSpec`: `toolExecutors` (local adapters that turn - provider tool calls into canonical `CuaAction`s via `CuaToolExecutorSpec`), - `coordinateSystem`, and optional `screenshot` policy alongside the existing - tool definitions, default prompt, and payload middleware. -- `resolveCuaRuntimeSpec(input, options?)` accepts `ComputerToolsOptions` and - forwards it to the provider module, so runtime consumers can narrow tool - definitions and executors (e.g. `{ actions: ["click"] }`). -- `registerCuaProviders()` is exported: importing the package still registers - the Yutori/Tzafon stream providers automatically, and this restores them - after pi-ai registry mutators (`clearApiProviders`, `resetApiProviders`, - `unregisterApiProviders`). -- `parseCuaModelRef` / `getCuaModel` accept `"gemini:"` refs as an alias for - `"google:"`, and unsupported-provider errors now list the valid providers. -- `CuaMouseButton` and `CuaDragMouseButton` closed unions type the `button` - field on click/mouse_down/mouse_up and drag actions. -- `yutori.YutoriOptions` and `tzafon.TzafonResponsesOptions` are exported and - aligned; both support `keepToolNames` to preserve caller tools that collide - with canonical action names on the wire. -- Yutori native action vocabulary exports: `YUTORI_N1_ACTION_TYPES`, - `YUTORI_N15_CORE_ACTION_TYPES`, `YUTORI_N15_EXPANDED_ACTION_TYPES`, - tool-set ids, `yutoriToolSetForModel`, `yutoriNativeActionsForModel`, and - `toCanonicalActions`; Tzafon exports `toCanonicalActions`, - `TzafonCanonicalAction`, `tzafonComputerUseOnPayload`, and - `tzafonToolCallId`. -- README and JSDoc coverage across the public surface: API key prerequisites - and helpers, error handling (`stopReason` semantics), a multi-turn - tool-result example, the complete export list, and per-provider canonical - action subsets. - -## 0.1.0 - -- Provider-qualified CUA model catalog with support annotations and curated overrides. -- Unified runtime-spec resolution for provider defaults (tools, prompts, payload middleware). -- Registers CUA provider adapters and exports canonical computer-use schemas/tool definitions. diff --git a/packages/ai/README.md b/packages/ai/README.md deleted file mode 100644 index 1bb7ee45..00000000 --- a/packages/ai/README.md +++ /dev/null @@ -1,288 +0,0 @@ -# `@onkernel/cua-ai` - -The model and tool-policy layer for Kernel computer-use agents, built on -`@earendil-works/pi-ai` 0.83.0. - -Use [`@onkernel/cua-agent`](../agent) when you also want Kernel-browser tool -execution. Use this package directly for model discovery, explicit tool catalog -construction, and provider transport composition. - -## Install - -```bash -npm install @onkernel/cua-ai -``` - -Requires Node 22.19 or newer. - -## Model catalog - -Model references are always provider-qualified: - -```ts -import { - getCuaModel, - listCuaModels, - parseCuaModelRef, -} from "@onkernel/cua-ai"; - -const model = getCuaModel("openai:gpt-5.6-sol"); -console.log(parseCuaModelRef("anthropic:claude-opus-5")); -console.table(listCuaModels("google")); -``` - -`gemini:` aliases `google:` and `moonshot:` aliases `moonshotai:`. The package -does not export a default model. See [models and native surfaces](docs/supported-models.md) -for which models have provider-native tools and which have known request limits. - -## Explicit tools - -All CUA-owned tools are available from one frozen namespace: - -```ts -import { cua } from "@onkernel/cua-ai"; - -const tools = [ - cua.tools.browser.snapshot(), - cua.tools.browser.click(), - cua.tools.computer.screenshot(), -]; -``` - -Nothing is inferred from the model and no fallback tools are appended. - -### Atomic browser tools - -```ts -cua.tools.browser.snapshot(); -cua.tools.browser.text(); -cua.tools.browser.find(); -cua.tools.browser.click(); -cua.tools.browser.hover(); -cua.tools.browser.drag(); -cua.tools.browser.fill(); -cua.tools.browser.scrollTo(); -cua.tools.browser.scroll(); -cua.tools.browser.type(); -cua.tools.browser.key(); -cua.tools.browser.navigate(); -cua.tools.browser.listTabs(); -cua.tools.browser.newTab(); -cua.tools.browser.screenshot(); -cua.tools.browser.evaluate(); -cua.tools.browser.waitFor(); -cua.tools.browser.act(); -``` - -`browser_act` retains the established browser-action schema. Atomic tools expose -operation-specific arguments directly—there is no outer action wrapper. - -### Atomic computer tools - -```ts -cua.tools.computer.click(); -cua.tools.computer.doubleClick(); -cua.tools.computer.mouseDown(); -cua.tools.computer.mouseUp(); -cua.tools.computer.type(); -cua.tools.computer.keypress(); -cua.tools.computer.scroll(); -cua.tools.computer.move(); -cua.tools.computer.drag(); -cua.tools.computer.wait(); -cua.tools.computer.screenshot(); -cua.tools.computer.zoom(); -cua.tools.computer.goto(); -cua.tools.computer.back(); -cua.tools.computer.forward(); -cua.tools.computer.url(); -cua.tools.computer.cursorPosition(); -``` - -Computer coordinates default to pixels. Callers can request an explicit -normalized contract: - -```ts -cua.toolsets.computer({ - coordinates: cua.coordinates.normalized([0, 1000]), -}); -``` - -### Toolsets, names, and batches - -```ts -cua.toolsets.browser(); -cua.toolsets.computer(); -cua.toolsets.mixed(); -cua.toolsets.browser({ namespace: "page" }); - -cua.tools.browser.snapshot({ name: "page_snapshot" }); -cua.tools.computer.click({ name: "os_click" }); - -cua.tools.computer.batch({ actions: ["click", "keypress", "screenshot"] }); -cua.tools.browser.batch({ actions: ["snapshot", "click", "wait_for", "text"] }); - -cua.tools.playwright(); -``` - -Batches are mechanical primitive lists. They have no branching, saved values, -references, or workflow DSL. - -## Provider-native composition - -Provider-native tools are selected explicitly and may coexist with ordinary -function tools. - -```ts -const tools = [ - cua.providers.anthropic.tools.computer({ - version: "20260701", - enableZoom: true, - }), - cua.tools.browser.snapshot(), -]; -``` - -Available groups: - -```ts -cua.providers.openai.tools.computer(); - -cua.providers.anthropic.source; -cua.providers.anthropic.tools.computer({ version: "20260701" }); -cua.providers.anthropic.tools.browser({ version: "20260701" }); - -cua.providers.google.source; -cua.providers.google.toolsets.browser({ exclude: ["right_click"] }); - -// Meta, xAI, and Moonshot use the ordinary CUA browser tools. -cua.toolsets.browser(); -``` - -The Google browser set exposes the current predefined action names and uses -normalized coordinates in `[0, 999]`. Its native `computer_use` declaration -excludes every unselected browser action. If Google emits an excluded name -anyway, the adapter returns a named exact-catalog error instead of forwarding -an undeclared tool call. - -Moonshot accepts the ordinary browser toolset, including `browser_wait_for`, -but rejects `browser_act`'s substantially larger function schema. Catalog -compilation rejects that specific combination before a provider request. - -Provider-native caller-visible names are fixed by protocol. Version/tool/model -mismatches fail during catalog compilation. If an Anthropic credential cannot -access `browser_20260701`, CUA retries with an equivalent `browser` function -tool and remembers that choice for the credential and process. Every -`cua.providers.*` tool surface exposes its first-party `source` (or versioned -`sources`), and every returned provider spec carries the applicable URL. - -## Catalog compilation - -`compileCuaToolCatalog()` is the identity and validation boundary used by -`@onkernel/cua-agent`: - -```ts -const catalog = compileCuaToolCatalog({ - model: "anthropic:claude-opus-5", - requestedTools: tools, // CUA specs and plain pi-ai Tool declarations - viewport: { width: 1440, height: 900 }, -}); - -catalog.entries; // identities, fingerprints, declarations, coordinates -catalog.toolDeclarations; // pi-ai Tool declarations for Context.tools -catalog.headers.merge(callerHeaders); -await catalog.payload.apply(payload, catalog.model); -catalog.incoming; -``` - -Compilation is declaration-only and deterministic: identical declaration, -model, and viewport inputs produce identical catalogs, and compilation never -constructs executable tools or retains the requested input objects. cua-ai has -no `pi-agent-core` dependency — `@onkernel/cua-agent` materializes specs -against a Kernel browser and owns implementation identity. - -A CUA-owned identity remains stable when its name is customized. Caller tools -receive `caller.` identities through the canonical `callerToolIdentity()` -helper shared with every consumer. Compilation rejects: - -- duplicate identities; -- exact or provider-normalized caller-visible name collisions; -- unsafe names; -- incompatible model/provider-native combinations; -- conflicting payload-transform write claims; -- partial provider-native selections that violate a provider contract. - -The catalog fingerprint includes model, order, identity, name, schema, and -coordinates. cua-agent composes these declaration fingerprints with its own -implementation identity, so a schema or executor replacement cannot -masquerade as a no-op. - -Generated payload processing has deterministic order: - -1. model preparation; -2. tool declaration serialization; -3. provider request fields; -4. caller `onPayload` (applied by `cua-agent`). - -Generated header requirements merge with caller headers. Comma-list headers are -unioned and deduplicated; exact-value conflicts throw. - -## Dynamic loading metadata - -Ordinary function tools are marked eligible only where pi 0.83.0 supports -deferred loading. Provider-native tools are eager-only. The catalog itself does -not guess when tools were added; a caller that adds tools mid-turn records the -addition through pi's active-tool change entries. - -## Provider behavior - -Transport is derived, not stamped on the model ahead of time: a selected -tool's provider binding may declare `requiresApi`, and `compileCuaToolCatalog` -returns a `catalog.model` carrying that api. Selecting tools whose bindings -require different transports fails to compile. - -- **OpenAI**: a model selected with only ordinary/CUA browser tools streams - through pi's builtin Responses transport and its automatic prompt caching. - Selecting `cua.providers.openai.tools.computer()` derives the CUA-owned - `openai-cua-computer` api instead, which a CUA adapter handles; that same - adapter also covers tool-search namespace round-trips regardless of api, - since pi's builtin transport does not replay them. -- **Anthropic**: exact native declarations, beta-header composition, and - adaptive model preparation. No api fork — every Anthropic model streams - through pi's builtin transport. -- **Google**: a model selected without Google's native browser toolset streams - through pi's builtin transport. Selecting - `cua.providers.google.toolsets.browser()` derives the CUA-owned - `google-cua-interactions` api, which serializes one `computer_use` - declaration plus explicit exclusions through the Interactions API adapter. -- **Meta/xAI/Moonshot**: ordinary function tools with serial tool calls when the - selected catalog mutates browser state. - -## API keys - -```ts -import { - cuaApiKeyEnvVarsForProvider, - getCuaEnvApiKeyForModel, - requireCuaEnvApiKeyForModel, -} from "@onkernel/cua-ai"; -``` - -Conventional variables are `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, -`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `XAI_API_KEY`, and -`MOONSHOT_API_KEY`. - -## Development - -```bash -npm run typecheck --workspace @onkernel/cua-ai -npm test --workspace @onkernel/cua-ai -npm run build --workspace @onkernel/cua-ai -``` - -See [`examples/quickstart.ts`](examples/quickstart.ts) for direct catalog/model -usage. - -## License - -MIT. diff --git a/packages/ai/package.json b/packages/ai/package.json deleted file mode 100644 index c43e91b6..00000000 --- a/packages/ai/package.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "name": "@onkernel/cua-ai", - "version": "0.10.0", - "description": "Kernel-curated computer-use model access built on pi-ai", - "license": "MIT", - "type": "module", - "main": "./dist/index.js", - "types": "./dist/index.d.ts", - "repository": { - "type": "git", - "url": "git+https://github.com/kernel/cua.git", - "directory": "packages/ai" - }, - "bugs": { - "url": "https://github.com/kernel/cua/issues" - }, - "homepage": "https://github.com/kernel/cua/tree/main/packages/ai#readme", - "exports": { - ".": { - "types": "./dist/index.d.ts", - "source": "./src/index.ts", - "import": "./dist/index.js" - } - }, - "files": [ - "dist", - "docs", - "examples", - "README.md", - "CHANGELOG.md" - ], - "publishConfig": { - "access": "public" - }, - "engines": { - "node": ">=22.19.0" - }, - "scripts": { - "build": "tsdown", - "typecheck": "tsc -b", - "clean": "tsc -b --clean && rm -rf dist dist-tsc", - "example:quickstart": "NODE_OPTIONS=--conditions=source tsx examples/quickstart.ts", - "test": "vitest --run", - "test:integration": "vitest --run --config vitest.integration.config.ts" - }, - "dependencies": { - "@earendil-works/pi-ai": "0.83.0", - "openai": "^6.26.0" - }, - "devDependencies": { - "tsdown": "^0.22.2", - "vitest": "^3.2.4" - } -} diff --git a/packages/ai/src/actions/index.ts b/packages/ai/src/actions/index.ts deleted file mode 100644 index a2d164e5..00000000 --- a/packages/ai/src/actions/index.ts +++ /dev/null @@ -1,47 +0,0 @@ -import type { TSchema } from "@earendil-works/pi-ai"; -import { CUA_BROWSER_ACTION_TYPES, createCuaBrowserActionSchemaByType, type CuaBrowserAction, type CuaBrowserActionType, type CuaBrowserSchemaOptions } from "./browser"; -import { CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE, CUA_COMPUTER_ACTION_TYPES, type CuaComputerAction, type CuaComputerActionType } from "./computer"; - -export * from "./browser"; -export * from "./computer"; - -/** Any canonical CUA action type, across the computer and browser planes. */ -export type CuaActionType = CuaComputerActionType | CuaBrowserActionType; - -/** Any canonical CUA action, across the computer and browser planes. */ -export type CuaAction = CuaComputerAction | CuaBrowserAction; - -/** Every canonical action type: the computer plane followed by the browser plane. */ -export const CUA_ALL_ACTION_TYPES: readonly CuaActionType[] = [...CUA_COMPUTER_ACTION_TYPES, ...CUA_BROWSER_ACTION_TYPES]; - -const COMPUTER_ACTION_TYPE_SET: ReadonlySet = new Set(CUA_COMPUTER_ACTION_TYPES); -const BROWSER_ACTION_TYPE_SET: ReadonlySet = new Set(CUA_BROWSER_ACTION_TYPES); - -/** Whether a canonical action type belongs to the computer plane. */ -export function isCuaComputerActionType(action: CuaActionType): action is CuaComputerActionType { - return COMPUTER_ACTION_TYPE_SET.has(action); -} - -/** Whether a canonical action type belongs to the browser plane. */ -export function isCuaBrowserActionType(action: CuaActionType): action is CuaBrowserActionType { - return BROWSER_ACTION_TYPE_SET.has(action); -} - -/** Whether a canonical action belongs to the browser plane. */ -export function isCuaBrowserAction(action: CuaAction): action is CuaBrowserAction { - return BROWSER_ACTION_TYPE_SET.has(action.type); -} - -/** Options for building canonical action schemas. */ -export interface CuaActionSchemaOptions { - /** browser-plane schema variants; see {@link CuaBrowserSchemaOptions}. Defaults to coordinates allowed. */ - browser?: CuaBrowserSchemaOptions; -} - -/** Build the full action-type → schema map for a schema-options combination. */ -export function cuaActionSchemaByType(options: CuaActionSchemaOptions = {}): Record { - return { - ...CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE, - ...createCuaBrowserActionSchemaByType(options.browser ?? { coordinates: true }), - }; -} diff --git a/packages/ai/src/api-keys.ts b/packages/ai/src/api-keys.ts deleted file mode 100644 index fba7ee83..00000000 --- a/packages/ai/src/api-keys.ts +++ /dev/null @@ -1,69 +0,0 @@ -import type { Api, Model } from "@earendil-works/pi-ai"; -import { parseCuaModelRef, providerForModel, type CuaModelRef } from "./models"; - -/** - * Environment variables for the providers CUA documents, in precedence order. - * - * Every provider pi-ai carries is selectable, and pi resolves each one's own - * credential when streaming. This table exists only so callers and the CLI can - * name the variable to set up front; a provider absent from it is not - * unsupported, it just has no CUA-side preflight. pi-ai does not export its - * own env-var registry, or this would read from that. - */ -const CUA_PROVIDER_API_KEY_ENV_VARS: Readonly> = { - openai: ["OPENAI_API_KEY"], - anthropic: ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"], - google: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], - xai: ["XAI_API_KEY"], - moonshotai: ["MOONSHOT_API_KEY"], - openrouter: ["OPENROUTER_API_KEY"], -}; - -/** Provider prefixes accepted as aliases for a pi-ai provider id. */ -const PROVIDER_ALIASES: Readonly> = { gemini: "google", moonshot: "moonshotai" }; - -/** - * List the environment variables checked for a provider's API key, in - * precedence order. Returns an empty list for a provider CUA does not document, - * whose credential pi resolves at request time instead. - */ -export function cuaApiKeyEnvVarsForProvider(provider: string): readonly string[] { - return CUA_PROVIDER_API_KEY_ENV_VARS[PROVIDER_ALIASES[provider] ?? provider] ?? []; -} - -/** Read a provider's API key from the environment, or return undefined when unset. */ -export function getCuaEnvApiKey(provider: string): string | undefined { - for (const envVar of cuaApiKeyEnvVarsForProvider(provider)) { - const value = process.env[envVar]; - if (value?.trim()) return value; - } - return undefined; -} - -/** - * Read a provider's API key from the environment, or throw naming the variables - * to set. Throws for a provider CUA documents no variables for — callers that - * accept any pi-ai provider should use {@link cuaApiKeyEnvVarsForProvider} to - * decide whether a preflight is possible at all. - */ -export function requireCuaEnvApiKey(provider: string): string { - const apiKey = getCuaEnvApiKey(provider); - if (apiKey) return apiKey; - const envVars = cuaApiKeyEnvVarsForProvider(provider); - if (envVars.length === 0) { - throw new Error(`No known API key environment variables for provider "${provider}"`); - } - throw new Error(`Missing API key for "${provider}". Set one of: ${envVars.join(", ")}`); -} - -/** {@link getCuaEnvApiKey} keyed by a model ref or concrete model instead of a provider name. */ -export function getCuaEnvApiKeyForModel(input: CuaModelRef | Model): string | undefined { - const provider = typeof input === "string" ? parseCuaModelRef(input).provider : providerForModel(input); - return getCuaEnvApiKey(provider); -} - -/** {@link requireCuaEnvApiKey} keyed by a model ref or concrete model instead of a provider name. */ -export function requireCuaEnvApiKeyForModel(input: CuaModelRef | Model): string { - const provider = typeof input === "string" ? parseCuaModelRef(input).provider : providerForModel(input); - return requireCuaEnvApiKey(provider); -} diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts deleted file mode 100644 index 2988e702..00000000 --- a/packages/ai/src/index.ts +++ /dev/null @@ -1,29 +0,0 @@ -export * from "@earendil-works/pi-ai"; - -export { - createCuaModels, - cuaModels, - GOOGLE_CUA_INTERACTIONS_API, - OPENAI_CUA_COMPUTER_API, - streamGoogleInteractions, - streamOpenAIResponses, - streamSimpleGoogleInteractions, - streamSimpleOpenAIResponses, -} from "./providers"; -export * from "./models"; -export * from "./api-keys"; -export * from "./actions/index"; -export type { - CuaSimpleStreamOptions, - ResponseThreadingOptions, - ResponsesThreadingOptions, -} from "./providers/common"; -export { - normalizeGotoUrl, - responseThreadingDelta, - responseThreadingEnabled, - threadResponsesRequest, -} from "./providers/common"; -export * from "./tool-catalog"; -export * from "./cua"; -export * from "./menu"; diff --git a/packages/ai/test/api-keys.test.ts b/packages/ai/test/api-keys.test.ts deleted file mode 100644 index 770ca168..00000000 --- a/packages/ai/test/api-keys.test.ts +++ /dev/null @@ -1,65 +0,0 @@ -import { afterEach, describe, expect, it } from "vitest"; -import { - cuaApiKeyEnvVarsForProvider, - getCuaEnvApiKey, - getCuaEnvApiKeyForModel, - requireCuaEnvApiKey, -} from "../src/index"; - -const ENV_KEYS = [ - "OPENAI_API_KEY", - "ANTHROPIC_OAUTH_TOKEN", - "ANTHROPIC_API_KEY", - "GOOGLE_API_KEY", - "GEMINI_API_KEY", - "META_API_KEY", - "XAI_API_KEY", - "MOONSHOT_API_KEY", - "OPENROUTER_API_KEY", -] as const; - -const ORIGINAL_ENV = new Map(ENV_KEYS.map((key) => [key, process.env[key]])); - -afterEach(() => { - for (const key of ENV_KEYS) { - const value = ORIGINAL_ENV.get(key); - if (value === undefined) delete process.env[key]; - else process.env[key] = value; - } -}); - -describe("cua api key helpers", () => { - it("maps provider names to expected environment variables", () => { - expect(cuaApiKeyEnvVarsForProvider("openai")).toEqual(["OPENAI_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("google")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("gemini")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("xai")).toEqual(["XAI_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("moonshotai")).toEqual(["MOONSHOT_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("moonshot")).toEqual(["MOONSHOT_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("openrouter")).toEqual(["OPENROUTER_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("unknown")).toEqual([]); - }); - - it("resolves provider api keys with fallback order", () => { - delete process.env.GOOGLE_API_KEY; - process.env.GEMINI_API_KEY = "gemini"; - expect(getCuaEnvApiKey("google")).toBe("gemini"); - process.env.GOOGLE_API_KEY = "google"; - expect(getCuaEnvApiKey("google")).toBe("google"); - }); - - it("resolves keys from model refs", () => { - process.env.OPENAI_API_KEY = "openai"; - expect(getCuaEnvApiKeyForModel("openai:gpt-5.5")).toBe("openai"); - process.env.XAI_API_KEY = "xai"; - expect(getCuaEnvApiKeyForModel("xai:grok-4.5")).toBe("xai"); - process.env.MOONSHOT_API_KEY = "moonshot"; - expect(getCuaEnvApiKeyForModel("moonshotai:kimi-k3")).toBe("moonshot"); - process.env.OPENROUTER_API_KEY = "openrouter"; - expect(getCuaEnvApiKeyForModel("openrouter:moonshotai/kimi-k3")).toBe("openrouter"); - }); - - it("throws readable errors when missing", () => { - delete process.env.META_API_KEY; - }); -}); diff --git a/packages/ai/tsconfig.build.json b/packages/ai/tsconfig.build.json deleted file mode 100644 index 386602f1..00000000 --- a/packages/ai/tsconfig.build.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "extends": "../../tsconfig.base.json", - "compilerOptions": { - "outDir": "./dist-tsc", - "rootDir": "./src", - "emitDeclarationOnly": true, - "sourceMap": false, - "declarationMap": false - }, - "include": ["src/**/*.ts"], - "exclude": ["node_modules", "dist", "**/*.d.ts", "src/**/*.d.ts"] -} diff --git a/packages/ai/tsconfig.json b/packages/ai/tsconfig.json deleted file mode 100644 index d8faaf50..00000000 --- a/packages/ai/tsconfig.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "extends": "./tsconfig.build.json" -} diff --git a/packages/ai/tsdown.config.ts b/packages/ai/tsdown.config.ts deleted file mode 100644 index f377486b..00000000 --- a/packages/ai/tsdown.config.ts +++ /dev/null @@ -1,11 +0,0 @@ -import { defineConfig } from "tsdown"; - -export default defineConfig({ - entry: ["src/index.ts"], - format: ["esm"], - platform: "node", - dts: true, - sourcemap: false, - clean: true, - outExtensions: () => ({ js: ".js", dts: ".d.ts" }), -}); diff --git a/packages/ai/vitest.integration.config.ts b/packages/ai/vitest.integration.config.ts deleted file mode 100644 index 01bc9e51..00000000 --- a/packages/ai/vitest.integration.config.ts +++ /dev/null @@ -1,13 +0,0 @@ -import { defineConfig } from "vitest/config"; - -export default defineConfig({ - server: { - host: "127.0.0.1", - }, - test: { - globals: true, - environment: "node", - testTimeout: 30000, - include: ["test/**/*.integration.test.ts", "test/**/*.live.test.ts"], - }, -}); diff --git a/packages/loop/CHANGELOG.md b/packages/loop/CHANGELOG.md new file mode 100644 index 00000000..4d9f11ea --- /dev/null +++ b/packages/loop/CHANGELOG.md @@ -0,0 +1,885 @@ +# Changelog + +## Unreleased + +- The three packages are one: `@onkernel/loop` replaces `@onkernel/cua-ai`, + `@onkernel/cua-agent`, and `@onkernel/cua-pi-extension`. `.` exports the + framework-neutral core (canonical actions, tool declarations, catalog + compiler, tool menu, tool manager, browser execution), `./pi` exports the pi + binding (`attach()`, model resolution, provider adapters, retry), and the pi + extension is registered from the package's own `pi.extensions` field. +- `Cua*` names are `Loop*`, or plain domain names where the concept is + computer use rather than this product: `CuaAction` is `ComputerUseAction`, + `CuaBrowserAction` is `BrowserAction`, `CuaComputerAction` is + `ComputerAction`. Tool identities move from `cua.*.v1` to `kloop.*.v1`, so a + transcript recorded against an older build no longer resumes. Model-facing + tool names (`browser_snapshot` and friends) are unchanged. +- The package root no longer re-exports `@earendil-works/pi-ai`. Import pi's + types and helpers from pi directly. +- The changelogs of the retired `@onkernel/cua-agent` and + `@onkernel/cua-pi-extension` packages are folded in below; their released + history stays in git. + +Breaking: `CuaAgent` and `CuaAgentHarness` are removed. The package hands back +plain pi objects; the caller constructs the agent. + +- Add `attach({ browser, client })`, returning a handle that compiles + (model, tools) pairs into plain pi objects: `model` carrying the transport its + tools derive, `tools` / `agentTools` materialized against the handle's browser + pool, `models` adding provider retry, required headers, the catalog's payload + transforms and the tool-result image bound, `activate(harness)` for the + behaviors that are pi event handlers rather than constructor options, and + `apply(harness)` to swap a running harness onto a new pair. The handle owns + what actually persists — the Kernel client and browser, the translator, the + raw-CDP executor, ref and frame state — so a spec materializes once across + repeat compiles. +- `getTools()`, `setTools()`, `setModel()`, and `setModelAndTools()` are gone + with the classes. A change compiles a new pair and applies it, so the current + selection belongs to the caller and there is no second copy of it to drift. + `compile()` throws before anything reaches pi, and `apply()` restores the + previous pair if pi rejects the new one, so the atomicity those methods + provided is preserved. `apply()` sets the model only when the derived + transport actually moved, so a tools-only change records no model change. +- The tool manager is immutable: one compiled pair per instance. Removing its + execution scope also removes cache-preserving deferred tool addition, which + only that mutation path produced; the transport still consumes + `addedToolNames` on a transcript that carries them. +- The tool-result image bound, payload transforms, and required headers now + follow whichever pair is active rather than the one a harness was built with. + pi fixes `models` at construction while those are per-catalog, so one + collection per handle is what makes a swap possible at all. +- A model ref absent from a supplied `Models` collection falls back to the + registry, and an id the registry lacks is synthesized. +- `responseThreading` (`LoopAttachOptions`) no longer affects OpenAI models: + OpenAI streams through pi-ai's builtin Responses transport and its automatic + prompt caching regardless of this flag. The option still governs Google's + `previous_response_id`-style continuation. +- Exempt OpenAI's native computer tool from the tool-result image replay limit. + Its `computer_call_output` items must each carry a screenshot, and stateless + replay no longer leaves them in provider-stored state. +- `browser_act` no longer spends a plan's whole deadline waiting for the effect + of a step whose action failed. An unresolvable ref throws immediately, but the + step's `expect` was still awaited afterwards, so a model that invented a ref + burned a full global timeout per attempt instead of being told to snapshot + first. A step whose action never dispatched now skips its own expectation and + stops the plan. Uncertain *delivery* still waits, because a lost + acknowledgement may mean the input landed and the expectation is how that is + discovered. +- Remove `LoopExecutionResources.viewport`. It only fed the removed catalog + viewport option; the same value is still on `resources.browser.viewport`. + +Breaking: the `cua` CLI is removed, and the pi extension replaces it. + +- Everything the CLI built because it needed an agent front-end — sessions and + resume, skills, the TUI, print and RPC modes, model selection — pi supplies. + The `cua act` model-free executor path and the `--print -o jsonl` telemetry + schema are gone with it. +- The extension contributes Kernel browser tools to pi's own agent session. Its + menu is eight entries, one per capability: `browser` and `computer` + (primitives plus their batch form), `browser-act`, `playwright`, and the four + provider-native surfaces — `anthropic-computer`, `anthropic-browser`, + `openai-computer`, `google-browser`. Packaging variants are deliberately + absent: `mixed`, the batch tools on their own, and the 37 individual tool + names offered nothing the eight entries do not. +- Tools are selected with `--browser-tools` and `/browser-tools`, coordinates + with `--browser-coordinates`, and browser configuration is one + `--browser-options` JSON object forwarded verbatim to Kernel's browser-create + call. A flag per create-call field grows every time the SDK does; JSON tracks + it for free. The only default is `timeout_seconds: 600`, and `stealth` is not + forced on. `--browser-session` attaches an existing browser and cannot be + combined with `--browser-options`. +- A selection is validated by compiling it for the active model, so an + incompatible tool deactivates with the catalog compiler's own reason instead + of failing at request time. `/browser-tools` with no argument lists every + selector for the current model with those reasons, deciding each entry's + availability by compiling it on its own and reporting pairwise conflicts + separately. A deactivated selection reports itself on stderr in print and RPC + modes, once per distinct reason, so a scripted run cannot silently lose its + tools and answer from memory with exit 0. +- Provider-native surfaces work because the extension owns the stream for the + providers it registers, swapping pi's registry model for the compiled + catalog's model — which carries the transport the selected tools derive — and + passing the incoming native-call plan. Without that, `requiresApi` never takes + effect and native calls arrive unnormalized. +- One browser is provisioned lazily per session on first tool execution and + deleted on shutdown if this session created it. Declaration compilation, + header generation, and payload transforms never provision a browser. + +- Fix OpenAI's native computer transport rejecting every request after a + screenshot-less action. A `computer_call_output` whose result carried no image + put the failure text in an `error` key, which the Responses API refuses outright + (`400 Unknown parameter: 'input[N].output.error'`), so one failed action poisoned + the rest of the conversation. The output now always carries a valid + `computer_screenshot`, and the failure text follows as a user message so the + model still learns what happened. A 1x1 placeholder is not enough — the + Responses API rejects it even though the vision endpoint accepts one. +- Anthropic's native browser and native computer tools can no longer be selected + together. Anthropic answers 400 because the browser tool addresses a viewport + coordinate frame and the computer tool a display frame; the catalog now refuses + the pair at compile time instead of on the wire. +- Google no longer carries a schema quirk. The Gemini API rejects the JSON Schema + keywords `const` and `additionalProperties` outright rather than ignoring them, + so a payload transform rewrites both for Google — `const: x` becomes a + single-value `enum`, which means the same thing. Gemini now accepts every + function tool CUA offers, including `browser_act` and `browser_wait_for`, which + the removed quirk had marked unavailable. + +- Add `cuaToolMenu(model, selected)`: every tool CUA can offer for a model, each + marked available or not with the compiler's own reason when it is not. It + decides availability by compiling the candidate catalog rather than restating + the compiler's rules, so the menu cannot drift from what + `compileCuaToolCatalog` accepts. Availability is relative to the current + selection, because two providers' native surfaces cannot coexist and a native + surface pins the transport. + +Breaking: the model allowlist is removed. + +- `listCuaModels()` returns pi-ai's whole catalog — 37 providers, ~1,150 models — + instead of a curated subset, and each entry now carries `nativeSurfaces` and + `vision` so callers can render what a model can do. +- `getCuaModel(ref)` resolves any model pi-ai carries, and synthesizes one for + an id the registry has not caught up with, using the sibling that shares the + longest id prefix and preferring the latest such sibling. Providers migrate + transports mid-generation, so a new id follows its nearest, newest relative. + Only an unqualified ref or a provider pi-ai does not carry is refused. +- `CUA_MODEL_ANNOTATIONS`, `CUA_PROVIDERS`, `isCuaProvider`, and the + `CuaProvider` union are gone; `CuaProvider` is now a provider id string and + `cuaProviders()` returns what pi-ai carries. `providerForModel` no longer + throws. +- Two tables replace the allowlist, neither of which decides whether a model may + run: `CUA_NATIVE_SURFACES` (which models have a provider-native computer or + browser tool, with first-party sources) and `CUA_MODEL_QUIRKS` (request-shape + limits, each carrying the documented limit or observed failure that justifies + it). `cuaModelCapabilities` reads the quirk table and defaults to permissive; + `cuaNativeSurfaces(model)` and `cuaModelQuirks(model)` are exported for menus + and diagnostics. + +Breaking: a model's transport is derived from the tools selected with it, rather +than stamped on the model. + +- `compileCuaToolCatalog` derives the compiled model's `api` from the selected + tools' provider bindings: a `CuaProviderBinding` may declare `requiresApi`, + and the returned `catalog.model` carries that transport. Selecting tools whose + bindings require different transports fails to compile with a named catalog + error. This makes transport a function of `(model, selected tools)` instead of + `(model)` alone. +- `getCuaModel("google:...")` no longer forces `google-cua-interactions`. A + Google model resolved without Google's native browser toolset selected now + keeps pi-ai's builtin `google-generative-ai` transport; selecting + `cua.providers.google.toolsets.browser()` still compiles to + `google-cua-interactions` as before. +- Add `OPENAI_CUA_COMPUTER_API` (`"openai-cua-computer"`). A model compiled with + `cua.providers.openai.tools.computer()` selected carries this api, and the + OpenAI provider wrapper dispatches to the CUA adapter on `model.api` alone. + The one remaining request-shape check, `requiresCuaOpenAINamespaceAdapter`, + covers only what cannot be derived from the model: a transcript carrying a + deferred tool-search addition or a replayed function-call namespace, neither + of which pi-ai's builtin transport round-trips. +- Remove `routeCuaApi`. Every provider now takes its transport from pi-ai's + registry or from the selected tools' `requiresApi`, so model resolution + returns pi-ai's data unmodified. Its last remaining branch patched grok-4.5's + thinking-level map, price tiers, and compat flags onto pi-ai's registry entry; + live runs on grok-4.5 at the default, `off`, and `xhigh` thinking levels + behave identically without it. The only lost detail is a >200k-token price + tier that pi's registry does not carry, which affects `usage.cost` reporting + for long requests and nothing else. + +Breaking: OpenAI models no longer carry a CUA-owned api id. + +- OpenAI models resolve to pi-ai's builtin `"openai-responses"` api instead of + the removed `openai-cua-responses`, and stream through pi's builtin Responses + transport (`store: false`, automatic prompt-cache-key matching) by default. + `OPENAI_CUA_RESPONSES_API` and the OpenAI adapter's `previous_response_id` + threading are removed; `previous_response_id` and `store: true` no longer + appear on any OpenAI request. +- OpenAI's native computer adapter now sends the same `prompt_cache_key`, + `prompt_cache_retention`, `prompt_cache_options`, and session-affinity headers + as the function-tool path. It previously relied on stored response state for + context reuse and sent no cache key of its own. +- Remove the xAI Responses fork. It existed only to thread + `previous_response_id` and to set `parallel_tool_calls: false`, which the tool + catalog already emits for the provider. `xai-cua-responses` is gone and Grok + streams through pi's builtin xAI provider; `XAI_CUA_RESPONSES_API`, + `streamXaiResponses`, and `streamSimpleXaiResponses` are no longer exported. +- Google keeps its continuation protocol: it threads a provider-specific field + with no builtin equivalent, and the shared helpers in `providers/common.ts` + are unchanged for it. + +Breaking: the Tzafon, Yutori, and Meta providers are removed. + +- Remove the `tzafon` and `yutori` providers: their model annotations and + overrides, `cua.providers.tzafon`, `cua.providers.yutori`, + `TZAFON_API_KEY`/`YUTORI_API_KEY`, and the exported `TZAFON_RESPONSES_API`, + `YUTORI_CHAT_COMPLETIONS_API`, `streamTzafonResponses`, + `streamSimpleTzafonResponses`, `streamYutori`, and `streamSimpleYutori` stream + functions. `createCuaModels()` no longer registers either provider, and refs + like `tzafon:tzafon.northstar-cua-fast` or `yutori:n1.5-latest` now fail to + resolve. Drops the `@tzafon/lightcone` dependency. +- Remove the `meta` provider. pi-ai ships no `meta` provider, so cua hand-wrote + a model entry and pointed pi's own Responses transport at `api.meta.ai`. + `META_API_KEY`, `META_RESPONSES_API`, `streamMetaResponses`, and + `streamSimpleMetaResponses` go with it. Meta was the last user of the + model-override mechanism, so `CUA_MODEL_OVERRIDES` and `cuaOverrideModels()` + are gone too, and `getCuaModel` no longer has a "supported but not registered" + fallback. Muse Spark remains available through pi's OpenRouter catalog as + `openrouter:meta/muse-spark-1.1`, annotated with explicit capabilities because + OpenRouter's provider-level defaults are conservative. +- `CuaProviderBinding` loses its `tzafon-native` and `yutori-native` variants, + and `CuaIncomingToolPlan` loses `tzafonComputerName` and `yutoriNames`. The + Yutori-only rule rejecting a partial n1 native action set is gone with them; + every surviving native toolset can be selected in part. +- `CompileCuaToolCatalogOptions.viewport` is removed. It existed only to fill + Tzafon's `display_width`/`display_height` declaration defaults, and no + surviving declaration reads it. + +## 0.10.0 - 2026-08-04 + +Breaking: upgrade `@earendil-works/pi-ai` to 0.83.0. + +- Remove the local `claude-opus-5`, `gemini-3.6-flash`, and + `gemini-3.5-flash-lite` model overrides: pi-ai 0.83.0's registry now carries + all three with the same metadata. Overrides remain only for the CUA-only + providers pi-ai does not ship (Meta, Tzafon, Yutori). +- Kimi K3 reasoning effort follows pi-ai's catalog metadata with no CUA + override: `low`/`high`/`max` map through `thinkingLevelMap` (the rest clamp + away), and requests carry `reasoning_effort` on Moonshot or OpenRouter's + nested `reasoning.effort`. +- Disable Tzafon native non-screenshot action loops: its Responses API requires + every `computer_call_output` to carry an image, while CUA returns screenshots + only when explicitly requested. Unsupported native actions now fail before + browser execution instead of entering a text-only loop that the API rejects. + +## 0.9.0 - 2026-08-03 + +- Add Kimi K3 through OpenRouter as `openrouter:moonshotai/kimi-k3`, + authenticated with `OPENROUTER_API_KEY`, while retaining direct Moonshot access + through `moonshotai:kimi-k3`. +- Share Kimi K3's CUA capability metadata across both transports: nested browser + schemas are accepted, the larger `browser_act` schema is rejected, and + state-mutating function tools disable parallel calls. +- Resolve schema and tool-call compatibility from concrete model capabilities + instead of provider-wide allowlists. Existing provider-family defaults are + preserved, and provider-native tools remain restricted to their native + transports. + +## 0.8.0 - 2026-07-31 + +Breaking: agent tools are now one explicit, identity-keyed catalog. The mode, +implicit-tool, and runtime-spec APIs are removed. + +- Add the frozen `cua` namespace with atomic browser/computer/Playwright tools, + `browser`/`computer`/`mixed` convenience toolsets, mechanical batch tools, + coordinate contracts, and provider-native tools/toolsets. +- Add `compileCuaToolCatalog()` with stable identities; exact requested-order + preservation; schema/catalog fingerprints; exact and + provider-normalized collision checks; model compatibility checks; inspectable + declarations; dynamic-loading eligibility; generated header composition; + ordered payload transforms; and incoming native-call plans. +- Catalog compilation is declaration-only and deterministic: it accepts CUA + specs and sanitized caller `Tool` declarations plus a viewport, returns + pi-ai `Tool` declarations (`catalog.toolDeclarations`) and provider plans, + and never constructs executable tools or retains the requested inputs. + `callerToolIdentity()` is the single canonical identity scheme for caller + tools, shared with cua-agent and cua-cli. The package no longer depends on + `@earendil-works/pi-agent-core`; materialization and implementation identity + live in `@onkernel/cua-agent`. +- Remove `resolveCuaRuntimeSpec`, `CuaRuntimeSpec`, `CuaMode`, mode inference, + legacy native-tool switches, implicit navigation tools, and provider-owned + default prompt selection from the public API. +- Provider-native declarations now compose by selected identity with ordinary + functions. Add fixed-version Anthropic computer/browser factories, OpenAI + native computer composition, Tzafon viewport-aware declaration replacement, + Google's current predefined browser toolset, and identity-scoped Yutori + native selection. Every provider surface exposes its first-party source. +- Add deterministic provider composition: generated model preparation, tool + serialization, provider fields, then caller payload hooks. + Header requirements merge without overwriting unrelated caller headers. +- Add a Google Interactions API adapter plus current `computer_use` browser + action names and `[0, 999]` coordinates. Exact-subset declarations exclude + every unselected current action, and excluded incoming calls fail with a + named catalog error. Support the documented `gemini-3.6-flash`, + `gemini-3.5-flash`, and `gemini-3.5-flash-lite` models. +- Add `anthropic:claude-opus-5` with its 1M-token context window, 128k output + limit, adaptive thinking levels, and July 2026 native-tool compatibility. + Native `browser_20260701` transparently retries through an equivalent + function-tool declaration when the active credential lacks beta access. +- Add the verified `openai:gpt-5.6-sol` model. +- Expose Google's predefined actions through `browser()` only. Remove legacy + Google actions and the Meta/xAI/Moonshot coordinate toolsets; those + custom-function providers use the standard CUA browser toolset. +- Validate large function schemas separately from ordinary nested schemas. + Moonshot retains CUA browser primitives and `browser_wait_for`, but catalog + compilation now rejects `browser_act`, whose larger schema its API refuses. +- Native tool execution metadata carries stop-on-first-failure policy without + introducing provider branches in cua-agent. +- Declare `engines.node` `>=22.19.0`. This is not a new requirement: every + `@earendil-works/pi-*` dependency already declares the same floor, so it was + previously enforced only transitively and never stated on this package. + +## 0.7.0 - 2026-07-17 + +- Added Moonshot Kimi K3 computer-use support: `moonshotai:kimi-k3` + (`moonshot:` accepted as a ref alias), authenticated with + `MOONSHOT_API_KEY`. Kimi streams through the OpenAI-compatible chat + completions transport with ordinary function tools. +- New `moonshot` provider namespace following the standard conventions + (`computerTools`, `coordinateSystem`, `buildMoonshotSystemPrompt`, + `providerModule`, …). Kimi's coordinate contract is normalized 0–1 + width/height fractions, matching the model's native visual grounding; + payload middleware disables parallel tool calls. +- Bumped `@earendil-works/pi-ai` to 0.80.10 (carries the Kimi K3 registry + entry). Grok 4.5's registry api id is now `openai-responses`; CUA routing + behavior is unchanged. + +## 0.6.0 - 2026-07-10 + +Breaking: response threading is now configured per request instead of through +`CUA_DISABLE_RESPONSE_THREADING`. + +- `CuaSimpleStreamOptions` now includes `disableResponseThreading`, allowing + OpenAI and Tzafon Responses API calls to send the complete current context + instead of continuing through `previous_response_id`. +- Removed the process-wide `CUA_DISABLE_RESPONSE_THREADING` environment + variable. `@onkernel/cua-agent` users can set `responseThreading: false` on + `CuaAgent` or `CuaAgentHarness`; lower-level callers can pass + `disableResponseThreading: true` in stream options. + +## 0.5.0 - 2026-07-09 + +Introduces action planes (modes) and Anthropic native computer-use tools. + +- New `mode` option (`"computer"` | `"browser"` | `"hybrid"`, exported as + `CuaMode`) on `resolveCuaRuntimeSpec`, `computerTools`, and the executor + builders selects which action plane(s) the model sees. `computer` is the + pre-modes default and stays byte-compatible. `browser` exposes the new + browser-plane canonical actions; `hybrid` exposes both planes deduplicated + to one tool per capability, with browser actions restricted to element refs + so the OS screenshot is the single coordinate frame. +- New browser-plane canonical actions (`CUA_BROWSER_ACTION_TYPES`): + `browser_snapshot`, `browser_find`, `browser_text`, `browser_click`, + `browser_fill`, `browser_scroll_to`, `browser_navigate`, + `browser_list_tabs`, `browser_new_tab`, `browser_screenshot`, + `browser_evaluate`, and friends, with per-mode tool naming + (`cuaToolNameForAction`), descriptions, schemas, and system prompts. +- New `nativeTool` option drives Anthropic models through their native + computer-use declarations: `computer_20260701` (computer mode, with + `enable_zoom`) behind `anthropic-beta: computer-use-2026-07-01`, and + `browser_20260701` (browser mode) behind + `anthropic-beta: browser-use-2026-07-01`. +- JavaScript execution is on by default: `browser_evaluate` is part of the + default browser/hybrid action sets, and native `browser_20260701` + declarations default `enable_javascript_exec` to true (an explicit value on + the spec wins). Opt out by passing an explicit `actions` list or native + tool spec. +- New `zoom` computer action (cropped display inspection). + +## 0.4.0 - 2026-07-07 + +Breaking: adopts pi-ai 0.80's instance-based `Models` API and drops the +global api-registry surface. + +- Requests now stream through a pi `Models` collection. New exports: + `createCuaModels(options?)` builds a collection with pi's builtin providers + plus CUA's adjustments (OpenAI routed through `openai-cua-responses`, + Google accepting `GOOGLE_API_KEY` or `GEMINI_API_KEY`, Tzafon and Yutori + registered); `cuaModels()` returns the shared default collection. +- Removed `registerCuaProviders()` and the import-time registration side + effect. Build a collection with `createCuaModels()` instead; nothing global + is mutated. +- The pi-ai re-export now follows pi-ai 0.80: the free functions `complete`, + `stream`, `completeSimple`, `streamSimple`, `getModel`, `getModels`, and + the api-registry (`registerApiProvider`, `getApiProvider`, + `resetApiProviders`, …) are gone. Call the equivalent methods on + `cuaModels()`. +- API keys resolve from the documented env-var convention through provider + auth when streaming via the collection; explicit `apiKey` stream options + still take precedence. +- Removed the `claude-sonnet-5` and `gpt-5.5` model overrides (pi-ai 0.80's + registry carries both) and the `gpt-5.5-2026-04-23` dated-snapshot + override. Dated snapshot refs no longer resolve — use the family id + (`openai:gpt-5.5`). +- Updated `@earendil-works/pi-ai` to 0.80.3. + +## 0.3.4 - 2026-06-30 + +- Adapt newer Anthropic models to the adaptive thinking payload format, including `claude-sonnet-5`, `claude-opus-4-8`, and `claude-opus-4-7`. + +## 0.3.3 - 2026-06-30 + +- Add computer-use support for the `claude-sonnet-5` Anthropic model. + +## 0.3.2 - 2026-06-24 + +- Add computer-use support for the `gemini-3.5-flash` Google model. + +## 0.3.1 - 2026-06-23 + +- Add the `playwright_execute` tool definition: `CuaPlaywrightSchema`, + `CUA_PLAYWRIGHT_TOOL_NAME`, `CUA_PLAYWRIGHT_TOOL_DESCRIPTION`, + `createCuaPlaywrightToolDefinition()`, and the `CuaPlaywrightInput` type. + +## 0.3.0 - 2026-06-12 + +- Add `CuaSimpleStreamOptions`: pi-ai `SimpleStreamOptions` plus the + `keepToolNames` extension the Yutori/Tzafon stream adapters consume, so + callers can pass it through `streamSimple` without a cast. + +## 0.2.2 - 2026-06-11 + +- Add computer-use support for `gpt-5.4-mini`, `gemini-3.1-flash-lite`, `tzafon.northstar-cua-fast-1.6`, and `tzafon.northstar-cua-fast-1.7-experiment`. +- Drop `gemini-3-pro-preview`, which Google has retired (the API now returns 404 for it). + +## 0.2.1 - 2026-06-11 + +- Add computer-use support for the `claude-fable-5` Anthropic model. + +## 0.2.0 - 2026-06-10 + +### Fixed + +- The published package is now importable under plain Node ESM. 0.1.0 shipped + extensionless relative imports in `dist/`, so `import "@onkernel/cua-ai"` + failed outside bundlers; `dist/` is now bundled with tsdown. +- The shipped `examples/quickstart.ts` imports `@onkernel/cua-ai` instead of a + `../src` path that does not exist in the tarball, checks `stopReason` so + provider errors are no longer silent, resolves its API key via + `requireCuaEnvApiKeyForModel`, and switches providers with the `CUA_MODEL` + env var. +- `docs/` (the supported-models list the README links to) is now included in + the npm tarball. +- A malformed Yutori tool call now degrades to an empty-arguments call instead + of failing the entire response, matching the existing Tzafon hardening. + +### Breaking changes + +- Provider namespaces follow one convention. Every namespace now exports + `computerTools({ actions? })` / `computerToolExecutors({ actions? })`, + `createActionSchema`, `coordinateSystem()`, `providerModule`, + `_CUA_ACTION_TYPES`, `_COMPUTER_INSTRUCTIONS`, a + `Action` type, and `ComputerToolsOptions`. This replaces 0.1.0's + `createComputerToolDefinitions(options)` / + `CreateComputerToolDefinitionsOptions`, the per-namespace + `COMPUTER_TOOL_COORDINATES` constants, `TZAFON_ACTION_TYPES` / + `YUTORI_ACTION_TYPES`, and the `OPENAI_BATCH_INSTRUCTIONS` / + `GEMINI_INSTRUCTIONS_RAW` / `TZAFON_INSTRUCTIONS_RAW` / + `YUTORI_INSTRUCTIONS_RAW` prompt constants. +- `CUA_BATCH_TOOL_NAME` is now `"computer_batch"` (was + `"batch_computer_actions"`), matching the batch tool Anthropic ships by + default. `anthropic.ANTHROPIC_BATCH_TOOL_NAME` carries the same new value; + the other per-namespace batch aliases (`TZAFON_BATCH_TOOL_NAME`, + `YUTORI_BATCH_TOOL_NAME`, `*_BATCH_DESCRIPTION`, `*BatchSchema`, + `*BatchInput`) were removed — use `CUA_BATCH_TOOL_NAME`, + `CUA_BATCH_TOOL_DESCRIPTION`, `CuaBatchSchema`, and `CuaBatchInput`. +- Anthropic tools are now the 13 canonical browser actions Anthropic supports + (no `back`/`forward`/`url`) plus a `computer_batch` batch tool by default; + pass `excludeBatch: true` to omit it. Unsupported `actions` entries throw. + `anthropic.ANTHROPIC_CUA_ACTION_TYPES` reflects the supported subset rather + than aliasing the full canonical list. +- Yutori models now use Yutori's documented native `tool_set` request field. + `streamYutori` strips canonical action tools from the outbound payload + (preserve specific tools via the `keepToolNames` stream option), selects the + n1.5 core tool set where applicable, and normalizes native tool calls back + to canonical names. `yutori.providerModule.toolDefinitions()` is `[]`; + `yutori.computerTools()` builds local mirrors for executor lookup, validates + `{ actions }` against the supported subset, and throws on unsupported + actions. `yutoriBuiltinToolsOnPayload` was replaced by + `yutoriNativeToolSetOnPayload`. The Yutori runtime spec also carries a + screenshot policy (append a 1280x800 webp screenshot to the latest message). +- Family model annotations now match only the family root plus numeric + revision or dated-snapshot suffixes (`claude-opus-4-7`, + `gpt-5.5-2026-04-23`). Named sibling variants such as `gpt-5.4-mini` are no + longer listed by `listCuaModels()` or accepted by `getCuaModel()` without + their own annotation. +- `google:gemini-2.5-computer-use-preview-10-2025` was removed from the + catalog: it rejects the standard function declarations this package sends + and requires Google's native `tools.computer_use` wrapper. Use + `google:gemini-3-flash-preview` or `google:gemini-3-pro-preview`. +- `streamTzafonResponses` no longer accepts a `maxOutputTokens` option — use + the standard `maxTokens` stream option. + +### Added + +- `CuaProviderModule` contract plus a `providerModule` export per namespace, + and a richer `CuaRuntimeSpec`: `toolExecutors` (local adapters that turn + provider tool calls into canonical `CuaAction`s via `CuaToolExecutorSpec`), + `coordinateSystem`, and optional `screenshot` policy alongside the existing + tool definitions, default prompt, and payload middleware. +- `resolveCuaRuntimeSpec(input, options?)` accepts `ComputerToolsOptions` and + forwards it to the provider module, so runtime consumers can narrow tool + definitions and executors (e.g. `{ actions: ["click"] }`). +- `registerCuaProviders()` is exported: importing the package still registers + the Yutori/Tzafon stream providers automatically, and this restores them + after pi-ai registry mutators (`clearApiProviders`, `resetApiProviders`, + `unregisterApiProviders`). +- `parseCuaModelRef` / `getCuaModel` accept `"gemini:"` refs as an alias for + `"google:"`, and unsupported-provider errors now list the valid providers. +- `CuaMouseButton` and `CuaDragMouseButton` closed unions type the `button` + field on click/mouse_down/mouse_up and drag actions. +- `yutori.YutoriOptions` and `tzafon.TzafonResponsesOptions` are exported and + aligned; both support `keepToolNames` to preserve caller tools that collide + with canonical action names on the wire. +- Yutori native action vocabulary exports: `YUTORI_N1_ACTION_TYPES`, + `YUTORI_N15_CORE_ACTION_TYPES`, `YUTORI_N15_EXPANDED_ACTION_TYPES`, + tool-set ids, `yutoriToolSetForModel`, `yutoriNativeActionsForModel`, and + `toCanonicalActions`; Tzafon exports `toCanonicalActions`, + `TzafonCanonicalAction`, `tzafonComputerUseOnPayload`, and + `tzafonToolCallId`. +- README and JSDoc coverage across the public surface: API key prerequisites + and helpers, error handling (`stopReason` semantics), a multi-turn + tool-result example, the complete export list, and per-provider canonical + action subsets. + +## 0.1.0 + +- Provider-qualified CUA model catalog with support annotations and curated overrides. +- Unified runtime-spec resolution for provider defaults (tools, prompts, payload middleware). +- Registers CUA provider adapters and exports canonical computer-use schemas/tool definitions. + +## Pre-collapse history: @onkernel/cua-agent + +### Unreleased + +- `browser_act` no longer spends a plan's whole deadline waiting for the effect of + a step whose action failed. An unresolvable ref throws immediately, but the + step's `expect` was still awaited afterwards, so a model that invented a ref + burned a full global timeout per attempt instead of being told to snapshot + first. A step whose action never dispatched now skips its own expectation and + stops the plan. Uncertain *delivery* still waits, because a lost acknowledgement + may mean the input landed and the expectation is how that is discovered. + +Breaking: `CuaAgent` and `CuaAgentHarness` are removed. cua-agent hands back +plain pi objects; the caller constructs the agent. + +- Add `attach({ browser, client })`, returning a handle that compiles + (model, tools) pairs into plain pi objects: `model` carrying the transport its + tools derive, `tools` / `agentTools` materialized against the handle's browser + pool, `models` adding provider retry, required headers, the catalog's payload + transforms and the tool-result image bound, `activate(harness)` for the + behaviors that are pi event handlers rather than constructor options, and + `apply(harness)` to swap a running harness onto a new pair. The handle owns + what actually persists — the Kernel client and browser, the translator, the + raw-CDP executor, ref and frame state — so a spec materializes once across + repeat compiles. +- `getTools()`, `setTools()`, `setModel()`, and `setModelAndTools()` are gone + with the classes. A change compiles a new pair and applies it, so the current + selection belongs to the caller and there is no second copy of it to drift. + `compile()` throws before anything reaches pi, and `apply()` restores the + previous pair if pi rejects the new one, so the atomicity those methods + provided is preserved. `apply()` sets the model only when the derived + transport actually moved, so a tools-only change records no model change. +- `CuaToolManager` is now immutable: one compiled pair per instance, with + `prepareTools`/`prepareModel`/`prepareModelAndTools`/`commit`/`getTools` and + the async-local execution scope removed. Removing the execution scope also + removes cache-preserving deferred tool addition: a tool that added tools mid + execution used to have those names recorded on its result as + `addedToolNames`, letting pi extend an OpenAI request without invalidating the + prompt-cache prefix. Nothing produced them outside that mutation path. The + transport still consumes `addedToolNames` on a transcript that carries them. +- The tool-result image bound, payload transforms, and required headers now + follow whichever pair is active rather than the one a harness was built with. + pi fixes `models` at construction while those are per-catalog, so one + collection per handle is what makes a swap possible at all. +- A model ref absent from a supplied `Models` collection falls back to the + registry, and an id the registry lacks is synthesized. +- The model streamed for a Google model depends on which tools it was compiled + with: selecting Google's native browser toolset compiles to the CUA-owned + Interactions API, while a Google model selected with only CDP browser tools + streams through pi's builtin Google transport. +- `responseThreading` (`CuaAttachOptions`) no longer affects OpenAI models: + OpenAI streams through pi-ai's builtin Responses transport and its automatic + prompt caching regardless of this flag. The option still governs Google's + `previous_response_id`-style continuation. +- Exempt OpenAI's native computer tool from the tool-result image replay limit. + Its `computer_call_output` items must each carry a screenshot, and stateless + replay no longer leaves them in provider-stored state. + +Breaking: Tzafon and Yutori support is removed. + +- Compiling a Tzafon or Yutori model ref now fails to resolve the model, and + `cua.providers.tzafon` / `cua.providers.yutori` no longer exist. +- Remove `CuaExecutionResources.viewport`. It only fed the removed catalog + viewport option; the same value is still on `resources.browser.viewport`. + +### 0.10.0 - 2026-08-04 + +Breaking: upgrade `@earendil-works/pi-agent-core` and `@earendil-works/pi-ai` +to 0.83.0 and adopt pi's context-first harness API. + +- `CuaAgentHarness` and `CuaAgentHarnessOptions` now take the tool context as + their first type parameter — `CuaAgentHarness` — mirroring pi's `AgentHarness` generic order. The + supplied `toolContext` is forwarded to pi untouched, and every executable + harness tool receives the exact object on each call. +- Executable harness tools are pi `AgentHarnessTool`s via the new + `CuaHarnessTool` union (a CUA spec or an `AgentHarnessTool`). + `CuaAgent` stays on the ordinary pi `AgentTool` (`CuaAgentTool`); the two + tool APIs are no longer conflated. +- Remove `CuaAgentHarnessOptions.env` and `CuaAgentHarness.env`. Execution + environments now travel through the tool context (for example + `toolContext: { env: new NodeExecutionEnv({ cwd }) }` for pi's + read/bash/edit/write tools). No alias is preserved. +- Remove `CuaSystemPromptCallback`; `systemPrompt` is pi's + `AgentHarnessSystemPrompt` through the harness options. +- Keep `streamFn` optional on `CuaAgentOptions` (CUA supplies its default + stream) even though pi 0.83.0 makes `AgentOptions.streamFn` required. +- Published declarations target pi's TypeBox 1.3 as-is; a downstream compile + test with `skipLibCheck: false` guards the packaged types. +- Preserve explicit Tzafon screenshot results in model context even when they + fall outside `toolResultImageReplayLimit`, because its native continuation + protocol requires those images. Other tool-result images remain bounded. + +### 0.9.0 - 2026-08-03 + +- Add OpenRouter Kimi K3 support through `@onkernel/cua-ai` 0.9.0, + including the browser-primitives-only example catalog used by the provider + matrix. +- Resolve `CuaAgentHarness` string model references against its supplied + `Models` collection during construction and `setModel()`, while preserving + the curated CUA model gate and fallback support for CUA model overrides. +- Keep `CuaAgent` aligned with pi's low-level `Agent`: callers can pass a + concrete OpenRouter model and inject `models.streamSimple` without adding a + `Models` dependency to the agent API. + +### 0.8.0 - 2026-07-31 + +Breaking: `CuaAgent` and `CuaAgentHarness` now require one exact `tools` list and +use composition instead of inheriting from pi's `Agent`/`AgentHarness`. + +- Add `getTools()` and atomic `setTools()`. Model changes recompile and + revalidate the full requested catalog. Empty catalogs are valid; + no tools or system-prompt text are inferred or appended. Catalog changes from + inside a tool require sequential execution, including model changes. +- Remove `mode`, `nativeTool`, `extraTools`, `playwright`, `setMode()` / + `getMode()`, and implicit `computer_use_extra` behavior. +- Add one shared `CuaExecutionResources` pool per agent/harness. Catalog and + model changes preserve the canonical translator, lazy raw-CDP browser + executor, refs, tabs, screenshots, and Playwright capability. +- Define and export `CuaAgentTool` here (moved out of cua-ai, which now + compiles declaration-only catalogs). cua-agent owns all `AgentTool` + materialization — each CUA spec is materialized exactly once per shared + execution-resource pool — and owns implementation identity for + cache-preserving deferred-tool decisions: a reused `execute` function keeps + its identity across wrappers, a new `execute` or freshly created spec object + is a conservative replacement, and the same objects stay stable across model + recompilation. +- Integrate pi 0.80.10 dynamic tool loading. Eligible additions made from inside + a running tool emit `addedToolNames`; outside-tool additions and all + provider-native changes are eager. Schema/executor replacements are treated + as real changes, not name-only no-ops. +- Refactor atomic tools to operation-specific argument objects while preserving + the existing `browser_act` schema. Export `formatBrowserActResult()` so direct + application surfaces can render the same bounded plan feedback as agents. +- Add mechanical `computer_batch` and `browser_batch` execution. Computer writes + coalesce across write-only runs and flush around reads; browser actions run + sequentially against shared ref state. Failure details include the failed + action index, completed reads, and skipped count. +- Return screenshots only for explicit screenshot or zoom actions. Ordinary + writes return status text, semantic tools return structured feedback, and + failed batches replace images from earlier explicit screenshot steps with + textual markers. +- Native multi-action turns stop after the first failed tool call. Every + remaining call in that assistant turn receives the configured error result + instead of executing against stale browser state. +- Update shared examples to use the same browser-oriented provider catalogs as + the CLI: explicit `browser_act` plans where the provider accepts the schema, + browser primitives alone for Moonshot, and Anthropic native-browser selection + with model fallback. +- Security: require `sharp` `^0.35.3` (was `^0.34.5`) to pick up the libvips + fixes for GHSA-f88m-g3jw-g9cj (CVE-2026-33327, CVE-2026-33328, CVE-2026-35590, + CVE-2026-35591). `sharp` decodes cloud-browser screenshots inside the + translator's `zoom()`, so this is the one advisory in this release that + touched attacker-influenced bytes. The APIs this package uses are unchanged by + sharp 0.35, and no source changes were needed. Two packaging notes for + installers: sharp 0.35 no longer ships an `install` lifecycle script, and it + no longer falls back to building from source — installing with + `--omit=optional`, or on a platform with no prebuilt `@img/sharp-*` binary, + now fails at import instead of silently compiling. sharp 0.35 requires Node + `>=20.9.0`, well below this package's floor. +- Declare `engines.node` `>=22.19.0`. This is not a new requirement: every + `@earendil-works/pi-*` dependency already declares the same floor, so it was + previously enforced only transitively and never stated on this package. + +### 0.7.0 - 2026-07-17 + +- `CuaAgent` and `CuaAgentHarness` support Moonshot Kimi K3 + (`moonshotai:kimi-k3`) via `@onkernel/cua-ai` 0.7.0, resolving auth from + `MOONSHOT_API_KEY`. Kimi's fractional coordinates are scaled to viewport + pixels by the existing translator. +- Bumped `@earendil-works/pi-ai` and `@earendil-works/pi-agent-core` to + 0.80.10; the wrapped `Models` collections forward the new + `checkAuth`/`getAvailable`/`login`/`logout` methods. + +### 0.6.0 - 2026-07-10 + +Adds explicit request-recovery and context-management policies while keeping +provider retries and exact-empty recovery disabled by default. + +- `retry` adds opt-in transient provider-request retries to `CuaAgent` and + `CuaAgentHarness`, with configurable attempt and backoff limits. Failed + partial streams are buffered and discarded before a clean retry is exposed. +- `toolResultImageReplayLimit` limits each model request to the newest four + tool-result images by default. It operates on a request-time projection and + leaves agent state and persisted sessions unchanged. Harness context hooks + settle before this limit is applied at the `Models` boundary. +- `responseThreading` replaces the process-wide environment switch with a + constructor option for OpenAI and Tzafon `previous_response_id` chaining. +- `emptyResponseRecovery` optionally follows a successful exact-empty response + with a bounded, caller-supplied pi `followUp()` message. Omitting it preserves + pi's normal completion behavior. +- Updated `@onkernel/cua-ai` to 0.6.0. + +### 0.5.0 - 2026-07-09 + +Adds the browser action plane and runtime mode switching. Breaking: the +`computerUseExtra` option is removed — the `computer_use_extra` navigation +helper is always registered. + +- New `BrowserExecutor`: drives the browser plane over CDP. Accessibility + snapshots with element refs (`[e12]`), node states + (checked/expanded/disabled/value/…), and cursor:pointer clickable hints + for elements with no interactive ARIA role; iframe and OOPIF stitching + with per-frame session-aware refs; StaticText dedupe and wrapper + collapsing; an unchanged-snapshot short-circuit; lexical `find`, `fill`, + CDP navigation and tab management; and a JavaScript dialog guard. Refs invalidate on real + navigations (`Page.frameNavigated`), self-heal via (role, name, nth) when + the page changes but the element is still unambiguous, and the ref table is + bounded (per-target cap, generation sweeps). `exportRefState()` / + `importRefState()` persist refs across processes against the same browser. +- `CuaAgent` and `CuaAgentHarness` accept `mode` (`"computer"` | `"browser"` + | `"hybrid"`) and `nativeTool`, and support runtime plane switching via + `setMode()` / `getMode()`. Mode switches preserve the requested activation + state of surviving tools and keep the translator — CDP connection, tabs, + and element refs — alive; the translator is only rebuilt when a model + switch changes the provider's coordinate system or screenshot transform. + Both switches roll back cleanly on failure. +- Post-action grounding captures and the navigation helper are mode-aware: + browser mode grounds on the viewport and routes navigation through CDP + (browser and hybrid modes both route `computer_use_extra` navigation over + the browser plane so refs invalidate correctly). +- Updated `@onkernel/cua-ai` to 0.5.0. + +### 0.4.0 - 2026-07-07 + +Breaking: follows pi-agent-core 0.80's `Models`-based harness. + +- `CuaAgentHarness` accepts an optional `models` (a pi `Models` collection) + and defaults to `cuaModels()` from `@onkernel/cua-ai`. The + `getApiKeyAndHeaders` option is gone — pi-agent-core 0.80 resolves auth + through provider auth on the collection; pass a custom `models` to override + resolution (e.g. in tests). +- `CuaAgent`'s default stream path is `cuaModels().streamSimple` instead of + pi-ai's removed global `streamSimple`. Custom `streamFn` options work + unchanged. +- Updated `@earendil-works/pi-agent-core` and `@earendil-works/pi-ai` to + 0.80.3 and `@onkernel/cua-ai` to 0.4.0. + +### 0.3.5 - 2026-06-24 + +- Update the `@onkernel/cua-ai` dependency to 0.3.2, adding computer-use + support for the `gemini-3.5-flash` Google model. + +### 0.3.4 - 2026-06-23 + +- Add an opt-in `playwright` option to `CuaAgent` and `CuaAgentHarness` that + exposes a `playwright_execute` tool, running Playwright/TypeScript against + the live browser session via the Kernel SDK. Results, stdout, and stderr + come back as tool content; SDK-reported failures surface as content rather + than throwing. Adds the `PlaywrightDetails` export. + +### 0.3.3 - 2026-06-12 + +- The action translator now consumes the canonical `CuaAction` union with an + exhaustive switch. Malformed action shapes fail loudly instead of silently + coercing (previously e.g. a click at 0,0); the documented mouse-button + coercion to `"left"` is unchanged. +- `prepareNextTurn` no longer rebuilds the turn context on every turn: it + keeps stock pi behavior until a user hook returns an update or a mid-run + model assignment requires a refresh. +- One translator instance per runtime is shared between the executor tools + and the provider screenshot capability. +- The `CuaAgentHarness` README quickstart showcases session-backed turns and + mid-session model switching; `computerUseExtra` is documented with its + rationale. +- Update the `@onkernel/cua-ai` dependency to 0.3.0. + +### 0.3.2 - 2026-06-11 + +- Update the `@onkernel/cua-ai` dependency to 0.2.2. + +### 0.3.1 - 2026-06-11 + +- Update the `@onkernel/cua-ai` dependency to 0.2.1. + +### 0.3.0 - 2026-06-10 + +- Replaces the vendored pi-agent-core snapshot with the released `@earendil-works/pi-agent-core@0.79.1` dependency. The full pi surface is still re-exported, but it now tracks the published package instead of a frozen fork. +- BREAKING: `harness.agent` is removed. It only existed in the vendored pre-release snapshot and never shipped in any pi-agent-core release; use `getModel()`, `getTools()`, and `getActiveTools()` instead. +- BREAKING: `steer()`, `followUp()`, `nextTurn()`, and `setStreamOptions()` on the harness now return promises and must be awaited. +- BREAKING: the harness `model_select` and `thinking_level_select` events are renamed `model_update` and `thinking_level_update`, and the `steeringMode`/`followUpMode` property accessors became `getSteeringMode()`/`setSteeringMode()`/`getFollowUpMode()`/`setFollowUpMode()` methods. +- BREAKING: `ExecutionEnv` is now `Result`-based. Custom env implementations return `Result` values instead of throwing. +- BREAKING: requires Node.js >= 22.19.0. +- `NodeExecutionEnv` now comes from `@earendil-works/pi-agent-core`'s `/node` subpath; importing it from `@onkernel/cua-agent` keeps working. +- Tool execution follows pi's throw-on-failure contract: failed browser actions throw an error labeled with the action instead of also encoding the failure into tool result content and details. +- Moves the yutori screenshot payload append into `@onkernel/cua-ai`'s payload middleware. +- Built ESM output uses explicit `.js` relative import specifiers so `dist` resolves under plain Node.js. + +### 0.2.0 - 2026-05-13 + +- Adds `CuaAgentHarness`, a provider-aware harness API with session-backed turns, resource and prompt helpers, active tool selection, and model switching. +- Keeps CUA runtime defaults in sync when changing models so provider-specific tools, prompts, and payload middleware update together. +- Improves browser keyboard shortcut translation for Kernel computer actions. + +### 0.1.0 + +- Class-first CUA runtime: `CuaAgent` and `CuaHarness` on top of pi-agent-core. +- Provider-neutral browser tool executors for canonical CUA tool names, backed by Kernel browser actions. +- Includes examples plus unit and live e2e coverage for common provider/model combinations. + +## Pre-collapse history: @onkernel/cua-pi-extension + +### Unreleased + +- Flags name the domain rather than an acronym: `--cua-tools` is now + `--browser-tools`, `--cua-coordinates` is `--browser-coordinates`, and the + commands are `/browser` and `/browser-tools`. +- Browser configuration is one `--browser-options` JSON object forwarded verbatim + to Kernel's browser-create call, replacing `--cua-profile-id`, + `--cua-profile-save-changes`, `--cua-proxy-id`, and `--cua-browser-timeout`. A + flag per create-call field grows every time the SDK does; JSON tracks it for + free. The only default is `timeout_seconds: 600`. `--browser-session` still + attaches an existing browser and cannot be combined with `--browser-options`. + Note that `stealth` is no longer forced on — pass it in the JSON if you want it. + +- `@onkernel/cua-cli` and the `cua` binary are removed. Everything the CLI built + because it needed an agent front-end — sessions and resume, skills, the TUI, + print and RPC modes, model selection — pi supplies, so the extension replaces + it rather than reimplementing it. The `cua act` model-free executor path and + the `--print -o jsonl` telemetry schema are gone with it. +- Add `@onkernel/cua-pi-extension`, an installable pi extension that contributes + Kernel browser tools to pi's own agent session. The menu is eight entries, one + per capability: `browser` and `computer` (primitives plus their batch form), + `browser-act`, `playwright`, and the four provider-native surfaces — + `anthropic-computer`, `anthropic-browser`, `openai-computer`, `google-browser`. + Packaging variants are deliberately absent: `mixed`, the batch tools on their + own, and the 37 individual tool names offered nothing the eight entries do not. +- A deactivated selection now reports itself on stderr in print and RPC modes, + once per distinct reason. Previously the reason reached only the TUI status + line, so a scripted run lost its tools silently, created no browser, and let + the model answer from memory with exit 0. +- `/cua-tools` decides each entry's availability by compiling it on its own, and + reports pairwise conflicts separately. It previously passed the current + selection to the tool menu, whose verdicts are relative to that selection, so a + selection that failed to compile marked every entry unavailable with its + error — including entries that then activated fine. +- Provider-native surfaces work because the extension owns the stream for the + providers it registers, swapping pi's registry model for the compiled catalog's + model — which carries the transport the selected tools derive — and passing the + incoming native-call plan. Without that, `requiresApi` never takes effect and + native calls arrive unnormalized. +- A selection is validated by compiling it for the active model, so an + incompatible tool deactivates with the catalog compiler's own reason instead of + failing at request time. `/cua-tools` with no argument lists every selector for + the current model with those reasons. +- One browser is provisioned lazily per session on first tool execution and + deleted on shutdown if this session created it. Declaration compilation, header + generation, and payload transforms never provision a browser. diff --git a/packages/loop/README.md b/packages/loop/README.md new file mode 100644 index 00000000..2263cbab --- /dev/null +++ b/packages/loop/README.md @@ -0,0 +1,541 @@ +# `@onkernel/loop` + +Kernel browser computer-use for pi: tool declarations, per-model catalog +compilation, Kernel-browser execution, the `attach()` binding for +`@earendil-works/pi-agent-core`, and a pi extension. + +Two entry points: + +| import | what it is | +| --- | --- | +| `@onkernel/loop` | The framework-neutral core: canonical actions, the tool namespace, catalog compilation, the tool menu, and Kernel-browser execution. | +| `@onkernel/loop/pi` | The pi binding: `attach()`, model resolution, transport derivation, provider adapters, and provider retry. | + +Installing the package into pi (`pi install npm:@onkernel/loop`) registers the +extension described under [pi extension](#pi-extension). + +## Install + +```bash +npm install @onkernel/loop @onkernel/sdk +``` + +Requires Node 22.19 or newer, `KERNEL_API_KEY` for browser execution, and the +selected model provider's API key. + +## `attach()` + +`attach()` binds a Kernel browser to the package's execution resources and +returns a handle. `compile()` turns a (model, tools) pair into plain pi objects; +you construct whatever pi agent you want with them. There is no agent class here. + +```ts +import Kernel from "@onkernel/sdk"; +import { loop } from "@onkernel/loop"; +import { Agent, attach } from "@onkernel/loop/pi"; + +const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); +const browser = await client.browsers.create({ stealth: true }); +const kb = attach({ client, browser }); + +const { model, agentTools, models } = kb.compile({ + model: "anthropic:claude-opus-5", + tools: loop.toolsets.browser(), +}); + +const agent = new Agent({ + streamFn: (selected, context, options) => models.streamSimple(selected, context, options), + initialState: { + model, + tools: [...agentTools], + systemPrompt: "Inspect and interact with the page using the requested tools.", + }, +}); + +try { + await agent.prompt("Open example.com and report the heading."); +} finally { + await kb.dispose(); + await client.browsers.deleteByID(browser.session_id); +} +``` + +The compiled `model` carries the transport its tools derive: selecting a +provider-native browser or computer surface can change `model.api`, so the pair +has to reach pi together. + +### With pi's `AgentHarness` + +Use pi's harness for session-backed transcripts, skills, prompt templates, +compaction, steering, and follow-ups. `activate()` registers the behaviors this +package owns that are pi event handlers rather than constructor options, and +points the handle's `models` at this catalog: + +```ts +import { loop } from "@onkernel/loop"; +import { AgentHarness, attach, InMemorySessionRepo } from "@onkernel/loop/pi"; + +const session = await new InMemorySessionRepo().create(); +const kb = attach({ client, browser }); +const compiled = kb.compile({ model: "openai:gpt-5.6-sol", tools: loop.toolsets.browser() }); + +const harness = new AgentHarness({ + session, + model: compiled.model, + models: compiled.models, + tools: [...compiled.tools], + activeToolNames: compiled.tools.map((tool) => tool.name), + systemPrompt: "Use the supplied browser tools.", +}); +compiled.activate(harness); + +await harness.prompt("Find the pricing page."); +``` + +To change the model or the tool list on a running harness, compile the new pair +and apply it: + +```ts +await kb.compile({ model: "google:gemini-3.6-flash", tools: loop.providers.google.toolsets.browser() }).apply(harness); +``` + +`apply()` moves the model and tools together, sets the model only when the +derived transport actually moved, and restores the previous pair if pi rejects +the new one. Changing the model or the tool list compiles a new pair; nothing +mutates in place, and one shared execution-resource pool survives every change, +so browser refs, tabs, connections, and translator state are not reset. + +`@onkernel/loop/pi` re-exports pi-agent-core's session, skill, prompt-template, +compaction, and execution-environment primitives used with the harness. + +### Tool context + +Executable harness tools are pi `AgentHarnessTool`s: `execute` receives the +harness's tool context as its last argument. Supply it once as `toolContext` +and pi delivers the exact object (or the result of a zero-argument provider) +to every tool call: + +```ts +import { loop } from "@onkernel/loop"; +import { + AgentHarness, + attach, + NodeExecutionEnv, + createBashTool, + createReadTool, + type ExecutionToolContext, +} from "@onkernel/loop/pi"; + +const compiled = kb.compile({ + model: "openai:gpt-5.6-sol", + tools: [createReadTool(), createBashTool(), ...loop.toolsets.browser()], +}); +const harness = new AgentHarness({ + session, + model: compiled.model, + models: compiled.models, + tools: [...compiled.tools], + activeToolNames: compiled.tools.map((tool) => tool.name), + toolContext: { env: new NodeExecutionEnv({ cwd: process.cwd() }) }, + systemPrompt: "Use the supplied tools.", +}); +compiled.activate(harness); +``` + +Compile for the same context the harness delivers, so a later swap stays +type-compatible. Loop specs and plain pi `AgentTool`s are accepted too — they +simply ignore the context. `compiled.agentTools` is the context-free view for +the low-level `Agent`. + +## Action feedback + +Tools return only requested feedback: + +- write actions return concise status text; +- read actions return their requested text or structured data; +- explicit screenshot and zoom actions return images; +- `browser_act` returns causal outcomes and a bounded successor diff; +- failed batches replace images from earlier explicit screenshot steps with + textual markers. + +`toolResultImageReplayLimit` controls how many recent tool-result images remain +in model context (`4` by default, or `false` to disable projection). OpenAI +native computer results are exempt because its protocol requires each +`computer_call_output` to carry a screenshot. + +## Custom tools + +Ordinary pi `AgentTool`s can appear anywhere in the exact list: + +```ts +import { Type } from "@earendil-works/pi-ai"; + +const lookup = { + name: "customer_lookup", + label: "Customer lookup", + description: "Look up a customer by id.", + parameters: Type.Object({ id: Type.String() }), + async execute(_id, { id }) { + return { content: [{ type: "text", text: await lookupCustomer(id) }], details: {} }; + }, +}; + +kb.compile({ model, tools: [lookup, ...loop.toolsets.browser()] }); +``` + +Caller tools receive identity `caller.` through the canonical +`callerToolIdentity()` helper and participate in the same collision and +fingerprint rules. The catalog compiler is declaration-only, so the tool manager +projects caller `AgentTool`s into fresh declarations, joins compiled entries back +by identity, and materializes each spec exactly once per shared +execution-resource pool — repeat compiles hand pi a stable implementation. + +## Model catalog + +Model references are always provider-qualified: + +```ts +import { + getLoopModel, + listLoopModels, + parseLoopModelRef, +} from "@onkernel/loop/pi"; + +const model = getLoopModel("openai:gpt-5.6-sol"); +console.log(parseLoopModelRef("anthropic:claude-opus-5")); +console.table(listLoopModels("google")); +``` + +`gemini:` aliases `google:` and `moonshot:` aliases `moonshotai:`. The package +does not export a default model. See [models and native surfaces](docs/supported-models.md) +for which models have provider-native tools and which have known request limits. + +## Explicit tools + +All Loop-owned tools are available from one frozen namespace: + +```ts +import { loop } from "@onkernel/loop"; + +const tools = [ + loop.tools.browser.snapshot(), + loop.tools.browser.click(), + loop.tools.computer.screenshot(), +]; +``` + +Nothing is inferred from the model and no fallback tools are appended. + +### Atomic browser tools + +```ts +loop.tools.browser.snapshot(); +loop.tools.browser.text(); +loop.tools.browser.find(); +loop.tools.browser.click(); +loop.tools.browser.hover(); +loop.tools.browser.drag(); +loop.tools.browser.fill(); +loop.tools.browser.scrollTo(); +loop.tools.browser.scroll(); +loop.tools.browser.type(); +loop.tools.browser.key(); +loop.tools.browser.navigate(); +loop.tools.browser.listTabs(); +loop.tools.browser.newTab(); +loop.tools.browser.screenshot(); +loop.tools.browser.evaluate(); +loop.tools.browser.waitFor(); +loop.tools.browser.act(); +``` + +`browser_act` retains the established browser-action schema. Atomic tools expose +operation-specific arguments directly—there is no outer action wrapper. + +### Atomic computer tools + +```ts +loop.tools.computer.click(); +loop.tools.computer.doubleClick(); +loop.tools.computer.mouseDown(); +loop.tools.computer.mouseUp(); +loop.tools.computer.type(); +loop.tools.computer.keypress(); +loop.tools.computer.scroll(); +loop.tools.computer.move(); +loop.tools.computer.drag(); +loop.tools.computer.wait(); +loop.tools.computer.screenshot(); +loop.tools.computer.zoom(); +loop.tools.computer.goto(); +loop.tools.computer.back(); +loop.tools.computer.forward(); +loop.tools.computer.url(); +loop.tools.computer.cursorPosition(); +``` + +Computer coordinates default to pixels. Callers can request an explicit +normalized contract: + +```ts +loop.toolsets.computer({ + coordinates: loop.coordinates.normalized([0, 1000]), +}); +``` + +### Toolsets, names, and batches + +```ts +loop.toolsets.browser(); +loop.toolsets.computer(); +loop.toolsets.mixed(); +loop.toolsets.browser({ namespace: "page" }); + +loop.tools.browser.snapshot({ name: "page_snapshot" }); +loop.tools.computer.click({ name: "os_click" }); + +loop.tools.computer.batch({ actions: ["click", "keypress", "screenshot"] }); +loop.tools.browser.batch({ actions: ["snapshot", "click", "wait_for", "text"] }); + +loop.tools.playwright(); +``` + +Batches are mechanical primitive lists. They have no branching, saved values, +references, or workflow DSL. + +## Provider-native composition + +Provider-native tools are selected explicitly and may coexist with ordinary +function tools. + +```ts +const tools = [ + loop.providers.anthropic.tools.computer({ + version: "20260701", + enableZoom: true, + }), + loop.tools.browser.snapshot(), +]; +``` + +Available groups: + +```ts +loop.providers.openai.tools.computer(); + +loop.providers.anthropic.source; +loop.providers.anthropic.tools.computer({ version: "20260701" }); +loop.providers.anthropic.tools.browser({ version: "20260701" }); + +loop.providers.google.source; +loop.providers.google.toolsets.browser({ exclude: ["right_click"] }); + +// Meta, xAI, and Moonshot use the ordinary Loop browser tools. +loop.toolsets.browser(); +``` + +The Google browser set exposes the current predefined action names and uses +normalized coordinates in `[0, 999]`. Its native `computer_use` declaration +excludes every unselected browser action. If Google emits an excluded name +anyway, the adapter returns a named exact-catalog error instead of forwarding +an undeclared tool call. + +Moonshot accepts the ordinary browser toolset, including `browser_wait_for`, +but rejects `browser_act`'s substantially larger function schema. Catalog +compilation rejects that specific combination before a provider request. + +Provider-native caller-visible names are fixed by protocol. Version/tool/model +mismatches fail during catalog compilation. If an Anthropic credential cannot +access `browser_20260701`, Loop retries with an equivalent `browser` function +tool and remembers that choice for the credential and process. Every +`loop.providers.*` tool surface exposes its first-party `source` (or versioned +`sources`), and every returned provider spec carries the applicable URL. + +## Catalog compilation + +`compileLoopToolCatalog()` is the identity and validation boundary every +consumer shares — `attach()`, the pi extension, and callers compiling a catalog +themselves: + +```ts +const catalog = compileLoopToolCatalog({ + model: "anthropic:claude-opus-5", + requestedTools: tools, // Loop specs and plain pi-ai Tool declarations + viewport: { width: 1440, height: 900 }, +}); + +catalog.entries; // identities, fingerprints, declarations, coordinates +catalog.toolDeclarations; // pi-ai Tool declarations for Context.tools +catalog.headers.merge(callerHeaders); +await catalog.payload.apply(payload, catalog.model); +catalog.incoming; +``` + +Compilation is declaration-only and deterministic: identical declaration, +model, and viewport inputs produce identical catalogs, and compilation never +constructs executable tools or retains the requested input objects. Execution is +a separate concern: `attach()` materializes specs against a Kernel browser and +owns implementation identity. + +A Loop-owned identity remains stable when its name is customized. Caller tools +receive `caller.` identities through the canonical `callerToolIdentity()` +helper shared with every consumer. Compilation rejects: + +- duplicate identities; +- exact or provider-normalized caller-visible name collisions; +- unsafe names; +- incompatible model/provider-native combinations; +- conflicting payload-transform write claims; +- partial provider-native selections that violate a provider contract. + +The catalog fingerprint includes model, order, identity, name, schema, and +coordinates. The tool manager composes these declaration fingerprints with its +own implementation identity, so a schema or executor replacement cannot +masquerade as a no-op. + +Generated payload processing has deterministic order: + +1. model preparation; +2. tool declaration serialization; +3. provider request fields; +4. caller `onPayload` (applied by `attach()`). + +Generated header requirements merge with caller headers. Comma-list headers are +unioned and deduplicated; exact-value conflicts throw. + +## Dynamic loading metadata + +Ordinary function tools are marked eligible only where pi 0.83.0 supports +deferred loading. Provider-native tools are eager-only. The catalog itself does +not guess when tools were added; a caller that adds tools mid-turn records the +addition through pi's active-tool change entries. + +## Provider behavior + +Transport is derived, not stamped on the model ahead of time: a selected +tool's provider binding may declare `requiresApi`, and `compileLoopToolCatalog` +returns a `catalog.model` carrying that api. Selecting tools whose bindings +require different transports fails to compile. + +- **OpenAI**: a model selected with only ordinary/Loop browser tools streams + through pi's builtin Responses transport and its automatic prompt caching. + Selecting `loop.providers.openai.tools.computer()` derives the Loop-owned + `openai-computer-use` api instead, which a Loop adapter handles; that same + adapter also covers tool-search namespace round-trips regardless of api, + since pi's builtin transport does not replay them. +- **Anthropic**: exact native declarations, beta-header composition, and + adaptive model preparation. No api fork — every Anthropic model streams + through pi's builtin transport. +- **Google**: a model selected without Google's native browser toolset streams + through pi's builtin transport. Selecting + `loop.providers.google.toolsets.browser()` derives the Loop-owned + `google-interactions` api, which serializes one `computer_use` + declaration plus explicit exclusions through the Interactions API adapter. +- **Meta/xAI/Moonshot**: ordinary function tools with serial tool calls when the + selected catalog mutates browser state. + +## API keys + +```ts +import { + loopApiKeyEnvVarsForProvider, + getLoopEnvApiKeyForModel, + requireLoopEnvApiKeyForModel, +} from "@onkernel/loop/pi"; +``` + +Conventional variables are `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, +`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `XAI_API_KEY`, and +`MOONSHOT_API_KEY`. + +## pi extension + +Installing this package into pi adds the same tools to pi's own agent session. pi +owns the agent loop, session, and UI; the extension contributes the tools, the +browser they run against, and the provider wiring provider-native surfaces need. +It does not start a second model loop, and it adds no implicit screenshots or +prompt instructions. + +```sh +pi install npm:@onkernel/loop + +pi -p --provider openai --model gpt-5.6-sol \ + --browser-tools browser,browser-act "Open example.com and report its heading" +``` + +`KERNEL_API_KEY` is required when a tool first executes, not at startup. +`KERNEL_BASE_URL` is honored. Neither is written to session entries or output. + +### The menu + +Eight entries, one per capability. Availability is per model, and `/browser-tools` +tells you which apply to the one you selected. + +| entry | tools | works on | +| --- | --- | --- | +| `browser` | CDP browser primitives plus the one-call `browser_batch` form | every provider | +| `computer` | canonical computer primitives plus `computer_batch` | every provider | +| `browser-act` | `browser_act`, the verified-plan tool | every provider except Moonshot, which rejects its schema size | +| `playwright` | `playwright_execute` | every provider | +| `anthropic-computer` | Anthropic's native computer tool | Anthropic only | +| `anthropic-browser` | Anthropic's native browser tool | Anthropic only | +| `openai-computer` | OpenAI's native computer tool | OpenAI only | +| `google-browser` | Google's predefined browser action set | Google only | + +`--browser-coordinates` selects `pixels` (default) or `normalized-1000` for the +`computer` entry's coordinate contract. + +### Commands + +- `/browser` — current selectors, active tools, and browser status. +- `/browser-tools` — with no argument, list every selector for the current model, + marking the selected ones and showing the compiler's own reason for any that + this model cannot take. With an argument, replace the selection. `none` clears + it. + +A selection is checked by compiling it, so a model that cannot take a tool +deactivates it with a reason rather than failing at request time. Switching +models re-checks, and restores a previously forced-off selection when the new +model can take it. In TUI mode the reason appears in the status line; print and +RPC have no status line, so it is written to stderr once per distinct reason. + +### Browser + +| flag | effect | +| --- | --- | +| `--browser-session` | attach an existing session; never deleted on exit | +| `--browser-options` | JSON forwarded verbatim to Kernel's browser-create call | + +```sh +pi -p --browser-tools browser \ + --browser-options '{"stealth":true,"profile":{"id":"p1","save_changes":true},"proxy_id":"px1"}' \ + "open example.com" +``` + +One JSON object rather than a flag per field, so it tracks the Kernel SDK without +this extension growing an option every time the SDK does. The only default is +`timeout_seconds: 600` — the failure it prevents is a browser vanishing mid-task. +`--browser-session` attaches an existing browser, so it cannot be combined with +`--browser-options`. + +One browser is provisioned lazily per session, on first tool execution. +Compiling declarations, generating headers, and transforming a payload never +provision one. An owned browser is deleted on session shutdown. + +## Development + +```bash +npm run typecheck --workspace @onkernel/loop +npm run build --workspace @onkernel/loop +npm test --workspace @onkernel/loop +``` + +Build before testing: the pi print/RPC test loads the extension the way pi does, +through this package's own entry points. + +See [`examples/`](examples) for direct catalog/model usage, direct-agent and +harness smoke tests, provider matrices, and the Anthropic-native compositions. + +## License + +MIT. diff --git a/packages/ai/docs/supported-models.md b/packages/loop/docs/supported-models.md similarity index 74% rename from packages/ai/docs/supported-models.md rename to packages/loop/docs/supported-models.md index bfa9c19d..0685b0c6 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/loop/docs/supported-models.md @@ -1,17 +1,17 @@ # Models and native surfaces -`@onkernel/cua-ai` accepts **any model pi-ai carries**, and any model id its +`@onkernel/loop` accepts **any model pi-ai carries**, and any model id its registry has not caught up with yet. There is no allowlist: a model id you pass resolves, and the provider decides whether it exists. Run -`listCuaModels(provider?)` for the live catalog. +`listLoopModels(provider?)` for the live catalog. -Two small tables in [`src/models.ts`](https://github.com/kernel/cua/blob/main/packages/ai/src/models.ts) +Two small tables in [`src/pi/models.ts`](https://github.com/kernel/cua/blob/main/packages/loop/src/pi/models.ts) describe what is *different* about particular models. Neither decides whether a model may run. ## Native surfaces -`CUA_NATIVE_SURFACES` records which models have a provider-native computer or +`COMPUTER_USE_NATIVE_SURFACES` records which models have a provider-native computer or browser tool, so the tool menu can offer it. Entries match either an exact id or a `family` — the family root plus suffixes made of hyphen-separated numeric segments, covering revisions and dated snapshots such as `claude-opus-4-7` or @@ -26,14 +26,14 @@ models and need their own entry. Each cites first-party documentation. | `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite` | browser | Anthropic's entries live in `providers/anthropic/capabilities.ts`, which is -version-gated separately; `cuaNativeSurfaces(model)` reads both sources. +version-gated separately; `computerUseNativeSurfaces(model)` reads both sources. A model with no native surface is not restricted — it drives a Kernel browser -with CUA's own CDP tools, which is the default for every provider. +with Loop's own CDP tools, which is the default for every provider. ## Quirks -`CUA_MODEL_QUIRKS` records request-shape limits. Anything absent gets the +`LOOP_MODEL_QUIRKS` records request-shape limits. Anything absent gets the permissive default and is allowed to try; the provider's own error is the feedback. Every entry exists because of a documented limit or an observed failure, and carries its reason inline. @@ -46,8 +46,8 @@ failure, and carries its reason inline. | `openrouter` | `meta/muse-spark-1.1` | serializes state mutations | | `xai` | all | serializes state mutations | -`cuaModelCapabilities(model)` applies provider-wide quirks first, then -model-specific ones. `cuaModelQuirks(model)` returns the entries that applied, +`loopModelCapabilities(model)` applies provider-wide quirks first, then +model-specific ones. `loopModelQuirks(model)` returns the entries that applied, for diagnostics and menu hints. ## Model ids pi-ai does not carry @@ -72,26 +72,26 @@ change here. **The catalog looks stale.** Bump `@earendil-works/pi-ai`. Its registry is generated from models.dev, so a newer pi-ai is how names, context windows, and -pricing get refreshed. Note that cua does not read pi's `models.json`: that is a -pi-coding-agent config file, and cua builds its `Models` collection from pi-ai -directly. A provider pi-ai does not ship is not selectable without registering -it in `src/providers.ts` — a deliberate decision, since the repo has removed +pricing get refreshed. Note that this package does not read pi's `models.json`: that is a +pi-coding-agent config file, and `createLoopModels()` builds its `Models` +collection from pi-ai directly. A provider pi-ai does not ship is not selectable without registering +it in `src/pi/providers.ts` — a deliberate decision, since the repo has removed four such providers rather than carry them unused. **A provider shipped or changed a native tool.** This is real adapter work, not a table edit. Probe what the model actually emits: ```bash -npx tsx packages/ai/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --limit 3 +npx tsx packages/loop/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --limit 3 ``` Update that provider's adapter under `src/providers/` to execute the actions the -probe returns, then add or adjust the `CUA_NATIVE_SURFACES` entry, citing the +probe returns, then add or adjust the `COMPUTER_USE_NATIVE_SURFACES` entry, citing the provider's documentation. Anthropic's computer tool version and its `computer-use-*` beta header are chosen by pi-ai per model, so a new dated version there usually means bumping pi-ai rather than editing this package. -**A model rejects a tool CUA sends.** Add a `CUA_MODEL_QUIRKS` entry with the +**A model rejects a tool Loop sends.** Add a `LOOP_MODEL_QUIRKS` entry with the observed error as its `reason`, scoped as narrowly as the evidence supports: a single model id over a family, a family over a whole provider. Remove a quirk when the provider lifts the limit — a stale quirk silently denies a model a tool diff --git a/packages/loop/examples/agent-openai-smoke.ts b/packages/loop/examples/agent-openai-smoke.ts new file mode 100644 index 00000000..69c0ffe7 --- /dev/null +++ b/packages/loop/examples/agent-openai-smoke.ts @@ -0,0 +1,46 @@ +import Kernel from "@onkernel/sdk"; +import { loop } from "../src/index"; +import { Agent, attach, type LoopModelRef, requireLoopEnvApiKeyForModel } from "../src/pi/index"; +import { logAgentEvent, logAssistant } from "./shared/logging"; +import { SCENARIOS } from "./shared/scenarios"; + +const modelRef = (process.env.MODEL_REF as LoopModelRef | undefined) ?? "openai:gpt-5.6-sol"; + +async function main(): Promise { + const kernelApiKey = process.env.KERNEL_API_KEY; + if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); + requireLoopEnvApiKeyForModel(modelRef); + const client = new Kernel({ apiKey: kernelApiKey }); + const browser = await client.browsers.create({ stealth: true }); + const kb = attach({ browser, client }); + + try { + // Prefer structured browser refs and semantic reads for the OpenAI smoke, + // and opt into verified dependent plans without changing the base toolset. + const compiled = kb.compile({ + model: modelRef, + tools: [...loop.toolsets.browser(), loop.tools.browser.act()], + }); + const agent = new Agent({ + streamFn: (selected, context, options) => compiled.models.streamSimple(selected, context, options), + initialState: { + model: compiled.model, + tools: [...compiled.agentTools], + systemPrompt: "Use the provided computer and browser tools to interact with the page.", + }, + }); + + agent.subscribe(logAgentEvent); + + const scenario = SCENARIOS[0]!; + console.log(`running scenario: ${scenario.name} model=${modelRef}`); + await agent.prompt(scenario.prompt); + const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant"); + logAssistant(assistant?.role === "assistant" ? assistant : undefined); + } finally { + await kb.dispose(); + await client.browsers.deleteByID(browser.session_id); + } +} + +void main(); diff --git a/packages/agent/examples/agent-provider-matrix.ts b/packages/loop/examples/agent-provider-matrix.ts similarity index 60% rename from packages/agent/examples/agent-provider-matrix.ts rename to packages/loop/examples/agent-provider-matrix.ts index e13b0472..f5fb018e 100644 --- a/packages/agent/examples/agent-provider-matrix.ts +++ b/packages/loop/examples/agent-provider-matrix.ts @@ -1,30 +1,30 @@ import Kernel from "@onkernel/sdk"; -import { - requireCuaEnvApiKeyForModel, - type CuaModelRef, -} from "@onkernel/cua-ai"; -import { CuaAgent } from "../src/index"; +import { Agent, attach, type LoopModelRef, requireLoopEnvApiKeyForModel } from "../src/pi/index"; import { logAgentEvent, logAssistant } from "./shared/logging"; import { SCENARIOS } from "./shared/scenarios"; import { toolsForModel } from "./shared/tools"; -const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.6-sol"; +const modelRef = (process.env.MODEL_REF as LoopModelRef | undefined) ?? "openai:gpt-5.6-sol"; const scenarioName = process.env.SCENARIO ?? SCENARIOS[0]!.name; async function main(): Promise { const kernelApiKey = process.env.KERNEL_API_KEY; if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); - requireCuaEnvApiKeyForModel(modelRef); + requireLoopEnvApiKeyForModel(modelRef); const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); const scenario = SCENARIOS.find((entry) => entry.name === scenarioName) ?? SCENARIOS[0]!; + const kb = attach({ browser, client }); try { - const agent = new CuaAgent({ - browser, - client, - tools: toolsForModel(modelRef), - initialState: { model: modelRef, systemPrompt: "Use the provided computer and browser tools to interact with the page." }, + const compiled = kb.compile({ model: modelRef, tools: toolsForModel(modelRef) }); + const agent = new Agent({ + streamFn: (selected, context, options) => compiled.models.streamSimple(selected, context, options), + initialState: { + model: compiled.model, + tools: [...compiled.agentTools], + systemPrompt: "Use the provided computer and browser tools to interact with the page.", + }, }); agent.subscribe(logAgentEvent); console.log(`model=${modelRef} scenario=${scenario.name}`); @@ -32,6 +32,7 @@ async function main(): Promise { const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant"); logAssistant(assistant?.role === "assistant" ? assistant : undefined); } finally { + await kb.dispose(); await client.browsers.deleteByID(browser.session_id); } } diff --git a/packages/agent/examples/anthropic-native-smoke.ts b/packages/loop/examples/anthropic-native-smoke.ts similarity index 59% rename from packages/agent/examples/anthropic-native-smoke.ts rename to packages/loop/examples/anthropic-native-smoke.ts index 5daf459b..ef7c6a8c 100644 --- a/packages/agent/examples/anthropic-native-smoke.ts +++ b/packages/loop/examples/anthropic-native-smoke.ts @@ -2,21 +2,21 @@ // // MODEL_REF=anthropic:claude-opus-5 CONFIG=native-browser tsx examples/anthropic-native-smoke.ts // -// CONFIG selects the requested catalog; CuaAgent never infers or appends tools. +// CONFIG selects the requested catalog; nothing is inferred or appended. import Kernel from "@onkernel/sdk"; -import { cua, requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai"; -import { CuaAgent, type CuaAgentTool } from "../src/index"; +import { loop, type LoopAgentTool } from "../src/index"; +import { Agent, attach, type LoopModelRef, requireLoopEnvApiKeyForModel } from "../src/pi/index"; import { logAgentEvent, logAssistant } from "./shared/logging"; -const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "anthropic:claude-opus-5"; +const modelRef = (process.env.MODEL_REF as LoopModelRef | undefined) ?? "anthropic:claude-opus-5"; const config = process.env.CONFIG ?? "computer"; -const CONFIGS: Record = { - computer: cua.toolsets.computer(), - browser: cua.toolsets.browser(), - mixed: cua.toolsets.mixed(), - "native-computer": [cua.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true })], - "native-browser": [cua.providers.anthropic.tools.browser({ version: "20260701" })], +const CONFIGS: Record = { + computer: loop.toolsets.computer(), + browser: loop.toolsets.browser(), + mixed: loop.toolsets.mixed(), + "native-computer": [loop.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true })], + "native-browser": [loop.providers.anthropic.tools.browser({ version: "20260701" })], }; const PROMPT = [ @@ -31,17 +31,18 @@ async function main(): Promise { if (!tools) throw new Error(`unknown CONFIG "${config}" (expected: ${Object.keys(CONFIGS).join(" | ")})`); const kernelApiKey = process.env.KERNEL_API_KEY; if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); - requireCuaEnvApiKeyForModel(modelRef); + requireLoopEnvApiKeyForModel(modelRef); const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); + const kb = attach({ browser, client }); try { - const agent = new CuaAgent({ - browser, - client, - tools, + const compiled = kb.compile({ model: modelRef, tools }); + const agent = new Agent({ + streamFn: (selected, context, options) => compiled.models.streamSimple(selected, context, options), initialState: { - model: modelRef, + model: compiled.model, + tools: [...compiled.agentTools], systemPrompt: "Use only the requested tools to interact with the browser.", }, }); @@ -52,6 +53,7 @@ async function main(): Promise { const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant"); logAssistant(assistant?.role === "assistant" ? assistant : undefined); } finally { + await kb.dispose(); await client.browsers.deleteByID(browser.session_id); } } diff --git a/packages/agent/examples/harness-openai-smoke.ts b/packages/loop/examples/harness-openai-smoke.ts similarity index 59% rename from packages/agent/examples/harness-openai-smoke.ts rename to packages/loop/examples/harness-openai-smoke.ts index e381413b..9eb7ddd6 100644 --- a/packages/agent/examples/harness-openai-smoke.ts +++ b/packages/loop/examples/harness-openai-smoke.ts @@ -1,31 +1,43 @@ import Kernel from "@onkernel/sdk"; -import { cua, requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai"; -import { CuaAgentHarness, InMemorySessionRepo } from "../src/index"; +import { loop } from "../src/index"; +import { + AgentHarness, + attach, + InMemorySessionRepo, + type LoopModelRef, + requireLoopEnvApiKeyForModel, +} from "../src/pi/index"; import { logAgentEvent, logAssistant } from "./shared/logging"; import { SCENARIOS } from "./shared/scenarios"; -const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.6-sol"; +const modelRef = (process.env.MODEL_REF as LoopModelRef | undefined) ?? "openai:gpt-5.6-sol"; async function main(): Promise { const kernelApiKey = process.env.KERNEL_API_KEY; if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); - requireCuaEnvApiKeyForModel(modelRef); + requireLoopEnvApiKeyForModel(modelRef); const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); + const kb = attach({ browser, client }); try { const sessionRepo = new InMemorySessionRepo(); const session = await sessionRepo.create({ id: "harness-openai-smoke" }); - const harness = new CuaAgentHarness({ - browser, - client, + // Prefer structured browser refs and semantic reads for the OpenAI smoke, + // and opt into verified dependent plans without changing the base toolset. + const compiled = kb.compile({ model: modelRef, + tools: [...loop.toolsets.browser(), loop.tools.browser.act()], + }); + const harness = new AgentHarness({ session, - // Prefer structured browser refs and semantic reads for the OpenAI smoke, - // and opt into verified dependent plans without changing the base toolset. - tools: [...cua.toolsets.browser(), cua.tools.browser.act()], + model: compiled.model, + models: compiled.models, + tools: [...compiled.tools], + activeToolNames: compiled.tools.map((tool) => tool.name), systemPrompt: "Use the provided computer and browser tools to interact with the page.", }); + compiled.activate(harness); harness.subscribe(logAgentEvent); @@ -40,6 +52,7 @@ async function main(): Promise { )[0]; logAssistant(lastAssistant ?? response); } finally { + await kb.dispose(); await client.browsers.deleteByID(browser.session_id); } } diff --git a/packages/agent/examples/harness-provider-matrix.ts b/packages/loop/examples/harness-provider-matrix.ts similarity index 69% rename from packages/agent/examples/harness-provider-matrix.ts rename to packages/loop/examples/harness-provider-matrix.ts index 1218945c..19a21eee 100644 --- a/packages/agent/examples/harness-provider-matrix.ts +++ b/packages/loop/examples/harness-provider-matrix.ts @@ -1,35 +1,40 @@ import Kernel from "@onkernel/sdk"; import { - requireCuaEnvApiKeyForModel, - type CuaModelRef, -} from "@onkernel/cua-ai"; -import { CuaAgentHarness, InMemorySessionRepo } from "../src/index"; + AgentHarness, + attach, + InMemorySessionRepo, + type LoopModelRef, + requireLoopEnvApiKeyForModel, +} from "../src/pi/index"; import { logAgentEvent, logAssistant } from "./shared/logging"; import { SCENARIOS } from "./shared/scenarios"; import { toolsForModel } from "./shared/tools"; -const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.6-sol"; +const modelRef = (process.env.MODEL_REF as LoopModelRef | undefined) ?? "openai:gpt-5.6-sol"; const scenarioName = process.env.SCENARIO ?? SCENARIOS[0]!.name; async function main(): Promise { const kernelApiKey = process.env.KERNEL_API_KEY; if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required"); - requireCuaEnvApiKeyForModel(modelRef); + requireLoopEnvApiKeyForModel(modelRef); const client = new Kernel({ apiKey: kernelApiKey }); const browser = await client.browsers.create({ stealth: true }); const scenario = SCENARIOS.find((entry) => entry.name === scenarioName) ?? SCENARIOS[0]!; + const kb = attach({ browser, client }); try { const sessionRepo = new InMemorySessionRepo(); const session = await sessionRepo.create({ id: `harness-provider-matrix-${scenario.name}` }); - const harness = new CuaAgentHarness({ - browser, - client, - model: modelRef, + const compiled = kb.compile({ model: modelRef, tools: toolsForModel(modelRef) }); + const harness = new AgentHarness({ session, - tools: toolsForModel(modelRef), + model: compiled.model, + models: compiled.models, + tools: [...compiled.tools], + activeToolNames: compiled.tools.map((tool) => tool.name), systemPrompt: "Use the provided computer and browser tools to interact with the page.", }); + compiled.activate(harness); harness.subscribe(logAgentEvent); console.log(`model=${modelRef} scenario=${scenario.name}`); const response = await harness.prompt(scenario.prompt); @@ -41,6 +46,7 @@ async function main(): Promise { )[0]; logAssistant(lastAssistant ?? response); } finally { + await kb.dispose(); await client.browsers.deleteByID(browser.session_id); } } diff --git a/packages/ai/examples/quickstart.ts b/packages/loop/examples/quickstart.ts similarity index 63% rename from packages/ai/examples/quickstart.ts rename to packages/loop/examples/quickstart.ts index 2c991aa8..913ed6a5 100644 --- a/packages/ai/examples/quickstart.ts +++ b/packages/loop/examples/quickstart.ts @@ -1,27 +1,22 @@ import { readFile } from "node:fs/promises"; -import { - compileCuaToolCatalog, - cua, - cuaModels, - requireCuaEnvApiKeyForModel, - type CuaModelRef, -} from "@onkernel/cua-ai"; +import { compileLoopToolCatalog, loop } from "../src/index"; +import { type LoopModelRef, loopModels, requireLoopEnvApiKeyForModel } from "../src/pi/index"; -// Switch providers by setting CUA_MODEL and the matching provider API key. -const modelRef = (process.env.CUA_MODEL ?? "openai:gpt-5.6-sol") as CuaModelRef; -const apiKey = requireCuaEnvApiKeyForModel(modelRef); +// Switch providers by setting LOOP_MODEL and the matching provider API key. +const modelRef = (process.env.LOOP_MODEL ?? "openai:gpt-5.6-sol") as LoopModelRef; +const apiKey = requireLoopEnvApiKeyForModel(modelRef); const screenshot = await readFile(new URL("./screenshot.png", import.meta.url)); // The caller requests the exact catalog. Nothing is inferred or appended. -// Compilation is declaration-only: cua-ai has no pi-agent-core dependency and -// never constructs executable tools. `@onkernel/cua-agent` materializes specs -// against a live Kernel browser when execution is needed. -const catalog = compileCuaToolCatalog({ +// Compilation is declaration-only: it never constructs executable tools. +// `attach()` materializes specs against a live Kernel browser when execution is +// needed. +const catalog = compileLoopToolCatalog({ model: modelRef, - requestedTools: [cua.tools.computer.click()], + requestedTools: [loop.tools.computer.click()], }); -const response = await cuaModels().complete( +const response = await loopModels().complete( catalog.model, { systemPrompt: "Call computer_click with the target coordinates. Do not describe the click in prose.", diff --git a/packages/ai/examples/screenshot.png b/packages/loop/examples/screenshot.png similarity index 100% rename from packages/ai/examples/screenshot.png rename to packages/loop/examples/screenshot.png diff --git a/packages/agent/examples/shared/logging.ts b/packages/loop/examples/shared/logging.ts similarity index 94% rename from packages/agent/examples/shared/logging.ts rename to packages/loop/examples/shared/logging.ts index 39d11702..7d68b5bc 100644 --- a/packages/agent/examples/shared/logging.ts +++ b/packages/loop/examples/shared/logging.ts @@ -1,4 +1,4 @@ -import type { AgentEvent, AgentHarnessEvent } from "../../src/index"; +import type { AgentEvent, AgentHarnessEvent } from "../../src/pi/index"; type AssistantLike = { content: Array<{ type: string; text?: string }>; diff --git a/packages/agent/examples/shared/scenarios.ts b/packages/loop/examples/shared/scenarios.ts similarity index 100% rename from packages/agent/examples/shared/scenarios.ts rename to packages/loop/examples/shared/scenarios.ts diff --git a/packages/agent/examples/shared/tools.ts b/packages/loop/examples/shared/tools.ts similarity index 51% rename from packages/agent/examples/shared/tools.ts rename to packages/loop/examples/shared/tools.ts index 33b6ae61..68bf06eb 100644 --- a/packages/agent/examples/shared/tools.ts +++ b/packages/loop/examples/shared/tools.ts @@ -1,37 +1,36 @@ +import { loop, type LoopAgentTool } from "../../src/index"; import { - cua, - cuaModelCapabilities, - getCuaModel, - parseCuaModelRef, - type CuaModelRef, -} from "@onkernel/cua-ai"; -import type { CuaAgentTool } from "../../src/index"; + getLoopModel, + loopModelCapabilities, + type LoopModelRef, + parseLoopModelRef, +} from "../../src/pi/index"; -function structuredBrowserTools(): CuaAgentTool[] { - return [...cua.toolsets.browser(), cua.tools.browser.act()]; +function structuredBrowserTools(): LoopAgentTool[] { + return [...loop.toolsets.browser(), loop.tools.browser.act()]; } /** * Interaction policy shared by the agent and harness provider matrices. Both * examples read it from here so the two cannot drift apart. */ -export function toolsForModel(model: CuaModelRef): CuaAgentTool[] { - const { provider, model: modelId } = parseCuaModelRef(model); +export function toolsForModel(model: LoopModelRef): LoopAgentTool[] { + const { provider, model: modelId } = parseLoopModelRef(model); switch (provider) { case "openai": // Favor refs, semantic reads, and verified plans over coordinate-only computer use. return structuredBrowserTools(); case "anthropic": // Claude 5 can use Anthropic's native browser tool; older models use portable - // CUA tools plus the explicit semantic action-plan surface. - return cua.providers.anthropic.supports.browser(modelId) - ? [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })] + // Loop tools plus the explicit semantic action-plan surface. + return loop.providers.anthropic.supports.browser(modelId) + ? [loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true })] : structuredBrowserTools(); case "google": // Current Gemini computer-use models expect Google's predefined browser actions. - return cua.providers.google.toolsets.browser(); + return loop.providers.google.toolsets.browser(); case "xai": - // No first-party native browser surface exists, so use CUA browser primitives + // No first-party native browser surface exists, so use Loop browser primitives // plus verified dependent plans. return structuredBrowserTools(); case "moonshotai": @@ -39,8 +38,8 @@ export function toolsForModel(model: CuaModelRef): CuaAgentTool[] { // Same as xai, minus browser_act where the model rejects that tool's // oversized schema. OpenRouter fronts several model families, so ask // the model rather than the provider. - return cuaModelCapabilities(getCuaModel(model)).acceptsLargeSchemas + return loopModelCapabilities(getLoopModel(model)).acceptsLargeSchemas ? structuredBrowserTools() - : cua.toolsets.browser(); + : loop.toolsets.browser(); } } diff --git a/packages/agent/package.json b/packages/loop/package.json similarity index 51% rename from packages/agent/package.json rename to packages/loop/package.json index 6fcccca9..df33c376 100644 --- a/packages/agent/package.json +++ b/packages/loop/package.json @@ -1,7 +1,7 @@ { - "name": "@onkernel/cua-agent", - "version": "0.10.0", - "description": "Kernel browser computer-use Agent and AgentHarness classes built on pi-agent-core", + "name": "@onkernel/loop", + "version": "0.11.0", + "description": "Kernel browser computer-use tools, catalog compilation, and pi bindings", "license": "MIT", "type": "module", "main": "./dist/index.js", @@ -9,21 +9,39 @@ "repository": { "type": "git", "url": "git+https://github.com/kernel/cua.git", - "directory": "packages/agent" + "directory": "packages/loop" }, "bugs": { "url": "https://github.com/kernel/cua/issues" }, - "homepage": "https://github.com/kernel/cua/tree/main/packages/agent#readme", + "homepage": "https://github.com/kernel/cua/tree/main/packages/loop#readme", + "keywords": [ + "pi-package", + "pi-extension", + "computer-use", + "kernel" + ], "exports": { ".": { "types": "./dist/index.d.ts", "source": "./src/index.ts", - "import": "./dist/index.js" + "default": "./dist/index.js" + }, + "./pi": { + "types": "./dist/pi/index.d.ts", + "source": "./src/pi/index.ts", + "default": "./dist/pi/index.js" } }, + "pi": { + "extensions": [ + "./src/pi-extension/index.ts" + ] + }, "files": [ "dist", + "src", + "docs", "examples", "README.md", "CHANGELOG.md" @@ -36,20 +54,31 @@ }, "scripts": { "build": "tsdown", - "clean": "tsc -b --clean && rm -rf dist dist-tsc", + "typecheck": "tsc -b", + "clean": "tsc -b --clean && rm -rf dist dist-published dist-tsc", + "example:quickstart": "NODE_OPTIONS=--conditions=source tsx examples/quickstart.ts", "example:agent": "NODE_OPTIONS=--conditions=source tsx examples/agent-openai-smoke.ts", "example:harness": "NODE_OPTIONS=--conditions=source tsx examples/harness-openai-smoke.ts", "test": "vitest --run", - "typecheck": "tsc -b" + "test:integration": "vitest --run --config vitest.integration.config.ts" }, "dependencies": { "@earendil-works/pi-agent-core": "0.83.0", "@earendil-works/pi-ai": "0.83.0", - "@onkernel/cua-ai": "0.10.0", "@onkernel/sdk": "0.49.0", + "openai": "^6.26.0", "sharp": "^0.35.3" }, + "peerDependencies": { + "@earendil-works/pi-coding-agent": "*" + }, + "peerDependenciesMeta": { + "@earendil-works/pi-coding-agent": { + "optional": true + } + }, "devDependencies": { + "@earendil-works/pi-coding-agent": "0.83.0", "tsdown": "^0.22.2", "vitest": "^3.2.4" } diff --git a/packages/ai/scripts/native-action-probe.ts b/packages/loop/scripts/native-action-probe.ts similarity index 100% rename from packages/ai/scripts/native-action-probe.ts rename to packages/loop/scripts/native-action-probe.ts diff --git a/packages/ai/src/actions/browser.ts b/packages/loop/src/core/actions/browser.ts similarity index 89% rename from packages/ai/src/actions/browser.ts rename to packages/loop/src/core/actions/browser.ts index aec22f98..5c9b2c9e 100644 --- a/packages/ai/src/actions/browser.ts +++ b/packages/loop/src/core/actions/browser.ts @@ -15,7 +15,7 @@ import { Type, type TSchema } from "@earendil-works/pi-ai"; * by `browser_snapshot` / `browser_find`; a stale ref is an error instructing the * model to re-snapshot. */ -export const CUA_BROWSER_ACTION_TYPES = [ +export const BROWSER_ACTION_TYPES = [ "browser_snapshot", "browser_act", "browser_wait_for", @@ -36,9 +36,9 @@ export const CUA_BROWSER_ACTION_TYPES = [ "browser_evaluate", ] as const; -export type CuaBrowserActionType = (typeof CUA_BROWSER_ACTION_TYPES)[number]; +export type BrowserActionType = (typeof BROWSER_ACTION_TYPES)[number]; -export interface CuaActionBrowserSnapshot { +export interface BrowserActionSnapshot { type: "browser_snapshot"; filter?: "all" | "interactive"; ref?: string; @@ -61,32 +61,32 @@ type LocationExpectation = { type: "url" | "title" } & ( | { equals?: string; contains: string; changed?: boolean } | { equals?: string; contains?: string; changed: boolean } ); -type CuaBrowserExpectationLeaf = +type BrowserExpectationLeaf = | { type: "text"; text: string; exists?: boolean } | RoleNameExpectation | RefExpectation | LocationExpectation; type NonEmptyArray = [T, ...T[]]; /** Semantic condition over accessible content, ref state, location, or all/any leaf groups. */ -export type CuaBrowserExpectation = CuaBrowserExpectationLeaf | { all: NonEmptyArray } | { any: NonEmptyArray }; +export type BrowserExpectation = BrowserExpectationLeaf | { all: NonEmptyArray } | { any: NonEmptyArray }; -export interface CuaActionBrowserWaitFor { +export interface BrowserActionWaitFor { type: "browser_wait_for"; - expect: CuaBrowserExpectation; + expect: BrowserExpectation; /** Semantic polling timeout; an in-flight browser read settles before timeout is reported. */ timeout_ms?: number; poll_ms?: number; tab_id?: string; } -type CuaBrowserActStepOptions = { +type BrowserActStepOptions = { /** Timeout for performing this step and verifying its expectation, capped by the plan deadline. An in-flight atomic input settles before timeout is reported. */ timeout_ms?: number; - expect?: CuaBrowserExpectation; + expect?: BrowserExpectation; }; /** Ref- or focus-based operation with an optional per-step semantic expectation. */ -export type CuaBrowserActStep = ( +export type BrowserActStep = ( | { type: "click"; ref: string; button?: "left" | "right" | "middle"; num_clicks?: 1 | 2 | 3; modifiers?: string[] } | { type: "hover"; ref: string } | { type: "fill"; ref: string; value: string | number | boolean } @@ -94,13 +94,13 @@ export type CuaBrowserActStep = ( | { type: "key"; text: string; repeat?: number } | { type: "scroll_to"; ref: string } | { type: "wait"; ms?: number } -) & CuaBrowserActStepOptions; +) & BrowserActStepOptions; /** Dependent action plan whose optional `expect` verifies the complete plan result. */ -export interface CuaActionBrowserAct { +export interface BrowserActionAct { type: "browser_act"; - steps: NonEmptyArray; - expect?: CuaBrowserExpectation; + steps: NonEmptyArray; + expect?: BrowserExpectation; /** Global timeout for performing every step and verifying the plan expectation. An in-flight atomic input settles before timeout is reported. */ timeout_ms?: number; poll_ms?: number; @@ -108,18 +108,18 @@ export interface CuaActionBrowserAct { tab_id?: string; } -export interface CuaActionBrowserText { +export interface BrowserActionText { type: "browser_text"; tab_id?: string; } -export interface CuaActionBrowserFind { +export interface BrowserActionFind { type: "browser_find"; query: string; tab_id?: string; } -export interface CuaActionBrowserClick { +export interface BrowserActionClick { type: "browser_click"; ref?: string; x?: number; @@ -130,7 +130,7 @@ export interface CuaActionBrowserClick { tab_id?: string; } -export interface CuaActionBrowserHover { +export interface BrowserActionHover { type: "browser_hover"; ref?: string; x?: number; @@ -138,27 +138,27 @@ export interface CuaActionBrowserHover { tab_id?: string; } -export interface CuaActionBrowserDrag { +export interface BrowserActionDrag { type: "browser_drag"; from: { x: number; y: number }; to: { x: number; y: number }; tab_id?: string; } -export interface CuaActionBrowserFill { +export interface BrowserActionFill { type: "browser_fill"; ref: string; value: string | number | boolean; tab_id?: string; } -export interface CuaActionBrowserScrollTo { +export interface BrowserActionScrollTo { type: "browser_scroll_to"; ref: string; tab_id?: string; } -export interface CuaActionBrowserScroll { +export interface BrowserActionScroll { type: "browser_scroll"; x: number; y: number; @@ -167,69 +167,69 @@ export interface CuaActionBrowserScroll { tab_id?: string; } -export interface CuaActionBrowserType { +export interface BrowserActionTypeText { type: "browser_type"; text: string; tab_id?: string; } -export interface CuaActionBrowserKey { +export interface BrowserActionKey { type: "browser_key"; text: string; repeat?: number; tab_id?: string; } -export interface CuaActionBrowserNavigate { +export interface BrowserActionNavigate { type: "browser_navigate"; /** A URL, or the sentinels "back" / "forward" for history navigation. */ url: string; tab_id?: string; } -export interface CuaActionBrowserListTabs { +export interface BrowserActionListTabs { type: "browser_list_tabs"; } -export interface CuaActionBrowserNewTab { +export interface BrowserActionNewTab { type: "browser_new_tab"; } -export interface CuaActionBrowserScreenshot { +export interface BrowserActionScreenshot { type: "browser_screenshot"; /** Optional crop region, [x0, y0, x1, y1] in viewport pixels. */ region?: [number, number, number, number]; tab_id?: string; } -export interface CuaActionBrowserEvaluate { +export interface BrowserActionEvaluate { type: "browser_evaluate"; code: string; tab_id?: string; } -export type CuaBrowserAction = - | CuaActionBrowserSnapshot - | CuaActionBrowserAct - | CuaActionBrowserWaitFor - | CuaActionBrowserText - | CuaActionBrowserFind - | CuaActionBrowserClick - | CuaActionBrowserHover - | CuaActionBrowserDrag - | CuaActionBrowserFill - | CuaActionBrowserScrollTo - | CuaActionBrowserScroll - | CuaActionBrowserType - | CuaActionBrowserKey - | CuaActionBrowserNavigate - | CuaActionBrowserListTabs - | CuaActionBrowserNewTab - | CuaActionBrowserScreenshot - | CuaActionBrowserEvaluate; +export type BrowserAction = + | BrowserActionSnapshot + | BrowserActionAct + | BrowserActionWaitFor + | BrowserActionText + | BrowserActionFind + | BrowserActionClick + | BrowserActionHover + | BrowserActionDrag + | BrowserActionFill + | BrowserActionScrollTo + | BrowserActionScroll + | BrowserActionTypeText + | BrowserActionKey + | BrowserActionNavigate + | BrowserActionListTabs + | BrowserActionNewTab + | BrowserActionScreenshot + | BrowserActionEvaluate; /** Options for building browser action schemas. */ -export interface CuaBrowserSchemaOptions { +export interface BrowserActionSchemaOptions { /** * Whether coordinate targeting is allowed on `browser_click` / `browser_hover` * and whether `browser_drag` / `browser_scroll` are expressible at all. Browser @@ -244,7 +244,7 @@ const TabId = () => Type.Optional(Type.String({ description: "Tab to act on. Def const RefProperty = () => Type.String({ description: "Element reference from browser_snapshot or browser_find, e.g. \"e12\"." }); -export function createCuaBrowserActionSchemaByType(options: CuaBrowserSchemaOptions): Record { +export function createBrowserActionSchemaByType(options: BrowserActionSchemaOptions): Record { const exists = () => Type.Optional(Type.Boolean({ description: "Whether the matching content must exist (default true)." })); const roleName = (required: "role" | "name") => Type.Object( { diff --git a/packages/ai/src/actions/computer.ts b/packages/loop/src/core/actions/computer.ts similarity index 75% rename from packages/ai/src/actions/computer.ts rename to packages/loop/src/core/actions/computer.ts index 181269b0..a7c06c17 100644 --- a/packages/ai/src/actions/computer.ts +++ b/packages/loop/src/core/actions/computer.ts @@ -8,7 +8,7 @@ import { Type, type TSchema } from "@earendil-works/pi-ai"; * OS screenshot frame. The browser-plane vocabulary lives in `./browser` and is * executed over CDP; the two planes never share a coordinate frame. */ -export const CUA_COMPUTER_ACTION_TYPES = [ +export const COMPUTER_ACTION_TYPES = [ "click", "double_click", "mouse_down", @@ -28,74 +28,74 @@ export const CUA_COMPUTER_ACTION_TYPES = [ "cursor_position", ] as const; -export type CuaComputerActionType = (typeof CUA_COMPUTER_ACTION_TYPES)[number]; +export type ComputerActionType = (typeof COMPUTER_ACTION_TYPES)[number]; /** * The default computer-mode toolset. This is the pre-modes canonical action list: * every computer action except `zoom`, which is only exposed by default in hybrid * mode and by Anthropic's native computer tool (`enable_zoom`). */ -export const CUA_DEFAULT_COMPUTER_ACTION_TYPES = CUA_COMPUTER_ACTION_TYPES.filter( - (action): action is Exclude => action !== "zoom", +export const DEFAULT_COMPUTER_ACTION_TYPES = COMPUTER_ACTION_TYPES.filter( + (action): action is Exclude => action !== "zoom", ); /** * Mouse buttons accepted by click, mouse_down, and mouse_up actions. The * executor coerces anything outside this set to "left". */ -export type CuaMouseButton = "left" | "right" | "middle" | "back" | "forward"; +export type MouseButton = "left" | "right" | "middle" | "back" | "forward"; /** * Mouse buttons accepted by drag actions. The executor coerces anything * outside this set to "left". */ -export type CuaDragMouseButton = "left" | "right" | "middle"; +export type DragMouseButton = "left" | "right" | "middle"; -export interface CuaActionClick { +export interface ComputerActionClick { type: "click"; /** OS screenshot pixels. Omitted (native mappings only) means the current cursor position. */ x?: number; y?: number; - button?: CuaMouseButton; + button?: MouseButton; hold_keys?: string[]; num_clicks?: number; } -export interface CuaActionDoubleClick { +export interface ComputerActionDoubleClick { type: "double_click"; x: number; y: number; hold_keys?: string[]; } -export interface CuaActionMouseDown { +export interface ComputerActionMouseDown { type: "mouse_down"; x?: number; y?: number; - button?: CuaMouseButton; + button?: MouseButton; hold_keys?: string[]; } -export interface CuaActionMouseUp { +export interface ComputerActionMouseUp { type: "mouse_up"; x?: number; y?: number; - button?: CuaMouseButton; + button?: MouseButton; hold_keys?: string[]; } -export interface CuaActionTypeText { +export interface ComputerActionTypeText { type: "type"; text: string; } -export interface CuaActionKeypress { +export interface ComputerActionKeypress { type: "keypress"; keys: string[]; duration?: number; } -export interface CuaActionScroll { +export interface ComputerActionScroll { type: "scroll"; x?: number; y?: number; @@ -104,73 +104,73 @@ export interface CuaActionScroll { hold_keys?: string[]; } -export interface CuaActionMove { +export interface ComputerActionMove { type: "move"; x: number; y: number; } -export interface CuaActionDrag { +export interface ComputerActionDrag { type: "drag"; path: Array<{ x: number; y: number }>; - button?: CuaDragMouseButton; + button?: DragMouseButton; hold_keys?: string[]; } -export interface CuaActionWait { +export interface ComputerActionWait { type: "wait"; ms?: number; } -export interface CuaActionScreenshot { +export interface ComputerActionScreenshot { type: "screenshot"; } /** Crop of the most recent OS screenshot; region is [x0, y0, x1, y1] in OS screenshot pixels. */ -export interface CuaActionZoom { +export interface ComputerActionZoom { type: "zoom"; region: [number, number, number, number]; } -export interface CuaActionGoto { +export interface ComputerActionGoto { type: "goto"; url: string; } -export interface CuaActionBack { +export interface ComputerActionBack { type: "back"; } -export interface CuaActionForward { +export interface ComputerActionForward { type: "forward"; } -export interface CuaActionUrl { +export interface ComputerActionUrl { type: "url"; } -export interface CuaActionCursorPosition { +export interface ComputerActionCursorPosition { type: "cursor_position"; } -export type CuaComputerAction = - | CuaActionClick - | CuaActionDoubleClick - | CuaActionMouseDown - | CuaActionMouseUp - | CuaActionTypeText - | CuaActionKeypress - | CuaActionScroll - | CuaActionMove - | CuaActionDrag - | CuaActionWait - | CuaActionScreenshot - | CuaActionZoom - | CuaActionGoto - | CuaActionBack - | CuaActionForward - | CuaActionUrl - | CuaActionCursorPosition; +export type ComputerAction = + | ComputerActionClick + | ComputerActionDoubleClick + | ComputerActionMouseDown + | ComputerActionMouseUp + | ComputerActionTypeText + | ComputerActionKeypress + | ComputerActionScroll + | ComputerActionMove + | ComputerActionDrag + | ComputerActionWait + | ComputerActionScreenshot + | ComputerActionZoom + | ComputerActionGoto + | ComputerActionBack + | ComputerActionForward + | ComputerActionUrl + | ComputerActionCursorPosition; const PointSchema = Type.Object( { @@ -180,7 +180,7 @@ const PointSchema = Type.Object( { additionalProperties: false }, ); -export const CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE = { +export const COMPUTER_ACTION_SCHEMA_BY_TYPE = { click: Type.Object( { type: Type.Literal("click"), @@ -296,6 +296,6 @@ export const CUA_COMPUTER_ACTION_SCHEMA_BY_TYPE = { forward: Type.Object({ type: Type.Literal("forward") }, { additionalProperties: false }), url: Type.Object({ type: Type.Literal("url") }, { additionalProperties: false }), cursor_position: Type.Object({ type: Type.Literal("cursor_position") }, { additionalProperties: false }), -} satisfies Record; +} satisfies Record; -export type CuaZoomRegion = CuaActionZoom["region"]; +export type ZoomRegion = ComputerActionZoom["region"]; diff --git a/packages/loop/src/core/actions/index.ts b/packages/loop/src/core/actions/index.ts new file mode 100644 index 00000000..b7789618 --- /dev/null +++ b/packages/loop/src/core/actions/index.ts @@ -0,0 +1,47 @@ +import type { TSchema } from "@earendil-works/pi-ai"; +import { BROWSER_ACTION_TYPES, createBrowserActionSchemaByType, type BrowserAction, type BrowserActionType, type BrowserActionSchemaOptions } from "./browser"; +import { COMPUTER_ACTION_SCHEMA_BY_TYPE, COMPUTER_ACTION_TYPES, type ComputerAction, type ComputerActionType } from "./computer"; + +export * from "./browser"; +export * from "./computer"; + +/** Any canonical action type, across the computer and browser planes. */ +export type ComputerUseActionType = ComputerActionType | BrowserActionType; + +/** Any canonical action, across the computer and browser planes. */ +export type ComputerUseAction = ComputerAction | BrowserAction; + +/** Every canonical action type: the computer plane followed by the browser plane. */ +export const COMPUTER_USE_ACTION_TYPES: readonly ComputerUseActionType[] = [...COMPUTER_ACTION_TYPES, ...BROWSER_ACTION_TYPES]; + +const COMPUTER_ACTION_TYPE_SET: ReadonlySet = new Set(COMPUTER_ACTION_TYPES); +const BROWSER_ACTION_TYPE_SET: ReadonlySet = new Set(BROWSER_ACTION_TYPES); + +/** Whether a canonical action type belongs to the computer plane. */ +export function isComputerActionType(action: ComputerUseActionType): action is ComputerActionType { + return COMPUTER_ACTION_TYPE_SET.has(action); +} + +/** Whether a canonical action type belongs to the browser plane. */ +export function isBrowserActionType(action: ComputerUseActionType): action is BrowserActionType { + return BROWSER_ACTION_TYPE_SET.has(action); +} + +/** Whether a canonical action belongs to the browser plane. */ +export function isBrowserAction(action: ComputerUseAction): action is BrowserAction { + return BROWSER_ACTION_TYPE_SET.has(action.type); +} + +/** Options for building canonical action schemas. */ +export interface ComputerUseActionSchemaOptions { + /** browser-plane schema variants; see {@link BrowserActionSchemaOptions}. Defaults to coordinates allowed. */ + browser?: BrowserActionSchemaOptions; +} + +/** Build the full action-type → schema map for a schema-options combination. */ +export function computerUseActionSchemaByType(options: ComputerUseActionSchemaOptions = {}): Record { + return { + ...COMPUTER_ACTION_SCHEMA_BY_TYPE, + ...createBrowserActionSchemaByType(options.browser ?? { coordinates: true }), + }; +} diff --git a/packages/agent/src/browser-result-format.ts b/packages/loop/src/core/browser-result-format.ts similarity index 100% rename from packages/agent/src/browser-result-format.ts rename to packages/loop/src/core/browser-result-format.ts diff --git a/packages/ai/src/menu.ts b/packages/loop/src/core/menu.ts similarity index 67% rename from packages/ai/src/menu.ts rename to packages/loop/src/core/menu.ts index 5a0c2308..d8a1227c 100644 --- a/packages/ai/src/menu.ts +++ b/packages/loop/src/core/menu.ts @@ -1,18 +1,18 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import { cua } from "./cua"; -import { getCuaModel, type CuaModelRef } from "./models"; -import { compileCuaToolCatalog, type CuaToolSpec } from "./tool-catalog"; +import { loop } from "./tools"; +import { getLoopModel, type LoopModelRef } from "../pi/models"; +import { compileLoopToolCatalog, type LoopToolSpec } from "./tool-catalog"; /** Where a menu entry comes from, for grouping in a picker. */ -export type CuaToolMenuGroup = "browser" | "computer" | "playwright" | "native"; +export type LoopToolMenuGroup = "browser" | "computer" | "playwright" | "native"; /** One offerable item: a single tool, or a native toolset selected as a unit. */ -export interface CuaToolMenuEntry { +export interface LoopToolMenuEntry { /** Stable key: the tool's catalog identity, or a `group:` key for a multi-tool entry. */ readonly key: string; /** Model-facing name, or a label for a multi-tool entry. */ readonly label: string; - readonly group: CuaToolMenuGroup; + readonly group: LoopToolMenuGroup; readonly description?: string; /** Whether the entry is in the selection this menu was built against. */ readonly selected: boolean; @@ -21,16 +21,16 @@ export interface CuaToolMenuEntry { /** Why it cannot be selected, verbatim from the catalog compiler. */ readonly unavailableReason?: string; /** The specs this entry contributes to a tool list. */ - readonly tools: readonly CuaToolSpec[]; + readonly tools: readonly LoopToolSpec[]; } /** - * Every tool CUA can offer for a model, marked available or not. + * Every tool Loop can offer for a model, marked available or not. * * Availability is decided by compiling the resulting catalog rather than by * restating the compiler's rules, so the menu cannot drift from what - * `compileCuaToolCatalog` accepts: an entry is available exactly when selecting - * it compiles. `compileCuaToolCatalog` is pure and declaration-only — it builds + * `compileLoopToolCatalog` accepts: an entry is available exactly when selecting + * it compiles. `compileLoopToolCatalog` is pure and declaration-only — it builds * no executable tools and retains none of its inputs — so probing it per entry * is cheap and free of side effects. * @@ -39,11 +39,11 @@ export interface CuaToolMenuEntry { * transport. Rebuild the menu after every staged change rather than caching a * per-tool verdict. */ -export function cuaToolMenu( - model: CuaModelRef | Model, - selected: readonly CuaToolSpec[] = [], -): CuaToolMenuEntry[] { - const resolved = typeof model === "string" ? getCuaModel(model) : model; +export function loopToolMenu( + model: LoopModelRef | Model, + selected: readonly LoopToolSpec[] = [], +): LoopToolMenuEntry[] { + const resolved = typeof model === "string" ? getLoopModel(model) : model; const selectedIdentities = new Set(selected.map((tool) => tool.identity)); return offerableEntries().map((entry) => { const isSelected = entry.tools.every((tool) => selectedIdentities.has(tool.identity)); @@ -64,9 +64,9 @@ export function cuaToolMenu( }); } -function compileFailure(model: Model, requestedTools: readonly CuaToolSpec[]): string | undefined { +function compileFailure(model: Model, requestedTools: readonly LoopToolSpec[]): string | undefined { try { - compileCuaToolCatalog({ model, requestedTools }); + compileLoopToolCatalog({ model, requestedTools }); return undefined; } catch (error) { return error instanceof Error ? error.message : String(error); @@ -76,30 +76,30 @@ function compileFailure(model: Model, requestedTools: readonly CuaToolSpec[ interface OfferableEntry { readonly key: string; readonly label: string; - readonly group: CuaToolMenuGroup; + readonly group: LoopToolMenuGroup; readonly description?: string; - readonly tools: readonly CuaToolSpec[]; + readonly tools: readonly LoopToolSpec[]; } /** * The full offerable surface, before any model is considered. Native entries - * are listed for every provider CUA has an adapter for; the compile probe is + * are listed for every provider Loop has an adapter for; the compile probe is * what decides which of them the selected model can actually take, so this list * carries no provider-name rule of its own. */ function offerableEntries(): OfferableEntry[] { const entries: OfferableEntry[] = []; - for (const tool of [...cua.toolsets.browser(), cua.tools.browser.act()]) { + for (const tool of [...loop.toolsets.browser(), loop.tools.browser.act()]) { entries.push(single(tool, "browser")); } - for (const tool of cua.toolsets.computer()) { + for (const tool of loop.toolsets.computer()) { entries.push(single(tool, "computer")); } - entries.push(single(cua.tools.playwright(), "playwright")); - entries.push(single(cua.providers.openai.tools.computer(), "native")); - entries.push(single(cua.providers.anthropic.tools.computer(), "native")); - entries.push(single(cua.providers.anthropic.tools.browser(), "native")); - const googleBrowser = cua.providers.google.toolsets.browser(); + entries.push(single(loop.tools.playwright(), "playwright")); + entries.push(single(loop.providers.openai.tools.computer(), "native")); + entries.push(single(loop.providers.anthropic.tools.computer(), "native")); + entries.push(single(loop.providers.anthropic.tools.browser(), "native")); + const googleBrowser = loop.providers.google.toolsets.browser(); entries.push({ key: "group:google.native.browser", label: "google native browser", @@ -110,7 +110,7 @@ function offerableEntries(): OfferableEntry[] { return entries; } -function single(tool: CuaToolSpec, group: CuaToolMenuGroup): OfferableEntry { +function single(tool: LoopToolSpec, group: LoopToolMenuGroup): OfferableEntry { const description = firstLine(tool.declaration.description); return { key: tool.identity, diff --git a/packages/agent/src/resources.ts b/packages/loop/src/core/resources.ts similarity index 92% rename from packages/agent/src/resources.ts rename to packages/loop/src/core/resources.ts index a6cc07f3..617948e4 100644 --- a/packages/agent/src/resources.ts +++ b/packages/loop/src/core/resources.ts @@ -1,18 +1,15 @@ import type { AgentTool, AgentToolResult } from "@earendil-works/pi-agent-core"; import type { ImageContent, TextContent } from "@earendil-works/pi-ai"; import type Kernel from "@onkernel/sdk"; -import { - type CuaAction, - type CuaCoordinateContract, - type CuaToolSpec, -} from "@onkernel/cua-ai"; +import type { ComputerUseAction } from "./actions/index"; +import type { LoopCoordinateContract, LoopToolSpec } from "./tool-catalog"; import { formatBrowserActResult } from "./browser-result-format"; import { BatchExecutionError, InternalComputerTranslator, type KernelBrowser, type PlaywrightExecutionResult } from "./translator/translator"; import type { BrowserExecutor } from "./translator/browser"; import type { BatchExecutionResult, BatchReadResult, BrowserWaitForResult } from "./translator/types"; -/** Structured execution metadata returned by materialized CUA tools. */ -export interface CuaExecutionDetails { +/** Structured execution metadata returned by materialized Loop tools. */ +export interface LoopExecutionDetails { statusText: string; readResults?: Array>; skippedActions?: number; @@ -31,12 +28,12 @@ type ToolContent = Array; * One per-agent browser resource pool. Tool catalogs may be rebuilt without * replacing this object, its lazy CDP connection, refs, or tab lifecycle. */ -export class CuaExecutionResources { +export class LoopExecutionResources { readonly browser: KernelBrowser; readonly client: Kernel; private readonly translator: InternalComputerTranslator; /** Each spec is materialized exactly once per resource pool. */ - private readonly materialized = new WeakMap(); + private readonly materialized = new WeakMap(); constructor(options: { browser: KernelBrowser; @@ -49,7 +46,7 @@ export class CuaExecutionResources { this.translator = new InternalComputerTranslator(options); } - materialize(spec: CuaToolSpec): AgentTool { + materialize(spec: LoopToolSpec): AgentTool { const cached = this.materialized.get(spec); if (cached) return cached; const definition = spec.declaration; @@ -69,7 +66,7 @@ export class CuaExecutionResources { return tool; } - async computer(actions: CuaAction[], coordinateContract: CuaCoordinateContract, signal?: AbortSignal): Promise { + async computer(actions: ComputerUseAction[], coordinateContract: LoopCoordinateContract, signal?: AbortSignal): Promise { return this.translator.executeBatch(actions, coordinateContract, signal); } @@ -85,7 +82,7 @@ export class CuaExecutionResources { this.translator.dispose(); } - private async executeActions(spec: CuaToolSpec, actions: CuaAction[], signal?: AbortSignal): Promise> { + private async executeActions(spec: LoopToolSpec, actions: ComputerUseAction[], signal?: AbortSignal): Promise> { if (spec.execution.kind !== "actions") throw new Error(`tool "${spec.name}" has no action executor`); let result: BatchExecutionResult; let failure: BatchExecutionError | undefined; @@ -127,7 +124,7 @@ export class CuaExecutionResources { }; } - private async executePlaywright(name: string, input: unknown): Promise> { + private async executePlaywright(name: string, input: unknown): Promise> { const parameters = asRecord(input); const code = parameters.code; if (typeof code !== "string") throw new Error(`${name} requires string code`); diff --git a/packages/ai/src/tool-catalog.ts b/packages/loop/src/core/tool-catalog.ts similarity index 82% rename from packages/ai/src/tool-catalog.ts rename to packages/loop/src/core/tool-catalog.ts index 60b7e1d2..5bbecab1 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/loop/src/core/tool-catalog.ts @@ -1,42 +1,42 @@ import type { Api, Model, Tool } from "@earendil-works/pi-ai"; -import type { CuaAction } from "./actions/index"; -import type { CuaModelRef } from "./models"; -import { cuaModelCapabilities, getCuaModel } from "./models"; -import { anthropicAdaptiveThinkingOnPayload } from "./providers/anthropic/adaptive-thinking"; +import type { ComputerUseAction } from "./actions/index"; +import type { LoopModelRef } from "../pi/models"; +import { loopModelCapabilities, getLoopModel } from "../pi/models"; +import { anthropicAdaptiveThinkingOnPayload } from "../pi/providers/anthropic/adaptive-thinking"; import { supportsAnthropicNativeBrowser, supportsAnthropicNativeComputer, -} from "./providers/anthropic/capabilities"; -import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; -import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; +} from "../pi/providers/anthropic/capabilities"; +import { GOOGLE_INTERACTIONS_API } from "../pi/providers/google/provider"; +import { OPENAI_COMPUTER_USE_API } from "../pi/providers/openai/provider"; -export const CUA_TOOL_SPEC_KIND = "@onkernel/cua-tool-spec/v1" as const; +export const LOOP_TOOL_SPEC_KIND = "@onkernel/loop-tool-spec/v1" as const; -export type CuaToolOrigin = "cua" | "provider-native"; -export type CuaToolTransport = "function" | "native"; -export type CuaToolDynamicLoading = "eligible" | "eager-only"; +export type LoopToolOrigin = "loop" | "provider-native"; +export type LoopToolTransport = "function" | "native"; +export type LoopToolDynamicLoading = "eligible" | "eager-only"; -export type CuaCoordinateContract = +export type LoopCoordinateContract = | { readonly type: "pixel" } | { readonly type: "normalized"; readonly range: readonly [number, number] }; -export type CuaToolExecution = +export type LoopToolExecution = | { readonly kind: "actions"; - readonly toActions: (input: unknown) => CuaAction[]; - readonly coordinates: CuaCoordinateContract; + readonly toActions: (input: unknown) => ComputerUseAction[]; + readonly coordinates: LoopCoordinateContract; readonly batch: boolean; /** Block later calls in the same assistant turn after this tool fails. */ readonly stopTurnOnFailureMessage?: string; } | { readonly kind: "playwright" }; -export type CuaProviderBinding = +export type LoopProviderBinding = | { readonly kind: "anthropic-native"; readonly declaration: Record; readonly beta: string; - readonly accessFallback?: CuaAnthropicBrowserFallback; + readonly accessFallback?: LoopAnthropicBrowserFallback; } | { readonly kind: "openai-native"; @@ -52,22 +52,22 @@ export type CuaProviderBinding = readonly requiresApi?: Api; }; -/** Declarative CUA tool. Identity is immutable and independent from its model-facing alias. */ -export interface CuaToolSpec { - readonly kind: typeof CUA_TOOL_SPEC_KIND; +/** Declarative Loop tool. Identity is immutable and independent from its model-facing alias. */ +export interface LoopToolSpec { + readonly kind: typeof LOOP_TOOL_SPEC_KIND; readonly identity: string; readonly preferredName: string; readonly name: string; - readonly origin: CuaToolOrigin; + readonly origin: LoopToolOrigin; /** First-party documentation for a provider-native tool surface. */ readonly source?: string; - readonly transport: CuaToolTransport; - readonly dynamicLoading: CuaToolDynamicLoading; + readonly transport: LoopToolTransport; + readonly dynamicLoading: LoopToolDynamicLoading; readonly declaration: Tool; - /** @internal Local execution policy consumed by @onkernel/cua-agent. */ - readonly execution: CuaToolExecution; + /** @internal Local execution policy consumed by the tool manager. */ + readonly execution: LoopToolExecution; /** @internal Provider transport contribution consumed by the catalog compiler. */ - readonly providerBinding?: CuaProviderBinding; + readonly providerBinding?: LoopProviderBinding; /** @internal True when the tool mutates shared browser state. */ readonly stateMutating: boolean; /** @internal Complex schemas are deliberately allowlisted by provider. */ @@ -82,14 +82,14 @@ export interface CuaToolSpec { } /** - * Sanitized declarative projection of a caller-owned tool. cua-ai never sees - * executors: callers pass plain pi-ai `Tool` declarations and the executing - * runtime (cua-agent) keeps the matching implementation. + * Sanitized declarative projection of a caller-owned tool. The compiler never + * sees executors: callers pass plain pi-ai `Tool` declarations and the executing + * runtime keeps the matching implementation. */ -export type CuaCallerToolDeclaration = Tool; +export type LoopCallerToolDeclaration = Tool; -/** Declarative catalog input: a CUA spec or a sanitized caller tool declaration. */ -export type CuaCatalogToolInput = CuaToolSpec | CuaCallerToolDeclaration; +/** Declarative catalog input: a Loop spec or a sanitized caller tool declaration. */ +export type LoopCatalogToolInput = LoopToolSpec | LoopCallerToolDeclaration; /** * Canonical identity scheme for caller-owned tools. Exported so every consumer @@ -99,31 +99,31 @@ export function callerToolIdentity(name: string): string { return `caller.${name}`; } -export interface CuaToolInfo { +export interface LoopToolInfo { identity: string; name: string; preferredName: string; - origin: "cua" | "provider-native" | "caller"; + origin: "loop" | "provider-native" | "caller"; source?: string; - transport: CuaToolTransport; - dynamicLoading: CuaToolDynamicLoading; + transport: LoopToolTransport; + dynamicLoading: LoopToolDynamicLoading; declaration: Tool | Record; - coordinates?: CuaCoordinateContract; + coordinates?: LoopCoordinateContract; } -export interface CuaHeaderRequirement { +export interface LoopHeaderRequirement { identity: string; name: string; value: string; merge: "exact" | "comma-set"; } -export interface CuaHeaderPlan { - readonly requirements: readonly CuaHeaderRequirement[]; +export interface LoopHeaderPlan { + readonly requirements: readonly LoopHeaderRequirement[]; merge(callerHeaders?: Record): Record | undefined; } -export interface CuaPayloadTransform { +export interface LoopPayloadTransform { identity: string; consumesToolIdentities?: readonly string[]; writes?: readonly string[]; @@ -131,21 +131,21 @@ export interface CuaPayloadTransform { apply(payload: unknown, model: Model, names: ReadonlyMap): unknown | Promise; } -export interface CuaPayloadPlan { - readonly transforms: readonly CuaPayloadTransform[]; +export interface LoopPayloadPlan { + readonly transforms: readonly LoopPayloadTransform[]; apply(payload: unknown, model: Model): Promise; } /** Function-tool fallback for an Anthropic native browser tool unavailable to the active credential. */ -export interface CuaAnthropicBrowserFallback { +export interface LoopAnthropicBrowserFallback { readonly beta: string; readonly nativeType: string; readonly declaration: Record; } -/** Identity-addressed native call dispatch passed to CUA custom provider streams. */ -export interface CuaIncomingToolPlan { - readonly anthropicBrowserFallback?: CuaAnthropicBrowserFallback; +/** Identity-addressed native call dispatch passed to Loop custom provider streams. */ +export interface LoopIncomingToolPlan { + readonly anthropicBrowserFallback?: LoopAnthropicBrowserFallback; readonly openaiComputerName?: string; readonly googleNames: Readonly>; /** Google predefined functions disabled by the exact selected native subset. */ @@ -153,37 +153,37 @@ export interface CuaIncomingToolPlan { readonly nativeToolNames: readonly string[]; } -export interface CuaToolCatalogEntry extends CuaToolInfo { +export interface LoopToolCatalogEntry extends LoopToolInfo { readonly schemaFingerprint: string; readonly fingerprint: string; } -export interface CuaToolCatalog { +export interface LoopToolCatalog { readonly model: Model; - readonly entries: readonly CuaToolCatalogEntry[]; + readonly entries: readonly LoopToolCatalogEntry[]; /** * Provider-facing pi-ai `Tool` declarations in entry order, suitable for * `Context.tools`. Native placeholders are swapped by `payload` transforms. */ readonly toolDeclarations: readonly Tool[]; - readonly headers: CuaHeaderPlan; - readonly payload: CuaPayloadPlan; - readonly incoming: CuaIncomingToolPlan; + readonly headers: LoopHeaderPlan; + readonly payload: LoopPayloadPlan; + readonly incoming: LoopIncomingToolPlan; readonly fingerprint: string; } -export interface CompileCuaToolCatalogOptions { - model: CuaModelRef | Model; - requestedTools: readonly CuaCatalogToolInput[]; +export interface CompileLoopToolCatalogOptions { + model: LoopModelRef | Model; + requestedTools: readonly LoopCatalogToolInput[]; } /** * Internal compilation state. The published catalog entry never retains the * requested spec/declaration objects or provider bindings used to compile it. */ -interface CuaCatalogEntryDraft extends CuaToolCatalogEntry { +interface LoopCatalogEntryDraft extends LoopToolCatalogEntry { readonly placeholder: Tool; - readonly providerBinding?: CuaProviderBinding; + readonly providerBinding?: LoopProviderBinding; readonly stateMutating?: boolean; readonly complexSchema?: boolean; readonly largeSchema?: boolean; @@ -203,9 +203,9 @@ const SAFE_TOOL_NAME = /^[A-Za-z][A-Za-z0-9_-]{0,63}$/; * no such tool selected keeps its ordinary registry `api`. Selecting tools * whose bindings require different transports fails to compile. */ -export function compileCuaToolCatalog(options: CompileCuaToolCatalogOptions): CuaToolCatalog { +export function compileLoopToolCatalog(options: CompileLoopToolCatalogOptions): LoopToolCatalog { const baseModel = typeof options.model === "string" - ? getCuaModel(options.model) + ? getLoopModel(options.model) : resetCatalogDerivedApi(options.model); const normalizedEntries = [...options.requestedTools].map(normalizeTool); const requiresApi = validateCatalog(baseModel, normalizedEntries); @@ -236,7 +236,7 @@ export function compileCuaToolCatalog(options: CompileCuaToolCatalogOptions): Cu } /** Strip internal compilation state from a published catalog entry. */ -function publishEntry(draft: CuaCatalogEntryDraft): CuaToolCatalogEntry { +function publishEntry(draft: LoopCatalogEntryDraft): LoopToolCatalogEntry { const { placeholder: _placeholder, providerBinding: _providerBinding, @@ -248,8 +248,8 @@ function publishEntry(draft: CuaCatalogEntryDraft): CuaToolCatalogEntry { return Object.freeze(entry); } -export function isCuaToolSpec(value: unknown): value is CuaToolSpec { - return Boolean(value && typeof value === "object" && (value as { kind?: unknown }).kind === CUA_TOOL_SPEC_KIND); +export function isLoopToolSpec(value: unknown): value is LoopToolSpec { + return Boolean(value && typeof value === "object" && (value as { kind?: unknown }).kind === LOOP_TOOL_SPEC_KIND); } export function modelSupportsDeferredTools(model: Model): boolean { @@ -264,8 +264,8 @@ export function modelSupportsDeferredTools(model: Model): boolean { return major > 4 || (major === 4 && minor >= 5); } -function normalizeTool(tool: CuaCatalogToolInput): CuaCatalogEntryDraft { - if (isCuaToolSpec(tool)) { +function normalizeTool(tool: LoopCatalogToolInput): LoopCatalogEntryDraft { + if (isLoopToolSpec(tool)) { const schemaFingerprint = stableStringify(tool.declaration.parameters); const fingerprint = stableStringify({ identity: tool.identity, @@ -313,7 +313,7 @@ function normalizeTool(tool: CuaCatalogToolInput): CuaCatalogEntryDraft { }); } -function resolveProviderFacingDeclarations(entries: readonly CuaCatalogEntryDraft[]): CuaCatalogEntryDraft[] { +function resolveProviderFacingDeclarations(entries: readonly LoopCatalogEntryDraft[]): LoopCatalogEntryDraft[] { const google = entries.filter((entry) => entry.providerBinding?.kind === "google-native"); const googleDeclaration = google.length > 0 ? (() => { const binding = google[0]!.providerBinding; @@ -337,10 +337,10 @@ function resolveProviderFacingDeclarations(entries: readonly CuaCatalogEntryDraf } /** Validate the requested catalog against the model and return the transport its selected tools require, if any. */ -function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { - const identities = new Map(); - const exactNames = new Map(); - const normalizedNames = new Map(); +function validateCatalog(model: Model, entries: readonly LoopCatalogEntryDraft[]): Api | undefined { + const identities = new Map(); + const exactNames = new Map(); + const normalizedNames = new Map(); const normalizeName = model.provider === "anthropic" ? (name: string) => name.toLowerCase() : (name: string) => name; for (const entry of entries) { @@ -369,15 +369,15 @@ function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDra function nameCollision( name: string, - first: CuaCatalogEntryDraft, - second: CuaCatalogEntryDraft, + first: LoopCatalogEntryDraft, + second: LoopCatalogEntryDraft, provider?: string, ): Error { const suffix = provider ? ` after ${provider} name normalization` : ""; return new Error(`tool name "${name}" is requested by both "${first.identity}" and "${second.identity}"${suffix}`); } -function validateToolCompatibility(model: Model, entry: CuaCatalogEntryDraft): void { +function validateToolCompatibility(model: Model, entry: LoopCatalogEntryDraft): void { const binding = entry.providerBinding; if (entry.origin === "provider-native" && !/^https:\/\//.test(entry.source ?? "")) { throw new Error(`${entry.identity} must cite first-party provider documentation`); @@ -389,7 +389,7 @@ function validateToolCompatibility(model: Model, entry: CuaCatalogEntryDraf throw new Error(`${entry.identity} requires a ${required} model; selected ${model.provider}:${model.id}`); } } - const capabilities = cuaModelCapabilities(model); + const capabilities = loopModelCapabilities(model); if (entry.complexSchema && !capabilities.acceptsComplexSchemas) { throw new Error(`provider ${model.provider} does not accept the schema used by "${entry.name}" (${entry.identity})`); } @@ -410,7 +410,7 @@ function validateAnthropicNativeModel(model: Model, identity: string): void } /** Validate the selected native tools agree on a provider and a transport, and return the transport they require, if any. */ -function validateToolsetCompatibility(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { +function validateToolsetCompatibility(model: Model, entries: readonly LoopCatalogEntryDraft[]): Api | undefined { const nativeProviderKinds = new Set( entries.flatMap((entry) => entry.providerBinding ? [entry.providerBinding.kind.split("-")[0]] : []), ); @@ -445,22 +445,22 @@ function validateToolsetCompatibility(model: Model, entries: readonly CuaCa } /** The transport a provider binding requires, if it declares one. Anthropic never forks transports and declares none. */ -function bindingRequiresApi(binding: CuaProviderBinding | undefined): readonly [Api] | readonly [] { +function bindingRequiresApi(binding: LoopProviderBinding | undefined): readonly [Api] | readonly [] { return binding && binding.kind !== "anthropic-native" && binding.requiresApi ? [binding.requiresApi] : []; } /** * Model-shaped default transport for each `requiresApi` this module can * derive, keyed by the derived api itself. A `Model` a caller passes to - * {@link compileCuaToolCatalog} may already carry one of these — e.g. a prior + * {@link compileLoopToolCatalog} may already carry one of these — e.g. a prior * catalog's `catalog.model`, fed back in with a different tool selection — so * derivation resets it here before re-validating, keeping compilation pure * with respect to the currently requested tools rather than pinning whatever * transport an earlier selection required. */ const CATALOG_DERIVED_API_DEFAULTS: Readonly> = { - [OPENAI_CUA_COMPUTER_API]: "openai-responses", - [GOOGLE_CUA_INTERACTIONS_API]: "google-generative-ai", + [OPENAI_COMPUTER_USE_API]: "openai-responses", + [GOOGLE_INTERACTIONS_API]: "google-generative-ai", }; function resetCatalogDerivedApi(model: Model): Model { @@ -468,7 +468,7 @@ function resetCatalogDerivedApi(model: Model): Model { return defaultApi ? { ...model, api: defaultApi } : model; } -function compileHeaderRequirements(entries: readonly CuaCatalogEntryDraft[]): CuaHeaderRequirement[] { +function compileHeaderRequirements(entries: readonly LoopCatalogEntryDraft[]): LoopHeaderRequirement[] { return entries.flatMap((entry) => { const binding = entry.providerBinding; return binding?.kind === "anthropic-native" @@ -477,7 +477,7 @@ function compileHeaderRequirements(entries: readonly CuaCatalogEntryDraft[]): Cu }); } -function createHeaderPlan(requirements: readonly CuaHeaderRequirement[]): CuaHeaderPlan { +function createHeaderPlan(requirements: readonly LoopHeaderRequirement[]): LoopHeaderPlan { const frozenRequirements = Object.freeze(requirements.map((requirement) => Object.freeze({ ...requirement }))); return Object.freeze({ requirements: frozenRequirements, @@ -511,8 +511,8 @@ function commaTokens(value: string | undefined): string[] { return value?.split(",").map((token) => token.trim()).filter(Boolean) ?? []; } -function compilePayloadTransforms(model: Model, entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform[] { - const transforms: CuaPayloadTransform[] = []; +function compilePayloadTransforms(model: Model, entries: readonly LoopCatalogEntryDraft[]): LoopPayloadTransform[] { + const transforms: LoopPayloadTransform[] = []; if (model.provider === "anthropic") { transforms.push({ identity: "provider.anthropic.model-preparation", @@ -547,7 +547,7 @@ function compilePayloadTransforms(model: Model, entries: readonly CuaCatalo transforms.push(createGeminiSchemaTransform()); } - if (cuaModelCapabilities(model).serializesStateMutations && entries.some((entry) => entry.stateMutating)) { + if (loopModelCapabilities(model).serializesStateMutations && entries.some((entry) => entry.stateMutating)) { transforms.push({ identity: `provider.${model.provider}.serial-tool-calls`, writes: ["parallel_tool_calls"], @@ -571,7 +571,7 @@ function compilePayloadTransforms(model: Model, entries: readonly CuaCatalo * never performs. Rewriting them is what lets Google take the same declarations * every other provider gets, verified against the live API. */ -function createGeminiSchemaTransform(): CuaPayloadTransform { +function createGeminiSchemaTransform(): LoopPayloadTransform { return { identity: "provider.google.function-declaration-schema", writes: ["tools.functionDeclarations"], @@ -614,10 +614,10 @@ function narrowToGeminiSchema(node: unknown): unknown { return result; } -function createGoogleTransform(entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform { +function createGoogleTransform(entries: readonly LoopCatalogEntryDraft[]): LoopPayloadTransform { const firstBinding = entries[0]!.providerBinding; if (firstBinding?.kind !== "google-native") throw new Error("invalid Google catalog entry"); - const selected = new Set(entries.map((entry) => (entry.providerBinding as Extract).nativeName)); + const selected = new Set(entries.map((entry) => (entry.providerBinding as Extract).nativeName)); const excludedPredefinedFunctions = firstBinding.allNativeNames.filter((name) => !selected.has(name)); return { identity: "provider.google.native.browser", @@ -638,8 +638,8 @@ function createGoogleTransform(entries: readonly CuaCatalogEntryDraft[]): CuaPay }; } -function validateTransformClaims(transforms: readonly CuaPayloadTransform[]): void { - const claims = new Map(); +function validateTransformClaims(transforms: readonly LoopPayloadTransform[]): void { + const claims = new Map(); for (const transform of transforms) { for (const write of transform.writes ?? []) { const existing = claims.get(write); @@ -653,10 +653,10 @@ function validateTransformClaims(transforms: readonly CuaPayloadTransform[]): vo function createPayloadPlan( model: Model, - transforms: readonly CuaPayloadTransform[], + transforms: readonly LoopPayloadTransform[], names: ReadonlyMap, -): CuaPayloadPlan { - const phases: Record = { +): LoopPayloadPlan { + const phases: Record = { "model-preparation": 0, "tool-declarations": 1, "provider-fields": 2, @@ -672,8 +672,8 @@ function createPayloadPlan( }); } -function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncomingToolPlan { - let anthropicBrowserFallback: CuaAnthropicBrowserFallback | undefined; +function compileIncomingPlan(entries: readonly LoopCatalogEntryDraft[]): LoopIncomingToolPlan { + let anthropicBrowserFallback: LoopAnthropicBrowserFallback | undefined; let openaiComputerName: string | undefined; let googleAllNativeNames: readonly string[] = []; const googleNames: Record = {}; diff --git a/packages/agent/src/tool-manager.ts b/packages/loop/src/core/tool-manager.ts similarity index 61% rename from packages/agent/src/tool-manager.ts rename to packages/loop/src/core/tool-manager.ts index 12d9fc11..e5945526 100644 --- a/packages/agent/src/tool-manager.ts +++ b/packages/loop/src/core/tool-manager.ts @@ -1,30 +1,29 @@ import type { AgentHarnessTool, AgentTool } from "@earendil-works/pi-agent-core"; import type { Api, Model } from "@earendil-works/pi-ai"; +import { getLoopModel, type LoopModelRef } from "../pi/models"; import { callerToolIdentity, - compileCuaToolCatalog, - getCuaModel, - isCuaToolSpec, - type CuaCatalogToolInput, - type CuaModelRef, - type CuaToolCatalog, - type CuaToolSpec, -} from "@onkernel/cua-ai"; -import { CuaExecutionResources } from "./resources"; + compileLoopToolCatalog, + isLoopToolSpec, + type LoopCatalogToolInput, + type LoopToolCatalog, + type LoopToolSpec, +} from "./tool-catalog"; +import { LoopExecutionResources } from "./resources"; /** - * Caller-owned tool: a declarative CUA spec materialized by this package, or an - * already executable pi `AgentTool`. Defined here because cua-agent is the only - * package that holds both halves; cua-ai compiles declaration-only catalogs. + * Caller-owned tool: a declarative Loop spec materialized by this package, or an + * already executable pi `AgentTool`. Defined here because the tool manager is the + * only place that holds both halves; the catalog compiler stays declaration-only. */ -export type CuaAgentTool = CuaToolSpec | AgentTool; +export type LoopAgentTool = LoopToolSpec | AgentTool; /** - * Caller-owned tool for a harness: a declarative CUA spec, or an executable pi + * Caller-owned tool for a harness: a declarative Loop spec, or an executable pi * `AgentHarnessTool` that receives the harness's tool context on every call. A * plain `AgentTool` is assignable, since it simply ignores the context. */ -export type CuaHarnessTool = CuaToolSpec | AgentHarnessTool; +export type LoopHarnessTool = LoopToolSpec | AgentHarnessTool; /** * One compiled (model, tools) pair: the caller's list joined back to its @@ -35,33 +34,33 @@ export type CuaHarnessTool = CuaToo * one, which is what lets a caller hand pi a fresh pair instead of mutating a * live catalog underneath it. */ -export class CuaToolManager = CuaAgentTool> { - readonly catalog: CuaToolCatalog; +export class LoopToolManager = LoopAgentTool> { + readonly catalog: LoopToolCatalog; private readonly executables: readonly (AgentTool | AgentHarnessTool)[]; - private readonly specs: ReadonlyMap; + private readonly specs: ReadonlyMap; constructor( - readonly resources: CuaExecutionResources, - model: CuaModelRef | Model, + readonly resources: LoopExecutionResources, + model: LoopModelRef | Model, requestedTools: readonly TRequested[], - resolveModel: (model: CuaModelRef) => Model = getCuaModel, + resolveModel: (model: LoopModelRef) => Model = getLoopModel, ) { - const inputs: CuaCatalogToolInput[] = []; + const inputs: LoopCatalogToolInput[] = []; const executables = new Map(); - const specs = new Map(); + const specs = new Map(); for (const tool of requestedTools) { - if (isCuaToolSpec(tool)) { + if (isLoopToolSpec(tool)) { inputs.push(tool); executables.set(tool.identity, tool); specs.set(tool.identity, tool); } else { // Fresh declaration-only projection: execute, label, prepareArguments, - // and executionMode never cross into cua-ai. + // and executionMode never cross into the catalog compiler. inputs.push({ name: tool.name, description: tool.description, parameters: tool.parameters }); executables.set(callerToolIdentity(tool.name), tool); } } - this.catalog = compileCuaToolCatalog({ + this.catalog = compileLoopToolCatalog({ model: typeof model === "string" ? resolveModel(model) : model, requestedTools: inputs, }); @@ -71,7 +70,7 @@ export class CuaToolManager = CuaAgentToo const executable = executables.get(entry.identity); if (!executable) throw new Error(`compiled catalog entry "${entry.identity}" has no matching requested tool`); executables.delete(entry.identity); - return isCuaToolSpec(executable) ? resources.materialize(executable) : (executable as AgentHarnessTool); + return isLoopToolSpec(executable) ? resources.materialize(executable) : (executable as AgentHarnessTool); }); if (executables.size > 0) { throw new Error(`requested tool(s) ${[...executables.keys()].join(", ")} missing from the compiled catalog`); @@ -90,7 +89,7 @@ export class CuaToolManager = CuaAgentToo } /** Execution metadata for one catalog identity. */ - specFor(identity: string): CuaToolSpec | undefined { + specFor(identity: string): LoopToolSpec | undefined { return this.specs.get(identity); } } diff --git a/packages/ai/src/cua.ts b/packages/loop/src/core/tools.ts similarity index 81% rename from packages/ai/src/cua.ts rename to packages/loop/src/core/tools.ts index 091a57e2..3a2a5c54 100644 --- a/packages/ai/src/cua.ts +++ b/packages/loop/src/core/tools.ts @@ -1,45 +1,45 @@ import { Type, type Tool, type TSchema } from "@earendil-works/pi-ai"; import { - CUA_COMPUTER_ACTION_TYPES, - createCuaBrowserActionSchemaByType, - type CuaAction, - type CuaBrowserActionType, - type CuaComputerActionType, + COMPUTER_ACTION_TYPES, + createBrowserActionSchemaByType, + type ComputerUseAction, + type BrowserActionType, + type ComputerActionType, } from "./actions/index"; -import { supportsAnthropicNativeBrowser } from "./providers/anthropic/capabilities"; -import { mapNativeBrowserInput, mapNativeComputerInput } from "./providers/anthropic/native"; -import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; -import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; +import { supportsAnthropicNativeBrowser } from "../pi/providers/anthropic/capabilities"; +import { mapNativeBrowserInput, mapNativeComputerInput } from "../pi/providers/anthropic/native"; +import { GOOGLE_INTERACTIONS_API } from "../pi/providers/google/provider"; +import { OPENAI_COMPUTER_USE_API } from "../pi/providers/openai/provider"; import { - CUA_TOOL_SPEC_KIND, - type CuaCoordinateContract, - type CuaProviderBinding, - type CuaToolDynamicLoading, - type CuaToolExecution, - type CuaToolOrigin, - type CuaToolSpec, - type CuaToolTransport, + LOOP_TOOL_SPEC_KIND, + type LoopCoordinateContract, + type LoopProviderBinding, + type LoopToolDynamicLoading, + type LoopToolExecution, + type LoopToolOrigin, + type LoopToolSpec, + type LoopToolTransport, } from "./tool-catalog"; -export interface CuaToolNameOptions { +export interface LoopToolNameOptions { /** Explicit model-facing alias. Provider-native fixed-name tools do not accept this option. */ name?: string; } -export interface CuaComputerToolOptions extends CuaToolNameOptions { - coordinates?: CuaCoordinateContract; +export interface LoopComputerToolOptions extends LoopToolNameOptions { + coordinates?: LoopCoordinateContract; } -export interface CuaToolsetOptions { +export interface LoopToolsetOptions { /** Deterministically prefixes every preferred name as `_`. */ namespace?: string; } -export interface CuaComputerToolsetOptions extends CuaToolsetOptions { - coordinates?: CuaCoordinateContract; +export interface LoopComputerToolsetOptions extends LoopToolsetOptions { + coordinates?: LoopCoordinateContract; } -export type CuaBrowserBatchAction = +export type LoopBrowserBatchAction = | "snapshot" | "text" | "find" @@ -58,13 +58,13 @@ export type CuaBrowserBatchAction = | "evaluate" | "wait_for"; -export interface CuaComputerBatchOptions extends CuaToolNameOptions { - actions: readonly CuaComputerActionType[]; - coordinates?: CuaCoordinateContract; +export interface LoopComputerBatchOptions extends LoopToolNameOptions { + actions: readonly ComputerActionType[]; + coordinates?: LoopCoordinateContract; } -export interface CuaBrowserBatchOptions extends CuaToolNameOptions { - actions: readonly CuaBrowserBatchAction[]; +export interface LoopBrowserBatchOptions extends LoopToolNameOptions { + actions: readonly LoopBrowserBatchAction[]; } const pixels = Object.freeze({ type: "pixel" as const }); @@ -75,20 +75,20 @@ const providerSources = Object.freeze({ google: "https://ai.google.dev/gemini-api/docs/computer-use", }); -function normalized(range: readonly [number, number]): CuaCoordinateContract { +function normalized(range: readonly [number, number]): LoopCoordinateContract { if (!Number.isFinite(range[0]) || !Number.isFinite(range[1]) || range[1] <= range[0]) { throw new Error("normalized coordinate range must contain two finite, ascending values"); } return Object.freeze({ type: "normalized", range: Object.freeze([range[0], range[1]] as const) }); } -function coordinateDescription(contract: CuaCoordinateContract): string { +function coordinateDescription(contract: LoopCoordinateContract): string { return contract.type === "pixel" ? "Coordinates are pixels in the latest OS screenshot." : `Coordinates are normalized to [${contract.range[0]}, ${contract.range[1]}] in the latest OS screenshot.`; } -const browserSchemas = createCuaBrowserActionSchemaByType({ coordinates: true }); +const browserSchemas = createBrowserActionSchemaByType({ coordinates: true }); const browserActionByFactory = { snapshot: "browser_snapshot", @@ -135,7 +135,7 @@ const computerFactoryAction = { type ComputerFactoryName = keyof typeof computerFactoryAction; -const browserDescriptions: Record = { +const browserDescriptions: Record = { browser_act: "Run 1–20 dependent browser actions with semantic expectations. Ref-based steps require current refs from browser_snapshot or browser_find; never invent refs, and re-snapshot after navigation or when a ref is stale. The plan stops at failed or unverifiable boundaries and returns causal outcomes plus a stable successor snapshot.", browser_snapshot: "Return an accessibility snapshot with element refs. Re-snapshot after navigation or when a ref is stale.", browser_text: "Return visible page text.", @@ -156,14 +156,14 @@ const browserDescriptions: Record = { browser_wait_for: "Wait for page text, element, URL, title, value, or state evidence without delivering input.", }; -function browserTool(factory: BrowserFactoryName, options: CuaToolNameOptions = {}): CuaToolSpec { +function browserTool(factory: BrowserFactoryName, options: LoopToolNameOptions = {}): LoopToolSpec { const action = browserActionByFactory[factory]; const preferredName = action; return createSpec({ - identity: `cua.browser.${action.slice("browser_".length).replaceAll("_", "-")}.v1`, + identity: `kloop.browser.${action.slice("browser_".length).replaceAll("_", "-")}.v1`, preferredName, name: options.name, - origin: "cua", + origin: "loop", declaration: { name: preferredName, description: browserDescriptions[action], @@ -171,7 +171,7 @@ function browserTool(factory: BrowserFactoryName, options: CuaToolNameOptions = }, execution: { kind: "actions", - toActions: (input) => [{ ...asInput(input), type: action } as CuaAction], + toActions: (input) => [{ ...asInput(input), type: action } as ComputerUseAction], coordinates: pixels, batch: false, }, @@ -181,16 +181,16 @@ function browserTool(factory: BrowserFactoryName, options: CuaToolNameOptions = }); } -function computerTool(factory: ComputerFactoryName, options: CuaComputerToolOptions = {}): CuaToolSpec { +function computerTool(factory: ComputerFactoryName, options: LoopComputerToolOptions = {}): LoopToolSpec { const action = computerFactoryAction[factory]; const preferredName = `computer_${action}`; const schema = computerSchema(action); const coordinates = options.coordinates ?? pixels; return createSpec({ - identity: `cua.computer.${action.replaceAll("_", "-")}.v1`, + identity: `kloop.computer.${action.replaceAll("_", "-")}.v1`, preferredName, name: options.name, - origin: "cua", + origin: "loop", declaration: { name: preferredName, description: `Execute one ${action} computer action. ${coordinateDescription(coordinates)}`, @@ -198,7 +198,7 @@ function computerTool(factory: ComputerFactoryName, options: CuaComputerToolOpti }, execution: { kind: "actions", - toActions: (input) => [{ ...asInput(input), type: action } as CuaAction], + toActions: (input) => [{ ...asInput(input), type: action } as ComputerUseAction], coordinates, batch: false, }, @@ -206,16 +206,16 @@ function computerTool(factory: ComputerFactoryName, options: CuaComputerToolOpti }); } -function computerBatch(options: CuaComputerBatchOptions): CuaToolSpec { +function computerBatch(options: LoopComputerBatchOptions): LoopToolSpec { if (!options.actions?.length) throw new Error("computer_batch actions must be non-empty"); const actions = uniqueKnownComputerActions(options.actions); const coordinates = options.coordinates ?? pixels; const actionSchemas = actions.map((action) => renameDiscriminator(computerActionSchema(action), "type", "action")); return createSpec({ - identity: "cua.computer.batch.v1", + identity: "kloop.computer.batch.v1", preferredName: "computer_batch", name: options.name, - origin: "cua", + origin: "loop", declaration: { name: "computer_batch", description: `Execute an explicit ordered sequence of computer-plane actions. Reads flush pending writes; execution stops at the first failure. ${coordinateDescription(coordinates)}`, @@ -225,7 +225,7 @@ function computerBatch(options: CuaComputerBatchOptions): CuaToolSpec { kind: "actions", toActions(input) { const value = asActionsInput(input); - return value.actions.map((action) => ({ ...action, type: requireString(action.action, "action") } as CuaAction)); + return value.actions.map((action) => ({ ...action, type: requireString(action.action, "action") } as ComputerUseAction)); }, coordinates, batch: true, @@ -234,7 +234,7 @@ function computerBatch(options: CuaComputerBatchOptions): CuaToolSpec { }); } -function browserBatch(options: CuaBrowserBatchOptions): CuaToolSpec { +function browserBatch(options: LoopBrowserBatchOptions): LoopToolSpec { if (!options.actions?.length) throw new Error("browser_batch actions must be non-empty"); const actions = [...new Set(options.actions)]; for (const action of actions) { @@ -245,10 +245,10 @@ function browserBatch(options: CuaBrowserBatchOptions): CuaToolSpec { return renameDiscriminator(browserSchemas[canonical], "type", "action", action); }); return createSpec({ - identity: "cua.browser.batch.v1", + identity: "kloop.browser.batch.v1", preferredName: "browser_batch", name: options.name, - origin: "cua", + origin: "loop", declaration: { name: "browser_batch", description: "Execute a mechanical ordered sequence of browser-plane operations over one shared ref table. Snapshot/find refresh refs before later actions; there is no interpolation or workflow syntax.", @@ -259,10 +259,10 @@ function browserBatch(options: CuaBrowserBatchOptions): CuaToolSpec { toActions(input) { const value = asActionsInput(input); return value.actions.map((action) => { - const canonical = browserBatchActionToCanonical(requireString(action.action, "action") as CuaBrowserBatchAction); + const canonical = browserBatchActionToCanonical(requireString(action.action, "action") as LoopBrowserBatchAction); if (!canonical) throw new Error(`unsupported browser_batch action "${String(action.action)}"`); const { action: _action, ...parameters } = action; - return { ...parameters, type: canonical } as CuaAction; + return { ...parameters, type: canonical } as ComputerUseAction; }); }, coordinates: pixels, @@ -273,12 +273,12 @@ function browserBatch(options: CuaBrowserBatchOptions): CuaToolSpec { }); } -function playwright(options: CuaToolNameOptions = {}): CuaToolSpec { +function playwright(options: LoopToolNameOptions = {}): LoopToolSpec { return createSpec({ - identity: "cua.playwright.v1", + identity: "kloop.playwright.v1", preferredName: "playwright_execute", name: options.name, - origin: "cua", + origin: "loop", declaration: { name: "playwright_execute", description: "Run Playwright/TypeScript against the live browser. page, context, and browser are in scope. No screenshot is returned automatically.", @@ -292,7 +292,7 @@ function playwright(options: CuaToolNameOptions = {}): CuaToolSpec { }); } -function anthropicNativeComputer(options: { version: "20260701"; enableZoom?: boolean; displayNumber?: number } = { version: "20260701" }): CuaToolSpec { +function anthropicNativeComputer(options: { version: "20260701"; enableZoom?: boolean; displayNumber?: number } = { version: "20260701" }): LoopToolSpec { if (options.version !== "20260701") throw new Error(`unsupported Anthropic native computer version "${String(options.version)}"`); const declaration = { type: "computer_20260701", @@ -312,7 +312,7 @@ function anthropicNativeComputer(options: { version: "20260701"; enableZoom?: bo }); } -function anthropicNativeBrowser(options: { version: "20260701"; javascript?: boolean } = { version: "20260701" }): CuaToolSpec { +function anthropicNativeBrowser(options: { version: "20260701"; javascript?: boolean } = { version: "20260701" }): LoopToolSpec { if (options.version !== "20260701") throw new Error(`unsupported Anthropic native browser version "${String(options.version)}"`); const declaration = { type: "browser_20260701", @@ -420,14 +420,14 @@ function anthropicNativeBrowserActionSchema(action: AnthropicNativeBrowserAction } } -function openaiNativeComputer(): CuaToolSpec { +function openaiNativeComputer(): LoopToolSpec { const declaration = { type: "computer" }; return providerNativeSpec({ identity: "provider.openai.native.computer.v1", name: "computer", source: providerSources.openai, declaration, - binding: { kind: "openai-native", declaration, requiresApi: OPENAI_CUA_COMPUTER_API }, + binding: { kind: "openai-native", declaration, requiresApi: OPENAI_COMPUTER_USE_API }, toActions: mapOpenAIComputerInput, coordinates: pixels, }); @@ -444,7 +444,7 @@ export interface GoogleBrowserToolsetOptions { exclude?: readonly string[]; } -function googleBrowserToolset(options: GoogleBrowserToolsetOptions = {}): CuaToolSpec[] { +function googleBrowserToolset(options: GoogleBrowserToolsetOptions = {}): LoopToolSpec[] { const unknown = (options.exclude ?? []).filter((name) => !GOOGLE_BROWSER_ACTIONS.includes(name as never)); if (unknown.length > 0) throw new Error(`unknown Google predefined browser action(s): ${unknown.join(", ")}`); const excluded = new Set(options.exclude ?? []); @@ -453,7 +453,7 @@ function googleBrowserToolset(options: GoogleBrowserToolsetOptions = {}): CuaToo name: nativeName, source: providerSources.google, declaration: { computerUse: { environment: "ENVIRONMENT_BROWSER" } }, - binding: { kind: "google-native", nativeName, allNativeNames: GOOGLE_BROWSER_ACTIONS, requiresApi: GOOGLE_CUA_INTERACTIONS_API }, + binding: { kind: "google-native", nativeName, allNativeNames: GOOGLE_BROWSER_ACTIONS, requiresApi: GOOGLE_INTERACTIONS_API }, toActions: (input) => mapGoogleAction(nativeName, asInput(input)), coordinates: normalized([0, 999]), })); @@ -464,11 +464,11 @@ function providerNativeSpec(options: { name: string; source: string; declaration: Record; - binding: CuaProviderBinding; - toActions: (input: unknown) => CuaAction[]; - coordinates: CuaCoordinateContract; + binding: LoopProviderBinding; + toActions: (input: unknown) => ComputerUseAction[]; + coordinates: LoopCoordinateContract; stopTurnOnFailureMessage?: string; -}): CuaToolSpec { +}): LoopToolSpec { return createSpec({ identity: options.identity, preferredName: options.name, @@ -497,21 +497,21 @@ function createSpec(options: { identity: string; preferredName: string; name?: string; - origin: CuaToolOrigin; + origin: LoopToolOrigin; source?: string; - transport?: CuaToolTransport; - dynamicLoading?: CuaToolDynamicLoading; + transport?: LoopToolTransport; + dynamicLoading?: LoopToolDynamicLoading; declaration: Tool; - execution: CuaToolExecution; - providerBinding?: CuaProviderBinding; + execution: LoopToolExecution; + providerBinding?: LoopProviderBinding; stateMutating: boolean; complexSchema?: boolean; largeSchema?: boolean; -}): CuaToolSpec { +}): LoopToolSpec { const name = options.name ?? options.preferredName; const declaration = Object.freeze({ ...options.declaration, name }); return Object.freeze({ - kind: CUA_TOOL_SPEC_KIND, + kind: LOOP_TOOL_SPEC_KIND, identity: options.identity, preferredName: options.preferredName, name, @@ -528,7 +528,7 @@ function createSpec(options: { }); } -function toolsetNameOptions(preferredName: string, options: CuaToolsetOptions): CuaToolNameOptions { +function toolsetNameOptions(preferredName: string, options: LoopToolsetOptions): LoopToolNameOptions { return options.namespace ? { name: namespaced(preferredName, options.namespace) } : {}; } @@ -536,23 +536,23 @@ function namespaced(preferredName: string, namespace: string | undefined): strin return namespace ? `${namespace}_${preferredName}` : preferredName; } -function computerSchema(action: CuaComputerActionType): TSchema { +function computerSchema(action: ComputerActionType): TSchema { return removeDiscriminator(computerActionSchema(action), "type"); } -function computerActionSchema(action: CuaComputerActionType): TSchema { - const schemas = Object.fromEntries(CUA_COMPUTER_ACTION_TYPES.map((name) => [name, computerSchemaWithType(name)])); +function computerActionSchema(action: ComputerActionType): TSchema { + const schemas = Object.fromEntries(COMPUTER_ACTION_TYPES.map((name) => [name, computerSchemaWithType(name)])); return schemas[action]!; } -function computerSchemaWithType(action: CuaComputerActionType): TSchema { +function computerSchemaWithType(action: ComputerActionType): TSchema { // Importing the canonical schema map directly would expose a mutable object through // the public namespace; rebuild the selected schema from the package's action map. const all = awaitlessComputerSchemas(); return all[action]; } -function awaitlessComputerSchemas(): Record { +function awaitlessComputerSchemas(): Record { // Kept as a function so factories never return the canonical object by reference. const point = { x: Type.Number(), y: Type.Number() }; return { @@ -602,14 +602,14 @@ function isSchemaRecord(value: unknown): value is Record { return Boolean(value && typeof value === "object" && !Array.isArray(value)); } -function uniqueKnownComputerActions(actions: readonly CuaComputerActionType[]): CuaComputerActionType[] { +function uniqueKnownComputerActions(actions: readonly ComputerActionType[]): ComputerActionType[] { const unique = [...new Set(actions)]; - for (const action of unique) if (!(CUA_COMPUTER_ACTION_TYPES as readonly string[]).includes(action)) throw new Error(`unsupported computer_batch action "${action}"`); + for (const action of unique) if (!(COMPUTER_ACTION_TYPES as readonly string[]).includes(action)) throw new Error(`unsupported computer_batch action "${action}"`); return unique; } -function browserBatchActionToCanonical(action: CuaBrowserBatchAction): CuaBrowserActionType | undefined { - const entries: Record = { +function browserBatchActionToCanonical(action: LoopBrowserBatchAction): BrowserActionType | undefined { + const entries: Record = { snapshot: "browser_snapshot", text: "browser_text", find: "browser_find", @@ -631,10 +631,10 @@ function browserBatchActionToCanonical(action: CuaBrowserBatchAction): CuaBrowse return entries[action]; } -function mapOpenAIComputerInput(input: unknown): CuaAction[] { +function mapOpenAIComputerInput(input: unknown): ComputerUseAction[] { const value = asInput(input); const actions = Array.isArray(value.actions) ? value.actions : value.action && typeof value.action === "object" ? [value.action] : [value]; - const result: CuaAction[] = []; + const result: ComputerUseAction[] = []; for (const action of actions) { if (!action || typeof action !== "object") continue; const current = action as Record; @@ -655,7 +655,7 @@ function mapOpenAIComputerInput(input: unknown): CuaAction[] { return result; } -function mapGoogleAction(name: string, input: Record): CuaAction[] { +function mapGoogleAction(name: string, input: Record): ComputerUseAction[] { const safety = input.safety_decision; if (safety && typeof safety === "object") { const decision = (safety as { decision?: unknown }).decision; @@ -676,7 +676,7 @@ function mapGoogleAction(name: string, input: Record): CuaActio case "move": return [{ type: "move", x: number(x), y: number(y) }]; case "type": return [ { type: "type", text: requireString(input.text, "text") }, - ...(input.press_enter === true ? [{ type: "keypress", keys: ["enter"] } as CuaAction] : []), + ...(input.press_enter === true ? [{ type: "keypress", keys: ["enter"] } as ComputerUseAction] : []), ]; case "scroll": { const amount = optionalNumber(input.magnitude_in_pixels) ?? 300; @@ -746,49 +746,49 @@ function point(value: unknown): { x: number; y: number } { } const browserTools = Object.freeze({ - snapshot: (options?: CuaToolNameOptions) => browserTool("snapshot", options), - text: (options?: CuaToolNameOptions) => browserTool("text", options), - find: (options?: CuaToolNameOptions) => browserTool("find", options), - click: (options?: CuaToolNameOptions) => browserTool("click", options), - hover: (options?: CuaToolNameOptions) => browserTool("hover", options), - drag: (options?: CuaToolNameOptions) => browserTool("drag", options), - fill: (options?: CuaToolNameOptions) => browserTool("fill", options), - scrollTo: (options?: CuaToolNameOptions) => browserTool("scrollTo", options), - scroll: (options?: CuaToolNameOptions) => browserTool("scroll", options), - type: (options?: CuaToolNameOptions) => browserTool("type", options), - key: (options?: CuaToolNameOptions) => browserTool("key", options), - navigate: (options?: CuaToolNameOptions) => browserTool("navigate", options), - listTabs: (options?: CuaToolNameOptions) => browserTool("listTabs", options), - newTab: (options?: CuaToolNameOptions) => browserTool("newTab", options), - screenshot: (options?: CuaToolNameOptions) => browserTool("screenshot", options), - evaluate: (options?: CuaToolNameOptions) => browserTool("evaluate", options), - waitFor: (options?: CuaToolNameOptions) => browserTool("waitFor", options), - act: (options?: CuaToolNameOptions) => browserTool("act", options), + snapshot: (options?: LoopToolNameOptions) => browserTool("snapshot", options), + text: (options?: LoopToolNameOptions) => browserTool("text", options), + find: (options?: LoopToolNameOptions) => browserTool("find", options), + click: (options?: LoopToolNameOptions) => browserTool("click", options), + hover: (options?: LoopToolNameOptions) => browserTool("hover", options), + drag: (options?: LoopToolNameOptions) => browserTool("drag", options), + fill: (options?: LoopToolNameOptions) => browserTool("fill", options), + scrollTo: (options?: LoopToolNameOptions) => browserTool("scrollTo", options), + scroll: (options?: LoopToolNameOptions) => browserTool("scroll", options), + type: (options?: LoopToolNameOptions) => browserTool("type", options), + key: (options?: LoopToolNameOptions) => browserTool("key", options), + navigate: (options?: LoopToolNameOptions) => browserTool("navigate", options), + listTabs: (options?: LoopToolNameOptions) => browserTool("listTabs", options), + newTab: (options?: LoopToolNameOptions) => browserTool("newTab", options), + screenshot: (options?: LoopToolNameOptions) => browserTool("screenshot", options), + evaluate: (options?: LoopToolNameOptions) => browserTool("evaluate", options), + waitFor: (options?: LoopToolNameOptions) => browserTool("waitFor", options), + act: (options?: LoopToolNameOptions) => browserTool("act", options), batch: browserBatch, }); const computerTools = Object.freeze({ - click: (options?: CuaComputerToolOptions) => computerTool("click", options), - doubleClick: (options?: CuaComputerToolOptions) => computerTool("doubleClick", options), - mouseDown: (options?: CuaComputerToolOptions) => computerTool("mouseDown", options), - mouseUp: (options?: CuaComputerToolOptions) => computerTool("mouseUp", options), - type: (options?: CuaComputerToolOptions) => computerTool("type", options), - keypress: (options?: CuaComputerToolOptions) => computerTool("keypress", options), - scroll: (options?: CuaComputerToolOptions) => computerTool("scroll", options), - move: (options?: CuaComputerToolOptions) => computerTool("move", options), - drag: (options?: CuaComputerToolOptions) => computerTool("drag", options), - wait: (options?: CuaComputerToolOptions) => computerTool("wait", options), - screenshot: (options?: CuaComputerToolOptions) => computerTool("screenshot", options), - zoom: (options?: CuaComputerToolOptions) => computerTool("zoom", options), - goto: (options?: CuaComputerToolOptions) => computerTool("goto", options), - back: (options?: CuaComputerToolOptions) => computerTool("back", options), - forward: (options?: CuaComputerToolOptions) => computerTool("forward", options), - url: (options?: CuaComputerToolOptions) => computerTool("url", options), - cursorPosition: (options?: CuaComputerToolOptions) => computerTool("cursorPosition", options), + click: (options?: LoopComputerToolOptions) => computerTool("click", options), + doubleClick: (options?: LoopComputerToolOptions) => computerTool("doubleClick", options), + mouseDown: (options?: LoopComputerToolOptions) => computerTool("mouseDown", options), + mouseUp: (options?: LoopComputerToolOptions) => computerTool("mouseUp", options), + type: (options?: LoopComputerToolOptions) => computerTool("type", options), + keypress: (options?: LoopComputerToolOptions) => computerTool("keypress", options), + scroll: (options?: LoopComputerToolOptions) => computerTool("scroll", options), + move: (options?: LoopComputerToolOptions) => computerTool("move", options), + drag: (options?: LoopComputerToolOptions) => computerTool("drag", options), + wait: (options?: LoopComputerToolOptions) => computerTool("wait", options), + screenshot: (options?: LoopComputerToolOptions) => computerTool("screenshot", options), + zoom: (options?: LoopComputerToolOptions) => computerTool("zoom", options), + goto: (options?: LoopComputerToolOptions) => computerTool("goto", options), + back: (options?: LoopComputerToolOptions) => computerTool("back", options), + forward: (options?: LoopComputerToolOptions) => computerTool("forward", options), + url: (options?: LoopComputerToolOptions) => computerTool("url", options), + cursorPosition: (options?: LoopComputerToolOptions) => computerTool("cursorPosition", options), batch: computerBatch, }); -function browserToolset(options: CuaToolsetOptions = {}): CuaToolSpec[] { +function browserToolset(options: LoopToolsetOptions = {}): LoopToolSpec[] { return [ browserTools.snapshot(toolsetNameOptions("browser_snapshot", options)), browserTools.text(toolsetNameOptions("browser_text", options)), @@ -810,8 +810,8 @@ function browserToolset(options: CuaToolsetOptions = {}): CuaToolSpec[] { ]; } -function computerToolset(options: CuaComputerToolsetOptions = {}): CuaToolSpec[] { - const toolOptions = (preferredName: string): CuaComputerToolOptions => ({ +function computerToolset(options: LoopComputerToolsetOptions = {}): LoopToolSpec[] { + const toolOptions = (preferredName: string): LoopComputerToolOptions => ({ ...toolsetNameOptions(preferredName, options), ...(options.coordinates ? { coordinates: options.coordinates } : {}), }); @@ -848,18 +848,18 @@ const providers = Object.freeze({ }), }); -/** Frozen, discoverable tool namespace shared by @onkernel/cua-ai and @onkernel/cua-agent. */ -export const cua = Object.freeze({ +/** Frozen, discoverable tool namespace of every tool this package declares. */ +export const loop = Object.freeze({ coordinates: Object.freeze({ pixels: () => pixels, normalized }), tools: Object.freeze({ browser: browserTools, computer: computerTools, playwright }), toolsets: Object.freeze({ browser: browserToolset, computer: computerToolset, - mixed(options: CuaComputerToolsetOptions = {}) { + mixed(options: LoopComputerToolsetOptions = {}) { return [...computerToolset(options), ...browserToolset(options)]; }, }), providers, }); -export type CuaNamespace = typeof cua; +export type LoopNamespace = typeof loop; diff --git a/packages/agent/src/translator/browser-act.ts b/packages/loop/src/core/translator/browser-act.ts similarity index 93% rename from packages/agent/src/translator/browser-act.ts rename to packages/loop/src/core/translator/browser-act.ts index 9c6d4685..ddbf5d19 100644 --- a/packages/agent/src/translator/browser-act.ts +++ b/packages/loop/src/core/translator/browser-act.ts @@ -1,4 +1,4 @@ -import type { CuaActionBrowserAct, CuaActionBrowserSnapshot, CuaBrowserActStep, CuaBrowserExpectation } from "@onkernel/cua-ai"; +import type { BrowserActionAct, BrowserActionSnapshot, BrowserActStep, BrowserExpectation } from "../actions/index"; import { diffObservations, type BrowserObservation, type BrowserPresentation } from "./browser-observation"; import type { BrowserExpectationEvaluation } from "./browser-wait"; import type { BrowserActExpectationEvidence, BrowserActResult, BrowserActStepResult, BrowserWaitForResult } from "./types"; @@ -26,10 +26,10 @@ export interface BrowserActRuntime { liveGeneration(frameId: string): number; liveNavigationEpoch(targetId: string): number; /** Stop starting input sub-operations when aborted and settle any in-flight atomic input before resolving. */ - executeStep(step: CuaBrowserActStep, tabId: string | undefined, signal: AbortSignal): Promise; - wait(expect: CuaBrowserExpectation, baseline: BrowserObservation, targetId: string, tabId?: string, timeoutMs?: number, pollMs?: number): Promise; - evaluate(expect: CuaBrowserExpectation, observation: BrowserObservation, baseline: BrowserObservation): BrowserExpectationEvaluation; - present(observation: BrowserObservation, action: CuaActionBrowserSnapshot): BrowserPresentation; + executeStep(step: BrowserActStep, tabId: string | undefined, signal: AbortSignal): Promise; + wait(expect: BrowserExpectation, baseline: BrowserObservation, targetId: string, tabId?: string, timeoutMs?: number, pollMs?: number): Promise; + evaluate(expect: BrowserExpectation, observation: BrowserObservation, baseline: BrowserObservation): BrowserExpectationEvaluation; + present(observation: BrowserObservation, action: BrowserActionSnapshot): BrowserPresentation; render(presentation: BrowserPresentation): string; } @@ -38,7 +38,7 @@ export interface BrowserActRuntime { * successor. Expectations are evaluated against observations captured before input; a * condition already matched before input is `preexisting`, never proof that input worked. */ -export async function runBrowserAct(action: CuaActionBrowserAct, runtime: BrowserActRuntime): Promise { +export async function runBrowserAct(action: BrowserActionAct, runtime: BrowserActRuntime): Promise { const finalStepIndex = action.steps.length - 1; const globalDeadline: ActDeadline = { at: Date.now() + (action.timeout_ms ?? DEFAULT_ACT_TIMEOUT_MS), reason: "global_timeout" }; let baseline: BrowserObservation; @@ -205,7 +205,7 @@ export async function runBrowserAct(action: CuaActionBrowserAct, runtime: Browse stoppedAt = undefined; } } - const complete: CuaActionBrowserSnapshot = { type: "browser_snapshot", tab_id: action.tab_id, depth: Number.MAX_SAFE_INTEGER }; + const complete: BrowserActionSnapshot = { type: "browser_snapshot", tab_id: action.tab_id, depth: Number.MAX_SAFE_INTEGER }; const presentation = runtime.present(observed, { type: "browser_snapshot", tab_id: action.tab_id, ...action.successor }); successor = { status: "observed", @@ -244,7 +244,7 @@ function stepOutcome(expectation: BrowserActExpectationEvidence | undefined, act * preexisting evidence, missing expectations, timeouts, and unsafe stops remain unknown. */ function planOutcome( - action: CuaActionBrowserAct, + action: BrowserActionAct, steps: readonly BrowserActStepResult[], finalExpectation: BrowserActExpectationEvidence | undefined, timedOut: boolean, @@ -260,7 +260,7 @@ function planOutcome( return verified && (!stopReason || verifiedNavigation) ? "worked" : "unknown"; } -function stepDeadline(step: CuaBrowserActStep, globalDeadline: ActDeadline): ActDeadline { +function stepDeadline(step: BrowserActStep, globalDeadline: ActDeadline): ActDeadline { if (step.timeout_ms === undefined) return globalDeadline; const at = Date.now() + step.timeout_ms; return at < globalDeadline.at ? { at, reason: "step_timeout" } : globalDeadline; @@ -380,7 +380,7 @@ function waitStopReason(result?: BrowserWaitForResult): BrowserActResult["stop_r return "control_flow"; } -function stepResult(index: number, step: CuaBrowserActStep, outcome: BrowserActStepResult["outcome"], diagnostics: string[], expectation?: BrowserActExpectationEvidence): BrowserActStepResult { +function stepResult(index: number, step: BrowserActStep, outcome: BrowserActStepResult["outcome"], diagnostics: string[], expectation?: BrowserActExpectationEvidence): BrowserActStepResult { return { index, type: step.type, outcome, diagnostics, ...(expectation ? { expectation } : {}) }; } diff --git a/packages/agent/src/translator/browser-document-reconciliation.ts b/packages/loop/src/core/translator/browser-document-reconciliation.ts similarity index 100% rename from packages/agent/src/translator/browser-document-reconciliation.ts rename to packages/loop/src/core/translator/browser-document-reconciliation.ts diff --git a/packages/agent/src/translator/browser-frame-collection.ts b/packages/loop/src/core/translator/browser-frame-collection.ts similarity index 100% rename from packages/agent/src/translator/browser-frame-collection.ts rename to packages/loop/src/core/translator/browser-frame-collection.ts diff --git a/packages/agent/src/translator/browser-observation.ts b/packages/loop/src/core/translator/browser-observation.ts similarity index 100% rename from packages/agent/src/translator/browser-observation.ts rename to packages/loop/src/core/translator/browser-observation.ts diff --git a/packages/agent/src/translator/browser-ref-lifecycle.ts b/packages/loop/src/core/translator/browser-ref-lifecycle.ts similarity index 99% rename from packages/agent/src/translator/browser-ref-lifecycle.ts rename to packages/loop/src/core/translator/browser-ref-lifecycle.ts index e113ccfc..fa461c9e 100644 --- a/packages/agent/src/translator/browser-ref-lifecycle.ts +++ b/packages/loop/src/core/translator/browser-ref-lifecycle.ts @@ -62,7 +62,7 @@ export interface GenerationCapture { } /** - * Serializable ref state, so refs minted in one process (e.g. a `cua + * Serializable ref state, so refs minted in one process (e.g. a `loop * snapshot` invocation) can be resolved in a later one against the same * browser. Session ids are process-local and deliberately not exported; * imported refs rebind lazily. Backend node ids stay valid for the life of diff --git a/packages/agent/src/translator/browser-wait.ts b/packages/loop/src/core/translator/browser-wait.ts similarity index 97% rename from packages/agent/src/translator/browser-wait.ts rename to packages/loop/src/core/translator/browser-wait.ts index d867b265..b77d3fa6 100644 --- a/packages/agent/src/translator/browser-wait.ts +++ b/packages/loop/src/core/translator/browser-wait.ts @@ -1,4 +1,4 @@ -import type { CuaBrowserExpectation } from "@onkernel/cua-ai"; +import type { BrowserExpectation } from "../actions/index"; import { observedNodes, staticTextRun, type AXNode, type BrowserObservation } from "./browser-observation"; import type { BrowserExpectationEvidence, BrowserWaitForResult, BrowserWaitReason } from "./types"; @@ -7,7 +7,7 @@ export interface BrowserExpectationEvaluation extends BrowserExpectationEvidence reason?: BrowserWaitReason; } -type RefExpectation = Extract; +type RefExpectation = Extract; /** Resolve a ref expectation against one structured observation. */ export type BrowserRefResolver = (expectation: RefExpectation, observation: BrowserObservation) => BrowserExpectationEvaluation; @@ -27,7 +27,7 @@ export interface BrowserWaitRuntime { /** Timeout, polling, target, and condition options for a semantic wait. */ export interface BrowserWaitOptions { - expect: CuaBrowserExpectation; + expect: BrowserExpectation; timeoutMs?: number; pollMs?: number; tabId?: string; @@ -52,7 +52,7 @@ function expectationNodes(observation: BrowserObservation): AXNode[] { /** Evaluate a semantic condition without minting refs. Unknown means the observation cannot prove the claim. */ export function evaluateBrowserExpectation( - expectation: CuaBrowserExpectation, + expectation: BrowserExpectation, observation: BrowserObservation, baseline: BrowserObservation, resolveRef: BrowserRefResolver, @@ -178,7 +178,7 @@ function satisfiedEvidence(initial: BrowserExpectationEvidence): BrowserWaitForR return initial.truth === true ? "preexisting" : "newly_verified"; } -function containsLocationExpectation(expectation: CuaBrowserExpectation): boolean { +function containsLocationExpectation(expectation: BrowserExpectation): boolean { if ("all" in expectation) return expectation.all.some(containsLocationExpectation); if ("any" in expectation) return expectation.any.some(containsLocationExpectation); return expectation.type === "url" || expectation.type === "title"; diff --git a/packages/agent/src/translator/browser.ts b/packages/loop/src/core/translator/browser.ts similarity index 96% rename from packages/agent/src/translator/browser.ts rename to packages/loop/src/core/translator/browser.ts index 8fbfe6fa..cb2685c1 100644 --- a/packages/agent/src/translator/browser.ts +++ b/packages/loop/src/core/translator/browser.ts @@ -1,21 +1,21 @@ -import { - normalizeGotoUrl, - type CuaActionBrowserAct, - type CuaActionBrowserClick, - type CuaActionBrowserDrag, - type CuaActionBrowserFill, - type CuaActionBrowserFind, - type CuaActionBrowserHover, - type CuaActionBrowserKey, - type CuaActionBrowserNavigate, - type CuaActionBrowserScroll, - type CuaActionBrowserScrollTo, - type CuaActionBrowserSnapshot, - type CuaActionBrowserWaitFor, - type CuaBrowserAction, - type CuaBrowserActStep, - type CuaBrowserExpectation, -} from "@onkernel/cua-ai"; +import { normalizeGotoUrl } from "../url"; +import type { + BrowserActionAct, + BrowserActionClick, + BrowserActionDrag, + BrowserActionFill, + BrowserActionFind, + BrowserActionHover, + BrowserActionKey, + BrowserActionNavigate, + BrowserActionScroll, + BrowserActionScrollTo, + BrowserActionSnapshot, + BrowserActionWaitFor, + BrowserAction, + BrowserActStep, + BrowserExpectation, +} from "../actions/index"; import { CdpConnection, type CdpEventMessage } from "./cdp"; import { BrowserDocumentReconciler, @@ -383,7 +383,7 @@ export class BrowserExecutor { this.lifecycle.importState(state.generations, state.refs, state.activeTargetId, state.documents); } - async execute(action: CuaBrowserAction, signal?: AbortSignal): Promise { + async execute(action: BrowserAction, signal?: AbortSignal): Promise { throwIfAborted(signal); const results = await this.dispatch(action, signal); const dialogs = this.drainDialogNotes(); @@ -391,7 +391,7 @@ export class BrowserExecutor { return results; } - private async dispatch(action: CuaBrowserAction, signal?: AbortSignal): Promise { + private async dispatch(action: BrowserAction, signal?: AbortSignal): Promise { switch (action.type) { case "browser_snapshot": return [{ type: "browser_text", label: "snapshot", text: await this.snapshot(action) }]; @@ -459,7 +459,7 @@ export class BrowserExecutor { return { data: Buffer.from(data, "base64"), mimeType: "image/png" }; } - private async snapshot(action: CuaActionBrowserSnapshot): Promise { + private async snapshot(action: BrowserActionSnapshot): Promise { return this.withObservation(action.tab_id, true, (observation) => this.renderObservation(this.presentObservation(observation, action)), ); @@ -469,11 +469,11 @@ export class BrowserExecutor { return this.withObservation(tabId, includeCursor, (observation) => observation); } - private waitFor(action: CuaActionBrowserWaitFor): Promise { + private waitFor(action: BrowserActionWaitFor): Promise { return this.waitForExpectation(action.expect, { timeoutMs: action.timeout_ms, pollMs: action.poll_ms, tabId: action.tab_id }); } - private waitForExpectation(expect: CuaBrowserExpectation, options: { timeoutMs?: number; pollMs?: number; tabId?: string; baseline?: BrowserObservation; targetId?: string }): Promise { + private waitForExpectation(expect: BrowserExpectation, options: { timeoutMs?: number; pollMs?: number; tabId?: string; baseline?: BrowserObservation; targetId?: string }): Promise { return waitForBrowserExpectation({ selectTarget: (tabId) => this.resolveTarget(tabId), observeTarget: (targetId) => this.observe(targetId, false), @@ -483,7 +483,7 @@ export class BrowserExecutor { }, { expect, ...options }); } - private act(action: CuaActionBrowserAct): Promise { + private act(action: BrowserActionAct): Promise { return runBrowserAct(action, { observe: (tabId) => this.observe(tabId, false), targetIds: async () => (await this.cdp.pageTargets()).map((target) => target.targetId).sort(), @@ -498,7 +498,7 @@ export class BrowserExecutor { }); } - private async executeActStep(step: CuaBrowserActStep, tabId: string | undefined, signal: AbortSignal): Promise { + private async executeActStep(step: BrowserActStep, tabId: string | undefined, signal: AbortSignal): Promise { throwIfAborted(signal); switch (step.type) { case "click": return this.click({ type: "browser_click", ref: step.ref, button: step.button, num_clicks: step.num_clicks, modifiers: step.modifiers, tab_id: tabId }, signal); @@ -517,7 +517,7 @@ export class BrowserExecutor { } private evaluateRefExpectation( - expectation: Extract, + expectation: Extract, observation: BrowserObservation, ): BrowserExpectationEvaluation { const entry = this.refs.get(expectation.ref); @@ -629,7 +629,7 @@ export class BrowserExecutor { } } - private presentObservation(observation: BrowserObservation, action: CuaActionBrowserSnapshot): BrowserPresentation { + private presentObservation(observation: BrowserObservation, action: BrowserActionSnapshot): BrowserPresentation { const refEntry = action.ref ? this.resolveRef(action.ref, observation.targetId) : undefined; let tree = observation.tree; let rootIds = tree.roots; @@ -838,7 +838,7 @@ export class BrowserExecutor { return ids; } - private async find(action: CuaActionBrowserFind): Promise { + private async find(action: BrowserActionFind): Promise { const candidates = await this.findCandidates(action.query, action.tab_id); if (candidates.length === 0) return `No elements matched ${JSON.stringify(action.query)}. Try snapshot for the full tree.`; return candidates @@ -879,7 +879,7 @@ export class BrowserExecutor { }); } - private async click(action: CuaActionBrowserClick, signal?: AbortSignal): Promise { + private async click(action: BrowserActionClick, signal?: AbortSignal): Promise { throwIfAborted(signal); const targetId = await this.resolveTarget(action.tab_id); throwIfAborted(signal); @@ -910,7 +910,7 @@ export class BrowserExecutor { } } - private async hover(action: CuaActionBrowserHover, signal?: AbortSignal): Promise { + private async hover(action: BrowserActionHover, signal?: AbortSignal): Promise { throwIfAborted(signal); const targetId = await this.resolveTarget(action.tab_id); throwIfAborted(signal); @@ -921,14 +921,14 @@ export class BrowserExecutor { await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: point.x, y: point.y }, point.session); } - private async drag(action: CuaActionBrowserDrag): Promise { + private async drag(action: BrowserActionDrag): Promise { const session = await this.session(tabOf(action)); await this.cdp.send("Input.dispatchMouseEvent", { type: "mousePressed", x: action.from.x, y: action.from.y, button: "left", clickCount: 1 }, session); await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: action.to.x, y: action.to.y, button: "left" }, session); await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseReleased", x: action.to.x, y: action.to.y, button: "left", clickCount: 1 }, session); } - private async fill(action: CuaActionBrowserFill, signal?: AbortSignal): Promise { + private async fill(action: BrowserActionFill, signal?: AbortSignal): Promise { throwIfAborted(signal); const targetId = await this.resolveTarget(action.tab_id); // Attach before resolving the ref (like click/hover) so any imported document @@ -957,7 +957,7 @@ export class BrowserExecutor { } } - private async scrollTo(action: CuaActionBrowserScrollTo, signal?: AbortSignal): Promise { + private async scrollTo(action: BrowserActionScrollTo, signal?: AbortSignal): Promise { throwIfAborted(signal); const targetId = await this.resolveTarget(action.tab_id); // Reconcile the imported document before resolving the ref; see fill(). @@ -969,7 +969,7 @@ export class BrowserExecutor { await this.scrollIntoView(entry, action.ref, session); } - private async scroll(action: CuaActionBrowserScroll): Promise { + private async scroll(action: BrowserActionScroll): Promise { const session = await this.session(tabOf(action)); const notches = action.amount ?? 3; const delta = Math.trunc(notches) * SCROLL_NOTCH_PX; @@ -978,7 +978,7 @@ export class BrowserExecutor { await this.cdp.send("Input.dispatchMouseEvent", { type: "mouseWheel", x: action.x, y: action.y, deltaX, deltaY }, session); } - private async key(action: CuaActionBrowserKey, signal?: AbortSignal): Promise { + private async key(action: BrowserActionKey, signal?: AbortSignal): Promise { throwIfAborted(signal); const session = await this.session(tabOf(action)); const repeat = Math.min(Math.max(1, Math.trunc(action.repeat ?? 1)), 100); @@ -1003,7 +1003,7 @@ export class BrowserExecutor { await this.cdp.send("Input.dispatchKeyEvent", { type: "keyUp", ...base }, session); } - private async navigate(action: CuaActionBrowserNavigate, signal?: AbortSignal): Promise { + private async navigate(action: BrowserActionNavigate, signal?: AbortSignal): Promise { throwIfAborted(signal); const targetId = await this.resolveTarget(action.tab_id); const session = await this.attach(targetId); @@ -1133,7 +1133,7 @@ export class BrowserExecutor { } private async resolvePoint( - action: CuaActionBrowserClick | CuaActionBrowserHover, + action: BrowserActionClick | BrowserActionHover, targetId: string, session: string, ): Promise<{ x: number; y: number; session: string }> { @@ -1508,7 +1508,7 @@ const FILL_FUNCTION = `function(value) { el.dispatchEvent(new Event("change", { bubbles: true })); }`; -const CURSOR_SCAN_GROUP = "cua-cursor-scan"; +const CURSOR_SCAN_GROUP = "loop-cursor-scan"; const CURSOR_POINTER_SCAN = `(() => { const matches = []; diff --git a/packages/agent/src/translator/cdp.ts b/packages/loop/src/core/translator/cdp.ts similarity index 100% rename from packages/agent/src/translator/cdp.ts rename to packages/loop/src/core/translator/cdp.ts diff --git a/packages/agent/src/translator/keys.ts b/packages/loop/src/core/translator/keys.ts similarity index 100% rename from packages/agent/src/translator/keys.ts rename to packages/loop/src/core/translator/keys.ts diff --git a/packages/agent/src/translator/translator.ts b/packages/loop/src/core/translator/translator.ts similarity index 85% rename from packages/agent/src/translator/translator.ts rename to packages/loop/src/core/translator/translator.ts index ff02ca12..957ac85a 100644 --- a/packages/agent/src/translator/translator.ts +++ b/packages/loop/src/core/translator/translator.ts @@ -1,24 +1,24 @@ import type Kernel from "@onkernel/sdk"; import type { BrowserCreateResponse, BrowserRetrieveResponse } from "@onkernel/sdk/resources/browsers"; -import { - isCuaBrowserAction, - normalizeGotoUrl, - type CuaCoordinateContract, - type CuaAction, - type CuaActionClick, - type CuaActionDoubleClick, - type CuaActionDrag, - type CuaActionMouseDown, - type CuaActionMouseUp, - type CuaActionMove, - type CuaActionScroll, - type CuaActionTypeText, - type CuaActionWait, - type CuaActionZoom, - type CuaBrowserAction, - type CuaDragMouseButton, - type CuaMouseButton, -} from "@onkernel/cua-ai"; +import { isBrowserAction } from "../actions/index"; +import type { LoopCoordinateContract } from "../tool-catalog"; +import { normalizeGotoUrl } from "../url"; +import type { + ComputerUseAction, + ComputerActionClick, + ComputerActionDoubleClick, + ComputerActionDrag, + ComputerActionMouseDown, + ComputerActionMouseUp, + ComputerActionMove, + ComputerActionScroll, + ComputerActionTypeText, + ComputerActionWait, + ComputerActionZoom, + BrowserAction, + DragMouseButton, + MouseButton, +} from "../actions/index"; import sharp from "sharp"; import { BrowserExecutor } from "./browser"; import { isKernelModifierKey, normalizeKernelKey, normalizeKernelKeyCombo } from "./keys"; @@ -106,8 +106,8 @@ export class InternalComputerTranslator { } async executeBatch( - actions: CuaAction[], - coordinateSystem: CuaCoordinateContract = { type: "pixel" }, + actions: ComputerUseAction[], + coordinateSystem: LoopCoordinateContract = { type: "pixel" }, signal?: AbortSignal, ): Promise { const result: BatchExecutionResult = { readResults: [] }; @@ -125,7 +125,7 @@ export class InternalComputerTranslator { actionIndex = index; throwIfAborted(signal); const action = actions[index]!; - if (isCuaBrowserAction(action)) { + if (isBrowserAction(action)) { await flush(); const reads = await this.browser().execute(action, signal); result.readResults.push(...reads); @@ -202,8 +202,8 @@ export class InternalComputerTranslator { /** Crop the OS screenshot to a region; coordinates stay in the full-screenshot frame. */ async zoom( - action: CuaActionZoom, - coordinateSystem: CuaCoordinateContract = { type: "pixel" }, + action: ComputerActionZoom, + coordinateSystem: LoopCoordinateContract = { type: "pixel" }, ): Promise<{ data: Buffer; mimeType: string }> { const screenshot = await this.screenshot(); const [rawX0, rawY0, rawX1, rawY1] = action.region; @@ -218,8 +218,8 @@ export class InternalComputerTranslator { } private toSdkAction( - action: Exclude, - coordinateSystem: CuaCoordinateContract, + action: Exclude, + coordinateSystem: LoopCoordinateContract, ): KernelBatchAction { switch (action.type) { case "click": @@ -251,9 +251,9 @@ export class InternalComputerTranslator { } private clickAction( - action: CuaActionClick | CuaActionDoubleClick | CuaActionMouseDown | CuaActionMouseUp, - extra: { button?: CuaMouseButton; num_clicks?: number; click_type?: "down" | "up" }, - coordinateSystem: CuaCoordinateContract, + action: ComputerActionClick | ComputerActionDoubleClick | ComputerActionMouseDown | ComputerActionMouseUp, + extra: { button?: MouseButton; num_clicks?: number; click_type?: "down" | "up" }, + coordinateSystem: LoopCoordinateContract, ): KernelBatchAction { const point = this.toViewportPoint(action.x ?? 0, action.y ?? 0, coordinateSystem); return { @@ -267,7 +267,7 @@ export class InternalComputerTranslator { }; } - private scrollAction(action: CuaActionScroll, coordinateSystem: CuaCoordinateContract): KernelBatchAction { + private scrollAction(action: ComputerActionScroll, coordinateSystem: LoopCoordinateContract): KernelBatchAction { const point = this.toViewportPoint(action.x ?? 0, action.y ?? 0, coordinateSystem); return { type: "scroll", @@ -281,12 +281,12 @@ export class InternalComputerTranslator { }; } - private moveAction(action: CuaActionMove, coordinateSystem: CuaCoordinateContract): KernelBatchAction { + private moveAction(action: ComputerActionMove, coordinateSystem: LoopCoordinateContract): KernelBatchAction { const point = this.toViewportPoint(action.x, action.y, coordinateSystem); return { type: "move_mouse", move_mouse: { x: point.x, y: point.y } }; } - private dragAction(action: CuaActionDrag, coordinateSystem: CuaCoordinateContract): KernelBatchAction { + private dragAction(action: ComputerActionDrag, coordinateSystem: LoopCoordinateContract): KernelBatchAction { return { type: "drag_mouse", drag_mouse: { @@ -303,7 +303,7 @@ export class InternalComputerTranslator { private toViewportPoint( x: number, y: number, - coordinateSystem: CuaCoordinateContract, + coordinateSystem: LoopCoordinateContract, ): { x: number; y: number } { if (coordinateSystem.type === "pixel") return { x: Math.trunc(x), y: Math.trunc(y) }; const [min, max] = coordinateSystem.range; @@ -328,25 +328,25 @@ export type PlaywrightExecutionResult = const PLAYWRIGHT_MAX_TIMEOUT_SEC = 300; -const CLICK_BUTTONS: ReadonlySet = new Set(["left", "right", "middle", "back", "forward"]); -const DRAG_BUTTONS: ReadonlySet = new Set(["left", "right", "middle"]); +const CLICK_BUTTONS: ReadonlySet = new Set(["left", "right", "middle", "back", "forward"]); +const DRAG_BUTTONS: ReadonlySet = new Set(["left", "right", "middle"]); // The wire schemas keep button as an open string for provider compatibility; -// per the documented CuaMouseButton contract, values outside the set coerce +// per the documented MouseButton contract, values outside the set coerce // to "left". -function mouseButton(value: string | undefined): CuaMouseButton { - return value !== undefined && CLICK_BUTTONS.has(value) ? (value as CuaMouseButton) : "left"; +function mouseButton(value: string | undefined): MouseButton { + return value !== undefined && CLICK_BUTTONS.has(value) ? (value as MouseButton) : "left"; } -function dragButton(value: string | undefined): CuaDragMouseButton { - return value !== undefined && DRAG_BUTTONS.has(value) ? (value as CuaDragMouseButton) : "left"; +function dragButton(value: string | undefined): DragMouseButton { + return value !== undefined && DRAG_BUTTONS.has(value) ? (value as DragMouseButton) : "left"; } -function typeText(action: CuaActionTypeText): KernelBatchAction { +function typeText(action: ComputerActionTypeText): KernelBatchAction { return { type: "type_text", type_text: { text: action.text } }; } -function waitAction(action: CuaActionWait): KernelBatchAction { +function waitAction(action: ComputerActionWait): KernelBatchAction { return { type: "sleep", sleep: { duration_ms: Math.trunc(action.ms ?? 1000) } }; } diff --git a/packages/agent/src/translator/types.ts b/packages/loop/src/core/translator/types.ts similarity index 100% rename from packages/agent/src/translator/types.ts rename to packages/loop/src/core/translator/types.ts diff --git a/packages/loop/src/core/url.ts b/packages/loop/src/core/url.ts new file mode 100644 index 00000000..680558e3 --- /dev/null +++ b/packages/loop/src/core/url.ts @@ -0,0 +1,7 @@ +/** Prefix a bare hostname/path before browser navigation. */ +export function normalizeGotoUrl(value: unknown): string | undefined { + if (typeof value !== "string") return undefined; + const url = value.trim(); + if (!url) return undefined; + return /^[a-z][a-z0-9+.-]*:\/\//i.test(url) ? url : `https://${url}`; +} diff --git a/packages/loop/src/index.ts b/packages/loop/src/index.ts new file mode 100644 index 00000000..9eede9d1 --- /dev/null +++ b/packages/loop/src/index.ts @@ -0,0 +1,34 @@ +export * from "./core/actions/index"; +export * from "./core/menu"; +export * from "./core/tool-catalog"; +export * from "./core/tools"; +export { normalizeGotoUrl } from "./core/url"; + +export type { LoopAgentTool, LoopHarnessTool } from "./core/tool-manager"; +export { LoopExecutionResources } from "./core/resources"; +export { formatBrowserActResult } from "./core/browser-result-format"; +export type { KernelBrowser } from "./core/translator/translator"; +export { InternalComputerTranslator } from "./core/translator/translator"; +export { CdpConnection } from "./core/translator/cdp"; +export { BrowserExecutor } from "./core/translator/browser"; +export type { BrowserFindCandidate } from "./core/translator/browser"; +export type { BrowserRefState } from "./core/translator/browser-ref-lifecycle"; +export type { + BatchExecutionResult, + BatchReadResult, + BrowserActExpectationEvidence, + BrowserActExpectationStatus, + BrowserActObservedSuccessor, + BrowserActOutcome, + BrowserActResult, + BrowserActStepResult, + BrowserActStopReason, + BrowserActSuccessor, + BrowserActUnavailableSuccessor, + BrowserExpectationEvidence, + BrowserExpectationState, + BrowserObservationDiff, + BrowserObservationDiffEntry, + BrowserWaitForResult, + BrowserWaitReason, +} from "./core/translator/types"; diff --git a/packages/pi-extension/src/browser-runtime.ts b/packages/loop/src/pi-extension/browser-runtime.ts similarity index 79% rename from packages/pi-extension/src/browser-runtime.ts rename to packages/loop/src/pi-extension/browser-runtime.ts index 6440363a..503fa31d 100644 --- a/packages/pi-extension/src/browser-runtime.ts +++ b/packages/loop/src/pi-extension/browser-runtime.ts @@ -1,5 +1,5 @@ import Kernel from "@onkernel/sdk"; -import { CuaExecutionResources } from "@onkernel/cua-agent"; +import { LoopExecutionResources } from "@onkernel/loop"; /** * Browser configuration, passed as one JSON object rather than a flag per field. @@ -28,15 +28,15 @@ export interface BrowserStatus { * Lazily provisions one browser for one pi session. Attached sessions are never * deleted. * - * This holds `CuaExecutionResources` rather than a full `attach()` handle: pi + * This holds `LoopExecutionResources` rather than a full `attach()` handle: pi * owns the model collection and the agent loop here, so the handle's `models` * wrapper, retry, and harness behaviors have nothing to attach to. What the * extension needs is the executor and the catalog compiler — the part of the * library that is not pi-shaped. */ -export class CuaBrowserRuntime { - private pending?: Promise; - private resources?: CuaExecutionResources; +export class LoopBrowserRuntime { + private pending?: Promise; + private resources?: LoopExecutionResources; private client?: Kernel; private status: BrowserStatus = {}; private closed = false; @@ -48,14 +48,14 @@ export class CuaBrowserRuntime { getStatus(): BrowserStatus { return { ...this.status }; } - async get(signal?: AbortSignal): Promise { - if (signal?.aborted) throw new Error("CUA browser provisioning cancelled"); - if (this.closed) throw new Error("CUA browser runtime is closed"); + async get(signal?: AbortSignal): Promise { + if (signal?.aborted) throw new Error("Loop browser provisioning cancelled"); + if (this.closed) throw new Error("Loop browser runtime is closed"); if (this.resources) return this.resources; this.pending ??= this.provision(); try { const resources = await this.pending; - if (this.closed) throw new Error("CUA browser runtime is closed"); + if (this.closed) throw new Error("Loop browser runtime is closed"); this.resources = resources; return resources; } catch (error) { @@ -63,9 +63,9 @@ export class CuaBrowserRuntime { throw error; } } - private async provision(): Promise { + private async provision(): Promise { const apiKey = this.env.KERNEL_API_KEY; - if (!apiKey) throw new Error("KERNEL_API_KEY is required when a CUA tool first executes"); + if (!apiKey) throw new Error("KERNEL_API_KEY is required when a Loop tool first executes"); const client = new Kernel({ apiKey, ...(this.env.KERNEL_BASE_URL ? { baseURL: this.env.KERNEL_BASE_URL } : {}) }); const attached = Boolean(this.options.sessionId); const browser = attached @@ -81,14 +81,14 @@ export class CuaBrowserRuntime { liveUrl: browser.browser_live_view_url, createdAt: browser.created_at, }; - return new CuaExecutionResources({ browser, client }); + return new LoopExecutionResources({ browser, client }); } async close(): Promise { if (this.closed) return; this.closed = true; // A shutdown can race the first tool call. Wait for provisioning so an owned // browser created after shutdown starts is still disposed and deleted. - let pendingResources: CuaExecutionResources | undefined; + let pendingResources: LoopExecutionResources | undefined; try { pendingResources = await this.pending; } catch { diff --git a/packages/pi-extension/src/index.ts b/packages/loop/src/pi-extension/index.ts similarity index 84% rename from packages/pi-extension/src/index.ts rename to packages/loop/src/pi-extension/index.ts index fab9c847..2cdfe55c 100644 --- a/packages/pi-extension/src/index.ts +++ b/packages/loop/src/pi-extension/index.ts @@ -1,29 +1,27 @@ import { fileURLToPath } from "node:url"; import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; -import { - createCuaModels, - type Api, - type CuaIncomingToolPlan, - type CuaToolCatalog, - type CuaToolSpec, - type Model, - type SimpleStreamOptions, - type StreamOptions, -} from "@onkernel/cua-ai"; +import type { Api, Model, SimpleStreamOptions, StreamOptions } from "@earendil-works/pi-ai"; +// This extension imports the rest of the package by name rather than by relative +// path: pi loads it as TypeScript through jiti, whose pi-ai alias rewrites the +// package root and cannot follow the deep `@earendil-works/pi-ai/api/*` imports +// the provider adapters make. By name, those adapters are loaded from the built +// package instead, the way any other consumer loads them. +import type { LoopIncomingToolPlan, LoopToolCatalog, LoopToolSpec } from "@onkernel/loop"; +import { createLoopModels } from "@onkernel/loop/pi"; import { allSelectableSpecs, compileSpecs, - CUA_SELECTORS, + LOOP_SELECTORS, expandSelection, parseSelection, selectorAvailability, - type CuaSelection, + type LoopSelection, } from "./selection"; -import { CuaBrowserRuntime, type BrowserOptions } from "./browser-runtime"; +import { LoopBrowserRuntime, type BrowserOptions } from "./browser-runtime"; import { CONFIG_ENTRY, restoreConfig, type PersistedConfig } from "./state"; import { availabilityText, statusText } from "./render"; -export default function cuaPiExtension(pi: ExtensionAPI): void { +export default function loopPiExtension(pi: ExtensionAPI): void { pi.registerFlag("browser-tools", { type: "string", description: "Comma-separated tool selectors; see /browser-tools for this model's menu" }); pi.registerFlag("browser-coordinates", { type: "string", description: "pixels or normalized-1000", default: "pixels" }); pi.registerFlag("browser-session", { type: "string", description: "Attach an existing Kernel browser session instead of creating one" }); @@ -41,8 +39,8 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { let initialized = false; let forcedInactive = false; let sessionActive = false; - let runtime: CuaBrowserRuntime | undefined; - let allSpecs = new Map(); + let runtime: LoopBrowserRuntime | undefined; + let allSpecs = new Map(); function configureDeclarations(): void { allSpecs = new Map(allSelectableSpecs(selection.coordinates).map((spec) => [spec.name, spec])); @@ -69,14 +67,14 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { }); } } - function ensureRuntime(): CuaBrowserRuntime { + function ensureRuntime(): LoopBrowserRuntime { if (!sessionActive) throw new Error("the browser runtime is unavailable outside an active pi session"); - return (runtime ??= new CuaBrowserRuntime(browserOptions)); + return (runtime ??= new LoopBrowserRuntime(browserOptions)); } - function currentSpecs(): CuaToolSpec[] { + function currentSpecs(): LoopToolSpec[] { return expandSelection(selection); } - function activeSpecs(): CuaToolSpec[] { + function activeSpecs(): LoopToolSpec[] { return currentSpecs().filter((spec) => activeNames.has(spec.name)); } function persistCommandSelection(): void { @@ -93,11 +91,11 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { const specs = currentSpecs(); const current = pi.getActiveTools(); const selectedNames = specs.map((spec) => spec.name); - const priorCua = current.filter((name) => allSpecs.has(name)); + const priorLoop = current.filter((name) => allSpecs.has(name)); // After an extension-forced incompatibility deactivation, restore the selected // set when the next model is compatible. A user /tools deactivation remains off. const desired = - !initialized || activateInitial || forcedInactive ? selectedNames : priorCua.filter((name) => selectedNames.includes(name)); + !initialized || activateInitial || forcedInactive ? selectedNames : priorLoop.filter((name) => selectedNames.includes(name)); try { if (desired.length && !ctx.model) throw new Error("no pi model is selected"); if (desired.length && ctx.model) { @@ -131,10 +129,10 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { } /** * The compiled catalog for the model pi is about to stream with, or undefined - * when no CUA tool is active. Compiling is pure and cheap, so this re-derives + * when no Loop tool is active. Compiling is pure and cheap, so this re-derives * per request rather than caching a catalog that a model switch could stale. */ - function streamCatalog(model: Model): CuaToolCatalog | undefined { + function streamCatalog(model: Model): LoopToolCatalog | undefined { if (!activeNames.size || compatibilityError) return undefined; try { return compileSpecs(model, activeSpecs()); @@ -144,7 +142,7 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { } /** - * Own the stream for every provider CUA wraps. + * Own the stream for every provider Loop wraps. * * This is what makes provider-native surfaces work inside pi. pi resolves and * streams its own registry model, but the transport a native surface needs is @@ -154,8 +152,8 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { * Anthropic's browser-beta fallback. pi's own resolved credential rides along * in `options.apiKey`. */ - function registerCuaProviders(): void { - const models = createCuaModels(); + function registerLoopProviders(): void { + const models = createLoopModels(); for (const id of ["anthropic", "openai", "google"]) { const base = models.getProvider(id); if (!base) continue; @@ -180,7 +178,7 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { ); } - registerCuaProviders(); + registerLoopProviders(); pi.registerCommand("browser", { description: "Show the selected browser tools and browser status", @@ -221,8 +219,8 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { // longer exists. Restoring must not throw: drop what is gone, keep the rest, // and say so — a resumed session that refuses to start is worse than one // that starts with fewer tools. - const known = saved.selectors.filter((selector) => CUA_SELECTORS.includes(selector)); - const dropped = saved.selectors.filter((selector) => !CUA_SELECTORS.includes(selector)); + const known = saved.selectors.filter((selector) => LOOP_SELECTORS.includes(selector)); + const dropped = saved.selectors.filter((selector) => !LOOP_SELECTORS.includes(selector)); if (dropped.length) { process.stderr.write(`browser tools: ignoring retired selector(s) from this session: ${dropped.join(", ")}\n`); } @@ -244,7 +242,7 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { pi.on("before_provider_headers", (event, ctx) => { // Reconcile first, and tolerate a catalog that no longer compiles: a model // switch can invalidate one after this turn's tools were serialized, and - // omitting CUA's headers is the correct outcome there. Without this the hook + // omitting Loop's headers is the correct outcome there. Without this the hook // throws, and a stale provider beta can survive into the request. reconcile(ctx); if (!activeNames.size || compatibilityError || !ctx.model) return; @@ -257,10 +255,10 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { pi.on("before_provider_request", async (event, ctx) => { reconcile(ctx); if (!activeNames.size || compatibilityError || !ctx.model) { - // setActiveTools() normally removes CUA declarations before serialization. + // setActiveTools() normally removes Loop declarations before serialization. // This hook is the final pre-wire guard for a model switch that invalidates // a catalog after pi has already built a payload for the turn. - return currentSpecs().length ? withoutCuaToolSchemas(event.payload, allSpecs) : undefined; + return currentSpecs().length ? withoutLoopToolSchemas(event.payload, allSpecs) : undefined; } return compileSpecs(ctx.model, activeSpecs()).payload.apply(event.payload, ctx.model); }); @@ -278,9 +276,9 @@ export default function cuaPiExtension(pi: ExtensionAPI): void { } /** Carry the compiled catalog's incoming native-call plan into pi's stream options. */ -function withPlan(options: T, catalog: CuaToolCatalog | undefined): T { +function withPlan(options: T, catalog: LoopToolCatalog | undefined): T { if (!catalog) return options; - return { ...options, cuaIncomingToolPlan: catalog.incoming } as T & { cuaIncomingToolPlan: CuaIncomingToolPlan }; + return { ...options, loopIncomingToolPlan: catalog.incoming } as T & { loopIncomingToolPlan: LoopIncomingToolPlan }; } function validateRawCliFlags(argv = process.argv.slice(2)): void { @@ -293,7 +291,7 @@ function validateRawCliFlags(argv = process.argv.slice(2)): void { parseSelection(read("browser-tools"), read("browser-coordinates") ?? "pixels"); parseBrowserOptions(read("browser-session"), read("browser-options")); } -function readFlags(pi: ExtensionAPI): { selection: CuaSelection; browserOptions: BrowserOptions } { +function readFlags(pi: ExtensionAPI): { selection: LoopSelection; browserOptions: BrowserOptions } { return { selection: parseSelection(asString(pi.getFlag("browser-tools")), asString(pi.getFlag("browser-coordinates"))), browserOptions: parseBrowserOptions(asString(pi.getFlag("browser-session")), asString(pi.getFlag("browser-options"))), @@ -333,20 +331,20 @@ function trim(value: string | undefined): string | undefined { const result = value?.trim(); return result || undefined; } -function withoutCuaToolSchemas(payload: unknown, cuaSpecs: ReadonlyMap): unknown { +function withoutLoopToolSchemas(payload: unknown, loopSpecs: ReadonlyMap): unknown { if (!isRecord(payload) || !Array.isArray(payload.tools)) return payload; const tools: unknown[] = []; for (const tool of payload.tools) { if (isRecord(tool) && Array.isArray(tool.functionDeclarations)) { const functionDeclarations = tool.functionDeclarations.filter((declaration) => { const name = serializedToolName(declaration); - return !name || !cuaSpecs.has(name); + return !name || !loopSpecs.has(name); }); if (functionDeclarations.length) tools.push({ ...tool, functionDeclarations }); continue; } const name = serializedToolName(tool); - if (!name || !cuaSpecs.has(name)) tools.push(tool); + if (!name || !loopSpecs.has(name)) tools.push(tool); } return { ...payload, tools }; } diff --git a/packages/pi-extension/src/render.ts b/packages/loop/src/pi-extension/render.ts similarity index 100% rename from packages/pi-extension/src/render.ts rename to packages/loop/src/pi-extension/render.ts diff --git a/packages/pi-extension/src/selection.ts b/packages/loop/src/pi-extension/selection.ts similarity index 73% rename from packages/pi-extension/src/selection.ts rename to packages/loop/src/pi-extension/selection.ts index d13f47a7..9dd83ccf 100644 --- a/packages/pi-extension/src/selection.ts +++ b/packages/loop/src/pi-extension/selection.ts @@ -1,10 +1,10 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import { compileCuaToolCatalog, cua, cuaToolMenu, type CuaToolCatalog, type CuaToolSpec } from "@onkernel/cua-ai"; +import { compileLoopToolCatalog, loop, loopToolMenu, type LoopToolCatalog, type LoopToolSpec } from "@onkernel/loop"; type Coordinates = "pixels" | "normalized-1000"; -type CoordinateSystem = ReturnType | ReturnType; +type CoordinateSystem = ReturnType | ReturnType; -export interface CuaSelection { +export interface LoopSelection { selectors: readonly string[]; coordinates: Coordinates; } @@ -32,23 +32,23 @@ const COMPUTER_BATCH_ACTIONS = [ * catalog's model, which carries the transport the selected tools derive, and * passes the incoming native-call plan. */ -const MENU: Readonly CuaToolSpec[]>> = Object.freeze({ - browser: () => [...cua.toolsets.browser(), cua.tools.browser.batch({ actions: BROWSER_BATCH_ACTIONS })], +const MENU: Readonly LoopToolSpec[]>> = Object.freeze({ + browser: () => [...loop.toolsets.browser(), loop.tools.browser.batch({ actions: BROWSER_BATCH_ACTIONS })], computer: (coordinates) => [ - ...cua.toolsets.computer({ coordinates }), - cua.tools.computer.batch({ actions: COMPUTER_BATCH_ACTIONS, coordinates }), + ...loop.toolsets.computer({ coordinates }), + loop.tools.computer.batch({ actions: COMPUTER_BATCH_ACTIONS, coordinates }), ], - "browser-act": () => [cua.tools.browser.act()], - playwright: () => [cua.tools.playwright()], - "anthropic-computer": () => [cua.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true })], - "anthropic-browser": () => [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })], - "openai-computer": () => [cua.providers.openai.tools.computer()], - "google-browser": () => cua.providers.google.toolsets.browser(), + "browser-act": () => [loop.tools.browser.act()], + playwright: () => [loop.tools.playwright()], + "anthropic-computer": () => [loop.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true })], + "anthropic-browser": () => [loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true })], + "openai-computer": () => [loop.providers.openai.tools.computer()], + "google-browser": () => loop.providers.google.toolsets.browser(), }); -export const CUA_SELECTORS: readonly string[] = Object.freeze(Object.keys(MENU)); +export const LOOP_SELECTORS: readonly string[] = Object.freeze(Object.keys(MENU)); -export function parseSelection(value: string | undefined, coordinates: string | undefined): CuaSelection { +export function parseSelection(value: string | undefined, coordinates: string | undefined): LoopSelection { const coordinateMode = coordinates ?? "pixels"; if (coordinateMode !== "pixels" && coordinateMode !== "normalized-1000") { throw new Error('--browser-coordinates must be "pixels" or "normalized-1000"'); @@ -60,7 +60,7 @@ export function parseSelection(value: string | undefined, coordinates: string | .filter(Boolean) ?? []; if (new Set(selectors).size !== selectors.length) throw new Error("--browser-tools contains duplicate selectors"); for (const selector of selectors) { - if (!CUA_SELECTORS.includes(selector)) throw new Error(`unknown browser tool selector "${selector}"`); + if (!LOOP_SELECTORS.includes(selector)) throw new Error(`unknown browser tool selector "${selector}"`); } return Object.freeze({ selectors: Object.freeze(selectors), coordinates: coordinateMode }); } @@ -72,17 +72,17 @@ export function parseSelection(value: string | undefined, coordinates: string | * registration — harmless, because a native declaration is replaced by the * catalog's payload transform and only the selected spec is ever executed. */ -export function allSelectableSpecs(coordinates: Coordinates): CuaToolSpec[] { - const result = new Map(); - for (const selector of CUA_SELECTORS) { +export function allSelectableSpecs(coordinates: Coordinates): LoopToolSpec[] { + const result = new Map(); + for (const selector of LOOP_SELECTORS) { for (const spec of expandSelection(parseSelection(selector, coordinates))) result.set(spec.name, spec); } return [...result.values()]; } -export function expandSelection(selection: CuaSelection): CuaToolSpec[] { - const coordinates = selection.coordinates === "pixels" ? cua.coordinates.pixels() : cua.coordinates.normalized([0, 1000]); - const result: CuaToolSpec[] = []; +export function expandSelection(selection: LoopSelection): LoopToolSpec[] { + const coordinates = selection.coordinates === "pixels" ? loop.coordinates.pixels() : loop.coordinates.normalized([0, 1000]); + const result: LoopToolSpec[] = []; for (const selector of selection.selectors) { const entry = MENU[selector]; if (!entry) throw new Error(`unknown browser tool selector "${selector}"`); @@ -101,8 +101,8 @@ export function expandSelection(selection: CuaSelection): CuaToolSpec[] { * what lets the extension validate a selection and generate headers before any * browser exists. */ -export function compileSpecs(model: Model, specs: readonly CuaToolSpec[]): CuaToolCatalog { - return compileCuaToolCatalog({ model, requestedTools: specs }); +export function compileSpecs(model: Model, specs: readonly LoopToolSpec[]): LoopToolCatalog { + return compileLoopToolCatalog({ model, requestedTools: specs }); } export interface SelectorAvailability { @@ -120,7 +120,7 @@ export interface SelectorAvailability { * selector *on its own*. * * Standalone is the right question here, and getting it wrong was a real bug: an - * earlier version passed the current selection to `cuaToolMenu`, whose verdicts + * earlier version passed the current selection to `loopToolMenu`, whose verdicts * are deliberately pairwise — relative to what is already selected. When the * current selection itself failed to compile, that failure became the reason on * every row, including rows that then activated fine. The one command whose job @@ -130,12 +130,12 @@ export interface SelectorAvailability { * coexist, and two providers' natives never can — so `conflictsWith` reports what * a selector cannot be *combined* with, separately from whether it is available. */ -export function selectorAvailability(model: Model, selection: CuaSelection): SelectorAvailability[] { +export function selectorAvailability(model: Model, selection: LoopSelection): SelectorAvailability[] { const selected = new Set(selection.selectors); - return CUA_SELECTORS.map((selector) => { + return LOOP_SELECTORS.map((selector) => { const specs = expandSelection({ selectors: [selector], coordinates: selection.coordinates }); const tools = specs.map((spec) => spec.name); - const conflictsWith = CUA_SELECTORS.filter((other) => { + const conflictsWith = LOOP_SELECTORS.filter((other) => { if (other === selector) return false; try { compileSpecs(model, expandSelection({ selectors: [selector, other], coordinates: selection.coordinates })); diff --git a/packages/pi-extension/src/state.ts b/packages/loop/src/pi-extension/state.ts similarity index 87% rename from packages/pi-extension/src/state.ts rename to packages/loop/src/pi-extension/state.ts index 11e91888..91a2c84d 100644 --- a/packages/pi-extension/src/state.ts +++ b/packages/loop/src/pi-extension/state.ts @@ -1,11 +1,11 @@ -import type { CuaSelection } from "./selection"; +import type { LoopSelection } from "./selection"; -export const CONFIG_ENTRY = "cua-pi-config-v1"; +export const CONFIG_ENTRY = "loop-pi-config-v1"; export interface PersistedConfig { version: 1; origin: "command"; selectors: string[]; - coordinates: CuaSelection["coordinates"]; + coordinates: LoopSelection["coordinates"]; browser?: { sessionId?: string; owned?: boolean; liveUrl?: string; createdAt?: string }; } export function restoreConfig(entries: readonly unknown[]): PersistedConfig | undefined { diff --git a/packages/loop/src/pi/api-keys.ts b/packages/loop/src/pi/api-keys.ts new file mode 100644 index 00000000..5ecf68fa --- /dev/null +++ b/packages/loop/src/pi/api-keys.ts @@ -0,0 +1,69 @@ +import type { Api, Model } from "@earendil-works/pi-ai"; +import { parseLoopModelRef, providerForModel, type LoopModelRef } from "./models"; + +/** + * Environment variables for the providers Loop documents, in precedence order. + * + * Every provider pi-ai carries is selectable, and pi resolves each one's own + * credential when streaming. This table exists only so callers can name the + * variable to set up front; a provider absent from it is not + * unsupported, it just has no Loop-side preflight. pi-ai does not export its + * own env-var registry, or this would read from that. + */ +const LOOP_PROVIDER_API_KEY_ENV_VARS: Readonly> = { + openai: ["OPENAI_API_KEY"], + anthropic: ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"], + google: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], + xai: ["XAI_API_KEY"], + moonshotai: ["MOONSHOT_API_KEY"], + openrouter: ["OPENROUTER_API_KEY"], +}; + +/** Provider prefixes accepted as aliases for a pi-ai provider id. */ +const PROVIDER_ALIASES: Readonly> = { gemini: "google", moonshot: "moonshotai" }; + +/** + * List the environment variables checked for a provider's API key, in + * precedence order. Returns an empty list for a provider Loop does not document, + * whose credential pi resolves at request time instead. + */ +export function loopApiKeyEnvVarsForProvider(provider: string): readonly string[] { + return LOOP_PROVIDER_API_KEY_ENV_VARS[PROVIDER_ALIASES[provider] ?? provider] ?? []; +} + +/** Read a provider's API key from the environment, or return undefined when unset. */ +export function getLoopEnvApiKey(provider: string): string | undefined { + for (const envVar of loopApiKeyEnvVarsForProvider(provider)) { + const value = process.env[envVar]; + if (value?.trim()) return value; + } + return undefined; +} + +/** + * Read a provider's API key from the environment, or throw naming the variables + * to set. Throws for a provider Loop documents no variables for — callers that + * accept any pi-ai provider should use {@link loopApiKeyEnvVarsForProvider} to + * decide whether a preflight is possible at all. + */ +export function requireLoopEnvApiKey(provider: string): string { + const apiKey = getLoopEnvApiKey(provider); + if (apiKey) return apiKey; + const envVars = loopApiKeyEnvVarsForProvider(provider); + if (envVars.length === 0) { + throw new Error(`No known API key environment variables for provider "${provider}"`); + } + throw new Error(`Missing API key for "${provider}". Set one of: ${envVars.join(", ")}`); +} + +/** {@link getLoopEnvApiKey} keyed by a model ref or concrete model instead of a provider name. */ +export function getLoopEnvApiKeyForModel(input: LoopModelRef | Model): string | undefined { + const provider = typeof input === "string" ? parseLoopModelRef(input).provider : providerForModel(input); + return getLoopEnvApiKey(provider); +} + +/** {@link requireLoopEnvApiKey} keyed by a model ref or concrete model instead of a provider name. */ +export function requireLoopEnvApiKeyForModel(input: LoopModelRef | Model): string { + const provider = typeof input === "string" ? parseLoopModelRef(input).provider : providerForModel(input); + return requireLoopEnvApiKey(provider); +} diff --git a/packages/agent/src/attach.ts b/packages/loop/src/pi/attach.ts similarity index 80% rename from packages/agent/src/attach.ts rename to packages/loop/src/pi/attach.ts index e6321151..1a9fc598 100644 --- a/packages/agent/src/attach.ts +++ b/packages/loop/src/pi/attach.ts @@ -5,27 +5,24 @@ import type { AgentTool, StreamFn, } from "@earendil-works/pi-agent-core"; -import { - type Api, - type Context, - cuaModels, - type CuaIncomingToolPlan, - type CuaModelRef, - getCuaModel, - parseCuaModelRef, - type CuaSimpleStreamOptions, - type Model, - type Models, - type SimpleStreamOptions, -} from "@onkernel/cua-ai"; +import type { + Api, + Context, + Model, + Models, + SimpleStreamOptions, +} from "@earendil-works/pi-ai"; import type Kernel from "@onkernel/sdk"; -import { resolveProviderRetryPolicy, type CuaRetryOptions, withProviderRetryModels } from "./provider-retry"; -import { CuaExecutionResources, type CuaExecutionDetails } from "./resources"; -import { CuaToolManager, type CuaHarnessTool } from "./tool-manager"; -import type { KernelBrowser } from "./translator/translator"; +import { LoopExecutionResources, type LoopExecutionDetails } from "../core/resources"; +import type { LoopIncomingToolPlan } from "../core/tool-catalog"; +import { LoopToolManager, type LoopHarnessTool } from "../core/tool-manager"; +import type { KernelBrowser } from "../core/translator/translator"; +import { getLoopModel, parseLoopModelRef, type LoopModelRef } from "./models"; +import { resolveProviderRetryPolicy, type LoopRetryOptions, withProviderRetryModels } from "./provider-retry"; +import { loopModels } from "./providers"; -/** A registered CUA model reference or an already resolved pi model. */ -export type CuaModelInput = CuaModelRef | Model; +/** A registered Loop model reference or an already resolved pi model. */ +export type LoopModelInput = LoopModelRef | Model; const DEFAULT_TOOL_RESULT_IMAGE_REPLAY_LIMIT = 4; const OMITTED_TOOL_RESULT_IMAGES = "[stale tool-result images omitted]"; @@ -34,7 +31,7 @@ const OMITTED_TOOL_RESULT_IMAGES = "[stale tool-result images omitted]"; export type ToolResultImageReplayLimit = number | false; /** Optional follow-up policy for otherwise empty successful assistant responses. */ -export interface CuaEmptyResponseRecoveryOptions { +export interface LoopEmptyResponseRecoveryOptions { /** User message queued to ask the model to continue. */ followUp: string; /** Maximum automatic follow-ups per prompt. */ @@ -42,20 +39,20 @@ export interface CuaEmptyResponseRecoveryOptions { } /** What a Kernel browser handle needs to know to stream and execute. */ -export interface CuaAttachOptions { +export interface LoopAttachOptions { browser: KernelBrowser; client: Kernel; - /** Defaults to the shared {@link cuaModels} collection. */ + /** Defaults to the shared {@link loopModels} collection. */ models?: Models; - retry?: CuaRetryOptions; + retry?: LoopRetryOptions; toolResultImageReplayLimit?: ToolResultImageReplayLimit; responseThreading?: boolean; - emptyResponseRecovery?: CuaEmptyResponseRecoveryOptions; + emptyResponseRecovery?: LoopEmptyResponseRecoveryOptions; onPayload?: SimpleStreamOptions["onPayload"]; } /** One compiled (model, tools) pair, ready to hand to pi. */ -export interface CuaCompiled { +export interface LoopCompiled { /** The model to stream with, carrying the transport its tools derive. */ readonly model: Model; /** Executable tools, materialized once against the handle's browser pool. */ @@ -63,7 +60,7 @@ export interface CuaCompiled { /** Same tools viewed as pi `AgentTool`s, for the low-level `Agent`. */ readonly agentTools: readonly AgentTool[]; /** - * The handle's `Models` collection, adding what CUA owns per request: + * The handle's `Models` collection, adding what Loop owns per request: * provider retry, required headers, the catalog's payload transforms, and * the tool-result image bound. Shared by every compile from this handle, * because pi fixes `models` at construction while those transforms are @@ -90,7 +87,7 @@ export interface CuaCompiled { } /** - * A Kernel browser bound to CUA's execution resources. + * A Kernel browser bound to Loop's execution resources. * * The handle is what persists: the Kernel client and browser, the canonical * computer translator, the lazily created raw-CDP executor, element-ref and @@ -99,20 +96,20 @@ export interface CuaCompiled { * and a spec materializes exactly once per handle, so repeat compiles keep tool * identity stable. */ -export interface CuaBrowserHandle { +export interface LoopBrowserHandle { compile(options: { - model: CuaModelInput; - tools: readonly CuaHarnessTool[]; - }): CuaCompiled; + model: LoopModelInput; + tools: readonly LoopHarnessTool[]; + }): LoopCompiled; /** The shared execution pool, for callers that need it directly. */ - readonly resources: CuaExecutionResources; - /** Same collection every {@link CuaCompiled.models} returns; see the note there. */ + readonly resources: LoopExecutionResources; + /** Same collection every {@link LoopCompiled.models} returns; see the note there. */ readonly models: Models; dispose(): Promise; } /** - * Bind a Kernel browser to CUA's execution resources and return a handle that + * Bind a Kernel browser to Loop's execution resources and return a handle that * compiles (model, tools) pairs into plain pi objects. * * ```ts @@ -121,25 +118,25 @@ export interface CuaBrowserHandle { * const harness = new AgentHarness({ model, tools, models, activeToolNames: tools.map((t) => t.name), session }); * ``` */ -export function attach(options: CuaAttachOptions): CuaBrowserHandle { - const resources = new CuaExecutionResources({ browser: options.browser, client: options.client }); +export function attach(options: LoopAttachOptions): LoopBrowserHandle { + const resources = new LoopExecutionResources({ browser: options.browser, client: options.client }); const imageReplayLimit = resolveToolResultImageReplayLimit(options.toolResultImageReplayLimit); const useResponseThreading = resolveResponseThreading(options.responseThreading); const recovery = resolveEmptyResponseRecovery(options.emptyResponseRecovery); - const retrying = withProviderRetryModels(options.models ?? cuaModels(), resolveProviderRetryPolicy(options.retry)); + const retrying = withProviderRetryModels(options.models ?? loopModels(), resolveProviderRetryPolicy(options.retry)); // pi fixes `models` at construction, but the headers, payload transforms and // incoming tool plan it applies are per-catalog. One collection per handle, // reading whichever pair is live, is what lets a caller swap the pair on a // running harness at all. - let live: CuaToolManager | undefined; - let lastCompiled: CuaToolManager | undefined; + let live: LoopToolManager | undefined; + let lastCompiled: LoopToolManager | undefined; let release: (() => void) | undefined; const models = withCatalogModels( retrying, () => { const manager = live ?? lastCompiled; - if (!manager) throw new Error("cua: compile a (model, tools) pair before streaming"); + if (!manager) throw new Error("loop: compile a (model, tools) pair before streaming"); return manager; }, imageReplayLimit, @@ -152,10 +149,10 @@ export function attach(options: CuaAttachOptions): CuaBrowserHandle { models, dispose: () => resources.dispose(), compile(request: { - model: CuaModelInput; - tools: readonly CuaHarnessTool[]; - }): CuaCompiled { - const manager = new CuaToolManager>( + model: LoopModelInput; + tools: readonly LoopHarnessTool[]; + }): LoopCompiled { + const manager = new LoopToolManager>( resources, request.model, request.tools, @@ -166,7 +163,7 @@ export function attach(options: CuaAttachOptions): CuaBrowserHandle { const tools = manager.harnessTools() as readonly AgentHarnessTool[]; const activate = (harness: AgentHarness): (() => void) => { release?.(); - const uninstall = installCuaBehaviors(harness, manager, recovery); + const uninstall = installLoopBehaviors(harness, manager, recovery); live = manager; // Identity-checked so calling a stale release cannot clear a newer // activation: only the pair still live releases anything. @@ -212,13 +209,13 @@ async function applyCompiled( } /** - * Wire the pi event handlers CUA owns. Kept separate from `compile()` because + * Wire the pi event handlers Loop owns. Kept separate from `compile()` because * they are handlers on a constructed harness, not constructor options. */ -export function installCuaBehaviors( +export function installLoopBehaviors( harness: AgentHarness, - manager: CuaToolManager, - recovery: CuaEmptyResponseRecoveryOptions | undefined, + manager: LoopToolManager, + recovery: LoopEmptyResponseRecoveryOptions | undefined, ): () => void { let turnFailed = false; let hasPendingQueue = false; @@ -253,12 +250,12 @@ export function installCuaBehaviors( } /** @internal */ -export const defaultCuaStream: StreamFn = (model, context, options) => cuaModels().streamSimple(model, context, options); +export const defaultLoopStream: StreamFn = (model, context, options) => loopModels().streamSimple(model, context, options); /** @internal */ -export function resolveModelFromCollection(ref: CuaModelRef, models: Models): Model { - const { provider, model: id } = parseCuaModelRef(ref); - return models.getModel(provider, id) ?? getCuaModel(ref); +export function resolveModelFromCollection(ref: LoopModelRef, models: Models): Model { + const { provider, model: id } = parseLoopModelRef(ref); + return models.getModel(provider, id) ?? getLoopModel(ref); } /** Whether a tools-only recompile actually changed the model pi streams with, so `setTools()` only pushes `setModel()` (and its session/event side effects) when the derived transport moved. */ @@ -270,7 +267,7 @@ export function modelTransportChanged(previous: Model, next: Model): b /** @internal */ export function withCatalogModels( models: Models, - liveManager: () => CuaToolManager, + liveManager: () => LoopToolManager, imageReplayLimit: ToolResultImageReplayLimit, responseThreading: boolean, handleOnPayload?: SimpleStreamOptions["onPayload"], @@ -287,7 +284,7 @@ export function withCatalogModels( ...options, headers: catalog.headers.merge(options?.headers), disableResponseThreading: responseThreading ? undefined : true, - cuaIncomingToolPlan: catalog.incoming, + loopIncomingToolPlan: catalog.incoming, onPayload: async (payload: unknown, model: Model) => { const generated = await catalog.payload.apply(payload, model); return callerOnPayload ? (await callerOnPayload(generated, model)) ?? generated : generated; @@ -323,7 +320,7 @@ export function resolveToolResultImageReplayLimit(limit: ToolResultImageReplayLi /** Native computer tool names whose screenshot history the provider protocol requires in full, regardless of the image replay limit. */ /** @internal */ -export function requiredImageToolNames(incoming: CuaIncomingToolPlan): ReadonlySet { +export function requiredImageToolNames(incoming: LoopIncomingToolPlan): ReadonlySet { return new Set(incoming.openaiComputerName ? [incoming.openaiComputerName] : []); } @@ -373,7 +370,7 @@ function projectModelContext( } /** @internal */ -export function resolveEmptyResponseRecovery(options: CuaEmptyResponseRecoveryOptions | undefined): CuaEmptyResponseRecoveryOptions | undefined { +export function resolveEmptyResponseRecovery(options: LoopEmptyResponseRecoveryOptions | undefined): LoopEmptyResponseRecoveryOptions | undefined { if (!options) return undefined; if (options.followUp.trim().length === 0) throw new Error("emptyResponseRecovery.followUp must not be blank"); if (!Number.isInteger(options.maxAttempts) || options.maxAttempts < 0) throw new Error("emptyResponseRecovery.maxAttempts must be a non-negative finite integer"); @@ -393,11 +390,11 @@ export function isEmptyAssistantResponse(message: AgentMessage): boolean { /** @internal */ export function hasExecutionError(details: unknown): boolean { - return Boolean(details && typeof details === "object" && (details as CuaExecutionDetails).isError === true); + return Boolean(details && typeof details === "object" && (details as LoopExecutionDetails).isError === true); } /** @internal */ -export function turnFailureStopMessage(manager: CuaToolManager): string | undefined { +export function turnFailureStopMessage(manager: LoopToolManager): string | undefined { for (const entry of manager.catalog.entries) { const execution = manager.specFor(entry.identity)?.execution; if (execution?.kind === "actions" && execution.stopTurnOnFailureMessage) return execution.stopTurnOnFailureMessage; diff --git a/packages/loop/src/pi/index.ts b/packages/loop/src/pi/index.ts new file mode 100644 index 00000000..6cec36d1 --- /dev/null +++ b/packages/loop/src/pi/index.ts @@ -0,0 +1,35 @@ +export * from "@earendil-works/pi-agent-core"; +export { NodeExecutionEnv } from "@earendil-works/pi-agent-core/node"; + +export * from "./api-keys"; +export * from "./models"; +export { + createLoopModels, + GOOGLE_INTERACTIONS_API, + loopModels, + OPENAI_COMPUTER_USE_API, + streamGoogleInteractions, + streamOpenAIResponses, + streamSimpleGoogleInteractions, + streamSimpleOpenAIResponses, +} from "./providers"; +export type { + LoopSimpleStreamOptions, + ResponseThreadingOptions, + ResponsesThreadingOptions, +} from "./providers/common"; +export { + responseThreadingDelta, + responseThreadingEnabled, + threadResponsesRequest, +} from "./providers/common"; +export { attach } from "./attach"; +export type { + LoopAttachOptions, + LoopBrowserHandle, + LoopCompiled, + LoopEmptyResponseRecoveryOptions, + LoopModelInput, + ToolResultImageReplayLimit, +} from "./attach"; +export type { LoopRetryOptions } from "./provider-retry"; diff --git a/packages/ai/src/models.ts b/packages/loop/src/pi/models.ts similarity index 73% rename from packages/ai/src/models.ts rename to packages/loop/src/pi/models.ts index a64b212d..2876c9af 100644 --- a/packages/ai/src/models.ts +++ b/packages/loop/src/pi/models.ts @@ -3,25 +3,25 @@ import { getBuiltinModel, getBuiltinModels, getBuiltinProviders } from "@earendi import { supportsAnthropicNativeBrowser, supportsAnthropicNativeComputer } from "./providers/anthropic/capabilities"; /** A pi-ai provider id. Any provider pi-ai carries can be selected. */ -export type CuaProvider = string; +export type LoopProvider = string; /** Provider-qualified model reference, e.g. `"openai:gpt-5.6-sol"` or `"google:gemini-3.6-flash"`. */ -export type CuaModelRef = `${string}:${string}`; +export type LoopModelRef = `${string}:${string}`; -/** A provider-native tool surface CUA can offer for a model. */ -export type CuaNativeSurface = "computer" | "browser"; +/** A provider-native tool surface Loop can offer for a model. */ +export type ComputerUseNativeSurface = "computer" | "browser"; -/** One entry returned by {@link listCuaModels}. */ -export interface CuaModelInfo { - /** Provider-qualified ref accepted by {@link getCuaModel}. */ - ref: CuaModelRef; - provider: CuaProvider; +/** One entry returned by {@link listLoopModels}. */ +export interface LoopModelInfo { + /** Provider-qualified ref accepted by {@link getLoopModel}. */ + ref: LoopModelRef; + provider: LoopProvider; /** Provider-native model id (the part after the colon). */ model: string; /** Human-readable model name. */ name: string; /** Provider-native tool surfaces available for this model, if any. */ - nativeSurfaces: readonly CuaNativeSurface[]; + nativeSurfaces: readonly ComputerUseNativeSurface[]; /** Whether the model accepts image input, i.e. whether screenshot-based tools are usable. */ vision: boolean; } @@ -35,12 +35,12 @@ export interface CuaModelInfo { * "gpt-5.5-2026-04-23"). Named variants like "gpt-5.4-mini" are distinct * models and need their own entry. */ -export type CuaModelMatch = +export type LoopModelMatch = | { readonly kind: "exact"; readonly id: string } | { readonly kind: "family"; readonly family: string }; -/** CUA tool-catalog capabilities for a concrete model. */ -export interface CuaModelCapabilities { +/** Loop tool-catalog capabilities for a concrete model. */ +export interface LoopModelCapabilities { readonly acceptsComplexSchemas: boolean; readonly acceptsLargeSchemas: boolean; readonly serializesStateMutations: boolean; @@ -51,28 +51,28 @@ export interface CuaModelCapabilities { * default, with the evidence for it. Entries exist to prevent a request the * provider would reject — never to express a preference. */ -export interface CuaModelQuirk { - readonly provider: CuaProvider; +export interface LoopModelQuirk { + readonly provider: LoopProvider; /** Omit to apply the quirk to every model from the provider. */ - readonly match?: CuaModelMatch; - readonly capabilities: Partial; + readonly match?: LoopModelMatch; + readonly capabilities: Partial; /** Why this quirk exists: the documented limit or the observed failure. */ readonly reason: string; } /** * Models with a provider-native computer or browser tool, and the first-party - * documentation for it. This table answers "can CUA offer a native tool for + * documentation for it. This table answers "can Loop offer a native tool for * this model", not "may this model run" — every model pi-ai carries runs, with - * CUA's own CDP browser tools. + * Loop's own CDP browser tools. * * Anthropic is absent deliberately: its native surfaces are version-gated in - * `providers/anthropic/capabilities.ts`, which {@link cuaNativeSurfaces} reads. + * `providers/anthropic/capabilities.ts`, which {@link computerUseNativeSurfaces} reads. */ -export const CUA_NATIVE_SURFACES: readonly { - readonly provider: CuaProvider; - readonly match: CuaModelMatch; - readonly surfaces: readonly CuaNativeSurface[]; +export const COMPUTER_USE_NATIVE_SURFACES: readonly { + readonly provider: LoopProvider; + readonly match: LoopModelMatch; + readonly surfaces: readonly ComputerUseNativeSurface[]; readonly source: string; }[] = [ { provider: "openai", match: { kind: "exact", id: "gpt-5.6-sol" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.6-sol" }, @@ -90,7 +90,7 @@ export const CUA_NATIVE_SURFACES: readonly { * feedback. Every entry below is a limit we have documentation for or have * observed against the live API. */ -export const CUA_MODEL_QUIRKS: readonly CuaModelQuirk[] = [ +export const LOOP_MODEL_QUIRKS: readonly LoopModelQuirk[] = [ { provider: "moonshotai", match: { kind: "exact", id: "kimi-k3" }, @@ -116,7 +116,7 @@ export const CUA_MODEL_QUIRKS: readonly CuaModelQuirk[] = [ }, ]; -const PERMISSIVE_CAPABILITIES: CuaModelCapabilities = Object.freeze({ +const PERMISSIVE_CAPABILITIES: LoopModelCapabilities = Object.freeze({ acceptsComplexSchemas: true, acceptsLargeSchemas: true, serializesStateMutations: false, @@ -132,69 +132,69 @@ const PROVIDER_ALIASES: Readonly> = { gemini: "google", m * `"moonshot:"` for `"moonshotai"`. Throws when the ref is unqualified or names * a provider pi-ai does not carry. */ -export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: string } { +export function parseLoopModelRef(ref: string): { provider: LoopProvider; model: string } { const idx = ref.indexOf(":"); if (idx <= 0 || idx === ref.length - 1) { - throw new Error(`CUA model ref must be provider-qualified as ":"; got "${ref}"`); + throw new Error(`Loop model ref must be provider-qualified as ":"; got "${ref}"`); } const prefix = ref.slice(0, idx); const provider = PROVIDER_ALIASES[prefix] ?? prefix; const model = ref.slice(idx + 1); - if (!cuaProviders().includes(provider)) { - throw new Error(`unknown provider "${prefix}" (pi-ai carries: ${cuaProviders().join(", ")})`); + if (!loopProviders().includes(provider)) { + throw new Error(`unknown provider "${prefix}" (pi-ai carries: ${loopProviders().join(", ")})`); } return { provider, model }; } -/** Join a provider and model id into a {@link CuaModelRef}. */ -export function formatCuaModelRef(provider: CuaProvider, model: string): CuaModelRef { - return `${provider}:${model}` as CuaModelRef; +/** Join a provider and model id into a {@link LoopModelRef}. */ +export function formatLoopModelRef(provider: LoopProvider, model: string): LoopModelRef { + return `${provider}:${model}` as LoopModelRef; } /** Every provider id pi-ai carries. */ -export function cuaProviders(): readonly CuaProvider[] { +export function loopProviders(): readonly LoopProvider[] { return getBuiltinProviders(); } /** * List the models pi-ai carries, optionally filtered to one provider, each - * annotated with the provider-native surfaces CUA can offer for it. + * annotated with the provider-native surfaces Loop can offer for it. */ -export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { - const providers = provider ? [PROVIDER_ALIASES[provider] ?? provider] : [...cuaProviders()]; - const byRef = new Map(); +export function listLoopModels(provider?: LoopProvider): LoopModelInfo[] { + const providers = provider ? [PROVIDER_ALIASES[provider] ?? provider] : [...loopProviders()]; + const byRef = new Map(); for (const p of providers) { for (const model of getBuiltinModels(p as never) as Model[]) { - const ref = formatCuaModelRef(p, model.id); + const ref = formatLoopModelRef(p, model.id); if (byRef.has(ref)) continue; byRef.set(ref, { ref, provider: p, model: model.id, name: model.name, - nativeSurfaces: cuaNativeSurfaces(model), + nativeSurfaces: computerUseNativeSurfaces(model), vision: model.input.includes("image"), }); } } - return [...byRef.values()].sort(compareCuaModels); + return [...byRef.values()].sort(compareLoopModels); } /** - * Resolve a {@link CuaModelRef} to a concrete pi-ai model. + * Resolve a {@link LoopModelRef} to a concrete pi-ai model. * * A ref pi-ai's registry does not carry is synthesized from the provider's * other models, so a model id works the day the provider ships it rather than * when models.dev catches up. Throws only for an unqualified ref or a provider * pi-ai does not carry. */ -export function getCuaModel(ref: CuaModelRef): Model { - const { provider, model: modelId } = parseCuaModelRef(ref); +export function getLoopModel(ref: LoopModelRef): Model { + const { provider, model: modelId } = parseLoopModelRef(ref); const fromRegistry = getBuiltinModel(provider as never, modelId as never) as Model | undefined; if (fromRegistry) return fromRegistry; - return synthesizeCuaModel(provider, modelId); + return synthesizeLoopModel(provider, modelId); } /** @@ -208,7 +208,7 @@ export function getCuaModel(ref: CuaModelRef): Model { * id should follow its nearest, newest relative rather than whichever model * happens to come first. */ -function synthesizeCuaModel(provider: CuaProvider, modelId: string): Model { +function synthesizeLoopModel(provider: LoopProvider, modelId: string): Model { const siblings = getBuiltinModels(provider as never) as Model[]; if (siblings.length === 0) { throw new Error(`provider "${provider}" carries no models to infer "${modelId}" from`); @@ -232,19 +232,19 @@ function sharedPrefixLength(a: string, b: string): number { } /** Return the provider id for a concrete model. */ -export function providerForModel(model: Model): CuaProvider { +export function providerForModel(model: Model): LoopProvider { return model.provider; } /** Provider-native tool surfaces available for a model, if any. */ -export function cuaNativeSurfaces(model: Model): readonly CuaNativeSurface[] { +export function computerUseNativeSurfaces(model: Model): readonly ComputerUseNativeSurface[] { if (model.provider === "anthropic") { - const surfaces: CuaNativeSurface[] = []; + const surfaces: ComputerUseNativeSurface[] = []; if (supportsAnthropicNativeComputer(model.id)) surfaces.push("computer"); if (supportsAnthropicNativeBrowser(model.id)) surfaces.push("browser"); return surfaces; } - for (const entry of CUA_NATIVE_SURFACES) { + for (const entry of COMPUTER_USE_NATIVE_SURFACES) { if (entry.provider === model.provider && matchesModelId(model.id, entry.match)) return entry.surfaces; } return []; @@ -254,9 +254,9 @@ export function cuaNativeSurfaces(model: Model): readonly CuaNativeSurface[ * Tool-catalog capabilities for a model: permissive unless a quirk says * otherwise. Provider-wide quirks apply first, then model-specific ones. */ -export function cuaModelCapabilities(model: Model): CuaModelCapabilities { +export function loopModelCapabilities(model: Model): LoopModelCapabilities { let capabilities = PERMISSIVE_CAPABILITIES; - for (const quirk of CUA_MODEL_QUIRKS) { + for (const quirk of LOOP_MODEL_QUIRKS) { if (quirk.provider !== model.provider) continue; if (quirk.match && !matchesModelId(model.id, quirk.match)) continue; capabilities = { ...capabilities, ...quirk.capabilities }; @@ -265,22 +265,22 @@ export function cuaModelCapabilities(model: Model): CuaModelCapabilities { } /** Find the quirks that apply to a model, for diagnostics and menu hints. */ -export function cuaModelQuirks(model: Model): readonly CuaModelQuirk[] { - return CUA_MODEL_QUIRKS.filter( +export function loopModelQuirks(model: Model): readonly LoopModelQuirk[] { + return LOOP_MODEL_QUIRKS.filter( (quirk) => quirk.provider === model.provider && (!quirk.match || matchesModelId(model.id, quirk.match)), ); } -function matchesModelId(modelId: string, match: CuaModelMatch): boolean { +function matchesModelId(modelId: string, match: LoopModelMatch): boolean { const id = modelId.toLowerCase(); - return match.kind === "exact" ? id === match.id.toLowerCase() : isCuaFamilyMatch(id, match.family.toLowerCase()); + return match.kind === "exact" ? id === match.id.toLowerCase() : isFamilyMatch(id, match.family.toLowerCase()); } // A family entry covers its root id plus suffixes made of hyphen-separated // numeric segments: revisions like "claude-opus-4-7" and dated snapshots like // "gpt-5.5-2026-04-23". Named sibling variants ("gpt-5.4-mini") are distinct // models and need their own entry. -function isCuaFamilyMatch(id: string, family: string): boolean { +function isFamilyMatch(id: string, family: string): boolean { if (id === family) return true; if (!id.startsWith(`${family}-`)) return false; return id @@ -289,7 +289,7 @@ function isCuaFamilyMatch(id: string, family: string): boolean { .every((segment) => /^\d+$/.test(segment)); } -function compareCuaModels(a: CuaModelInfo, b: CuaModelInfo): number { +function compareLoopModels(a: LoopModelInfo, b: LoopModelInfo): number { if (a.provider !== b.provider) return a.provider.localeCompare(b.provider); return a.model.localeCompare(b.model); } diff --git a/packages/agent/src/provider-retry.ts b/packages/loop/src/pi/provider-retry.ts similarity index 96% rename from packages/agent/src/provider-retry.ts rename to packages/loop/src/pi/provider-retry.ts index bfb27f19..2dde8d3e 100644 --- a/packages/agent/src/provider-retry.ts +++ b/packages/loop/src/pi/provider-retry.ts @@ -17,11 +17,11 @@ const DEFAULT_BASE_DELAY_MS = 2_000; const MAX_TIMER_DELAY_MS = 2_147_483_647; /** - * Controls CUA retries around a single provider request. + * Controls Loop retries around a single provider request. * Enabling provider request retries as well can multiply network attempts. */ -export interface CuaRetryOptions { - /** Enable CUA-level retries. Disabled by default. */ +export interface LoopRetryOptions { + /** Enable Loop-level retries. Disabled by default. */ enabled?: boolean; /** Number of additional provider requests. Defaults to 3. */ maxRetries?: number; @@ -41,7 +41,7 @@ type RetryStreamFn = ( options?: SimpleStreamOptions, ) => ReturnType; -export function resolveProviderRetryPolicy(options?: CuaRetryOptions): ResolvedRetryPolicy { +export function resolveProviderRetryPolicy(options?: LoopRetryOptions): ResolvedRetryPolicy { const enabled = options?.enabled ?? false; const maxRetries = options?.maxRetries ?? DEFAULT_MAX_RETRIES; const baseDelayMs = options?.baseDelayMs ?? DEFAULT_BASE_DELAY_MS; diff --git a/packages/ai/src/providers.ts b/packages/loop/src/pi/providers.ts similarity index 53% rename from packages/ai/src/providers.ts rename to packages/loop/src/pi/providers.ts index 3e3fadbe..6bd03b8e 100644 --- a/packages/ai/src/providers.ts +++ b/packages/loop/src/pi/providers.ts @@ -11,24 +11,24 @@ import { type StreamOptions, } from "@earendil-works/pi-ai"; import { builtinModels } from "@earendil-works/pi-ai/providers/all"; -import { cuaApiKeyEnvVarsForProvider } from "./api-keys"; +import { loopApiKeyEnvVarsForProvider } from "./api-keys"; import { withAnthropicBrowserFallback } from "./providers/anthropic/browser-fallback"; -import { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; -import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAICuaComputer, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; +import { GOOGLE_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; +import { OPENAI_COMPUTER_USE_API, requiresOpenAINamespaceAdapter, streamOpenAIComputerUse, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; /** - * Build the pi `Models` collection CUA streams through: pi's builtin - * providers, adjusted for CUA: + * Build the pi `Models` collection Loop streams through: pi's builtin + * providers, adjusted for Loop: * * - `anthropic` retries an inaccessible native browser beta through the * selected tool's equivalent function declaration. * - `openai` streams through pi's builtin `openai-responses` transport and its * automatic prompt caching by default; a model compiled with OpenAI's native - * computer tool carries `openai-cua-computer` instead, which this wrapper - * routes to the CUA adapter. The one dispatch that cannot be derived from + * computer tool carries `openai-computer-use` instead, which this wrapper + * routes to the Loop adapter. The one dispatch that cannot be derived from * `model.api` is a transcript carrying a deferred tool-search addition or a - * replayed function-call namespace (see {@link requiresCuaOpenAINamespaceAdapter}). - * - `google` intercepts `google-cua-interactions` — carried only by a model + * replayed function-call namespace (see {@link requiresOpenAINamespaceAdapter}). + * - `google` intercepts `google-interactions` — carried only by a model * compiled with Google's native computer-use toolset — for current native * computer use, and resolves API keys from `GOOGLE_API_KEY` or `GEMINI_API_KEY`. * A Google model compiled without that toolset streams through pi's builtin @@ -37,70 +37,70 @@ import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenA * Responses transport, and the catalog supplies its serial-tool-call field. * - `moonshotai` is pi's builtin provider untouched: Kimi streams through the * plain OpenAI-compatible chat completions transport with `MOONSHOT_API_KEY`. - * - `meta` is a CUA-only provider pi does not ship. It speaks the OpenAI + * - `meta` is a Loop-only provider pi does not ship. It speaks the OpenAI * Responses wire protocol, so it registers pi's builtin transport against * Meta's base URL and credentials. * * Each call returns an independent collection; register additional providers - * or credentials on it freely. Use {@link cuaModels} for the shared default. + * or credentials on it freely. Use {@link loopModels} for the shared default. */ -export function createCuaModels(options?: CreateModelsOptions): MutableModels { +export function createLoopModels(options?: CreateModelsOptions): MutableModels { const models = builtinModels(options); const anthropic = models.getProvider("anthropic"); if (anthropic) models.setProvider(withAnthropicBrowserFallback(anthropic)); const openai = models.getProvider("openai"); - if (openai) models.setProvider(withOpenAICuaAdapter(openai)); + if (openai) models.setProvider(withOpenAIComputerUseAdapter(openai)); const google = models.getProvider("google"); - if (google) models.setProvider(withGoogleCuaInteractions(google)); + if (google) models.setProvider(withGoogleInteractions(google)); return models; } -let defaultCuaModels: MutableModels | undefined; +let defaultLoopModels: MutableModels | undefined; /** - * Shared default {@link createCuaModels} collection, created on first use. + * Shared default {@link createLoopModels} collection, created on first use. * - * cua-agent streams through this instance unless given - * another one. Auth resolves from the documented CUA env-var convention (see - * `cuaApiKeyEnvVarsForProvider`); pass an explicit `options.apiKey` per + * `attach()` streams through this instance unless given another one. Auth + * resolves from the documented env-var convention (see + * `loopApiKeyEnvVarsForProvider`); pass an explicit `options.apiKey` per * request to override. */ -export function cuaModels(): MutableModels { - return (defaultCuaModels ??= createCuaModels()); +export function loopModels(): MutableModels { + return (defaultLoopModels ??= createLoopModels()); } -// The compiled catalog's model.api decides dispatch: OPENAI_CUA_COMPUTER_API -// routes to the CUA adapter, everything else falls through to pi's builtin -// "openai-responses" provider. requiresCuaOpenAINamespaceAdapter is the one +// The compiled catalog's model.api decides dispatch: OPENAI_COMPUTER_USE_API +// routes to the Loop adapter, everything else falls through to pi's builtin +// "openai-responses" provider. requiresOpenAINamespaceAdapter is the one // exception that cannot be derived from the model — see its doc comment. -function withOpenAICuaAdapter(base: Provider): Provider { +function withOpenAIComputerUseAdapter(base: Provider): Provider { return { ...base, stream: (model: Model, context: Context, options?: StreamOptions) => - model.api === OPENAI_CUA_COMPUTER_API - ? streamOpenAICuaComputer(model as never, context, options) - : requiresCuaOpenAINamespaceAdapter(context) + model.api === OPENAI_COMPUTER_USE_API + ? streamOpenAIComputerUse(model as never, context, options) + : requiresOpenAINamespaceAdapter(context) ? streamOpenAIResponses(model as never, context, options) : base.stream(model, context, options), streamSimple: (model: Model, context: Context, options?: SimpleStreamOptions) => - model.api === OPENAI_CUA_COMPUTER_API - ? streamOpenAICuaComputer(model as never, context, options) - : requiresCuaOpenAINamespaceAdapter(context) + model.api === OPENAI_COMPUTER_USE_API + ? streamOpenAIComputerUse(model as never, context, options) + : requiresOpenAINamespaceAdapter(context) ? streamSimpleOpenAIResponses(model as never, context, options) : base.streamSimple(model, context, options), }; } -function withGoogleCuaInteractions(base: Provider): Provider { +function withGoogleInteractions(base: Provider): Provider { return { ...base, - auth: { ...base.auth, apiKey: envApiKeyAuth("Google API key", cuaApiKeyEnvVarsForProvider("google")) }, + auth: { ...base.auth, apiKey: envApiKeyAuth("Google API key", loopApiKeyEnvVarsForProvider("google")) }, stream: (model: Model, context: Context, options?: StreamOptions) => - model.api === GOOGLE_CUA_INTERACTIONS_API + model.api === GOOGLE_INTERACTIONS_API ? streamGoogleInteractions(model as never, context, options) : base.stream(model, context, options), streamSimple: (model: Model, context: Context, options?: SimpleStreamOptions) => - model.api === GOOGLE_CUA_INTERACTIONS_API + model.api === GOOGLE_INTERACTIONS_API ? streamSimpleGoogleInteractions(model as never, context, options) : base.streamSimple(model, context, options), }; @@ -108,5 +108,5 @@ function withGoogleCuaInteractions(base: Provider): Provider { -export { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions }; -export { OPENAI_CUA_COMPUTER_API, streamOpenAIResponses, streamSimpleOpenAIResponses }; +export { GOOGLE_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions }; +export { OPENAI_COMPUTER_USE_API, streamOpenAIResponses, streamSimpleOpenAIResponses }; diff --git a/packages/ai/src/providers/anthropic/adaptive-thinking.ts b/packages/loop/src/pi/providers/anthropic/adaptive-thinking.ts similarity index 91% rename from packages/ai/src/providers/anthropic/adaptive-thinking.ts rename to packages/loop/src/pi/providers/anthropic/adaptive-thinking.ts index 24e20d54..c71230f8 100644 --- a/packages/ai/src/providers/anthropic/adaptive-thinking.ts +++ b/packages/loop/src/pi/providers/anthropic/adaptive-thinking.ts @@ -1,8 +1,8 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import type { CuaPayloadHook } from "../common"; +import type { LoopPayloadHook } from "../common"; /** Convert manual thinking budgets to Anthropic adaptive-thinking effort. */ -export const anthropicAdaptiveThinkingOnPayload: CuaPayloadHook = (payload, model) => { +export const anthropicAdaptiveThinkingOnPayload: LoopPayloadHook = (payload, model) => { if (!isAdaptiveThinkingModel(model) || !isRecord(payload)) return undefined; const thinking = payload.thinking; if (!isRecord(thinking) || thinking.type !== "enabled") return undefined; diff --git a/packages/ai/src/providers/anthropic/browser-fallback.ts b/packages/loop/src/pi/providers/anthropic/browser-fallback.ts similarity index 90% rename from packages/ai/src/providers/anthropic/browser-fallback.ts rename to packages/loop/src/pi/providers/anthropic/browser-fallback.ts index 83e625a0..8bb051d7 100644 --- a/packages/ai/src/providers/anthropic/browser-fallback.ts +++ b/packages/loop/src/pi/providers/anthropic/browser-fallback.ts @@ -11,8 +11,8 @@ import { type SimpleStreamOptions, type StreamOptions, } from "@earendil-works/pi-ai"; -import type { CuaAnthropicBrowserFallback } from "../../tool-catalog"; -import type { CuaSimpleStreamOptions } from "../common"; +import type { LoopAnthropicBrowserFallback } from "../../../core/tool-catalog"; +import type { LoopSimpleStreamOptions } from "../common"; const supportedCredentials = new Set(); const unsupportedCredentials = new Set(); @@ -25,13 +25,13 @@ export function withAnthropicBrowserFallback(base: Provider): Provider { base.stream.bind(base), model, context, - options as CuaSimpleStreamOptions | undefined, + options as LoopSimpleStreamOptions | undefined, ), streamSimple: (model, context, options) => streamWithFallback( base.streamSimple.bind(base), model, context, - options as CuaSimpleStreamOptions | undefined, + options as LoopSimpleStreamOptions | undefined, ), }; } @@ -46,9 +46,9 @@ function streamWithFallback( start: StartStream, model: Model, context: Context, - options: CuaSimpleStreamOptions | undefined, + options: LoopSimpleStreamOptions | undefined, ): AssistantMessageEventStream { - const fallback = options?.cuaIncomingToolPlan?.anthropicBrowserFallback; + const fallback = options?.loopIncomingToolPlan?.anthropicBrowserFallback; if (!fallback) return start(model, context, options); const credential = credentialKey(model, options); if (supportedCredentials.has(credential)) return start(model, context, options); @@ -64,8 +64,8 @@ async function detectAccessAndRelay( start: StartStream, model: Model, context: Context, - options: CuaSimpleStreamOptions, - fallback: CuaAnthropicBrowserFallback, + options: LoopSimpleStreamOptions, + fallback: LoopAnthropicBrowserFallback, credential: string, ): Promise { let committed = false; @@ -120,9 +120,9 @@ async function relay(output: AssistantMessageEventStream, source: AssistantMessa } function fallbackOptions( - options: CuaSimpleStreamOptions, - fallback: CuaAnthropicBrowserFallback, -): CuaSimpleStreamOptions { + options: LoopSimpleStreamOptions, + fallback: LoopAnthropicBrowserFallback, +): LoopSimpleStreamOptions { const originalOnPayload = options.onPayload; return { ...options, @@ -134,7 +134,7 @@ function fallbackOptions( }; } -function replaceNativeDeclaration(payload: unknown, fallback: CuaAnthropicBrowserFallback): unknown { +function replaceNativeDeclaration(payload: unknown, fallback: LoopAnthropicBrowserFallback): unknown { if (!isRecord(payload) || !Array.isArray(payload.tools)) return payload; let replaced = false; const tools = payload.tools.map((tool) => { @@ -164,7 +164,7 @@ function removeHeaderToken( /** Return whether an Anthropic provider error specifically denies the selected native browser beta. */ export function isAnthropicBrowserAccessError( message: string | undefined, - fallback: Pick, + fallback: Pick, ): boolean { if (!message) return false; const value = message.toLowerCase(); diff --git a/packages/ai/src/providers/anthropic/capabilities.ts b/packages/loop/src/pi/providers/anthropic/capabilities.ts similarity index 100% rename from packages/ai/src/providers/anthropic/capabilities.ts rename to packages/loop/src/pi/providers/anthropic/capabilities.ts diff --git a/packages/ai/src/providers/anthropic/native.ts b/packages/loop/src/pi/providers/anthropic/native.ts similarity index 96% rename from packages/ai/src/providers/anthropic/native.ts rename to packages/loop/src/pi/providers/anthropic/native.ts index fb123cde..7e018d14 100644 --- a/packages/ai/src/providers/anthropic/native.ts +++ b/packages/loop/src/pi/providers/anthropic/native.ts @@ -1,4 +1,4 @@ -import type { CuaAction, CuaMouseButton } from "../../actions/index"; +import type { ComputerUseAction, MouseButton } from "../../../core/actions/index"; interface NativeInput { action: string; @@ -15,7 +15,7 @@ function asNativeInput(args: unknown): NativeInput { const MAX_KEY_REPEAT = 100; /** Map one `computer_20260701` tool input onto canonical computer-plane actions. */ -export function mapNativeComputerInput(input: NativeInput): CuaAction[] { +export function mapNativeComputerInput(input: NativeInput): ComputerUseAction[] { switch (input.action) { case "screenshot": return [{ type: "screenshot" }]; @@ -65,7 +65,7 @@ export function mapNativeComputerInput(input: NativeInput): CuaAction[] { } /** Map one `browser_20260701` tool input onto canonical browser-plane actions. */ -export function mapNativeBrowserInput(input: NativeInput): CuaAction[] { +export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] { const tab = tabId(input); switch (input.action) { case "navigate": @@ -133,7 +133,7 @@ export function mapNativeBrowserInput(input: NativeInput): CuaAction[] { } } -function click(input: NativeInput, button: CuaMouseButton): CuaAction & { type: "click" } { +function click(input: NativeInput, button: MouseButton): ComputerUseAction & { type: "click" } { const point = input.coordinate === undefined ? {} : coordinate(input.coordinate, "coordinate"); return { type: "click", ...point, button, ...holdKeys(input.text) }; } diff --git a/packages/ai/src/providers/common.ts b/packages/loop/src/pi/providers/common.ts similarity index 79% rename from packages/ai/src/providers/common.ts rename to packages/loop/src/pi/providers/common.ts index 4facab45..5546e0dc 100644 --- a/packages/ai/src/providers/common.ts +++ b/packages/loop/src/pi/providers/common.ts @@ -7,20 +7,12 @@ import type { SimpleStreamOptions, StreamOptions, } from "@earendil-works/pi-ai"; -import type { CuaIncomingToolPlan } from "../tool-catalog"; +import type { LoopIncomingToolPlan } from "../../core/tool-catalog"; -/** Prefix a bare hostname/path before browser navigation. */ -export function normalizeGotoUrl(value: unknown): string | undefined { - if (typeof value !== "string") return undefined; - const url = value.trim(); - if (!url) return undefined; - return /^[a-z][a-z0-9+.-]*:\/\//i.test(url) ? url : `https://${url}`; -} - -/** pi stream options plus CUA adapter request controls. */ -export interface CuaSimpleStreamOptions extends SimpleStreamOptions, ResponseThreadingOptions { +/** pi stream options plus Loop adapter request controls. */ +export interface LoopSimpleStreamOptions extends SimpleStreamOptions, ResponseThreadingOptions { /** @internal Identity-addressed native call dispatch for custom providers. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; + loopIncomingToolPlan?: LoopIncomingToolPlan; } /** Per-request control for Responses API continuation. */ @@ -78,4 +70,4 @@ export function responseThreadingDelta(messages: readonly Message[], api: Api): return { deltaMessages: [...messages] }; } -export type CuaPayloadHook = (payload: unknown, model: Model) => unknown | Promise; +export type LoopPayloadHook = (payload: unknown, model: Model) => unknown | Promise; diff --git a/packages/ai/src/providers/google/provider.ts b/packages/loop/src/pi/providers/google/provider.ts similarity index 94% rename from packages/ai/src/providers/google/provider.ts rename to packages/loop/src/pi/providers/google/provider.ts index dfbc9ae4..4c4ddbda 100644 --- a/packages/ai/src/providers/google/provider.ts +++ b/packages/loop/src/pi/providers/google/provider.ts @@ -12,14 +12,14 @@ import { type ThinkingLevel, type ToolCall, } from "@earendil-works/pi-ai"; -import type { CuaIncomingToolPlan } from "../../tool-catalog"; +import type { LoopIncomingToolPlan } from "../../../core/tool-catalog"; import { responseThreadingDelta, responseThreadingEnabled, type ResponseThreadingOptions, } from "../common"; -export const GOOGLE_CUA_INTERACTIONS_API = "google-cua-interactions"; +export const GOOGLE_INTERACTIONS_API = "google-interactions"; const GOOGLE_NATIVE_ALIASES: Readonly> = Object.freeze({ "screenshot:take_screenshot": "take_screenshot", @@ -31,10 +31,10 @@ const GOOGLE_NATIVE_ALIASES: Readonly> = Object.freeze({ export interface GoogleInteractionsOptions extends StreamOptions, ResponseThreadingOptions { reasoning?: ThinkingLevel; /** @internal Identity-addressed native action dispatch. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; + loopIncomingToolPlan?: LoopIncomingToolPlan; } -export const streamGoogleInteractions: StreamFunction = ( +export const streamGoogleInteractions: StreamFunction = ( model, context, options, @@ -44,7 +44,7 @@ export const streamGoogleInteractions: StreamFunction = +export const streamSimpleGoogleInteractions: StreamFunction = (model, context, options) => streamGoogleInteractions(model, context, options); async function run( @@ -58,7 +58,7 @@ async function run( const apiKey = options?.apiKey || process.env.GOOGLE_API_KEY || process.env.GEMINI_API_KEY; if (!apiKey) throw new Error("missing Google API key"); const threading = responseThreadingEnabled(options) - ? responseThreadingDelta(context.messages, GOOGLE_CUA_INTERACTIONS_API) + ? responseThreadingDelta(context.messages, GOOGLE_INTERACTIONS_API) : { deltaMessages: [...context.messages] }; let payload: Record = { model: model.id, @@ -100,9 +100,9 @@ async function run( const contents = Array.isArray(interaction.steps) ? interaction.steps : Array.isArray(interaction.outputs) ? interaction.outputs : []; - const nativeToolNames = new Set(options?.cuaIncomingToolPlan?.nativeToolNames ?? []); + const nativeToolNames = new Set(options?.loopIncomingToolPlan?.nativeToolNames ?? []); const ordinaryToolNames = new Set((context.tools ?? []).map((tool) => tool.name).filter((name) => !nativeToolNames.has(name))); - for (const raw of contents) emitContent(stream, output, raw, options?.cuaIncomingToolPlan, ordinaryToolNames); + for (const raw of contents) emitContent(stream, output, raw, options?.loopIncomingToolPlan, ordinaryToolNames); if (output.content.some((content) => content.type === "toolCall")) output.stopReason = "toolUse"; if (interaction.status === "incomplete") output.stopReason = "length"; if (interaction.status === "failed" || interaction.status === "cancelled") { @@ -183,7 +183,7 @@ function emitContent( stream: ReturnType, output: AssistantMessage, raw: unknown, - incoming: CuaIncomingToolPlan | undefined, + incoming: LoopIncomingToolPlan | undefined, ordinaryToolNames: ReadonlySet, ): void { if (!isRecord(raw)) return; @@ -217,7 +217,7 @@ function emitContent( function resolveGoogleToolName( name: string, - incoming: CuaIncomingToolPlan | undefined, + incoming: LoopIncomingToolPlan | undefined, ordinaryToolNames: ReadonlySet, ): string { // Caller function names are exact and take precedence over provider-observed diff --git a/packages/ai/src/providers/openai/provider.ts b/packages/loop/src/pi/providers/openai/provider.ts similarity index 95% rename from packages/ai/src/providers/openai/provider.ts rename to packages/loop/src/pi/providers/openai/provider.ts index 4221b12a..3a107007 100644 --- a/packages/ai/src/providers/openai/provider.ts +++ b/packages/loop/src/pi/providers/openai/provider.ts @@ -22,11 +22,11 @@ import { import { createGrammarToolInputProperties } from "@earendil-works/pi-ai/api/constrained-sampling"; import { clampOpenAIPromptCacheKey } from "@earendil-works/pi-ai/api/openai-prompt-cache"; import { buildBaseOptions } from "@earendil-works/pi-ai/api/simple-options"; -import type { CuaIncomingToolPlan } from "../../tool-catalog"; -import type { CuaSimpleStreamOptions } from "../common"; +import type { LoopIncomingToolPlan } from "../../../core/tool-catalog"; +import type { LoopSimpleStreamOptions } from "../common"; -/** CUA-owned api id for OpenAI's native computer tool, derived onto the model by compileCuaToolCatalog when that tool is selected. */ -export const OPENAI_CUA_COMPUTER_API = "openai-cua-computer"; +/** Loop-owned api id for OpenAI's native computer tool, derived onto the model by compileLoopToolCatalog when that tool is selected. */ +export const OPENAI_COMPUTER_USE_API = "openai-computer-use"; /** * 64x64 black PNG, used when a computer action produced no screenshot so the @@ -37,21 +37,21 @@ const BLANK_SCREENSHOT_DATA_URL = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgA export interface OpenAIResponsesOptions extends PiOpenAIResponsesOptions { /** @internal Identity-addressed native dispatch compiled from selected tools. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; + loopIncomingToolPlan?: LoopIncomingToolPlan; } /** The request fields both the function-tool and native-computer paths read, whichever option shape the caller passed. */ type OpenAIRequestOptions = Pick; /** - * The one dispatch cua-ai's OpenAI provider wrapper cannot derive from the + * The one dispatch the OpenAI provider wrapper cannot derive from the * model's `api`: whether the transcript carries state (a deferred tool-search * addition, or a replayed function-call namespace) that only this adapter * round-trips, because pi-ai's builtin `openai-responses` transport does not * parse or replay `function_call`'s `namespace` field. Every other dispatch — * including OpenAI's native computer tool — is decided by `model.api` instead. */ -export function requiresCuaOpenAINamespaceAdapter(context: Context): boolean { +export function requiresOpenAINamespaceAdapter(context: Context): boolean { for (const message of context.messages) { if (message.role === "toolResult" && (message.addedToolNames?.length ?? 0) > 0) return true; if (message.role === "assistant" && message.content.some((part) => part.type === "toolCall" && toolCallNamespace(part))) return true; @@ -62,13 +62,13 @@ export function requiresCuaOpenAINamespaceAdapter(context: Context): boolean { export const streamOpenAIResponses: StreamFunction<"openai-responses", OpenAIResponsesOptions> = (model, context, options) => streamOpenAIFunctionTools(model, context, options); -export const streamSimpleOpenAIResponses: StreamFunction<"openai-responses", CuaSimpleStreamOptions> = (model, context, options) => { +export const streamSimpleOpenAIResponses: StreamFunction<"openai-responses", LoopSimpleStreamOptions> = (model, context, options) => { const base = buildBaseOptions(model, context, options, options?.apiKey); const clampedReasoning = options?.reasoning ? clampThinkingLevel(model, options.reasoning) : undefined; return streamOpenAIFunctionTools(model, context, { ...base, reasoningEffort: clampedReasoning === "off" ? undefined : clampedReasoning, - cuaIncomingToolPlan: options?.cuaIncomingToolPlan, + loopIncomingToolPlan: options?.loopIncomingToolPlan, }); }; @@ -287,21 +287,21 @@ function applyServiceTierPricing( usage.cost.total = usage.cost.input + usage.cost.output + usage.cost.cacheRead + usage.cost.cacheWrite; } -/** Responses adapter for models the catalog compiled onto {@link OPENAI_CUA_COMPUTER_API}, i.e. those whose selected tools include OpenAI's native computer. */ -export const streamOpenAICuaComputer: StreamFunction = (model, context, options) => +/** Responses adapter for models the catalog compiled onto {@link OPENAI_COMPUTER_USE_API}, i.e. those whose selected tools include OpenAI's native computer. */ +export const streamOpenAIComputerUse: StreamFunction = (model, context, options) => streamOpenAINativeComputer(model as unknown as Model<"openai-responses">, context, options); function streamOpenAINativeComputer( model: Model<"openai-responses">, context: Context, - options: OpenAIResponsesOptions | CuaSimpleStreamOptions | undefined, + options: OpenAIResponsesOptions | LoopSimpleStreamOptions | undefined, ) { const stream = createAssistantMessageEventStream(); const output = initialAssistantMessage(model); void (async () => { try { const apiKey = openAIApiKey(options); - const nativeName = options?.cuaIncomingToolPlan?.openaiComputerName; + const nativeName = options?.loopIncomingToolPlan?.openaiComputerName; if (!nativeName) throw new Error("OpenAI native computer incoming plan is missing"); const placement = splitDeferredTools(context); let payload: Record = { @@ -465,7 +465,7 @@ function convertMessages(messages: readonly Context["messages"][number][], nativ return [tool]; }); if (added.length > 0) { - const callId = `cua_tool_load_${message.toolCallId}`.replace(/[^A-Za-z0-9_-]/g, "_").slice(0, 64); + const callId = `loop_tool_load_${message.toolCallId}`.replace(/[^A-Za-z0-9_-]/g, "_").slice(0, 64); input.push({ type: "tool_search_call", call_id: callId, execution: "client", status: "completed", arguments: { query: added.map((tool) => tool.name).join(" "), limit: added.length } }); input.push({ type: "tool_search_output", call_id: callId, execution: "client", status: "completed", tools: convertTools(added, true) }); } diff --git a/packages/ai/test/anthropic-browser-fallback.test.ts b/packages/loop/test/anthropic-browser-fallback.test.ts similarity index 85% rename from packages/ai/test/anthropic-browser-fallback.test.ts rename to packages/loop/test/anthropic-browser-fallback.test.ts index 98aa5f6c..e84ff796 100644 --- a/packages/ai/test/anthropic-browser-fallback.test.ts +++ b/packages/loop/test/anthropic-browser-fallback.test.ts @@ -8,13 +8,9 @@ import { type SimpleStreamOptions, } from "@earendil-works/pi-ai"; import { describe, expect, it } from "vitest"; -import { - compileCuaToolCatalog, - cua, - getCuaModel, - type CuaSimpleStreamOptions, -} from "../src/index"; -import { withAnthropicBrowserFallback } from "../src/providers/anthropic/browser-fallback"; +import { getLoopModel, type LoopSimpleStreamOptions } from "../src/pi/index"; +import { compileLoopToolCatalog, loop } from "../src/index"; +import { withAnthropicBrowserFallback } from "../src/pi/providers/anthropic/browser-fallback"; const context: Context = { systemPrompt: "", @@ -24,10 +20,10 @@ const context: Context = { describe("Anthropic native browser access fallback", () => { it("retries with the equivalent function tool and remembers the credential", async () => { - const model = getCuaModel("anthropic:claude-opus-5"); - const catalog = compileCuaToolCatalog({ + const model = getLoopModel("anthropic:claude-opus-5"); + const catalog = compileLoopToolCatalog({ model, - requestedTools: [cua.providers.anthropic.tools.browser()], + requestedTools: [loop.providers.anthropic.tools.browser()], }); const payloads: Array<{ tools: unknown[]; headers: SimpleStreamOptions["headers"] }> = []; let calls = 0; @@ -41,10 +37,10 @@ describe("Anthropic native browser access fallback", () => { : message(selectedModel, "toolUse"); }); const provider = withAnthropicBrowserFallback(base); - const options: CuaSimpleStreamOptions = { + const options: LoopSimpleStreamOptions = { apiKey: "fallback-test-key", headers: catalog.headers.merge({ "anthropic-beta": "other-beta" }), - cuaIncomingToolPlan: catalog.incoming, + loopIncomingToolPlan: catalog.incoming, onPayload: (payload) => catalog.payload.apply(payload, model), }; @@ -62,10 +58,10 @@ describe("Anthropic native browser access fallback", () => { }); it("does not hide unrelated provider errors", async () => { - const model = getCuaModel("anthropic:claude-opus-5"); - const catalog = compileCuaToolCatalog({ + const model = getLoopModel("anthropic:claude-opus-5"); + const catalog = compileLoopToolCatalog({ model, - requestedTools: [cua.providers.anthropic.tools.browser()], + requestedTools: [loop.providers.anthropic.tools.browser()], }); let calls = 0; const provider = withAnthropicBrowserFallback(fakeProvider(async (selectedModel) => { @@ -75,7 +71,7 @@ describe("Anthropic native browser access fallback", () => { const result = await provider.streamSimple(model, { ...context, tools: [...catalog.toolDeclarations] }, { apiKey: "non-access-error-test-key", - cuaIncomingToolPlan: catalog.incoming, + loopIncomingToolPlan: catalog.incoming, }).result(); expect(calls).toBe(1); diff --git a/packages/ai/test/anthropic-native.integration.test.ts b/packages/loop/test/anthropic-native.integration.test.ts similarity index 75% rename from packages/ai/test/anthropic-native.integration.test.ts rename to packages/loop/test/anthropic-native.integration.test.ts index 3ad553f1..3337db82 100644 --- a/packages/ai/test/anthropic-native.integration.test.ts +++ b/packages/loop/test/anthropic-native.integration.test.ts @@ -1,9 +1,6 @@ import { describe, expect, it } from "vitest"; -import { - compileCuaToolCatalog, - createCuaModels, - cua, -} from "../src/index"; +import { createLoopModels } from "../src/pi/index"; +import { compileLoopToolCatalog, loop } from "../src/index"; const apiKey = process.env.ANTHROPIC_API_KEY; const liveIt = apiKey ? it : it.skip; @@ -11,13 +8,13 @@ const liveIt = apiKey ? it : it.skip; const cases = [ { name: "computer", - tool: cua.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true }), + tool: loop.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true }), prompt: "Use the computer tool to take one screenshot.", expectedAction: "screenshot", }, { name: "browser", - tool: cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true }), + tool: loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true }), prompt: "Use the browser tool to navigate to example.com.", expectedAction: "navigate", }, @@ -26,11 +23,11 @@ const cases = [ describe("Anthropic early-access native tools", () => { for (const current of cases) { liveIt(`${current.name} survives catalog and pi-ai serialization`, async () => { - const catalog = compileCuaToolCatalog({ + const catalog = compileLoopToolCatalog({ model: "anthropic:claude-opus-5", requestedTools: [current.tool], }); - const response = await createCuaModels().complete( + const response = await createLoopModels().complete( catalog.model, { systemPrompt: "Use only the explicitly supplied tool.", @@ -41,7 +38,7 @@ describe("Anthropic early-access native tools", () => { apiKey, maxTokens: 1_024, headers: catalog.headers.merge(), - cuaIncomingToolPlan: catalog.incoming, + loopIncomingToolPlan: catalog.incoming, onPayload: (payload, model) => catalog.payload.apply(payload, model), }, ); diff --git a/packages/ai/test/anthropic-payload.test.ts b/packages/loop/test/anthropic-payload.test.ts similarity index 68% rename from packages/ai/test/anthropic-payload.test.ts rename to packages/loop/test/anthropic-payload.test.ts index 64ebc6ec..c44ccfdc 100644 --- a/packages/ai/test/anthropic-payload.test.ts +++ b/packages/loop/test/anthropic-payload.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; -import { getCuaModel } from "../src/models"; -import { anthropicAdaptiveThinkingOnPayload } from "../src/providers/anthropic/adaptive-thinking"; +import { getLoopModel } from "../src/pi/models"; +import { anthropicAdaptiveThinkingOnPayload } from "../src/pi/providers/anthropic/adaptive-thinking"; describe("anthropicAdaptiveThinkingOnPayload", () => { it("converts Sonnet 5 manual thinking to adaptive thinking with effort", () => { @@ -10,14 +10,14 @@ describe("anthropicAdaptiveThinkingOnPayload", () => { messages: [], }; - expect(anthropicAdaptiveThinkingOnPayload(payload, getCuaModel("anthropic:claude-sonnet-5"))).toEqual({ + expect(anthropicAdaptiveThinkingOnPayload(payload, getLoopModel("anthropic:claude-sonnet-5"))).toEqual({ thinking: { type: "adaptive" }, output_config: { other: true, effort: "medium" }, messages: [], }); }); - it("converts adaptive-thinking Anthropic CUA models", () => { + it("converts adaptive-thinking Anthropic Loop models", () => { const payload = { thinking: { type: "enabled", budget_tokens: 8_192 } }; for (const ref of [ @@ -28,21 +28,21 @@ describe("anthropicAdaptiveThinkingOnPayload", () => { "anthropic:claude-opus-4-7", "anthropic:claude-fable-5", ] as const) { - expect(anthropicAdaptiveThinkingOnPayload(payload, getCuaModel(ref))).toMatchObject({ + expect(anthropicAdaptiveThinkingOnPayload(payload, getLoopModel(ref))).toMatchObject({ thinking: { type: "adaptive" }, output_config: { effort: "medium" }, }); } }); - it("leaves older manual-thinking Anthropic CUA models unchanged", () => { + it("leaves older manual-thinking Anthropic Loop models unchanged", () => { const payload = { thinking: { type: "enabled", budget_tokens: 8_192 } }; - expect(anthropicAdaptiveThinkingOnPayload(payload, getCuaModel("anthropic:claude-sonnet-4-5"))).toBeUndefined(); + expect(anthropicAdaptiveThinkingOnPayload(payload, getLoopModel("anthropic:claude-sonnet-4-5"))).toBeUndefined(); }); it("maps old budget levels to supported Sonnet 5 effort levels", () => { - const model = getCuaModel("anthropic:claude-sonnet-5"); + const model = getLoopModel("anthropic:claude-sonnet-5"); const effortFor = (budget_tokens: number) => (anthropicAdaptiveThinkingOnPayload({ thinking: { type: "enabled", budget_tokens } }, model) as { output_config: { effort: string } }) .output_config.effort; diff --git a/packages/loop/test/api-keys.test.ts b/packages/loop/test/api-keys.test.ts new file mode 100644 index 00000000..e1323438 --- /dev/null +++ b/packages/loop/test/api-keys.test.ts @@ -0,0 +1,65 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { + getLoopEnvApiKey, + getLoopEnvApiKeyForModel, + loopApiKeyEnvVarsForProvider, + requireLoopEnvApiKey, +} from "../src/pi/index"; + +const ENV_KEYS = [ + "OPENAI_API_KEY", + "ANTHROPIC_OAUTH_TOKEN", + "ANTHROPIC_API_KEY", + "GOOGLE_API_KEY", + "GEMINI_API_KEY", + "META_API_KEY", + "XAI_API_KEY", + "MOONSHOT_API_KEY", + "OPENROUTER_API_KEY", +] as const; + +const ORIGINAL_ENV = new Map(ENV_KEYS.map((key) => [key, process.env[key]])); + +afterEach(() => { + for (const key of ENV_KEYS) { + const value = ORIGINAL_ENV.get(key); + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } +}); + +describe("loop api key helpers", () => { + it("maps provider names to expected environment variables", () => { + expect(loopApiKeyEnvVarsForProvider("openai")).toEqual(["OPENAI_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("google")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("gemini")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("xai")).toEqual(["XAI_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("moonshotai")).toEqual(["MOONSHOT_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("moonshot")).toEqual(["MOONSHOT_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("openrouter")).toEqual(["OPENROUTER_API_KEY"]); + expect(loopApiKeyEnvVarsForProvider("unknown")).toEqual([]); + }); + + it("resolves provider api keys with fallback order", () => { + delete process.env.GOOGLE_API_KEY; + process.env.GEMINI_API_KEY = "gemini"; + expect(getLoopEnvApiKey("google")).toBe("gemini"); + process.env.GOOGLE_API_KEY = "google"; + expect(getLoopEnvApiKey("google")).toBe("google"); + }); + + it("resolves keys from model refs", () => { + process.env.OPENAI_API_KEY = "openai"; + expect(getLoopEnvApiKeyForModel("openai:gpt-5.5")).toBe("openai"); + process.env.XAI_API_KEY = "xai"; + expect(getLoopEnvApiKeyForModel("xai:grok-4.5")).toBe("xai"); + process.env.MOONSHOT_API_KEY = "moonshot"; + expect(getLoopEnvApiKeyForModel("moonshotai:kimi-k3")).toBe("moonshot"); + process.env.OPENROUTER_API_KEY = "openrouter"; + expect(getLoopEnvApiKeyForModel("openrouter:moonshotai/kimi-k3")).toBe("openrouter"); + }); + + it("throws readable errors when missing", () => { + delete process.env.META_API_KEY; + }); +}); diff --git a/packages/agent/test/attach-session.test.ts b/packages/loop/test/attach-session.test.ts similarity index 83% rename from packages/agent/test/attach-session.test.ts rename to packages/loop/test/attach-session.test.ts index 1e24f873..17a1ffdb 100644 --- a/packages/agent/test/attach-session.test.ts +++ b/packages/loop/test/attach-session.test.ts @@ -1,29 +1,27 @@ import { describe, expect, it } from "vitest"; import { - createAssistantMessageEventStream, - createCuaModels, - cua, - getCuaModel, - GOOGLE_CUA_INTERACTIONS_API, type AssistantMessage, type Context, + createAssistantMessageEventStream, type Model, type Models, -} from "@onkernel/cua-ai"; -import type Kernel from "@onkernel/sdk"; +} from "@earendil-works/pi-ai"; import { Agent, AgentHarness, - attach, - InMemorySessionRepo, type AgentMessage, type AgentTool, - type CuaHarnessTool, - type CuaModelInput, - type KernelBrowser, + attach, + createLoopModels, + getLoopModel, + GOOGLE_INTERACTIONS_API, + InMemorySessionRepo, + type LoopModelInput, type Session, type StreamFn, -} from "../src/index"; +} from "../src/pi/index"; +import { type KernelBrowser, loop, type LoopHarnessTool } from "../src/index"; +import type Kernel from "@onkernel/sdk"; const browser = { session_id: "browser_123", viewport: { width: 1440, height: 900 } } as KernelBrowser; const client = {} as Kernel; @@ -65,7 +63,7 @@ function callerTool(name: string, execute?: AgentTool["execute"]): AgentTool { } function modelsFromStream(streamFn: StreamFn, provider = "openai"): Models { - const models = createCuaModels(); + const models = createLoopModels(); models.setProvider({ id: provider, name: "scripted", @@ -82,13 +80,13 @@ function modelsFromStream(streamFn: StreamFn, provider = "openai"): Models { * harness, and recompile-then-apply to change it. */ async function openSession(options: { - model: CuaModelInput; - tools: readonly CuaHarnessTool[]; + model: LoopModelInput; + tools: readonly LoopHarnessTool[]; models?: Models; }): Promise<{ harness: AgentHarness; session: Session; - select: (model: CuaModelInput, tools: readonly CuaHarnessTool[]) => Promise; + select: (model: LoopModelInput, tools: readonly LoopHarnessTool[]) => Promise; }> { const session = await new InMemorySessionRepo().create(); const handle = attach({ browser, client, models: options.models }); @@ -112,21 +110,21 @@ async function openSession(options: { } describe("compiling a pair", () => { - it("compiles exact native-only, native-plus-CUA, Playwright-only, and browser-act-only catalogs", () => { + it("compiles exact native-only, native-plus-Loop, Playwright-only, and browser-act-only catalogs", () => { const custom = callerTool("customer_lookup"); const cases = [ { model: "anthropic:claude-opus-5" as const, - tools: [cua.providers.anthropic.tools.browser(), custom], + tools: [loop.providers.anthropic.tools.browser(), custom], names: ["browser", "customer_lookup"], }, { model: "openai:gpt-5.5" as const, - tools: [cua.providers.openai.tools.computer(), cua.tools.browser.snapshot(), cua.tools.browser.act()], + tools: [loop.providers.openai.tools.computer(), loop.tools.browser.snapshot(), loop.tools.browser.act()], names: ["computer", "browser_snapshot", "browser_act"], }, - { model: "openai:gpt-5.5" as const, tools: [cua.tools.playwright()], names: ["playwright_execute"] }, - { model: "openai:gpt-5.5" as const, tools: [cua.tools.browser.act()], names: ["browser_act"] }, + { model: "openai:gpt-5.5" as const, tools: [loop.tools.playwright()], names: ["playwright_execute"] }, + { model: "openai:gpt-5.5" as const, tools: [loop.tools.browser.act()], names: ["browser_act"] }, ]; const handle = attach({ browser, client }); for (const entry of cases) { @@ -135,10 +133,10 @@ describe("compiling a pair", () => { }); it("resolves a ref through the supplied collection, falling back to the registry", () => { - const models = createCuaModels(); + const models = createLoopModels(); const openai = models.getProvider("openai")!; - const first = { ...getCuaModel("openai:gpt-5.5"), baseUrl: "https://first.example" }; - const second = { ...getCuaModel("openai:gpt-5.6-sol"), baseUrl: "https://second.example" }; + const first = { ...getLoopModel("openai:gpt-5.5"), baseUrl: "https://first.example" }; + const second = { ...getLoopModel("openai:gpt-5.6-sol"), baseUrl: "https://second.example" }; models.setProvider({ ...openai, getModels: () => [first, second] }); const handle = attach({ browser, client, models }); @@ -153,14 +151,14 @@ describe("compiling a pair", () => { const keep = callerTool("keep"); const { harness, select } = await openSession({ model: "openai:gpt-5.5", tools: [keep] }); - await expect(select("openai:gpt-5.5", [cua.providers.anthropic.tools.browser()])).rejects.toThrow(/requires a anthropic model/); + await expect(select("openai:gpt-5.5", [loop.providers.anthropic.tools.browser()])).rejects.toThrow(/requires a anthropic model/); expect(harness.getModel().id).toBe("gpt-5.5"); expect(harness.getTools().map((tool) => tool.name)).toEqual(["keep"]); }); it("retains protocol-required OpenAI native computer screenshot results outside the image replay limit", async () => { const contexts: Context[] = []; - const model = getCuaModel("openai:gpt-5.5"); + const model = getLoopModel("openai:gpt-5.5"); const image = { type: "image" as const, data: "c2NyZWVuc2hvdA==", mimeType: "image/png" }; const messages: AgentMessage[] = [ assistant(model, [{ type: "toolCall", id: "native-shot", name: "computer", arguments: { action: { type: "screenshot" } } }], "toolUse"), @@ -174,7 +172,7 @@ describe("compiling a pair", () => { toolResultImageReplayLimit: 0, models: modelsFromStream(scriptedStream([(selected) => assistant(selected)], contexts)), }); - const compiled = handle.compile({ model, tools: [cua.providers.openai.tools.computer(), callerTool("ordinary")] }); + const compiled = handle.compile({ model, tools: [loop.providers.openai.tools.computer(), callerTool("ordinary")] }); const agent = new Agent({ streamFn: (selected, context, options) => compiled.models.streamSimple(selected, context, options), initialState: { model: compiled.model, tools: [...compiled.agentTools], messages }, @@ -194,7 +192,7 @@ describe("applying a pair to a live harness", () => { const script = scriptedStream([(selected) => assistant(selected)]); const { harness, select } = await openSession({ model: "google:gemini-3.6-flash", - tools: [cua.tools.browser.snapshot()], + tools: [loop.tools.browser.snapshot()], models: modelsFromStream((model, context, options) => { streamedApis.push(model.api); return script(model, context, options); @@ -202,11 +200,11 @@ describe("applying a pair to a live harness", () => { }); expect(harness.getModel().api).toBe("google-generative-ai"); - await select("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); - expect(harness.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + await select("google:gemini-3.6-flash", loop.providers.google.toolsets.browser()); + expect(harness.getModel().api).toBe(GOOGLE_INTERACTIONS_API); await harness.prompt("go"); - expect(streamedApis).toEqual([GOOGLE_CUA_INTERACTIONS_API]); + expect(streamedApis).toEqual([GOOGLE_INTERACTIONS_API]); }); it("records no model change when the derived transport is unchanged", async () => { @@ -220,11 +218,11 @@ describe("applying a pair to a live harness", () => { it("records one model change for a switch that moves both the model and its transport", async () => { const { harness, session, select } = await openSession({ model: "google:gemini-3.6-flash", - tools: cua.providers.google.toolsets.browser(), + tools: loop.providers.google.toolsets.browser(), }); - expect(harness.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + expect(harness.getModel().api).toBe(GOOGLE_INTERACTIONS_API); - await select("openai:gpt-5.5", [cua.tools.browser.snapshot()]); + await select("openai:gpt-5.5", [loop.tools.browser.snapshot()]); expect(harness.getModel().api).toBe("openai-responses"); expect((await session.getBranch()).filter((entry) => entry.type === "model_change")).toHaveLength(1); @@ -239,7 +237,7 @@ describe("applying a pair to a live harness", () => { }, "google"); const session = await new InMemorySessionRepo().create(); const handle = attach({ browser, client, models }); - const first = handle.compile({ model: "google:gemini-3.6-flash", tools: [cua.tools.browser.snapshot()] }); + const first = handle.compile({ model: "google:gemini-3.6-flash", tools: [loop.tools.browser.snapshot()] }); const harness = new AgentHarness({ session, model: first.model, @@ -271,13 +269,13 @@ describe("applying a pair to a live harness", () => { }; }) as typeof harness.subscribe; - await handle.compile({ model: "google:gemini-3.6-flash", tools: cua.providers.google.toolsets.browser() }).apply(harness); + await handle.compile({ model: "google:gemini-3.6-flash", tools: loop.providers.google.toolsets.browser() }).apply(harness); const afterSwap = live; // The caller still holds the first pair's release. Calling it must neither // strand `models` with no live catalog nor drop the handle's grip on the // pair that *is* live — otherwise the next activation cannot release it. releaseFirst(); - await handle.compile({ model: "google:gemini-3.6-flash", tools: [cua.tools.browser.snapshot()] }).apply(harness); + await handle.compile({ model: "google:gemini-3.6-flash", tools: [loop.tools.browser.snapshot()] }).apply(harness); expect(live).toBe(afterSwap); await harness.prompt("go"); @@ -295,7 +293,7 @@ describe("applying a pair to a live harness", () => { }); const { harness } = await openSession({ model: "anthropic:claude-opus-5", - tools: [cua.providers.anthropic.tools.browser(), failing, succeeding], + tools: [loop.providers.anthropic.tools.browser(), failing, succeeding], models: modelsFromStream(scriptedStream([ (model) => assistant(model, [{ type: "toolCall", id: "fail", name: "failing", arguments: {} }], "toolUse"), (model) => assistant(model, [{ type: "toolCall", id: "succeed", name: "succeeding", arguments: {} }], "toolUse"), diff --git a/packages/agent/test/attach.test.ts b/packages/loop/test/attach.test.ts similarity index 80% rename from packages/agent/test/attach.test.ts rename to packages/loop/test/attach.test.ts index bb5a71bc..8ea36fc9 100644 --- a/packages/agent/test/attach.test.ts +++ b/packages/loop/test/attach.test.ts @@ -1,18 +1,24 @@ import { describe, expect, it } from "vitest"; import { - createAssistantMessageEventStream, - createCuaModels, - cua, - GOOGLE_CUA_INTERACTIONS_API, - OPENAI_CUA_COMPUTER_API, type AssistantMessage, type Context, - type CuaSimpleStreamOptions, + createAssistantMessageEventStream, type Model, -} from "@onkernel/cua-ai"; +} from "@earendil-works/pi-ai"; +import { + Agent, + AgentHarness, + attach, + createLoopModels, + GOOGLE_INTERACTIONS_API, + InMemorySessionRepo, + type LoopSimpleStreamOptions, + OPENAI_COMPUTER_USE_API, + type StreamFn, +} from "../src/pi/index"; +import { type KernelBrowser, loop } from "../src/index"; import type Kernel from "@onkernel/sdk"; -import { Agent, AgentHarness, attach, InMemorySessionRepo, type KernelBrowser, type StreamFn } from "../src/index"; -import { installCuaBehaviors } from "../src/attach"; +import { installLoopBehaviors } from "../src/pi/attach"; const browser = { session_id: "browser_123", viewport: { width: 1440, height: 900 } } as KernelBrowser; const client = {} as Kernel; @@ -30,9 +36,9 @@ function assistant(model: Model): AssistantMessage { }; } -function recordingStream(seen: { model: Model; context: Context; options?: CuaSimpleStreamOptions }[]): StreamFn { +function recordingStream(seen: { model: Model; context: Context; options?: LoopSimpleStreamOptions }[]): StreamFn { return (model, context, options) => { - seen.push({ model, context: { ...context, tools: context.tools?.slice() }, options: options as CuaSimpleStreamOptions }); + seen.push({ model, context: { ...context, tools: context.tools?.slice() }, options: options as LoopSimpleStreamOptions }); const stream = createAssistantMessageEventStream(); const message = assistant(model); stream.push({ type: "start", partial: message }); @@ -47,7 +53,7 @@ describe("attach", () => { const handle = attach({ browser, client }); const compiled = handle.compile({ model: "openai:gpt-5.5", - tools: [cua.tools.browser.snapshot(), cua.tools.browser.click()], + tools: [loop.tools.browser.snapshot(), loop.tools.browser.click()], }); expect(compiled.model.api).toBe("openai-responses"); @@ -58,18 +64,18 @@ describe("attach", () => { it("derives the transport from the selected tools", () => { const handle = attach({ browser, client }); - const cdp = handle.compile({ model: "google:gemini-3.6-flash", tools: [cua.tools.browser.snapshot()] }); - const native = handle.compile({ model: "google:gemini-3.6-flash", tools: cua.providers.google.toolsets.browser() }); + const cdp = handle.compile({ model: "google:gemini-3.6-flash", tools: [loop.tools.browser.snapshot()] }); + const native = handle.compile({ model: "google:gemini-3.6-flash", tools: loop.providers.google.toolsets.browser() }); expect(cdp.model.api).toBe("google-generative-ai"); - expect(native.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + expect(native.model.api).toBe(GOOGLE_INTERACTIONS_API); }); it("materializes a spec once per handle, across compiles", () => { const handle = attach({ browser, client }); - const snapshot = cua.tools.browser.snapshot(); + const snapshot = loop.tools.browser.snapshot(); handle.compile({ model: "openai:gpt-5.5", tools: [snapshot] }); - handle.compile({ model: "openai:gpt-5.5", tools: [snapshot, cua.tools.browser.click()] }); + handle.compile({ model: "openai:gpt-5.5", tools: [snapshot, loop.tools.browser.click()] }); // The executable is cached per pool and per spec. Each compile wraps it to // install the execution scope, so the wrapper differs while the tool @@ -78,10 +84,10 @@ describe("attach", () => { expect(handle.resources.materialize(snapshot)).toBe(handle.resources.materialize(snapshot)); }); - it("drives a plain pi Agent with no CUA agent class", async () => { - const seen: { model: Model; context: Context; options?: CuaSimpleStreamOptions }[] = []; + it("drives a plain pi Agent with no Loop agent class", async () => { + const seen: { model: Model; context: Context; options?: LoopSimpleStreamOptions }[] = []; const handle = attach({ browser, client }); - const compiled = handle.compile({ model: "openai:gpt-5.5", tools: [cua.tools.browser.snapshot()] }); + const compiled = handle.compile({ model: "openai:gpt-5.5", tools: [loop.tools.browser.snapshot()] }); const agent = new Agent({ streamFn: recordingStream(seen), @@ -94,10 +100,10 @@ describe("attach", () => { expect(seen[0]!.context.tools?.map((tool) => tool.name)).toEqual(["browser_snapshot"]); }); - it("drives a plain pi AgentHarness, with CUA's behaviors installed", async () => { - const seen: { model: Model; context: Context; options?: CuaSimpleStreamOptions }[] = []; + it("drives a plain pi AgentHarness, with Loop's behaviors installed", async () => { + const seen: { model: Model; context: Context; options?: LoopSimpleStreamOptions }[] = []; const handle = attach({ browser, client, models: modelsFromStream(recordingStream(seen)) }); - const compiled = handle.compile({ model: "openai:gpt-5.5", tools: [cua.tools.browser.snapshot()] }); + const compiled = handle.compile({ model: "openai:gpt-5.5", tools: [loop.tools.browser.snapshot()] }); const session = await new InMemorySessionRepo().create(); const harness = new AgentHarness({ @@ -117,9 +123,9 @@ describe("attach", () => { }); it("streams the live catalog's tool plan after a swap, not the one the harness was built with", async () => { - const seen: { model: Model; context: Context; options?: CuaSimpleStreamOptions }[] = []; + const seen: { model: Model; context: Context; options?: LoopSimpleStreamOptions }[] = []; const handle = attach({ browser, client, models: modelsFromStream(recordingStream(seen)) }); - const first = handle.compile({ model: "openai:gpt-5.5", tools: [cua.tools.browser.snapshot()] }); + const first = handle.compile({ model: "openai:gpt-5.5", tools: [loop.tools.browser.snapshot()] }); const session = await new InMemorySessionRepo().create(); const harness = new AgentHarness({ @@ -131,7 +137,7 @@ describe("attach", () => { }); first.activate(harness); - const second = handle.compile({ model: "openai:gpt-5.5", tools: [cua.providers.openai.tools.computer()] }); + const second = handle.compile({ model: "openai:gpt-5.5", tools: [loop.providers.openai.tools.computer()] }); await second.apply(harness); await harness.prompt("go"); @@ -139,8 +145,8 @@ describe("attach", () => { // and tool plan it carries are per-catalog, so a per-compile collection // would keep sending the first catalog's plan for the rest of the session. expect(seen).toHaveLength(1); - expect(seen[0]!.options?.cuaIncomingToolPlan?.openaiComputerName).toBe("computer"); - expect(seen[0]!.model.api).toBe(OPENAI_CUA_COMPUTER_API); + expect(seen[0]!.options?.loopIncomingToolPlan?.openaiComputerName).toBe("computer"); + expect(seen[0]!.model.api).toBe(OPENAI_COMPUTER_USE_API); }); it("spends an empty-response retry only when the follow-up is queued", async () => { @@ -159,7 +165,7 @@ describe("attach", () => { }, }; const manager = { catalog: { entries: [] }, specFor: () => undefined }; - installCuaBehaviors(stub as never, manager as never, { followUp: "continue", maxAttempts: 1 }); + installLoopBehaviors(stub as never, manager as never, { followUp: "continue", maxAttempts: 1 }); const emptyTurn = { type: "turn_end", message: { role: "assistant", stopReason: "stop", content: [] } }; await expect(emit(emptyTurn)).rejects.toThrow("queue closed"); @@ -181,7 +187,7 @@ describe("attach", () => { }); function modelsFromStream(streamFn: StreamFn) { - const models = createCuaModels(); + const models = createLoopModels(); models.setProvider({ id: "openai", name: "scripted", diff --git a/packages/agent/test/browser-act-fail-fast.test.ts b/packages/loop/test/browser-act-fail-fast.test.ts similarity index 96% rename from packages/agent/test/browser-act-fail-fast.test.ts rename to packages/loop/test/browser-act-fail-fast.test.ts index bff55f8e..afe2b3f1 100644 --- a/packages/agent/test/browser-act-fail-fast.test.ts +++ b/packages/loop/test/browser-act-fail-fast.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { runBrowserAct, type BrowserActRuntime } from "../src/translator/browser-act"; +import { type BrowserActRuntime, runBrowserAct } from "../src/core/translator/browser-act"; // `boundary()` compares the observation's navigation epoch and per-frame // generations against what the runtime reports live, so a fixture has to agree diff --git a/packages/agent/test/browser-cross-process.live.test.ts b/packages/loop/test/browser-cross-process.live.test.ts similarity index 87% rename from packages/agent/test/browser-cross-process.live.test.ts rename to packages/loop/test/browser-cross-process.live.test.ts index 977737b7..2ba639ce 100644 --- a/packages/agent/test/browser-cross-process.live.test.ts +++ b/packages/loop/test/browser-cross-process.live.test.ts @@ -1,8 +1,8 @@ import Kernel from "@onkernel/sdk"; import { describe, expect, it } from "vitest"; -import type { CuaBrowserAction } from "@onkernel/cua-ai"; -import { BrowserExecutor } from "../src/translator/browser"; -import type { BatchReadResult } from "../src/translator/types"; +import type { BrowserAction } from "../src/index"; +import { BrowserExecutor } from "../src/core/translator/browser"; +import type { BatchReadResult } from "../src/core/translator/types"; /** * Live regression for cross-process named-session ref staleness (Defect 2). @@ -10,10 +10,10 @@ import type { BatchReadResult } from "../src/translator/types"; * Faithfully exercises the process boundary that unit tests can only fake: * each BrowserExecutor opens its own CDP connection to the same Kernel * browser, so exported ref state is reconciled against real `loaderId`s and - * real navigations. Gated on CUA_E2E_LIVE=1 + KERNEL_API_KEY so it never + * real navigations. Gated on LOOP_E2E_LIVE=1 + KERNEL_API_KEY so it never * provisions a browser in a normal test run. */ -const LIVE = process.env.CUA_E2E_LIVE === "1"; +const LIVE = process.env.LOOP_E2E_LIVE === "1"; const KERNEL_API_KEY = process.env.KERNEL_API_KEY; const PAGE_A = "https://example.com/"; @@ -46,7 +46,7 @@ describe.skipIf(!(LIVE && KERNEL_API_KEY))("BrowserExecutor cross-process docume // Process 1: land on page A, mint a ref, export ref state. const first = new BrowserExecutor(cdpWsUrl); await first.execute({ type: "browser_navigate", url: PAGE_A }); - const snapshot = textOf(await first.execute({ type: "browser_snapshot", filter: "interactive" } as CuaBrowserAction)); + const snapshot = textOf(await first.execute({ type: "browser_snapshot", filter: "interactive" } as BrowserAction)); const ref = /\[(e\d+)\]/.exec(snapshot)?.[1]; expect(ref, `expected a minted ref in:\n${snapshot}`).toBeTruthy(); const state = first.exportRefState(); @@ -56,7 +56,7 @@ describe.skipIf(!(LIVE && KERNEL_API_KEY))("BrowserExecutor cross-process docume // Process 2: same browser, document unchanged -> imported ref resolves. const second = new BrowserExecutor(cdpWsUrl); second.importRefState(state); - const scoped = textOf(await second.execute({ type: "browser_snapshot", ref } as CuaBrowserAction)); + const scoped = textOf(await second.execute({ type: "browser_snapshot", ref } as BrowserAction)); expect(scoped.length).toBeGreaterThan(0); second.close(); @@ -65,7 +65,7 @@ describe.skipIf(!(LIVE && KERNEL_API_KEY))("BrowserExecutor cross-process docume // resolving by process-local generation against a possibly-reused node id. const legacy = new BrowserExecutor(cdpWsUrl); legacy.importRefState({ ...state, documents: undefined }); - await expect(legacy.execute({ type: "browser_click", ref } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(legacy.execute({ type: "browser_click", ref } as BrowserAction)).rejects.toThrow(/stale/); legacy.close(); // A navigation changes the document after the ref was minted, standing @@ -79,7 +79,7 @@ describe.skipIf(!(LIVE && KERNEL_API_KEY))("BrowserExecutor cross-process docume // against a different document with possibly-reused backend node ids. const third = new BrowserExecutor(cdpWsUrl); third.importRefState(state); - await expect(third.execute({ type: "browser_click", ref } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(third.execute({ type: "browser_click", ref } as BrowserAction)).rejects.toThrow(/stale/); third.close(); }); }, diff --git a/packages/agent/test/browser-frame-collection.test.ts b/packages/loop/test/browser-frame-collection.test.ts similarity index 93% rename from packages/agent/test/browser-frame-collection.test.ts rename to packages/loop/test/browser-frame-collection.test.ts index 030d3c07..f11508a0 100644 --- a/packages/agent/test/browser-frame-collection.test.ts +++ b/packages/loop/test/browser-frame-collection.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; -import { isExpectedFrameCollectionError } from "../src/translator/browser-frame-collection"; -import { CdpProtocolError } from "../src/translator/cdp"; +import { isExpectedFrameCollectionError } from "../src/core/translator/browser-frame-collection"; +import { CdpProtocolError } from "../src/core/translator/cdp"; // The allow-list is the sole boundary between a transiently-inaccessible frame // (retried, then omitted) and a hard, loud FrameCollectionError. Pin every diff --git a/packages/agent/test/browser-ref-lifecycle.test.ts b/packages/loop/test/browser-ref-lifecycle.test.ts similarity index 98% rename from packages/agent/test/browser-ref-lifecycle.test.ts rename to packages/loop/test/browser-ref-lifecycle.test.ts index 08cdd681..8fceb9eb 100644 --- a/packages/agent/test/browser-ref-lifecycle.test.ts +++ b/packages/loop/test/browser-ref-lifecycle.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { type RefEntry, RefGenerationLifecycle } from "../src/translator/browser-ref-lifecycle"; +import { type RefEntry, RefGenerationLifecycle } from "../src/core/translator/browser-ref-lifecycle"; function frameRef(overrides: Partial = {}): RefEntry { return { diff --git a/packages/pi-extension/test/browser-runtime.test.ts b/packages/loop/test/browser-runtime.test.ts similarity index 75% rename from packages/pi-extension/test/browser-runtime.test.ts rename to packages/loop/test/browser-runtime.test.ts index 9e6316fb..9c0cef91 100644 --- a/packages/pi-extension/test/browser-runtime.test.ts +++ b/packages/loop/test/browser-runtime.test.ts @@ -6,14 +6,14 @@ vi.mock("@onkernel/sdk", () => ({ browsers = { create: state.create, retrieve: state.retrieve, deleteByID: state.deleteByID }; }, })); -vi.mock("@onkernel/cua-agent", () => ({ - CuaExecutionResources: class { +vi.mock("@onkernel/loop", () => ({ + LoopExecutionResources: class { dispose = state.dispose; constructor(_options: unknown) {} }, })); -import { CuaBrowserRuntime } from "../src/browser-runtime"; +import { LoopBrowserRuntime } from "../src/pi-extension/browser-runtime"; const owned = { session_id: "owned", created_at: "2026-01-01T00:00:00Z", browser_live_view_url: "https://live" }; @@ -24,10 +24,10 @@ beforeEach(() => { state.dispose.mockReset(); }); -describe("CuaBrowserRuntime", () => { +describe("LoopBrowserRuntime", () => { it("creates one shared owned browser for concurrent first calls and deletes it on close", async () => { state.create.mockResolvedValue(owned); - const runtime = new CuaBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }); + const runtime = new LoopBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }); const [first, second] = await Promise.all([runtime.get(), runtime.get()]); expect(first).toBe(second); expect(state.create).toHaveBeenCalledTimes(1); @@ -40,7 +40,7 @@ describe("CuaBrowserRuntime", () => { it("deletes an owned browser when resource disposal fails", async () => { state.create.mockResolvedValue(owned); state.dispose.mockRejectedValue(new Error("dispose failed")); - const runtime = new CuaBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }); + const runtime = new LoopBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }); await runtime.get(); await expect(runtime.close()).rejects.toThrow("dispose failed"); expect(state.deleteByID).toHaveBeenCalledWith("owned"); @@ -48,7 +48,7 @@ describe("CuaBrowserRuntime", () => { it("does not delete an attached browser", async () => { state.retrieve.mockResolvedValue({ ...owned, session_id: "attached" }); - const runtime = new CuaBrowserRuntime( + const runtime = new LoopBrowserRuntime( { sessionId: "attached", timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }, ); @@ -65,7 +65,7 @@ describe("CuaBrowserRuntime", () => { resolve = done; }), ); - const runtime = new CuaBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }); + const runtime = new LoopBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }); const pending = runtime.get().catch(() => undefined); const closing = runtime.close(); resolve(owned); @@ -77,8 +77,8 @@ describe("CuaBrowserRuntime", () => { const cancelled = new AbortController(); cancelled.abort(); await expect( - new CuaBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }).get(cancelled.signal), + new LoopBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, { KERNEL_API_KEY: "test" }).get(cancelled.signal), ).rejects.toThrow("cancelled"); - await expect(new CuaBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, {}).get()).rejects.toThrow("KERNEL_API_KEY"); + await expect(new LoopBrowserRuntime({ timeoutSeconds: 60, saveProfileChanges: false }, {}).get()).rejects.toThrow("KERNEL_API_KEY"); }); }); diff --git a/packages/agent/test/browser-wait.test.ts b/packages/loop/test/browser-wait.test.ts similarity index 96% rename from packages/agent/test/browser-wait.test.ts rename to packages/loop/test/browser-wait.test.ts index 61be42b6..2a90e447 100644 --- a/packages/agent/test/browser-wait.test.ts +++ b/packages/loop/test/browser-wait.test.ts @@ -1,7 +1,16 @@ import { describe, expect, it } from "vitest"; -import type { CuaBrowserExpectation } from "@onkernel/cua-ai"; -import { buildNthIndex, type AXNode, type BrowserObservation, type RenderContext } from "../src/translator/browser-observation"; -import { evaluateBrowserExpectation, waitForBrowserExpectation, type BrowserRefResolver } from "../src/translator/browser-wait"; +import type { BrowserExpectation } from "../src/index"; +import { + type AXNode, + type BrowserObservation, + buildNthIndex, + type RenderContext, +} from "../src/core/translator/browser-observation"; +import { + type BrowserRefResolver, + evaluateBrowserExpectation, + waitForBrowserExpectation, +} from "../src/core/translator/browser-wait"; const nodes = (names: string[]): AXNode[] => [ { nodeId: "root", role: { value: "RootWebArea" }, childIds: names.map((_, index) => `n${index}`) }, @@ -36,7 +45,7 @@ describe("evaluateBrowserExpectation", () => { [{ type: "url", contains: "a.test" }, true], [{ type: "title", equals: "A" }, true], [{ type: "url", changed: false }, true], - ] as Array<[CuaBrowserExpectation, boolean]>) ("evaluates %#", (condition, truth) => { + ] as Array<[BrowserExpectation, boolean]>) ("evaluates %#", (condition, truth) => { const current = observation(["Save"]); expect(evaluateBrowserExpectation(condition, current, current, missingRef).truth).toBe(truth); }); @@ -55,7 +64,7 @@ describe("evaluateBrowserExpectation", () => { [false, undefined, { all: [{ type: "text", text: "Save", exists: false }, { type: "text", text: "Missing" }] }, false], [true, undefined, { any: [{ type: "text", text: "Save" }, { type: "text", text: "Missing" }] }, true], [false, undefined, { any: [{ type: "text", text: "No" }, { type: "text", text: "Missing" }] }, undefined], - ] as Array<[boolean, undefined, CuaBrowserExpectation, boolean | undefined]>) ("combines three-valued condition %#", (_a, _b, condition, truth) => { + ] as Array<[boolean, undefined, BrowserExpectation, boolean | undefined]>) ("combines three-valued condition %#", (_a, _b, condition, truth) => { const current = observation(["Save"], false); expect(evaluateBrowserExpectation(condition, current, current, missingRef).truth).toBe(truth); }); @@ -221,7 +230,7 @@ describe("waitForBrowserExpectation", () => { it.each([ { any: [{ type: "url", contains: "/done" }, { type: "text", text: "Never" }] }, { all: [{ type: "url", contains: "/done" }, { type: "text", text: "Ready" }] }, - ] as CuaBrowserExpectation[])("allows mixed location condition %# to satisfy after navigation", async (expectation) => { + ] as BrowserExpectation[])("allows mixed location condition %# to satisfy after navigation", async (expectation) => { let time = 0, reads = 0; const result = await waitForBrowserExpectation({ selectTarget: async () => "page", diff --git a/packages/agent/test/cdp.test.ts b/packages/loop/test/cdp.test.ts similarity index 97% rename from packages/agent/test/cdp.test.ts rename to packages/loop/test/cdp.test.ts index 8bc1edff..4695b921 100644 --- a/packages/agent/test/cdp.test.ts +++ b/packages/loop/test/cdp.test.ts @@ -1,5 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { CdpConnection, CdpProtocolError } from "../src/translator/cdp"; +import { CdpConnection, CdpProtocolError } from "../src/core/translator/cdp"; class FakeSocket { static instances: FakeSocket[] = []; diff --git a/packages/agent/test/e2e.live.test.ts b/packages/loop/test/e2e.live.test.ts similarity index 95% rename from packages/agent/test/e2e.live.test.ts rename to packages/loop/test/e2e.live.test.ts index 2b9355e9..59db5b65 100644 --- a/packages/agent/test/e2e.live.test.ts +++ b/packages/loop/test/e2e.live.test.ts @@ -2,16 +2,16 @@ import Kernel from "@onkernel/sdk"; import { describe, expect, it } from "vitest"; import { Agent, - AgentHarness, - attach, - cua, - InMemorySessionRepo, type AgentEvent, + AgentHarness, type AgentHarnessEvent, type AgentMessage, -} from "../src/index"; + attach, + InMemorySessionRepo, +} from "../src/pi/index"; +import { loop } from "../src/index"; -const LIVE = process.env.CUA_E2E_LIVE === "1"; +const LIVE = process.env.LOOP_E2E_LIVE === "1"; const KERNEL_API_KEY = process.env.KERNEL_API_KEY; type ProviderCase = { @@ -49,7 +49,7 @@ const cases: ProviderCase[] = [ ].join("\n"), expectToolCalls: true, timeoutMs: 120_000, - ciOptInEnvVar: "CUA_E2E_OPENAI", + ciOptInEnvVar: "LOOP_E2E_OPENAI", }, { name: "anthropic", @@ -62,7 +62,7 @@ const cases: ProviderCase[] = [ ].join("\n"), expectToolCalls: true, timeoutMs: 120_000, - ciOptInEnvVar: "CUA_E2E_ANTHROPIC", + ciOptInEnvVar: "LOOP_E2E_ANTHROPIC", }, { name: "gemini", @@ -75,7 +75,7 @@ const cases: ProviderCase[] = [ ].join("\n"), expectToolCalls: true, timeoutMs: 300_000, - ciOptInEnvVar: "CUA_E2E_GEMINI", + ciOptInEnvVar: "LOOP_E2E_GEMINI", }, { name: "xai", @@ -139,7 +139,7 @@ function shouldRunSwitchCase(c: ModelSwitchCase): boolean { } function structuredBrowserTools() { - return [...cua.toolsets.browser(), cua.tools.browser.act()]; + return [...loop.toolsets.browser(), loop.tools.browser.act()]; } function toolsForCase(c: ProviderCase) { @@ -149,11 +149,11 @@ function toolsForCase(c: ProviderCase) { return structuredBrowserTools(); case "moonshotai": // Moonshot's API rejects `browser_act`'s schema; Kimi runs primitives only. - return cua.toolsets.browser(); + return loop.toolsets.browser(); case "anthropic": - return [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]; + return [loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]; case "gemini": - return cua.providers.google.toolsets.browser(); + return loop.providers.google.toolsets.browser(); } } @@ -169,7 +169,7 @@ const modelSwitchPrompt = [ ].join("\n"); function modelSwitchTools() { - return [cua.tools.computer.screenshot({ name: "screenshot" })]; + return [loop.tools.computer.screenshot({ name: "screenshot" })]; } function createRunStats(): RunStats { @@ -255,7 +255,7 @@ function toolErrorMessage(result: unknown): string | undefined { return text || undefined; } -describe("Cua live e2e", () => { +describe("Loop live e2e", () => { for (const c of cases) { const test = shouldRunCase(c) ? it : it.skip; diff --git a/packages/agent/test/example-provider-matrix.test.ts b/packages/loop/test/example-provider-matrix.test.ts similarity index 87% rename from packages/agent/test/example-provider-matrix.test.ts rename to packages/loop/test/example-provider-matrix.test.ts index 940b9825..8ae5219a 100644 --- a/packages/agent/test/example-provider-matrix.test.ts +++ b/packages/loop/test/example-provider-matrix.test.ts @@ -1,7 +1,5 @@ -import { - compileCuaToolCatalog, - type CuaModelRef, -} from "@onkernel/cua-ai"; +import type { LoopModelRef } from "../src/pi/index"; +import { compileLoopToolCatalog } from "../src/index"; import { describe, expect, it } from "vitest"; import { toolsForModel } from "../examples/shared/tools"; @@ -13,7 +11,7 @@ import { toolsForModel } from "../examples/shared/tools"; * Limited to models the registry can resolve, so Anthropic's older non-native * fallback branch is covered by the tool menu's availability tests instead. */ -const models: readonly CuaModelRef[] = [ +const models: readonly LoopModelRef[] = [ "openai:gpt-5.6-sol", "anthropic:claude-opus-5", "anthropic:claude-sonnet-5", @@ -28,7 +26,7 @@ describe("example provider matrix tool policy", () => { it("compiles a valid catalog for every model the matrices advertise", () => { for (const model of models) { expect( - () => compileCuaToolCatalog({ model, requestedTools: toolsForModel(model) }), + () => compileLoopToolCatalog({ model, requestedTools: toolsForModel(model) }), model, ).not.toThrow(); } diff --git a/packages/pi-extension/test/extension.test.ts b/packages/loop/test/extension.test.ts similarity index 94% rename from packages/pi-extension/test/extension.test.ts rename to packages/loop/test/extension.test.ts index 22e827ba..c8636045 100644 --- a/packages/pi-extension/test/extension.test.ts +++ b/packages/loop/test/extension.test.ts @@ -1,12 +1,12 @@ import { fileURLToPath } from "node:url"; import type { Api, Model, Provider } from "@earendil-works/pi-ai"; import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; -import { getCuaModel } from "@onkernel/cua-ai"; +import { getLoopModel } from "../src/pi/index"; import { describe, expect, it, vi } from "vitest"; -import { CuaBrowserRuntime } from "../src/browser-runtime"; -import { allSelectableSpecs, expandSelection, parseSelection } from "../src/selection"; -import extension from "../src/index"; +import { LoopBrowserRuntime } from "../src/pi-extension/browser-runtime"; +import { allSelectableSpecs, expandSelection, parseSelection } from "../src/pi-extension/selection"; +import extension from "../src/pi-extension/index"; type Handler = (event: unknown, ctx: ExtensionContext) => unknown; @@ -31,7 +31,7 @@ interface FakePi { readonly active: string[]; } -const extensionPath = fileURLToPath(new URL("../src/index.ts", import.meta.url)); +const extensionPath = fileURLToPath(new URL("../src/pi-extension/index.ts", import.meta.url)); function makePi(flags: Record): FakePi { const handlers = new Map(); @@ -92,7 +92,7 @@ const ctx = { sessionManager: { getBranch: () => [] }, ui: { setStatus() {}, notify() {} }, } as unknown as ExtensionContext; -const anthropicCtx = { ...ctx, model: getCuaModel("anthropic:claude-fable-5") } as ExtensionContext; +const anthropicCtx = { ...ctx, model: getLoopModel("anthropic:claude-fable-5") } as ExtensionContext; describe("pi extension activation", () => { it("reads parsed flags at session_start, installs selectable batch tools, and preserves unrelated tools", async () => { @@ -115,7 +115,7 @@ describe("pi extension activation", () => { expect(() => getHandler(pi, "session_start")({}, ctx)).toThrow('unknown browser tool selector "nope"'); }); - it("registers the CUA Anthropic provider and serializes native computer use", async () => { + it("registers the Loop Anthropic provider and serializes native computer use", async () => { const pi = makePi({ "browser-tools": "anthropic-computer", "browser-coordinates": "pixels", @@ -135,7 +135,7 @@ describe("pi extension activation", () => { }); it("keeps the browser out of the request path, and blocks execution after shutdown", async () => { - const get = vi.spyOn(CuaBrowserRuntime.prototype, "get"); + const get = vi.spyOn(LoopBrowserRuntime.prototype, "get"); try { const pi = makePi({ "browser-tools": "anthropic-computer", @@ -161,7 +161,7 @@ describe("pi extension activation", () => { } }); - it("applies provider transforms only for the active CUA subset", async () => { + it("applies provider transforms only for the active Loop subset", async () => { const pi = makePi({ "browser-tools": "playwright", "browser-coordinates": "pixels", @@ -193,7 +193,7 @@ describe("pi extension activation", () => { expect(pi.entries).toEqual([ { type: "custom", - customType: "cua-pi-config-v1", + customType: "loop-pi-config-v1", data: expect.objectContaining({ origin: "command", selectors: ["computer"] }), }, ]); @@ -218,7 +218,7 @@ describe("pi extension activation", () => { getBranch: () => [ { type: "custom", - customType: "cua-pi-config-v1", + customType: "loop-pi-config-v1", data: { version: 1, selectors: ["computer"], coordinates: "normalized-1000" }, }, ], @@ -230,7 +230,7 @@ describe("pi extension activation", () => { expect(legacy.active).not.toContain("computer_click"); }); - it("removes stale incompatible CUA schemas from the provider payload", async () => { + it("removes stale incompatible Loop schemas from the provider payload", async () => { // A provider-native surface is the incompatibility that survives the model // allowlist's removal: an unknown provider now compiles fine, but Anthropic's // native computer still cannot reach an OpenAI model. @@ -295,7 +295,7 @@ describe("pi extension activation", () => { getBranch: () => [ { type: "custom", - customType: "cua-pi-config-v1", + customType: "loop-pi-config-v1", data: { version: 1, origin: "command", selectors: ["browser-batch", "computer"], coordinates: "pixels" }, }, ], @@ -329,7 +329,7 @@ describe("pi extension activation", () => { getBranch: () => [ { type: "custom", - customType: "cua-pi-config-v1", + customType: "loop-pi-config-v1", data: { version: 1, origin: "command", selectors: ["browser-batch"], coordinates: "pixels" }, }, ], @@ -384,7 +384,7 @@ describe("pi extension activation", () => { // An OpenAI model so a native selector it cannot take shows up unavailable. const listingCtx = { ...ctx, - model: getCuaModel("openai:gpt-5.6-sol"), + model: getLoopModel("openai:gpt-5.6-sol"), ui: { setStatus() {}, notify: (text: string) => notices.push(text) }, } as unknown as ExtensionContext; extension(pi.api); diff --git a/packages/ai/test/google-provider.test.ts b/packages/loop/test/google-provider.test.ts similarity index 91% rename from packages/ai/test/google-provider.test.ts rename to packages/loop/test/google-provider.test.ts index b907c7f8..4fd46775 100644 --- a/packages/ai/test/google-provider.test.ts +++ b/packages/loop/test/google-provider.test.ts @@ -1,11 +1,11 @@ import type { AssistantMessage, Model, ToolCall } from "@earendil-works/pi-ai"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { getCuaModel } from "../src/index"; -import * as google from "../src/providers/google/provider"; +import { getLoopModel } from "../src/pi/index"; +import * as google from "../src/pi/providers/google/provider"; -// This adapter is exercised directly (not through compileCuaToolCatalog's +// This adapter is exercised directly (not through compileLoopToolCatalog's // derivation), so the model must carry the Interactions api itself. -const model: Model = { ...getCuaModel("google:gemini-3.6-flash"), api: google.GOOGLE_CUA_INTERACTIONS_API }; +const model: Model = { ...getLoopModel("google:gemini-3.6-flash"), api: google.GOOGLE_INTERACTIONS_API }; const incoming = { googleNames: { click: "click" }, googleExcludedNames: ["take_screenshot"], @@ -35,7 +35,7 @@ describe("Google Interactions computer-use adapter", () => { systemPrompt: "Use the browser.", messages: [{ role: "user", content: "click search", timestamp: 1 }], tools: [clickTool], - }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); expect(fetch.mock.calls[0]?.[0]).toBe("https://generativelanguage.googleapis.com/v1beta/interactions"); const call = message.content.find((content): content is ToolCall => content.type === "toolCall"); @@ -63,7 +63,7 @@ describe("Google Interactions computer-use adapter", () => { const message = await google.streamGoogleInteractions(model, { messages: [{ role: "user", content: "take a screenshot", timestamp: 1 }], tools: [screenshotTool], - }, { apiKey: "test", cuaIncomingToolPlan: screenshotIncoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: screenshotIncoming }).result(); expect(message.stopReason).toBe("error"); expect(message.errorMessage).toBe( @@ -88,7 +88,7 @@ describe("Google Interactions computer-use adapter", () => { const message = await google.streamGoogleInteractions(model, { messages: [{ role: "user", content: "take screenshots", timestamp: 1 }], tools: [screenshotTool], - }, { apiKey: "test", cuaIncomingToolPlan: screenshotIncoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: screenshotIncoming }).result(); expect(message.content.filter((content): content is ToolCall => content.type === "toolCall").map((call) => call.name)).toEqual([ "take_screenshot", @@ -111,7 +111,7 @@ describe("Google Interactions computer-use adapter", () => { const message = await google.streamGoogleInteractions(model, { messages: [{ role: "user", content: "click", timestamp: 1 }], tools: [clickTool], - }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); expect(message.stopReason).toBe("error"); expect(message.errorMessage).toBe( @@ -133,7 +133,7 @@ describe("Google Interactions computer-use adapter", () => { const message = await google.streamGoogleInteractions(model, { messages: [{ role: "user", content: "take screenshots", timestamp: 1 }], tools: [screenshotTool], - }, { apiKey: "test", cuaIncomingToolPlan: screenshotIncoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: screenshotIncoming }).result(); expect(message.content.filter((content): content is ToolCall => content.type === "toolCall").map((call) => call.name)).toEqual([ "take_screenshot", @@ -155,7 +155,7 @@ describe("Google Interactions computer-use adapter", () => { const message = await google.streamGoogleInteractions(model, { messages: [{ role: "user", content: "call my function", timestamp: 1 }], tools: [ordinaryScreenshot, screenshotTool], - }, { apiKey: "test", cuaIncomingToolPlan: screenshotIncoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: screenshotIncoming }).result(); expect(message.content.filter((content): content is ToolCall => content.type === "toolCall")).toEqual([ expect.objectContaining({ name: "screenshot", arguments: { full: true } }), @@ -197,7 +197,7 @@ describe("Google Interactions computer-use adapter", () => { }, ], tools: [clickTool], - }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); expect(message.content).toContainEqual({ type: "text", text: "Done" }); const request = JSON.parse(String(fetch.mock.calls[0]?.[1]?.body)) as Record; diff --git a/packages/agent/test/harness-context.test.ts b/packages/loop/test/harness-context.test.ts similarity index 91% rename from packages/agent/test/harness-context.test.ts rename to packages/loop/test/harness-context.test.ts index 89e3f062..01881604 100644 --- a/packages/agent/test/harness-context.test.ts +++ b/packages/loop/test/harness-context.test.ts @@ -2,27 +2,23 @@ import { mkdtempSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import { type AssistantMessage, createAssistantMessageEventStream, type Model } from "@earendil-works/pi-ai"; import { - createAssistantMessageEventStream, - createCuaModels, - type AssistantMessage, - type Model, -} from "@onkernel/cua-ai"; -import type Kernel from "@onkernel/sdk"; -import { + AgentHarness, + type AgentHarnessTool, + attach, createBashTool, createEditTool, + createLoopModels, createReadTool, createWriteTool, - AgentHarness, - attach, + type ExecutionToolContext, InMemorySessionRepo, NodeExecutionEnv, - type AgentHarnessTool, - type ExecutionToolContext, - type KernelBrowser, type StreamFn, -} from "../src/index"; +} from "../src/pi/index"; +import type Kernel from "@onkernel/sdk"; +import type { KernelBrowser } from "../src/index"; const browser = { session_id: "browser_123", viewport: { width: 1440, height: 900 } } as KernelBrowser; const client = {} as Kernel; @@ -60,7 +56,7 @@ function scriptedStream(turns: Array<(model: Model) => AssistantMessage> } function modelsFromStream(streamFn: StreamFn, provider = "openai") { - const models = createCuaModels(); + const models = createLoopModels(); models.setProvider({ id: provider, name: "scripted", @@ -98,7 +94,7 @@ describe("harness tool context", () => { env: ExecutionToolContext["env"]; requestId: string; } - const cwd = mkdtempSync(join(tmpdir(), "cua-harness-context-")); + const cwd = mkdtempSync(join(tmpdir(), "loop-harness-context-")); const toolContext: CustomContext = { env: new NodeExecutionEnv({ cwd }), requestId: "req-1" }; const received: CustomContext[] = []; const custom: AgentHarnessTool = { @@ -127,10 +123,10 @@ describe("harness tool context", () => { }); it("runs pi's native read/write/edit/bash tools against the context's execution env", async () => { - const cwd = mkdtempSync(join(tmpdir(), "cua-harness-tools-")); + const cwd = mkdtempSync(join(tmpdir(), "loop-harness-tools-")); const harness = await openHarness({ models: modelsFromStream(scriptedStream([ - (model) => assistant(model, [{ type: "toolCall", id: "write-1", name: "write", arguments: { path: "notes.txt", content: "hello cua\n" } }], "toolUse"), + (model) => assistant(model, [{ type: "toolCall", id: "write-1", name: "write", arguments: { path: "notes.txt", content: "hello loop\n" } }], "toolUse"), (model) => assistant(model, [{ type: "toolCall", id: "edit-1", name: "edit", arguments: { path: "notes.txt", edits: [{ oldText: "hello", newText: "goodbye" }] } }], "toolUse"), (model) => assistant(model, [{ type: "toolCall", id: "read-1", name: "read", arguments: { path: "notes.txt" } }], "toolUse"), (model) => assistant(model, [{ type: "toolCall", id: "bash-1", name: "bash", arguments: { command: "cat notes.txt" } }], "toolUse"), @@ -147,10 +143,10 @@ describe("harness tool context", () => { await harness.prompt("write, edit, read, and cat a file"); - expect(readFileSync(join(cwd, "notes.txt"), "utf8")).toBe("goodbye cua\n"); + expect(readFileSync(join(cwd, "notes.txt"), "utf8")).toBe("goodbye loop\n"); expect(results.get("write")?.isError).toBe(false); expect(results.get("edit")?.isError).toBe(false); - expect(results.get("read")?.content.some((block) => block.text?.includes("goodbye cua"))).toBe(true); - expect(results.get("bash")?.content.some((block) => block.text?.includes("goodbye cua"))).toBe(true); + expect(results.get("read")?.content.some((block) => block.text?.includes("goodbye loop"))).toBe(true); + expect(results.get("bash")?.content.some((block) => block.text?.includes("goodbye loop"))).toBe(true); }); }); diff --git a/packages/agent/test/keys.test.ts b/packages/loop/test/keys.test.ts similarity index 96% rename from packages/agent/test/keys.test.ts rename to packages/loop/test/keys.test.ts index fdc4db46..682eaa06 100644 --- a/packages/agent/test/keys.test.ts +++ b/packages/loop/test/keys.test.ts @@ -3,7 +3,7 @@ import { normalizeKernelKey, normalizeKernelKeyCombo, normalizeKernelKeySequence, -} from "../src/translator/keys"; +} from "../src/core/translator/keys"; describe("Kernel key normalization", () => { it("normalizes common provider key names to X11 keysyms", () => { diff --git a/packages/ai/test/menu.test.ts b/packages/loop/test/menu.test.ts similarity index 71% rename from packages/ai/test/menu.test.ts rename to packages/loop/test/menu.test.ts index 1115a418..5038a9ad 100644 --- a/packages/ai/test/menu.test.ts +++ b/packages/loop/test/menu.test.ts @@ -1,10 +1,11 @@ import { describe, expect, it } from "vitest"; -import { compileCuaToolCatalog, cuaToolMenu, getCuaModel, type CuaModelRef } from "../src/index"; +import { getLoopModel, type LoopModelRef } from "../src/pi/index"; +import { compileLoopToolCatalog, loopToolMenu } from "../src/index"; // One with a native browser surface, one with a native computer surface, one // with neither, one carrying a quirk, and one synthesized from an id pi-ai's // registry does not carry. -const MODELS: CuaModelRef[] = [ +const MODELS: LoopModelRef[] = [ "google:gemini-3.6-flash", "openai:gpt-5.5", "anthropic:claude-opus-5", @@ -12,14 +13,14 @@ const MODELS: CuaModelRef[] = [ "xai:grok-4.6", ]; -describe("cuaToolMenu", () => { +describe("loopToolMenu", () => { it("marks an entry available exactly when selecting it compiles", () => { for (const ref of MODELS) { - const model = getCuaModel(ref); - for (const entry of cuaToolMenu(ref)) { + const model = getLoopModel(ref); + for (const entry of loopToolMenu(ref)) { let compiles = true; try { - compileCuaToolCatalog({ model, requestedTools: entry.tools }); + compileLoopToolCatalog({ model, requestedTools: entry.tools }); } catch { compiles = false; } @@ -29,22 +30,22 @@ describe("cuaToolMenu", () => { }); it("quotes the compiler's own message as the reason", () => { - const act = cuaToolMenu("moonshotai:kimi-k3").find((entry) => entry.label === "browser_act"); + const act = loopToolMenu("moonshotai:kimi-k3").find((entry) => entry.label === "browser_act"); expect(act?.available).toBe(false); expect(act?.unavailableReason).toContain("does not accept the schema size"); // Google used to fail here. It no longer does: the two JSON Schema keywords // the Gemini API rejects are narrowed by the catalog's payload transform, so // browser_wait_for compiles like any other function tool. - const waitFor = cuaToolMenu("google:gemini-3.6-flash").find((entry) => entry.label === "browser_wait_for"); + const waitFor = loopToolMenu("google:gemini-3.6-flash").find((entry) => entry.label === "browser_wait_for"); expect(waitFor?.available).toBe(true); }); it("offers a model's own native surfaces and no other provider's", () => { - const nativeFor = (ref: CuaModelRef) => - cuaToolMenu(ref).filter((entry) => entry.group === "native" && entry.available).map((entry) => entry.label); + const nativeFor = (ref: LoopModelRef) => + loopToolMenu(ref).filter((entry) => entry.group === "native" && entry.available).map((entry) => entry.label); - // Anthropic's surfaces are version-gated outside CUA_NATIVE_SURFACES, so a + // Anthropic's surfaces are version-gated outside COMPUTER_USE_NATIVE_SURFACES, so a // menu reading that table directly would report these unavailable. expect(nativeFor("anthropic:claude-opus-5")).toEqual(["computer", "browser"]); expect(nativeFor("openai:gpt-5.5")).toEqual(["computer"]); @@ -53,14 +54,14 @@ describe("cuaToolMenu", () => { }); it("re-evaluates against the current selection, because some rules are pairwise", () => { - const google = cuaToolMenu("google:gemini-3.6-flash"); + const google = loopToolMenu("google:gemini-3.6-flash"); const nativeBrowser = google.find((entry) => entry.key === "group:google.native.browser")!; expect(nativeBrowser.available).toBe(true); expect(nativeBrowser.selected).toBe(false); - // Selecting Google's native set pins the transport, so a CUA browser tool + // Selecting Google's native set pins the transport, so a Loop browser tool // that compiles on its own no longer does beside it. - const withNative = cuaToolMenu("google:gemini-3.6-flash", nativeBrowser.tools); + const withNative = loopToolMenu("google:gemini-3.6-flash", nativeBrowser.tools); expect(withNative.find((entry) => entry.key === "group:google.native.browser")?.selected).toBe(true); const snapshot = withNative.find((entry) => entry.label === "browser_snapshot")!; const snapshotAlone = google.find((entry) => entry.label === "browser_snapshot")!; @@ -69,15 +70,15 @@ describe("cuaToolMenu", () => { }); it("reports what is already selected", () => { - const menu = cuaToolMenu("openai:gpt-5.5"); + const menu = loopToolMenu("openai:gpt-5.5"); const snapshot = menu.find((entry) => entry.label === "browser_snapshot")!; - const withSnapshot = cuaToolMenu("openai:gpt-5.5", snapshot.tools); + const withSnapshot = loopToolMenu("openai:gpt-5.5", snapshot.tools); expect(withSnapshot.find((entry) => entry.label === "browser_snapshot")?.selected).toBe(true); expect(withSnapshot.filter((entry) => entry.selected)).toHaveLength(1); }); it("covers the whole offerable surface, grouped", () => { - const menu = cuaToolMenu("openai:gpt-5.5"); + const menu = loopToolMenu("openai:gpt-5.5"); const groups = new Set(menu.map((entry) => entry.group)); expect(groups).toEqual(new Set(["browser", "computer", "playwright", "native"])); expect(menu.some((entry) => entry.label === "playwright_execute")).toBe(true); diff --git a/packages/ai/test/models.test.ts b/packages/loop/test/models.test.ts similarity index 54% rename from packages/ai/test/models.test.ts rename to packages/loop/test/models.test.ts index 515b7a19..8f530b55 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/loop/test/models.test.ts @@ -1,47 +1,47 @@ import { describe, expect, it } from "vitest"; import { - CUA_MODEL_QUIRKS, - CUA_NATIVE_SURFACES, - type CuaModelRef, - cuaModelCapabilities, - cuaNativeSurfaces, - cuaProviders, - formatCuaModelRef, - getCuaModel, - listCuaModels, - parseCuaModelRef, -} from "../src/index"; - -describe("CUA model refs", () => { + COMPUTER_USE_NATIVE_SURFACES, + computerUseNativeSurfaces, + formatLoopModelRef, + getLoopModel, + listLoopModels, + LOOP_MODEL_QUIRKS, + loopModelCapabilities, + type LoopModelRef, + loopProviders, + parseLoopModelRef, +} from "../src/pi/index"; + +describe("Loop model refs", () => { it("parses and formats provider-qualified refs", () => { - expect(parseCuaModelRef("openai:gpt-5.5")).toEqual({ provider: "openai", model: "gpt-5.5" }); - expect(formatCuaModelRef("openrouter", "meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); + expect(parseLoopModelRef("openai:gpt-5.5")).toEqual({ provider: "openai", model: "gpt-5.5" }); + expect(formatLoopModelRef("openrouter", "meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); }); it("rejects unqualified refs and unknown providers, but not unknown models", () => { - expect(() => getCuaModel("gpt-5.5" as never)).toThrow(/provider-qualified/); - expect(() => getCuaModel("bogus:model" as never)).toThrow(/unknown provider/); + expect(() => getLoopModel("gpt-5.5" as never)).toThrow(/provider-qualified/); + expect(() => getLoopModel("bogus:model" as never)).toThrow(/unknown provider/); // A model id pi-ai's registry has not caught up with still resolves: the // provider decides whether it exists, not a table in this package. - expect(getCuaModel("openai:gpt-3.5" as never).id).toBe("gpt-3.5"); + expect(getLoopModel("openai:gpt-3.5" as never).id).toBe("gpt-3.5"); }); it("names pi-ai's providers in the unknown-provider error", () => { - expect(() => parseCuaModelRef("bogus:model")).toThrow(/unknown provider "bogus" \(pi-ai carries: /); + expect(() => parseLoopModelRef("bogus:model")).toThrow(/unknown provider "bogus" \(pi-ai carries: /); }); it("accepts gemini: as an alias for google:", () => { - expect(parseCuaModelRef("gemini:gemini-3.6-flash")).toEqual({ + expect(parseLoopModelRef("gemini:gemini-3.6-flash")).toEqual({ provider: "google", model: "gemini-3.6-flash", }); - const model = getCuaModel("gemini:gemini-3.6-flash" as never); + const model = getLoopModel("gemini:gemini-3.6-flash" as never); expect(model.provider).toBe("google"); expect(model.id).toBe("gemini-3.6-flash"); }); it("lists curated model refs without a default", () => { - const models = listCuaModels(); + const models = listLoopModels(); expect(models.some((model) => model.ref === "openai:gpt-5.6-sol")).toBe(true); expect(models.some((model) => model.ref === "openai:gpt-5.5")).toBe(true); expect(models.some((model) => model.ref === "anthropic:claude-opus-5")).toBe(true); @@ -51,7 +51,7 @@ describe("CUA model refs", () => { }); it("returns pi-ai registry entries verbatim", () => { - const opus = getCuaModel("anthropic:claude-opus-5"); + const opus = getLoopModel("anthropic:claude-opus-5"); expect(opus).toMatchObject({ provider: "anthropic", api: "anthropic-messages", @@ -62,19 +62,19 @@ describe("CUA model refs", () => { }); expect(opus.compat).toMatchObject({ forceAdaptiveThinking: true, supportsTemperature: false }); - expect(getCuaModel("google:gemini-3.6-flash")).toMatchObject({ + expect(getLoopModel("google:gemini-3.6-flash")).toMatchObject({ provider: "google", api: "google-generative-ai", contextWindow: 1_048_576, }); - const muse = getCuaModel("openrouter:meta/muse-spark-1.1"); + const muse = getLoopModel("openrouter:meta/muse-spark-1.1"); expect(muse.provider).toBe("openrouter"); expect(muse.baseUrl).toBe("https://openrouter.ai/api/v1"); }); it("returns pi-ai's Grok catalog entry", () => { - const grok = getCuaModel("xai:grok-4.5"); + const grok = getLoopModel("xai:grok-4.5"); expect(grok.provider).toBe("xai"); expect(grok.api).toBe("openai-responses"); expect(grok.baseUrl).toBe("https://api.x.ai/v1"); @@ -83,15 +83,15 @@ describe("CUA model refs", () => { }); it("uses pi-ai's Kimi catalog entries for both transports", () => { - const direct = getCuaModel("moonshotai:kimi-k3"); - const routed = getCuaModel("openrouter:moonshotai/kimi-k3"); + const direct = getLoopModel("moonshotai:kimi-k3"); + const routed = getLoopModel("openrouter:moonshotai/kimi-k3"); expect(direct).toMatchObject({ provider: "moonshotai", id: "kimi-k3", api: "openai-completions" }); expect(routed).toMatchObject({ provider: "openrouter", id: "moonshotai/kimi-k3", api: "openai-completions" }); - expect(listCuaModels("openrouter").map((model) => model.ref)).toContain("openrouter:moonshotai/kimi-k3"); + expect(listLoopModels("openrouter").map((model) => model.ref)).toContain("openrouter:moonshotai/kimi-k3"); }); it("uses pi-ai's Kimi catalog entry", () => { - const kimi = getCuaModel("moonshotai:kimi-k3"); + const kimi = getLoopModel("moonshotai:kimi-k3"); expect(kimi.provider).toBe("moonshotai"); expect(kimi.api).toBe("openai-completions"); expect(kimi.baseUrl).toBe("https://api.moonshot.ai/v1"); @@ -106,78 +106,78 @@ describe("CUA model refs", () => { }); it("accepts the moonshot: alias for moonshotai refs", () => { - expect(parseCuaModelRef("moonshot:kimi-k3")).toEqual({ provider: "moonshotai", model: "kimi-k3" }); - expect(getCuaModel("moonshot:kimi-k3" as CuaModelRef).id).toBe("kimi-k3"); + expect(parseLoopModelRef("moonshot:kimi-k3")).toEqual({ provider: "moonshotai", model: "kimi-k3" }); + expect(getLoopModel("moonshot:kimi-k3" as LoopModelRef).id).toBe("kimi-k3"); }); it("resolves every model to its ordinary registry transport, independent of tool selection", () => { - // getCuaModel() never derives a tool-driven transport: OPENAI_CUA_COMPUTER_API - // and GOOGLE_CUA_INTERACTIONS_API are only ever carried by a model that - // compileCuaToolCatalog compiled with the matching native tool selected + // getLoopModel() never derives a tool-driven transport: OPENAI_COMPUTER_USE_API + // and GOOGLE_INTERACTIONS_API are only ever carried by a model that + // compileLoopToolCatalog compiled with the matching native tool selected // (see tool-catalog.test.ts's transport derivation coverage). - expect(getCuaModel("openai:gpt-5.6-sol").api).toBe("openai-responses"); - expect(getCuaModel("openai:gpt-5.5").api).toBe("openai-responses"); - expect(getCuaModel("openai:gpt-5.4-mini").api).toBe("openai-responses"); - expect(getCuaModel("google:gemini-3.6-flash").api).toBe("google-generative-ai"); - expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); + expect(getLoopModel("openai:gpt-5.6-sol").api).toBe("openai-responses"); + expect(getLoopModel("openai:gpt-5.5").api).toBe("openai-responses"); + expect(getLoopModel("openai:gpt-5.4-mini").api).toBe("openai-responses"); + expect(getLoopModel("google:gemini-3.6-flash").api).toBe("google-generative-ai"); + expect(getLoopModel("xai:grok-4.5").api).toBe("openai-responses"); }); it("synthesizes models pi-ai's registry does not carry", () => { // pi-ai's registry (generated from models.dev) carries family roots, not // dated snapshots, and lags a provider's newest ids. Both still resolve, // inheriting the transport and base URL from the provider's other models. - const snapshot = getCuaModel("openai:gpt-5.5-2026-04-23"); + const snapshot = getLoopModel("openai:gpt-5.5-2026-04-23"); expect(snapshot.id).toBe("gpt-5.5-2026-04-23"); expect(snapshot.provider).toBe("openai"); - expect(snapshot.api).toBe(getCuaModel("openai:gpt-5.5").api); - expect(snapshot.baseUrl).toBe(getCuaModel("openai:gpt-5.5").baseUrl); + expect(snapshot.api).toBe(getLoopModel("openai:gpt-5.5").api); + expect(snapshot.baseUrl).toBe(getLoopModel("openai:gpt-5.5").baseUrl); // The motivating case: a model the provider has shipped and models.dev // has not picked up yet. - expect(getCuaModel("xai:grok-4.6").id).toBe("grok-4.6"); + expect(getLoopModel("xai:grok-4.6").id).toBe("grok-4.6"); }); it("synthesizes from the nearest, newest sibling", () => { // xAI carries grok-4.3 on chat completions and grok-4.5 on Responses, so // picking the wrong sibling would send a new Grok to the wrong transport. - expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); - expect(getCuaModel("xai:grok-4.6").api).toBe("openai-responses"); - expect(getCuaModel("xai:grok-4.6").baseUrl).toBe(getCuaModel("xai:grok-4.5").baseUrl); - expect(getCuaModel("anthropic:claude-opus-6").api).toBe("anthropic-messages"); + expect(getLoopModel("xai:grok-4.5").api).toBe("openai-responses"); + expect(getLoopModel("xai:grok-4.6").api).toBe("openai-responses"); + expect(getLoopModel("xai:grok-4.6").baseUrl).toBe(getLoopModel("xai:grok-4.5").baseUrl); + expect(getLoopModel("anthropic:claude-opus-6").api).toBe("anthropic-messages"); }); }); describe("native surfaces", () => { it("cites first-party documentation for every entry", () => { - for (const entry of CUA_NATIVE_SURFACES) { + for (const entry of COMPUTER_USE_NATIVE_SURFACES) { expect(entry.source).toMatch(/^https?:\/\//); expect(entry.surfaces.length).toBeGreaterThan(0); } }); it("matches family roots, dated snapshots, and numeric revisions", () => { - expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.5"))).toEqual(["computer"]); - expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.5-2026-04-23"))).toEqual(["computer"]); - expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-mini"))).toEqual(["computer"]); - expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-opus-5"))).toEqual(["computer", "browser"]); - expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-opus-5-20260724"))).toEqual(["computer", "browser"]); + expect(computerUseNativeSurfaces(getLoopModel("openai:gpt-5.5"))).toEqual(["computer"]); + expect(computerUseNativeSurfaces(getLoopModel("openai:gpt-5.5-2026-04-23"))).toEqual(["computer"]); + expect(computerUseNativeSurfaces(getLoopModel("openai:gpt-5.4-mini"))).toEqual(["computer"]); + expect(computerUseNativeSurfaces(getLoopModel("anthropic:claude-opus-5"))).toEqual(["computer", "browser"]); + expect(computerUseNativeSurfaces(getLoopModel("anthropic:claude-opus-5-20260724"))).toEqual(["computer", "browser"]); }); it("does not match adjacent families or named sibling variants", () => { - expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-nano"))).toEqual([]); - expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-pro"))).toEqual([]); - expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-3-5-sonnet"))).toEqual([]); + expect(computerUseNativeSurfaces(getLoopModel("openai:gpt-5.4-nano"))).toEqual([]); + expect(computerUseNativeSurfaces(getLoopModel("openai:gpt-5.4-pro"))).toEqual([]); + expect(computerUseNativeSurfaces(getLoopModel("anthropic:claude-3-5-sonnet"))).toEqual([]); }); it("reports no native surface for models that have none, without refusing them", () => { - expect(cuaNativeSurfaces(getCuaModel("moonshotai:kimi-k3"))).toEqual([]); - expect(cuaNativeSurfaces(getCuaModel("xai:grok-4.5"))).toEqual([]); - // A model with no native surface still resolves and runs on CUA's own tools. - expect(getCuaModel("xai:grok-4.5").provider).toBe("xai"); + expect(computerUseNativeSurfaces(getLoopModel("moonshotai:kimi-k3"))).toEqual([]); + expect(computerUseNativeSurfaces(getLoopModel("xai:grok-4.5"))).toEqual([]); + // A model with no native surface still resolves and runs on Loop's own tools. + expect(getLoopModel("xai:grok-4.5").provider).toBe("xai"); }); it("surfaces the flag on catalog listings", () => { - const google = listCuaModels("google"); + const google = listLoopModels("google"); const flash = google.find((model) => model.model === "gemini-3.6-flash"); expect(flash?.nativeSurfaces).toEqual(["browser"]); expect(flash?.vision).toBe(true); @@ -187,20 +187,20 @@ describe("native surfaces", () => { describe("model quirks", () => { it("explains why every quirk exists", () => { - for (const quirk of CUA_MODEL_QUIRKS) { + for (const quirk of LOOP_MODEL_QUIRKS) { expect(quirk.reason.length).toBeGreaterThan(20); expect(Object.keys(quirk.capabilities).length).toBeGreaterThan(0); } }); it("defaults to permissive for a model with no quirk", () => { - expect(cuaModelCapabilities(getCuaModel("openai:gpt-5.6-sol"))).toEqual({ + expect(loopModelCapabilities(getLoopModel("openai:gpt-5.6-sol"))).toEqual({ acceptsComplexSchemas: true, acceptsLargeSchemas: true, serializesStateMutations: false, }); // Including a model pi-ai's registry does not carry. - expect(cuaModelCapabilities(getCuaModel("xai:grok-4.6")).acceptsComplexSchemas).toBe(true); + expect(loopModelCapabilities(getLoopModel("xai:grok-4.6")).acceptsComplexSchemas).toBe(true); }); it("keeps the limits we have evidence for", () => { @@ -208,33 +208,33 @@ describe("model quirks", () => { // JSON Schema keywords it does not know — `const` and `additionalProperties` — // and the catalog narrows both for it, so the same declarations every other // provider gets are accepted. Verified live against the Gemini API. - expect(cuaModelCapabilities(getCuaModel("google:gemini-3.6-flash")).acceptsComplexSchemas).toBe(true); - expect(cuaModelCapabilities(getCuaModel("google:gemini-3.6-flash")).acceptsLargeSchemas).toBe(true); + expect(loopModelCapabilities(getLoopModel("google:gemini-3.6-flash")).acceptsComplexSchemas).toBe(true); + expect(loopModelCapabilities(getLoopModel("google:gemini-3.6-flash")).acceptsLargeSchemas).toBe(true); // Observed live: Kimi K3 rejects the request once browser_act is attached. - expect(cuaModelCapabilities(getCuaModel("moonshotai:kimi-k3")).acceptsLargeSchemas).toBe(false); - expect(cuaModelCapabilities(getCuaModel("openrouter:moonshotai/kimi-k3")).acceptsLargeSchemas).toBe(false); + expect(loopModelCapabilities(getLoopModel("moonshotai:kimi-k3")).acceptsLargeSchemas).toBe(false); + expect(loopModelCapabilities(getLoopModel("openrouter:moonshotai/kimi-k3")).acceptsLargeSchemas).toBe(false); // Muse Spark accepts the large schema but serializes state mutations. - const muse = cuaModelCapabilities(getCuaModel("openrouter:meta/muse-spark-1.1")); + const muse = loopModelCapabilities(getLoopModel("openrouter:meta/muse-spark-1.1")); expect(muse.acceptsLargeSchemas).toBe(true); expect(muse.serializesStateMutations).toBe(true); }); it("applies a provider-wide quirk to every model from that provider", () => { - expect(cuaModelCapabilities(getCuaModel("xai:grok-4.5")).serializesStateMutations).toBe(true); - expect(cuaModelCapabilities(getCuaModel("xai:grok-4.6")).serializesStateMutations).toBe(true); + expect(loopModelCapabilities(getLoopModel("xai:grok-4.5")).serializesStateMutations).toBe(true); + expect(loopModelCapabilities(getLoopModel("xai:grok-4.6")).serializesStateMutations).toBe(true); }); }); describe("catalog passthrough", () => { it("exposes every provider pi-ai carries", () => { - expect(cuaProviders().length).toBeGreaterThan(20); - expect(cuaProviders()).toContain("openai"); - expect(cuaProviders()).toContain("groq"); - expect(cuaProviders()).toContain("zai"); + expect(loopProviders().length).toBeGreaterThan(20); + expect(loopProviders()).toContain("openai"); + expect(loopProviders()).toContain("groq"); + expect(loopProviders()).toContain("zai"); }); - it("lists models no CUA table mentions", () => { - const all = listCuaModels(); + it("lists models no Loop table mentions", () => { + const all = listLoopModels(); expect(all.length).toBeGreaterThan(100); // Previously refused for want of a table entry. expect(all.some((model) => model.ref === "xai:grok-4.3")).toBe(true); diff --git a/packages/ai/test/openai-adapter-routing.test.ts b/packages/loop/test/openai-adapter-routing.test.ts similarity index 85% rename from packages/ai/test/openai-adapter-routing.test.ts rename to packages/loop/test/openai-adapter-routing.test.ts index d258922c..9fabfb6d 100644 --- a/packages/ai/test/openai-adapter-routing.test.ts +++ b/packages/loop/test/openai-adapter-routing.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from "vitest"; import type { ToolCall } from "@earendil-works/pi-ai"; -import { createCuaModels, getCuaModel, OPENAI_CUA_COMPUTER_API } from "../src/index"; +import { createLoopModels, getLoopModel, OPENAI_COMPUTER_USE_API } from "../src/pi/index"; const { responsesCreate } = vi.hoisted(() => ({ responsesCreate: vi.fn() })); @@ -23,14 +23,14 @@ async function* responseEvents(response: Record) { yield { type: "response.completed", response }; } -// pi's builtin transport iterates an event stream; the CUA native-computer +// pi's builtin transport iterates an event stream; the Loop native-computer // adapter reads the raw response object's fields directly. Serve both from // one mock so a single test file can exercise either dispatch path. function responsePayload(response: Record): AsyncIterable & Record { return Object.assign(responseEvents(response), response); } -const model = getCuaModel("openai:gpt-5.5"); +const model = getLoopModel("openai:gpt-5.5"); const tools = [{ name: "lookup", description: "lookup", parameters: { type: "object" } as never }]; describe("OpenAI adapter routing", () => { @@ -49,13 +49,13 @@ describe("OpenAI adapter routing", () => { status: "completed", }], }); - const message = await createCuaModels().streamSimple(model, { + const message = await createLoopModels().streamSimple(model, { messages: [{ role: "user", content: "look it up", timestamp: 1 }], tools, }, { apiKey: "test", sessionId: "session_1" }).result(); // pi-ai 0.83.0's builtin Responses path does not parse the namespace - // field on function_call items, unlike the CUA adapter. + // field on function_call items, unlike the Loop adapter. const call = message.content.find((part): part is ToolCall => part.type === "toolCall") as (ToolCall & { namespace?: string }) | undefined; expect(call?.name).toBe("lookup"); expect(call?.namespace).toBeUndefined(); @@ -68,7 +68,7 @@ describe("OpenAI adapter routing", () => { expect(payload.previous_response_id).toBeUndefined(); }); - it("reaches the CUA adapter when the model carries OPENAI_CUA_COMPUTER_API", async () => { + it("reaches the Loop adapter when the model carries OPENAI_COMPUTER_USE_API", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_2", output: [{ @@ -77,25 +77,25 @@ describe("OpenAI adapter routing", () => { action: { type: "click", x: 10, y: 20 }, }], }); - // compileCuaToolCatalog derives this api onto the model whenever OpenAI's + // compileLoopToolCatalog derives this api onto the model whenever OpenAI's // native computer tool is selected; the provider wrapper dispatches on it // alone, with no request-shape sniffing. - const computerModel = { ...model, api: OPENAI_CUA_COMPUTER_API }; - const message = await createCuaModels().streamSimple(computerModel, { + const computerModel = { ...model, api: OPENAI_COMPUTER_USE_API }; + const message = await createLoopModels().streamSimple(computerModel, { messages: [{ role: "user", content: "click it", timestamp: 1 }], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", - cuaIncomingToolPlan: { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, + loopIncomingToolPlan: { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, } as never).result(); - // Only the CUA native-computer adapter understands computer_call items; + // Only the Loop native-computer adapter understands computer_call items; // pi's builtin transport has no case for them and would emit nothing. const call = message.content.find((part): part is ToolCall => part.type === "toolCall"); expect(call).toMatchObject({ name: "computer", arguments: { action: { type: "click", x: 10, y: 20 } } }); }); - it("reaches the CUA adapter when the transcript carries a deferred tool-search addition", async () => { + it("reaches the Loop adapter when the transcript carries a deferred tool-search addition", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_3", status: "completed", @@ -110,7 +110,7 @@ describe("OpenAI adapter routing", () => { status: "completed", }], }); - const message = await createCuaModels().streamSimple(model, { + const message = await createLoopModels().streamSimple(model, { messages: [ { role: "user", content: "load and look it up", timestamp: 1 }, { @@ -126,23 +126,23 @@ describe("OpenAI adapter routing", () => { tools, }, { apiKey: "test" } as never).result(); - // The CUA adapter round-trips the namespace pi's builtin drops. + // The Loop adapter round-trips the namespace pi's builtin drops. const call = message.content.find((part): part is ToolCall => part.type === "toolCall") as (ToolCall & { namespace?: string }) | undefined; expect(call?.namespace).toBe("deferred_tools"); }); it("keeps cache-relevant payload fields identical across a mid-conversation escalation", async () => { - // A turn can move from pi's builtin transport to the CUA adapter mid-session + // A turn can move from pi's builtin transport to the Loop adapter mid-session // (a deferred tool gets added). If that switch changed `store` or the cache // key, it would silently invalidate the matched prompt-cache prefix. responsesCreate.mockReturnValue({ id: "resp_parity", status: "completed", usage: {}, output: [] }); - await createCuaModels().streamSimple(model, { + await createLoopModels().streamSimple(model, { messages: [{ role: "user", content: "look it up", timestamp: 1 }], tools, }, { apiKey: "test", sessionId: "session_parity" }).result(); const builtinPayload = responsesCreate.mock.calls.at(-1)?.[0] as Record; - await createCuaModels().streamSimple(model, { + await createLoopModels().streamSimple(model, { messages: [ { role: "user", content: "load and look it up", timestamp: 1 }, { @@ -166,7 +166,7 @@ describe("OpenAI adapter routing", () => { it("pairs replayed namespaces by call id, not ordinal, across an aborted assistant turn", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_4", usage: {}, output: [] }); - await createCuaModels().streamSimple(model, { + await createLoopModels().streamSimple(model, { messages: [ { role: "user", content: "load and look it up", timestamp: 1 }, { diff --git a/packages/ai/test/openai-native-provider.test.ts b/packages/loop/test/openai-native-provider.test.ts similarity index 89% rename from packages/ai/test/openai-native-provider.test.ts rename to packages/loop/test/openai-native-provider.test.ts index 2371af1d..dd3dab15 100644 --- a/packages/ai/test/openai-native-provider.test.ts +++ b/packages/loop/test/openai-native-provider.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it, vi } from "vitest"; import type { Model, ToolCall } from "@earendil-works/pi-ai"; -import { getCuaModel } from "../src/index"; -import * as openai from "../src/providers/openai/provider"; +import { getLoopModel } from "../src/pi/index"; +import * as openai from "../src/pi/providers/openai/provider"; const { responsesCreate } = vi.hoisted(() => ({ responsesCreate: vi.fn() })); @@ -15,10 +15,10 @@ vi.mock("openai", () => ({ }, })); -const model = getCuaModel("openai:gpt-5.5") as Model<"openai-responses">; +const model = getLoopModel("openai:gpt-5.5") as Model<"openai-responses">; // The catalog derives this api when OpenAI's native computer tool is selected; // the provider wrapper routes it to the adapter under test. -const nativeModel = { ...model, api: openai.OPENAI_CUA_COMPUTER_API } as unknown as Model<"openai-responses">; +const nativeModel = { ...model, api: openai.OPENAI_COMPUTER_USE_API } as unknown as Model<"openai-responses">; const incoming = { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }; describe("OpenAI native computer Responses adapter", () => { @@ -33,12 +33,12 @@ describe("OpenAI native computer Responses adapter", () => { pending_safety_checks: [{ id: "check_1", code: "malicious_instructions" }], }], }); - const message = await openai.streamOpenAICuaComputer(nativeModel, { + const message = await openai.streamOpenAIComputerUse(nativeModel, { messages: [], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", - cuaIncomingToolPlan: incoming, + loopIncomingToolPlan: incoming, onPayload: (payload) => ({ ...(payload as Record), tools: [{ type: "computer" }] }), }).result(); const call = message.content.find((part): part is ToolCall => part.type === "toolCall"); @@ -58,10 +58,10 @@ describe("OpenAI native computer Responses adapter", () => { it("sends the same prompt-cache fields as the function-tool path", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_cache", usage: {}, output: [] }); - await openai.streamOpenAICuaComputer(nativeModel, { + await openai.streamOpenAIComputerUse(nativeModel, { messages: [{ role: "user", content: "go", timestamp: 1 }], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], - }, { apiKey: "test", sessionId: "session_native", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", sessionId: "session_native", loopIncomingToolPlan: incoming }).result(); const payload = responsesCreate.mock.calls.at(-1)?.[0] as Record; expect(payload.prompt_cache_key).toBe("session_native"); @@ -81,18 +81,18 @@ describe("OpenAI native computer Responses adapter", () => { arguments: '{"query":"status"}', }], }); - const first = await openai.streamOpenAICuaComputer(nativeModel, { + const first = await openai.streamOpenAIComputerUse(nativeModel, { messages: [{ role: "user", content: "look it up", timestamp: 1 }], tools: [ { name: "computer", description: "placeholder", parameters: { type: "object" } as never }, { name: "lookup", description: "lookup", parameters: { type: "object" } as never }, ], - }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); const call = first.content.find((part): part is ToolCall => part.type === "toolCall") as ToolCall & { namespace?: string }; expect(call.namespace).toBe("deferred_tools"); responsesCreate.mockReturnValueOnce({ id: "resp_done", usage: {}, output: [] }); - await openai.streamOpenAICuaComputer(nativeModel, { + await openai.streamOpenAIComputerUse(nativeModel, { messages: [ { role: "user", content: "look it up", timestamp: 1 }, first, @@ -110,7 +110,7 @@ describe("OpenAI native computer Responses adapter", () => { { name: "computer", description: "placeholder", parameters: { type: "object" } as never }, { name: "lookup", description: "lookup", parameters: { type: "object" } as never }, ], - }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); const payload = responsesCreate.mock.calls.at(-1)?.[0] as { input: Array>; store?: unknown; previous_response_id?: unknown }; expect(payload.input).toContainEqual(expect.objectContaining({ @@ -125,7 +125,7 @@ describe("OpenAI native computer Responses adapter", () => { it("serializes native results as computer_call_output and ordinary results as function output", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_2", usage: {}, output: [] }); - await openai.streamOpenAICuaComputer(nativeModel, { + await openai.streamOpenAIComputerUse(nativeModel, { messages: [ { role: "assistant", @@ -153,7 +153,7 @@ describe("OpenAI native computer Responses adapter", () => { tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", - cuaIncomingToolPlan: incoming, + loopIncomingToolPlan: incoming, onPayload: (payload) => ({ ...(payload as Record), tools: [{ type: "computer" }] }), }).result(); @@ -176,13 +176,13 @@ describe("OpenAI native computer Responses adapter", () => { describe("computer_call_output serialization", () => { async function sendWithResult(content: Array<{ type: string; [k: string]: unknown }>, isError: boolean) { responsesCreate.mockReturnValueOnce({ id: "resp_ser", usage: {}, output: [] }); - await openai.streamOpenAICuaComputer(nativeModel, { + await openai.streamOpenAIComputerUse(nativeModel, { messages: [ { role: "assistant", content: [{ type: "toolCall", id: "c1", name: "computer", arguments: {} }], stopReason: "toolUse" }, { role: "toolResult", toolCallId: "c1", toolName: "computer", content, isError, timestamp: 1 }, ] as never, tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], - }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); return JSON.stringify(responsesCreate.mock.calls.at(-1)?.[0]); } diff --git a/packages/pi-extension/test/pi-modes.test.ts b/packages/loop/test/pi-modes.test.ts similarity index 96% rename from packages/pi-extension/test/pi-modes.test.ts rename to packages/loop/test/pi-modes.test.ts index 89ee767d..b69ae55d 100644 --- a/packages/pi-extension/test/pi-modes.test.ts +++ b/packages/loop/test/pi-modes.test.ts @@ -184,12 +184,12 @@ async function runRpc( } describe("pi modes", () => { - it("runs a deterministic CUA browser tool in print and RPC modes", async () => { + it("runs a deterministic Loop browser tool in print and RPC modes", async () => { const server = await startFakeServer(); - const directory = await mkdtemp(join(tmpdir(), "cua-pi-mode-")); + const directory = await mkdtemp(join(tmpdir(), "loop-pi-mode-")); try { const agentDir = await fakeProviderConfig(directory, server.url); - const extension = fileURLToPath(new URL("../src/index.ts", import.meta.url)); + const extension = fileURLToPath(new URL("../src/pi-extension/index.ts", import.meta.url)); const env = { ...process.env, OPENAI_API_KEY: "test-key", diff --git a/packages/agent/test/provider-retry.test.ts b/packages/loop/test/provider-retry.test.ts similarity index 99% rename from packages/agent/test/provider-retry.test.ts rename to packages/loop/test/provider-retry.test.ts index edc89cfe..7e71c851 100644 --- a/packages/agent/test/provider-retry.test.ts +++ b/packages/loop/test/provider-retry.test.ts @@ -13,7 +13,7 @@ import { resolveProviderRetryPolicy, withProviderRetry, withProviderRetryModels, -} from "../src/provider-retry"; +} from "../src/pi/provider-retry"; const model = { id: "test", diff --git a/packages/pi-extension/test/provider-stream.test.ts b/packages/loop/test/provider-stream.test.ts similarity index 75% rename from packages/pi-extension/test/provider-stream.test.ts rename to packages/loop/test/provider-stream.test.ts index c1c29ac8..75de0a0f 100644 --- a/packages/pi-extension/test/provider-stream.test.ts +++ b/packages/loop/test/provider-stream.test.ts @@ -1,14 +1,14 @@ import type { Api, Model } from "@earendil-works/pi-ai"; import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; -import { getCuaModel } from "@onkernel/cua-ai"; +import { getLoopModel } from "../src/pi/index"; import { describe, expect, it, vi } from "vitest"; -// Only createCuaModels is replaced: the wrapped providers echo back what the +// Only createLoopModels is replaced: the wrapped providers echo back what the // extension forwarded, so the test observes the model and options that would go -// on the wire without a network call. Everything else in cua-ai stays real, and +// on the wire without a network call. Everything else in the package stays real, and // this mock is scoped to this file so it cannot weaken assertions elsewhere. -vi.mock("@onkernel/cua-ai", async (importOriginal) => { - const actual = await importOriginal(); +vi.mock("@onkernel/loop/pi", async (importOriginal) => { + const actual = await importOriginal(); const echo = (id: string) => ({ id, name: `echo-${id}`, @@ -19,14 +19,14 @@ vi.mock("@onkernel/cua-ai", async (importOriginal) => { }); return { ...actual, - createCuaModels: () => ({ - ...actual.createCuaModels(), + createLoopModels: () => ({ + ...actual.createLoopModels(), getProvider: (id: string) => (["anthropic", "openai", "google"].includes(id) ? echo(id) : undefined), }), }; }); -import extension from "../src/index"; +import extension from "../src/pi-extension/index"; interface FakeProvider { id: string; @@ -67,24 +67,24 @@ describe("provider stream ownership", () => { "browser-coordinates": "pixels", }); extension(pi.api); - const openaiCtx = { ...ctx, model: getCuaModel("openai:gpt-5.6-sol") } as ExtensionContext; + const openaiCtx = { ...ctx, model: getLoopModel("openai:gpt-5.6-sol") } as ExtensionContext; await pi.handlers.get("session_start")!({}, openaiCtx); expect(pi.active).toContain("computer"); // pi resolves its own registry model, whose api is the builtin transport. The // registered provider has to put the *compiled* api on the wire instead, or - // the CUA adapter never runs and `computer_call` items never normalize. + // the Loop adapter never runs and `computer_call` items never normalize. const provider = pi.providers.find((candidate) => candidate.id === "openai"); expect(provider).toBeDefined(); - const registryModel = getCuaModel("openai:gpt-5.6-sol"); + const registryModel = getLoopModel("openai:gpt-5.6-sol"); expect(registryModel.api).toBe("openai-responses"); const streamed = provider!.streamSimple(registryModel, { messages: [] } as never, { apiKey: "from-pi" } as never) as unknown as { model: Model; - options: { cuaIncomingToolPlan?: { openaiComputerName?: string }; apiKey?: string }; + options: { loopIncomingToolPlan?: { openaiComputerName?: string }; apiKey?: string }; }; - expect(streamed.model.api).toBe("openai-cua-computer"); - expect(streamed.options.cuaIncomingToolPlan?.openaiComputerName).toBe("computer"); + expect(streamed.model.api).toBe("openai-computer-use"); + expect(streamed.options.loopIncomingToolPlan?.openaiComputerName).toBe("computer"); // pi's resolved credential must survive the swap. expect(streamed.options.apiKey).toBe("from-pi"); }); diff --git a/packages/ai/test/providers.test.ts b/packages/loop/test/providers.test.ts similarity index 68% rename from packages/ai/test/providers.test.ts rename to packages/loop/test/providers.test.ts index c38ca2d9..aac9c1fd 100644 --- a/packages/ai/test/providers.test.ts +++ b/packages/loop/test/providers.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; -import { createCuaModels, cuaModels } from "../src/index"; +import { createLoopModels, loopModels } from "../src/pi/index"; -describe("createCuaModels", () => { - it("registers a streamable provider for every CUA provider", () => { - const models = createCuaModels(); +describe("createLoopModels", () => { + it("registers a streamable provider for every Loop provider", () => { + const models = createLoopModels(); for (const id of ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]) { const provider = models.getProvider(id); expect(provider, id).toBeDefined(); @@ -12,8 +12,8 @@ describe("createCuaModels", () => { } }); - it("lists CUA provider catalogs", () => { - const models = createCuaModels(); + it("lists Loop provider catalogs", () => { + const models = createLoopModels(); expect(models.getModel("xai", "grok-4.5")?.api).toBe("openai-responses"); expect(models.getModel("moonshotai", "kimi-k3")?.api).toBe("openai-completions"); expect(models.getModel("openrouter", "moonshotai/kimi-k3")?.api).toBe("openai-completions"); @@ -22,13 +22,13 @@ describe("createCuaModels", () => { }); it("keeps builtin catalogs on wrapped providers", () => { - const models = createCuaModels(); + const models = createLoopModels(); const openaiIds = models.getModels("openai").map((m) => m.id); expect(openaiIds).toContain("gpt-5.4"); // OpenAI models keep pi's builtin "openai-responses" api id on both the - // collection and getCuaModel(); the wrapped provider only intercepts a - // model carrying OPENAI_CUA_COMPUTER_API or a namespace round-trip (see - // requiresCuaOpenAINamespaceAdapter). + // collection and getLoopModel(); the wrapped provider only intercepts a + // model carrying OPENAI_COMPUTER_USE_API or a namespace round-trip (see + // requiresOpenAINamespaceAdapter). expect(models.getModel("openai", "gpt-5.4")?.api).toBe("openai-responses"); const xaiIds = models.getModels("xai").map((m) => m.id); @@ -37,17 +37,17 @@ describe("createCuaModels", () => { }); it("returns independent collections", () => { - const a = createCuaModels(); - const b = createCuaModels(); + const a = createLoopModels(); + const b = createLoopModels(); a.deleteProvider("google"); expect(a.getProvider("google")).toBeUndefined(); expect(b.getProvider("google")).toBeDefined(); }); }); -describe("cuaModels", () => { +describe("loopModels", () => { it("memoizes the default collection", () => { - expect(cuaModels()).toBe(cuaModels()); - expect(cuaModels().getProvider("openai")).toBeDefined(); + expect(loopModels()).toBe(loopModels()); + expect(loopModels().getProvider("openai")).toBeDefined(); }); }); diff --git a/packages/agent/test/published-declarations.test.ts b/packages/loop/test/published-declarations.test.ts similarity index 78% rename from packages/agent/test/published-declarations.test.ts rename to packages/loop/test/published-declarations.test.ts index f174c95d..793e58d4 100644 --- a/packages/agent/test/published-declarations.test.ts +++ b/packages/loop/test/published-declarations.test.ts @@ -5,20 +5,27 @@ import { join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import { describe, it } from "vitest"; -const agentRoot = fileURLToPath(new URL("..", import.meta.url)); -const repoRoot = resolve(agentRoot, "..", ".."); +const packageRoot = fileURLToPath(new URL("..", import.meta.url)); +const repoRoot = resolve(packageRoot, "..", ".."); /** * Downstream consumer compiled against the packaged declarations (dist/) with - * skipLibCheck disabled, so transitive declaration issues in cua-agent, - * cua-ai, pi, or TypeBox surface here instead of after publish. + * skipLibCheck disabled, so transitive declaration issues in this package, pi, + * or TypeBox surface here instead of after publish. */ const CONSUMER = ` import { Type } from "typebox"; +import { + loop, + type LoopAgentTool, + type LoopHarnessTool, + type KernelBrowser, +} from "@onkernel/loop"; import { Agent, AgentHarness, attach, + getLoopModel, InMemorySessionRepo, NodeExecutionEnv, createBashTool, @@ -26,12 +33,8 @@ import { createReadTool, createWriteTool, type AgentHarnessTool, - type CuaAgentTool, - type CuaHarnessTool, type ExecutionToolContext, - type KernelBrowser, -} from "@onkernel/cua-agent"; -import { cua, getCuaModel } from "@onkernel/cua-ai"; +} from "@onkernel/loop/pi"; import type Kernel from "@onkernel/sdk"; declare const browser: KernelBrowser; @@ -51,8 +54,8 @@ const custom: AgentHarnessTool = { }, }; -const harnessTools: readonly CuaHarnessTool[] = [ - ...cua.toolsets.browser(), +const harnessTools: readonly LoopHarnessTool[] = [ + ...loop.toolsets.browser(), createReadTool(), createBashTool(), createEditTool(), @@ -60,7 +63,7 @@ const harnessTools: readonly CuaHarnessTool[] = [ custom, ]; -const agentTools: readonly CuaAgentTool[] = [...cua.toolsets.browser()]; +const agentTools: readonly LoopAgentTool[] = [...loop.toolsets.browser()]; async function build() { const session = await new InMemorySessionRepo().create(); @@ -77,7 +80,7 @@ async function build() { }); const release = compiled.activate(harness); // A swap compiles for the same tool context the harness delivers. - await handle.compile({ model: getCuaModel("openai:gpt-5.6-sol"), tools: [] }).apply(harness); + await handle.compile({ model: getLoopModel("openai:gpt-5.6-sol"), tools: [] }).apply(harness); const lowLevel = handle.compile({ model: "openai:gpt-5.6-sol", tools: agentTools }); const agent = new Agent({ @@ -108,7 +111,7 @@ function run(command: string, args: string[], cwd: string): void { // skipLibCheck: false checks the whole reachable declaration graph. Two // vendored SDK typings cannot compile under it in a hoisted node_modules -// layout, independent of cua: @anthropic-ai/sdk unions import() fallbacks +// layout, independent of loop: @anthropic-ai/sdk unions import() fallbacks // for nested layouts that cannot resolve, and @google/genai references an // optional MCP package and DOM event globals. Tolerate exactly those two // files' errors; anything else — above all our dist declarations, pi's, or @@ -134,10 +137,12 @@ function runTsc(tsconfigDir: string): void { describe("published declarations", () => { it("compile a downstream consumer with skipLibCheck: false", { timeout: 300_000 }, () => { - run("npm", ["run", "build", "--workspace", "@onkernel/cua-ai"], repoRoot); - run("npm", ["run", "build", "--workspace", "@onkernel/cua-agent"], repoRoot); - - const dir = mkdtempSync(join(tmpdir(), "cua-declarations-")); + const dir = mkdtempSync(join(tmpdir(), "loop-declarations-")); + // Built beside dist/ rather than over it, because the pi extension tests + // import dist/ while this test runs, and inside the package so the emitted + // declarations resolve pi and TypeBox exactly as the published ones do. + const out = join(packageRoot, "dist-published"); + run("npx", ["tsdown", "--out-dir", out], packageRoot); writeFileSync(join(dir, "consumer.ts"), CONSUMER); writeFileSync(join(dir, "tsconfig.json"), JSON.stringify({ compilerOptions: { @@ -152,8 +157,8 @@ describe("published declarations", () => { noEmit: true, baseUrl: repoRoot, paths: { - "@onkernel/cua-ai": ["packages/ai/dist/index.d.ts"], - "@onkernel/cua-agent": ["packages/agent/dist/index.d.ts"], + "@onkernel/loop": [join(out, "index.d.ts")], + "@onkernel/loop/pi": [join(out, "pi", "index.d.ts")], "*": ["./node_modules/*"], }, }, diff --git a/packages/pi-extension/test/published-package.test.ts b/packages/loop/test/published-package.test.ts similarity index 67% rename from packages/pi-extension/test/published-package.test.ts rename to packages/loop/test/published-package.test.ts index cad7a964..8d96f22a 100644 --- a/packages/pi-extension/test/published-package.test.ts +++ b/packages/loop/test/published-package.test.ts @@ -5,11 +5,12 @@ import { describe, expect, it } from "vitest"; describe("published pi package", () => { it("ships a discoverable TypeScript extension manifest and runtime dependencies", async () => { const pkg = JSON.parse(await readFile(resolve(import.meta.dirname, "../package.json"), "utf8")); - expect(pkg.pi.extensions).toEqual(["./src/index.ts"]); + expect(pkg.pi.extensions).toEqual(["./src/pi-extension/index.ts"]); + // pi reads the extension's TypeScript directly, so the tarball ships src alongside dist. expect(pkg.files).toContain("src"); expect(pkg.dependencies).toMatchObject({ - "@onkernel/cua-ai": pkg.version, - "@onkernel/cua-agent": pkg.version, + "@earendil-works/pi-agent-core": expect.any(String), + "@earendil-works/pi-ai": expect.any(String), "@onkernel/sdk": expect.any(String), }); expect(pkg.peerDependencies.typebox).toBeUndefined(); diff --git a/packages/agent/test/resources.test.ts b/packages/loop/test/resources.test.ts similarity index 80% rename from packages/agent/test/resources.test.ts rename to packages/loop/test/resources.test.ts index f8f36c32..91aa4920 100644 --- a/packages/agent/test/resources.test.ts +++ b/packages/loop/test/resources.test.ts @@ -1,9 +1,8 @@ import { describe, expect, it, vi } from "vitest"; -import { cua, type CuaBrowserAction } from "@onkernel/cua-ai"; +import { type BrowserAction, type KernelBrowser, loop, LoopExecutionResources } from "../src/index"; import type Kernel from "@onkernel/sdk"; -import { CuaExecutionResources, type KernelBrowser } from "../src/index"; -import type { BrowserExecutor } from "../src/translator/browser"; -import type { BatchReadResult } from "../src/translator/types"; +import type { BrowserExecutor } from "../src/core/translator/browser"; +import type { BatchReadResult } from "../src/core/translator/types"; const browser = { session_id: "browser_123", @@ -32,10 +31,10 @@ function setup(options: { failBatch?: boolean; failPlaywright?: boolean; success : { success: true, result: "ok" } }, }, } as unknown as Kernel; - const executed: CuaBrowserAction[] = []; + const executed: BrowserAction[] = []; const browserScreenshot = vi.fn(async () => ({ data: Buffer.from("viewport"), mimeType: "image/png" })); const browserExecutor = { - async execute(action: CuaBrowserAction): Promise { + async execute(action: BrowserAction): Promise { executed.push(action); if (action.type === "browser_snapshot") return [{ type: "browser_text", label: "snapshot", text: "button Save [e1]" }]; if (action.type === "browser_text") return [{ type: "browser_text", label: "text", text: "Saved" }]; @@ -73,14 +72,14 @@ function setup(options: { failBatch?: boolean; failPlaywright?: boolean; success close() {}, } as unknown as BrowserExecutor; const createBrowserExecutor = vi.fn(() => browserExecutor); - const resources = new CuaExecutionResources({ browser, client, createBrowserExecutor }); + const resources = new LoopExecutionResources({ browser, client, createBrowserExecutor }); return { resources, batches, executed, createBrowserExecutor, captureScreenshot, browserScreenshot }; } -describe("CuaExecutionResources results and batch boundaries", () => { +describe("LoopExecutionResources results and batch boundaries", () => { it("flushes computer writes around ordered reads without adding actions", async () => { const { resources, batches } = setup(); - const spec = cua.tools.computer.batch({ actions: ["click", "screenshot", "keypress"] }); + const spec = loop.tools.computer.batch({ actions: ["click", "screenshot", "keypress"] }); const tool = resources.materialize(spec); const result = await tool.execute("batch", { actions: [ { action: "click", x: 10, y: 20 }, @@ -97,7 +96,7 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("retains completed textual reads, replaces error images, and records skipped actions", async () => { const { resources } = setup({ failBatch: true }); - const spec = cua.tools.computer.batch({ actions: ["screenshot", "click", "url", "cursor_position"] }); + const spec = loop.tools.computer.batch({ actions: ["screenshot", "click", "url", "cursor_position"] }); const tool = resources.materialize(spec); const result = await tool.execute("batch", { actions: [ { action: "screenshot" }, @@ -114,7 +113,7 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("keeps browser_batch mechanical and returns ordered reads without a fallback image", async () => { const { resources, executed } = setup(); - const spec = cua.tools.browser.batch({ actions: ["snapshot", "click", "text"] }); + const spec = loop.tools.browser.batch({ actions: ["snapshot", "click", "text"] }); const tool = resources.materialize(spec); const result = await tool.execute("browser-batch", { actions: [ { action: "snapshot" }, @@ -130,7 +129,7 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("replaces prior screenshots when a semantic browser batch condition fails", async () => { const { resources } = setup(); - const spec = cua.tools.browser.batch({ actions: ["screenshot", "wait_for"] }); + const spec = loop.tools.browser.batch({ actions: ["screenshot", "wait_for"] }); const result = await resources.materialize(spec).execute("browser-batch", { actions: [ { action: "screenshot" }, { action: "wait_for", expect: { type: "text", text: "Ready" } }, @@ -144,8 +143,8 @@ describe("CuaExecutionResources results and batch boundaries", () => { }); it.each([ - ["browser_wait_for", cua.tools.browser.waitFor(), { expect: { type: "text", text: "Ready" } }], - ["browser_act", cua.tools.browser.act(), { steps: [{ type: "wait" }] }], + ["browser_wait_for", loop.tools.browser.waitFor(), { expect: { type: "text", text: "Ready" } }], + ["browser_act", loop.tools.browser.act(), { steps: [{ type: "wait" }] }], ] as const)("marks a failed standalone %s result as an error", async (_name, spec, input) => { const { resources } = setup(); const result = await resources.materialize(spec).execute("standalone", input); @@ -162,7 +161,7 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("keeps a worked browser_act navigation boundary successful", async () => { const { resources } = setup({ successfulAct: true }); - const result = await resources.materialize(cua.tools.browser.act()).execute("act", { + const result = await resources.materialize(loop.tools.browser.act()).execute("act", { steps: [{ type: "click", x: 10, y: 20 }], }); expect(result.details).toMatchObject({ statusText: "Actions executed successfully." }); @@ -172,17 +171,17 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("shares one lazy browser executor across independently materialized tools", async () => { const { resources, createBrowserExecutor } = setup(); - await resources.materialize(cua.tools.browser.snapshot()).execute("snapshot", {}); - await resources.materialize(cua.tools.browser.click()).execute("click", { ref: "e1" }); + await resources.materialize(loop.tools.browser.snapshot()).execute("snapshot", {}); + await resources.materialize(loop.tools.browser.click()).execute("click", { ref: "e1" }); expect(createBrowserExecutor).toHaveBeenCalledTimes(1); }); it("returns status text for writes without capturing screenshots", async () => { const { resources, captureScreenshot, browserScreenshot } = setup(); - const click = await resources.materialize(cua.tools.browser.click()).execute("click", { ref: "e1" }); + const click = await resources.materialize(loop.tools.browser.click()).execute("click", { ref: "e1" }); expect(click.content).toEqual([{ type: "text", text: "Actions executed successfully." }]); - const navigate = await resources.materialize(cua.tools.browser.navigate()).execute("navigate", { url: "https://example.test" }); + const navigate = await resources.materialize(loop.tools.browser.navigate()).execute("navigate", { url: "https://example.test" }); expect(navigate.content).toEqual([{ type: "text", text: "Navigated" }]); expect(captureScreenshot).not.toHaveBeenCalled(); expect(browserScreenshot).not.toHaveBeenCalled(); @@ -190,7 +189,7 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("keeps Playwright execution failures as model-readable content", async () => { const { resources } = setup({ failPlaywright: true }); - const result = await resources.materialize(cua.tools.playwright()).execute("playwright", { code: "throw new Error('boom')" }); + const result = await resources.materialize(loop.tools.playwright()).execute("playwright", { code: "throw new Error('boom')" }); expect(result.content).toEqual([ { type: "text", text: "stderr:\ntrace" }, { type: "text", text: "error: page evaluation failed" }, @@ -205,7 +204,7 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("returns status text for provider-native writes without capturing a screenshot", async () => { const { resources, captureScreenshot } = setup(); - const spec = cua.providers.google.toolsets.browser().find((tool) => tool.name === "click")!; + const spec = loop.providers.google.toolsets.browser().find((tool) => tool.name === "click")!; const result = await resources.materialize(spec).execute("click", { x: 100, y: 200 }); expect(result.content).toEqual([{ type: "text", text: "Actions executed successfully." }]); expect(captureScreenshot).not.toHaveBeenCalled(); @@ -213,10 +212,10 @@ describe("CuaExecutionResources results and batch boundaries", () => { it("materializes each spec exactly once per resource pool", () => { const { resources } = setup(); - const spec = cua.tools.browser.snapshot(); + const spec = loop.tools.browser.snapshot(); const first = resources.materialize(spec); expect(resources.materialize(spec)).toBe(first); // A freshly created spec object is conservatively a different implementation. - expect(resources.materialize(cua.tools.browser.snapshot())).not.toBe(first); + expect(resources.materialize(loop.tools.browser.snapshot())).not.toBe(first); }); }); diff --git a/packages/pi-extension/test/selection.test.ts b/packages/loop/test/selection.test.ts similarity index 80% rename from packages/pi-extension/test/selection.test.ts rename to packages/loop/test/selection.test.ts index da79fea8..420a7ee7 100644 --- a/packages/pi-extension/test/selection.test.ts +++ b/packages/loop/test/selection.test.ts @@ -1,10 +1,16 @@ -import { getCuaModel } from "@onkernel/cua-ai"; +import { getLoopModel } from "../src/pi/index"; import { describe, expect, it } from "vitest"; -import { compileSpecs, CUA_SELECTORS, expandSelection, parseSelection, selectorAvailability } from "../src/selection"; -import { DEFAULT_BROWSER_TIMEOUT_SECONDS } from "../src/browser-runtime"; -import { parseBrowserOptions } from "../src/index"; +import { + compileSpecs, + expandSelection, + LOOP_SELECTORS, + parseSelection, + selectorAvailability, +} from "../src/pi-extension/selection"; +import { DEFAULT_BROWSER_TIMEOUT_SECONDS } from "../src/pi-extension/browser-runtime"; +import { parseBrowserOptions } from "../src/pi-extension/index"; -describe("CUA pi selectors", () => { +describe("Loop pi selectors", () => { it("has stable exact browser and computer entry membership, batch included", () => { expect(expandSelection(parseSelection("browser", "pixels")).map((tool) => tool.name)).toEqual([ "browser_snapshot", @@ -47,7 +53,7 @@ describe("CUA pi selectors", () => { ]); }); it("offers exactly the eight menu entries", () => { - expect([...CUA_SELECTORS]).toEqual([ + expect([...LOOP_SELECTORS]).toEqual([ "browser", "computer", "browser-act", @@ -69,16 +75,16 @@ describe("CUA pi selectors", () => { }); it("compiles Anthropic native computer use only for supported Anthropic models", () => { const specs = expandSelection(parseSelection("anthropic-computer", "pixels")); - const catalog = compileSpecs(getCuaModel("anthropic:claude-fable-5"), specs); + const catalog = compileSpecs(getLoopModel("anthropic:claude-fable-5"), specs); expect(specs.map((tool) => tool.name)).toEqual(["computer"]); expect(catalog.entries.map((entry) => entry.transport)).toEqual(["native"]); expect(catalog.headers.requirements).toContainEqual(expect.objectContaining({ value: "computer-use-2026-07-01" })); - expect(() => compileSpecs(getCuaModel("openai:gpt-5.6-sol"), specs)).toThrow("requires a anthropic model"); + expect(() => compileSpecs(getLoopModel("openai:gpt-5.6-sol"), specs)).toThrow("requires a anthropic model"); }); it("offers every provider-native surface as its own selector", () => { for (const selector of ["anthropic-computer", "anthropic-browser", "openai-computer", "google-browser"]) { - expect(CUA_SELECTORS).toContain(selector); + expect(LOOP_SELECTORS).toContain(selector); expect(expandSelection(parseSelection(selector, "pixels")).length).toBeGreaterThan(0); } }); @@ -87,20 +93,20 @@ describe("CUA pi selectors", () => { // This api is what the extension must put on the wire; pi's registry model // carries the builtin transport instead. const openai = expandSelection(parseSelection("openai-computer", "pixels")); - expect(compileSpecs(getCuaModel("openai:gpt-5.6-sol"), openai).model.api).toBe("openai-cua-computer"); + expect(compileSpecs(getLoopModel("openai:gpt-5.6-sol"), openai).model.api).toBe("openai-computer-use"); const google = expandSelection(parseSelection("google-browser", "pixels")); - expect(compileSpecs(getCuaModel("google:gemini-3.6-flash"), google).model.api).toBe("google-cua-interactions"); + expect(compileSpecs(getLoopModel("google:gemini-3.6-flash"), google).model.api).toBe("google-interactions"); // And the incoming plan is what normalizes the calls that come back. - expect(compileSpecs(getCuaModel("openai:gpt-5.6-sol"), openai).incoming.openaiComputerName).toBe("computer"); - expect(compileSpecs(getCuaModel("anthropic:claude-opus-5"), expandSelection(parseSelection("anthropic-browser", "pixels"))).incoming + expect(compileSpecs(getLoopModel("openai:gpt-5.6-sol"), openai).incoming.openaiComputerName).toBe("computer"); + expect(compileSpecs(getLoopModel("anthropic:claude-opus-5"), expandSelection(parseSelection("anthropic-browser", "pixels"))).incoming .anthropicBrowserFallback).toBeDefined(); }); it("reports selector availability per model with the compiler's own reason", () => { const empty = parseSelection(undefined, "pixels"); - const byName = new Map(selectorAvailability(getCuaModel("openai:gpt-5.6-sol"), empty).map((entry) => [entry.selector, entry])); + const byName = new Map(selectorAvailability(getLoopModel("openai:gpt-5.6-sol"), empty).map((entry) => [entry.selector, entry])); expect(byName.get("browser")?.available).toBe(true); expect(byName.get("playwright")?.available).toBe(true); @@ -120,12 +126,12 @@ describe("CUA pi selectors", () => { // Anthropic answers 400: the browser tool's viewport coordinate frame is // incompatible with the computer tool's display frame. const both = expandSelection(parseSelection("anthropic-computer,anthropic-browser", "pixels")); - expect(() => compileSpecs(getCuaModel("anthropic:claude-opus-5"), both)).toThrow(/cannot be selected alongside/); + expect(() => compileSpecs(getLoopModel("anthropic:claude-opus-5"), both)).toThrow(/cannot be selected alongside/); }); it("reports that pairwise conflict as a conflict, not as unavailability", () => { const byName = new Map( - selectorAvailability(getCuaModel("anthropic:claude-opus-5"), parseSelection(undefined, "pixels")).map((e) => [e.selector, e]), + selectorAvailability(getLoopModel("anthropic:claude-opus-5"), parseSelection(undefined, "pixels")).map((e) => [e.selector, e]), ); expect(byName.get("anthropic-computer")?.available).toBe(true); expect(byName.get("anthropic-computer")?.conflictsWith).toContain("anthropic-browser"); @@ -136,7 +142,7 @@ describe("CUA pi selectors", () => { it("no longer marks every row unavailable when the current selection fails to compile", () => { // The regression this replaces: a failing selection's error became the reason // on every row, including rows that then activated fine. - const model = getCuaModel("anthropic:claude-opus-5"); + const model = getLoopModel("anthropic:claude-opus-5"); const failing = parseSelection("anthropic-computer,anthropic-browser", "pixels"); expect(() => compileSpecs(model, expandSelection(failing))).toThrow(); diff --git a/packages/ai/test/tool-catalog.test.ts b/packages/loop/test/tool-catalog.test.ts similarity index 68% rename from packages/ai/test/tool-catalog.test.ts rename to packages/loop/test/tool-catalog.test.ts index d4e380c6..b19449d4 100644 --- a/packages/ai/test/tool-catalog.test.ts +++ b/packages/loop/test/tool-catalog.test.ts @@ -1,20 +1,13 @@ -import { Type, type Tool } from "@earendil-works/pi-ai"; +import { type Tool, Type } from "@earendil-works/pi-ai"; import { describe, expect, it } from "vitest"; -import { - callerToolIdentity, - compileCuaToolCatalog, - cua, - getCuaModel, - GOOGLE_CUA_INTERACTIONS_API, - OPENAI_CUA_COMPUTER_API, - type CuaToolSpec, -} from "../src/index"; - -function compile(model: Parameters[0]["model"], requestedTools: Parameters[0]["requestedTools"]) { - return compileCuaToolCatalog({ model, requestedTools }); +import { getLoopModel, GOOGLE_INTERACTIONS_API, OPENAI_COMPUTER_USE_API } from "../src/pi/index"; +import { callerToolIdentity, compileLoopToolCatalog, loop, type LoopToolSpec } from "../src/index"; + +function compile(model: Parameters[0]["model"], requestedTools: Parameters[0]["requestedTools"]) { + return compileLoopToolCatalog({ model, requestedTools }); } -/** Sanitized caller declaration: cua-ai never receives executable members. */ +/** Sanitized caller declaration: the compiler never receives executable members. */ function callerTool(name: string): Tool { return { name, @@ -23,46 +16,46 @@ function callerTool(name: string): Tool { }; } -describe("cua tool namespace", () => { - it("is frozen and exposes exact CUA toolset members", () => { - expect(Object.isFrozen(cua)).toBe(true); - expect(cua.toolsets.browser().map((tool) => tool.name)).toEqual([ +describe("loop tool namespace", () => { + it("is frozen and exposes exact Loop toolset members", () => { + expect(Object.isFrozen(loop)).toBe(true); + expect(loop.toolsets.browser().map((tool) => tool.name)).toEqual([ "browser_snapshot", "browser_text", "browser_find", "browser_click", "browser_hover", "browser_drag", "browser_fill", "browser_scroll_to", "browser_scroll", "browser_type", "browser_key", "browser_navigate", "browser_list_tabs", "browser_new_tab", "browser_screenshot", "browser_evaluate", "browser_wait_for", ]); - expect(cua.toolsets.computer().map((tool) => tool.name)).toEqual([ + expect(loop.toolsets.computer().map((tool) => tool.name)).toEqual([ "computer_click", "computer_double_click", "computer_mouse_down", "computer_mouse_up", "computer_type", "computer_keypress", "computer_scroll", "computer_move", "computer_drag", "computer_wait", "computer_screenshot", "computer_goto", "computer_back", "computer_forward", "computer_url", "computer_cursor_position", ]); - expect(cua.toolsets.mixed().map((tool) => tool.name)).toEqual([ - ...cua.toolsets.computer().map((tool) => tool.name), - ...cua.toolsets.browser().map((tool) => tool.name), + expect(loop.toolsets.mixed().map((tool) => tool.name)).toEqual([ + ...loop.toolsets.computer().map((tool) => tool.name), + ...loop.toolsets.browser().map((tool) => tool.name), ]); }); it("applies deterministic namespaces without changing identity", () => { - const [plain] = cua.toolsets.browser(); - const [namespaced] = cua.toolsets.browser({ namespace: "page" }); + const [plain] = loop.toolsets.browser(); + const [namespaced] = loop.toolsets.browser({ namespace: "page" }); expect(namespaced.name).toBe("page_browser_snapshot"); expect(namespaced.identity).toBe(plain.identity); }); it("requires explicit non-empty batch action lists", () => { - expect(() => cua.tools.computer.batch({ actions: [] })).toThrow(/non-empty/); - expect(() => cua.tools.browser.batch({ actions: [] })).toThrow(/non-empty/); - expect(cua.tools.computer.batch({ actions: ["click", "screenshot"] }).declaration.parameters).toMatchObject({ + expect(() => loop.tools.computer.batch({ actions: [] })).toThrow(/non-empty/); + expect(() => loop.tools.browser.batch({ actions: [] })).toThrow(/non-empty/); + expect(loop.tools.computer.batch({ actions: ["click", "screenshot"] }).declaration.parameters).toMatchObject({ type: "object", }); - const browserBatch = cua.tools.browser.batch({ actions: ["snapshot", "click", "wait_for", "text"] }); + const browserBatch = loop.tools.browser.batch({ actions: ["snapshot", "click", "wait_for", "text"] }); expect(browserBatch.name).toBe("browser_batch"); expect(JSON.stringify(browserBatch.declaration.parameters)).not.toMatch(/saveAs|\$ref|workflow|branch/i); }); it("exposes Google's exact current predefined browser action set", () => { - expect(cua.providers.google.toolsets.browser().map((tool) => tool.name)).toEqual([ + expect(loop.providers.google.toolsets.browser().map((tool) => tool.name)).toEqual([ "click", "double_click", "triple_click", "middle_click", "right_click", "mouse_down", "mouse_up", "move", "type", "drag_and_drop", "wait", "press_key", "key_down", "key_up", "hotkey", "take_screenshot", "scroll", "go_back", "navigate", "go_forward", @@ -70,15 +63,15 @@ describe("cua tool namespace", () => { }); it("cites first-party documentation for every provider tool surface", () => { - expect("toolsets" in cua.providers.anthropic).toBe(false); - expect("legacyBrowser" in cua.providers.google.toolsets).toBe(false); - const surfaces: Array<[string, CuaToolSpec[]]> = [ - [cua.providers.openai.source, [cua.providers.openai.tools.computer()]], - [cua.providers.anthropic.source, [ - cua.providers.anthropic.tools.browser(), - cua.providers.anthropic.tools.computer(), + expect("toolsets" in loop.providers.anthropic).toBe(false); + expect("legacyBrowser" in loop.providers.google.toolsets).toBe(false); + const surfaces: Array<[string, LoopToolSpec[]]> = [ + [loop.providers.openai.source, [loop.providers.openai.tools.computer()]], + [loop.providers.anthropic.source, [ + loop.providers.anthropic.tools.browser(), + loop.providers.anthropic.tools.computer(), ]], - [cua.providers.google.source, cua.providers.google.toolsets.browser()], + [loop.providers.google.source, loop.providers.google.toolsets.browser()], ]; for (const [source, tools] of surfaces) { expect(source).toMatch(/^https:\/\//); @@ -87,20 +80,20 @@ describe("cua tool namespace", () => { } }); - it("uses the same CUA-authored browser toolset with custom-function providers", () => { + it("uses the same Loop-authored browser toolset with custom-function providers", () => { for (const model of ["xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:meta/muse-spark-1.1"] as const) { - const catalog = compile(model, cua.toolsets.browser()); + const catalog = compile(model, loop.toolsets.browser()); expect(catalog.entries[0]).toMatchObject({ - identity: "cua.browser.snapshot.v1", + identity: "kloop.browser.snapshot.v1", name: "browser_snapshot", - origin: "cua", + origin: "loop", }); expect(catalog.entries.at(-1)?.name).toBe("browser_wait_for"); } }); }); -describe("compileCuaToolCatalog", () => { +describe("compileLoopToolCatalog", () => { it("accepts an exact empty catalog", () => { const catalog = compile("openai:gpt-5.5", []); expect(catalog.entries).toEqual([]); @@ -108,7 +101,7 @@ describe("compileCuaToolCatalog", () => { }); it("never exposes requested, executable, spec, or executor state", () => { - const catalog = compile("openai:gpt-5.5", [cua.tools.browser.snapshot(), callerTool("custom")]); + const catalog = compile("openai:gpt-5.5", [loop.tools.browser.snapshot(), callerTool("custom")]); expect("requested" in catalog).toBe(false); expect("agentTools" in catalog).toBe(false); for (const entry of catalog.entries) { @@ -145,9 +138,9 @@ describe("compileCuaToolCatalog", () => { it("preserves exact requested order and inspectable identities", () => { const custom = callerTool("customer_lookup"); - const catalog = compile("anthropic:claude-opus-5", [cua.tools.browser.snapshot(), custom]); + const catalog = compile("anthropic:claude-opus-5", [loop.tools.browser.snapshot(), custom]); expect(catalog.entries.map((entry) => [entry.identity, entry.name, entry.origin])).toEqual([ - ["cua.browser.snapshot.v1", "browser_snapshot", "cua"], + ["kloop.browser.snapshot.v1", "browser_snapshot", "loop"], ["caller.customer_lookup", "customer_lookup", "caller"], ]); expect(catalog.toolDeclarations.map((tool) => tool.name)).toEqual(["browser_snapshot", "customer_lookup"]); @@ -161,13 +154,13 @@ describe("compileCuaToolCatalog", () => { it("rejects duplicate identities and exact name collisions", () => { expect(() => compile("openai:gpt-5.5", [ - cua.tools.browser.snapshot(), - cua.tools.browser.snapshot({ name: "page_snapshot" }), - ])).toThrow(/identity "cua\.browser\.snapshot\.v1"/); + loop.tools.browser.snapshot(), + loop.tools.browser.snapshot({ name: "page_snapshot" }), + ])).toThrow(/identity "kloop\.browser\.snapshot\.v1"/); expect(() => compile("openai:gpt-5.5", [ - cua.tools.browser.act(), + loop.tools.browser.act(), callerTool("browser_act"), - ])).toThrow('tool name "browser_act" is requested by both "cua.browser.act.v1" and "caller.browser_act"'); + ])).toThrow('tool name "browser_act" is requested by both "kloop.browser.act.v1" and "caller.browser_act"'); }); it("rejects Anthropic OAuth-normalized name collisions case-insensitively", () => { @@ -178,28 +171,28 @@ describe("compileCuaToolCatalog", () => { // Moonshot's API accepts browser_wait_for (~15KB) but rejects the request // outright once browser_act's (~124KB) schema is attached, so the oversized // schema is gated separately from merely-complex ones. - expect(() => compile("moonshotai:kimi-k3", [cua.tools.browser.act()])) - .toThrow('provider moonshotai does not accept the schema size of "browser_act" (cua.browser.act.v1)'); - expect(() => compile("moonshotai:kimi-k3", [cua.tools.browser.waitFor()])).not.toThrow(); - expect(() => compile("moonshotai:kimi-k3", cua.toolsets.browser())).not.toThrow(); + expect(() => compile("moonshotai:kimi-k3", [loop.tools.browser.act()])) + .toThrow('provider moonshotai does not accept the schema size of "browser_act" (kloop.browser.act.v1)'); + expect(() => compile("moonshotai:kimi-k3", [loop.tools.browser.waitFor()])).not.toThrow(); + expect(() => compile("moonshotai:kimi-k3", loop.toolsets.browser())).not.toThrow(); }); it("still accepts browser_act on providers that take its schema size", () => { for (const model of ["openai:gpt-5.5", "anthropic:claude-opus-5", "xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { - expect(() => compile(model, [cua.tools.browser.act()]), model).not.toThrow(); + expect(() => compile(model, [loop.tools.browser.act()]), model).not.toThrow(); } }); it("rejects unsafe names and incompatible native tools", () => { expect(() => compile("openai:gpt-5.5", [callerTool("bad name")])).toThrow(/must match/); - expect(() => compile("openai:gpt-5.5", [cua.providers.anthropic.tools.computer()])).toThrow(/requires a anthropic model/); + expect(() => compile("openai:gpt-5.5", [loop.providers.anthropic.tools.computer()])).toThrow(/requires a anthropic model/); }); it("replaces only the selected OpenAI identity placeholder", async () => { const catalog = compile("openai:gpt-5.5", [ - cua.providers.openai.tools.computer(), + loop.providers.openai.tools.computer(), callerTool("click"), - cua.tools.browser.click(), + loop.tools.browser.click(), ]); const payload = { tools: [ @@ -219,8 +212,8 @@ describe("compileCuaToolCatalog", () => { it("composes Anthropic native browser declarations, access fallback, and ordinary functions", async () => { const catalog = compile("anthropic:claude-opus-5", [ - cua.providers.anthropic.tools.browser(), - cua.tools.browser.snapshot(), + loop.providers.anthropic.tools.browser(), + loop.tools.browser.snapshot(), ]); expect(catalog.headers.merge({ "anthropic-beta": "other-beta" })).toEqual({ "anthropic-beta": "other-beta,browser-use-2026-07-01", @@ -240,7 +233,7 @@ describe("compileCuaToolCatalog", () => { }); it("serializes Google's current native declaration and keeps custom functions", async () => { - const selected = cua.providers.google.toolsets.browser({ exclude: ["right_click", "triple_click"] }); + const selected = loop.providers.google.toolsets.browser({ exclude: ["right_click", "triple_click"] }); const catalog = compile("google:gemini-3.6-flash", [...selected, callerTool("custom")]); const next = await catalog.payload.apply({ tools: [ ...selected.map((tool) => ({ type: "function", name: tool.name })), @@ -267,7 +260,7 @@ describe("compileCuaToolCatalog", () => { }); it("excludes every other Google browser function from a take_screenshot-only catalog", async () => { - const current = cua.providers.google.toolsets.browser(); + const current = loop.providers.google.toolsets.browser(); const screenshot = current.find((tool) => tool.name === "take_screenshot")!; const expectedExcludedNames = current.map((tool) => tool.name).filter((name) => name !== screenshot.name); const catalog = compile("google:gemini-3.6-flash", [screenshot]); @@ -288,40 +281,40 @@ describe("compileCuaToolCatalog", () => { it("rejects browser_act but accepts browser primitives for both Kimi transports", () => { for (const model of ["moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3"] as const) { - expect(() => compile(model, [cua.tools.browser.act()])).toThrow(/schema size/); - expect(() => compile(model, cua.toolsets.browser())).not.toThrow(); + expect(() => compile(model, [loop.tools.browser.act()])).toThrow(/schema size/); + expect(() => compile(model, loop.toolsets.browser())).not.toThrow(); } }); it("serializes state-mutating catalogs with serial tool calls", async () => { for (const model of ["xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3", "openrouter:meta/muse-spark-1.1"] as const) { - const catalog = compile(model, cua.toolsets.browser()); + const catalog = compile(model, loop.toolsets.browser()); await expect(catalog.payload.apply({ parallel_tool_calls: true }, catalog.model)).resolves.toMatchObject({ parallel_tool_calls: false }); } }); it("rejects incompatible model changes", () => { - const nativeTools: Array<[CuaToolSpec[], string]> = [ - [[cua.providers.anthropic.tools.browser()], "anthropic"], - [[cua.providers.openai.tools.computer()], "openai"], - [[cua.providers.google.toolsets.browser()[0]!], "google"], + const nativeTools: Array<[LoopToolSpec[], string]> = [ + [[loop.providers.anthropic.tools.browser()], "anthropic"], + [[loop.providers.openai.tools.computer()], "openai"], + [[loop.providers.google.toolsets.browser()[0]!], "google"], ]; for (const [tools, provider] of nativeTools) { expect(() => compile("openrouter:moonshotai/kimi-k3", tools)).toThrow(new RegExp(`requires a ${provider} model`)); } - const requested = [cua.providers.anthropic.tools.browser()]; + const requested = [loop.providers.anthropic.tools.browser()]; expect(() => compile("openai:gpt-5.5", requested)).toThrow(/requires a anthropic model/); }); it("fingerprints coordinate replacements independently from name and schema", () => { - const pixels = compile("openai:gpt-5.5", [cua.tools.computer.click()]); - const normalized = compile("openai:gpt-5.5", [cua.tools.computer.click({ coordinates: cua.coordinates.normalized([0, 1000]) })]); + const pixels = compile("openai:gpt-5.5", [loop.tools.computer.click()]); + const normalized = compile("openai:gpt-5.5", [loop.tools.computer.click({ coordinates: loop.coordinates.normalized([0, 1000]) })]); expect(pixels.entries[0]?.schemaFingerprint).toBe(normalized.entries[0]?.schemaFingerprint); expect(pixels.entries[0]?.fingerprint).not.toBe(normalized.entries[0]?.fingerprint); }); it("produces deterministic fingerprints for identical declaration and model inputs", () => { - const compileInputs = () => [cua.tools.browser.snapshot(), cua.tools.computer.click(), callerTool("custom")]; + const compileInputs = () => [loop.tools.browser.snapshot(), loop.tools.computer.click(), callerTool("custom")]; const first = compile("openai:gpt-5.5", compileInputs()); const second = compile("openai:gpt-5.5", compileInputs()); expect(second.fingerprint).toBe(first.fingerprint); @@ -331,61 +324,61 @@ describe("compileCuaToolCatalog", () => { }); describe("transport derivation", () => { - it("keeps an OpenAI model on its registry api when only CUA browser tools are selected", () => { - const catalog = compile("openai:gpt-5.5", cua.toolsets.browser()); + it("keeps an OpenAI model on its registry api when only Loop browser tools are selected", () => { + const catalog = compile("openai:gpt-5.5", loop.toolsets.browser()); expect(catalog.model.api).toBe("openai-responses"); }); - it("derives OPENAI_CUA_COMPUTER_API when OpenAI's native computer tool is selected", () => { - const catalog = compile("openai:gpt-5.5", [cua.providers.openai.tools.computer()]); - expect(catalog.model.api).toBe(OPENAI_CUA_COMPUTER_API); + it("derives OPENAI_COMPUTER_USE_API when OpenAI's native computer tool is selected", () => { + const catalog = compile("openai:gpt-5.5", [loop.providers.openai.tools.computer()]); + expect(catalog.model.api).toBe(OPENAI_COMPUTER_USE_API); }); it("keeps a Google model on pi's builtin transport when only CDP browser tools are selected", () => { - const catalog = compile("google:gemini-3.6-flash", [cua.tools.browser.snapshot(), cua.tools.browser.click()]); + const catalog = compile("google:gemini-3.6-flash", [loop.tools.browser.snapshot(), loop.tools.browser.click()]); expect(catalog.model.api).toBe("google-generative-ai"); }); - it("derives GOOGLE_CUA_INTERACTIONS_API when Google's native browser toolset is selected", () => { - const catalog = compile("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); - expect(catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + it("derives GOOGLE_INTERACTIONS_API when Google's native browser toolset is selected", () => { + const catalog = compile("google:gemini-3.6-flash", loop.providers.google.toolsets.browser()); + expect(catalog.model.api).toBe(GOOGLE_INTERACTIONS_API); }); it("rejects a catalog whose selected tools require conflicting transports", () => { - const [click, scroll] = cua.providers.google.toolsets.browser(); - const conflicting: CuaToolSpec = { + const [click, scroll] = loop.providers.google.toolsets.browser(); + const conflicting: LoopToolSpec = { ...scroll!, identity: "test.conflicting-transport.v1", - providerBinding: { kind: "google-native", nativeName: "conflict", allNativeNames: ["conflict"], requiresApi: OPENAI_CUA_COMPUTER_API }, + providerBinding: { kind: "google-native", nativeName: "conflict", allNativeNames: ["conflict"], requiresApi: OPENAI_COMPUTER_USE_API }, }; expect(() => compile("google:gemini-3.6-flash", [click!, conflicting])).toThrow(/incompatible provider transports/); }); it("re-derives from a model object that already carries a stale derived api, instead of pinning it", () => { - const nativeCatalog = compile("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); - expect(nativeCatalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + const nativeCatalog = compile("google:gemini-3.6-flash", loop.providers.google.toolsets.browser()); + expect(nativeCatalog.model.api).toBe(GOOGLE_INTERACTIONS_API); - const recompiled = compile(nativeCatalog.model, [cua.tools.browser.snapshot(), cua.tools.browser.click()]); + const recompiled = compile(nativeCatalog.model, [loop.tools.browser.snapshot(), loop.tools.browser.click()]); expect(recompiled.model.api).toBe("google-generative-ai"); - const reselected = compile(recompiled.model, cua.providers.google.toolsets.browser()); - expect(reselected.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + const reselected = compile(recompiled.model, loop.providers.google.toolsets.browser()); + expect(reselected.model.api).toBe(GOOGLE_INTERACTIONS_API); }); it("re-derives an OpenAI model object that already carries a stale derived api, instead of pinning it", () => { - const nativeCatalog = compile("openai:gpt-5.5", [cua.providers.openai.tools.computer()]); - expect(nativeCatalog.model.api).toBe(OPENAI_CUA_COMPUTER_API); + const nativeCatalog = compile("openai:gpt-5.5", [loop.providers.openai.tools.computer()]); + expect(nativeCatalog.model.api).toBe(OPENAI_COMPUTER_USE_API); - const recompiled = compile(nativeCatalog.model, cua.toolsets.browser()); + const recompiled = compile(nativeCatalog.model, loop.toolsets.browser()); expect(recompiled.model.api).toBe("openai-responses"); }); }); describe("Gemini function-declaration schema", () => { it("rewrites the two keywords the Gemini API rejects", async () => { - const catalog = compileCuaToolCatalog({ - model: getCuaModel("google:gemini-3.6-flash"), - requestedTools: [cua.tools.browser.waitFor()], + const catalog = compileLoopToolCatalog({ + model: getLoopModel("google:gemini-3.6-flash"), + requestedTools: [loop.tools.browser.waitFor()], }); const raw = { tools: [ @@ -412,12 +405,12 @@ describe("Gemini function-declaration schema", () => { it("narrows the flat tool shape the Interactions transport emits", async () => { // Selecting a native surface alongside a function tool derives the Interactions // transport, which serializes tools flat instead of under functionDeclarations. - const catalog = compileCuaToolCatalog({ - model: getCuaModel("google:gemini-3.6-flash"), - requestedTools: [...cua.providers.google.toolsets.browser(), cua.tools.browser.waitFor()], + const catalog = compileLoopToolCatalog({ + model: getLoopModel("google:gemini-3.6-flash"), + requestedTools: [...loop.providers.google.toolsets.browser(), loop.tools.browser.waitFor()], }); const flat = { - tools: [{ type: "function", name: "browser_wait_for", parameters: cua.tools.browser.waitFor().declaration.parameters }], + tools: [{ type: "function", name: "browser_wait_for", parameters: loop.tools.browser.waitFor().declaration.parameters }], }; const sent = JSON.stringify(await catalog.payload.apply(flat, catalog.model)); expect(JSON.stringify(flat)).toContain('"const"'); @@ -426,9 +419,9 @@ describe("Gemini function-declaration schema", () => { }); it("leaves other providers' declarations untouched", async () => { - const catalog = compileCuaToolCatalog({ - model: getCuaModel("openai:gpt-5.6-sol"), - requestedTools: [cua.tools.browser.waitFor()], + const catalog = compileLoopToolCatalog({ + model: getLoopModel("openai:gpt-5.6-sol"), + requestedTools: [loop.tools.browser.waitFor()], }); const raw = { tools: [{ functionDeclarations: [{ name: "browser_wait_for", parameters: catalog.toolDeclarations[0]!.parameters }] }] }; const sent = JSON.stringify(await catalog.payload.apply(raw, catalog.model)); diff --git a/packages/agent/test/tool-manager.test.ts b/packages/loop/test/tool-manager.test.ts similarity index 69% rename from packages/agent/test/tool-manager.test.ts rename to packages/loop/test/tool-manager.test.ts index 15cc28df..c9f5c863 100644 --- a/packages/agent/test/tool-manager.test.ts +++ b/packages/loop/test/tool-manager.test.ts @@ -1,19 +1,20 @@ import { describe, expect, it, vi } from "vitest"; -import { callerToolIdentity, cua, GOOGLE_CUA_INTERACTIONS_API } from "@onkernel/cua-ai"; -import type Kernel from "@onkernel/sdk"; +import { type AgentTool, GOOGLE_INTERACTIONS_API } from "../src/pi/index"; import { - CuaExecutionResources, - type AgentTool, - type CuaAgentTool, + callerToolIdentity, type KernelBrowser, + loop, + type LoopAgentTool, + LoopExecutionResources, } from "../src/index"; -import { CuaToolManager } from "../src/tool-manager"; +import type Kernel from "@onkernel/sdk"; +import { LoopToolManager } from "../src/core/tool-manager"; const browser = { session_id: "browser_123", viewport: { width: 1440, height: 900 } } as KernelBrowser; const client = {} as Kernel; function setup() { - return new CuaExecutionResources({ browser, client }); + return new LoopExecutionResources({ browser, client }); } function callerTool(name: string, execute?: AgentTool["execute"], executionMode?: AgentTool["executionMode"]): AgentTool { @@ -27,10 +28,10 @@ function callerTool(name: string, execute?: AgentTool["execute"], executionMode? }; } -describe("CuaToolManager declaration projection", () => { +describe("LoopToolManager declaration projection", () => { it("projects caller AgentTools into fresh declaration-only compile inputs", () => { const tool = callerTool("lookup"); - const manager = new CuaToolManager(setup(), "openai:gpt-5.5", [tool]); + const manager = new LoopToolManager(setup(), "openai:gpt-5.5", [tool]); const [declaration] = manager.catalog.toolDeclarations; expect(declaration).toEqual({ name: "lookup", description: "lookup tool", parameters: tool.parameters }); expect(declaration).not.toBe(tool); @@ -50,11 +51,11 @@ describe("CuaToolManager declaration projection", () => { }); -describe("CuaToolManager identity join", () => { +describe("LoopToolManager identity join", () => { it("joins specs and caller tools strictly by identity across mixed ordering", async () => { const calls: string[] = []; - const snapshot = cua.tools.browser.snapshot(); - const renamedClick = cua.tools.browser.click({ name: "page_click" }); + const snapshot = loop.tools.browser.snapshot(); + const renamedClick = loop.tools.browser.click({ name: "page_click" }); const alpha = callerTool("alpha", async () => { calls.push("alpha"); return { content: [{ type: "text", text: "alpha" }], details: {} }; @@ -63,13 +64,13 @@ describe("CuaToolManager identity join", () => { calls.push("zeta"); return { content: [{ type: "text", text: "zeta" }], details: {} }; }); - const manager = new CuaToolManager(setup(), "openai:gpt-5.5", [zeta, renamedClick, alpha, snapshot]); + const manager = new LoopToolManager(setup(), "openai:gpt-5.5", [zeta, renamedClick, alpha, snapshot]); expect(manager.catalog.entries.map((entry) => entry.identity)).toEqual([ callerToolIdentity("zeta"), - "cua.browser.click.v1", + "kloop.browser.click.v1", callerToolIdentity("alpha"), - "cua.browser.snapshot.v1", + "kloop.browser.snapshot.v1", ]); const installed = manager.agentTools(); expect(installed.map((tool) => tool.name)).toEqual(["zeta", "page_click", "alpha", "browser_snapshot"]); @@ -80,31 +81,31 @@ describe("CuaToolManager identity join", () => { expect(calls).toEqual(["alpha"]); await installed[0]!.execute("call-2", {}); expect(calls).toEqual(["alpha", "zeta"]); - expect(manager.specFor("cua.browser.click.v1")).toBe(renamedClick); + expect(manager.specFor("kloop.browser.click.v1")).toBe(renamedClick); expect(manager.specFor(callerToolIdentity("alpha"))).toBeUndefined(); }); }); -describe("CuaToolManager transport derivation", () => { +describe("LoopToolManager transport derivation", () => { it("derives the compiled model's api from the tools selected with it", () => { const resources = setup(); - const cdp = new CuaToolManager(resources, "google:gemini-3.6-flash", [cua.tools.browser.snapshot()]); - const native = new CuaToolManager(resources, "google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); + const cdp = new LoopToolManager(resources, "google:gemini-3.6-flash", [loop.tools.browser.snapshot()]); + const native = new LoopToolManager(resources, "google:gemini-3.6-flash", loop.providers.google.toolsets.browser()); expect(cdp.catalog.model.api).toBe("google-generative-ai"); - expect(native.catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + expect(native.catalog.model.api).toBe(GOOGLE_INTERACTIONS_API); }); }); -describe("CuaToolManager materialization", () => { +describe("LoopToolManager materialization", () => { it("materializes each spec exactly once, however many pairs it is compiled into", () => { const resources = setup(); const spy = vi.spyOn(resources, "materialize"); - const spec = cua.tools.browser.snapshot(); + const spec = loop.tools.browser.snapshot(); - new CuaToolManager(resources, "openai:gpt-5.5", [spec]); - new CuaToolManager(resources, "openai:gpt-5.6-sol", [spec]); - new CuaToolManager(resources, "openai:gpt-5.6-sol", [spec, callerTool("added")]); + new LoopToolManager(resources, "openai:gpt-5.5", [spec]); + new LoopToolManager(resources, "openai:gpt-5.6-sol", [spec]); + new LoopToolManager(resources, "openai:gpt-5.6-sol", [spec, callerTool("added")]); // The executable is cached per pool and per spec object, so pi sees one // stable implementation across every recompile. diff --git a/packages/agent/test/translator-browser.test.ts b/packages/loop/test/translator-browser.test.ts similarity index 96% rename from packages/agent/test/translator-browser.test.ts rename to packages/loop/test/translator-browser.test.ts index 9401b07e..bff32117 100644 --- a/packages/agent/test/translator-browser.test.ts +++ b/packages/loop/test/translator-browser.test.ts @@ -1,16 +1,24 @@ import type Kernel from "@onkernel/sdk"; import sharp from "sharp"; import { describe, expect, it, vi } from "vitest"; -import type { CuaBrowserAction } from "@onkernel/cua-ai"; -import { BrowserExecutor } from "../src/translator/browser"; -import type { BrowserRefState } from "../src/translator/browser-ref-lifecycle"; -import { CdpProtocolError, type CdpConnection } from "../src/translator/cdp"; -import { formatBrowserActResult } from "../src/browser-result-format"; -import { runBrowserAct, type BrowserActRuntime } from "../src/translator/browser-act"; -import { evaluateBrowserExpectation, waitForBrowserExpectation } from "../src/translator/browser-wait"; -import { diffObservations, type BrowserObservation, type BrowserPresentation } from "../src/translator/browser-observation"; -import { InternalComputerTranslator, type KernelBrowser } from "../src/translator/translator"; -import type { BatchReadResult, BrowserActResult, BrowserWaitForResult } from "../src/translator/types"; +import type { BrowserAction } from "../src/index"; +import { BrowserExecutor } from "../src/core/translator/browser"; +import type { BrowserRefState } from "../src/core/translator/browser-ref-lifecycle"; +import { type CdpConnection, CdpProtocolError } from "../src/core/translator/cdp"; +import { formatBrowserActResult } from "../src/core/browser-result-format"; +import { type BrowserActRuntime, runBrowserAct } from "../src/core/translator/browser-act"; +import { evaluateBrowserExpectation, waitForBrowserExpectation } from "../src/core/translator/browser-wait"; +import { + type BrowserObservation, + type BrowserPresentation, + diffObservations, +} from "../src/core/translator/browser-observation"; +import { InternalComputerTranslator, type KernelBrowser } from "../src/core/translator/translator"; +import type { + BatchReadResult, + BrowserActResult, + BrowserWaitForResult, +} from "../src/core/translator/types"; const browser = { session_id: "browser_123", cdp_ws_url: "wss://example.test/cdp" } as KernelBrowser; @@ -31,9 +39,9 @@ function createClient() { } function createFakeBrowserExecutor() { - const executed: CuaBrowserAction[] = []; + const executed: BrowserAction[] = []; const executor = { - execute: async (action: CuaBrowserAction): Promise => { + execute: async (action: BrowserAction): Promise => { executed.push(action); if (action.type === "browser_text") return [{ type: "browser_text", label: "text", text: "hello" }]; return []; @@ -416,7 +424,7 @@ describe("browser_act orchestration", () => { const rt = runtime([ observation("before"), observation("before"), observation("after"), observation("after"), ], [waitResult("newly_verified")]); - const presentations: CuaBrowserAction[] = []; + const presentations: BrowserAction[] = []; const originalPresent = rt.present; rt.present = (state, snapshot) => { presentations.push(snapshot); @@ -512,7 +520,7 @@ describe("browser_act orchestration", () => { it("stops a mixed batch after a failed semantic wait", async () => { const { client } = createClient(); const executed: string[] = []; - const executor = { execute: async (action: CuaBrowserAction) => { + const executor = { execute: async (action: BrowserAction) => { executed.push(action.type); return action.type === "browser_wait_for" ? [{ type: "browser_wait_for", result: { status: "timed_out", evidence: "failed", initial: { truth: false, details: [] }, final: { truth: false, details: [] }, elapsed_ms: 20, details: [] } } as BatchReadResult] : []; } } as unknown as BrowserExecutor; @@ -524,7 +532,7 @@ describe("browser_act orchestration", () => { it("stops a mixed batch after a worked plan's terminal navigation boundary", async () => { const { client } = createClient(); const executed: string[] = []; - const executor = { execute: async (action: CuaBrowserAction) => { + const executor = { execute: async (action: BrowserAction) => { executed.push(action.type); return action.type === "browser_act" ? [{ type: "browser_act", result: { outcome: "worked", steps: [], stopped_at: 0, stop_reason: "navigation", successor: { status: "observed", text: "new page", url: "https://example.test/new", title: "New", diff: { changed: true, added: [], removed: [], url: { before: "https://example.test/old", after: "https://example.test/new" } } } } } as BatchReadResult] : []; } } as unknown as BrowserExecutor; @@ -849,7 +857,7 @@ function refsOf(executor: BrowserExecutor): Map { } async function snapshotText(executor: BrowserExecutor, action: Record = {}): Promise { - const results = await executor.execute({ type: "browser_snapshot", ...action } as CuaBrowserAction); + const results = await executor.execute({ type: "browser_snapshot", ...action } as BrowserAction); const read = results[0]!; if (read.type !== "browser_text") throw new Error("expected browser_text read result"); return read.text; @@ -872,7 +880,7 @@ describe("BrowserExecutor ref lifecycle", () => { const executor = new BrowserExecutor(cdp); await snapshotText(executor); expect(refsOf(executor).size).toBe(1); - await executor.execute({ type: "browser_navigate", url: "https://b.test" } as CuaBrowserAction); + await executor.execute({ type: "browser_navigate", url: "https://b.test" } as BrowserAction); expect(refsOf(executor).size).toBe(0); }); @@ -883,7 +891,7 @@ describe("BrowserExecutor ref lifecycle", () => { expect(refsOf(executor).size).toBe(1); failOn("Page.navigate"); - await expect(executor.execute({ type: "browser_navigate", url: "https://b.test" } as CuaBrowserAction)).rejects.toThrow(/rejected/); + await expect(executor.execute({ type: "browser_navigate", url: "https://b.test" } as BrowserAction)).rejects.toThrow(/rejected/); // A page-initiated navigation right after the failed command must still invalidate. emit({ method: "Page.frameNavigated", params: { frame: { id: "F0" } }, sessionId: "session-1" }); @@ -897,11 +905,11 @@ describe("BrowserExecutor ref lifecycle", () => { expect(text).toContain('button "Save" [e1]'); emit({ method: "Page.frameNavigated", params: { frame: { id: "F2", parentId: "F1" } }, sessionId: "session-1" }); - await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent")).toBe(true); emit({ method: "Page.frameNavigated", params: { frame: { id: "F1" } }, sessionId: "session-1" }); - await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); expect(refsOf(executor).size).toBe(0); }); @@ -928,23 +936,23 @@ describe("BrowserExecutor ref lifecycle", () => { const sent = new Promise((resolve) => { commandSent = resolve; }); fake.setSendHook((method) => { if (method === "Page.navigate") commandSent(); }); const executor = new BrowserExecutor(fake.cdp); - const navigation = executor.execute({ type: "browser_navigate", url: "https://a.test/#section" } as CuaBrowserAction); + const navigation = executor.execute({ type: "browser_navigate", url: "https://a.test/#section" } as BrowserAction); await sent; fake.emit({ method: "Page.navigatedWithinDocument", params: { frameId: "TARGET-1", url: "https://a.test/#section" }, sessionId: "session-1" }); await navigation; await snapshotText(executor); fake.emit({ method: "Page.frameNavigated", params: { frame: { id: "TARGET-1" } }, sessionId: "session-1" }); - await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); }); it("does not double-bump the generation for its own navigate", async () => { const { cdp } = createFakeCdp(BUTTON_TREE); const executor = new BrowserExecutor(cdp); - await executor.execute({ type: "browser_navigate", url: "https://b.test" } as CuaBrowserAction); + await executor.execute({ type: "browser_navigate", url: "https://b.test" } as BrowserAction); const text = await snapshotText(executor); expect(text).toContain('button "Save" [e1]'); - await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).resolves.toEqual([]); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).resolves.toEqual([]); }); }); @@ -953,7 +961,7 @@ describe("BrowserExecutor navigation stabilization", () => { const fake = createFakeCdp(BUTTON_TREE); const executor = new BrowserExecutor(fake.cdp); - await expect(executor.execute({ type: "browser_navigate", url: "https://fast.test" } as CuaBrowserAction)).resolves.toEqual([ + await expect(executor.execute({ type: "browser_navigate", url: "https://fast.test" } as BrowserAction)).resolves.toEqual([ expect.objectContaining({ type: "browser_text", label: "navigate" }), ]); @@ -972,7 +980,7 @@ describe("BrowserExecutor navigation stabilization", () => { fake.setSendHook((method) => { if (method === "Page.navigate") commandSent(); }); const executor = new BrowserExecutor(fake.cdp); let settled = false; - const navigation = executor.execute({ type: "browser_navigate", url: "https://new.test" } as CuaBrowserAction) + const navigation = executor.execute({ type: "browser_navigate", url: "https://new.test" } as BrowserAction) .then((result) => { settled = true; return result; }); await sent; @@ -993,7 +1001,7 @@ describe("BrowserExecutor navigation stabilization", () => { fake.setSendHook((method) => { if (method === "Page.navigate") commandSent(); }); const executor = new BrowserExecutor(fake.cdp); let settled = false; - const navigation = executor.execute({ type: "browser_navigate", url: "https://redirect.test" } as CuaBrowserAction) + const navigation = executor.execute({ type: "browser_navigate", url: "https://redirect.test" } as BrowserAction) .then((result) => { settled = true; return result; }); await sent; @@ -1056,7 +1064,7 @@ describe("BrowserExecutor navigation stabilization", () => { fake.setSendHook((method) => { if (method === "Page.navigate") commandSent(); }); const executor = new BrowserExecutor(fake.cdp); const controller = new AbortController(); - const navigation = executor.execute({ type: "browser_navigate", url: "https://slow.test" } as CuaBrowserAction, controller.signal); + const navigation = executor.execute({ type: "browser_navigate", url: "https://slow.test" } as BrowserAction, controller.signal); await sent; await flushMicrotasks(); controller.abort(new Error("stop navigation")); @@ -1073,7 +1081,7 @@ describe("BrowserExecutor navigation stabilization", () => { fake.setNavigationAutoLifecycle(false); const executor = new BrowserExecutor(fake.cdp); await snapshotText(executor); - const navigation = executor.execute({ type: "browser_navigate", url: "https://never-loads.test" } as CuaBrowserAction); + const navigation = executor.execute({ type: "browser_navigate", url: "https://never-loads.test" } as BrowserAction); await flushMicrotasks(12); expect(fake.sent.some((command) => command.method === "Page.navigate")).toBe(true); expect(refsOf(executor).size).toBe(0); @@ -1091,7 +1099,7 @@ describe("BrowserExecutor navigation stabilization", () => { fake.setHistoryNavigationMode("load"); const executor = new BrowserExecutor(fake.cdp); - await expect(executor.execute({ type: "browser_navigate", url: "back" } as CuaBrowserAction)).resolves.toEqual([ + await expect(executor.execute({ type: "browser_navigate", url: "back" } as BrowserAction)).resolves.toEqual([ expect.objectContaining({ type: "browser_text", label: "navigate", text: expect.stringContaining("Navigated back") }), ]); expect(fake.sent).toContainEqual(expect.objectContaining({ method: "Page.navigateToHistoryEntry", params: { entryId: 1 } })); @@ -1110,7 +1118,7 @@ describe("BrowserExecutor navigation stabilization", () => { const controller = new AbortController(); let settled = false; let result: Awaited> | undefined; - const navigation = executor.execute({ type: "browser_navigate", url: direction } as CuaBrowserAction, controller.signal) + const navigation = executor.execute({ type: "browser_navigate", url: direction } as BrowserAction, controller.signal) .then((value) => { settled = true; result = value; return value; }); await sent; @@ -1271,7 +1279,7 @@ describe("BrowserExecutor stale-ref self-healing", () => { ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), ]); - await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(sent.some((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded" && cmd.params.backendNodeId === 99)).toBe(true); expect(sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent")).toBe(true); }); @@ -1289,7 +1297,7 @@ describe("BrowserExecutor stale-ref self-healing", () => { ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), ax({ nodeId: "3", role: "button", name: "Save", backendDOMNodeId: 100, parentId: "1" }), ]); - await executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e2" } as BrowserAction); expect(sent.some((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded" && cmd.params.backendNodeId === 100)).toBe(true); }); @@ -1304,7 +1312,7 @@ describe("BrowserExecutor stale-ref self-healing", () => { ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), ax({ nodeId: "2", role: "textbox", name: "Email", backendDOMNodeId: 99, parentId: "1" }), ]); - await executor.execute({ type: "browser_fill", ref: "e1", value: "a@b.c" } as CuaBrowserAction); + await executor.execute({ type: "browser_fill", ref: "e1", value: "a@b.c" } as BrowserAction); expect(sent.some((cmd) => cmd.method === "DOM.resolveNode" && cmd.params.backendNodeId === 99)).toBe(true); expect(sent.some((cmd) => cmd.method === "Runtime.callFunctionOn")).toBe(true); }); @@ -1318,7 +1326,7 @@ describe("BrowserExecutor stale-ref self-healing", () => { ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), ax({ nodeId: "3", role: "button", name: "Save", backendDOMNodeId: 100, parentId: "1" }), ]); - await expect(executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); }); it("refuses to heal a duplicate ref when the cohort shrank", async () => { @@ -1333,7 +1341,7 @@ describe("BrowserExecutor stale-ref self-healing", () => { ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), ax({ nodeId: "2", role: "button", name: "Save", backendDOMNodeId: 99, parentId: "1" }), ]); - await expect(executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e2" } as BrowserAction)).rejects.toThrow(/stale/); }); }); @@ -1347,7 +1355,7 @@ describe("BrowserExecutor fill", () => { const { cdp, sent } = createFakeCdp(FILL_TREE); const executor = new BrowserExecutor(cdp); await snapshotText(executor); - await executor.execute({ type: "browser_fill", ref: "e1", value: "a@b.c" } as CuaBrowserAction); + await executor.execute({ type: "browser_fill", ref: "e1", value: "a@b.c" } as BrowserAction); const call = sent.find((cmd) => cmd.method === "Runtime.callFunctionOn"); const declaration = call?.params.functionDeclaration as string; const fillFn = new Function(`return (${declaration})`)() as (value: unknown) => void; @@ -1386,7 +1394,7 @@ describe("BrowserExecutor fill", () => { } as unknown as CdpConnection; const executor = new BrowserExecutor(wrapped); await snapshotText(executor); - await expect(executor.execute({ type: "browser_fill", ref: "e1", value: "x" } as CuaBrowserAction)).rejects.toThrow( + await expect(executor.execute({ type: "browser_fill", ref: "e1", value: "x" } as BrowserAction)).rejects.toThrow( /^browser_fill failed: element is not a form control$/, ); }); @@ -1412,7 +1420,7 @@ describe("BrowserExecutor cursor-pointer hints", () => { const { cdp, failOn, sent } = createFakeCdp(BUTTON_TREE); failOn("Runtime.evaluate"); const executor = new BrowserExecutor(cdp); - const results = await executor.execute({ type: "browser_find", query: "save button" } as CuaBrowserAction); + const results = await executor.execute({ type: "browser_find", query: "save button" } as BrowserAction); expect((results[0] as { text: string }).text).toContain('button "Save" [e1]'); expect(sent.some((command) => command.method === "Runtime.evaluate")).toBe(false); }); @@ -1422,13 +1430,13 @@ describe("BrowserExecutor dialog guard", () => { it("dismisses confirm/prompt dialogs and surfaces the message on the next action", async () => { const { cdp, emit, sent } = createFakeCdp(); const executor = new BrowserExecutor(cdp); - await executor.execute({ type: "browser_text" } as CuaBrowserAction); + await executor.execute({ type: "browser_text" } as BrowserAction); emit({ method: "Page.javascriptDialogOpening", params: { type: "confirm", message: "Delete item?" }, sessionId: "session-1" }); const handled = sent.find((cmd) => cmd.method === "Page.handleJavaScriptDialog"); expect(handled).toEqual({ method: "Page.handleJavaScriptDialog", params: { accept: false }, sessionId: "session-1" }); - const results = await executor.execute({ type: "browser_text" } as CuaBrowserAction); + const results = await executor.execute({ type: "browser_text" } as BrowserAction); expect(results).toEqual([ { type: "browser_text", label: "text", text: "hello" }, { type: "browser_text", label: "dialog", text: 'Dismissed a JavaScript confirm dialog (answered No/cancel): "Delete item?"' }, @@ -1438,14 +1446,14 @@ describe("BrowserExecutor dialog guard", () => { it("accepts alert and beforeunload dialogs so navigation can proceed", async () => { const { cdp, emit, sent } = createFakeCdp(); const executor = new BrowserExecutor(cdp); - await executor.execute({ type: "browser_text" } as CuaBrowserAction); + await executor.execute({ type: "browser_text" } as BrowserAction); emit({ method: "Page.javascriptDialogOpening", params: { type: "beforeunload", message: "" }, sessionId: "session-1" }); emit({ method: "Page.javascriptDialogOpening", params: { type: "alert", message: "Saved!" }, sessionId: "session-1" }); const handled = sent.filter((cmd) => cmd.method === "Page.handleJavaScriptDialog"); expect(handled.map((cmd) => cmd.params)).toEqual([{ accept: true }, { accept: true }]); - const results = await executor.execute({ type: "browser_text" } as CuaBrowserAction); + const results = await executor.execute({ type: "browser_text" } as BrowserAction); expect(results[1]).toEqual({ type: "browser_text", label: "dialog", @@ -1462,7 +1470,7 @@ describe("BrowserExecutor snapshot diffing", () => { const executor = new BrowserExecutor(cdp); expect(await snapshotText(executor)).toContain('button "Save" [e1]'); expect(await snapshotText(executor)).toBe(UNCHANGED); - await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction); setNodes([ ax({ nodeId: "1", role: "RootWebArea", name: "Page", childIds: ["2"] }), ax({ nodeId: "2", role: "button", name: "Delete", backendDOMNodeId: 43, parentId: "1" }), @@ -1490,7 +1498,7 @@ describe("BrowserExecutor semantic waits", () => { { targetId: "TARGET-2", type: "page", title: "Two", url: "https://b.test/" }, ]); const executor = new BrowserExecutor(fake.cdp); - await executor.execute({ type: "browser_text", tab_id: "TARGET-2" } as CuaBrowserAction); + await executor.execute({ type: "browser_text", tab_id: "TARGET-2" } as BrowserAction); fake.setAxReadHook((read) => { if (read === 2) fake.emit({ method: "Page.navigatedWithinDocument", params: { frameId }, sessionId }); }); @@ -1500,7 +1508,7 @@ describe("BrowserExecutor semantic waits", () => { expect: { type: "text", text: "Ready" }, timeout_ms: 20, poll_ms: 1, - } as CuaBrowserAction); + } as BrowserAction); expect(read).toMatchObject({ type: "browser_wait_for", result: { status: "timed_out", evidence: "failed" } }); }); @@ -1521,7 +1529,7 @@ describe("BrowserExecutor semantic waits", () => { expect: { type: "ref", ref: "e1", ...expected }, timeout_ms: 2, poll_ms: 1, - } as CuaBrowserAction); + } as BrowserAction); expect(read).toMatchObject({ type: "browser_wait_for", result: { status } }); }); @@ -1546,14 +1554,14 @@ describe("BrowserExecutor semantic waits", () => { expect: { type: "text", text: "Ready" }, timeout_ms: 100, poll_ms: 10, - } as CuaBrowserAction); + } as BrowserAction); expect(read).toMatchObject({ type: "browser_wait_for", result: { status: "satisfied", evidence: "newly_verified" } }); }); }); describe("BrowserExecutor action plans", () => { async function act(executor: BrowserExecutor, action: Record) { - const [read] = await executor.execute({ type: "browser_act", timeout_ms: 100, poll_ms: 10, ...action } as CuaBrowserAction); + const [read] = await executor.execute({ type: "browser_act", timeout_ms: 100, poll_ms: 10, ...action } as BrowserAction); if (read?.type !== "browser_act") throw new Error("expected browser_act result"); return read.result; } @@ -1675,10 +1683,10 @@ describe("BrowserExecutor iframe stitching", () => { const executor = new BrowserExecutor(cdp); const text = await snapshotText(executor); expect(text).toBe(['RootWebArea "Page"', " Iframe [e1]", ' RootWebArea "Embed"', ' button "Inside" [e2]'].join("\n")); - await executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e2" } as BrowserAction); emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-SP", parentId: "F0" } }, sessionId: "session-1" }); - await expect(executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e2" } as BrowserAction)).rejects.toThrow(/stale/); }); const OOPIF_PAGE = [ @@ -1726,7 +1734,7 @@ describe("BrowserExecutor iframe stitching", () => { expect(text).toContain('button "Top" [e1]'); expect(text).toContain(' button "Pay" [e3]'); - await executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e3" } as BrowserAction); const scrolled = sent.find((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded" && cmd.params.backendNodeId === 70); expect(scrolled?.sessionId).toBe("session-oop"); const pressed = sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); @@ -1742,7 +1750,7 @@ describe("BrowserExecutor iframe stitching", () => { const text = await snapshotText(executor); expect(text).toContain(' button "Pay" [e3]'); - await executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e3" } as BrowserAction); const owner = fake.sent.find((cmd) => cmd.method === "DOM.getFrameOwner" && cmd.params.frameId === "FRAME-OOP"); expect(owner?.sessionId).toBe("session-1"); const pressed = fake.sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); @@ -1756,7 +1764,7 @@ describe("BrowserExecutor iframe stitching", () => { const executor = new BrowserExecutor(fake.cdp); await snapshotText(executor); - await executor.execute({ type: "browser_hover", ref: "e3" } as CuaBrowserAction); + await executor.execute({ type: "browser_hover", ref: "e3" } as BrowserAction); const moved = fake.sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mouseMoved"); expect(moved?.sessionId).toBe("session-1"); expect([moved?.params.x, moved?.params.y]).toEqual([105, 205]); @@ -1770,7 +1778,7 @@ describe("BrowserExecutor iframe stitching", () => { // e1 is the top-level "Top" button (backend node 40): read through the page // session, so its quads are already top-level and must not be shifted. - await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(fake.sent.some((cmd) => cmd.method === "DOM.getFrameOwner")).toBe(false); const pressed = fake.sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); expect([pressed?.params.x, pressed?.params.y]).toEqual([5, 5]); @@ -1793,7 +1801,7 @@ describe("BrowserExecutor iframe stitching", () => { const executor = new BrowserExecutor(fake.cdp); await snapshotText(executor); - await executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e2" } as BrowserAction); expect(fake.sent.some((cmd) => cmd.method === "DOM.getFrameOwner")).toBe(false); const pressed = fake.sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); expect([pressed?.params.x, pressed?.params.y]).toEqual([5, 5]); @@ -1816,7 +1824,7 @@ describe("BrowserExecutor iframe stitching", () => { const reboundCdp = nestedSetup(); const rebound = new BrowserExecutor(reboundCdp.cdp); rebound.importRefState(state); - const [worked] = await rebound.execute({ type: "browser_act", steps: [{ type: "click", ref: "e3" }] } as CuaBrowserAction); + const [worked] = await rebound.execute({ type: "browser_act", steps: [{ type: "click", ref: "e3" }] } as BrowserAction); expect(worked).toMatchObject({ type: "browser_act", result: { steps: [{ diagnostics: ["action dispatched"] }] } }); expect(reboundCdp.sent.find((command) => command.method === "DOM.scrollIntoViewIfNeeded" && command.params.backendNodeId === 80)?.sessionId).toBe("session-oop"); @@ -1825,7 +1833,7 @@ describe("BrowserExecutor iframe stitching", () => { stale.importRefState(state); await snapshotText(stale); staleCdp.emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-INNER", parentId: "FRAME-OOP" } }, sessionId: "session-oop" }); - const [failed] = await stale.execute({ type: "browser_act", steps: [{ type: "click", ref: "e3" }] } as CuaBrowserAction); + const [failed] = await stale.execute({ type: "browser_act", steps: [{ type: "click", ref: "e3" }] } as BrowserAction); expect(failed).toMatchObject({ type: "browser_act", result: { outcome: "didnt", stop_reason: "stale_ref" } }); }); @@ -1835,7 +1843,7 @@ describe("BrowserExecutor iframe stitching", () => { await snapshotText(executor); emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-INNER", parentId: "FRAME-OOP" } }, sessionId: "session-oop" }); - await expect(executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e3" } as BrowserAction)).rejects.toThrow(/stale/); }); it.each([ @@ -1864,7 +1872,7 @@ describe("BrowserExecutor iframe stitching", () => { type: "browser_wait_for", expect: { type: "text", text: "Pay" }, timeout_ms: 20, - } as CuaBrowserAction); + } as BrowserAction); expect(read).toMatchObject({ type: "browser_wait_for", result: { status: "satisfied", evidence: "preexisting" } }); }); @@ -1872,12 +1880,12 @@ describe("BrowserExecutor iframe stitching", () => { const { cdp, sent } = setupOopif(); const executor = new BrowserExecutor(cdp); - const results = await executor.execute({ type: "browser_find", query: "pay button" } as CuaBrowserAction); + const results = await executor.execute({ type: "browser_find", query: "pay button" } as BrowserAction); const text = (results[0] as { text: string }).text; expect(text).toContain('button "Pay" [e'); const ref = /\[(e\d+)\]/.exec(text)![1]!; - await executor.execute({ type: "browser_click", ref } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref } as BrowserAction); const resolved = sent.find((cmd) => cmd.method === "DOM.getBoxModel" && cmd.params.backendNodeId === 70); expect(resolved?.sessionId).toBe("session-oop"); }); @@ -1888,8 +1896,8 @@ describe("BrowserExecutor iframe stitching", () => { await snapshotText(executor); emit({ method: "Page.frameNavigated", params: { frame: { id: "FRAME-OOP" } }, sessionId: "session-oop" }); - await expect(executor.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); - await executor.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await expect(executor.execute({ type: "browser_click", ref: "e3" } as BrowserAction)).rejects.toThrow(/stale/); + await executor.execute({ type: "browser_click", ref: "e1" } as BrowserAction); const text = await snapshotText(executor); expect(text).toContain('button "Pay" [e'); @@ -1907,7 +1915,7 @@ describe("BrowserExecutor iframe stitching", () => { await snapshotText(executor); fake.emit({ method: "Page.frameDetached", params: { frameId: "FRAME-SP", reason: "swap" }, sessionId: "session-1" }); - await expect(executor.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(executor.execute({ type: "browser_click", ref: "e2" } as BrowserAction)).rejects.toThrow(/stale/); fake.setFrameTree("FRAME-SP", [ax({ nodeId: "new", role: "button", name: "New", backendDOMNodeId: 61 })]); expect(await snapshotText(executor)).toContain('button "New" [e'); @@ -2048,7 +2056,7 @@ describe("BrowserExecutor observation fencing", () => { }); const executor = new BrowserExecutor(fake.cdp); const results = await executor.execute( - (type === "browser_snapshot" ? { type } : { type, query: "stable button" }) as CuaBrowserAction, + (type === "browser_snapshot" ? { type } : { type, query: "stable button" }) as BrowserAction, ); const text = (results[0] as { text: string }).text; expect(text).toContain('button "Stable" [e1]'); @@ -2119,7 +2127,7 @@ describe("BrowserExecutor observation fencing", () => { }); const executor = new BrowserExecutor(fake.cdp); await expect( - executor.execute((type === "browser_snapshot" ? { type } : { type, query: "save" }) as CuaBrowserAction), + executor.execute((type === "browser_snapshot" ? { type } : { type, query: "save" }) as BrowserAction), ).rejects.toThrow(/observation changed/i); expect(fake.sent.filter((command) => command.method === "Accessibility.getFullAXTree")).toHaveLength(3); expect(refsOf(executor).size).toBe(0); @@ -2131,7 +2139,7 @@ describe("BrowserExecutor multi-click", () => { const { cdp, sent } = createFakeCdp(BUTTON_TREE); const executor = new BrowserExecutor(cdp); await snapshotText(executor); - for (const num_clicks of [0, 1.5, 4]) await expect(executor.execute({ type: "browser_click", ref: "e1", num_clicks } as CuaBrowserAction)).rejects.toThrow(/integer between 1 and 3/); + for (const num_clicks of [0, 1.5, 4]) await expect(executor.execute({ type: "browser_click", ref: "e1", num_clicks } as BrowserAction)).rejects.toThrow(/integer between 1 and 3/); expect(sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent")).toBe(false); }); @@ -2139,7 +2147,7 @@ describe("BrowserExecutor multi-click", () => { const { cdp, sent } = createFakeCdp(BUTTON_TREE); const executor = new BrowserExecutor(cdp); await snapshotText(executor); - await executor.execute({ type: "browser_click", ref: "e1", num_clicks: 2 } as CuaBrowserAction); + await executor.execute({ type: "browser_click", ref: "e1", num_clicks: 2 } as BrowserAction); const mouse = sent.filter((cmd) => cmd.method === "Input.dispatchMouseEvent").map((cmd) => cmd.params); expect(mouse.map((params) => [params.type, params.clickCount])).toEqual([ @@ -2161,7 +2169,7 @@ describe("BrowserExecutor ref state export/import", () => { const { cdp, sent } = createFakeCdp(BUTTON_TREE); const second = new BrowserExecutor(cdp); second.importRefState(state); - await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e1" } as BrowserAction); const pressed = sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); expect(pressed).toBeDefined(); }); @@ -2183,7 +2191,7 @@ describe("BrowserExecutor ref state export/import", () => { secondFake.setFrameTree("FRAME-SP", [ax({ nodeId: "f1", role: "button", name: "Pay", backendDOMNodeId: 70 })]); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await second.execute({ type: "browser_click", ref: "e2" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e2" } as BrowserAction); expect(secondFake.sent.some((command) => command.method === "DOM.getBoxModel" && command.params.backendNodeId === 70)).toBe(true); }); @@ -2198,7 +2206,7 @@ describe("BrowserExecutor ref state export/import", () => { expect(await snapshotText(second)).toContain('button "Save" [e2]'); emit({ method: "Page.frameNavigated", params: { frame: { id: "F0" } }, sessionId: "session-1" }); - await expect(second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); }); }); @@ -2227,7 +2235,7 @@ describe("BrowserExecutor cross-process document identity", () => { secondFake.setLoaderId("L1"); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); expect(refsOf(second).size).toBe(0); }); @@ -2242,7 +2250,7 @@ describe("BrowserExecutor cross-process document identity", () => { secondFake.setLoaderId("L0"); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e1" } as BrowserAction); const pressed = secondFake.sent.find((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed"); expect(pressed).toBeDefined(); }); @@ -2262,7 +2270,7 @@ describe("BrowserExecutor cross-process document identity", () => { secondFake.setLoaderId("L1"); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_fill", ref: "e1", value: "x" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_fill", ref: "e1", value: "x" } as BrowserAction)).rejects.toThrow(/stale/); expect(refsOf(second).size).toBe(0); expect(secondFake.sent.some((cmd) => cmd.method === "Runtime.callFunctionOn")).toBe(false); }); @@ -2278,7 +2286,7 @@ describe("BrowserExecutor cross-process document identity", () => { secondFake.setLoaderId("L0"); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await second.execute({ type: "browser_fill", ref: "e1", value: "x" } as CuaBrowserAction); + await second.execute({ type: "browser_fill", ref: "e1", value: "x" } as BrowserAction); expect(secondFake.sent.some((cmd) => cmd.method === "Runtime.callFunctionOn")).toBe(true); }); @@ -2293,7 +2301,7 @@ describe("BrowserExecutor cross-process document identity", () => { secondFake.setLoaderId("L1"); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_scroll_to", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_scroll_to", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); expect(refsOf(second).size).toBe(0); expect(secondFake.sent.some((cmd) => cmd.method === "DOM.scrollIntoViewIfNeeded")).toBe(false); }); @@ -2305,7 +2313,7 @@ describe("BrowserExecutor cross-process document identity", () => { await snapshotText(first); const state = first.exportRefState(); - // Middle invocation imports and re-exports without ever attaching (e.g. `cua url`). + // Middle invocation imports and re-exports without ever attaching (e.g. `loop url`). const middleFake = createFakeCdp(BUTTON_TREE); middleFake.setLoaderId("L0"); const middle = new BrowserExecutor(middleFake.cdp); @@ -2318,7 +2326,7 @@ describe("BrowserExecutor cross-process document identity", () => { lastFake.setLoaderId("L1"); const last = new BrowserExecutor(lastFake.cdp); last.importRefState(relayed); - await expect(last.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(last.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); }); // Legacy state predates document identity; generation alone is process-local @@ -2337,7 +2345,7 @@ describe("BrowserExecutor cross-process document identity", () => { const secondFake = createFakeCdp(BUTTON_TREE); const second = new BrowserExecutor(secondFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_click", ref: "e1" } as BrowserAction)).rejects.toThrow(/stale/); expect(refsOf(second).size).toBe(0); expect(secondFake.sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent")).toBe(false); @@ -2401,11 +2409,11 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_click", ref: "e3" } as BrowserAction)).rejects.toThrow(/stale/); // Only the OOPIF's refs were dropped; the main-frame "Top" and parent // Iframe refs survive. expect([...refsOf(second).keys()].sort()).toEqual(["e1", "e2"]); - await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(importFake.sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed")).toBe(true); }); @@ -2415,8 +2423,8 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); - await second.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e1" } as BrowserAction); + await second.execute({ type: "browser_click", ref: "e3" } as BrowserAction); const oopifPress = importFake.sent.find( (cmd) => cmd.method === "DOM.getBoxModel" && cmd.params.backendNodeId === 70 && cmd.sessionId === "session-oop", ); @@ -2431,7 +2439,7 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - await second.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e3" } as BrowserAction); const resolved = importFake.sent.find( (cmd) => cmd.method === "DOM.getBoxModel" && cmd.params.backendNodeId === 70 && cmd.sessionId === "session-oop", ); @@ -2444,8 +2452,8 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); - await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await expect(second.execute({ type: "browser_click", ref: "e3" } as BrowserAction)).rejects.toThrow(/stale/); + await second.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(importFake.sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed")).toBe(true); }); @@ -2457,7 +2465,7 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - const action = (type === "browser_fill" ? { type, ref: "e3", value: "x" } : { type, ref: "e3" }) as CuaBrowserAction; + const action = (type === "browser_fill" ? { type, ref: "e3", value: "x" } : { type, ref: "e3" }) as BrowserAction; await expect(second.execute(action)).rejects.toThrow(/stale/); // The changed OOPIF document stales the ref before any mutation touches it. expect(importFake.sent.some((cmd) => cmd.method === "Runtime.callFunctionOn")).toBe(false); @@ -2483,8 +2491,8 @@ describe("BrowserExecutor cross-process frame document identity", () => { const lastFake = oopifProcess({ main: "M0", oopif: "O1" }); const last = new BrowserExecutor(lastFake.cdp); last.importRefState(relayed); - await expect(last.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); - await last.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await expect(last.execute({ type: "browser_click", ref: "e3" } as BrowserAction)).rejects.toThrow(/stale/); + await last.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(lastFake.sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed")).toBe(true); }); @@ -2494,7 +2502,7 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - await second.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e3" } as BrowserAction); const pageTrees = importFake.sent.filter((cmd) => cmd.method === "Page.getFrameTree" && cmd.sessionId === "session-1"); const oopifTrees = importFake.sent.filter((cmd) => cmd.method === "Page.getFrameTree" && cmd.sessionId === "session-oop"); expect(pageTrees).toHaveLength(1); @@ -2544,9 +2552,9 @@ describe("BrowserExecutor cross-process frame document identity", () => { const second = new BrowserExecutor(importFake.cdp); second.importRefState(state); - await expect(second.execute({ type: "browser_click", ref: payA } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_click", ref: payA } as BrowserAction)).rejects.toThrow(/stale/); // The sibling frame's ref still resolves against its unchanged document. - await second.execute({ type: "browser_click", ref: payB } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: payB } as BrowserAction); expect(importFake.sent.some((cmd) => cmd.method === "DOM.getBoxModel" && cmd.params.backendNodeId === 61)).toBe(true); }); @@ -2560,9 +2568,9 @@ describe("BrowserExecutor cross-process frame document identity", () => { second.importRefState(state); // Unverifiable ⇒ stale, even though the live OOPIF document is unchanged. - await expect(second.execute({ type: "browser_click", ref: "e3" } as CuaBrowserAction)).rejects.toThrow(/stale/); + await expect(second.execute({ type: "browser_click", ref: "e3" } as BrowserAction)).rejects.toThrow(/stale/); // The frame that *does* carry identity is preserved. - await second.execute({ type: "browser_click", ref: "e1" } as CuaBrowserAction); + await second.execute({ type: "browser_click", ref: "e1" } as BrowserAction); expect(importFake.sent.some((cmd) => cmd.method === "Input.dispatchMouseEvent" && cmd.params.type === "mousePressed")).toBe(true); }); diff --git a/packages/agent/test/translator.test.ts b/packages/loop/test/translator.test.ts similarity index 95% rename from packages/agent/test/translator.test.ts rename to packages/loop/test/translator.test.ts index 4ed9f2af..bc64f482 100644 --- a/packages/agent/test/translator.test.ts +++ b/packages/loop/test/translator.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import type Kernel from "@onkernel/sdk"; -import type { CuaBrowserAction } from "@onkernel/cua-ai"; -import type { BrowserExecutor } from "../src/translator/browser"; -import { InternalComputerTranslator, type KernelBrowser } from "../src/translator/translator"; +import type { BrowserAction } from "../src/index"; +import type { BrowserExecutor } from "../src/core/translator/browser"; +import { InternalComputerTranslator, type KernelBrowser } from "../src/core/translator/translator"; const browser = { session_id: "browser_123" } as KernelBrowser; @@ -95,9 +95,9 @@ describe("InternalComputerTranslator", () => { it("stops a batch when a semantic browser wait does not satisfy", async () => { const { client } = createClient(); - const calls: CuaBrowserAction[] = []; + const calls: BrowserAction[] = []; const browserExecutor = { - execute: async (action: CuaBrowserAction) => { + execute: async (action: BrowserAction) => { calls.push(action); if (action.type !== "browser_wait_for") return []; return [{ diff --git a/packages/loop/tsconfig.build.json b/packages/loop/tsconfig.build.json new file mode 100644 index 00000000..da5abfed --- /dev/null +++ b/packages/loop/tsconfig.build.json @@ -0,0 +1,20 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "outDir": "./dist-tsc", + "rootDir": "./src", + "emitDeclarationOnly": true, + "sourceMap": false, + "declarationMap": false, + "baseUrl": ".", + // The pi extension imports this package by name rather than by relative + // path; see the note in src/pi-extension/index.ts. Point those specifiers + // at source so a typecheck does not require a build first. + "paths": { + "@onkernel/loop": ["./src/index.ts"], + "@onkernel/loop/pi": ["./src/pi/index.ts"] + } + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist", "**/*.d.ts", "src/**/*.d.ts"] +} diff --git a/packages/agent/tsconfig.json b/packages/loop/tsconfig.json similarity index 100% rename from packages/agent/tsconfig.json rename to packages/loop/tsconfig.json diff --git a/packages/agent/tsdown.config.ts b/packages/loop/tsdown.config.ts similarity index 82% rename from packages/agent/tsdown.config.ts rename to packages/loop/tsdown.config.ts index f377486b..3fca3ac6 100644 --- a/packages/agent/tsdown.config.ts +++ b/packages/loop/tsdown.config.ts @@ -1,7 +1,7 @@ import { defineConfig } from "tsdown"; export default defineConfig({ - entry: ["src/index.ts"], + entry: ["src/index.ts", "src/pi/index.ts"], format: ["esm"], platform: "node", dts: true, diff --git a/packages/ai/vitest.config.ts b/packages/loop/vitest.config.ts similarity index 50% rename from packages/ai/vitest.config.ts rename to packages/loop/vitest.config.ts index 25582a14..736a603f 100644 --- a/packages/ai/vitest.config.ts +++ b/packages/loop/vitest.config.ts @@ -1,12 +1,25 @@ +import { fileURLToPath } from "node:url"; import { configDefaults, defineConfig } from "vitest/config"; export default defineConfig({ server: { host: "127.0.0.1", }, + resolve: { + // The pi extension imports this package by name, because pi loads it as + // TypeScript through jiti and jiti's pi-ai alias cannot follow the deep + // `@earendil-works/pi-ai/api/*` imports the provider adapters make. Unit + // tests point that name back at source so they do not need a build. + alias: { + "@onkernel/loop/pi": fileURLToPath(new URL("./src/pi/index.ts", import.meta.url)), + "@onkernel/loop": fileURLToPath(new URL("./src/index.ts", import.meta.url)), + }, + }, test: { globals: true, environment: "node", + // The pi print/RPC test spawns a real pi process and waits on a fake + // provider, which is slower than a unit test but still bounded. testTimeout: 30000, // Unit runs cover every test file except the opt-in suites; use // vitest.integration.config.ts to run those. diff --git a/packages/loop/vitest.integration.config.ts b/packages/loop/vitest.integration.config.ts new file mode 100644 index 00000000..861d004c --- /dev/null +++ b/packages/loop/vitest.integration.config.ts @@ -0,0 +1,21 @@ +import { fileURLToPath } from "node:url"; +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + server: { + host: "127.0.0.1", + }, + resolve: { + // Same source alias the unit config sets; see the note there. + alias: { + "@onkernel/loop/pi": fileURLToPath(new URL("./src/pi/index.ts", import.meta.url)), + "@onkernel/loop": fileURLToPath(new URL("./src/index.ts", import.meta.url)), + }, + }, + test: { + globals: true, + environment: "node", + testTimeout: 30000, + include: ["test/**/*.integration.test.ts", "test/**/*.live.test.ts"], + }, +}); diff --git a/packages/pi-extension/CHANGELOG.md b/packages/pi-extension/CHANGELOG.md deleted file mode 100644 index e9e428eb..00000000 --- a/packages/pi-extension/CHANGELOG.md +++ /dev/null @@ -1,48 +0,0 @@ -# Changelog - -## Unreleased - -- Flags name the domain rather than an acronym: `--cua-tools` is now - `--browser-tools`, `--cua-coordinates` is `--browser-coordinates`, and the - commands are `/browser` and `/browser-tools`. -- Browser configuration is one `--browser-options` JSON object forwarded verbatim - to Kernel's browser-create call, replacing `--cua-profile-id`, - `--cua-profile-save-changes`, `--cua-proxy-id`, and `--cua-browser-timeout`. A - flag per create-call field grows every time the SDK does; JSON tracks it for - free. The only default is `timeout_seconds: 600`. `--browser-session` still - attaches an existing browser and cannot be combined with `--browser-options`. - Note that `stealth` is no longer forced on — pass it in the JSON if you want it. - -- `@onkernel/cua-cli` and the `cua` binary are removed. Everything the CLI built - because it needed an agent front-end — sessions and resume, skills, the TUI, - print and RPC modes, model selection — pi supplies, so the extension replaces - it rather than reimplementing it. The `cua act` model-free executor path and - the `--print -o jsonl` telemetry schema are gone with it. -- Add `@onkernel/cua-pi-extension`, an installable pi extension that contributes - Kernel browser tools to pi's own agent session. The menu is eight entries, one - per capability: `browser` and `computer` (primitives plus their batch form), - `browser-act`, `playwright`, and the four provider-native surfaces — - `anthropic-computer`, `anthropic-browser`, `openai-computer`, `google-browser`. - Packaging variants are deliberately absent: `mixed`, the batch tools on their - own, and the 37 individual tool names offered nothing the eight entries do not. -- A deactivated selection now reports itself on stderr in print and RPC modes, - once per distinct reason. Previously the reason reached only the TUI status - line, so a scripted run lost its tools silently, created no browser, and let - the model answer from memory with exit 0. -- `/cua-tools` decides each entry's availability by compiling it on its own, and - reports pairwise conflicts separately. It previously passed the current - selection to the tool menu, whose verdicts are relative to that selection, so a - selection that failed to compile marked every entry unavailable with its - error — including entries that then activated fine. -- Provider-native surfaces work because the extension owns the stream for the - providers it registers, swapping pi's registry model for the compiled catalog's - model — which carries the transport the selected tools derive — and passing the - incoming native-call plan. Without that, `requiresApi` never takes effect and - native calls arrive unnormalized. -- A selection is validated by compiling it for the active model, so an - incompatible tool deactivates with the catalog compiler's own reason instead of - failing at request time. `/cua-tools` with no argument lists every selector for - the current model with those reasons. -- One browser is provisioned lazily per session on first tool execution and - deleted on shutdown if this session created it. Declaration compilation, header - generation, and payload transforms never provision a browser. diff --git a/packages/pi-extension/README.md b/packages/pi-extension/README.md deleted file mode 100644 index 1cd6c396..00000000 --- a/packages/pi-extension/README.md +++ /dev/null @@ -1,115 +0,0 @@ -# `@onkernel/cua-pi-extension` - -An installable [pi](https://pi.dev) extension that adds Kernel browser tools to -pi's existing agent session. pi owns the agent loop, session, and UI; this -extension contributes the tools, the browser they run against, and the provider -wiring that provider-native surfaces need. - -It does not start a second model loop, and it adds no implicit screenshots or -prompt instructions. - -## Install - -```sh -pi install ./packages/pi-extension -# or, once published -pi install npm:@onkernel/cua-pi-extension -``` - -`KERNEL_API_KEY` is required when a tool first executes, not at startup. -`KERNEL_BASE_URL` is honored. Neither is written to session entries or output. - -## Use - -No selector means no Kernel tool is active and no browser is provisioned. - -```sh -pi -p --provider openai --model gpt-5.6-sol \ - --browser-tools browser,browser-act "Open example.com and report its heading" - -pi --mode rpc --no-session --provider openai --model gpt-5.6-sol --browser-tools browser - -pi -p --provider anthropic --model claude-opus-5 --browser-tools anthropic-computer \ - "Open example.com and report its heading" -``` - -### The menu - -Eight entries, one per capability. Availability is per model, and `/browser-tools` -tells you which apply to the one you selected. - -| entry | tools | works on | -| --- | --- | --- | -| `browser` | CDP browser primitives plus the one-call `browser_batch` form | every provider | -| `computer` | canonical computer primitives plus `computer_batch` | every provider | -| `browser-act` | `browser_act`, the verified-plan tool | every provider except Moonshot, which rejects its schema size | -| `playwright` | `playwright_execute` | every provider | -| `anthropic-computer` | Anthropic's native computer tool | Anthropic only | -| `anthropic-browser` | Anthropic's native browser tool | Anthropic only | -| `openai-computer` | OpenAI's native computer tool | OpenAI only | -| `google-browser` | Google's predefined browser action set | Google only | - -`anthropic-browser` and `anthropic-computer` cannot be selected together: -Anthropic rejects the pair because the browser tool addresses a viewport -coordinate frame and the computer tool a display frame. The catalog compiler -refuses it before the request goes out, and `/browser-tools` reports it as a conflict -rather than as unavailability. - -`--browser-coordinates` selects `pixels` (default) or `normalized-1000` for the -`computer` entry's coordinate contract. - -### Commands - -- `/browser` — current selectors, active tools, and browser status. -- `/browser-tools` — with no argument, list every selector for the current model, - marking the selected ones and showing the compiler's own reason for any that - this model cannot take. With an argument, replace the selection. `none` clears - it. - -A selection is checked by compiling it, so a model that cannot take a tool -deactivates it with a reason rather than failing at request time. Switching -models re-checks, and restores a previously forced-off selection when the new -model can take it. - -In TUI mode the reason appears in the status line. In print and RPC modes there -is no status line, so the reason is written to **stderr** — once per distinct -reason. Without that, a deactivated selection is invisible: the tools are gone, -no browser is created, and the model answers from memory with exit 0. - -### Browser - -| flag | effect | -| --- | --- | -| `--browser-session` | attach an existing session; never deleted on exit | -| `--browser-options` | JSON forwarded verbatim to Kernel's browser-create call | - -`--browser-options` is one JSON object rather than a flag per field, so it tracks -the Kernel SDK without this extension growing an option every time the SDK does: - -```sh -pi -p --browser-tools browser \ - --browser-options '{"stealth":true,"profile":{"id":"p1","save_changes":true},"proxy_id":"px1"}' \ - "open example.com" -``` - -The only default is `timeout_seconds: 600` — the failure it prevents is a browser -vanishing mid-task. Override it in the same JSON. `--browser-session` attaches an -existing browser, so it cannot be combined with `--browser-options`. - -One browser is provisioned lazily per session, on first tool execution. -Compiling declarations, generating headers, and transforming a payload never -provision one. An owned browser is deleted on session shutdown. - -## Development - -```bash -npm run typecheck --workspace @onkernel/cua-pi-extension -npm test --workspace @onkernel/cua-pi-extension -``` - -The test suite includes an end-to-end run that spawns real `pi` in print and RPC -modes against a fake provider and Kernel server. - -## License - -MIT diff --git a/packages/pi-extension/package.json b/packages/pi-extension/package.json deleted file mode 100644 index 9ecb18f0..00000000 --- a/packages/pi-extension/package.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "name": "@onkernel/cua-pi-extension", - "version": "0.10.0", - "description": "Kernel browser tools for pi", - "license": "MIT", - "type": "module", - "repository": { - "type": "git", - "url": "git+https://github.com/kernel/cua.git", - "directory": "packages/pi-extension" - }, - "homepage": "https://github.com/kernel/cua/tree/main/packages/pi-extension#readme", - "keywords": [ - "pi-package", - "pi-extension", - "computer-use", - "kernel" - ], - "pi": { - "extensions": [ - "./src/index.ts" - ] - }, - "files": [ - "src", - "README.md", - "CHANGELOG.md", - "package.json" - ], - "publishConfig": { - "access": "public" - }, - "engines": { - "node": ">=22.19.0" - }, - "scripts": { - "build": "tsc -b", - "typecheck": "tsc -b", - "clean": "tsc -b --clean && rm -rf dist-tsc", - "test": "vitest --run" - }, - "dependencies": { - "@onkernel/cua-agent": "0.10.0", - "@onkernel/cua-ai": "0.10.0", - "@onkernel/sdk": "0.49.0" - }, - "peerDependencies": { - "@earendil-works/pi-agent-core": "*", - "@earendil-works/pi-ai": "*", - "@earendil-works/pi-coding-agent": "*" - }, - "devDependencies": { - "@earendil-works/pi-agent-core": "0.83.0", - "@earendil-works/pi-ai": "0.83.0", - "@earendil-works/pi-coding-agent": "0.83.0", - "vitest": "^3.2.4" - } -} diff --git a/packages/pi-extension/tsconfig.build.json b/packages/pi-extension/tsconfig.build.json deleted file mode 100644 index a6715524..00000000 --- a/packages/pi-extension/tsconfig.build.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "extends": "../../tsconfig.base.json", - "compilerOptions": { - "outDir": "./dist-tsc", - "rootDir": "./src", - "emitDeclarationOnly": true, - "sourceMap": false, - "declarationMap": false - }, - "include": [ - "src/**/*.ts" - ], - "exclude": [ - "node_modules", - "dist", - "**/*.d.ts", - "src/**/*.d.ts" - ], - "references": [ - { "path": "../ai" }, - { "path": "../agent" } - ] -} diff --git a/packages/pi-extension/tsconfig.json b/packages/pi-extension/tsconfig.json deleted file mode 100644 index d8faaf50..00000000 --- a/packages/pi-extension/tsconfig.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "extends": "./tsconfig.build.json" -} diff --git a/packages/pi-extension/vitest.config.ts b/packages/pi-extension/vitest.config.ts deleted file mode 100644 index fac3fe58..00000000 --- a/packages/pi-extension/vitest.config.ts +++ /dev/null @@ -1,15 +0,0 @@ -import { defineConfig } from "vitest/config"; - -export default defineConfig({ - // This environment does not resolve "localhost". - server: { - host: "127.0.0.1", - }, - test: { - globals: true, - environment: "node", - // The pi print/RPC test spawns a real pi process and waits on a fake - // provider, which is slower than a unit test but still bounded. - testTimeout: 30000, - }, -}); diff --git a/tsconfig.json b/tsconfig.json index 311212bb..34b327be 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -1,9 +1,7 @@ { "files": [], "references": [ - { "path": "./packages/ai" }, - { "path": "./packages/agent" }, - { "path": "./packages/ptywright" }, - { "path": "./packages/pi-extension" } + { "path": "./packages/loop" }, + { "path": "./packages/ptywright" } ] } From 986ea0fbd37fe697de49ac3ceb823e58378bc2d6 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Fri, 14 Aug 2026 20:33:51 +0000 Subject: [PATCH 2/6] Point the probe and the docs skill at the merged package The native action probe still looked for its fixture under packages/ai, so a real xai probe failed from the repo root, and the docs skill still described the three-package layout. --- .agents/skills/update-docs/SKILL.md | 12 ++++++------ docs/agent-tool-configuration-spec.md | 3 ++- packages/loop/scripts/native-action-probe.ts | 10 +++++----- 3 files changed, 13 insertions(+), 12 deletions(-) diff --git a/.agents/skills/update-docs/SKILL.md b/.agents/skills/update-docs/SKILL.md index 3c33f1f6..db193f19 100644 --- a/.agents/skills/update-docs/SKILL.md +++ b/.agents/skills/update-docs/SKILL.md @@ -40,18 +40,18 @@ For every doc update: Start with these source-of-truth checks: -- Package topology: `package.json`, `tsconfig.json`, and `packages/*/package.json`. -- Design invariants: `@onkernel/cua-ai` owns provider-specific policy (catalog, tool schemas, payload transforms); `@onkernel/cua-agent` is provider-neutral runtime glue around `pi-agent-core` (no provider names in `packages/agent/src`); provider differences reach the agent as compiled `CuaToolCatalog` data; `@onkernel/cua-pi-extension` composes both inside a pi session. -- Model layer: `packages/ai/src/index.ts`, `cua.ts`, `tool-catalog.ts`, `getCuaModel`/`listCuaModels`/`parseCuaModelRef`, provider adapters, and `api-keys.ts`. -- Execution layer: `packages/agent/src/index.ts`, `attach.ts`, `tool-manager.ts`, `resources.ts`, and the canonical CUA tool executors against `@onkernel/sdk`. -- Extension runtime flow: `packages/pi-extension/src/index.ts`, `selection.ts`, `browser-runtime.ts`, `state.ts`, and `render.ts`. +- Package topology: `package.json`, `tsconfig.json`, and `packages/*/package.json`. `@onkernel/loop` is one package with two entry points (`.` and `./pi`) and a `pi.extensions` field. +- Design invariants: `packages/loop/src/core` is the framework-neutral core (canonical actions, tool declarations, catalog compilation, tool menu, tool manager, Kernel-browser execution); `packages/loop/src/pi` owns provider-specific policy (model resolution, transport derivation, adapters, payload transforms, headers, retry); provider differences reach execution as compiled `LoopToolCatalog` data rather than provider conditionals in the translator. +- Neutral core: `packages/loop/src/index.ts`, `core/tools.ts`, `core/actions/`, `core/tool-catalog.ts`, `core/menu.ts`, `core/tool-manager.ts`, `core/resources.ts`, and `core/translator/`. +- pi binding: `packages/loop/src/pi/index.ts`, `pi/attach.ts`, `pi/models.ts` (`getLoopModel`/`listLoopModels`/`parseLoopModelRef`), `pi/providers/`, `pi/provider-retry.ts`, and `pi/api-keys.ts`. +- Extension runtime flow: `packages/loop/src/pi-extension/index.ts`, `selection.ts`, `browser-runtime.ts`, `state.ts`, and `render.ts`. - TUI test infrastructure: `packages/ptywright/package.json`, `src/index.ts`, `src/session.ts`, `src/terminal.ts`, and `README.md`. - External drift: provider computer-use docs, `@earendil-works/pi-*` versions, and `@onkernel/sdk` versions in package manifests. Questions `architecture.md` should answer after each update: - What owns the canonical action vocabulary and the model catalog? -- Where is the cua-ai vs cua-agent ownership boundary, and how do provider differences reach the agent without provider conditionals in `packages/agent/src`? +- Where is the `src/core` vs `src/pi` boundary, and how do provider differences reach execution without provider conditionals in the translator? - Where does Kernel SDK browser execution happen? - What does the pi extension compose at runtime, and which selectors does it offer? - Which package is dev/test infrastructure only? diff --git a/docs/agent-tool-configuration-spec.md b/docs/agent-tool-configuration-spec.md index 27ff0f86..b8e9e34c 100644 --- a/docs/agent-tool-configuration-spec.md +++ b/docs/agent-tool-configuration-spec.md @@ -7,7 +7,8 @@ rule it establishes still holds; what changed is where the array lives. `CuaAgen and its in-tool `executionMode: "sequential"` guard described below no longer exist. Retained as the record of why the tool array is explicit and required. -**Scope:** `@onkernel/loop` (written when this code lived in two packages) +**Scope:** `@onkernel/loop` (written when this code lived in two packages) + **Compatibility:** Not a goal; these packages are alpha and may make breaking API changes. ## Summary diff --git a/packages/loop/scripts/native-action-probe.ts b/packages/loop/scripts/native-action-probe.ts index 343d4282..7fdbe37f 100644 --- a/packages/loop/scripts/native-action-probe.ts +++ b/packages/loop/scripts/native-action-probe.ts @@ -71,9 +71,9 @@ function parseArgs(argv: string[]): Args { function usage(): never { console.log(`Usage: - npx tsx packages/ai/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json - npx tsx packages/ai/scripts/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 - npx tsx packages/ai/scripts/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 + npx tsx packages/loop/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json + npx tsx packages/loop/scripts/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 + npx tsx packages/loop/scripts/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 `); process.exit(0); } @@ -246,9 +246,9 @@ async function probeXai(model: string, prompt: ProbePrompt): Promise { const path = [ join(process.cwd(), "examples", "screenshot.png"), - join(process.cwd(), "packages", "ai", "examples", "screenshot.png"), + join(process.cwd(), "packages", "loop", "examples", "screenshot.png"), ].find(existsSync); - if (!path) throw new Error("could not find packages/ai/examples/screenshot.png"); + if (!path) throw new Error("could not find packages/loop/examples/screenshot.png"); return readFile(path); } From cc6f90bddd80b8ac8e7aa4d1110dc70ddc7cf5a1 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:54:00 +0000 Subject: [PATCH 3/6] Fix multi-turn replay and per-action screenshots on OpenAI's native computer path --- packages/loop/src/core/tools.ts | 4 ++++ .../loop/src/pi/providers/openai/provider.ts | 5 ++++- .../loop/test/openai-native-provider.test.ts | 17 +++++++++++++++++ packages/loop/test/resources.test.ts | 10 ++++++++++ 4 files changed, 35 insertions(+), 1 deletion(-) diff --git a/packages/loop/src/core/tools.ts b/packages/loop/src/core/tools.ts index 3a2a5c54..25a7cb30 100644 --- a/packages/loop/src/core/tools.ts +++ b/packages/loop/src/core/tools.ts @@ -652,6 +652,10 @@ function mapOpenAIComputerInput(input: unknown): ComputerUseAction[] { default: throw new Error(`unsupported OpenAI computer action "${type}"`); } } + // OpenAI's computer-use loop expects a screenshot back from every + // `computer_call`: without one the adapter has to send a placeholder, leaving + // the model blind after each action and re-screenshotting to recover. + if (result.length > 0 && result[result.length - 1]!.type !== "screenshot") result.push({ type: "screenshot" }); return result; } diff --git a/packages/loop/src/pi/providers/openai/provider.ts b/packages/loop/src/pi/providers/openai/provider.ts index 3a107007..70ec88fe 100644 --- a/packages/loop/src/pi/providers/openai/provider.ts +++ b/packages/loop/src/pi/providers/openai/provider.ts @@ -402,7 +402,10 @@ function convertMessages(messages: readonly Context["messages"][number][], nativ } if (message.role === "assistant") { const text = message.content.filter((part): part is TextContent => part.type === "text").map((part) => part.text).join("\n"); - if (text) input.push({ role: "assistant", content: [{ type: "input_text", text }] }); + // Responses accepts only `output_text` (or `refusal`) on an assistant + // input item; `input_text` is rejected outright, so replaying a prior + // assistant reply this way 400s every request after the first turn. + if (text) input.push({ role: "assistant", content: [{ type: "output_text", text }] }); for (const part of message.content) { if (part.type !== "toolCall") continue; if (part.name === nativeName) { diff --git a/packages/loop/test/openai-native-provider.test.ts b/packages/loop/test/openai-native-provider.test.ts index dd3dab15..533725c0 100644 --- a/packages/loop/test/openai-native-provider.test.ts +++ b/packages/loop/test/openai-native-provider.test.ts @@ -203,4 +203,21 @@ describe("computer_call_output serialization", () => { expect(sent).toContain("data:image/png;base64,aW1n"); expect(sent).not.toContain("produced no screenshot"); }); + + it("replays assistant text as output_text", async () => { + responsesCreate.mockReturnValueOnce({ id: "resp_turn2", usage: {}, output: [] }); + await openai.streamOpenAIComputerUse(nativeModel, { + messages: [ + { role: "user", content: "say hi" }, + { role: "assistant", content: [{ type: "text", text: "hi" }], stopReason: "stop" }, + { role: "user", content: "say hi again" }, + ] as never, + tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], + }, { apiKey: "test", loopIncomingToolPlan: incoming }).result(); + + // Responses rejects `input_text` on an assistant item, so replaying it that + // way answers 400 on every turn after the first. + const payload = responsesCreate.mock.calls.at(-1)?.[0] as { input: Array> }; + expect(payload.input).toContainEqual({ role: "assistant", content: [{ type: "output_text", text: "hi" }] }); + }); }); diff --git a/packages/loop/test/resources.test.ts b/packages/loop/test/resources.test.ts index 91aa4920..a43b79d9 100644 --- a/packages/loop/test/resources.test.ts +++ b/packages/loop/test/resources.test.ts @@ -210,6 +210,16 @@ describe("LoopExecutionResources results and batch boundaries", () => { expect(captureScreenshot).not.toHaveBeenCalled(); }); + it("returns a screenshot for every OpenAI native computer action", async () => { + const { resources, captureScreenshot } = setup(); + const result = await resources.materialize(loop.providers.openai.tools.computer()) + .execute("computer", { action: { type: "click", x: 10, y: 20 } }); + // OpenAI's computer-use loop expects one screenshot back per computer_call; + // without it the model is blind after each action. + expect(result.content.some((block) => block.type === "image")).toBe(true); + expect(captureScreenshot).toHaveBeenCalledTimes(1); + }); + it("materializes each spec exactly once per resource pool", () => { const { resources } = setup(); const spec = loop.tools.browser.snapshot(); From ca9cca05b4d29918265318d8701181a7ba78033f Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:54:00 +0000 Subject: [PATCH 4/6] Gate native surfaces per model, narrow Gemini schemas on the SDK payload shape, and widen the Moonshot quirk --- packages/loop/src/core/tool-catalog.ts | 53 ++++++++++++++++--------- packages/loop/src/pi/models.ts | 12 ++++-- packages/loop/test/tool-catalog.test.ts | 28 +++++++++++++ 3 files changed, 72 insertions(+), 21 deletions(-) diff --git a/packages/loop/src/core/tool-catalog.ts b/packages/loop/src/core/tool-catalog.ts index 5bbecab1..261476f4 100644 --- a/packages/loop/src/core/tool-catalog.ts +++ b/packages/loop/src/core/tool-catalog.ts @@ -1,7 +1,7 @@ import type { Api, Model, Tool } from "@earendil-works/pi-ai"; import type { ComputerUseAction } from "./actions/index"; -import type { LoopModelRef } from "../pi/models"; -import { loopModelCapabilities, getLoopModel } from "../pi/models"; +import type { ComputerUseNativeSurface, LoopModelRef } from "../pi/models"; +import { computerUseNativeSurfaces, loopModelCapabilities, getLoopModel } from "../pi/models"; import { anthropicAdaptiveThinkingOnPayload } from "../pi/providers/anthropic/adaptive-thinking"; import { supportsAnthropicNativeBrowser, @@ -397,6 +397,7 @@ function validateToolCompatibility(model: Model, entry: LoopCatalogEntryDra throw new Error(`provider ${model.provider} does not accept the schema size of "${entry.name}" (${entry.identity})`); } if (binding?.kind === "anthropic-native") validateAnthropicNativeModel(model, entry.identity); + else if (binding) validateNativeSurfaceModel(model, entry.identity, binding.kind === "google-native" ? "browser" : "computer"); } function validateAnthropicNativeModel(model: Model, identity: string): void { @@ -409,6 +410,16 @@ function validateAnthropicNativeModel(model: Model, identity: string): void } } +// A provider enables its native surface per model, not per provider: OpenAI's +// computer tool and Google's `computer_use` both answer 400 on a model the +// surface is not enabled for. COMPUTER_USE_NATIVE_SURFACES is what the menu +// reads, so gate compilation on it too rather than letting the request fail on +// the wire. +function validateNativeSurfaceModel(model: Model, identity: string, surface: ComputerUseNativeSurface): void { + if (computerUseNativeSurfaces(model).includes(surface)) return; + throw new Error(`${identity} does not support model "${model.id}": ${model.provider} does not offer a native ${surface} surface for it`); +} + /** Validate the selected native tools agree on a provider and a transport, and return the transport they require, if any. */ function validateToolsetCompatibility(model: Model, entries: readonly LoopCatalogEntryDraft[]): Api | undefined { const nativeProviderKinds = new Set( @@ -577,26 +588,32 @@ function createGeminiSchemaTransform(): LoopPayloadTransform { writes: ["tools.functionDeclarations"], phase: "tool-declarations", apply(payload) { - if (!isRecord(payload) || !Array.isArray(payload.tools)) return payload; - // Google serializes function tools two ways: the Generative Language API - // nests them under `functionDeclarations`, while the Interactions transport - // emits flat `{ type: "function", parameters }` entries. Narrow both, because - // selecting a native surface alongside a function tool derives the second - // shape and a shape-specific transform would silently skip it. - return { - ...payload, - tools: payload.tools.map((tool) => { - if (!isRecord(tool)) return tool; - if (Array.isArray(tool.functionDeclarations)) { - return { ...tool, functionDeclarations: tool.functionDeclarations.map(narrowToGeminiSchema) }; - } - return "parameters" in tool ? { ...tool, parameters: narrowToGeminiSchema(tool.parameters) } : tool; - }), - }; + if (!isRecord(payload)) return payload; + // Google serializes function tools three ways: the Generative Language API + // nests them under `config.tools[].functionDeclarations`, its raw request + // shape puts the same list at the top level, and the Interactions transport + // emits flat `{ type: "function", parameters }` entries. Narrow all of them, + // because selecting a native surface alongside a function tool derives the + // last shape and a shape-specific transform would silently skip the others. + if (Array.isArray(payload.tools)) return { ...payload, tools: narrowGeminiTools(payload.tools) }; + if (isRecord(payload.config) && Array.isArray(payload.config.tools)) { + return { ...payload, config: { ...payload.config, tools: narrowGeminiTools(payload.config.tools) } }; + } + return payload; }, }; } +function narrowGeminiTools(tools: readonly unknown[]): unknown[] { + return tools.map((tool) => { + if (!isRecord(tool)) return tool; + if (Array.isArray(tool.functionDeclarations)) { + return { ...tool, functionDeclarations: tool.functionDeclarations.map(narrowToGeminiSchema) }; + } + return "parameters" in tool ? { ...tool, parameters: narrowToGeminiSchema(tool.parameters) } : tool; + }); +} + const GEMINI_UNSUPPORTED_SCHEMA_KEYWORDS = new Set(["additionalProperties", "$schema", "$defs", "definitions"]); function narrowToGeminiSchema(node: unknown): unknown { diff --git a/packages/loop/src/pi/models.ts b/packages/loop/src/pi/models.ts index 2876c9af..a60cdee5 100644 --- a/packages/loop/src/pi/models.ts +++ b/packages/loop/src/pi/models.ts @@ -82,6 +82,7 @@ export const COMPUTER_USE_NATIVE_SURFACES: readonly { { provider: "google", match: { kind: "exact", id: "gemini-3.6-flash" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, { provider: "google", match: { kind: "exact", id: "gemini-3.5-flash" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, { provider: "google", match: { kind: "exact", id: "gemini-3.5-flash-lite" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, + { provider: "google", match: { kind: "exact", id: "gemini-2.5-computer-use-preview-10-2025" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, ]; /** @@ -91,17 +92,22 @@ export const COMPUTER_USE_NATIVE_SURFACES: readonly { * observed against the live API. */ export const LOOP_MODEL_QUIRKS: readonly LoopModelQuirk[] = [ + { + provider: "moonshotai", + capabilities: { acceptsLargeSchemas: false }, + reason: "Moonshot answers 400 \"schema exceeds maximum allowed size\" once browser_act's schema is attached; observed on kimi-k2.5 and kimi-k3.", + }, { provider: "moonshotai", match: { kind: "exact", id: "kimi-k3" }, - capabilities: { acceptsLargeSchemas: false, serializesStateMutations: true }, - reason: "Kimi K3 rejects the request outright once browser_act's schema is attached.", + capabilities: { serializesStateMutations: true }, + reason: "Kimi K3 serializes state mutations.", }, { provider: "openrouter", match: { kind: "exact", id: "moonshotai/kimi-k3" }, capabilities: { acceptsLargeSchemas: false, serializesStateMutations: true }, - reason: "Same Kimi K3 limit, reached through OpenRouter.", + reason: "Same Kimi K3 limits, reached through OpenRouter.", }, { provider: "openrouter", diff --git a/packages/loop/test/tool-catalog.test.ts b/packages/loop/test/tool-catalog.test.ts index b19449d4..6b464679 100644 --- a/packages/loop/test/tool-catalog.test.ts +++ b/packages/loop/test/tool-catalog.test.ts @@ -188,6 +188,16 @@ describe("compileLoopToolCatalog", () => { expect(() => compile("openai:gpt-5.5", [loop.providers.anthropic.tools.computer()])).toThrow(/requires a anthropic model/); }); + it("gates OpenAI and Google native surfaces on the model, not just the provider", () => { + expect(() => compile("openai:gpt-5.5", [loop.providers.openai.tools.computer()])).not.toThrow(); + expect(() => compile("google:gemini-3.6-flash", loop.providers.google.toolsets.browser())).not.toThrow(); + // Both providers answer 400 for a model the surface is not enabled for. + expect(() => compile("openai:gpt-4.1", [loop.providers.openai.tools.computer()])) + .toThrow(/does not offer a native computer surface/); + expect(() => compile("google:gemini-2.5-flash", loop.providers.google.toolsets.browser())) + .toThrow(/does not offer a native browser surface/); + }); + it("replaces only the selected OpenAI identity placeholder", async () => { const catalog = compile("openai:gpt-5.5", [ loop.providers.openai.tools.computer(), @@ -402,6 +412,24 @@ describe("Gemini function-declaration schema", () => { expect(sent).toContain('"enum":["text"]'); }); + it("narrows the nested config.tools shape the Generative Language SDK builds", async () => { + // pi-ai hands `@google/genai` params to onPayload, which carry the function + // declarations under `config.tools` rather than at the top level. + const catalog = compileLoopToolCatalog({ + model: getLoopModel("google:gemini-3.6-flash"), + requestedTools: [loop.tools.browser.waitFor()], + }); + const params = { + model: "gemini-3.6-flash", + config: { tools: [{ functionDeclarations: catalog.toolDeclarations.map((tool) => ({ name: tool.name, parameters: tool.parameters })) }] }, + }; + const sent = JSON.stringify(await catalog.payload.apply(params, catalog.model)); + expect(JSON.stringify(params)).toContain('"const"'); + expect(sent).not.toContain('"const"'); + expect(sent).not.toContain('"additionalProperties"'); + expect(sent).toContain('"enum":["text"]'); + }); + it("narrows the flat tool shape the Interactions transport emits", async () => { // Selecting a native surface alongside a function tool derives the Interactions // transport, which serializes tools flat instead of under functionDeclarations. From d2b41710c208b794446e01f78cdb42c01928ebac Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:54:00 +0000 Subject: [PATCH 5/6] Run the handle's onPayload alongside pi's own --- packages/loop/src/pi/attach.ts | 11 ++++++++--- packages/loop/test/attach.test.ts | 30 ++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/packages/loop/src/pi/attach.ts b/packages/loop/src/pi/attach.ts index 1a9fc598..59c3e462 100644 --- a/packages/loop/src/pi/attach.ts +++ b/packages/loop/src/pi/attach.ts @@ -279,15 +279,20 @@ export function withCatalogModels( ); const optionsFor = (options: T): T => { const catalog = liveManager().catalog; - const callerOnPayload = options?.onPayload ?? handleOnPayload; + // The handle's hook and a per-request hook both run: pi's harness always + // sets its own `onPayload` in stream options, so preferring one would drop + // the handle's hook on every harness request. + const requestOnPayload = options?.onPayload; return { ...options, headers: catalog.headers.merge(options?.headers), disableResponseThreading: responseThreading ? undefined : true, loopIncomingToolPlan: catalog.incoming, onPayload: async (payload: unknown, model: Model) => { - const generated = await catalog.payload.apply(payload, model); - return callerOnPayload ? (await callerOnPayload(generated, model)) ?? generated : generated; + let next = await catalog.payload.apply(payload, model); + if (handleOnPayload) next = (await handleOnPayload(next, model)) ?? next; + if (requestOnPayload) next = (await requestOnPayload(next, model)) ?? next; + return next; }, } as T; }; diff --git a/packages/loop/test/attach.test.ts b/packages/loop/test/attach.test.ts index 8ea36fc9..ccb2eb42 100644 --- a/packages/loop/test/attach.test.ts +++ b/packages/loop/test/attach.test.ts @@ -149,6 +149,36 @@ describe("attach", () => { expect(seen[0]!.model.api).toBe(OPENAI_COMPUTER_USE_API); }); + it("runs the handle's onPayload even when the request carries its own", async () => { + const seen: { model: Model; context: Context; options?: LoopSimpleStreamOptions }[] = []; + const applied: string[] = []; + const handle = attach({ + browser, + client, + models: modelsFromStream(recordingStream(seen)), + onPayload: (payload) => { + applied.push("handle"); + return payload as Record; + }, + }); + const compiled = handle.compile({ model: "openai:gpt-5.5", tools: [loop.tools.browser.snapshot()] }); + const session = await new InMemorySessionRepo().create(); + const harness = new AgentHarness({ + session, + model: compiled.model, + models: compiled.models, + tools: [...compiled.tools], + activeToolNames: compiled.tools.map((tool) => tool.name), + }); + compiled.activate(harness); + await harness.prompt("go"); + + // pi's harness always sets its own onPayload, so the handle's hook only + // survives if both run. + await seen[0]!.options?.onPayload?.({}, compiled.model); + expect(applied).toEqual(["handle"]); + }); + it("spends an empty-response retry only when the follow-up is queued", async () => { const attempted: string[] = []; let reject = true; From 289e372425395397c128713d4404a16cbd9032a0 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:54:00 +0000 Subject: [PATCH 6/6] Correct the catalog, quirk, and native-surface documentation --- packages/loop/README.md | 30 ++++++++++++++++++-------- packages/loop/docs/supported-models.md | 20 +++++++++++------ 2 files changed, 34 insertions(+), 16 deletions(-) diff --git a/packages/loop/README.md b/packages/loop/README.md index 2263cbab..a03e5dcd 100644 --- a/packages/loop/README.md +++ b/packages/loop/README.md @@ -161,7 +161,12 @@ Tools return only requested feedback: `toolResultImageReplayLimit` controls how many recent tool-result images remain in model context (`4` by default, or `false` to disable projection). OpenAI native computer results are exempt because its protocol requires each -`computer_call_output` to carry a screenshot. +`computer_call_output` to carry a screenshot, so every native computer action +returns one. + +`emptyResponseRecovery: { followUp, maxAttempts }` queues a follow-up when a +turn ends with a successful tool call but no assistant text — Google's native +browser surface does that occasionally. It is off by default. ## Custom tools @@ -300,6 +305,9 @@ loop.tools.browser.batch({ actions: ["snapshot", "click", "wait_for", "text"] }) loop.tools.playwright(); ``` +A toolset carries that surface's primitives, not every tool it has: `browser_act`, +`computer_zoom`, and the batch forms are selected explicitly. + Batches are mechanical primitive lists. They have no branching, saved values, references, or workflow DSL. @@ -344,6 +352,11 @@ Moonshot accepts the ordinary browser toolset, including `browser_wait_for`, but rejects `browser_act`'s substantially larger function schema. Catalog compilation rejects that specific combination before a provider request. +A provider enables its native surface per model, not for its whole catalog, so +selecting one for a model without it fails during catalog compilation rather +than on the wire. [Models and native surfaces](docs/supported-models.md) lists +which models carry which surface. + Provider-native caller-visible names are fixed by protocol. Version/tool/model mismatches fail during catalog compilation. If an Anthropic credential cannot access `browser_20260701`, Loop retries with an equivalent `browser` function @@ -361,7 +374,6 @@ themselves: const catalog = compileLoopToolCatalog({ model: "anthropic:claude-opus-5", requestedTools: tools, // Loop specs and plain pi-ai Tool declarations - viewport: { width: 1440, height: 900 }, }); catalog.entries; // identities, fingerprints, declarations, coordinates @@ -371,9 +383,9 @@ await catalog.payload.apply(payload, catalog.model); catalog.incoming; ``` -Compilation is declaration-only and deterministic: identical declaration, -model, and viewport inputs produce identical catalogs, and compilation never -constructs executable tools or retains the requested input objects. Execution is +Compilation is declaration-only and deterministic: identical declaration and +model inputs produce identical catalogs, and compilation never constructs +executable tools or retains the requested input objects. Execution is a separate concern: `attach()` materializes specs against a Kernel browser and owns implementation identity. @@ -477,10 +489,10 @@ tells you which apply to the one you selected. | `computer` | canonical computer primitives plus `computer_batch` | every provider | | `browser-act` | `browser_act`, the verified-plan tool | every provider except Moonshot, which rejects its schema size | | `playwright` | `playwright_execute` | every provider | -| `anthropic-computer` | Anthropic's native computer tool | Anthropic only | -| `anthropic-browser` | Anthropic's native browser tool | Anthropic only | -| `openai-computer` | OpenAI's native computer tool | OpenAI only | -| `google-browser` | Google's predefined browser action set | Google only | +| `anthropic-computer` | Anthropic's native computer tool | Anthropic models with that native surface | +| `anthropic-browser` | Anthropic's native browser tool | Anthropic models with that native surface | +| `openai-computer` | OpenAI's native computer tool | OpenAI models with that native surface | +| `google-browser` | Google's predefined browser action set | Google models with that native surface | `--browser-coordinates` selects `pixels` (default) or `normalized-1000` for the `computer` entry's coordinate contract. diff --git a/packages/loop/docs/supported-models.md b/packages/loop/docs/supported-models.md index 0685b0c6..ac1f90c4 100644 --- a/packages/loop/docs/supported-models.md +++ b/packages/loop/docs/supported-models.md @@ -12,7 +12,9 @@ model may run. ## Native surfaces `COMPUTER_USE_NATIVE_SURFACES` records which models have a provider-native computer or -browser tool, so the tool menu can offer it. Entries match either an exact id or +browser tool, so the tool menu can offer it and catalog compilation can refuse +it for a model the provider has not enabled it for — every provider answers 400 +for that combination. Entries match either an exact id or a `family` — the family root plus suffixes made of hyphen-separated numeric segments, covering revisions and dated snapshots such as `claude-opus-4-7` or `gpt-5.5-2026-04-23`. Named sibling variants like `gpt-5.4-mini` are distinct @@ -23,9 +25,9 @@ models and need their own entry. Each cites first-party documentation. | `anthropic` | `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser | | `anthropic` | `claude-fable-5` family | computer | | `openai` | `gpt-5.6-sol`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.5` | computer | -| `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite` | browser | +| `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-2.5-computer-use-preview-10-2025` | browser | -Anthropic's entries live in `providers/anthropic/capabilities.ts`, which is +Anthropic's entries live in `src/pi/providers/anthropic/capabilities.ts`, which is version-gated separately; `computerUseNativeSurfaces(model)` reads both sources. A model with no native surface is not restricted — it drives a Kernel browser @@ -40,12 +42,16 @@ failure, and carries its reason inline. | provider | models | limit | | --- | --- | --- | -| `google` | all | rejects `browser_wait_for`'s schema shape; the Gemini API accepts a subset of JSON Schema for function declarations | -| `moonshotai` | `kimi-k3` | rejects the request once `browser_act`'s schema is attached; serializes state mutations | -| `openrouter` | `moonshotai/kimi-k3` | the same Kimi limit, reached through OpenRouter | +| `moonshotai` | all | rejects the request once `browser_act`'s schema is attached | +| `moonshotai` | `kimi-k3` | serializes state mutations | +| `openrouter` | `moonshotai/kimi-k3` | the same Kimi limits, reached through OpenRouter | | `openrouter` | `meta/muse-spark-1.1` | serializes state mutations | | `xai` | all | serializes state mutations | +Google is absent on purpose: the Gemini function-declaration dialect is a subset +of JSON Schema, and a payload transform rewrites `const` and +`additionalProperties` into what it accepts, so no tool has to be withheld. + `loopModelCapabilities(model)` applies provider-wide quirks first, then model-specific ones. `loopModelQuirks(model)` returns the entries that applied, for diagnostics and menu hints. @@ -85,7 +91,7 @@ a table edit. Probe what the model actually emits: npx tsx packages/loop/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --limit 3 ``` -Update that provider's adapter under `src/providers/` to execute the actions the +Update that provider's adapter under `src/pi/providers/` to execute the actions the probe returns, then add or adjust the `COMPUTER_USE_NATIVE_SURFACES` entry, citing the provider's documentation. Anthropic's computer tool version and its `computer-use-*` beta header are chosen by pi-ai per model, so a new dated