Skip to content

feat: add AgentKit CUA computer-use toolset - #1219

Merged
zakahan merged 4 commits into
volcengine:mainfrom
helenyang88:feat/agentkit-cua-stdio
Nov 17, 2026
Merged

zakahan merged 4 commits into
volcengine:mainfrom
helenyang88:feat/agentkit-cua-stdio

Conversation

@helenyang88

@helenyang88 helenyang88 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add ComputerUseToolset / create_computer_use_toolset so VeADK agents can discover and call the independently distributed AgentKit CUA MCP server over stdio. Add a standalone example for customers using a standard MCP client instead of VeADK.

  • Discover tools with tools/list before model execution, without provisioning a sandbox. The MCP package exposes a template-defined catalog and resolves/creates the sandbox lazily on an actual action or explicit ensure_sandbox call.
  • Derive routing from the real ADK context, persist returned sandbox bindings through session-state events, and convert MCP screenshots into model image content.
  • Reject CUA actions in multi-tool-call model responses with model-readable errors. Do not retry uncertain mutations. Discovery failures stop execution rather than silently removing tools.
  • Require deployment configuration through CUA_MCP_WHEEL_URL, or explicit connection_params for preinstalled/offline execution. There is no built-in wheel URL or automatic STG-variable forwarding.
  • Add examples/cua_without_veadk/ with independent dependencies, an actual model/tool loop, screenshot handling, binding persistence and bilingual instructions. It contains no metrics instrumentation, test directory, or cache files.

Compatibility and boundaries

  • Existing run_code, general Ark model handling and AIO tools are unchanged by these commits.
  • The CUA MCP server and sandbox implementation are maintained separately; this PR does not deploy or modify either one.
  • Explicit STG clients must pass the environment to the revised MCP server and use separate conversation state. The default VeADK launcher remains unchanged in this respect.
  • Multi-call rejection covers CUA calls only; other tools in the same model response may execute. The standalone example rejects the entire multi-call response before dispatch.
  • The standalone example persists only the sandbox binding, not full model history, and does not include an HTTP server or Runtime image.

Verification

  • uv run --extra dev pre-commit run --all-files: passed, including Ruff and secret scanning.
  • pytest -q tests/tools/cua_mcp tests/tools/builtin_tools/test_run_code.py: 73 passed on the submitted example revision with the optional MCP dependency installed.
  • Standalone example: independent dependency installation, CLI help, syntax and bilingual local-link checks passed. Offline stdio/model checks passed during development; their fixture files were removed from the deliverable at the owner's request.
  • One live standalone smoke completed: real model -> public MCP wheel -> lazy sandbox creation -> screenshot -> model description. The saved binding matched the platform instance; the test instance was cleaned up. This is not all-tool, STG or hosted Runtime certification.

Review notes

  • tests/tools/cua_mcp skips when the independent MCP package is not installed; a skipped CI run is not CUA verification.
  • Use an explicitly selected wheel revision and verify downloaded SHA256 separately; a URL fragment alone is not an integrity guarantee for every uvx version.
  • Local verification results above do not imply that GitHub Actions have run. Fork workflows require maintainer approval when GitHub requests it.

@zakahan zakahan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is by an automated agent, not a human.

I reviewed commit 742c28d (vs 38d3c6e6), and also checked how the branch relates to origin/main.

Two items must be fixed before this can merge.

  1. The branch is 38 commits behind origin/main, and this PR was opened before PRs #1220, #1222 and #1218 merged. GitHub compares against the merge base and shows only these 20 files, but merging the feat/agentkit-cua-stdio head as it stands (or letting auto-merge do a fast-forward, because the merge base is 38d3c6e) would silently return main to a much older state, where the changes described in the next sentence are undone. It would revert query routing, the generic built-in tools, and the ADK 2.11 support.

  2. Please drop the files that the two commits also carry but that have nothing to do with CUA. Please keep only veadk/tools/builtin_tools/computer_use.py, veadk/tools/computer_use/*, tests/tools/cua_mcp/* and examples/cua_without_veadk/*, plus the two examples/README* additions. The unrelated changes are also visible when comparing the head against origin/main:

    • pyproject.toml, uv.lock, tests/test_runtime_dependency_compatibility.py: ADK range is reverted to <2.8.0 and openpyxl and python-docx are removed.
    • veadk/agent.py, veadk/routing/*, veadk/decision/*, veadk/configs/routing_configs.py, and the matching docs and tests: these reverted routing.
    • veadk/tools/__init__.py, veadk/tools/builtin_tools/*: these reverted #1222's built-in-tool registry.
    • veadk/models/ark_llm.py, veadk/utils/misc.py, veadk/utils/patches.py, veadk/runner.py, veadk/cli/studio_start.py, veadk/content_safety/*, pytest.ini, and the routing and built-in-tool tests: these changes were unauthorised.

The CUA code itself looks solid. Declining mutating tools when another tool is called in the same model turn, failing closed on discovery and catalog errors, persisting the binding through state deltas, and testing at the Toolset level are all good.

Smaller points that can wait for a follow-up:

  • In tests/tools/cua_mcp/test_serial_guidance.py, test_disabled_template_never_recommends_batch uses sys, but the module does not import it, so that test fails with a NameError instead of asserting anything.
  • _ComputerTool.run_async reports every transport failure as "outcome may be unknown; not retried". For failures raised before any request is sent (such as a subprocess start failure), the model is told the outcome is unknown when in fact nothing ran. Check whether these two cases can be told apart.
  • tests/tools/cua_mcp/conftest.py skips every test when agentkit-cua-mcp is not installed, so CI will not exercise this suite unless the pinned wheel is installed for it. Please make sure some pipeline actually installs the wheel and runs the tests.

@zakahan zakahan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is by an automated agent, not a human.

Correction to my earlier review: the "merging this would revert query routing and other changes" point was wrong.

I re-checked by doing a real local 3-way merge of 742c28d into the current origin/main (4afae36): git merge --no-ff 742c28d completed with no conflicts, and the resulting tree changes only these 20 files (+2624 lines, no deletions). Merging never takes the PR branch state as-is; it combines both sides from the merge base, so the routing, built-in-tool and ADK 2.11 changes on main are kept.

What actually holds:

  1. The two commits also carry outdated copies of files that main has since changed: pyproject.toml, uv.lock, veadk/tools/__init__.py, veadk/agent.py, veadk/models/ark_llm.py and others. The merge above did not contain those copies, but they are still noise in the diff and make this branch conflict-prone. Please rebase onto main and keep only the CUA files plus the two examples/README* additions.

  2. The small points below still stand: the missing import sys in test_serial_guidance.py, and the CI suite skipping when the agentkit-cua-mcp wheel is not installed.

Please ignore the earlier rollback claim.

@zakahan
zakahan merged commit 622e7a2 into volcengine:main Nov 17, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants