Repository navigation
feat: add AgentKit CUA computer-use toolset - #1219
Conversation
zakahan
left a comment
There was a problem hiding this comment.
This is by an automated agent, not a human.
I reviewed commit 742c28d (vs 38d3c6e6), and also checked how the branch relates to origin/main.
Two items must be fixed before this can merge.
-
The branch is 38 commits behind
origin/main, and this PR was opened before PRs #1220, #1222 and #1218 merged. GitHub compares against the merge base and shows only these 20 files, but merging thefeat/agentkit-cua-stdiohead as it stands (or letting auto-merge do a fast-forward, because the merge base is 38d3c6e) would silently returnmainto a much older state, where the changes described in the next sentence are undone. It would revert query routing, the generic built-in tools, and the ADK 2.11 support. -
Please drop the files that the two commits also carry but that have nothing to do with CUA. Please keep only
veadk/tools/builtin_tools/computer_use.py,veadk/tools/computer_use/*,tests/tools/cua_mcp/*andexamples/cua_without_veadk/*, plus the twoexamples/README*additions. The unrelated changes are also visible when comparing the head againstorigin/main:pyproject.toml,uv.lock,tests/test_runtime_dependency_compatibility.py: ADK range is reverted to<2.8.0and openpyxl and python-docx are removed.veadk/agent.py,veadk/routing/*,veadk/decision/*,veadk/configs/routing_configs.py, and the matching docs and tests: these reverted routing.veadk/tools/__init__.py,veadk/tools/builtin_tools/*: these reverted #1222's built-in-tool registry.veadk/models/ark_llm.py,veadk/utils/misc.py,veadk/utils/patches.py,veadk/runner.py,veadk/cli/studio_start.py,veadk/content_safety/*,pytest.ini, and the routing and built-in-tool tests: these changes were unauthorised.
The CUA code itself looks solid. Declining mutating tools when another tool is called in the same model turn, failing closed on discovery and catalog errors, persisting the binding through state deltas, and testing at the Toolset level are all good.
Smaller points that can wait for a follow-up:
- In
tests/tools/cua_mcp/test_serial_guidance.py,test_disabled_template_never_recommends_batchusessys, but the module does not import it, so that test fails with aNameErrorinstead of asserting anything. _ComputerTool.run_asyncreports every transport failure as "outcome may be unknown; not retried". For failures raised before any request is sent (such as a subprocess start failure), the model is told the outcome is unknown when in fact nothing ran. Check whether these two cases can be told apart.tests/tools/cua_mcp/conftest.pyskips every test whenagentkit-cua-mcpis not installed, so CI will not exercise this suite unless the pinned wheel is installed for it. Please make sure some pipeline actually installs the wheel and runs the tests.
zakahan
left a comment
There was a problem hiding this comment.
This is by an automated agent, not a human.
Correction to my earlier review: the "merging this would revert query routing and other changes" point was wrong.
I re-checked by doing a real local 3-way merge of 742c28d into the current origin/main (4afae36): git merge --no-ff 742c28d completed with no conflicts, and the resulting tree changes only these 20 files (+2624 lines, no deletions). Merging never takes the PR branch state as-is; it combines both sides from the merge base, so the routing, built-in-tool and ADK 2.11 changes on main are kept.
What actually holds:
-
The two commits also carry outdated copies of files that
mainhas since changed:pyproject.toml,uv.lock,veadk/tools/__init__.py,veadk/agent.py,veadk/models/ark_llm.pyand others. The merge above did not contain those copies, but they are still noise in the diff and make this branch conflict-prone. Please rebase ontomainand keep only the CUA files plus the twoexamples/README*additions. -
The small points below still stand: the missing
import sysintest_serial_guidance.py, and the CI suite skipping when theagentkit-cua-mcpwheel is not installed.
Please ignore the earlier rollback claim.
Summary
Add
ComputerUseToolset/create_computer_use_toolsetso VeADK agents can discover and call the independently distributed AgentKit CUA MCP server over stdio. Add a standalone example for customers using a standard MCP client instead of VeADK.tools/listbefore model execution, without provisioning a sandbox. The MCP package exposes a template-defined catalog and resolves/creates the sandbox lazily on an actual action or explicitensure_sandboxcall.CUA_MCP_WHEEL_URL, or explicitconnection_paramsfor preinstalled/offline execution. There is no built-in wheel URL or automatic STG-variable forwarding.examples/cua_without_veadk/with independent dependencies, an actual model/tool loop, screenshot handling, binding persistence and bilingual instructions. It contains no metrics instrumentation, test directory, or cache files.Compatibility and boundaries
run_code, general Ark model handling and AIO tools are unchanged by these commits.Verification
uv run --extra dev pre-commit run --all-files: passed, including Ruff and secret scanning.pytest -q tests/tools/cua_mcp tests/tools/builtin_tools/test_run_code.py: 73 passed on the submitted example revision with the optional MCP dependency installed.Review notes
tests/tools/cua_mcpskips when the independent MCP package is not installed; a skipped CI run is not CUA verification.