Repository navigation
feat(tools): add generic built-in tools for documents, media, and uti… - #1222
Conversation
…lities Add eleven framework-agnostic built-in tools so agents and migrated workflows do not need bespoke adapters for common capabilities: - http_request: SSRF-guarded HTTP(S) client with redirect re-validation, host allowlist, env-referenced header secrets, and a body size cap - document_parse: PDF/Word/Excel/CSV/HTML/JSON/Markdown extraction for local files and public URLs, with Content-Type aware detection - doc_generate: Word/Excel/CSV/Markdown artifacts - mindmap_generate: Mermaid and SVG from a Markdown outline - qrcode_generate: PNG QR codes - image_transform: resize/crop/rotate/flip/thumbnail/grayscale/convert - image_understand and ocr: vision calls through the configured model - speech_to_text: Volcengine flash ASR - calculator and datetime: safe arithmetic and timezone-aware date math Register them in the built-in tool registry, mirror the declarations in the Studio cold-start catalog, document both locales, and add unit tests (100% line coverage for the new modules). web_fetch now exposes assert_public_url so outbound HTTP tools share one SSRF policy.
…nners The capability endpoint answers 202 while its background lookup runs. Under the CI unit-test matrix (pytest -n 16 on a 2-vCPU runner) the route's 1.8s initial wait can expire even with every runtime call stubbed, which made test_mpa_update_rejects_incompatible_manifest_before_launch flaky. Poll the endpoint until it settles, mirroring the existing capability helper.
cuericlee
left a comment
There was a problem hiding this comment.
Review: generic built-in tools (ee5f7c2)
Reviewed the eleven new tools end-to-end, the registry change, the Studio declarations file, and the new dependency pins. Overall this is solid: tools return {"error": ...} instead of raising, heavy parsers are imported lazily, inputs are type/range-checked with bool guards, SSRF checks are centralized in assert_public_url and re-run on every redirect hop (http_request/document_parse/image_transform), downloads are streamed with byte caps (and Content-Length treated as advisory in image_transform), the calculator is an AST allowlist evaluator, and the registry contract test pins the advertised names. python-docx/openpyxl bounds are reasonable.
Comments below, ordered by importance. #1 is a concrete credential-exposure gap; the rest are hardening and consistency items.
1. Secret/custom headers are forwarded across cross-host and downgrade redirects
_perform_request uses one headers dict for every hop (http_request.py:145-195). Each hop is SSRF-validated, but nothing stops a public, attacker-controlled host from appearing in the redirect chain. A request to https://api.example.com carrying Authorization or an env:NAME-resolved key that answers 302 Location: https://attacker.test/... (307/308 also preserve the body) causes the secret — and for 307/308 the request body — to be sent to the attacker; https:// → http:// redirects additionally expose it in cleartext. Browsers and requests strip auth headers on cross-origin/downgrade redirects for exactly this reason.
Suggested: when the redirect target's scheme downgrades or its host (ideally registrable domain) differs, drop Authorization, Cookie, Proxy-Authorization and any header resolved from env:, or drop all non-default custom headers and only re-apply them for same-host hops. Worth a test with an env-resolved header and a 302 to a different host asserting it never arrives.
2. image_transform resize has no output-size cap
_op_resize accepts any positive int for width/height (image_transform.py:169-187). Upscaling a small source (or rotate(expand=True) on a large image) lets model-supplied arguments allocate an arbitrarily large bitmap — e.g. resize to 100000×100000 — an OOM driven entirely by tool arguments, independent of the 20MB input cap. Pillow's decompression-bomb guard protects input decoding, not output allocation. Suggest a max side / total-pixels bound with a clear error (and the same check applied to thumbnail's resulting bounds is unnecessary since those only shrink).
3. document_parse caps remote downloads but not local files
_download_url streams to 20MB; the local branch does path.read_bytes() with no cap (document_parse.py:402-408), then hands bytes to pypdfium2/openpyxl. A model-supplied path to a multi-GB (or zip-bomb xlsx) file is read fully into memory and parsed. For parity and predictable memory, apply the same cap to local reads and surface a truncation/error result. Separately, xlsx zip-bomb expansion inside openpyxl is unconstrained (read_only helps, but doesn't bound expanded sheet data) — at minimum note the trust boundary; a library-level limit is optional.
4. CSV/Excel formula injection in doc_generate
_cell_value and the CSV writer emit model-provided strings verbatim. Values beginning with =, +, -, @ (optionally prefixed by tab/CR) become formulas when the generated xlsx/csv is opened in Excel; generated documents here typically embed data the model extracted from untrusted input. Standard mitigation: prefix such cells with a single quote for xlsx/csv, or wrap in text — applies to both sheets and rows. If deliberately left to the caller, it should be stated in the tool docstring and the builtin-tool docs.
5. Studio declarations are hand-synced copies
frontend/server/studio_tools/builtin_declarations.py duplicates each tool's description and parameter schema (431 added lines), and nothing in tests compares them to the live FunctionTool declarations/registry. Descriptions already drift in small ways (formatting/wording). Suggest a contract test that, for every registered built-in present in the declarations map, compares the JSON-schema property names/required set against tool._get_declaration(), so a parameter change to an implementation fails CI until the Studio copy is updated.
6. Vision URL handling differs from the other tools
vision._image_part passes remote http(s) URLs straight to the model provider without assert_public_url, while speech_to_text and the download-capable tools validate the host (vision.py:92-94). It is the provider that fetches rather than the SDK, so the risk is indirect, but the behavior should be deliberate and documented (or aligned), especially for deployments whose vision endpoint resolves internal names. Also, the per-image 20MB cap allows an aggregate request of ~160MB base64 (8 images) with no total budget — a coarse aggregate cap would bound request size.
Nits / questions
http_requestsends no explicit User-Agent (httpx default advertises the client/version); fine if intentional, but the other new fetchers all set one.assert_public_url→getaddrinfois synchronous. It's fine in the sync tools (http_request,document_parse), butimage_transform._download_urland the vision path are async and block the event loop on DNS; trivially avoidable withasyncio.to_threadif the loop matters.- The DNS-validation/connection TOCTOU already documented on
web_fetchapplies to the three new callers too; a shared follow-up that pins the socket to the validated IP would close it for all of them. doc_generate/image_transform/mindmap_generate/qrcode_generatecreatemkdtemp/mkstempoutputs that are never cleaned whentool_contextis None. In production ADK always injects the context, so this mainly accumulates files in tests and direct/embedded use; consider a best-effort cleanup or documenting ownership.document_parsetruncation is silent: a remote body capped at 20MB can fail parsing with only the parser error; including atruncatedhint (ashttp_requestdoes) would help the model diagnose it.datetime_tool.pydefines both the public function and its module concept underdatetime; the registry import path handles it, just flagging thatfrom datetime import ...in any future code in that module needs the aliased form already used here.
Verification note
Static review only (plus a quick local Pillow check that rotate-fillcolor on an L image is tolerated). Tests cover the matrix I would have asked for in http_request (SSRF, allowlist per hop, env headers, redirect method/body semantics, truncation), and the malformed-input suites for the other tools are thorough; CI on ee5f7c20 was still partially pending at review time (Gate / codex-smoke / sidecar gates), so merge should wait on those. I did not run the suite or exercise real ASR/vision endpoints.
There was a problem hiding this comment.
补充上一条英文 review(#1–#6 结论一致,以上条为准):
#1 的可利用性已本地复现:用 fake client + DNS 打桩,对 https://safe.example/x 携带 Authorization: Bearer SECRET 和经 env: 解析的 X-Api-Key,响应 302 → https://evil.example/capture 后,第二跳同时携带了这两个头(https→http 降级时还会明文发送),建议本 PR 修复并补跨域用例。
另一个小 nit:_perform_request 对 303 无条件改 GET,HEAD 请求也会被改(httpx 对 303 保留 HEAD),影响很小,顺手对齐即可。
验证环境说明:沙箱内零依赖跑 test_calculator.py 全过;test_datetime_tool.py 有 6 条失败,定位为评审沙箱自身 tzdata 异常(ZoneInfo("UTC") 偏移为 +08:00),CI test (3.12) 为 pass,非本 PR 缺陷。
- http_request: drop Authorization, Cookie, Proxy-Authorization, and env:-resolved headers on cross-origin or https->http redirects, while keeping them for same-origin hops and same-host https upgrades; keep HEAD as HEAD on 303. - image_transform: bound resize/rotate output canvases at 40,000,000 pixels, cap local reads at the download limit, and move DNS checks off the event loop. - vision: SSRF-check remote image URLs before forwarding them to the provider, cap local reads, add a 40,000,000-byte aggregate local payload limit, and offload blocking DNS/file work. - doc_generate: write formula-like xlsx cells as text and prefix CSV cells with a quote, and report truncated output. - document_parse: cap local file reads, flag remote downloads cut by the byte cap, and document the xlsx expansion trust boundary. - studio_tools: refresh the deferred declarations mirror to match the updated tool docstrings. - docs: document the new redirect, size, SSRF, and truncation behavior in both languages.
cuericlee
left a comment
There was a problem hiding this comment.
Re-review of 0be969d0 — resolved
Pulled the fix commit and verified each prior finding against the new code, with runtime repros (not just static reading). All six substantive points are addressed; CI is fully green (test (3.10)/(3.12), Backend/Frontend gates, gitleaks, License, codex-smoke, Gate all SUCCESS).
- Credential replay across redirects — fixed and reproduced.
_headers_for_redirectmirrors httpx semantics. Runtime check with a fake client: cross-host 302 now dropsAuthorization,Cookie,Proxy-Authorizationand everyenv:-resolved header (X-Api-Key) while keeping plain headers; same-hosthttp→httpsupgrade keeps credentials; a stripped header stays stripped on later hops. The new tests cover same-origin, explicit-default-port, upgrade, cross-host, downgrade, and multi-hop cases. The 303/HEAD nit is also fixed (HEAD now stays HEAD). - Output-size cap in image_transform — fixed.
_MAX_OUTPUT_PIXELS = 40,000,000guards resize in all three branches, and rotate pre-computes the expanded canvas via_rotated_sizebefore allocation (verified: 50000×50000, single-side upscale, and a 45° rotation of a 6000×6000 source are all rejected; 100×100 still allowed). Local file reads are now capped too, and blocking DNS moved toasyncio.to_thread. - document_parse local read cap — fixed. Local
>20MBnow returns an error (reproduced), and remote over-cap bodies setmetadata["download_truncated"]instead of failing opaquely. The xlsx zip-expansion boundary is documented explicitly (20MB input cap is the bound) — fine as a stated trust decision. - Formula injection — fixed and verified end-to-end. Generated an xlsx with
=/+/-/@cells and re-opened it: every one reads back asdata_type='s'(string), never a formula; CSV prefixes a'. The newtruncatedflag is wired into the result and declared in the schema. - Studio declarations hand-synced — this commit updated the affected descriptions/schemas by hand, but there is still no automated contract test comparing the declarations against the live function signatures. Non-blocking; it will drift again eventually, so a parameter/required-set parity test remains worth doing as a follow-up.
- Vision SSRF + aggregate budget — fixed and reproduced.
vision._image_partnow callsassert_public_urlbefore forwarding a remote URL:169.254.169.254and127.0.0.1are rejected, a public URL is still forwarded with zero local bytes, and local images share a 40MB aggregate budget (per-image 20MB). DNS/file reads moved off the event loop.
Verification: runtime repros above, plus pytest tests/tools/builtin_tools/test_{http_request,image_transform,doc_generate,vision_tools}.py → 273 passed locally. Remaining unaddressed nits are cosmetic (http_request still sends no explicit User-Agent; temp files without a tool_context are not cleaned up) and don't affect merge.
Approving.
…lities
Add eleven framework-agnostic built-in tools so agents and migrated workflows do not need bespoke adapters for common capabilities:
Register them in the built-in tool registry, mirror the declarations in the Studio cold-start catalog, document both locales, and add unit tests (100% line coverage for the new modules). web_fetch now exposes assert_public_url so outbound HTTP tools share one SSRF policy.