Skip to content

feat(tools): add generic built-in tools for documents, media, and uti… - #1222

Merged
wujiaming-ai merged 3 commits into
mainfrom
feat/generic-builtin-tools
Nov 16, 2026
Merged

wujiaming-ai merged 3 commits into
mainfrom
feat/generic-builtin-tools

Conversation

@wujiaming-ai

Copy link
Copy Markdown
Collaborator

…lities

Add eleven framework-agnostic built-in tools so agents and migrated workflows do not need bespoke adapters for common capabilities:

  • http_request: SSRF-guarded HTTP(S) client with redirect re-validation, host allowlist, env-referenced header secrets, and a body size cap
  • document_parse: PDF/Word/Excel/CSV/HTML/JSON/Markdown extraction for local files and public URLs, with Content-Type aware detection
  • doc_generate: Word/Excel/CSV/Markdown artifacts
  • mindmap_generate: Mermaid and SVG from a Markdown outline
  • qrcode_generate: PNG QR codes
  • image_transform: resize/crop/rotate/flip/thumbnail/grayscale/convert
  • image_understand and ocr: vision calls through the configured model
  • speech_to_text: Volcengine flash ASR
  • calculator and datetime: safe arithmetic and timezone-aware date math

Register them in the built-in tool registry, mirror the declarations in the Studio cold-start catalog, document both locales, and add unit tests (100% line coverage for the new modules). web_fetch now exposes assert_public_url so outbound HTTP tools share one SSRF policy.

…lities

Add eleven framework-agnostic built-in tools so agents and migrated
workflows do not need bespoke adapters for common capabilities:

- http_request: SSRF-guarded HTTP(S) client with redirect re-validation,
  host allowlist, env-referenced header secrets, and a body size cap
- document_parse: PDF/Word/Excel/CSV/HTML/JSON/Markdown extraction for
  local files and public URLs, with Content-Type aware detection
- doc_generate: Word/Excel/CSV/Markdown artifacts
- mindmap_generate: Mermaid and SVG from a Markdown outline
- qrcode_generate: PNG QR codes
- image_transform: resize/crop/rotate/flip/thumbnail/grayscale/convert
- image_understand and ocr: vision calls through the configured model
- speech_to_text: Volcengine flash ASR
- calculator and datetime: safe arithmetic and timezone-aware date math

Register them in the built-in tool registry, mirror the declarations in
the Studio cold-start catalog, document both locales, and add unit tests
(100% line coverage for the new modules). web_fetch now exposes
assert_public_url so outbound HTTP tools share one SSRF policy.
…nners

The capability endpoint answers 202 while its background lookup runs. Under
the CI unit-test matrix (pytest -n 16 on a 2-vCPU runner) the route's 1.8s
initial wait can expire even with every runtime call stubbed, which made
test_mpa_update_rejects_incompatible_manifest_before_launch flaky. Poll the
endpoint until it settles, mirroring the existing capability helper.

@cuericlee cuericlee left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: generic built-in tools (ee5f7c2)

Reviewed the eleven new tools end-to-end, the registry change, the Studio declarations file, and the new dependency pins. Overall this is solid: tools return {"error": ...} instead of raising, heavy parsers are imported lazily, inputs are type/range-checked with bool guards, SSRF checks are centralized in assert_public_url and re-run on every redirect hop (http_request/document_parse/image_transform), downloads are streamed with byte caps (and Content-Length treated as advisory in image_transform), the calculator is an AST allowlist evaluator, and the registry contract test pins the advertised names. python-docx/openpyxl bounds are reasonable.

Comments below, ordered by importance. #1 is a concrete credential-exposure gap; the rest are hardening and consistency items.

1. Secret/custom headers are forwarded across cross-host and downgrade redirects

_perform_request uses one headers dict for every hop (http_request.py:145-195). Each hop is SSRF-validated, but nothing stops a public, attacker-controlled host from appearing in the redirect chain. A request to https://api.example.com carrying Authorization or an env:NAME-resolved key that answers 302 Location: https://attacker.test/... (307/308 also preserve the body) causes the secret — and for 307/308 the request body — to be sent to the attacker; https:// → http:// redirects additionally expose it in cleartext. Browsers and requests strip auth headers on cross-origin/downgrade redirects for exactly this reason.

Suggested: when the redirect target's scheme downgrades or its host (ideally registrable domain) differs, drop Authorization, Cookie, Proxy-Authorization and any header resolved from env:, or drop all non-default custom headers and only re-apply them for same-host hops. Worth a test with an env-resolved header and a 302 to a different host asserting it never arrives.

2. image_transform resize has no output-size cap

_op_resize accepts any positive int for width/height (image_transform.py:169-187). Upscaling a small source (or rotate(expand=True) on a large image) lets model-supplied arguments allocate an arbitrarily large bitmap — e.g. resize to 100000×100000 — an OOM driven entirely by tool arguments, independent of the 20MB input cap. Pillow's decompression-bomb guard protects input decoding, not output allocation. Suggest a max side / total-pixels bound with a clear error (and the same check applied to thumbnail's resulting bounds is unnecessary since those only shrink).

3. document_parse caps remote downloads but not local files

_download_url streams to 20MB; the local branch does path.read_bytes() with no cap (document_parse.py:402-408), then hands bytes to pypdfium2/openpyxl. A model-supplied path to a multi-GB (or zip-bomb xlsx) file is read fully into memory and parsed. For parity and predictable memory, apply the same cap to local reads and surface a truncation/error result. Separately, xlsx zip-bomb expansion inside openpyxl is unconstrained (read_only helps, but doesn't bound expanded sheet data) — at minimum note the trust boundary; a library-level limit is optional.

4. CSV/Excel formula injection in doc_generate

_cell_value and the CSV writer emit model-provided strings verbatim. Values beginning with =, +, -, @ (optionally prefixed by tab/CR) become formulas when the generated xlsx/csv is opened in Excel; generated documents here typically embed data the model extracted from untrusted input. Standard mitigation: prefix such cells with a single quote for xlsx/csv, or wrap in text — applies to both sheets and rows. If deliberately left to the caller, it should be stated in the tool docstring and the builtin-tool docs.

5. Studio declarations are hand-synced copies

frontend/server/studio_tools/builtin_declarations.py duplicates each tool's description and parameter schema (431 added lines), and nothing in tests compares them to the live FunctionTool declarations/registry. Descriptions already drift in small ways (formatting/wording). Suggest a contract test that, for every registered built-in present in the declarations map, compares the JSON-schema property names/required set against tool._get_declaration(), so a parameter change to an implementation fails CI until the Studio copy is updated.

6. Vision URL handling differs from the other tools

vision._image_part passes remote http(s) URLs straight to the model provider without assert_public_url, while speech_to_text and the download-capable tools validate the host (vision.py:92-94). It is the provider that fetches rather than the SDK, so the risk is indirect, but the behavior should be deliberate and documented (or aligned), especially for deployments whose vision endpoint resolves internal names. Also, the per-image 20MB cap allows an aggregate request of ~160MB base64 (8 images) with no total budget — a coarse aggregate cap would bound request size.

Nits / questions

  • http_request sends no explicit User-Agent (httpx default advertises the client/version); fine if intentional, but the other new fetchers all set one.
  • assert_public_url → getaddrinfo is synchronous. It's fine in the sync tools (http_request, document_parse), but image_transform._download_url and the vision path are async and block the event loop on DNS; trivially avoidable with asyncio.to_thread if the loop matters.
  • The DNS-validation/connection TOCTOU already documented on web_fetch applies to the three new callers too; a shared follow-up that pins the socket to the validated IP would close it for all of them.
  • doc_generate/image_transform/mindmap_generate/qrcode_generate create mkdtemp/mkstemp outputs that are never cleaned when tool_context is None. In production ADK always injects the context, so this mainly accumulates files in tests and direct/embedded use; consider a best-effort cleanup or documenting ownership.
  • document_parse truncation is silent: a remote body capped at 20MB can fail parsing with only the parser error; including a truncated hint (as http_request does) would help the model diagnose it.
  • datetime_tool.py defines both the public function and its module concept under datetime; the registry import path handles it, just flagging that from datetime import ... in any future code in that module needs the aliased form already used here.

Verification note

Static review only (plus a quick local Pillow check that rotate-fillcolor on an L image is tolerated). Tests cover the matrix I would have asked for in http_request (SSRF, allowlist per hop, env headers, redirect method/body semantics, truncation), and the malformed-input suites for the other tools are thorough; CI on ee5f7c20 was still partially pending at review time (Gate / codex-smoke / sidecar gates), so merge should wait on those. I did not run the suite or exercise real ASR/vision endpoints.

@cuericlee cuericlee left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充上一条英文 review(#1–#6 结论一致,以上条为准):

#1 的可利用性已本地复现:用 fake client + DNS 打桩,对 https://safe.example/x 携带 Authorization: Bearer SECRET 和经 env: 解析的 X-Api-Key,响应 302 → https://evil.example/capture 后,第二跳同时携带了这两个头(https→http 降级时还会明文发送),建议本 PR 修复并补跨域用例。

另一个小 nit:_perform_request 对 303 无条件改 GET,HEAD 请求也会被改(httpx 对 303 保留 HEAD),影响很小,顺手对齐即可。

验证环境说明:沙箱内零依赖跑 test_calculator.py 全过;test_datetime_tool.py 有 6 条失败,定位为评审沙箱自身 tzdata 异常(ZoneInfo("UTC") 偏移为 +08:00),CI test (3.12) 为 pass,非本 PR 缺陷。

- http_request: drop Authorization, Cookie, Proxy-Authorization, and
  env:-resolved headers on cross-origin or https->http redirects, while
  keeping them for same-origin hops and same-host https upgrades; keep
  HEAD as HEAD on 303.
- image_transform: bound resize/rotate output canvases at 40,000,000
  pixels, cap local reads at the download limit, and move DNS checks off
  the event loop.
- vision: SSRF-check remote image URLs before forwarding them to the
  provider, cap local reads, add a 40,000,000-byte aggregate local
  payload limit, and offload blocking DNS/file work.
- doc_generate: write formula-like xlsx cells as text and prefix CSV
  cells with a quote, and report truncated output.
- document_parse: cap local file reads, flag remote downloads cut by the
  byte cap, and document the xlsx expansion trust boundary.
- studio_tools: refresh the deferred declarations mirror to match the
  updated tool docstrings.
- docs: document the new redirect, size, SSRF, and truncation behavior
  in both languages.

@cuericlee cuericlee left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of 0be969d0 — resolved

Pulled the fix commit and verified each prior finding against the new code, with runtime repros (not just static reading). All six substantive points are addressed; CI is fully green (test (3.10)/(3.12), Backend/Frontend gates, gitleaks, License, codex-smoke, Gate all SUCCESS).

  1. Credential replay across redirects — fixed and reproduced. _headers_for_redirect mirrors httpx semantics. Runtime check with a fake client: cross-host 302 now drops Authorization, Cookie, Proxy-Authorization and every env:-resolved header (X-Api-Key) while keeping plain headers; same-host http→https upgrade keeps credentials; a stripped header stays stripped on later hops. The new tests cover same-origin, explicit-default-port, upgrade, cross-host, downgrade, and multi-hop cases. The 303/HEAD nit is also fixed (HEAD now stays HEAD).
  2. Output-size cap in image_transform — fixed. _MAX_OUTPUT_PIXELS = 40,000,000 guards resize in all three branches, and rotate pre-computes the expanded canvas via _rotated_size before allocation (verified: 50000×50000, single-side upscale, and a 45° rotation of a 6000×6000 source are all rejected; 100×100 still allowed). Local file reads are now capped too, and blocking DNS moved to asyncio.to_thread.
  3. document_parse local read cap — fixed. Local >20MB now returns an error (reproduced), and remote over-cap bodies set metadata["download_truncated"] instead of failing opaquely. The xlsx zip-expansion boundary is documented explicitly (20MB input cap is the bound) — fine as a stated trust decision.
  4. Formula injection — fixed and verified end-to-end. Generated an xlsx with =/+/-/@ cells and re-opened it: every one reads back as data_type='s' (string), never a formula; CSV prefixes a '. The new truncated flag is wired into the result and declared in the schema.
  5. Studio declarations hand-synced — this commit updated the affected descriptions/schemas by hand, but there is still no automated contract test comparing the declarations against the live function signatures. Non-blocking; it will drift again eventually, so a parameter/required-set parity test remains worth doing as a follow-up.
  6. Vision SSRF + aggregate budget — fixed and reproduced. vision._image_part now calls assert_public_url before forwarding a remote URL: 169.254.169.254 and 127.0.0.1 are rejected, a public URL is still forwarded with zero local bytes, and local images share a 40MB aggregate budget (per-image 20MB). DNS/file reads moved off the event loop.

Verification: runtime repros above, plus pytest tests/tools/builtin_tools/test_{http_request,image_transform,doc_generate,vision_tools}.py → 273 passed locally. Remaining unaddressed nits are cosmetic (http_request still sends no explicit User-Agent; temp files without a tool_context are not cleaned up) and don't affect merge.

Approving.

@wujiaming-ai
wujiaming-ai merged commit a2c339b into main Nov 16, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants