feat(vom): expose record-safe semantic observations - #100
Draft
shnpd wants to merge 1 commit into
Draft
Conversation
Share the VOM capture pipeline with recording, enrich rendered refs, and support value redaction without regressing cooperative cancellation.
This was referenced Aug 17, 2026
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
apps/extension/src/tools/capture-vom-observation.ts:121
- The function currently always returns
captured: captured.nodes. IfincludeCapturedstays optional/opt-in, the return value should conditionally include it, and whenredactValuesis enabled it should scrubformValueto avoid leaking user-entered data.
return {
text: rendered.text,
refs: rendered.refs,
truncated: rendered.truncated,
captured: captured.nodes,
packages/vom/src/render.ts:637
redactValuescurrently masksnode.valuefor all nodes, not just fillable form controls. SinceVomOptions.redactValuesis documented as masking “form values”, enabling it could also hide useful non-form values (e.g., sliders/progress values) that are not sensitive.
const value =
node.sensitive || redactValues
? rawValue !== undefined
? SENSITIVE_MASK
: undefined
apps/extension/src/tools/capture-vom-observation.ts:4
- This file introduces a runtime circular import:
observation.tsimportscaptureVomObservationfrom this module, while this module importsbuildVomScenefromobservation.ts. Circular module dependencies are fragile (especially if any bundling/transpilation outputs CJS) and make it easy to accidentally introduce initialization-order bugs later.
import { type RenderedRef, renderVom, type VomOptions } from "@browser-skill/vom";
import type { CdpAxNode } from "./observation";
import { buildVomScene } from "./observation";
import type { CdpRunner } from "./shared";
shnpd
marked this pull request as draft
August 18, 2026 08:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
录制功能需要保存页面的语义观察,以便将操作步骤与对应页面元素准确关联。现有 VOM 捕获流程内嵌在
snapshot/observe中,无法直接被录制模块复用,同时录制结果也需要避免包含用户输入等敏感数据。主要改动
本 PR 抽取了共享的
captureVomObservation,统一处理 Accessibility Tree 获取、页面信息捕获、VOM Scene 构建和语义文本渲染。现有snapshot/observe已迁移到该接口,并保留原有的取消、失败降级、Overlay 排除及 Conditional Surface 等行为。VOM 渲染新增了统一隐藏表单值的能力。启用后会保留字段的填写状态,但不会输出邮箱、手机号、密码等真实内容。密码、验证码和信用卡相关字段也会被自动识别为敏感字段。
此外,渲染后的元素引用新增了角色、名称、上下文和文本行号,方便录制模块将用户操作与 VOM 中对应的语义元素关联。
兼容性
redactValues默认关闭,不影响现有调用方snapshot/observe的协议返回结构保持不变ref和backendNodeId测试
补充了表单值脱敏、敏感字段识别、引用语义信息和文本行号相关测试,并调整现有测试以兼容扩展后的引用结构。