…refs
The web/agent-browser backend mints refs in tree order and skips
non-interactive nodes, so its `@eN` refs are not dense. agent-device was
dropping that ref in `normalizeAgentBrowserSnapshot` and then re-minting a
dense positional `e${index+1}` in `attachRefs`. The ref an agent reads off
the snapshot (dense, positional) therefore did not equal the ref the backend
resolves on the next action (tree-ordered). On the ShopDemo login screen the
username textbox displayed as one ref while the backend's ref for the same
position pointed at the passcode field, so `fill @e3` landed in the wrong
input.
Preserve the backend ref on each web node and make `attachRefs` keep a
node's existing `ref` when present, falling back to dense numbering for
backends that do not mint refs (iOS/Android/maestro are unaffected).
Summary
Fixes a web ref mismatch where the ref an agent reads off a snapshot does not equal the ref the agent-browser backend resolves on the next action, causing taps/fills to land in the wrong element.
Repro
On the ShopDemo web login screen (driven through
agent-devicewith the web platform):snapshotshows the username textbox as one ref and the passcode textbox as the next ref.fill @<ref> "..."aimed at the username lands in the passcode field (verified against live CDP DOM ground truth:#login-usernamestayed empty while#login-passwordreceived the text).Root cause
The web/agent-browser backend mints refs in tree order and skips non-interactive nodes (e.g. the leading
genericcontainer), so its@eNrefs are not dense — there are gaps.agent-device was dropping that backend ref in
normalizeAgentBrowserSnapshotand then re-minting a dense positionale${index+1}inattachRefs. The ref an agent reads off the rendered snapshot (dense, by printed position) therefore did not equal the ref the backend resolves on the next action (tree-ordered, by its own map). When a node was skipped, the two numbering schemes drifted apart and the same@eNstring referred to two different elements on the two sides of the round trip.Fix
packages/kernel/src/snapshot.ts: add an optionalreftoRawSnapshotNodeand makeattachRefskeep a node's existingrefwhen present, falling back to densee${index+1}otherwise. Backends that don't mint refs (iOS/Android/maestro) are unaffected — they still get dense numbering.packages/platform-web/src/agent-browser-snapshot.ts: carry each web node's backend ref (draft.ref, which agent-browser already uses for box/geometry fetches) onto the node so the displayed ref equals the actionable ref.No code derives an index from a ref's numeric value, so non-dense refs are safe downstream (
findNodeByRef/normalizeRefoperate on the ref string).Tests
packages/kernel/src/snapshot-attach-refs.test.ts(new): preserves a backend ref; re-mints dense refs for nodes without one.packages/platform-web/.../agent-browser-snapshot.test.ts: new case reproducing the non-dense login shape (Usernamee2, Passcodee3, Sign-ine4) and asserting the refs survive normalization.packages/platform-web/.../agent-browser-provider.test.ts:expectedNodenow carries the preservedref.Full
pnpm typecheckgreen;unit-coresuites forpackages/platform-web,packages/kernel, core snapshot, interaction, and CLI snapshot-output tests all pass.