Skip to content

feat: add semantic UI snapshots#40

Merged
kitlangton merged 1 commit into
mainfrom
feat/semantic-snapshot
Jul 19, 2026
Merged

feat: add semantic UI snapshots#40
kitlangton merged 1 commit into
mainfrom
feat/semantic-snapshot

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

What

Expose OpenCode's versioned semantic UI tree through Drive.

Programs can now call ui.snapshot(), poll one exact node with ui.getNode(), and pass the resulting semantic node to ui.click(). Nodes carry stable IDs, optional occurrence identity, role, label, hierarchy, component-owned state, and a transient renderer handle.

Semantic snapshots and atomic semantic clicks are negotiated independently. Older OpenCode checkouts keep ordinary UI control, while unsupported semantic operations fail locally with typed UiCapabilityError instead of sending an unsafe request.

How

  • Copy the canonical protocol-v1 semantic schemas, ui.snapshot RPC, and ui.click.semantic capability from OpenCode.
  • Add typed snapshot and exact polling operations, immediate ambiguity errors, capability errors, and semantic-node click support to driver/ui.ts.
  • Carry { id, instance, element } on semantic clicks so OpenCode validates live identity before mouse dispatch.
  • Add --command.ui.snapshot with the same optional-capability guard as the programmatic API.
  • Document the API in the package README, driver reference, and agent skill.
  • Add a minor Changeset for opencode-drive.

Scope

This PR consumes the canonical producer shipped by OpenCode PRs #37802 and #37808. It does not add semantic IDs for more TUI components or implement the separate arbitrary-tool lifecycle protocol.

Testing

  • bun run release:validate in packages/drive
  • 162 Effect tests passed
  • 53 CLI tests passed
  • lint and typecheck passed
  • dry-run package inspection passed with 94 expected files
  • bun run test in apps/catalog: 28 passed
  • Focused stale-handle, capability-negotiation, semantic-click identity, and exact-node tests passed
  • OpenCode producer CI passed on Linux and Windows
  • git diff --check

Flow

sequenceDiagram
    participant Program as Drive program
    participant Drive
    participant OpenCode as OpenCode UI endpoint
    participant TUI

    Drive->>OpenCode: simulation.handshake(optional ui.snapshot, ui.click.semantic)
    Program->>Drive: ui.getNode(query)
    loop until one exact match
        Drive->>OpenCode: ui.snapshot
        OpenCode->>TUI: read live semantic tree
        TUI-->>Drive: versioned nodes
    end
    Program->>Drive: ui.click(node)
    Drive->>OpenCode: ui.click(target, position, semantic identity)
    OpenCode->>TUI: validate identity and dispatch click
    TUI-->>Program: updated UI state
Loading

@kitlangton
kitlangton merged commit 7caebeb into main Jul 19, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant