Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ anchor: '@file'
created: 2026-10-20T04:25:55Z
norm: '1'
sig: 9cb6c3e49e9e89a4
body_hash: b768f54da9b8c5a6
raw_hash: 5c185423536e6725
body_hash: d7ab3a923729569f
raw_hash: 490623e9c71688bc
vouched: 2026-10-24T23:36:33Z
lines: 1-831
---

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
id: 01M5R4BQAAV5GVFM0TEDFXYVBZ
anchor: fn textRead
created: 2026-10-24T23:36:29Z
norm: '1'
---

a scoped text read walks from the snapshot element's AXUIElement, so element depths start at 0 there; the table read still walks the window because its row IDs map to snapshot IDs by depth
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,12 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no
- The signed `desktop-tools-client` and `desktop-tools-mcp` check the host again before sending anything: a process at the socket path that isn't the same user's and signed by the same team gets no requests, and the call fails with `HOST_UNTRUSTED`. The Peekaboo-based client did this, and docs/embed.md still said so. ([#191](https://github.com/githubnext/desktop-tools/issues/191))
- The app and `desktop-tools-build --native2` ship Peekaboo's and AXorcist's MIT notices again, in `Licenses/`, since the Swift host contains code ported from them. docs/licenses.md describes native2. ([#192](https://github.com/githubnext/desktop-tools/issues/192))
- When `npx` runs `desktop-tools-mcp` against a Desktop Tools app of another version that has no MCP server of its own (before 0.17.0), every failed call names both versions and says to install the matching app, since updating the `npx` tag doesn't update the app. ([#179](https://github.com/githubnext/desktop-tools/issues/179))

### Changed

- An agent run as a background job, with no terminal or app above it, is asked about as the nearest program signed by a developer team that runs it, such as Claude Code, and the prompt says it's a program, not an app. When there is no such program, the refusal says so and why, instead of only that the caller couldn't be identified. A caller's team counts only when Apple issued its certificate. Setup docs note that background jobs don't load nvm. ([#194](https://github.com/githubnext/desktop-tools/issues/194))
- A background click on a web menu row, button or tab in a WebKit app, such as a WKWebView or Electrobun app or Safari, no longer presses a control nested inside it. WebKit's Accessibility press goes to the first native button inside the element, so a click on a project row in Ace pressed its hover-only remove button. Such an element now gets a pointer click at the point when its window is frontmost and is refused otherwise, and every press result names what was pressed in `pressed`. ([#195](https://github.com/githubnext/desktop-tools/issues/195))
- `desktop_read` with a `scope` walks that element's subtree directly, so a busy window can't spend the read's budget before reaching it, and returns the subtree's full text. A top-level `snapshot` and `element`, which desktop_read never had, are read as that scope instead of being ignored and reading the whole window; an `element` alone is refused with how to pass a scope. ([#204](https://github.com/githubnext/desktop-tools/issues/204))

## [0.17.1] - 2026-10-20

Expand Down
40 changes: 33 additions & 7 deletions native2/Sources/DesktopHost/TextRead.swift
Original file line number Diff line number Diff line change
Expand Up @@ -559,7 +559,15 @@ extension Tools {
guard let pid = arguments["pid"]?.int.flatMap(Int32.init(exactly:)), pid > 0,
let windowID = arguments["window"]?.int, windowID > 0
else { throw ToolError("Choose a positive application PID and window ID from the desktop inventory.") }
let scope = try arguments["scope"].map(scopeArgument)
var scope = try arguments["scope"].map(scopeArgument)
// desktop_read has no top-level element or snapshot; a caller passing them meant a scope, and reading the
// whole window instead would look like an answer (#204).
if arguments["element"] != nil || arguments["snapshot"] != nil {
guard scope == nil, let snapshot = arguments["snapshot"], let element = arguments["element"] else {
throw ToolError("Read one element's subtree with scope: the snapshot_id and an element ID from an earlier desktop_inspect of this window, as scope {snapshot, element}.")
}
scope = try scopeArgument(.object([("snapshot", snapshot), ("element", element)]))
}
var web = true
if let value = arguments["web_accessibility"] {
guard let flag = value.bool else { throw ToolError("Choose web_accessibility true or false.") }
Expand Down Expand Up @@ -614,19 +622,25 @@ extension Tools {
var source: ScopeSource?
if let scope { source = try await scopeSource(scope, pid: pid, windowID: windowID) }
let generation = try readTarget(pid: pid, windowID: windowID)
// A scope is walked from its own element, so a busy window can't spend the budget before the walk reaches
// it, and the subtree's text comes back whole (#204).
let root = try await scopeRoot(scope, source, generation: generation)
let (tree, change) = try await webRead(pid: pid, allowed: web, tree: { $0 }) {
try await run { try Mac.elements(pid: pid, windowID: windowID, budget: wideBudget) }
if let root {
return try await run { Mac.elements(root: root.reference, budget: wideBudget) }
}
return try await run { try Mac.elements(pid: pid, windowID: windowID, budget: wideBudget) }
}
guard Mac.generation(pid) == generation else {
throw ToolError("DESKTOP_ERROR: The native read did not confirm the requested process and window. Refresh the window list and try again.")
}
let elements = preorder(tree.elements)
var selected = elements
let selected = elements
var readScope: JSON?
if let scope, let source {
let position = try locateScope(source, in: elements, generation: generation)
selected = subtree(elements, at: position)
let root = elements[position]
if let scope, let root {
guard let first = elements.first, first.attributes["role"] == root.attributes["role"] else {
throw ToolError("DESKTOP_ERROR: The scope element is gone or no longer readable. Inspect the window again and choose the element from that result.")
}
readScope = .object([
"snapshot": JSON(scope.snapshot), "element": JSON(scope.element),
"role": root.attributes["role"].map(JSON.string), "label": root.label.map(JSON.string),
Expand Down Expand Up @@ -657,6 +671,18 @@ extension Tools {
])
}

/// The scope's element in its snapshot, whose live reference a scoped text read walks from.
private static func scopeRoot(_ scope: ScopeRequest?, _ source: ScopeSource?, generation: UInt64) async throws -> Element? {
guard let scope, let source else { return nil }
guard source.generation == generation else {
throw ToolError("DESKTOP_ERROR: The application restarted since the scope snapshot. Inspect the window again.")
}
guard let element = await Snapshots.shared.get(scope.snapshot)?.elements[scope.element] else {
throw ToolError("DESKTOP_ERROR: The scope snapshot no longer belongs to the running desktop. Inspect the window again.")
}
return element
}

/// The table built from a table read, with the read's window, scope and traversal limits.
private static func tableReading(_ read: TableRead, rowOffset: Int, maxRows: Int) -> ToolResult {
let table: [(String, JSON)]
Expand Down
9 changes: 9 additions & 0 deletions native2/Sources/DesktopMac/Elements.swift
Original file line number Diff line number Diff line change
Expand Up @@ -443,6 +443,15 @@ extension Mac {
return windows.contains { AX.windowID($0) == windowID } || openMenu(app, windowID: windowID) != nil
}

/// Walks the subtree under one element an earlier walk emitted, so the whole budget goes to that subtree.
/// Depths start at 0 at `root`. An empty tree means the element is gone or no longer readable.
public static func elements(root: AXUIElement, budget: ElementBudget) -> ElementTree {
let deadline = ContinuousClock.now + .seconds(budget.seconds - min(0.25, max(0.01, budget.seconds * 0.1)))
var walk = Walk(deadline: deadline, budget: budget)
walk.process(root, depth: 0)
return walk.tree
}

/// Peekaboo's focus receipt: the one element that says it has focus. Several can, as a focused table's cells
/// do, and then only a complete walk names one: the app's focused element, unchanged since before the walk.
static func focusedID(_ tree: ElementTree, app: AXUIElement, before: AXUIElement?) -> String? {
Expand Down
Loading