From 0bababb06ef3ac04ed74a25abd84e8e7793b3542 Mon Sep 17 00:00:00 2001 From: iamnbutler Date: Sat, 24 Oct 2026 19:44:34 -0400 Subject: [PATCH 1/2] native2: inspect spends its budget on what's on screen first (#178) The walk sets aside elements entirely outside the window's frame, with their subtrees, and reads them after everything on screen, then numbers elements in tree pre-order as before. Set-aside children don't count toward the per-node child limit. A result over the size limit drops offscreen elements first, and offscreen_omitted counts what was left out. Why: native2/Sources/DesktopMac/Elements.swift#fn walk | with visibleFirst, elements are emitted out of order and renumbered by their child-index path at the end; anything that reads element IDs during the walk sees provisional IDs --- ...XFH8WD7ADT-ui-elements-are-ordered-by-p.md | 0 ...XMVG0X39TG-ui-elements-are-ordered-by-p.md | 9 +++ CHANGELOG.md | 2 +- native2/Sources/DesktopCore/Results.swift | 56 ++++++------- native2/Sources/DesktopHost/Inspect.swift | 20 ++++- native2/Sources/DesktopMac/Elements.swift | 80 ++++++++++++++++--- native2/Tests/CoreTests/CoreTests.swift | 12 +++ 7 files changed, 137 insertions(+), 42 deletions(-) rename .lane/{memory => attic}/native2/Sources/DesktopHost/Inspect.swift/01M5BKJWVVGTA2MXXFH8WD7ADT-ui-elements-are-ordered-by-p.md (100%) create mode 100644 .lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md diff --git a/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5BKJWVVGTA2MXXFH8WD7ADT-ui-elements-are-ordered-by-p.md b/.lane/attic/native2/Sources/DesktopHost/Inspect.swift/01M5BKJWVVGTA2MXXFH8WD7ADT-ui-elements-are-ordered-by-p.md similarity index 100% rename from .lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5BKJWVVGTA2MXXFH8WD7ADT-ui-elements-are-ordered-by-p.md rename to .lane/attic/native2/Sources/DesktopHost/Inspect.swift/01M5BKJWVVGTA2MXXFH8WD7ADT-ui-elements-are-ordered-by-p.md diff --git a/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md b/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md new file mode 100644 index 00000000..98e5e804 --- /dev/null +++ b/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md @@ -0,0 +1,9 @@ +--- +id: 01M5R4TGZ74Z2041XMVG0X39TG +anchor: '@file' +created: 2026-10-24T23:44:34Z +norm: '1' +supersedes: 01M5BKJWVVGTA2MXXFH8WD7ADT +--- + +ui_elements are ordered by Peekaboo's type groups with menus first. When bounding, elements outside the window are dropped first, then non-scope roles, then scope roles (#178); this deliberately differs from Node's truncation diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e730ad6..efa45c20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,6 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no - `desktop_inspect` takes its screenshot after reading the window's Accessibility tree, not during it. Reading some SwiftUI lists makes them load rows and grow, which moved content after the screenshot was taken, so a point read off it could click whatever moved under it. An inspection takes about as long as its tree read longer. ([#159](https://github.com/githubnext/desktop-tools/issues/159)) - `return` is accepted as a key name wherever keys are named (`desktop_key`, `desktop_system_key`), as an alias of `enter`. `desktop_system_key` results name the app that was frontmost when the keys were sent (`frontmost`), and `desktop_wait`'s description states its argument rules: `element_appears` and `element_gone` need an element with a role plus a title, value or identifier, and `timeout_ms` is an integer from 100 to 45000. ([#199](https://github.com/githubnext/desktop-tools/issues/199)) - A background point click on a tab in Chrome's vertical tab strip switches to that tab. Chrome lays empty groups over its vertical tabs, and the Accessibility hit test returns one of those, so the click was refused with "No pressable accessibility element". When the element at the point is such an empty group, the click now presses the smallest pressable element beneath it, such as the tab or its close button. ([#202](https://github.com/githubnext/desktop-tools/issues/202)) - - An open menu's window, such as a SwiftUI `MenuBarExtra` status item's NSMenu or one of its submenus, can be inspected: `desktop_inspect` reads the open `AXMenu` and lists its items with `checked` for a checkmark, so they can be clicked by element. `desktop_menubar_click` on an item whose menu is open closes the menu without choosing an item (`closed_menu`), and `desktop_element_action` `AXCancel` on the menu closes it too. A window opened by a status item is `inspectable` only when it has Accessibility elements to read. ([#193](https://github.com/githubnext/desktop-tools/issues/193)) - Typing refused because secure keyboard entry is on now names the app and window holding it (`secure_input_holder`: pid, name, bundle ID, window title), read from the window server's session, such as a system password dialog behind the target, and asks to have the user dismiss it. ([#176](https://github.com/githubnext/desktop-tools/issues/176)) - `desktop_hover` and foreground `desktop_drag` no longer count a window that lets the pointer through, such as a screen recorder's frame, as covering the target: the cover check asks the window server which window takes pointer events at the point. An unverified hover or a cover refusal now names the window found there. `desktop_focus` and the other window and app tools say which `target` field is missing or wrong, such as a pid and window passed at the top level, and a window tool no longer asks for an application's target. ([#196](https://github.com/githubnext/desktop-tools/issues/196)) @@ -29,6 +28,7 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no - An agent run as a background job, with no terminal or app above it, is asked about as the nearest program signed by a developer team that runs it, such as Claude Code, and the prompt says it's a program, not an app. When there is no such program, the refusal says so and why, instead of only that the caller couldn't be identified. A caller's team counts only when Apple issued its certificate. Setup docs note that background jobs don't load nvm. ([#194](https://github.com/githubnext/desktop-tools/issues/194)) - A background click on a web menu row, button or tab in a WebKit app, such as a WKWebView or Electrobun app or Safari, no longer presses a control nested inside it. WebKit's Accessibility press goes to the first native button inside the element, so a click on a project row in Ace pressed its hover-only remove button. Such an element now gets a pointer click at the point when its window is frontmost and is refused otherwise, and every press result names what was pressed in `pressed`. ([#195](https://github.com/githubnext/desktop-tools/issues/195)) - `desktop_read` with a `scope` walks that element's subtree directly, so a busy window can't spend the read's budget before reaching it, and returns the subtree's full text. A top-level `snapshot` and `element`, which desktop_read never had, are read as that scope instead of being ignored and reading the whole window; an `element` alone is refused with how to pass a scope. ([#204](https://github.com/githubnext/desktop-tools/issues/204)) +- `desktop_inspect` spends its element budget on what's inside the window first. Elements entirely outside the window, such as a long page's content scrolled past or below the fold, are walked only after everything on screen, and a result over the size limit drops them before any on-screen element. `offscreen_omitted` counts the offscreen elements left out. ([#178](https://github.com/githubnext/desktop-tools/issues/178)) ## [0.17.1] - 2026-10-20 diff --git a/native2/Sources/DesktopCore/Results.swift b/native2/Sources/DesktopCore/Results.swift index da5d5872..727bf134 100644 --- a/native2/Sources/DesktopCore/Results.swift +++ b/native2/Sources/DesktopCore/Results.swift @@ -26,38 +26,40 @@ public struct ToolResult: Sendable { } } -/// Fits `field` within `limit` bytes by dropping its last values, those `keep` accepts only after all others. The -/// limit applies to the compact result a model reads; the text returned stays whole for internal reads. -public func bounded(_ data: JSON, field: String, limit: Int = maxText, keep: ((JSON) -> Bool)? = nil) throws -> String { +/// Fits `field` within `limit` bytes by dropping its last values, those `keep` accepts only after all others. With +/// `dropFirst`, the values its predicate accepts go before any others, and how many went is added to its count +/// field. The limit applies to the compact result a model reads; the text returned stays whole for internal reads. +public func bounded( + _ data: JSON, field: String, limit: Int = maxText, dropFirst: (predicate: (JSON) -> Bool, count: String)? = nil, + keep: ((JSON) -> Bool)? = nil +) throws -> String { guard let source = data[field]?.array else { throw ToolError("Native inspection returned no \(field) array.") } - var values = source - var value = data - value.set(field, .array(values)) func size(_ value: JSON) -> Int { compact(value, allWindows: true).text.utf8.count } - func truncated(_ kept: [JSON]) -> JSON { + guard size(data) > limit else { return data.text } + // The order values are dropped in: by tier, and within a tier from the end. + let tier = { (value: JSON) -> Int in + (dropFirst?.predicate(value) == true ? 0 : 2) + (keep?(value) == true ? 1 : 0) + } + let tiers = source.map(tier) + let order = source.indices.sorted { tiers[$0] != tiers[$1] ? tiers[$0] < tiers[$1] : $0 > $1 } + func truncated(dropping count: Int) -> JSON { + let dropped = Set(order.prefix(count)) var shorter = data - shorter.set(field, .array(kept)) + shorter.set(field, .array(source.indices.filter { !dropped.contains($0) }.map { source[$0] })) shorter.set("truncated", true) - shorter.set("omitted", JSON(source.count - kept.count)) - return shorter - } - // Without `keep`, dropping from the end keeps the longest prefix that fits, so search for it - // rather than writing the whole result once per dropped value. - if keep == nil, size(value) > limit { - var fits = 0 - var over = values.count - while over - fits > 1 { - let mid = (fits + over) / 2 - if size(truncated(Array(values[.. 0 { shorter.set(dropFirst.count, JSON((data[dropFirst.count]?.int ?? 0) + first)) } } - values = Array(values[.. limit, !values.isEmpty { - let index = keep.flatMap { keep in values.lastIndex { !keep($0) } } ?? values.count - 1 - values.remove(at: index) - value = truncated(values) + // Dropping more never grows the result, so search for the fewest drops that fit. + guard size(truncated(dropping: source.count)) <= limit else { throw ToolError("Native inspection metadata exceeds the result limit.") } + var fits = source.count, over = 0 + while fits - over > 1 { + let mid = (fits + over) / 2 + if size(truncated(dropping: mid)) <= limit { fits = mid } else { over = mid } } - guard size(value) <= limit else { throw ToolError("Native inspection metadata exceeds the result limit.") } - return value.text + return truncated(dropping: fits).text } diff --git a/native2/Sources/DesktopHost/Inspect.swift b/native2/Sources/DesktopHost/Inspect.swift index 6ba28a75..102ca09b 100644 --- a/native2/Sources/DesktopHost/Inspect.swift +++ b/native2/Sources/DesktopHost/Inspect.swift @@ -98,7 +98,23 @@ extension Tools { /// An observation bounded to the result limit, keeping the containers an agent can scope into. static func boundedObservation(_ data: JSON, limit: Int = maxText) throws -> String { - try bounded(data, field: "ui_elements", limit: limit) { element in + // Elements entirely outside the window go first, since acting on them needs a scroll anyway (#178). + let bounds = data["coordinate_context"]?["logical_bounds"]?.array + let window = bounds.flatMap { pair -> CGRect? in + guard pair.count == 2, let origin = pair[0].array, let size = pair[1].array, origin.count == 2, size.count == 2, + let x = origin[0].number, let y = origin[1].number, let width = size[0].number, let height = size[1].number + else { return nil } + return CGRect(x: x, y: y, width: width, height: height) + } + let offscreen: ((JSON) -> Bool)? = window.map { window in + { element in + guard let frame = element["bounds"], let x = frame["x"]?.number, let y = frame["y"]?.number, + let width = frame["width"]?.number, let height = frame["height"]?.number + else { return false } + return !CGRect(x: x, y: y, width: width, height: height).intersects(window) + } + } + return try bounded(data, field: "ui_elements", limit: limit, dropFirst: offscreen.map { ($0, "offscreen_omitted") }) { element in element["attributes"]?["role"]?.string.map(scopeRoles.contains) ?? false } } @@ -127,6 +143,7 @@ extension Tools { if let scope = request.scope { source = try await scopeSource(scope, pid: pid, windowID: windowID) } var budget = ElementBudget() if source != nil { (budget.maxDepth, budget.maxElements, budget.maxChildren, budget.seconds) = (64, 4000, 1000, 18) } + budget.visibleFirst = true let accessibility = !request.pixels let ((window, image, elements, captureDone), web) = try await webRead(pid: pid, allowed: accessibility && request.web, tree: \.2) { try await read(pid: pid, windowID: windowID, generation: generation, budget: accessibility ? budget : nil, started: started) @@ -176,6 +193,7 @@ extension Tools { "element_count": JSON(source == nil ? elements.elements.count : listed.count), "scope": scope, "traversed_element_count": source.map { _ in JSON(elements.elements.count) }, + "offscreen_omitted": elements.offscreenSkipped > 0 ? JSON(elements.offscreenSkipped) : nil, "ui_elements": .array(listed.map(json)), "coordinate_context": .object([ ("version", 1), diff --git a/native2/Sources/DesktopMac/Elements.swift b/native2/Sources/DesktopMac/Elements.swift index f8e1bcb4..1b9dd800 100644 --- a/native2/Sources/DesktopMac/Elements.swift +++ b/native2/Sources/DesktopMac/Elements.swift @@ -12,6 +12,10 @@ public struct ElementBudget: Sendable { public var maxElements = 200 public var maxChildren = 100 public var seconds: Double = 15 + /// Walk what intersects the window first: an element entirely outside the window's frame, with its subtree, + /// is read only after everything inside it, so a long page's offscreen content can't spend the budget (#178). + /// Element IDs still number the result in tree pre-order. + public var visibleFirst = false public init() {} } @@ -38,6 +42,8 @@ public struct ElementTree: @unchecked Sendable { public var maxChildrenPerNodeReached = false public var deadlineReached = false public var incompleteAccessibilityRead = false + /// With `visibleFirst`, elements outside the window that the budget left unread; their descendants aren't counted. + public var offscreenSkipped = 0 public init() {} @@ -210,6 +216,11 @@ private struct Walk { var tree = ElementTree() var visited: [AXUIElement] = [] var columnHeaders: [AXUIElement] = [] + /// Each emitted element's position in the tree, as child indices from the root, which orders them in pre-order. + var paths: [[Int]] = [] + /// The window's frame, once read, when offscreen subtrees wait for a second pass. + var visible: CGRect? + var deferred: [(element: AXUIElement, depth: Int, path: [Int])] = [] /// Only Notification Center's custom action names are listed; elsewhere they are arbitrary app text. var listsCustomActions = false @@ -268,18 +279,56 @@ private struct Walk { return (result, false) } - mutating func process(_ element: AXUIElement, depth: Int) { - guard ContinuousClock.now < deadline else { tree.deadlineReached = true; return } - guard depth < budget.maxDepth else { tree.maxDepthReached = true; return } - guard tree.elements.count < budget.maxElements else { tree.maxElementCountReached = true; return } - guard !visited.contains(where: { CFEqual($0, element) }) else { return } - visited.append(element) + /// Returns true when it set the element aside for the second pass. + @discardableResult + mutating func process(_ element: AXUIElement, depth: Int, path: [Int] = [], deferring: Bool = false) -> Bool { + guard ContinuousClock.now < deadline else { tree.deadlineReached = true; return false } + guard depth < budget.maxDepth else { tree.maxDepthReached = true; return false } + guard tree.elements.count < budget.maxElements else { tree.maxElementCountReached = true; return false } + guard !visited.contains(where: { CFEqual($0, element) }) else { return false } AXUIElementSetMessagingTimeout(element, timeout() ?? 0.01) let (read, incomplete) = descriptor(element) + if deferring, depth > 0, !incomplete, let read, let visible, !read.frame.intersects(visible) { + deferred.append((element, depth, path)) + return true + } + visited.append(element) if incomplete { tree.incompleteAccessibilityRead = true } - if let descriptor = read, !incomplete { emit(descriptor, element, depth: depth) } - processChildren(element, depth: depth) + if let descriptor = read, !incomplete { + if depth == 0, budget.visibleFirst { visible = descriptor.frame } + emit(descriptor, element, depth: depth) + paths.append(path) + } + processChildren(element, depth: depth, path: path, deferring: deferring) + return false + } + + /// Walks from `root`, then, with `visibleFirst`, the offscreen subtrees it set aside, and numbers the elements + /// in tree pre-order. + mutating func walk(_ root: AXUIElement) { + process(root, depth: 0, deferring: budget.visibleFirst) + guard !deferred.isEmpty else { return } + let waiting = deferred + deferred = [] + for (index, item) in waiting.enumerated() { + guard tree.elements.count < budget.maxElements, ContinuousClock.now < deadline else { + tree.offscreenSkipped = waiting.count - index + if tree.elements.count >= budget.maxElements { tree.maxElementCountReached = true } else { tree.deadlineReached = true } + break + } + process(item.element, depth: item.depth, path: item.path) + } + let order = paths.indices.sorted { paths[$0].lexicographicallyPrecedes(paths[$1]) } + var elements: [Element] = [] + for (position, index) in order.enumerated() { + let old = tree.elements[index] + let id = "elem_\(position)" + elements.append(Element(id: id, type: old.type, label: old.label, value: old.value, frame: old.frame, isEnabled: old.isEnabled, + isSelected: old.isSelected, attributes: old.attributes, reference: old.reference)) + } + tree.elements = elements + paths = order.map { paths[$0] } } mutating func emit(_ descriptor: Descriptor, _ element: AXUIElement, depth: Int) { @@ -342,13 +391,18 @@ private struct Walk { )) } - mutating func processChildren(_ element: AXUIElement, depth: Int) { + mutating func processChildren(_ element: AXUIElement, depth: Int, path: [Int], deferring: Bool) { let read = children(element) if read.incomplete { tree.incompleteAccessibilityRead = true } - if read.elements.count > budget.maxChildren { tree.maxChildrenPerNodeReached = true } - let limited = read.elements.prefix(budget.maxChildren) + // Set-aside children don't count toward the per-node limit, so a long list scrolled to its middle reaches + // the rows on screen; it scans at most ten times the limit. + let sorting = deferring && visible != nil + let limited = read.elements.prefix(sorting ? budget.maxChildren * 10 : budget.maxChildren) + if read.elements.count > limited.count { tree.maxChildrenPerNodeReached = true } + var taken = 0 for (index, child) in limited.enumerated() { - process(child, depth: depth + 1) + guard taken < budget.maxChildren else { tree.maxChildrenPerNodeReached = true; return } + if !process(child, depth: depth + 1, path: path + [index], deferring: deferring) { taken += 1 } guard index < limited.count - 1 else { break } if tree.elements.count >= budget.maxElements { tree.maxElementCountReached = true; return } if ContinuousClock.now >= deadline { tree.deadlineReached = true; return } @@ -382,7 +436,7 @@ extension Mac { var walk = Walk(deadline: deadline, budget: budget) walk.listsCustomActions = CustomActions.isNotificationCenter(pid) walk.tree.windowTitle = AX.string(window, kAXTitleAttribute) - walk.process(window, depth: 0) + walk.walk(window) if walk.tree.elements.isEmpty { throw ToolError("internalError: Exact window \(windowID) returned no usable Accessibility elements. Warning: AX tree incomplete at incomplete accessibility read. Retry once after a short wait.") } diff --git a/native2/Tests/CoreTests/CoreTests.swift b/native2/Tests/CoreTests/CoreTests.swift index cf96e2ec..2d986c18 100644 --- a/native2/Tests/CoreTests/CoreTests.swift +++ b/native2/Tests/CoreTests/CoreTests.swift @@ -40,6 +40,18 @@ import Testing #expect(try JSON(parsing: text)["items"] == ["keep", "keep"]) } + @Test func dropsFirstValuesBeforeOthers() throws { + let filler = { (name: String) in JSON(name + String(repeating: "x", count: 20)) } + let data: JSON = .object([("items", [filler("a"), filler("off1"), "keep", filler("b"), filler("off2")])]) + let text = try bounded(data, field: "items", limit: 110, dropFirst: ({ $0.string?.hasPrefix("off") == true }, "offscreen")) { + $0 == "keep" + } + let result = try JSON(parsing: text) + #expect(result["items"] == [filler("a"), "keep", filler("b")]) + #expect(result["omitted"]?.int == 2) + #expect(result["offscreen"]?.int == 2) + } + @Test func leavesSmallResultsAlone() throws { let data: JSON = .object([("items", [1, 2])]) #expect(try bounded(data, field: "items") == #"{"items":[1,2]}"#) From 9f886abcb93bbccf8c2cca8d11fe3eb13f1cc510 Mon Sep 17 00:00:00 2001 From: iamnbutler Date: Sat, 24 Oct 2026 19:44:43 -0400 Subject: [PATCH 2/2] lane: sync fix-inspect-offscreen memory --- ...R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md | 4 ++++ ...R4TH8PZ9DG1WYQFPN470XT-with-visiblefirst-elements-a.md | 8 ++++++++ 2 files changed, 12 insertions(+) create mode 100644 .lane/memory/native2/Sources/DesktopMac/Elements.swift/01M5R4TH8PZ9DG1WYQFPN470XT-with-visiblefirst-elements-a.md diff --git a/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md b/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md index 98e5e804..e5634fea 100644 --- a/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md +++ b/.lane/memory/native2/Sources/DesktopHost/Inspect.swift/01M5R4TGZ74Z2041XMVG0X39TG-ui-elements-are-ordered-by-p.md @@ -3,6 +3,10 @@ id: 01M5R4TGZ74Z2041XMVG0X39TG anchor: '@file' created: 2026-10-24T23:44:34Z norm: '1' +sig: 9cb6c3e49e9e89a4 +body_hash: 746fb88a0eadce35 +raw_hash: cce651be3a11598d +lines: 1-325 supersedes: 01M5BKJWVVGTA2MXXFH8WD7ADT --- diff --git a/.lane/memory/native2/Sources/DesktopMac/Elements.swift/01M5R4TH8PZ9DG1WYQFPN470XT-with-visiblefirst-elements-a.md b/.lane/memory/native2/Sources/DesktopMac/Elements.swift/01M5R4TH8PZ9DG1WYQFPN470XT-with-visiblefirst-elements-a.md new file mode 100644 index 00000000..36b65b53 --- /dev/null +++ b/.lane/memory/native2/Sources/DesktopMac/Elements.swift/01M5R4TH8PZ9DG1WYQFPN470XT-with-visiblefirst-elements-a.md @@ -0,0 +1,8 @@ +--- +id: 01M5R4TH8PZ9DG1WYQFPN470XT +anchor: fn walk +created: 2026-10-24T23:44:34Z +norm: '1' +--- + +with visibleFirst, elements are emitted out of order and renumbered by their child-index path at the end; anything that reads element IDs during the walk sees provisional IDs