diff --git a/.lane/memory/native2/Sources/DesktopMac/Keyboard.swift/01M5R5PTGK64T7AGEX72PSHF40-a-sandboxed-open-panel-repor.md b/.lane/memory/native2/Sources/DesktopMac/Keyboard.swift/01M5R5PTGK64T7AGEX72PSHF40-a-sandboxed-open-panel-repor.md new file mode 100644 index 00000000..ff6fca07 --- /dev/null +++ b/.lane/memory/native2/Sources/DesktopMac/Keyboard.swift/01M5R5PTGK64T7AGEX72PSHF40-a-sandboxed-open-panel-repor.md @@ -0,0 +1,8 @@ +--- +id: 01M5R5PTGK64T7AGEX72PSHF40 +anchor: fn requireKeyboardFocus +created: 2026-10-25T00:00:01Z +norm: '1' +--- + +a sandboxed Open panel reports no AXFocusedUIElement for its app, and keys posted to the app or the panel service pid don't reach it in the background (checked live); background clicks do work diff --git a/CHANGELOG.md b/CHANGELOG.md index b8aa8fdf..f70a399e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,9 +15,7 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no - An open menu's window, such as a SwiftUI `MenuBarExtra` status item's NSMenu or one of its submenus, can be inspected: `desktop_inspect` reads the open `AXMenu` and lists its items with `checked` for a checkmark, so they can be clicked by element. `desktop_menubar_click` on an item whose menu is open closes the menu without choosing an item (`closed_menu`), and `desktop_element_action` `AXCancel` on the menu closes it too. A window opened by a status item is `inspectable` only when it has Accessibility elements to read. ([#193](https://github.com/githubnext/desktop-tools/issues/193)) - Typing refused because secure keyboard entry is on now names the app and window holding it (`secure_input_holder`: pid, name, bundle ID, window title), read from the window server's session, such as a system password dialog behind the target, and asks to have the user dismiss it. ([#176](https://github.com/githubnext/desktop-tools/issues/176)) - `desktop_hover` and foreground `desktop_drag` no longer count a window that lets the pointer through, such as a screen recorder's frame, as covering the target: the cover check asks the window server which window takes pointer events at the point. An unverified hover or a cover refusal now names the window found there. `desktop_focus` and the other window and app tools say which `target` field is missing or wrong, such as a pid and window passed at the top level, and a window tool no longer asks for an application's target. ([#196](https://github.com/githubnext/desktop-tools/issues/196)) - -### Fixed - +- A sandboxed app's Open or Save panel is described accurately: the open and save panel service's own windows, including Go to Folder, are listed as `open_save_panel_service` with a hint to use the app's panel window, and keys into the panel, which background input can't reach, are refused with how to click it or enter a path with `desktop_dialog_answer` instead of a generic focus refusal. A new recipe, "Choose a file or folder in an Open panel", walks through it. ([#200](https://github.com/githubnext/desktop-tools/issues/200)) - `desktop_scroll` scrolls web content it used to refuse with "no supported route". A Chrome or other Chromium page gets arrow keys, one per unit, at the window's keyboard focus when that focus is on the page and not in a field, and the result reads back that the page moved; with focus in a field it is refused with a hint to click the page or send `desktop_key` `pagedown`. A WKWebView app that loads WebKit from a library beside its executable, such as an Electrobun app, gets the background wheel Safari gets. ([#197](https://github.com/githubnext/desktop-tools/issues/197), [#201](https://github.com/githubnext/desktop-tools/issues/201)) - `desktop_type` into a web field reads the field back until it settles, up to 2 seconds, so a busy page such as a React filter is confirmed instead of reported `TYPING_UNVERIFIED`. When the field still doesn't match, the error says what it read back, in `read_back`. Replacing a Chrome field's text no longer appends to it: Chrome ignores a background Command-A, so the field's text is also selected through Accessibility, and typing stops with the field's contents reported when it can't be cleared. ([#203](https://github.com/githubnext/desktop-tools/issues/203)) - The signed `desktop-tools-client` and `desktop-tools-mcp` check the host again before sending anything: a process at the socket path that isn't the same user's and signed by the same team gets no requests, and the call fails with `HOST_UNTRUSTED`. The Peekaboo-based client did this, and docs/embed.md still said so. ([#191](https://github.com/githubnext/desktop-tools/issues/191)) diff --git a/docs/recipes.md b/docs/recipes.md index 320bdbb0..ee986af0 100644 --- a/docs/recipes.md +++ b/docs/recipes.md @@ -32,6 +32,24 @@ If the app has **File > Export as PDF…**, as Preview, Pages, Safari and Notes - More than one dialog open in the app makes `desktop_dialog` refuse; pass the window target. - An existing file at the path makes the panel ask to replace it. Read it with `desktop_dialog` and answer deliberately. +## Choose a file or folder in an Open panel + +An app's **File > Open…** (⌘O), or its "add a project" or "choose folder" button, shows an Open panel. In a sandboxed app the panel's content is drawn by the open and save panel service, a separate process, while the app lists the panel as its own window. + +1. Open the panel: `desktop_menu` with the app target and `path: ["File", "Open…"]`, or click the app's own button. Expect `completed`. +2. `desktop_wait` with `until: "dialog_appears"` and `pid`. Expect `matched`. +3. `desktop_dialog` with the app target. Expect `is_file_dialog` true, the panel's buttons such as **Open** and **Cancel**, and its window target. Use that window target for the next calls. +4. `desktop_dialog_answer` with `path: "/Users/me/code/project"`. This is foreground: it activates the app and types the path into Go to Folder. Expect `completed` once the panel reads back the folder; an Open panel's file choice is `unknown`, so read the panel again with `desktop_dialog`. +5. `desktop_dialog_answer` with `button: "Open"`, or the panel's default button's title (some apps name it Choose or Add). + +**Verify:** `desktop_wait` with `until: "dialog_gone"`, then check the app shows the file or folder. + +**Goes wrong:** + +- The panel service's own window is listed under its process, not the app's, with availability `open_save_panel_service`. Act on the app's window with the same title and bounds instead. +- Background keys don't reach the panel's content, so `desktop_key` (such as ⇧⌘G for Go to Folder) and typing are refused. Clicks do work in the background: inspect the app's panel window and `desktop_click` a sidebar location or a row, then press **Open** with `desktop_dialog_answer`. +- The panel's sidebar can fill much of an inspection; `desktop_dialog` lists the buttons and text fields directly. + ## Pick a color in the Colors panel 1. Select what the color applies to: for text in TextEdit, `desktop_select` the text. diff --git a/native2/Sources/DesktopCore/Availability.swift b/native2/Sources/DesktopCore/Availability.swift index 3ebe4195..8cdb8a56 100644 --- a/native2/Sources/DesktopCore/Availability.swift +++ b/native2/Sources/DesktopCore/Availability.swift @@ -8,6 +8,10 @@ public func windowAvailability(_ window: JSON, sessionLocked: Bool, appHidden: B return .object([("state", "unavailable"), ("reason", "session_locked"), ("hint", "The macOS GUI session is locked. Unlock the active user session, then observe the screen before retrying.")]) } + if window["observation_reason"]?.string == "open_save_panel_service" { + return .object([("state", "unavailable"), ("reason", "open_save_panel_service"), + ("hint", "This window belongs to the service that draws another app's Open or Save panel, and to its sheets such as Go to Folder. Use the app that opened the panel: its desktop_windows lists the panel with the same title and bounds, which desktop_inspect and desktop_click reach in the background. To go to a folder or file, pass the app's target to desktop_dialog, then desktop_dialog_answer with path.")]) + } if window["is_minimized"]?.bool == true { return .object([("state", "unavailable"), ("reason", "minimized"), ("hint", JSON("The window is minimized. Restore it with desktop_restore, using its target from desktop_windows, \(reinspect)"))]) diff --git a/native2/Sources/DesktopMac/Keyboard.swift b/native2/Sources/DesktopMac/Keyboard.swift index 039a86ac..b3b43ea6 100644 --- a/native2/Sources/DesktopMac/Keyboard.swift +++ b/native2/Sources/DesktopMac/Keyboard.swift @@ -138,9 +138,22 @@ extension Mac { "Invalid target: Exact-window keyboard focus is not on the target window at dispatch. " + backgroundFocusRemediation, hint: "Observe the intended target before retrying this operation." ) - guard let focusedWindow = AX.element(AX.value(app, kAXFocusedWindowAttribute)), AX.windowID(focusedWindow) == window.windowID, - let focused = AX.element(AX.value(app, kAXFocusedUIElementAttribute)) + guard let focusedWindow = AX.element(AX.value(app, kAXFocusedWindowAttribute)), AX.windowID(focusedWindow) == window.windowID else { throw refusal } + guard let focused = AX.element(AX.value(app, kAXFocusedUIElementAttribute)) else { + // A sandboxed app's Open or Save panel is drawn by another process, which holds its focus and doesn't + // take background keys (#200). + if hostsRemoteContent(focusedWindow, pid: window.pid) { + var remote = Refusal( + "DESKTOP_ACTION_FAILED", + "The window's keyboard focus is in content another process draws, such as a sandboxed app's Open or Save panel, which background keys don't reach. Nothing was sent.", + hint: "Click the panel's elements with desktop_click, which works in the background. To go to a folder or file, use desktop_dialog with the app's target, then desktop_dialog_answer with path, which types it in the foreground." + ) + remote.reason = "operation_unsupported" + throw remote + } + throw refusal + } if let sheets = AX.value(focusedWindow, "AXSheets") as? [AXUIElement], !sheets.isEmpty { throw refusal } AXUIElementSetMessagingTimeout(focused, 0.2) if let owner = AX.element(AX.value(focused, kAXWindowAttribute)), AX.windowID(owner) != window.windowID { throw refusal } @@ -154,6 +167,22 @@ extension Mac { return focused } + /// Whether some element under `window` belongs to another process, as a remote view's content does. + static func hostsRemoteContent(_ window: AXUIElement, pid: pid_t) -> Bool { + var queue = [window] + var read = 0 + while !queue.isEmpty, read < 200 { + let element = queue.removeFirst() + read += 1 + AXUIElementSetMessagingTimeout(element, 0.2) + for child in AX.value(element, kAXChildrenAttribute) as? [AXUIElement] ?? [] { + if let owner = AX.pid(child), owner > 0, owner != pid { return true } + queue.append(child) + } + } + return false + } + /// Whether the focused element is a password field, which only a password fill may type into. static func secureFocus(_ focused: AXUIElement) -> Bool { AX.string(focused, kAXRoleAttribute) == "AXSecureTextField" || AX.string(focused, kAXSubroleAttribute) == "AXSecureTextField" diff --git a/native2/Sources/DesktopMac/Windows.swift b/native2/Sources/DesktopMac/Windows.swift index adb1e931..4862562a 100644 --- a/native2/Sources/DesktopMac/Windows.swift +++ b/native2/Sources/DesktopMac/Windows.swift @@ -57,14 +57,16 @@ private struct Row { private func plural(_ n: Int, _ noun: String) -> String { "\(n) \(noun)\(n == 1 ? "" : "s")" } +let openSavePanelService = "com.apple.appkit.xpc.openAndSavePanelService" + extension Mac { /// One app's windows, as desktop_windows lists them: CoreGraphics' rows enriched by Accessibility, then /// windows only Accessibility knows. Each carries the evidence a later action checks before it acts. public static func windows(pid: pid_t) async throws -> JSON { - let app = await MainActor.run { () -> (name: String, hidden: Bool)? in + let app = await MainActor.run { () -> (name: String, hidden: Bool, bundle: String?)? in guard let app = NSWorkspace.shared.runningApplications.first(where: { $0.processIdentifier == pid && !$0.isTerminated }) else { return nil } - return (app.localizedName ?? app.bundleIdentifier ?? "Unknown", app.isHidden) + return (app.localizedName ?? app.bundleIdentifier ?? "Unknown", app.isHidden, app.bundleIdentifier) } guard let app else { throw ToolError("notFound: Application 'PID:\(pid)' not found.") } guard let generation = startIdentity(pid).value else { @@ -94,7 +96,11 @@ extension Mac { var omittedCG = 0 var seen = Set() if merged { - let unmatched = axIncomplete + // The open and save panel service draws a sandboxed app's Open or Save panel and its sheets, such as Go to + // Folder, for that app, which lists the panel as its own window; Accessibility reaches it only there (#200). + let unmatched = app.bundle == openSavePanelService + ? ("pixels_only", "open_save_panel_service") + : axIncomplete ? ("unknown", "accessibility_enumeration_incomplete") : ("pixels_only", "no_matching_accessibility_window") for cg in owned where seen.insert(cg.id).inserted { diff --git a/native2/Tests/CoreTests/CoreTests.swift b/native2/Tests/CoreTests/CoreTests.swift index e7b29bd7..cf96e2ec 100644 --- a/native2/Tests/CoreTests/CoreTests.swift +++ b/native2/Tests/CoreTests/CoreTests.swift @@ -54,6 +54,13 @@ import Testing #expect(result["space_id"] == 7) } + @Test func panelServiceWindowPointsToTheApp() { + let row: JSON = .object([("is_on_screen", false), ("observation_capability", "pixels_only"), ("observation_reason", "open_save_panel_service")]) + let result = windowAvailability(row, sessionLocked: false, appHidden: false) + #expect(result["reason"] == "open_save_panel_service") + #expect(result["state"] == "unavailable") + } + @Test func lockedComesFirst() { let row: JSON = .object([("is_minimized", true)]) #expect(windowAvailability(row, sessionLocked: true, appHidden: true)["reason"] == "session_locked")