native2: describe out-of-process Open panels accurately (#200)

.lane/memory/native2/Sources/DesktopMac/Keyboard.swift/01M5R5PTGK64T7AGEX72PSHF40-a-sandboxed-open-panel-repor.md

@@ -0,0 +1,8 @@
+---
+id: 01M5R5PTGK64T7AGEX72PSHF40
+anchor: fn requireKeyboardFocus
+created: 2026-10-25T00:00:01Z
+norm: '1'
+---
+
+a sandboxed Open panel reports no AXFocusedUIElement for its app, and keys posted to the app or the panel service pid don't reach it in the background (checked live); background clicks do work

CHANGELOG.md

@@ -15,9 +15,7 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no
 - An open menu's window, such as a SwiftUI `MenuBarExtra` status item's NSMenu or one of its submenus, can be inspected: `desktop_inspect` reads the open `AXMenu` and lists its items with `checked` for a checkmark, so they can be clicked by element. `desktop_menubar_click` on an item whose menu is open closes the menu without choosing an item (`closed_menu`), and `desktop_element_action` `AXCancel` on the menu closes it too. A window opened by a status item is `inspectable` only when it has Accessibility elements to read. ([#193](https://github.com/githubnext/desktop-tools/issues/193))
 - Typing refused because secure keyboard entry is on now names the app and window holding it (`secure_input_holder`: pid, name, bundle ID, window title), read from the window server's session, such as a system password dialog behind the target, and asks to have the user dismiss it. ([#176](https://github.com/githubnext/desktop-tools/issues/176))
 - `desktop_hover` and foreground `desktop_drag` no longer count a window that lets the pointer through, such as a screen recorder's frame, as covering the target: the cover check asks the window server which window takes pointer events at the point. An unverified hover or a cover refusal now names the window found there. `desktop_focus` and the other window and app tools say which `target` field is missing or wrong, such as a pid and window passed at the top level, and a window tool no longer asks for an application's target. ([#196](https://github.com/githubnext/desktop-tools/issues/196))
-
-### Fixed
-
+- A sandboxed app's Open or Save panel is described accurately: the open and save panel service's own windows, including Go to Folder, are listed as `open_save_panel_service` with a hint to use the app's panel window, and keys into the panel, which background input can't reach, are refused with how to click it or enter a path with `desktop_dialog_answer` instead of a generic focus refusal. A new recipe, "Choose a file or folder in an Open panel", walks through it. ([#200](https://github.com/githubnext/desktop-tools/issues/200))
 - `desktop_scroll` scrolls web content it used to refuse with "no supported route". A Chrome or other Chromium page gets arrow keys, one per unit, at the window's keyboard focus when that focus is on the page and not in a field, and the result reads back that the page moved; with focus in a field it is refused with a hint to click the page or send `desktop_key` `pagedown`. A WKWebView app that loads WebKit from a library beside its executable, such as an Electrobun app, gets the background wheel Safari gets. ([#197](https://github.com/githubnext/desktop-tools/issues/197), [#201](https://github.com/githubnext/desktop-tools/issues/201))
 - `desktop_type` into a web field reads the field back until it settles, up to 2 seconds, so a busy page such as a React filter is confirmed instead of reported `TYPING_UNVERIFIED`. When the field still doesn't match, the error says what it read back, in `read_back`. Replacing a Chrome field's text no longer appends to it: Chrome ignores a background Command-A, so the field's text is also selected through Accessibility, and typing stops with the field's contents reported when it can't be cleared. ([#203](https://github.com/githubnext/desktop-tools/issues/203))
 - The signed `desktop-tools-client` and `desktop-tools-mcp` check the host again before sending anything: a process at the socket path that isn't the same user's and signed by the same team gets no requests, and the call fails with `HOST_UNTRUSTED`. The Peekaboo-based client did this, and docs/embed.md still said so. ([#191](https://github.com/githubnext/desktop-tools/issues/191))

docs/recipes.md

@@ -32,6 +32,24 @@ If the app has **File > Export as PDF…**, as Preview, Pages, Safari and Notes
 - More than one dialog open in the app makes `desktop_dialog` refuse; pass the window target.
 - An existing file at the path makes the panel ask to replace it. Read it with `desktop_dialog` and answer deliberately.
 
+## Choose a file or folder in an Open panel
+
+An app's **File > Open…** (⌘O), or its "add a project" or "choose folder" button, shows an Open panel. In a sandboxed app the panel's content is drawn by the open and save panel service, a separate process, while the app lists the panel as its own window.
+
+1. Open the panel: `desktop_menu` with the app target and `path: ["File", "Open…"]`, or click the app's own button. Expect `completed`.
+2. `desktop_wait` with `until: "dialog_appears"` and `pid`. Expect `matched`.
+3. `desktop_dialog` with the app target. Expect `is_file_dialog` true, the panel's buttons such as **Open** and **Cancel**, and its window target. Use that window target for the next calls.
+4. `desktop_dialog_answer` with `path: "/Users/me/code/project"`. This is foreground: it activates the app and types the path into Go to Folder. Expect `completed` once the panel reads back the folder; an Open panel's file choice is `unknown`, so read the panel again with `desktop_dialog`.
+5. `desktop_dialog_answer` with `button: "Open"`, or the panel's default button's title (some apps name it Choose or Add).
+
+**Verify:** `desktop_wait` with `until: "dialog_gone"`, then check the app shows the file or folder.
+
+**Goes wrong:**
+
+- The panel service's own window is listed under its process, not the app's, with availability `open_save_panel_service`. Act on the app's window with the same title and bounds instead.
+- Background keys don't reach the panel's content, so `desktop_key` (such as ⇧⌘G for Go to Folder) and typing are refused. Clicks do work in the background: inspect the app's panel window and `desktop_click` a sidebar location or a row, then press **Open** with `desktop_dialog_answer`.
+- The panel's sidebar can fill much of an inspection; `desktop_dialog` lists the buttons and text fields directly.
+
 ## Pick a color in the Colors panel
 
 1. Select what the color applies to: for text in TextEdit, `desktop_select` the text.

native2/Sources/DesktopCore/Availability.swift

@@ -8,6 +8,10 @@ public func windowAvailability(_ window: JSON, sessionLocked: Bool, appHidden: B
 		return .object([("state", "unavailable"), ("reason", "session_locked"),
 		                ("hint", "The macOS GUI session is locked. Unlock the active user session, then observe the screen before retrying.")])
 	}
+	if window["observation_reason"]?.string == "open_save_panel_service" {
+		return .object([("state", "unavailable"), ("reason", "open_save_panel_service"),
+		                ("hint", "This window belongs to the service that draws another app's Open or Save panel, and to its sheets such as Go to Folder. Use the app that opened the panel: its desktop_windows lists the panel with the same title and bounds, which desktop_inspect and desktop_click reach in the background. To go to a folder or file, pass the app's target to desktop_dialog, then desktop_dialog_answer with path.")])
+	}
 	if window["is_minimized"]?.bool == true {
 		return .object([("state", "unavailable"), ("reason", "minimized"),
 		                ("hint", JSON("The window is minimized. Restore it with desktop_restore, using its target from desktop_windows, \(reinspect)"))])

native2/Sources/DesktopMac/Keyboard.swift

@@ -138,9 +138,22 @@ extension Mac {
 			"Invalid target: Exact-window keyboard focus is not on the target window at dispatch. " + backgroundFocusRemediation,
 			hint: "Observe the intended target before retrying this operation."
 		)
-		guard let focusedWindow = AX.element(AX.value(app, kAXFocusedWindowAttribute)), AX.windowID(focusedWindow) == window.windowID,
-		      let focused = AX.element(AX.value(app, kAXFocusedUIElementAttribute))
+		guard let focusedWindow = AX.element(AX.value(app, kAXFocusedWindowAttribute)), AX.windowID(focusedWindow) == window.windowID
 		else { throw refusal }
+		guard let focused = AX.element(AX.value(app, kAXFocusedUIElementAttribute)) else {
+			// A sandboxed app's Open or Save panel is drawn by another process, which holds its focus and doesn't
+			// take background keys (#200).
+			if hostsRemoteContent(focusedWindow, pid: window.pid) {
+				var remote = Refusal(
+					"DESKTOP_ACTION_FAILED",
+					"The window's keyboard focus is in content another process draws, such as a sandboxed app's Open or Save panel, which background keys don't reach. Nothing was sent.",
+					hint: "Click the panel's elements with desktop_click, which works in the background. To go to a folder or file, use desktop_dialog with the app's target, then desktop_dialog_answer with path, which types it in the foreground."
+				)
+				remote.reason = "operation_unsupported"
+				throw remote
+			}
+			throw refusal
+		}
 		if let sheets = AX.value(focusedWindow, "AXSheets") as? [AXUIElement], !sheets.isEmpty { throw refusal }
 		AXUIElementSetMessagingTimeout(focused, 0.2)
 		if let owner = AX.element(AX.value(focused, kAXWindowAttribute)), AX.windowID(owner) != window.windowID { throw refusal }
@@ -154,6 +167,22 @@ extension Mac {
 		return focused
 	}
 
+	/// Whether some element under `window` belongs to another process, as a remote view's content does.
+	static func hostsRemoteContent(_ window: AXUIElement, pid: pid_t) -> Bool {
+		var queue = [window]
+		var read = 0
+		while !queue.isEmpty, read < 200 {
+			let element = queue.removeFirst()
+			read += 1
+			AXUIElementSetMessagingTimeout(element, 0.2)
+			for child in AX.value(element, kAXChildrenAttribute) as? [AXUIElement] ?? [] {
+				if let owner = AX.pid(child), owner > 0, owner != pid { return true }
+				queue.append(child)
+			}
+		}
+		return false
+	}
+
 	/// Whether the focused element is a password field, which only a password fill may type into.
 	static func secureFocus(_ focused: AXUIElement) -> Bool {
 		AX.string(focused, kAXRoleAttribute) == "AXSecureTextField" || AX.string(focused, kAXSubroleAttribute) == "AXSecureTextField"

native2/Sources/DesktopMac/Windows.swift

@@ -57,14 +57,16 @@ private struct Row {
 
 private func plural(_ n: Int, _ noun: String) -> String { "\(n) \(noun)\(n == 1 ? "" : "s")" }
 
+let openSavePanelService = "com.apple.appkit.xpc.openAndSavePanelService"
+
 extension Mac {
 	/// One app's windows, as desktop_windows lists them: CoreGraphics' rows enriched by Accessibility, then
 	/// windows only Accessibility knows. Each carries the evidence a later action checks before it acts.
 	public static func windows(pid: pid_t) async throws -> JSON {
-		let app = await MainActor.run { () -> (name: String, hidden: Bool)? in
+		let app = await MainActor.run { () -> (name: String, hidden: Bool, bundle: String?)? in
 			guard let app = NSWorkspace.shared.runningApplications.first(where: { $0.processIdentifier == pid && !$0.isTerminated })
 			else { return nil }
-			return (app.localizedName ?? app.bundleIdentifier ?? "Unknown", app.isHidden)
+			return (app.localizedName ?? app.bundleIdentifier ?? "Unknown", app.isHidden, app.bundleIdentifier)
 		}
 		guard let app else { throw ToolError("notFound: Application 'PID:\(pid)' not found.") }
 		guard let generation = startIdentity(pid).value else {
@@ -94,7 +96,11 @@ extension Mac {
 			var omittedCG = 0
 			var seen = Set<Int>()
 			if merged {
-				let unmatched = axIncomplete
+				// The open and save panel service draws a sandboxed app's Open or Save panel and its sheets, such as Go to
+				// Folder, for that app, which lists the panel as its own window; Accessibility reaches it only there (#200).
+				let unmatched = app.bundle == openSavePanelService
+					? ("pixels_only", "open_save_panel_service")
+					: axIncomplete
 					? ("unknown", "accessibility_enumeration_incomplete")
 					: ("pixels_only", "no_matching_accessibility_window")
 				for cg in owned where seen.insert(cg.id).inserted {

native2/Tests/CoreTests/CoreTests.swift

@@ -54,6 +54,13 @@ import Testing
 		#expect(result["space_id"] == 7)
 	}
 
+	@Test func panelServiceWindowPointsToTheApp() {
+		let row: JSON = .object([("is_on_screen", false), ("observation_capability", "pixels_only"), ("observation_reason", "open_save_panel_service")])
+		let result = windowAvailability(row, sessionLocked: false, appHidden: false)
+		#expect(result["reason"] == "open_save_panel_service")
+		#expect(result["state"] == "unavailable")
+	}
+
 	@Test func lockedComesFirst() {
 		let row: JSON = .object([("is_minimized", true)])
 		#expect(windowAvailability(row, sessionLocked: true, appHidden: true)["reason"] == "session_locked")