native2: never press a web row's nested control on a click (#195)

.lane/memory/native2/Sources/DesktopMac/Pointer.swift/01M5BM221JZ273DQQTVKJVZBQW-routed-events-name-the-windo.md

@@ -4,8 +4,9 @@ anchor: '@file'
 created: 2026-10-20T03:00:42Z
 norm: '1'
 sig: 27abcba5e975f16e
-body_hash: b18a2fb9835fe963
-raw_hash: 49a25acbe0d7524f
+body_hash: f71b819700237df6
+raw_hash: f635761a1c53b4f3
+vouched: 2026-10-24T23:52:56Z
 lines: 1-360
 ---
 

.lane/memory/native2/Sources/DesktopMac/Pointer.swift/01M5BVVPB930XQY34PWD5NYMMY-the-point-click-descendant-w.md

@@ -4,8 +4,9 @@ anchor: '@file'
 created: 2026-10-20T05:17:02Z
 norm: '1'
 sig: 27abcba5e975f16e
-body_hash: b18a2fb9835fe963
-raw_hash: 49a25acbe0d7524f
+body_hash: f71b819700237df6
+raw_hash: f635761a1c53b4f3
+vouched: 2026-10-24T23:41:12Z
 lines: 1-360
 ---
 

.lane/memory/native2/Sources/DesktopMac/Pointer.swift/01M5R4M887B8G2FFRZ216HDQ5B-webkit-s-axpress-on-an-aria.md

@@ -0,0 +1,8 @@
+---
+id: 01M5R4M887B8G2FFRZ216HDQ5B
+anchor: pressNestedSafely
+created: 2026-10-24T23:41:08Z
+norm: '1'
+---
+
+WebKit's AXPress on an ARIA menuitem/button/tab presses the first native <button>/<select>/checkbox inside it; never AXPress such a web element

.lane/memory/native2/Sources/DesktopMac/Pointer.swift/01M5R4M88EK12KWBEMDKDYA2H3-a-routed-single-click-into-a.md

@@ -0,0 +1,8 @@
+---
+id: 01M5R4M88EK12KWBEMDKDYA2H3
+anchor: pressNestedSafely
+created: 2026-10-24T23:41:08Z
+norm: '1'
+---
+
+a routed single click into a WKWebView window that isn't key in the active app is swallowed as first mouse, so it is only sent when the window is frontmost

CHANGELOG.md

@@ -24,6 +24,7 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no
 ### Changed
 
 - An agent run as a background job, with no terminal or app above it, is asked about as the nearest program signed by a developer team that runs it, such as Claude Code, and the prompt says it's a program, not an app. When there is no such program, the refusal says so and why, instead of only that the caller couldn't be identified. A caller's team counts only when Apple issued its certificate. Setup docs note that background jobs don't load nvm. ([#194](https://github.com/githubnext/desktop-tools/issues/194))
+- A background click on a web menu row, button or tab in a WebKit app, such as a WKWebView or Electrobun app or Safari, no longer presses a control nested inside it. WebKit's Accessibility press goes to the first native button inside the element, so a click on a project row in Ace pressed its hover-only remove button. Such an element now gets a pointer click at the point when its window is frontmost and is refused otherwise, and every press result names what was pressed in `pressed`. ([#195](https://github.com/githubnext/desktop-tools/issues/195))
 
 ## [0.17.1] - 2026-10-20
 

docs/tools.md

@@ -265,7 +265,7 @@ These are the rest of each tool's contract, beyond its description.
 - **After an action,** the automatic observation uses accessibility mode. If it fails, inspect again with pixels mode.
 - **In a Chromium or Electron app,** it first turns on the app's web accessibility, once per launch, unless `web_accessibility` is false. The result's `web_accessibility` says what happened; see [Input](#input).
 
-**`desktop_click`.** A single left click on a supported editable text field, by element or point, also requests keyboard focus. The result says whether focus was verified; use `desktop_select` to choose a range or caret position. Points stay bound to the captured window, even when the returned screenshot was scaled. A point click presses the Accessibility element there, and is refused before input when that element belongs to another window, so a covered window can't be clicked through. A sheet that Accessibility lists only inside its parent window, such as System Settings' Search Privacy sheet, or an Open panel on it, takes point clicks from a pixels snapshot of the sheet's own window.
+**`desktop_click`.** A single left click on a supported editable text field, by element or point, also requests keyboard focus. The result says whether focus was verified; use `desktop_select` to choose a range or caret position. Points stay bound to the captured window, even when the returned screenshot was scaled. A point click presses the Accessibility element there, and is refused before input when that element belongs to another window, so a covered window can't be clicked through. The result's `pressed` names the role and label of the element that got the press. WebKit presses a web menu item, button or tab through the first native button inside it, so one that holds a nested control, such as a menu row with a hover-only remove button, is never pressed through Accessibility: it gets a real click routed to its window at the point when that window is frontmost, and is otherwise refused, naming the `nested_control`. A sheet that Accessibility lists only inside its parent window, such as System Settings' Search Privacy sheet, or an Open panel on it, takes point clicks from a pixels snapshot of the sheet's own window.
 
 **`desktop_scroll`.** `amount` is in native scroll units. Their distance depends on the route the target supports, not on pixels.
 

native2/Sources/DesktopHost/Click.swift

@@ -29,7 +29,7 @@ extension Tools {
 			Hooks.showPointer(.click(at: global, kind: kind, window: window.windowID))
 			switch (kind, target) {
 			case (.single, let element?):
-				do { return try Mac.clickElement(element.reference, window: window) } catch is Unsupported {
+				do { return try Mac.clickElement(element.reference, window: window, at: global) } catch is Unsupported {
 					return try Mac.clickPoint(global, window: window)
 				}
 			case (.single, nil):

native2/Sources/DesktopMac/Pointer.swift

@@ -136,13 +136,16 @@ extension Mac {
 
 	/// An element click as Peekaboo's action route performs it: focus a text input, else AXPress, else
 	/// focus whatever accepts it. Throws Unsupported when none applies.
-	public static func clickElement(_ element: AXUIElement, window: ClickWindow) throws -> Delivery {
+	/// `point` is where the element is aimed at, its observed center; with it, a web element WebKit would press
+	/// through a nested control is clicked at that point instead (pressNestedSafely).
+	public static func clickElement(_ element: AXUIElement, window: ClickWindow, at point: CGPoint? = nil) throws -> Delivery {
 		AXUIElementSetMessagingTimeout(element, 1)
 		let role = AX.string(element, kAXRoleAttribute) ?? ""
 		if textInputRoles.contains(role), settable(element, kAXFocusedAttribute) {
 			return try focus(element, window: window)
 		}
 		if actions(element).contains(kAXPressAction) {
+			if let point { return try pressNestedSafely(element, role: role, at: point, window: window) }
 			return try press(element, window: window)
 		}
 		if focusableRoles.contains(role) && settable(element, kAXFocusedAttribute) || settable(element, kAXValueAttribute) {
@@ -193,7 +196,9 @@ extension Mac {
 		}
 		switch how {
 		case "focus": return try focus(element, window: window)
-		case "press": return try press(element, window: window)
+		case "press":
+			// A point click can land on a parent or child of what the caller saw there, so the result names it.
+			return try pressNestedSafely(element, role: AX.string(element, kAXRoleAttribute) ?? "", at: point, window: window)
 		default:
 			try window.check()
 			let error = AXUIElementSetAttributeValue(element, kAXSelectedAttribute as CFString, kCFBooleanTrue)
@@ -202,6 +207,116 @@ extension Mac {
 		}
 	}
 
+	// MARK: Presses WebKit would redirect
+
+	/// AXPress, unless WebKit would press a different control. WebCore's AccessibilityObject::press presses an
+	/// ARIA menu item, button, tab, toggle or list item through the first native button, select or checkbox inside
+	/// it (nativeActionElement), so pressing a menu row that holds a hover-only remove button presses remove (#195).
+	/// Such an element gets a real click routed to its window at the point instead, where the page's own hit
+	/// testing decides, and only when Accessibility finds the element itself at that point. The result names what
+	/// got the input.
+	static func pressNestedSafely(_ element: AXUIElement, role: String, at point: CGPoint, window: ClickWindow) throws -> Delivery {
+		let pressed = pressedJSON(element, role: role)
+		guard webPressRedirects(role: role), !routesThroughSkyLight(window.pid), AXWebTextMarkers(element: element) != nil,
+		      let control = nativeControl(in: element, role: { AX.string($0, kAXRoleAttribute) ?? "" }, children: { AX.value($0, kAXChildrenAttribute) as? [AXUIElement] ?? [] })
+		else {
+			var delivery = try press(element, window: window)
+			delivery.fields.append(("pressed", pressed))
+			return delivery
+		}
+		let nested = pressedJSON(control, role: AX.string(control, kAXRoleAttribute) ?? "")
+		let what = describe(element, role: role), inner = describe(control, role: AX.string(control, kAXRoleAttribute) ?? "")
+		let application = AXUIElementCreateApplication(window.pid)
+		AXUIElementSetMessagingTimeout(application, 1)
+		var hit: AXUIElement?
+		guard AXUIElementCopyElementAtPosition(application, Float(point.x), Float(point.y), &hit) == .success, let hit,
+		      isPlainPart(hit, of: element)
+		else {
+			var refusal = Refusal(
+				"DESKTOP_ACTION_FAILED",
+				"An Accessibility press on \(what) would press \(inner) inside it, and the point (\(Int(point.x)), \(Int(point.y))) is not on \(what) itself, so nothing was clicked.",
+				hint: "Inspect again and click a point on the visible part of \(what), away from \(inner); click \(inner) only to use it.",
+				fromHost: true
+			)
+			refusal.reason = "operation_unsupported"
+			refusal.fields = [("nested_control", nested)]
+			throw refusal
+		}
+		// WebKit drops the first click into a window that isn't key in the active app (acceptsFirstMouse), so a
+		// routed click reaches the page only when the window is in front.
+		guard NSRunningApplication(processIdentifier: window.pid)?.isActive == true,
+		      AX.element(AX.value(application, kAXFocusedWindowAttribute)).flatMap(AX.windowID) == window.windowID
+		else {
+			var refusal = Refusal(
+				"DESKTOP_ACTION_FAILED",
+				"An Accessibility press on \(what) would press \(inner) inside it, and a background pointer click doesn't reach this page while its window isn't the frontmost window, so nothing was clicked.",
+				hint: "Bring the window forward with desktop_focus (foreground) and click again, or reach \(what) with the keyboard.",
+				fromHost: true
+			)
+			refusal.reason = "operation_unsupported"
+			refusal.fields = [("nested_control", nested)]
+			throw refusal
+		}
+		var delivery = try routedClick(at: point, kind: .single, window: window)
+		delivery.fields += [
+			("pressed", pressed), ("nested_control", nested),
+			("note", JSON("Sent as a pointer click at the point, since an Accessibility press on \(what) would press \(inner) inside it.")),
+		]
+		return delivery
+	}
+
+	/// The roles WebCore's actionElement checks for a native control inside, as macOS names them.
+	static func webPressRedirects(role: String) -> Bool {
+		["AXMenuItem", "AXButton", "AXPopUpButton", "AXRadioButton", "AXCheckBox", "AXMenuButton", "AXGroup"].contains(role)
+	}
+
+	/// The first native-looking control below a node, depth first as WebCore searches, within 6 levels and 128 nodes.
+	/// Accessibility doesn't say which buttons are native, so any button, pop-up, checkbox or radio button counts.
+	static func nativeControl<Node>(in root: Node, role: (Node) -> String, children: (Node) -> [Node]) -> Node? {
+		var visited = 0
+		func search(_ node: Node, depth: Int) -> Node? {
+			guard depth < 6 else { return nil }
+			for child in children(node) {
+				visited += 1
+				if visited > 128 { return nil }
+				if ["AXButton", "AXPopUpButton", "AXCheckBox", "AXRadioButton", "AXMenuButton"].contains(role(child)) { return child }
+				if let found = search(child, depth: depth + 1) { return found }
+			}
+			return nil
+		}
+		return search(root, depth: 0)
+	}
+
+	/// Whether the element at a point is the element, or a part of it that is no control of its own.
+	private static func isPlainPart(_ hit: AXUIElement, of element: AXUIElement) -> Bool {
+		var current = hit
+		for _ in 0..<8 {
+			if CFEqual(current, element) { return true }
+			if ["AXButton", "AXPopUpButton", "AXCheckBox", "AXRadioButton", "AXMenuButton", "AXLink", "AXTextField", "AXTextArea"].contains(AX.string(current, kAXRoleAttribute) ?? "") {
+				return false
+			}
+			guard let parent = AX.element(AX.value(current, kAXParentAttribute)) else { return false }
+			current = parent
+		}
+		return false
+	}
+
+	private static func label(_ element: AXUIElement) -> String {
+		for attribute in [kAXTitleAttribute, kAXDescriptionAttribute, kAXValueAttribute] {
+			if let text = AX.string(element, attribute)?.trimmingCharacters(in: .whitespacesAndNewlines), !text.isEmpty { return text }
+		}
+		return ""
+	}
+
+	private static func pressedJSON(_ element: AXUIElement, role: String) -> JSON {
+		.object([("role", JSON(role)), ("label", JSON(label(element)))])
+	}
+
+	private static func describe(_ element: AXUIElement, role: String) -> String {
+		let text = label(element)
+		return text.isEmpty ? "the \(role)" : "the \(role) \"\(text)\""
+	}
+
 	/// AXShowMenu, which blocks while the menu is open; a menu still open after half a second is still running.
 	public static func showMenu(_ element: AXUIElement, window: ClickWindow) throws -> Delivery {
 		guard actions(element).contains("AXShowMenu") else { throw Unsupported() }

native2/Tests/RuntimeTests/PressRedirectTests.swift

@@ -0,0 +1,48 @@
+@testable import DesktopMac
+import Testing
+
+/// WebKit presses an ARIA menu item or button through the first native control inside it, so a point click on a
+/// menu row must not AXPress a row that holds a remove button (#195).
+struct PressRedirectTests {
+	final class Node {
+		let role: String
+		let children: [Node]
+		init(_ role: String, _ children: [Node] = []) {
+			self.role = role
+			self.children = children
+		}
+	}
+
+	static func control(in node: Node) -> Node? {
+		Mac.nativeControl(in: node, role: { $0.role }, children: { $0.children })
+	}
+
+	@Test
+	func `a menu row holding a remove button is found to redirect, depth first like WebCore`() {
+		let remove = Node("AXButton")
+		let row = Node("AXMenuItem", [Node("AXStaticText"), Node("AXGroup", [remove]), Node("AXCheckBox")])
+		#expect(Mac.webPressRedirects(role: "AXMenuItem"))
+		#expect(Self.control(in: row) === remove)
+	}
+
+	@Test
+	func `a row with only text, or a link, has no control to redirect to`() {
+		#expect(Self.control(in: Node("AXMenuItem", [Node("AXStaticText"), Node("AXGroup", [Node("AXImage")])])) == nil)
+		#expect(Self.control(in: Node("AXMenuItem", [Node("AXLink", [Node("AXStaticText")])])) == nil)
+	}
+
+	@Test
+	func `text fields and links are pressed directly, whatever they hold`() {
+		for role in ["AXTextField", "AXLink", "AXStaticText", "AXCell"] {
+			#expect(!Mac.webPressRedirects(role: role))
+		}
+	}
+
+	@Test
+	func `the search stops at six levels`() {
+		var node = Node("AXButton")
+		for _ in 0..<6 { node = Node("AXGroup", [node]) }
+		#expect(Self.control(in: Node("AXMenuItem", [node])) == nil)
+		#expect(Self.control(in: Node("AXMenuItem", node.children)) != nil)
+	}
+}