Capture the native element tree per action in Selenium and Nightwatch

.changeset/native-page-source-capture.md

@@ -0,0 +1,6 @@
+---
+"@wdio/selenium-devtools": patch
+"@wdio/nightwatch-devtools": patch
+---
+
+Capture the element list and accessibility tree per action on native Appium sessions. In trace mode each action of a native app now reads the app's page source beside its screenshot, as the WebdriverIO service does, so a native trace carries element data, locators and the A11y tab instead of a screenshot alone. The page source is the costly part of a native capture, most of it on iOS, and it shows as the `pageSource` probe in the `DEVTOOLS_CAPTURE_TIMING=1` report, which now marks these captures native.

CLAUDE.md

@@ -397,7 +397,8 @@ Documented divergences from the conventions above. They exist today as debt to b
 ### What a per-action trace capture costs
 
 - **Measure before proposing a change: `DEVTOOLS_CAPTURE_TIMING=1`.** Every adapter then times each probe of each capture (issue and settle, so a probe's own service time is separable from its wait behind the probes the driver served first) and, when the trace is written, prints a per-probe table and appends the raw records to `capture-timing-<pid>.ndjson` beside the trace. The JS recording is `core/capture-timing.ts`, wired once in `captureActionSnapshot` and reported from `finalizeTraceExport`, so no adapter carries more than the capture's trigger (action, wait poll, matcher poll, final); Python mirrors it in `capture_timing.py`, held to the same summary by `mirror-vectors.json`. Off, it costs a flag read per capture. The per-platform commands are in the README's "Measuring capture cost".
-- **One capture per action means two driver round trips per action, and the platform decides which half is expensive.** Measured on Appium 3.7.0 / UiAutomator2 (emulator-5554, Android API 37, 1080×2424): `GET /screenshot` 1.18–2.22 s (steady ~1.20 s) at 1.86 MB, `GET /source` (page-source XML) 0.09–0.53 s at 40 KB, `window/rect` 0.015 s — the screenshot dominates by ~10× on Android. **iOS is unmeasured**; the issue claims the split inverts there, and the native example (`examples/wdio/mocha/wdio.native.conf.ts`) is what makes it measurable. Measured per action: two captures 2.41 s against one capture 1.19 s; end-to-end on the native example 19.1 s against 12.4 s, with live mode (no per-action capture) at 5.9 s. That is the case for the `beforeCommand` capture, and the reason an eager post-action capture is not worth re-adding: it cost a second capture per action and existed only to be patched by a `readyState` poll that could not tell a document that had not navigated yet from one that was loading.
+- **One capture per action means two driver round trips per action, and the platform decides which half is expensive.** Measured on Appium 3.7.0 / UiAutomator2 (emulator-5554, Android API 37, 1080×2424): `GET /screenshot` 1.18–2.22 s (steady ~1.20 s) at 1.86 MB, `GET /source` (page-source XML) 0.09–0.53 s at 40 KB, `window/rect` 0.015 s; the screenshot dominates by ~10× on Android. A later timing run (v11.2.0, `DEVTOOLS_CAPTURE_TIMING=1`, a loaded machine, so rough) split an Android WebdriverIO capture about evenly between the two (384-471 ms per capture) and put the page source at **817 ms of a 938 ms iOS capture (87%)**: the split inverts on iOS. Measured per action: two captures 2.41 s against one capture 1.19 s; end-to-end on the native example 19.1 s against 12.4 s, with live mode (no per-action capture) at 5.9 s. That is the case for the `beforeCommand` capture, and the reason an eager post-action capture is not worth re-adding: it cost a second capture per action and existed only to be patched by a `readyState` poll that could not tell a document that had not navigated yet from one that was loading.
+- **A native capture is the screenshot plus the page source, in every JS adapter.** A native app serves no page script, url or title, so core's `captureActionSnapshot` builds the element list, the accessibility tree and the locators from the page-source XML instead (`fromPageSource`), and an adapter only supplies the reader: the service `getPageSource` (or its direct probe), Selenium the unpatched original (`getDriverOriginals().getPageSource`, so the read is never a row of the user's), Nightwatch `GET /source` over the raw WebDriver transport. The Selenium and Nightwatch adapters ask core's `SessionCapturer.nativeApp` (`nativeAppOf`), which carries the platform the XML is read by (none for a Mac2 or tvOS app, read as Android). A reader passed is also what marks the capture `native` in the timing report and times its `pageSource` probe. The read is the costly half of a native capture (above), so trace mode on iOS pays for it on every action. Python takes the screenshot alone on a native app (`_action_capture_enabled` skips the page reads and reads no source), so its native traces carry no element list or accessibility tree.
 - **Locator generation from a native page source must stay linear, and iOS is where it did not.** Every native action snapshot builds element locators from the page-source XML (`core/locators`), and each uniqueness check was a `//*[@name="…"]` XPath run by the `xpath` library over the whole document, several per element. Measured on the iOS Settings page source tiled up: 38 KB 82 ms, 188 KB 1.4 s, 752 KB 23.5 s, which is what a reporter saw as trace mode adding 1.5 min to a 2 min 20 s iOS test while Android, with smaller flatter trees, added 10 s. `locators/xpath-index.ts` indexes the document once per `generateAllElementLocators` walk (`withXPathIndex`) and answers `//*`, `//Tag` and `//Tag[@a="x" and …]` from it: 15 ms, 49 ms, 268 ms for the same three, identical elements. It is scoped to the walk so a caller of the exported `evaluateXPath` that edits its document is never answered stale, and it declines a namespaced document, where XPath name tests resolve namespaces and the index does not. The page-source request itself remains: 0.75-1.2 s per action on XCUITest even for Settings, against 0.09-0.5 s on UiAutomator2.
 - A filmstrip poll against a native session stacks requests: `setInterval` never waits for its async handler, a native screenshot takes ~1.2 s against the 200 ms default, and a serialised driver serves that queue ahead of the test's own commands — measured, a **15 ms command took 4.5–7.8 s**. `ScreencastRecorderBase`'s `#pollInFlight` latch bounds it to one outstanding shot, and `#pollGeneration` stops a shot orphaned by `stop()` from appending after the stop or clearing its successor's latch.
 - **`start()` and `stop()` are serialised against each other, because unserialised neither owns what it arms.** A `stop()` landing mid-handshake returned at its `#isRecording` guard — still false, the handshake not having finished — so it never reached its own CDP teardown, and the `start()` it interrupted then returned on the generation mismatch without stopping what it had armed: the session and its frame listener stayed live after teardown. The frames themselves were moot — every adapter builds a fresh recorder per session (`screencast-lifecycle.ts` does so even across `reloadSession`), so the leaked session and its browser-side stream were the damage, not buffer pollution. The obvious repair — stopping the session on that mismatch path — is wrong, and it is the shape to keep out: it guards a field both CDP overrides write at handshake **start** (`#cdpSession` / `#cdp`) with a flag they set only at handshake **end** (`#cdpActive`), so a start arriving in that window is indistinguishable from no start at all and the stale path tears down the *newer* recording's session instead. Overlapping starts are the adapters' normal mode rather than a corner: Selenium's `driverPatcher` documents the second test's body racing an in-flight `screencast.start()`, and Nightwatch's session rotation is fire-and-forget from a command hook. Queued, a stop always runs against a start that has finished arming and owns what it armed, which makes the mismatch branch unreachable and deleted — the tick-phase generation checks in `#startPolling` are **not**, since a shot orphaned by a stop still lands afterwards. Two consequences: `stop()` now waits on an in-flight handshake, and every awaited driver primitive in that handshake is ceilinged at `SCREENCAST_HANDSHAKE_TIMEOUT_MS` (the service's `getPuppeteer()`/`pages()`/`createCDPSession()`/`Page.startScreencast` and the stop-side `Page.stopScreencast`, Selenium's `createCDPConnection`, the polling path's first screenshot) — a driver that wedges gives up after the ceiling, falls back to polling, or reports the screencast unavailable when polling was what wedged. Nothing is claimed until the handshake has answered, so a timed-out start leaves nothing armed — and a session or connection that completes after the ceiling is detached (or its socket closed) when it lands, so no orphan outlives the recording. And a stop landing during a polling session's first screenshot now records that shot before tearing the loop down, where it previously cancelled the start outright; the interval is still never left armed underneath the stop.

README.md

@@ -380,7 +380,7 @@ services: [[DevToolsHookService, {
 
 #### Measuring capture cost
 
-A trace-mode run is slower than a live one because, for every action, the adapter reads the page through the driver: url, title, accessibility tree, element list and a screenshot, plus the page source on a native app. Set `DEVTOOLS_CAPTURE_TIMING=1` on a trace-mode run to see where that time goes. It only measures; nothing is captured differently, and with it unset nothing is timed or written.
+A trace-mode run is slower than a live one because, for every action, the adapter reads the page through the driver: url, title, accessibility tree, element list and a screenshot. A native app has none of the page reads: its capture is a screenshot plus the page source, from which the element list and accessibility tree are built, and the page source is the costly half (about half of an Android capture, and 87% of an iOS one, measured on WebdriverIO). Set `DEVTOOLS_CAPTURE_TIMING=1` on a trace-mode run to see where that time goes. It only measures; nothing is captured differently, and with it unset nothing is timed or written.
 
 When the trace is written, the run prints a table to stderr and appends the raw records, one line per capture, to `capture-timing-<pid>.ndjson` beside the trace (one file per worker process):
 
@@ -400,7 +400,7 @@ raw records: /out/capture-timing-1.ndjson
 - **Columns.** The JS adapters issue every read of a capture at once and the driver serves them in order, so a read's time from issue to settle includes waiting behind the reads before it. `svc` subtracts that wait: it is the read's own cost. `process` is the in-process step that turns the reads into a snapshot (page-source parsing and locator generation on native). `capture` is the whole capture.
 - **Busy and wall.** `busy` is the time at least one capture was running, overlapping captures counted once; `wall` is the time since the process started, or since the previous report in the same process.
 - **Triggers.** `action` is a capture for a user command, `wait-poll` one for a command a wait issued, `matcher-poll` one taken while a WebdriverIO `expect` matcher was polling, and `final` the WebdriverIO capture of the last action's result. Matcher polls are told apart on WebdriverIO only, and only with `captureAssertions` on; Selenium and Nightwatch matchers expose no hook to tell them by.
-- **Python.** It reads one thing at a time, so `svc` equals the read's duration; it records the screenshot and the two page reads, and has no url, title or `process` rows.
+- **Python.** It reads one thing at a time, so `svc` equals the read's duration; it records the screenshot and the two page reads, and has no url, title or `process` rows. On a native app it reads the screenshot alone.
 
 To measure each platform from the repo's examples (the desktop examples other than Selenium's drive Chrome only):
 
@@ -453,26 +453,29 @@ A hybrid app is the case capabilities alone cannot answer, because the answer ch
 | Mobile browser sessions (Chrome on Android, Safari on iOS) | ✓ | ✓ | ✓ | ✓ |
 | iOS (XCUITest): native apps and Safari | ✓ | ✓ | ✓ | ✓ |
 | Native session detected, page-side capture skipped | ✓ | ✓ | ✓ | ✓ |
-| Follows a hybrid app into its webview | ◐ ² | ◐ ³ | ◐ ³ | ◐ ³ |
-| Native app network through appium-interceptor (`appium:intercept`) | ✓ ⁴ | ✓ ⁴ | ✓ ⁴ | ✓ ⁴ |
+| Native app element list and accessibility tree per action (trace mode) | ✓ | ✓ | ✓ | ✕ ² |
+| Follows a hybrid app into its webview | ◐ ³ | ◐ ⁴ | ◐ ⁴ | ◐ ⁴ |
+| Native app network through appium-interceptor (`appium:intercept`) | ✓ ⁵ | ✓ ⁵ | ✓ ⁵ | ✓ ⁵ |
 
 ✓ supported. ◐ partial. ✕ not supported. A number points to its note below the table. Each supported cell is backed by the cross-adapter conformance suite, a named test or a manual check, listed in [the feature list](https://github.com/webdriverio/devtools/blob/main/packages/conformance/src/features.ts).
 
 ¹ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari.
 
-² Follows `switchContext('WEBVIEW_...')`; the object form (`{ title }` / `{ url }`) and `switchAppiumContext` leave capture on the native side.
+² On a native app the Python adapter takes only a screenshot per action; it reads no page source, so the trace has no element list or accessibility tree.
 
-³ Decides from the startup capabilities, so a hybrid app's webview half is captured as if it were still native.
+³ Follows `switchContext('WEBVIEW_...')`; the object form (`{ title }` / `{ url }`) and `switchAppiumContext` leave capture on the native side.
 
-⁴ Android only; appium-interceptor is an Android plugin.
+⁴ Decides from the startup capabilities, so a hybrid app's webview half is captured as if it were still native.
+
+⁵ Android only; appium-interceptor is an Android plugin.
 
 <!-- capability-matrix:readme-mobile:end -->
 
 **In the dashboard**, a phone capture is framed as the device — a full-height device column with the action list and dock beside it — in both live mode and the trace player. The trace's `context-options` records the platform, model and OS version, read back so the player labels frames without guessing.
 
 **What a mobile trace does not contain.** Per-action snapshots are issued from inside the command hook, and Appium serialises a probe behind the command it is observing — so where a session has a document to probe, the per-action snapshot is skipped. In practice that means **a hybrid app's webview actions carry no per-action element data, accessibility tree or settle screenshot**. Native sessions, mobile browsers and every desktop session are unaffected. Command rows and their screenshots, console and the archive itself are always captured; a mobile browser's network comes from the page's Resource Timing and fetch/XHR patch rather than from BiDi, and a native session has no network capture, because the driver exposes none. These absences are by design; they are not a capture failure.
 
-On a native session, element data and the A11y panel come from the platform's XML tree rather than the DOM, so locators read as `android=new UiSelector()…` / `-ios predicate string:…` rather than CSS or XPath over HTML.
+On a native session, element data and the A11y panel come from the platform's XML tree (its page source, read once per action in trace mode by the WebdriverIO, Selenium and Nightwatch adapters; the Python adapter captures a screenshot alone) rather than the DOM, so locators read as `android=new UiSelector()…` / `-ios predicate string:…` rather than CSS or XPath over HTML.
 
 A runnable example per adapter lives under [examples/](examples/) — `pnpm demo:wdio:mobile`, `:selenium:mobile`, `:nightwatch:mobile`, `:python:mobile`. Each drives the Clock app that ships with every Android system image, keying a duration on the timer keypad without ever starting a timer, so none needs an `.apk` (iOS drives Settings via `DEVTOOLS_MOBILE_PLATFORM=ios`); `DEVTOOLS_MOBILE=web` drives the device's own browser instead — Chrome on Android, Safari on iOS. [examples/MOBILE.md](examples/MOBILE.md) states the prerequisites and switches in one place. A single-adapter WDIO variant also lives at [examples/wdio/mocha/wdio.native.conf.ts](examples/wdio/mocha/wdio.native.conf.ts) (`pnpm demo:wdio:native`), which reads its Appium endpoint from `APPIUM_HOST` / `APPIUM_PORT` / `APPIUM_DEVICE`. In short, for a local emulator:
 

packages/conformance/src/capability-reasons.ts

@@ -38,3 +38,11 @@ export const NIGHTWATCH_METADATA_BY_HAND =
   'No test asserts the capabilities in the metadata the adapter sends; check the Metadata tab with `pnpm demo:nightwatch`.'
 export const SERVICE_MANIFEST_BY_HAND =
   'No test covers the service turning the manifest on when an Allure reporter is configured; check by hand with one configured.'
+export const PYTHON_NATIVE_SCREENSHOT =
+  'On a native app the Python adapter takes only a screenshot per action; it reads no page source, so the trace has no element list or accessibility tree.'
+export const PYTHON_NO_MANIFEST =
+  'The Python adapter writes no artifacts manifest.'
+export const SERVICE_SCREENSHOT_ALWAYS =
+  'No option: every command row takes a screenshot, except on Appium, where only native sessions in live mode do.'
+export const SELENIUM_SCREENSHOT_SWITCH_BY_HAND =
+  'No unit test asserts that `captureScreenshots: false` skips the row screenshot; check on the Selenium example.'