Skip to content

chore: throw on Frame.expect / Page.expectScreenshot failure#40801

Open
dgozman wants to merge 4 commits into
microsoft:mainfrom
dgozman:chore-expect-throws
Open

chore: throw on Frame.expect / Page.expectScreenshot failure#40801
dgozman wants to merge 4 commits into
microsoft:mainfrom
dgozman:chore-expect-throws

Conversation

@dgozman
Copy link
Copy Markdown
Collaborator

@dgozman dgozman commented May 12, 2026

Summary

  • Frame.expect and Page.expectScreenshot throw on failure instead of returning the diff payload. Failure details ride on a new per-method errorDetails: block in protocol.yml; the dispatcher always sends errorDetails (validated against the schema, defaulting to {}) for any method that defines one, so its presence on the wire is the matcher-error discriminator.
  • The unused noAutoWaiting option on Frame.expect is gone, which lets the one-shot/retry blocks stop threading the wrapper error through internal catches.
  • Server emits bare messages; the client uniformly prefixes Error: when surfacing them as errorMessage. YAML parse errors and JS-error/strict-mode messages now end up with the same shape inside the matcher's failure header.

dgozman added 3 commits May 12, 2026 14:25
Frame.expect now throws ExpectError on mismatch instead of returning a
result. Carries strongly-typed details (received, timedOut) validated
against a new FrameExpectErrorDetails schema in protocol.yml. The
dispatcher routes details through a generic per-method ErrorDetails
validator next to the response error on the wire, and the client
surfaces them on the rejected PlaywrightError, letting Frame._expect
rebuild ExpectResult. _wrapApiCall skips its apiName-prefix and call-log
decoration when the error carries details so matchers format their own
output.
Nothing ever sets it, so the option only existed to gate code paths that
can't be triggered. Removing it lets the one-shot/retry blocks stop
threading ExpectError through internal catches: the single throw site
is now the outer catch that wraps the failure into a typed ExpectError.
…rmat

Page.expectScreenshot now throws on mismatch with bolted-on details,
mirroring Frame.expect. The protocol's `expectScreenshot.returns` is
narrowed to `{ actual? }` and an `errorDetails:` block carries the diff
payload.

The dispatcher always emits `errorDetails` (validated against the
per-method schema, defaulting to `{}`) for any method that defines one,
so the wire's presence of `errorDetails` becomes the matcher-error
discriminator. The throw site no longer needs to bolt empty details.

Frame.expect's outer catch strips the `Error: ` prefix from messages
raised inside `evaluate` (V8 / createStacklessError already prepended
it); the client unconditionally re-adds it, so YAML parse errors and
strict-mode violations end up with the same `Error: …` shape inside the
matcher's failure message.
@github-actions
Copy link
Copy Markdown
Contributor

Test results for "MCP"

23 failed
❌ [chrome] › mcp/annotate.spec.ts:57 › should capture multiple screenshots in one annotation @mcp-windows-latest-chrome
❌ [chrome] › mcp/annotate.spec.ts:110 › should abort annotation when last screenshot is removed @mcp-windows-latest-chrome
❌ [chromium] › mcp/annotate.spec.ts:173 › user-initiated annotate downloads zip with feedback.md @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:230 › should capture annotations via show --annotate @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:251 › should start dashboard and annotate when no dashboard is running @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:273 › should enter annotate mode on fresh dashboard.tsx mount with -s --annotate @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:297 › should annotate via direct browser_annotate MCP call @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:330 › should annotate when context has no fixed viewport @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:367 › should cancel browser_annotate when the MCP request is aborted @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:398 › should cancel browser_annotate when the MCP client disconnects @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:427 › should switch screencast to -s session on show --annotate @mcp-windows-latest-chromium
❌ [chromium] › mcp/annotate.spec.ts:476 › should disengage annotate mode when --annotate client disconnects @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-core.spec.ts:31 › close @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-devtools.spec.ts:217 › video-start-stop @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-devtools.spec.ts:231 › video-chapter @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-json.spec.ts:177 › close-all after open returns closed sessions @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:44 › close named session @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:56 › close-all @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:82 › delete-data named session @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:99 › session stops when browser exits @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:113 › session reopen with different config @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:130 › workspace isolation - sessions in different workspaces are isolated @mcp-windows-latest-chromium
❌ [chromium] › mcp/cli-session.spec.ts:162 › list --all lists sessions from all workspaces @mcp-windows-latest-chromium

7047 passed, 1068 skipped


Merge workflow run.

@github-actions
Copy link
Copy Markdown
Contributor

Test results for "tests 1"

18 failed
❌ [chromium-library] › library/trace-viewer.spec.ts:352 › should show params and return value @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/trace-viewer.spec.ts:352 › should show params and return value @chromium-ubuntu-22.04-arm-node20
❌ [chromium-library] › library/trace-viewer.spec.ts:352 › should show params and return value @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/trace-viewer.spec.ts:352 › should show params and return value @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/trace-viewer.spec.ts:352 › should show params and return value @chromium-ubuntu-22.04-node22
❌ [default] › update-snapshots.spec.ts:70 › should update all snapshots 3-way @vscode-extension
❌ [default] › update-snapshots.spec.ts:70 › should update all snapshots overwrite @vscode-extension
❌ [default-reuse] › update-snapshots.spec.ts:70 › should update all snapshots 3-way @vscode-extension
❌ [default-reuse] › update-snapshots.spec.ts:70 › should update all snapshots overwrite @vscode-extension
❌ [default-trace] › update-snapshots.spec.ts:70 › should update all snapshots 3-way @vscode-extension
❌ [default-trace] › update-snapshots.spec.ts:70 › should update all snapshots overwrite @vscode-extension
❌ [firefox-library] › library/trace-viewer.spec.ts:352 › should show params and return value @firefox-ubuntu-22.04-node20
❌ [webkit-library] › library/trace-viewer.spec.ts:352 › should show params and return value @webkit-ubuntu-22.04-node20
❌ [playwright-test] › update-aria-snapshot.spec.ts:545 › update-snapshots all › should create new baseline for matching snapshot @ubuntu-latest-node20
❌ [playwright-test] › update-aria-snapshot.spec.ts:545 › update-snapshots all › should create new baseline for matching snapshot @macos-latest-node20
❌ [playwright-test] › update-aria-snapshot.spec.ts:545 › update-snapshots all › should create new baseline for matching snapshot @ubuntu-latest-node24
❌ [playwright-test] › update-aria-snapshot.spec.ts:545 › update-snapshots all › should create new baseline for matching snapshot @ubuntu-latest-node22
❌ [playwright-test] › update-aria-snapshot.spec.ts:545 › update-snapshots all › should create new baseline for matching snapshot @windows-latest-node20

3 flaky ⚠️ [chromium-library] › library/popup.spec.ts:261 › should not throw when click closes popup `@ubuntu-22.04-chromium-tip-of-tree`
⚠️ [chromium-library] › library/video.spec.ts:275 › screencast › should capture navigation `@chromium-ubuntu-22.04-node22`
⚠️ [firefox-page] › page/page-goto.spec.ts:483 › js redirect overrides url bar navigation `@firefox-ubuntu-22.04-node20`

41746 passed, 850 skipped


Merge workflow run.

Restores `received` in the FrameExpect success response so matchers and
trace viewer can use it. `toMatchAriaSnapshot` needs it to write
rebaselines in `--update-snapshots=all` mode; the trace viewer's "Return
value" section uses it to display the matched text.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant