[ostenbom]

Description

Runs the auth e2e suite (e2e/playwright.auth.config.ts, 29 tests) on plain Playwright and on Endform side by side, on every PR and push and on demand. It adds the one fix the suite needed to run on remote machines. Below the results is a 30-day look at the main e2e suite in sanity-io/sanity CI, which is what this comparison is meant to inform.

Changes

• .github/workflows/e2e-auth.yml
  • A runner: [playwright, endform] matrix: the same commit, built the same way, tested by both runners.
  • workflow_dispatch inputs repeat_each and retries, passed to both runners.
  • Endform runs in the sanity organization (OQXolTtG) and authenticates with the ENDFORM_API_KEY secret. The CLI is pinned to endform@0.81.5.
  • Endform jobs skip the browser install.
• e2e/helpers/domPropLeaks/vocabulary.ts, e2e/scripts/writeDomPropVocabulary.ts, e2e/endform.config.ts
  • The problem: the DOM prop leak guard added in fix(core): stop non-DOM props leaking to DOM elements and guard against regressions sanity-io/sanity#15109 reads react-dom and @emotion/is-prop-valid from the studio's node_modules when a browser context starts. Endform runners have the e2e package but not the studio's node_modules, so the first Endform run failed all 29 tests in the fixture (run).
  • The fix: the guard's input is plain JSON, so pnpm --filter e2e dom-prop-vocabulary writes it on the CI machine and endform.config.ts ships it to the runners. loadDomPropVocabulary() reads it when ENDFORM=true. The guard runs unchanged.
  • The main e2e suite uses the same guard and needs the same fix.
• e2e/playwright.auth.config.ts: trace: 'retain-on-failure' under Endform only, so the dashboard has a trace for every failed attempt.

Nothing changes when the suite runs natively: no ENDFORM variable, same config, same commands.

Results: auth suite, Playwright vs Endform

Commit 27f4828, ubuntu-latest, 13 dispatched runs on 27 Oct 2026. Every one of the 899 test executions per runner passed, with no failures and no flaky retries on either side. Endform recorded all of them in org OQXolTtG on the first attempt.

Normal runs (11 runs, 29 tests, retries from config):

Median (range)	Playwright	Endform
Tests, as reported by the runner	43.6 s (30.1–46.9)	31.9 s (29.5–34.0)
Test step	45 s (30–48)	40 s (36–43), including writing the vocabulary and the CLI download
Whole job	106 s (85–123)	97 s (85–113)
Endform billable runner minutes		7.5 per run (7.0–8.1)

Each test repeated 10 times, retries 0 (2 runs, 290 executions each):

Run	Playwright test step	Endform test step	Endform billable runner minutes
40571436591	288 s	62 s	170
40571440726	486 s	66 s	184

How to read these numbers:

• At 29 tests, setup dominates. Install and build take about 55 s of each job, so Endform saves about 9 s per job.
• At 290 executions, Endform stays near one minute. Playwright on one 4-core runner takes 5–8 minutes. The main suite runs 326 tests per CI run, so this is the regime it would be in.
• Endform bills more machine time than GitHub for a suite this small, since every test gets its own machine.
• Retries: this suite had no flakes in either runner, so the comparison says nothing about flakiness. The main suite is where that question lives.

Reproduce: gh workflow run e2e-auth.yml --ref endform/auth-suite-comparison -f repeat_each=10 -f retries=0

Context: the main e2e suite in sanity-io/sanity CI

Data from the GitHub Actions API for sanity-io/sanity, 27 Sep to 26 Oct 2026, at upstream main 4a1847e3. Failure causes were classified with the repo's own e2e/scripts/flakeReport/ classifier, over all 175 main runs from 13 to 27 Oct. Its verdicts match the weekly E2E flake report on all 55 runs both cover.

Suites

Suite	Tests per CI run	CI shape	Needs
Main e2e (e2e.yml), required	163 × chromium and firefox = 326	8 jobs on ubuntu-8core, retries: 2	Vercel preview, staging api.sanity.work, 2 new datasets per run
Auth e2e (e2e-auth.yml)	29, chromium	1 job	Nothing; all auth APIs mocked
Embedded e2e (e2e-embedded.yml)	1 × 2 browsers	2 jobs	Staging, shared dataset
Unit (test.yml), required	9,865 × Node 22, 24, 26	12 jobs	Nothing
Browser components (browser-tests.yml), required	315 × 3 browsers	3 jobs	Nothing

Reliability and cost of e2e.yml

Measure	Value
Failed runs on main	43.2% over 30 days (152 of 352); 32% over 13–27 Oct. For comparison, about 64% in the week of 21 Sep
Failed first attempts, all branches	49.0%
Next-worst test workflow on main	Chromatic at 8.2%; unit tests 4.5%, auth e2e 2.0%
Green main runs that needed Playwright retries	112 of 119 (94%), 13–27 Oct
Retry-recovered test executions vs terminal failures on main	3,004 vs 191 (13–27 Oct)
Manual re-runs	401 runs in 30 days, some up to 10 attempts; 274 of 358 failed first attempts went green on re-run. auto-retry-failed-jobs.yml has been disabled since Jan 2026
Green run duration	Median 8m30s. The slowest shard's Playwright step is 2m29s; install, dataset setup and Vercel deploy take 4m08s before the first shard starts
Runner time	Median 28 runner-minutes per run, about 90% on ubuntu-8core. About 108,000 runner-minutes over 30 days, all branches

Why main runs failed (56 of 175 runs, 13–27 Oct)

Cause	Runs
Staging API degraded: 33 runs showed the "Too many requests" (429) dialog on every hard failure's last attempt	38
Setup or infrastructure: 7 broken studio builds in deploy-preview, plus a web server start, an npm timeout and an artifact download	10
Mixed degraded and healthy	3
Test or app with a healthy API	3
Unknown or other	2

Over single attempts, 85% of 3,825 failed attempts showed the 429 dialog, and 11% saw a healthy API. Staging rate-limits per IP, and concurrent PR runs share GitHub's runner IP range.

Most frequently flaky tests on main (13–27 Oct)

Test	Recovered on retry	Failed for good	Main error
tasks.spec › creates a task for a release version (chromium)	84	8	Task list item not visible; API healthy
tasks.spec › creates a task targeting the active document (chromium)	82	3	Same
tasks.spec › resolves a task from the tasks list (chromium)	75	6	Same
expanded.spec › maximized document with enhanced object dialog (both browsers)	96	6	not.toBeVisible times out
vision.spec › listen to changes (both browsers)	67	8	Result not visible; partly 429
unpublish.spec › unpublish a published document (both browsers)	35	18	429 on nearly every attempt
delete.spec › deleted document shows the right name (both browsers)	48	15	429, then waitFor timeout

What moving the main suite to Endform would and wouldn't change

• Would change:
  • The 8 shard jobs collapse into one job, removing a browser install and a globalSetup warm-up of up to 120 s from each shard.
  • Every test attempt gets a trace, history and flake rate, and tests can be quarantined.
• Wouldn't change:
  • The Vercel deploy and dataset setup before tests start.
  • Staging's per-IP rate limiting. Runner traffic would come from Endform's IPs instead of GitHub's, and the effect of that has to be measured.
• Already fits: fullyParallel is on and tests create their own documents.
• To fix first: shared per-user key-value state (sort order, layout, inspect mode), and the one serial block in DisplayedDocument.spec.ts.

Testing

• pnpm vitest run --project=e2e e2e/helpers: 40 tests pass.
• The 13 workflow runs above, plus the failing run before the vocabulary fix.

Notes for release

Not for release. CI and test tooling only, on the endformdev/sanity fork.

🤖 Generated with Claude Code

[github-actions]

📦 Bundle Stats — sanity

Compared against main (4a1847e3) · v6.20.0 (npm)

sanity

Metric	Value	vs main (4a1847e)	vs v6.20.0
Internal (raw)	4.66 MB	-	+48.0 KB, +1.0%
Internal (gzip)	1.17 MB	-	+16.2 KB, +1.4%
Bundled (raw)	13.79 MB	-	+61.0 KB, +0.4%
Bundled (gzip)	3.28 MB	-	+19.8 KB, +0.6%
Import time	1.80s	+19ms, +1.1%	+4ms, +0.2%

bin:sanity

Metric	Value	vs main (4a1847e)	vs v6.20.0
Internal (raw)	7.1 KB	-	-
Internal (gzip)	2.9 KB	-	-
Bundled (raw)	7.1 KB	-	-
Bundled (gzip)	2.8 KB	-	-
Import time	5ms	+0ms, +1.3%	+0ms, +1.8%

🗺️ View treemap · Artifacts

Details

• Import time regressions over 10% are flagged with ⚠️
• Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.