E2e reliability and performance#2201
Conversation
05040a8 to
e7a39b8
Compare
Adds checkAccessibility(page, selector, options) to mendix-helpers.
Wraps @axe-core/playwright with sensible defaults (wcag21aa tags).
Usage:
import { checkAccessibility } from "@mendix/run-e2e/mendix-helpers";
await checkAccessibility(page, ".mx-name-widget1");
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without explicit exports, Node cannot resolve @mendix/run-e2e/fixtures or @mendix/run-e2e/mendix-helpers from widget spec files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…idle The previous implementation resolved too early — mx.session exists before sub-page widgets render. Now waits for: 1. domcontentloaded 2. mx.session + no progress indicator + .mx-page exists 3. networkidle (ensures widget data fetches complete) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…der)
Codemod regex only matched `{ test, expect }` ordering. Six specs used
`{ expect, test }` and were left importing from @playwright/test — no
fixture cleanup meant session exhaustion after ~5 tests.
Also fix codemod regex to handle both orderings for future runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make mendixSession worker-scoped so each worker holds 1 Mendix session, preventing session exhaustion with parallel workers (4 workers < 5 license limit). - Fix waitForFunction call: pass timeout as 3rd arg (options), not 2nd (arg). Previously actionTimeout (10s) was used instead of intended 60s. - Set local workers to 4 (safe with worker-scoped sessions). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- FilteringSingle: replace allTextContents() after toHaveText() with single toHaveText([...array]) call that retries atomically. - FilteringMulti: remove waitForTimeout(300) and allTextContents() pattern, use toHaveText()/toContainText() for auto-retry assertions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…specs - video-player-web: remove redundant waitForMendixReady (fixture handles it). - checkbox-radio-selection-web: replace networkidle waits with waitForMendixApp. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…itForDataReady networkidle hangs forever on pages with streaming content (video embeds, websockets). The mx.session + .mx-page check is sufficient for readiness. Add waitForDataReady helper for tests that explicitly need networkidle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration to shared fixtures and waitForMendixApp is complete across all specs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
8d2f812 to
cf624f0
Compare
…eenshot validation
…test Replace textContent()+toBe() with toHaveText() locator assertion so the paging status check retries until the filter is applied. Fix row loop to use expected.length and .nth(i) — rows.length on a Playwright Locator is undefined, so the loop was silently never executing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
||
| | Don't | Do Instead | Why | | ||
| | --------------------------------------- | ------------------------------------------ | --------------------------------------- | | ||
| | `.nth(N)` on ambiguous selectors | `.mx-name-*` attribute selectors | nth fragile to DOM order | |
There was a problem hiding this comment.
Not sure how this works. .mx-name- is for widget, if we need to find nth option in a list of a row in the data grid, this won't help.
There was a problem hiding this comment.
I think that's fine, the don't is more to avoid ambiguous selectors. For a list row in the datagrid2 will still need a widget class name and a more complex CSS selector.
| | --------------------------------------- | ------------------------------------------ | --------------------------------------- | | ||
| | `.nth(N)` on ambiguous selectors | `.mx-name-*` attribute selectors | nth fragile to DOM order | | ||
| | Complex CSS selectors | `page.locator(".mx-name-widgetName")` | mx-name attributes are stable, semantic | | ||
| | `page.click("text=...")` for navigation | `page.locator(".mx-name-navItem").click()` | Text fragile to i18n/copy changes | |
There was a problem hiding this comment.
I believe having more human-visible selector is better, so we should encourage role/text based selectors. Am I wrong?
There was a problem hiding this comment.
I don't think using text is a good approach. The class or role is a more stable selector. In most of the tests, we try to use classes. I'm not against roles, but the class is more unique, so that avoids finding more than one on the page.
There was a problem hiding this comment.
I think we should guide it more towards using it when needed, css selector to select widget + text selector to select an element in it is better than a complex CSS selector to do the same. For example guiding towards a widget with css selector, and then selecting a dropdown option based on text. And we control the tested environment and if texts change, there is a reason behind it, and we probably should catch it.
There was a problem hiding this comment.
Actually, I think it's the opposite because this way we can see when the order of the DOM changes. Looking only for a text can lead to a false positive. To be honest, ideally, that should be checked in unit snapshot testing. I liked the idea of the compose selector. We can suggest, as you mentioned, a CSS selector/ role + text/caption/label. It can use the playwright docs as a reference: https://playwright.dev/docs/locators.
| await page.goto("/"); | ||
| await waitForMendixApp(page); |
There was a problem hiding this comment.
Doc says: "Auto-wraps page.goto() to call waitForMendixApp()", so is this wait redundant?
| - [ ] No per-test screenshot threshold overrides | ||
| - [ ] No manual `afterEach` logout | ||
| - [ ] Locators use `.mx-name-*` attributes where possible | ||
| - [ ] Tests tagged `@smoke` if they cover critical paths |
There was a problem hiding this comment.
That was introduced in this PR. We're not using smoke tags.
| ## E2E Test Rules (Playwright) | ||
|
|
||
| - Import from `@mendix/run-e2e/fixtures` (not `@playwright/test`) | ||
| - Wait with `waitForMendixApp(page)`, never `waitForTimeout` or `networkidle` | ||
| - Use web-first assertions: `toBeVisible`, `toHaveCount`, `toContainText`, `toHaveCSS` | ||
| - Locators: prefer `.mx-name-*` attributes over nth() or text selectors | ||
| - Screenshots: no per-test threshold overrides, ensure element visible first | ||
| - No manual afterEach logout — fixture handles session lifecycle | ||
| - Tag critical-path tests with `@smoke` | ||
| - See `docs/requirements/e2e-test-guidelines.md` for full rules + template | ||
|
|
There was a problem hiding this comment.
This feels like duplication, should we just point it out to the document
Pull request type
No code changes (changes to documentation, CI, metadata, etc.)
Test related change (New E2E test, test automation, etc.)
Description
This PR improves E2E test reliability and reduces nightly run duration through a set of infrastructure, stability, and parallelization improvements.
Shared fixtures & Mendix helpers (
run-e2e)waitForMendixApp(page)helper to replacewaitForLoadState("networkidle"), which was unreliable and caused flaky testscheckAccessibilityshared helperE2E_SUITE=smokeenv var and@smoketagFlakiness fixes
page.waitForTimeout()fixed delays with event-based waitswaitForLoadState("networkidle")across all specsnth(1)assertion in badge-button close page testScreenshot baseline hardening
threshold: 0.1andanimations: "disabled"in playwright configESLint rules
eslint-plugin-playwrightrules to prevent flaky patterns from creeping back:no-wait-for-timeout(error),no-networkidle(warn),prefer-web-first-assertions(warn)Parallelization
Turbo
cypress/**,cypress.config.*) replaced with Playwright equivalents (e2e/**,playwright.config.*) for correct cache invalidationWhat should be covered while testing?