The functional Playwright project tests the production app with the pinned browser engines. The visual project adds screenshot comparisons in bundled Chromium. Each test has an isolated browser context; the shared fixture fails on uncaught page errors. Failure screenshots, traces and the HTML report go under ignored sandbox/playwright/. The validation workflow is configured to upload these as playwright-evidence for seven days; local runs do not establish a hosted CI result.
Use Node from .nvmrc and the existing npm installation:
npm run check
npx playwright install chromium
npm run test:e2ePlaywright builds current source before starting its own preview server. Port 4175 must be available; it deliberately refuses to reuse another server that might serve an older build. The build timeout includes engine preparation; prepare engines separately first if starting from an empty cache.
Run a focused project or scenario during development:
npm run test:e2e -- --project=functional --grep 'downloads and reopens'
npm run test:e2e -- --project=visual
PLAYWRIGHT_BROWSER_CHANNEL=chrome npm run test:e2e -- --project=functionalThe browser-channel override applies only to functional checks. Visual checks always use the Chromium version installed for the locked Playwright version. The checked-in baselines were generated on Linux; use the same OS, browser and fonts for comparisons. Other platforms need separately reviewed baselines, not wider mismatch thresholds. Visual tests fix the viewport, locale, timezone, system theme and reduced-motion preference, and wait for the example diagram and local fonts before capture.
The required checks job always runs formatting, architecture, TypeScript, unit/script tests and the dependency audit. Browser coverage is selected separately:
| Trigger | Browser coverage |
|---|---|
| Non-release PR | The @critical real-engine scenarios: entry, source-defined views, invalid drafts, rename and cancellation, native view creation, typed part/connection authoring, source undo/redo, exact save/reopen, engine failure recovery, IndexedDB restore, exact requirements/satisfaction browsing and no-view exploration |
| Non-release PR with code, assets, examples, tooling or unknown paths | All visual comparisons as well as the critical scenarios |
| Documentation-only non-release PR | Critical scenarios; no visual comparisons |
PR targeting main, release/**, hotfix/**; weekly Monday at 04:00 UTC; manual dispatch; or push to main, release/**, hotfix/** |
Complete functional and visual suites |
There is no duplicate run on a merge push to develop; its PR must pass checks. Scheduled runs use the repository's default branch. The full run must pass before release, and remains available manually on any branch. Scheduled failures need triage; they do not retroactively block an already merged PR. The reduced PR suite intentionally defers the extended preservation/failure matrix, language assistance, engine identity investigations and example-library sweeps to full runs.
The selector in scripts/ci-test-plan.mjs only exempts Markdown documentation, root license/notice files and CODEOWNERS from visual checks. Missing/empty change lists retain visual coverage; a missing PR base SHA or failed Git comparison fails the job. No whole-workflow path filters leave required checks pending. Critical traces/screenshots use a separate output directory so the visual run does not erase them. Baselines and assertions are unchanged; full local runs still select every test. Browser workers remain at one until isolation and resource use are measured.
Reproduce the PR browser gate locally:
npm run test:e2e -- --project=functional --grep @critical
npm run test:e2e -- --project=visual # also required for code changesWhen adding a semantic command, keep a real-engine acceptance path in the critical set. Use npm run test:e2e -- --project=functional --grep @critical --list to review membership. Moving a regression out of the PR gate needs an explicit coverage decision; never remove a tag merely to avoid a failure.
The example catalog supplies independent source projects, per-view engine expectations and visual targets. npm run test:examples runs its metadata/inventory/index checks and all library browser scenarios. These scenarios observe the actual worker replies, verify named views and node/relationship multisets, check rendered/partial/unavailable app behavior, and download/reopen exact sources. Unavailable renderers are tested as explicit app behavior rather than skipped. Current diagram PNGs and engine JSON are retained per view under sandbox/playwright/results/examples-example-library-*/ and attached to the HTML report. These captures record current behavior; they are not visual-comparison baselines or evidence that every listed visual target is satisfied.
default-example.spec.ts explicitly imports the retained air-workshop fixture and checks: Diagram mode, the workshop context’s exact parts/ports/connections and explicit navigation into Pneumatics, fourteen named views, all nine unchanged source files and independent file/view selection. ready() enters /app, imports the legacy compressor bundle from its source files, waits for the real-engine overview, then explicitly opens Pneumatics as the fixture for existing diagram regressions; legacy import tests use startExample() directly; rover startup tests load the actual built-in example; waitForStartup() is only a readiness assertion, and reload scenarios choose their fixture deliberately; focused history and palette tests explicitly import the smaller Workshop fixture through openWorkshop(). The example-library scenario exercises all fourteen compressor views and exact source download/reopen, including unavailable renderer messages.
The identity investigation runs the compiled production workers directly to compare model/view identities across source revisions. Run npm run test:e2e -- --project=functional identity.spec.ts for the source-change matrix and Toolkit UTF-8 splice/correspondence probe. These are worker/SDK contract checks; they do not assert that the UI retains layout after applying source. Full source inputs, evaluations, edit reports and candidate mappings are retained as JSON attachments in each scenario's output directory.
The application preservation scenarios are separate: run npm run test:e2e -- --project=functional preservation.spec.ts authoring.spec.ts for reviewed definition/usage renames, bounded source edits, typed part creation and explicit port connections. They compare exact accepted sources and semantic occurrence keys across both views, source undo/redo and fresh import. Reordering, invalid drafts, duplicate names, a blocked engine asset and cancellation retain explicit failure/preservation assertions. Controlled workspace tests cover malformed evidence, stale source/geometry proposals and unavailable/ambiguous view mappings independently.
Run npm run test:e2e -- --project=functional panels.spec.ts toolbar.spec.ts for the panel/icon workflows. They resize all dividers with pointer/keyboard input, exercise bounded editor minimums, collapse/restore with focus recovery, reload/reset UI preferences and handle blocked/corrupt storage. The assembly case compares exact portable bundles, drafts, source DOM identity, selection, camera and both saved view positions. Divider movement preserves the camera; explicit editor mode buttons retain fit-to-pane behavior. Toolbar checks exercise tooltips, source undo/redo, zoom, fit and locked dragging. They also compare the source/mode group centres with their actual panes through resizing, verify SVG centring inside buttons and check contextual edge-rail placement. The single-editor path asserts separation between Diagram and Workspace layout and checks that a constrained desktop editor wraps the modes below source actions.
Run npm run test:e2e -- --project=functional model-tree.spec.ts for ownership-tree navigation. The pinned startup model supplies action definitions with identically named parameters in different owners. Checks compare each inspected declaration/type identity, keyboard focus versus selection, expand/collapse, repeated diagram selection and unchanged source/draft/view/camera/project data after browsing an invalid draft. Existing semantic tests explicitly expand the tree before selecting a hidden declaration; tree selection assertions use aria-selected. Four visual scenarios capture the expanded behavior branch at desktop/narrow widths in both themes.
Run npm run test:e2e -- --project=functional source-editor.spec.ts for CodeMirror editing, indentation/bracket closing, local undo/redo, discard isolation, find/replace followed by real-engine apply and toolbar history, a 1,500-line draft through file switches and fresh reopen, CRLF/Unicode source reveal/edit/save, and blocked typing/undo while an actual engine asset is held pending. Existing semantic and preservation checks now read the editor through its actual select-all/copy commands; clipboard permission is granted only to each isolated test context. readSource() focuses and selects the whole document, including virtualized lines; copySourceSelection() reads an existing revealed range. fillSource() uses select-all plus text input rather than filling rendered DOM lines. Project download assertions separately check exact raw source and line endings. No private CodeMirror instance or hidden mirrored text field is used. Split → Code checks also assert full source-area use and restoration of the split at desktop/narrow sizes. Source/search captures cover desktop/narrow widths in both themes, alongside the changed split-pane captures.
Run npm run test:e2e -- --project=functional diagnostics.spec.ts for real-engine issue navigation with pointer/keyboard, independent source-file choice, exact draft and applied-source retention, both saved views, camera, correction, source undo/redo and fresh reopen. Three additional controlled-response scenarios exercise missing/unknown/out-of-range locations, file-only navigation and a rejected rename candidate whose text differs from the live draft. These normalized response injections establish location-policy behavior; the first scenario supplies real-engine diagnostic evidence. Four source/Issues captures cover desktop/narrow widths in both themes.
Run npm run test:e2e -- --project=functional language-assistance.spec.ts recovery.spec.ts for the source assistance and recovery workflows. Language scenarios use the pinned Toolkit server for symbol information, cross-file definitions with CRLF/Unicode offsets and completion followed by real-engine application and source history. Held and failed engine assets exercise cancellation, retained drafts and a successful fresh query after the fault is removed. Recovery scenarios read the actual IndexedDB record, restore exact drafts and saved layouts after reload, discard malformed envelopes explicitly, retry denied storage and retain both live drafts during a competing-tab conflict. The recovery dialog also exercises keyboard focus containment/return and the persisted enabled preference. Additional cases verify that recovered clean geometry stays unsaved until download, discarded drafts are removed from the durable record, and a real-engine rejected restore retains the current project and pending saved copy. See storage and query policies.
Run npm run test:e2e -- --project=functional shell-entry.spec.ts recovery.spec.ts default-example.spec.ts for entry and replacement behavior. These scenarios check landing/launcher without implicit model work, explicit example and direct app links, New/Open, modal focus, cancellation, actual parser failure and exact retained work through launcher/home/Back/Forward navigation. Recovery scenarios start from the genuinely empty launcher after reload instead of relying on an automatically opened example. The diagram's Named view selector is checked independently of the active source document. Visual entry scenarios capture landing/launcher at desktop, narrow and mobile widths in both themes, alongside the workbench and its dialogs.
semanticPositions() compares saved doubles exactly using semantic subject names plus complete usage contexts. Anonymous declarations must have a zero-length name, a matching recorded origin identity and in-bounds UTF-8 source bytes; their keys retain that exact source identity/content and full context instead of inventing a name or ignoring the saved position. expectRenderedPositions() compares actual node transforms with the browser's own CSS serialization of expected coordinates; CSSOM rounds transforms to six significant digits, so raw JavaScript doubles cannot be compared directly with parsed CSS values. The helper retains its original 0.0001 coordinate threshold after normalization. This does not relax the exact portable-project comparison or test identity through renderer IDs.
Write a concrete expected outcome before implementation. Explore the actual UI to choose accessible locators; prefer role/name or label over CSS structure. Keep special renderer coordinate reads in tests/e2e/support.ts. Compare model-space positions, since camera fit/zoom can change screen coordinates without changing saved geometry.
Use real engines for supported semantic command acceptance, source application, and save/reopen. Controlled workspace responses remain useful for stale/forged proposals and cancellation. Network failure tests must establish that the injected failure occurred and check retained work plus recovery after removing the fault. A mocked engine result is not evidence of engine conformance.
requirements.spec.ts exercises the native capacity requirement graph and a warning-free traceability fixture through the actual pinned workers. Checks distinguish typing, direct ownership, positive and asserted-negated satisfaction, preserve repeated satisfier identity, reveal exact node/edge source, retain invalid drafts and independent layouts through fresh reopen, and recover from an injected engine asset failure with source undo/redo. Its critical scenario checks declared relationships without interpreting them as execution results. The compressor control remains a generic requirements tree and preserves all nine files plus its structural layout. Four requirement visual scenarios cover desktop/narrow widths in both themes, retaining six captures: two fitted overviews and four focused, readable card/Inspector states.
diagram-style.spec.ts checks structural label hierarchy, selection outline, keyboard port selection and physical connection endpoints through appearance changes, comparing the exact portable project before/after. It checks OUT-label contrast across all six palettes in light/dark. Selected structural-card baselines cover desktop/narrow widths in both themes.
Visual checks complement source, identity and geometry assertions. Current baselines cover the default palette in light/dark workbenches, expanded Appearance dialogs and nested assembly diagrams, plus a 768 px workbench and dialog. Resized panels with the Issues dock have desktop and narrow captures in both themes. Part and connection forms have desktop and narrow snapshots in both light and dark themes; the connection form repeats complete selected endpoints in wrapping text below its selectors. Functional checks exercise all palettes and custom colors, structural containment, physical connector endpoints, keyboard port selection and group/child movement through save/reopen. Full visual parity across every palette and browser is not established. Keyboard checks cover Appearance, inherited-port/source interaction, authoring-dialog focus containment/return and a stable workspace focus fallback when an accepted edit removes its original trigger. They do not cover the complete editor. Automated accessibility scanning and Firefox/WebKit coverage are not currently configured.
exploration.spec.ts covers no-view multi-file imports, project/package roots, repeated usage context, Unicode/CRLF declaration reveal and explicit definition-versus-typed-usage structure through the pinned worker. It compares exact source/drafts/presentations before and after browsing, including an invalid draft and named-view positions through portable reopen. Asset failure injection proves the old scope and project survive and a fresh query can succeed. Controlled workspace tests cover stale/cancelled results independently. Exploration checks also assert actual nested package/usage bounds, nonoverlapping parallel relationship labels, exact keyboard edge source reveal without source mutation, compact sibling layout and a full wrapping qualified scope at narrow width. Four focused exploration visual scenarios cover overview/structure at desktop/narrow sizes in both themes and keyboard traversal between scope and representation controls.
npx playwright show-report sandbox/playwright/report
npx playwright show-trace sandbox/playwright/results/<failed-test>/trace.zipInspect the trace, assertion and screenshot to determine whether the application or test is wrong. Repair the cause and preserve the acceptance criterion. Retries, skipped tests, broader pixel tolerances or baseline replacement do not explain a regression.
For an intentional visual change or a new screenshot, generate only the affected baselines:
npm run test:e2e -- --project=visual --grep 'workbench and appearance' --update-snapshotsInspect each generated image and available diff before retaining it. Keep expected PNGs alongside the visual spec; transient captures and experiments belong in sandbox/. Rerun the affected tests without --update-snapshots. Never update snapshots automatically in CI or treat successful baseline generation as a passing comparison.
For a critical regression, demonstrate that the assertion detects a deliberate, bounded fault, restore the original behavior, and rerun. Keep fault experiments isolated from unrelated user edits. Report the injected fault, expected failure and restored passing result without claiming broader coverage.
overview-routing.spec.ts checks the assembly overview against a retained native-worker artifact, samples actual SVG routes against card bodies/headers, checks thirteen label rectangles, reveals every exact declaration by keyboard and preserves the portable bundle. Four assembly visual scenarios cover overview and structure at desktop/narrow widths in light/dark. Renderer geometry reads stay in support.ts.
The initial-exploration failure regression lets the actual parse worker load, fails the following exploration WASM request, then retries with the asset restored. It checks unchanged source/presentations/history and keyboard focus on Scope. Four visual cases retain the retry state at desktop/narrow widths in both themes.
rover.spec.ts checks the actual default example, inherited source-declared attribute values, exact architecture connections, all six source files, all five named views, declared requirement links and source/geometry preservation for both native behavior families through portable reopen. startExample() explicitly chooses the retained air-workshop fixture for existing regressions; waitForRoverStartup() asserts the product default independently. Behavior captures cover desktop/narrow widths and both themes.
engine-upgrade.spec.ts reopens the retained 0.10 positioned project through the current real worker and compares its exact bundle, correspondence and rendered geometry. Its broader fixture retains node/port identities, origins and complete contexts for sixteen supported named views across four old-engine example projects. This is bounded compatibility evidence, not a universal cross-version identity promise.
requirements-data.spec.ts checks table content, repeated qualified satisfiers, positive/negative declarations, unlinked rows, CSV, source reveal, actual verification through a cross-file alias, retained invalid drafts/positions and portable reopen. A real asset failure must occur before retry assertions. Controlled workspace tests separately reject stale/cancelled inspection replies without replacing work. Matrices over 10,000 cells require narrowing the search.
The language-assistance suite additionally checks real Toolkit references and exact source navigation, formatting as a draft, local undo, accepted source undo/redo and formatting failure preservation. The visual project includes requirements tables/matrices at desktop/narrow widths in both themes.
Drag helpers wait for stable heading and viewport geometry across animation frames before issuing one pointer gesture. This avoids reading a screen coordinate while a mode change is still fitting the camera; it does not retry a failed drag. The legacy workspace reopen scenario uses the same browser-normalized rendered-position helper as the newer preservation tests, while separately requiring the moved symbols and exact portable position doubles to survive reopen.
The 0.12 assembly overview fixture records thirteen source-backed edges (ownership/reference plus typing) rather than the old six type-collapsed edges. Its routing regression still requires every route/label to avoid cards and reveals each exact declaration or type reference by keyboard. Changed example tree expectations retain explicit source targets/gaps and exact endpoint/declaration fixtures; structural boundary connection counts are not relaxed.
editor-tour.spec.ts exercises the optional guide with an imported project and an unapplied draft, verifies keyboard dismissal/focus return and checks every popup stays in the viewport at phone width with Explorer collapsed. It also checks that collapsed Explorer and Code-only layouts receive instructions for revealing hidden controls without changing the layout, that each step has an accessible description and a polite live region, and that dismissing by choosing another control preserves focus on that control. The visual project captures the first step at desktop/narrow widths in both themes. Screen-reader announcements still require manual assistive-technology verification.
recovery.spec.ts exercises the competing-tab conflict through both explicit Restore and Discard decisions. It checks the Retry explanation, exact stored timestamp, both live drafts and the unchanged foreign IndexedDB record before a decision. Restore additionally cancels dirty-work confirmation before accepting replacement; Discard retains the local project and resumes saving its queued copy. Visual recovery-conflict and pending-copy captures use a controlled foreign IndexedDB record with a fixed timestamp, while exercising the real manager’s compare-and-swap and Retry paths. They cover desktop/narrow widths in both themes, dialog bounds and keyboard focus return; they are presentation evidence, not a substitute for the real two-tab preservation scenarios.
The critical recovery restore scenario now writes distinguishable successive drafts, waits for the latest exact durable project, reloads and inspects the timestamp and expandable file summary before restoring. Applied/draft byte counts are checked against independently computed UTF-8 sizes, including Unicode, and inspecting the summary must leave the durable record unchanged. The restored source and both saved view layouts are compared exactly with that latest snapshot. Pending-copy visual captures expand the file list. These checks establish bounded successful-save behavior, not a reproduction of the report’s earlier condensed recovery copy.
recovery-error-lifetime.spec.ts restores a real IndexedDB copy whose source parses cleanly but fails exploration origin validation. It retains that current operational error across another parse-clean failing revision, verifies recovery no longer owns the accepted replacement’s visualization failure, and clears the obsolete error only after a later model actually explores successfully. It compares exact accepted source and retains desktop/390px light/dark captures. This is separate from rejected project validation, which the recovery suite still checks as a pending recovery action.
The inherited-member critical regression imports a positioned project captured with the previous real engine. It checks exact inherited declaration/context identities, legacy IDs and displayed positions, keyboard Properties/source navigation, independent inherited-node movement, both view layouts, rejected draft preservation and fresh-context portable reopen. Automatic layout alone does not write presentation positions; the test explicitly moves nodes before requiring saved geometry.