Skip to content

Replace resemble.js with pixelmatch as the comparison engine#1151

Open
wswebcreation wants to merge 16 commits into
mainfrom
ws/investigate-flakiness
Open

Replace resemble.js with pixelmatch as the comparison engine#1151
wswebcreation wants to merge 16 commits into
mainfrom
ws/investigate-flakiness

Conversation

@wswebcreation

@wswebcreation wswebcreation commented Jun 3, 2026

Copy link
Copy Markdown
Member

What this does

This replaces the whole image stack used for visual comparison.

  • resemble.js is removed as the comparison engine and replaced by pixelmatch.
  • jimp is removed as the image manipulation library. PNG decode/encode now goes through fast-png, and the few operations we still need (crop, composite, canvas creation, opacity, rotate, resize) live in a small internal utils/imageUtils.ts.
  • The vendored resemble.jimp.cjs file (~1200 lines) is deleted.

Why

People kept hitting flaky visual tests caused by rendering noise that no human can see: sub-pixel font hinting, 1px anti-aliasing at element edges, tiny shadow changes between runs. None of these are real regressions, but they made tests fail at random and that kills trust in the suite.

The root cause sat in resemble.js:

  • It compares images with a per-channel RGB tolerance. RGB is not perceptually uniform, so the same tolerance means something completely different on the red channel than on the blue channel.
  • On large images it has a sampling optimisation that silently skips around 31% of pixels (every 6th row and column, including the entire first row), so real differences could be missed while noise still slipped through.

Pixelmatch compares in the YIQ colour space, which is weighted by how sensitive the human eye actually is (luminance first, then orange-blue, then purple-green). Its anti-aliasing detection cross-checks both images, so a pixel is only treated as anti-aliasing if both the baseline and the actual look like an anti-aliased edge there. The result: invisible noise passes, real regressions still fail.

On top of removing Jimp we drop a large dependency with native-ish baggage in favour of two small, focused libraries.

Behaviour and compatibility

  • The public method API is unchanged. checkScreen, checkElement, checkFullPageScreen, the matchers and the report output all work the same way.
  • The ignore compare options (nothing, less, antialiasing, colors, alpha) are still supported. They are mapped onto pixelmatch thresholds.
  • Diff images, the JSON report, diffPixels bounding boxes and mismatch percentages keep the same shape.
  • Because the comparison maths is different, mismatch percentages will not be identical to the old engine, so existing baselines may need to be re-accepted once. See the release note.

Fixes included in this PR

  • Top-row artifact on full page screenshots. Jimp's contain() centred the image, shifting content by 1px and creating a false diff across the top row. Replaced with buffer-level padding that anchors content at (0,0).
  • Ignored region 1px under-coverage. The device-pixel size of an ignored region used Math.floor, which could drop a pixel when cssSize * DPR had a fractional part. Width and height now use Math.ceil so the full element is always covered. Position still uses Math.floor.
  • Pixelmatch threshold aligned with the old resemble tolerances and a consistent diff colour.
  • Image dimensions are normalised before comparison so different-sized baseline/actual pairs no longer throw.
  • BiDi emulated tests bump ignoreRegionPadding to better cover elements.

@changeset-bot

changeset-bot Bot commented Jun 3, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3a8a1f5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@wdio/image-comparison-core Major
@wdio/visual-service Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@wswebcreation wswebcreation changed the title Ws/investigate flakiness Replace resemble.js with pixelmatch as the comparison engine Jun 3, 2026
* refactor: remove resemble engine and make pixelmatch the sole comparison engine

* feat: add fast-png dependency and imageUtils pure-JS pixel utility module

* chore: linting fix

* chore: add missing coverage

* feat: overlay diff pixels on actual screenshot and eliminate intermediate PNG encode/decode

- fix restore blockout transparency and correct landscape bezel rotation
-

* chore: add new temp images

* chore:now use the pixelmatch not layout testing baseline

* tests: fix tests

* Merge branch 'main' into ws/replace-jimp

* chore: add cron option to enable/disable trigger

* chore: fix flow

* Merge branch 'main' into ws/replace-jimp

* Merge branch 'main' into ws/replace-jimp

* Merge branch 'ws/investigate-flakiness' into ws/replace-jimp

* fix: fix ocr test

* chore: fix OCR test

* chore: remove images

* chore: revert lt config
@wswebcreation wswebcreation marked this pull request as ready for review June 27, 2026 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant