Skip to content

feat!: support CodeceptJS 3.7+/4 and Visual Regression Tracker 6 - #53

Merged
pashidlos merged 5 commits into
masterfrom
feat/codeceptjs-4
Nov 2, 2026
Merged

pashidlos merged 5 commits into
masterfrom
feat/codeceptjs-4

Conversation

@pashidlos

@pashidlos pashidlos commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Modernises the CodeceptJS helper for current CodeceptJS and Visual Regression Tracker 6, and releases it like agent-playwright. This is a breaking change, so it releases as 5.0.0.

Before merging (owner)

  1. Tag the last release. The repo has no git tags; 4.6.1 was published by hand. Without a tag, semantic-release computes 1.0.0, and publishing that would move npm's latest below 4.6.1.

    git tag v4.6.1 58a02b1   # "release 4.6.1"
    git push origin v4.6.1
  2. Add a trusted publisher on npmjs.com for @visual-regression-tracker/agent-codeceptjs (Settings → Trusted publishing → GitHub Actions):

    • organization: Visual-Regression-Tracker
    • repository: agent-codeceptjs
    • workflow filename: release.yml
    • environment: leave empty

    Without it, the first release fails in @semantic-release/npm (there is no NPM_TOKEN).

  3. Squash-merge with the PR title as the commit subject. The feat!: is what makes it a major release.

Why 4.6.1 fails with the Playwright helper

vrtTrack used the return value of the browser helper's saveScreenshot(). In CodeceptJS 3 and 4, the Playwright and Puppeteer helpers only write the file and return undefined, so the step failed with Cannot read properties of undefined (reading 'toString'). Only WebDriver returned the buffer, which is why examples-js uses WebDriver today.

The agent

  • Screenshots:
    • Playwright and Puppeteer: page.screenshot() directly. It honours fullScreen, falling back to the helper's fullPageScreenshots.
    • WebDriver and Appium: saveScreenshot() into a temp file, which keeps CodeceptJS's full-page stitching. The file is removed afterwards.
    • TestCafe and WebDriverIO, which CodeceptJS dropped, are no longer supported.
  • Helper base class: extends @codeceptjs/helper instead of the codecept_helper global. CodeceptJS 4 deprecates the global and drops it with noGlobals; a noGlobals: true run works.
  • sdk-js ^5.8.0:
    • vrtTrack passes through os, browser, viewport, device, customTags, diffTollerancePercent, ignoreAreas, comment and retryCount.
    • The helper config takes sourceBranch (deprecated baselineBranchName still works).
    • Settings missing from the helper config come from vrt.json and VRT_* variables, including VRT_SOURCEBRANCH. With only require, everything comes from there.
  • Uploads: screenshots go as multipart (imageBuffer/imagePath) instead of base64 JSON, so full-page shots no longer need a raised BODY_PARSER_JSON_LIMIT.
  • Statuses: results are handled by sdk-js, as in agent-playwright. new and unresolved fail the step, or are only logged with enableSoftAssert. sdk-js 5.8.0's TestStatus knows new, ok, unresolved and autoApproved. Any other status, such as failed or approved, passes without an error there; that is an sdk-js matter.
  • Package and tooling:
    • Peer codeceptjs ^3.7.0 || ^4.0.0, optional so it isn't installed for development. engines.node >=20 (CodeceptJS 4 itself needs 22.12+).
    • TypeScript 6, Jest 30, ts-jest. files: ["dist"] replaces .npmignore.
    • 17 unit tests, 100% coverage. README rewritten.

Repo setup

  • ci.yml: build and unit tests on Node 20, 22 and 24, on pull requests and pushes to master.
  • release.yml, copied from agent-playwright:
    • semantic-release on push to master;
    • npm trusted publishing (id-token: write, no token) with provenance;
    • v-prefixed tags as in agent-playwright;
    • conventionalcommits preset, so feat!: counts as breaking. It is pinned to preset 9 because preset 10 can't render notes with semantic-release 25's writer.
  • dependabot.yml: npm weekly with limit 5, in groups npm-minor-patch, npm-major and npm-security; GitHub Actions weekly, grouped.
  • npm pkg fix reports nothing.

Verification

  • npm run build and npm run test:cov pass locally on Node 24.
  • Real runs against a visualregressiontracker/api:6.0.0-rc1 stack (the examples-js compose file), using an npm pack of this branch on the-internet.herokuapp.com:
    • CodeceptJS 4.2.2 + Playwright: a baseline was approved on branch release. A run on feature with VRT_SOURCEBRANCH=release and hard asserts passed. Both runs were ok, with baselineBranchName: release.
    • Fields: customTags, comment, the ignore areas, os/browser/viewport/device were stored on the server.
    • Hard assert: a missing baseline failed the step.
    • CodeceptJS 3.7.9 + Playwright passed the same run.
    • CodeceptJS 4.2.2 + WebDriver and + Puppeteer both tracked. Puppeteer against the Playwright baseline correctly failed with "Difference found".
    • Config from VRT_* only, a partial helper config plus VRT_*, and noGlobals: true all passed.
  • npx semantic-release --dry-run on this branch (branch allowed locally, npm plugin left out because OIDC exists only in Actions):
    • with a local v4.6.1 tag: 5.0.0, with notes generated;
    • without the tag: 1.0.0, hence step 1.

Follow-up: Visual-Regression-Tracker/examples-js gets a draft PR that switches the CodeceptJS example back to the Playwright helper on this major.

Phase 2 additions (runtime floor, licence, release tooling)

  • Breaking: merge after the server 6.0 release, together with the runtime-floor PRs feat!: require Node.js 26 or later sdk-js#355, feat!: require Node.js 26 or later agent-cypress#282 and feat!: require Node.js 26 or later agent-playwright#174.
  • engines.node is now >=22 (was >=20); the CI matrix is Node 22, 24 and 26.
  • Licence: Apache-2.0, as a LICENSE file and in the license field (was ISC).
  • version is the 0.0.0-development placeholder, as in the other three repos.
  • semantic-release runs pinned through npx (25.0.9 with the conventionalcommits preset 9.3.1) and is no longer a devDependency.
    • Why: its bundled npm brought undici and ip-address advisories into the lockfile, where nothing can patch them.
    • glob and minimatch are overridden for jest.
    • npm audit now shows only deepmerge (low, through jest-config). No patched version of it exists.
  • The README follows the shared template, adding a configuration table, results and statuses, errors, Releasing and License.
  • Release: with the owner's v4.6.1 tag simulated, the dry run gives 5.0.0. One major up from 4.6.1, this repo cannot reach 6.0.0 the way the other three can.
  • Real run against api:6.0.0-rc1, with the examples-js #104 CodeceptJS/Playwright example on this branch and sdk-js #354 packed: 1 passed, 2 test runs created.

🤖 Generated with Claude Code

pashidlos and others added 3 commits November 2, 2026 21:29
- Take screenshots from page.screenshot() for Playwright and Puppeteer:
  their saveScreenshot() returns nothing, so 4.6.1 failed with
  "Cannot read properties of undefined (reading 'toString')".
  WebDriver and Appium save to a temp file through saveScreenshot().
- Extend @codeceptjs/helper instead of the codecept_helper global,
  which CodeceptJS 4 deprecates and drops with noGlobals.
- Move to sdk-js ^5.8.0: sourceBranch, comment, customTags and
  retryCount are passed through, and settings missing from the helper
  config come from vrt.json and VRT_* variables.
- Upload screenshots as multipart instead of base64 JSON, so large
  full-page shots no longer need a raised BODY_PARSER_JSON_LIMIT.
- TypeScript 6, Jest 30, unit tests; Node.js >= 20.

BREAKING CHANGE: requires CodeceptJS 3.7+ or 4, Node.js 20+ and sdk-js 5
(Visual Regression Tracker server 5+). The TestCafe and WebDriverIO
helpers are no longer supported. VRT_* variables and vrt.json now fill
settings missing from the helper config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI builds and tests on Node.js 20, 22 and 24. Pushes to master release
with semantic-release (conventionalcommits preset, v-prefixed tags as in
agent-playwright) and publish to npm through trusted publishing with
provenance. Dependabot checks npm and GitHub Actions weekly in groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ender

Preset 10 needs conventional-changelog-writer 9, while semantic-release
25.0.9 ships writer 8, so generateNotes failed with "Missing helper".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pashidlos and others added 2 commits November 2, 2026 23:07
- LICENSE file and license field are Apache-2.0, like the other VRT clients.
- semantic-release leaves devDependencies: its bundled npm carried
  vulnerable undici and ip-address that no override can patch. release.yml
  runs pinned semantic-release 25.0.9 with the conventionalcommits preset 9.3.1.
- glob and minimatch are overridden past their advisories for jest.
- package.json version is a placeholder (0.0.0-development).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Node.js 20 is end-of-life. CI tests 22, 24 and 26. The README follows the
template shared by sdk-js and the other agents.

BREAKING CHANGE: Node.js 20 is no longer supported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pashidlos pashidlos added the breaking change Major release; merge after the server 6.0 release label Nov 2, 2026
@pashidlos
pashidlos merged commit d42c16f into master Nov 2, 2026
3 checks passed
@pashidlos
pashidlos deleted the feat/codeceptjs-4 branch November 2, 2026 22:31
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 5.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change Major release; merge after the server 6.0 release released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant