feat!: add generic e2e support to check-store workflow

[digitalrisedorset]

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://github.com/graycoreio/github-actions-magento2/blob/main/CONTRIBUTING.md#commit
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• Other... Please describe:

What is the current behavior?

check-store.yaml currently supports:

• coding-standard
• unit-test
• smoke-test

There is currently no generic mechanism for executing optional store-defined E2E test suites within the reusable workflow.

Fixes: N/A

What is the new behavior?

Adds optional e2e-test support to check-store.yaml.

The workflow now:

• provisions a Node runtime
• installs npm dependencies when a package.json exists
• executes npm run test:e2e

This keeps E2E implementation details owned by the consuming store while allowing Mage-OS CI orchestration to execute generic Node-based E2E suites.

A minimal TypeScript-based E2E fixture was also added to validate the orchestration flow.

Does this PR introduce a breaking change?

• Yes
• No

Other information

This intentionally keeps the E2E contract lightweight and framework-agnostic for now.

The workflow orchestrates execution but does not enforce:

• Playwright
• Cypress
• browser-specific tooling
• E2E file structure conventions

This allows consuming stores to define their own E2E implementation details while reusing the existing Mage-OS orchestration and Magento service bootstrapping.

[damienwebdev]

@herve

So, what I'd like to see is that we can run this against a store even if that store doesn't have the e2e tests installed.

So detect-e2e currently gates this against running if one isn't found, let's actually install the sample data and the e2e tests and attempt to run them.

[digitalrisedorset]

@herve

So, what I'd like to see is that we can run this against a store even if that store doesn't have the e2e tests installed.

So detect-e2e currently gates this against running if one isn't found, let's actually install the sample data and the e2e tests and attempt to run them.

@damienwebdev I've pushed a follow-up commit.

The workflow now:

• Continues normally when no E2E suite is present.
• Installs sample data when E2E is enabled.
• Installs the MageOS-adapted Playwright suite.
• Executes the Playwright tests.
• Removes the Node version pin.

I think this gets us much closer to the flow we discussed earlier today. Could you take another look when you have a moment? I suspect these changes might need tuning from where we are but this structure should be what we are aiming at. Once it works, I'll start 2 new activties:

• adding more Playwrights test on a weekly basis to start adding genuine tests coverage (this is something anyone can add to as well)
• start a E2E adapter so that any E2E tests suite can work. Lucas might want to join to add his test suite to work as an alternative

[digitalrisedorset]

@damienwebdev I've removed the guards you mentioned. I understand this is so that we can discover fully the E2E execution path and identify what still needs to be wired up. I imagine we may reinstate some of these guards once E2E tests are fully integrated.

[damienwebdev]

@Vinai asked two things:

• Is it possible, to use the built-in playwright version vs. installing our own?
• Is it possible to disable npm scripts to just lower the security surface concern?

[digitalrisedorset]

@Vinai asked two things:

• Is it possible, to use the built-in playwright version vs. installing our own?
• Is it possible to disable npm scripts to just lower the security surface concern?

@Vinai , @damienwebdev My thoughts were that MageOS would own the E2E tests and the execution environment (Node, Playwright version, dependency installation, etc.) would be a separate concern.
I understand the concern around keeping the Node/npm security surface area as small as possible, and I'm happy to investigate those options once we've validated the E2E flow itself.

[damienwebdev]

@digitalrisedorset

Thanks for the continued work on this. E2e test code is now running. Before I re-review, I need to see a green run of the Check Store Test workflow executed in your own fork, since fork PRs here don't run the full CI automatically.

Here's how to run it on digitalrisedorset/github-actions-magento2:

1. Enable Actions on your fork, go to the Actions tab and click "I understand my workflows, go ahead and enable them" if you haven't already.
2. Push your changes to feat/check-store-e2e-support on your fork.
3. Trigger the workflow, either:
   • Actions tab → "Check Store Test" → "Run workflow" → select branch feat/check-store-e2e-support, or
   • open a draft PR within your fork (feat/check-store-e2e-support → digitalrisedorset:main) so the pull_request trigger fires on every push.

Two notes:

• The pin in check-store.yaml pointing resolve-check-config at digitalrisedorset/github-actions-magento2/resolve-check-config@feat/check-store-e2e-support is correct for validation, it's what makes the fork run exercise your modified action instead of the published one on main. Keep it while you iterate, but it must be reverted to graycoreio/github-actions-magento2/resolve-check-config@main before this can merge.

Once you have a green run, link it here and I'll pick the review back up.

[digitalrisedorset]

@digitalrisedorset

Thanks for the continued work on this. E2e test code is now running. Before I re-review, I need to see a green run of the Check Store Test workflow executed in your own fork, since fork PRs here don't run the full CI automatically.

Here's how to run it on digitalrisedorset/github-actions-magento2:

1. Enable Actions on your fork, go to the Actions tab and click "I understand my workflows, go ahead and enable them" if you haven't already.

2. Push your changes to feat/check-store-e2e-support on your fork.

3. Trigger the workflow, either:

   • Actions tab → "Check Store Test" → "Run workflow" → select branch feat/check-store-e2e-support, or
   • open a draft PR within your fork (feat/check-store-e2e-support → digitalrisedorset:main) so the pull_request trigger fires on every push.

Two notes:

• The pin in check-store.yaml pointing resolve-check-config at digitalrisedorset/github-actions-magento2/resolve-check-config@feat/check-store-e2e-support is correct for validation, it's what makes the fork run exercise your modified action instead of the published one on main. Keep it while you iterate, but it must be reverted to graycoreio/github-actions-magento2/resolve-check-config@main before this can merge.

Once you have a green run, link it here and I'll pick the review back up.

thanks @damienwebdev for your support through this CI pipeline work. Today, I tracked down the issue preventing me from running the workflow in my fork. My fork was significantly behind upstream, so the Check Store workflow was not available. After syncing the fork, I was able to run Check Store Test successfully in my own fork as requested.
The latest run is green. I understand this is when you wanted to review again and start to investigate how the final use case (having CI to allow the merchant to run his/her own E2E test suite) can work

[michielgerritsen]

@digitalrisedorset a few points I noticed after looking at this PR.

Failing tests

The tests are failing because of this error:

    ╔══════════════════════════════════════════════════════╗
    ║ Host system is missing dependencies to run browsers. ║
    ║ Missing libraries:                                   ║
    ║     libgtk-4.so.1                                    ║
    ║     libgraphene-1.0.so.0                             ║
    ║     libopus.so.0                                     ║
    ║     libgstallocators-1.0.so.0                        ║
// etc

Playwright is a bit weird in the sense that an npm ci/npm install is not enough, you also need to run this command:

npx playwright install --with-deps

If you are only running Chromium browser, which is most cases is sufficient, you can also only install Chromium, which saves a bunch of seconds of not having to download and install webkit/firefox/safari/?:

npx playwright install --with-deps chromium

Artifact

I would recommend adding a step to upload the artifact:

      - uses: actions/upload-artifact@v4
        if: always() && steps.playwright.outcome == 'failure'
        with:
          name: playwright-report
          path: playwright-report/

Double-check the path, but that uploads a report where you can trace back exactly what the browser did, how the page looked, what was clicked on, etc, etc. As if you would run it locally.

Playwright defaults

I always update the playwright.config.ts with a few defaults that are off from the Playwright default:

The reporter:

reporter: process.env.CI
    ? [['github'], ['list', {printSteps: true}], ['html', {open: 'never'}]]
    : [['list', {printSteps: true}], ['html', {open: 'never'}]],

• GitHub: This outputs the failures back into overview of the current GitHub run.
• List: Output the steps it's currently doing. Otherwise, you have to wait until the full test suite has run to know that something has failed. This prints the result of each step as it is executed. Mainly for when watching the log go.
• Html: The report required for the previous step.

maxFailures: 2,

If there are two failed tests, quit running tests. When you have like 50 tests, and the first few are already failing, there is usually no value in running them all.

[digitalrisedorset]

@digitalrisedorset a few points I noticed after looking at this PR.

Failing tests

The tests are failing because of this error:

    ╔══════════════════════════════════════════════════════╗
    ║ Host system is missing dependencies to run browsers. ║
    ║ Missing libraries:                                   ║
    ║     libgtk-4.so.1                                    ║
    ║     libgraphene-1.0.so.0                             ║
    ║     libopus.so.0                                     ║
    ║     libgstallocators-1.0.so.0                        ║
// etc

Playwright is a bit weird in the sense that an npm ci/npm install is not enough, you also need to run this command:

npx playwright install --with-deps

If you are only running Chromium browser, which is most cases is sufficient, you can also only install Chromium, which saves a bunch of seconds of not having to download and install webkit/firefox/safari/?:

npx playwright install --with-deps chromium

Artifact

I would recommend adding a step to upload the artifact:

      - uses: actions/upload-artifact@v4
        if: always() && steps.playwright.outcome == 'failure'
        with:
          name: playwright-report
          path: playwright-report/

Double-check the path, but that uploads a report where you can trace back exactly what the browser did, how the page looked, what was clicked on, etc, etc. As if you would run it locally.

Playwright defaults

I always update the playwright.config.ts with a few defaults that are off from the Playwright default:

The reporter:

reporter: process.env.CI
    ? [['github'], ['list', {printSteps: true}], ['html', {open: 'never'}]]
    : [['list', {printSteps: true}], ['html', {open: 'never'}]],

• GitHub: This outputs the failures back into overview of the current GitHub run.
• List: Output the steps it's currently doing. Otherwise, you have to wait until the full test suite has run to know that something has failed. This prints the result of each step as it is executed. Mainly for when watching the log go.
• Html: The report required for the previous step.

maxFailures: 2,

If there are two failed tests, quit running tests. When you have like 50 tests, and the first few are already failing, there is usually no value in running them all.

Thanks @michielgerritsen for the suggestions, I have added these along with some other attempts today, I hope we have moved forward.

Here's where I am now:

• Added the Playwright reporters (github, list, html) and maxFailures.
• Uploading the Playwright report and test artifacts from the CI workflow.
• Added Playwright browser dependencies to be installed.
• Reduced the scope to a few smoke tests (healthcheck, homepage, category and product) rather than the full suite.

The browser now starts correctly and the tests execute, so we've moved past the infrastructure issues.

The current failure is a 404 on the PLP/PDP, which makes me think this is now an environment or sample data issue rather than a Playwright one. I'm currently installing the sample data using:

bin/magento sampledata:deploy
composer update
bin/magento setup:upgrade
bin/magento indexer:reindex

but the step appears to complete almost instantly, which doesn't seem right, so I suspect the sample data may not actually be available.

I'm going to pause here for now rather than continue debugging blindly. I think we've made good progress on getting the E2E framework into the pipeline, and I'd rather understand the environment issue properly before spending more time chasing it.

Thanks again for your help!

[digitalrisedorset]

@damienwebdev , @michielgerritsen thanks both for your support. The Playwright tests are failing because the sample data are not being installed. Now, this goes slightly beyond the initial scope that I was having in focus.

Since changing permissions and magento settings is security sensitive and likely requires other parts of the system to be adjusted, I was wondering if you could take a look at the error in the workflow and advise as to what may be the best place to change