feat(e2e): add plugin sanity check test (RHIDP-13508)

[gustavolira]

Summary

Implements enhanced sanity checks for all enabled plugins in the RHDH deployment as part of RHIDP-13508.

Changes

• New test file: e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts

  • Validates all enabled packages from default.packages.yaml
  • Checks package name format for all enabled plugins
  • Validates disabled packages list is parseable

• Integration: Added to SHOWCASE_SANITY_PLUGINS project in playwright.config.ts

• Mock file: Created default.packages.yaml in repo root for local testing

  • Real file is injected during CI deployment
  • Mock allows local test execution

Test Behavior

The test runs in two parts:

1. Enabled packages validation: Reads all enabled packages and validates their format
2. Disabled packages validation: Ensures the disabled list is parseable

Currently validates package structure. Future enhancement can add actual plugin loading using @red-hat-developer-hub/cli-module-install-dynamic-plugins.

Testing

Local

yarn playwright test plugin-sanity-check --project showcase-sanity-plugins

CI

Test runs automatically in nightly showcase-sanity-plugins job where RHDH is deployed.

Jira

• Implements: RHIDP-13508
• Part of epic: RHIDP-13496 (Test Layer Infrastructure)

Checklist

• Test created with proper component annotation
• Integrated into existing Playwright project
• ESM compatibility fixed (__dirname polyfill)
• Mock file created for local testing
• Test passes locally with mock data
• Follows playwright-locators best practices
• Follows ci-e2e-testing conventions

[rhdh-qodo-merge]

PR Summary by Qodo

feat(e2e): add Playwright plugin sanity-check test
🧪 Tests ✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

Description

• Add Playwright sanity test that parses default.packages.yaml and validates plugin entries.
• Wire the new spec into the SHOWCASE_SANITY_PLUGINS Playwright project for nightly runs.
• Provide a mock default.packages.yaml in repo root to enable local execution.

Diagram

graph TD
  Config["playwright.config.ts"] --> Project["SHOWCASE_SANITY_PLUGINS"] --> Runner["Playwright runner"] --> Test["plugin-sanity-check.spec.ts"] --> Yaml[("default.packages.yaml")] --> Validate["Validate package entries"]
  CI["Nightly CI deployment"] --> Yaml

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Resolve/install validation (require.resolve / dynamic import)

• ➕ Catches missing deps and broken package exports, not just naming issues
• ➕ Aligns better with the header comment about “can be loaded”
• ➖ May be flaky if the test environment doesn’t include all plugin packages
• ➖ Could increase runtime and require dependency management in CI images

2. Use dynamic plugin installer CLI in the test

• ➕ Validates the real dynamic-plugin acquisition path end-to-end
• ➕ Closer to how RHDH consumes plugins in deployment
• ➖ Adds infra complexity (network access, caching, time)
• ➖ Harder to keep deterministic in CI without pinning and mirrors

3. Schema/structure-only validation + stronger rules

• ➕ Keeps the test lightweight and deterministic
• ➕ Can still catch common config errors (missing fields, duplicates, invalid npm scope/name regex)
• ➖ Won’t detect runtime load failures or missing packages

Recommendation: The PR’s lightweight approach is reasonable for a nightly “sanity” gate, but the spec currently only checks package-name scoping (starts with @) and disabled-list structure. Consider either (a) tightening the structure validation (required fields, non-empty lists, duplicate detection, npm package-name regex) and adjusting the test header comment to match behavior, or (b) adding an optional require.resolve/import-based check when the environment guarantees plugin deps are present. Full CLI-based installation is higher fidelity but likely too heavy unless the team is ready to invest in caching and determinism.

Files changed (3) +149 / -0

Tests (1) +137 / -0

plugin-sanity-check.spec.tsAdd Playwright spec to validate enabled/disabled plugin entries +137/-0

Add Playwright spec to validate enabled/disabled plugin entries

• Introduces a new Playwright spec that reads 'default.packages.yaml', validates enabled plugin package name format (scoped), and asserts the disabled package list is parseable and well-formed. Adds a 'component=plugins' annotation and emits a simple console summary.

e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts

Other (2) +12 / -0

default.packages.yamlAdd mock default plugin package list for local runs +11/-0

Add mock default plugin package list for local runs

• Adds a repo-root 'default.packages.yaml' with enabled/disabled plugin entries to support local execution of the new Playwright sanity test. Notes that the real file is injected during CI deployment.

default.packages.yaml

playwright.config.tsInclude plugin sanity spec in SHOWCASE_SANITY_PLUGINS project +1/-0

Include plugin sanity spec in SHOWCASE_SANITY_PLUGINS project

• Extends the 'SHOWCASE_SANITY_PLUGINS' project 'testMatch' list to include 'plugin-sanity-check.spec.ts', ensuring it runs as part of the sanity-plugins CI job.

e2e-tests/playwright.config.ts

[github-actions]

The container image build workflow finished with status: cancelled.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.25%. Comparing base (0125ff6) to head (6628116).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4967      +/-   ##
==========================================
- Coverage   55.82%   55.25%   -0.58%     
==========================================
  Files         121      109      -12     
  Lines        2350     2132     -218     
  Branches      539      537       -2     
==========================================
- Hits         1312     1178     -134     
+ Misses       1033      954      -79     
+ Partials        5        0       -5     

Flag	Coverage Δ
rhdh	55.25% <ø> (-0.58%)	⬇️

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0125ff6...6628116. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used

✅ Tickets: [RHIDP-13508](https://redhat.atlassian.net/browse/RHIDP-13508)

1. No package resolution performed ✓ Resolved 🐞 Bug ≡ Correctness

Description

The test "All enabled packages can be resolved" never resolves/imports any packages; it only checks
that the string starts with "@" and then records success. This can let CI pass even when a listed
plugin package is missing/unresolvable, making the new sanity check misleading.

Code

e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts[R44-83]

+  test("All enabled packages can be resolved", async () => {
+    // Read default.packages.yaml from rhdh repo root
+    const defaultPackagesPath = join(__dirname, "../../../default.packages.yaml");
+    const yamlContent = readFileSync(defaultPackagesPath, "utf8");
+    const config = yaml.parse(yamlContent) as DefaultPackagesConfig;
+
+    const enabledPackages = config.packages.enabled;
+    console.log(`\n📦 Testing ${enabledPackages.length} enabled packages...\n`);
+
+    const results: {
+      package: string;
+      status: "success" | "failed";
+      error?: string;
+    }[] = [];
+
+    for (const pkg of enabledPackages) {
+      const packageName = pkg.package;
+
+      try {
+        // Attempt to resolve the package
+        // Note: We can't actually import dynamic plugins here as they require
+        // a Backstage runtime, but we can at least verify the package name format
+        // and that it's listed in package.json dependencies
+
+        // Validate package name format
+        if (!packageName.startsWith("@")) {
+          throw new Error("Package name must be scoped (start with @)");
+        }
+
+        // For now, just verify the package is properly formatted
+        // Future enhancement: Use @red-hat-developer-hub/cli-module-install-dynamic-plugins
+        // to actually download and verify the plugins load
+
+        results.push({
+          package: packageName,
+          status: "success",
+        });
+
+        console.log(`✅ ${packageName}`);
+      } catch (error) {

Relevance

⭐⭐ Medium

Mixed history: team accepts improving E2E assertions/coverage (#3147,#4526) but rejected similar
“add runtime validation” checks (#4519).

PR-#3147
PR-#4526
PR-#4519

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The test’s core loop never calls any module-resolution API; it only validates a prefix and then
pushes a success result, so it cannot detect missing/unresolvable packages. Additionally, the E2E
test package itself does not declare the example plugin packages as dependencies, reinforcing that
the current implementation cannot be performing real resolution.

e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts[44-83]
e2e-tests/package.json[57-71]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The test name/comments say it verifies packages can be resolved/loaded, but the implementation only checks a naming convention (`startsWith('@')`) and always marks entries as `success` if that passes. This can silently miss real breakages (missing package, typo, unpublished artifact), because no `resolve`/`import` is ever attempted.

### Issue Context
- This is a Playwright E2E "sanity" test intended for CI signal quality. If the test claims resolvability but doesn’t actually do it, it provides misleading green runs.

### Fix Focus Areas
- e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts[44-83]

### Suggested fix
Choose one (A is lowest-risk for now):

**A) Rename + adjust messaging to match reality**
- Rename the test to something like `All enabled packages have valid package identifiers`.
- Update the header comment and inline comments to remove "resolve/import" language.
- Update summary logs to reflect "validated format" rather than "loaded/resolved".

**B) Actually resolve packages (only if intended + feasible)**
- Implement a real resolution check (e.g., `createRequire(import.meta.url).resolve(packageName)` or Node’s `import.meta.resolve` where available) and fail on thrown errors.
- If doing this, ensure the packages being checked are expected to be present in the environment (dependencies or fetched artifacts), otherwise the test will become permanently red.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-4967
• quay.io/rhdh-community/rhdh:pr-4967-a40db8b8

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[gustavolira]

/test ?

[gustavolira]

/test e2e-ocp-helm-nightly

[gustavolira]

/test e2e-ocp-helm-nightly

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-4967
• quay.io/rhdh-community/rhdh:pr-4967-cbaf0751

[gustavolira]

/test e2e-ocp-helm-nightly

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-4967
• quay.io/rhdh-community/rhdh:pr-4967-6628116f

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[gustavolira]

/test e2e-ocp-helm-nightly

[openshift-ci]

@gustavolira: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-helm-nightly	60a37f4	link	false	/test e2e-ocp-helm-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.