Skip to content

[HOTE-803] feat: Add some improvements to E2E tests #355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mikeeq wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

[HOTE-803] feat: Add some improvements to E2E tests #355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2575a9f
[HOTE-803] feat: Add some improvements to E2E tests
mikeeq Apr 14, 2026
fb69751
add timeout
mikeeq Apr 14, 2026
0865ba6
add parallelistation to tf
mikeeq Apr 14, 2026
dbdf9ca
remove unneeded depends_on
mikeeq Apr 14, 2026
9207b17
Replace trivy
mikeeq Apr 16, 2026
af74217
Add other things to pre-commit
mikeeq Apr 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 49 additions & 1 deletion .github/workflows/playwright-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,13 +91,17 @@ jobs:
playwright-${{ runner.os }}-

- name: "Install Playwright browsers"
timeout-minutes: 3
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 3-minute step timeout for npx playwright install --with-deps is likely too low on cache-miss runners (apt deps + browser downloads can exceed this), causing avoidable CI failures; increase/remove the timeout or apply it only to the download portion when cache is warm.

Suggested change
timeout-minutes: 3

Copilot uses AI. Check for mistakes.
working-directory: tests
run: npx playwright install --with-deps
Comment on lines 93 to 96
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timeout-minutes: 3 for npx playwright install --with-deps is likely to cause intermittent CI failures on cache misses or slower runners because browser downloads can exceed 3 minutes. Consider increasing this timeout or making it conditional on the cache hit status.

Copilot uses AI. Check for mistakes.

- name: "Start the application"
if: env.TARGET_ENV == 'local'
run: |
npm run start
npm run local:start
env:
BUILDKIT_PROGRESS: plain # or "quiet" to fully suppress build output
DOCKER_CLI_HINTS: false # removes "What's next?" hints
Comment on lines 98 to +104
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR title/description focuses on E2E test improvements, but this change set also introduces dependency vulnerability scanning (Grype/OSV) and refactors local Terraform; update the PR description/title to reflect the broader scope so reviewers know what to focus on.

Copilot uses AI. Check for mistakes.

- name: "Show application status"
if: env.TARGET_ENV == 'local'
Expand Down Expand Up @@ -179,6 +183,34 @@ jobs:
docker compose -f local-environment/docker-compose.yml logs "$service" > "tests/testResults/docker-compose-${service}.log" 2>&1
done

- name: "Detect flaky tests"
if: always()
run: |
RESULTS_FILE="tests/testResults/test-results.json"
if [ ! -f "$RESULTS_FILE" ]; then
echo "No test results JSON found, skipping flaky detection"
exit 0
fi

# Check top-level stats for flaky count first
FLAKY_COUNT=$(jq '.stats.flaky // 0' "$RESULTS_FILE" 2>/dev/null || echo 0)
echo "Flaky test count: $FLAKY_COUNT"

if [ "$FLAKY_COUNT" -gt 0 ]; then
# Extract flaky test details: specs contain tests, status is on test objects
FLAKY_TESTS=$(jq -r '
[.. | .specs?[]? | {title: .title, file: .file, line: .line, tests: [.tests[]? | select(.status == "flaky") | .projectName]} | select(.tests | length > 0)]
| .[] | "[\(.tests | join(", "))] \(.title) (\(.file):\(.line))"
' "$RESULTS_FILE" 2>/dev/null)

echo "::warning::Flaky tests detected (passed on retry):"
while IFS= read -r test; do
echo "::warning:: Flaky: $test"
done <<< "$FLAKY_TESTS"
else
echo "No flaky tests detected"
fi

- name: "Publish Test Results"
uses: dorny/test-reporter@v3
if: always()
Expand Down Expand Up @@ -210,6 +242,22 @@ jobs:
echo ":warning: No test results found" >> $GITHUB_STEP_SUMMARY
fi

# Append flaky test details to summary
RESULTS_FILE="tests/testResults/test-results.json"
if [ -f "$RESULTS_FILE" ]; then
FLAKY_COUNT=$(jq '.stats.flaky // 0' "$RESULTS_FILE" 2>/dev/null || echo 0)
if [ "$FLAKY_COUNT" -gt 0 ]; then
echo "" >> $GITHUB_STEP_SUMMARY
echo "### :warning: Flaky Tests ($FLAKY_COUNT)" >> $GITHUB_STEP_SUMMARY
echo "These tests failed initially but passed on retry:" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
jq -r '
[.. | .specs?[]? | {title: .title, file: .file, line: .line, tests: [.tests[]? | select(.status == "flaky") | .projectName]} | select(.tests | length > 0)]
| .[] | "- **[\(.tests | join(", "))] \(.title)** (\(.file):\(.line))"
' "$RESULTS_FILE" 2>/dev/null >> $GITHUB_STEP_SUMMARY
fi
fi

- name: "Upload test results"
uses: actions/upload-artifact@v7
if: always()
Expand Down
11 changes: 5 additions & 6 deletions .mise.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,6 @@ install_before = "7d"
[settings.python]
compile = false

Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the [settings.aqua] cosign = false stanza changes how mise installs aqua-backed tools; in this repo grype/osv-scanner are still resolved via the aqua backend (see mise.lock), so this may reintroduce GitHub/cosign verification calls and break installs in CI or restricted networks. Consider restoring the setting or documenting/validating the new expected behaviour.

Suggested change
[settings.aqua]
cosign = false

Copilot uses AI. Check for mistakes.
# Disable to avoid calling Github API
[settings.aqua]
cosign = false

[tools]
# Custom registries are not included in mise-versions
# https://mise-versions.jdx.dev/
Expand Down Expand Up @@ -40,8 +36,11 @@ python = "3.14.2"

## Security scanning

# https://github.com/aquasecurity/trivy/releases
"aqua:aquasecurity/trivy" = "v0.69.3"
# https://github.com/anchore/grype/releases
grype = "0.111.0"

# https://github.com/google/osv-scanner/releases
osv-scanner = "2.3.5"

# https://github.com/gitleaks/gitleaks/releases
gitleaks = "8.30.1"
Expand Down
56 changes: 52 additions & 4 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,46 @@ repos:
args: [--markdown-linebreak-ext=md]
- id: end-of-file-fixer
- id: check-yaml
args: [--allow-multiple-documents]
- id: check-json
- id: check-toml
- id: check-added-large-files
args: ["--maxkb=500"]
- id: check-case-conflict
- id: check-merge-conflict
- id: check-symlinks
- id: detect-private-key
- id: check-executables-have-shebangs
- id: forbid-submodules
- id: mixed-line-ending
args: [--fix=lf]
- id: no-commit-to-branch
args: [--branch, main, --branch, master, --branch, develop]

# # https://github.com/google/osv-scanner/releases
# - repo: https://github.com/google/osv-scanner
# rev: v2.3.5
# hooks:
# - id: osv-scanner

- repo: local
hooks:
- id: trivy-fs-scan
name: Trivy filesystem scan
entry: trivy fs --scanners vuln --severity HIGH,CRITICAL --exit-code 1 .
# - id: grype-fs-scan
# name: Grype filesystem vulnerability scan
# entry: scripts/grype-scan.sh
# language: script
# pass_filenames: false
# always_run: true
# require_serial: true

# - id: osv-scanner
# name: OSV-Scanner dependency vulnerability scan
# entry: osv-scanner scan --recursive --format table .
# language: system
# pass_filenames: false

- id: grype
name: grype
entry: bash -c 'echo $PWD; whereis grype; grype --version; grype "dir:$PWD" --quiet -o template -t $PWD/scripts/config/grype-table.tmpl --name hometest-service -c $PWD/scripts/config/grype.yaml'
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The grype hook entry uses unquoted $PWD paths (e.g. -t $PWD/scripts/...), which will break if the repo path contains spaces and makes the hook less portable; quote those arguments or call scripts/grype-scan.sh instead of inlining a long bash -c command.

Suggested change
entry: bash -c 'echo $PWD; whereis grype; grype --version; grype "dir:$PWD" --quiet -o template -t $PWD/scripts/config/grype-table.tmpl --name hometest-service -c $PWD/scripts/config/grype.yaml'
entry: bash -c 'echo "$PWD"; whereis grype; grype --version; grype "dir:$PWD" --quiet -o template -t "$PWD/scripts/config/grype-table.tmpl" --name hometest-service -c "$PWD/scripts/config/grype.yaml"'

Copilot uses AI. Check for mistakes.
language: system
pass_filenames: false
Comment on lines +50 to 52
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The grype pre-commit hook runs a long inline bash -c command with unquoted $PWD paths, which will break if the repo path contains spaces and makes the hook harder to maintain. Prefer calling the dedicated scripts/grype-scan.sh (or the existing scripts/reports/scan-vulnerabilities.sh) and ensure all paths are quoted; also drop the echo $PWD/whereis debug output unless it’s needed.

Suggested change
entry: bash -c 'echo $PWD; whereis grype; grype --version; grype "dir:$PWD" --quiet -o template -t $PWD/scripts/config/grype-table.tmpl --name hometest-service -c $PWD/scripts/config/grype.yaml'
language: system
pass_filenames: false
entry: scripts/grype-scan.sh
language: script
pass_filenames: false
always_run: true
require_serial: true

Copilot uses AI. Check for mistakes.

Comment on lines +48 to 53
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grype-fs-scan and the additional grype hook both run a full filesystem Grype scan on every commit, which will double the runtime of pre-commit (and CI pre-commit) without adding coverage; consider keeping just one hook (ideally the scripted one) to avoid redundant scanning.

Suggested change
- id: grype
name: grype
entry: bash -c 'echo $PWD; whereis grype; grype --version; grype "dir:$PWD" --quiet -o template -t $PWD/scripts/config/grype-table.tmpl --name hometest-service -c $PWD/scripts/config/grype.yaml'
language: system
pass_filenames: false

Copilot uses AI. Check for mistakes.
Expand Down Expand Up @@ -103,10 +130,31 @@ repos:
args:
- --args=-recursive

- id: terraform_docs
files: ^local-environment/infra
args:
- --hook-config=--path-to-file=README.md
- --hook-config=--add-to-existing-file=true
- --hook-config=--create-file-if-not-exist=true
- --hook-config=--recursive
- --hook-config=--recursive-path=modules
- --hook-config=--output-file=README.md
- --hook-config=--output-mode=inject
- --hook-config=--output-template='<!-- BEGIN_TF_DOCS -->\n{{ .Content }}\n<!-- END_TF_DOCS -->'
- --hook-config=--working-dir=terraform

- id: terraform_validate
files: ^local-environment/infra

- repo: https://github.com/sqlfluff/sqlfluff
rev: 4.1.0 # pin to a specific version
hooks:
- id: sqlfluff-lint
args: ["--config", ".sqlfluff"] # optional, if you keep config elsewhere
- id: sqlfluff-fix
args: ["--config", ".sqlfluff"] # auto-fix style issues

- repo: https://github.com/rhysd/actionlint
rev: v1.7.12
hooks:
- id: actionlint
Loading
Loading