feat: Add random test execution verification system

[paulbalandan]

Description

This PR introduces a comprehensive system for verifying that CodeIgniter4 component tests pass when executed in random order, addressing issue #9968. The system includes automated CI/CD integration and detailed failure diagnostics to help identify and fix test interdependencies.

What Changed

New Files:

1. .github/scripts/run-random-tests.sh (894 lines)

   • Bash script that orchestrates parallel random test execution
   • Configurable concurrency, timeouts, and repeat runs
   • Detailed failure diagnostics with test predecessor tracking
   • Security hardened with input validation and array-based command construction

2. .github/scripts/random-tests-config.txt

   • Configuration file listing verified components (currently: API, CLI)
   • Comments guide for incrementally enabling more components

3. .github/workflows/test-random-execution.yml (224 lines)

   • GitHub Actions workflow with database matrix testing
   • Tests 5 database platforms: MySQLi, Postgre, SQLSRV, SQLite3, Oracle
   • Service containers for MySQL 8.0, PostgreSQL, MSSQL 2025, Oracle XE, Redis, Memcached
   • Workflow callable with 5 customizable inputs

Why This Change

Test execution order dependencies are a common source of flaky tests and difficult-to-reproduce bugs. This system:

1. Prevents regressions: CI automatically catches new test dependencies
2. Provides diagnostics: Failed test predecessor tracking helps identify interdependencies
3. Scales gradually: Components can be verified incrementally via config file
4. Supports debugging: Local execution with configurable timeouts and verbosity
5. Ensures database compatibility: Matrix testing across 5 database platforms

Key Features

Script Capabilities:

• Parallel execution with auto-detected or manual concurrency limits
• Per-component timeout with graceful termination (SIGTERM → SIGKILL)
• Random seed generation and logging for reproducibility
• PHPUnit event log parsing to identify failed test and its predecessor
• Timeout-aware messaging (differentiates timeout vs actual test failure)
• Component name validation to prevent path traversal/injection
• Inflection for grammatically correct output (component/components)

Workflow Features:

• Push/PR triggers for automatic verification on code changes
• workflow_call support for manual/external invocation
• 5 inputs: quiet, component, max-jobs, repeat, timeout
• Database matrix: 5x parallelization across platforms

Example Usage:

# Test all configured components once
.github/scripts/run-random-tests.sh

# Test Database component 10 times with timeout
.github/scripts/run-random-tests.sh --component Database --repeat 10 --timeout 600

# Quiet mode with 4 parallel jobs
.github/scripts/run-random-tests.sh --quiet --max-jobs 4

# Check which tests ran before failure
cat build/random-tests/random_test_order_*.txt

Implementation Notes

Security Considerations:

• Component names validated with regex: ^[a-zA-Z0-9/_-]+$
• Rejects path traversal: .., //, leading /
• PHPUnit arguments use Bash arrays (prevents command injection)
• No eval usage throughout script

Performance:

• Auto-detects CPU cores via nproc/sysctl
• --no-coverage flag speeds up PHPUnit execution
• Results processed in elapsed-time order (fastest first)

Compatibility:

• Requires: bash, find, sort, sed, grep, bc, date, PHPUnit
• Optional: timeout/gtimeout (falls back to manual watchdog)
• Tested on: ubuntu-24.04 with PHP 8.5

Testing Strategy

Incremental Rollout:

• Config file starts with 2 verified components (API, CLI)
• Additional components uncommented as they pass verification
• Failed components investigated and fixed before enabling

CI Integration:

• Workflow triggers on changes to:
  • Script itself
  • Config file
  • Workflow definition
  • phpunit.xml.dist
  • PHP files in system/ or tests/ directories

Local Testing:

# Syntax check
bash -n .github/scripts/run-random-tests.sh

# Smoke test with aggressive timeout
.github/scripts/run-random-tests.sh --component CLI --timeout 1

# Full verification
.github/scripts/run-random-tests.sh

Related Issues

Addresses #9968 - Tests should pass in random execution order

This PR provides the infrastructure for verifying random test execution. Subsequent PRs will fix individual component tests that fail under random ordering.

Breaking Changes

None. This is purely additive CI/CD infrastructure.

Additional Context

Future Work:

• Enable remaining components as they are verified
• Potential extraction to codeigniter4/.github for reuse in other repos

Script Development:
Implementation progressed through 9 distinct phases with comprehensive testing and refinement at each stage, ensuring robust error handling and user-friendly output.

Checklist

• Securely signed commits
• Component(s) with PHPDoc blocks, only if necessary or adds value (without duplication)
  • N/A: Bash script with inline comments; comprehensive function headers present
• Unit testing, with >80% coverage
  • N/A: Bash script; functionality validated via smoke tests and syntax checks
• User guide updated
  • Not applicable for internal CI/CD tooling; usage documented in script header comments
• Conforms to style guide
  • Bash style: 4-space indentation, proper quoting, array usage, no eval

[Copilot]

Pull request overview

Adds CI infrastructure to verify CodeIgniter4 component tests are order-independent by repeatedly executing component test suites with randomized PHPUnit ordering, both locally (script) and in GitHub Actions (workflow), with diagnostics to help identify inter-test dependencies (issue #9968).

Changes:

• Introduces a Bash orchestrator to run component test suites in parallel with randomized order, timeouts, and failure diagnostics.
• Adds a component allowlist config file to enable incremental rollout of random-order verification.
• Adds a GitHub Actions workflow that runs the random-order verification across a database platform matrix and supports workflow_call inputs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/test-random-execution.yml	New CI workflow to run random-order verification across DB matrix and accept reusable-workflow inputs.
.github/scripts/run-random-tests.sh	New runner script to execute component tests in randomized order with concurrency, timeouts, and diagnostics.
.github/scripts/random-tests-config.txt	Initial component list for incremental enablement (currently API, CLI).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[michalsn]

I don't have enough experience with Bash to comment on the implementation details, but the overall approach makes sense to me.

Can we add instructions for running these tests locally to tests/README.md, similar to the notes in the PR description?