Skip to content

Migrate and expand Semaphore integration test suite - #4

Merged
fiftin merged 18 commits into
mainfrom
codex/migrate-existing-suite
Sep 3, 2026
Merged

Migrate and expand Semaphore integration test suite#4
fiftin merged 18 commits into
mainfrom
codex/migrate-existing-suite

Conversation

@dantro86

@dantro86 dantro86 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • merge the existing Semaphore test automation history into the official integration-tests repository without squashing it
  • preserve the repository-owned Bash output regression and Cloudflare R2/Worker infrastructure
  • expand API, security, task lifecycle, runner, UI, configuration, and upgrade coverage
  • keep scenario data in focused typed fixtures, including a dedicated SemaphoreShellOutputFixtures
  • add a separate read-only externalTest task for user-managed Semaphore installations
  • make the background-child output regression reliable on both local and persistent runners

The repositories share commit 531b7c0, so this PR uses a regular Git merge rather than copying the current tree. The official repository's seven unique commits and the legacy suite's thirteen unique commits remain reachable in history.

Validation

  • ./gradlew --no-daemon qualityGate — passed
  • scripts/run-external-tests.sh --no-daemon against a local Semaphore stand — 2 passed
  • core API suite exercised on SQLite, PostgreSQL, MySQL, MariaDB, and PostgreSQL with a persistent runner
  • persistent runner registration, health, routing, lifecycle, and security scenarios passed
  • SQLite v2.19.7 -> v2.19.8 seed and verify phases passed
  • PostgreSQL v2.19.7 -> v2.19.8 seed and verify phases passed
  • runner-safe background-child regression passed after increasing the child lifetime to 60 seconds and keeping the completion limit at 30 seconds

GitHub Actions evidence

  • PR CI — framework quality gate and Allure HTML artifact passed; the SQLite suite is red only on the task-output regression described below
  • configuration matrix — TOTP, Git over HTTPS, encryption rotation, OIDC, proxy + OIDC, and the production PostgreSQL/persistent-runner profile passed; database/SSH/LDAP failures all contain the same missing-stdout signature
  • release upgrade — SQLite and PostgreSQL reached the post-upgrade suite after successful seed/verify phases; Allure HTML artifact passed and the post-upgrade failure is limited to the same task-output regression

Known product failure

The pinned stable release images can lose short or early task stdout. This directly fails ShellOutputTest and can also hide output markers used by task-stop, Git, and SSH diagnostic assertions. The affected reports consistently show a successful task/fixture setup with stdout missing from the API response; configuration startup, authentication, database, runner, OIDC, Git HTTPS, TOTP, and encryption flows are otherwise healthy.

The product fix was merged in semaphoreui/semaphore#4159, but it is not included in the latest stable release (v2.19.12). The published semaphoreui/semaphore:develop image is currently an old 2-16-stable build, so it cannot be used as a reliable replacement. The regression assertions are intentionally not disabled or weakened.

Existing PRs

  • Simplifies local integration-test development #2 can be rebased after this migration; its local fixture repository and Ansible working-directory coverage remain a separate concern.
  • External #3 overlaps the new external test layer. Its Docker runner and report publishing pieces should be reviewed and moved separately instead of merging the whole branch into this migration.

@fiftin
fiftin merged commit 28f203e into main Sep 3, 2026
10 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants