Fail shiftstack testconfig when testing stages marked UNSTABLE

[tusharjadhav3302]

Summary

• After rsyncing shiftstackclient artifacts, fail the shiftstack testconfig when test_stage_unstable_flag is present in the artifact tree.
• Allows the inner ocp_testing.yaml playbook to complete all testing stages while still failing the outer Zuul job for test regressions.

Companion changes (required together)

1. shiftstack-qa — testing stages write test_stage_unstable_flag: Write test_stage_unstable_flag for post-run POST_FAILURE (yellow Zuul UI) shiftstack/shiftstack-qa#36
2. ci-framework-jobs — failure-message: unstable on monolithic shiftstack jobs for yellow UI: https://gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/-/merge_requests/3556

Test plan

• Land all three PRs/MRs together
• TP check with failing openstack-test stage → Zuul job unstable/yellow
• Passing run → green
• Soft verification-only issues → green (no test flag)

Made with Cursor

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign imatza-rh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• roles/shiftstack/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tusharjadhav3302]

Test Project MR : https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/2642

[evallesp]

We need to add the role name at the commit tittle. So please change the tittle to: [shiftstack]

[michburk]

/ok-to-test

[michburk]

Also, this pull request has the following error: 'Cannot update this protected ref.' This is because changes to the cifmw must get merged to the main branch first. Once merged to main, there is a small suite of tests that run daily, and if all of those tests pass, all of the commits included in that testing will get automatically to the stable branch.

Because of this, please target the main branch instead of the stable branch in this pr.

Thanks!

[tusharjadhav3302]

Closing in favor of post-run POST_FAILURE (!3556)

We validated the yellow/unstable Zuul UI using a different integration point than this PR. This PR can be closed without merge — the flag write side (shiftstack-qa #36) and the consumer side (ci-framework-jobs !3556) are sufficient.

What this PR proposed

Fail the run phase inside ci-framework after artifact rsync when test_stage_unstable_flag exists in shiftstackclient artifacts. Companion failure-message: unstable on the Zuul job was intended to colour the build yellow.

What we implemented instead

Layer	Approach
shiftstack-qa #36	Testing stages write test_stage_unstable_flag; inner playbook completes (no end-of-playbook fail)
ci-framework-jobs !3556	Post-run check-test-unstable-post.yaml finds the flag on controller-0 and fails with POST_FAILURE:
ci-framework #4111 (this PR)	Not used

Why post-run (!3556) instead of run-phase fail (this PR)

1. Correct Zuul semantics — Run phase stays SUCCESS when only testing stages fail; post-run POST_FAILURE produces the yellow/unstable UI we need. Run-phase failure tends toward red FAILURE unless additional Zuul workarounds apply.

2. Works for phased and monolithic jobs — Post-run is wired on both periodic-osasinfra-shiftstack_ci-osp_verification and shiftstack-phased-run-tests. A run-phase hook inside the shiftstack testconfig role is harder to align across phased reuse (deploy-edpm-reuse) and monolithic paths without coupling ci-framework to Zuul job shape.

3. Right ownership — Zuul post-run behaviour belongs in ci-framework-jobs, not in the generic ci-framework shiftstack test role. Keeps ci-framework focused on test execution; jobs repo owns how Zuul surfaces results.

4. failure-message: unstable was insufficient — Early experiments with failure-message: unstable on the job did not reliably yield yellow UI on our SF instance. Post-run POST_FAILURE with an explicit fail message did, and was validated end-to-end.

5. Validated on serval70 — TP build https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/zuul/t/components-integration/build/7d875d9d51834190ac7f48df65d9be12

   • Run: SUCCESS
   • Post-run: POST_FAILURE: … Warning! The openshift test Suite failed.
   • Zuul UI: yellow/unstable

What stays the same

shiftstack-qa #36 still writes test_stage_unstable_flag exactly as this PR assumed on the write side. Only the consumer moved from ci-framework run fail → ci-framework-jobs post-run check.

Active MRs

• shiftstack-qa: Write test_stage_unstable_flag for post-run POST_FAILURE (yellow Zuul UI) shiftstack/shiftstack-qa#36
• ci-framework-jobs: https://gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/-/merge_requests/3556
• Jira: https://redhat.atlassian.net/browse/OSPRH-34940

No functional gap from closing this PR — please close when convenient. Thanks for the run-phase prototype; it helped clarify that post-run POST_FAILURE is the better fit for our Zuul setup.

[tusharjadhav3302]

Closing this pull request.