Skip to content

Retry transient oc exec errors during shiftstack pod polling - #4150

Open
tusharjadhav3302 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-transient-retry
Open

Retry transient oc exec errors during shiftstack pod polling#4150
tusharjadhav3302 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-transient-retry

Conversation

@tusharjadhav3302

@tusharjadhav3302 tusharjadhav3302 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Treat known transient oc exec stderr patterns as retryable while polling long-running shiftstack tests in shiftstackclient
  • Prevents false post-deployment failures when the API/admission webhook returns transient errors (for example SCCExecRestrictions timeouts) while the background command is still running
  • Improves the unreachable-pod failure message with the polling attempt limit

Context

shiftstack-rhel9-rhoso18.0-ocp4.22 on serval70 failed in post-deployment after ~32 minutes even though:

  • RHOSO deploy and openstack-edpm were healthy
  • The nested OCP 4.22 install inside shiftstackclient continued and completed successfully afterward

Failure signature:

admission plugin "security.openshift.io/SCCExecRestrictions" failed to complete validation in 13s

Related Jira: https://redhat.atlassian.net/browse/OSPRH-36061

Test plan

  • Re-run shiftstack-rhel9-rhoso18.0-ocp4.22 on serval70 and confirm post-deployment survives transient oc exec polling errors during osp_verification_4.22_nightly.yaml
  • Confirm non-transient failures (for example pod NotFound) still fail promptly

Made with Cursor

@qodo-code-review

qodo-code-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign eurijon for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@centosinfra-prod-github-app

centosinfra-prod-github-app Bot commented Aug 29, 2026

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b4f2059d92234a2898c1b7c94a2ac513

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 33m 28s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 31m 09s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 51m 37s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 20m 17s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 00s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 17m 08s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 41m 41s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 03s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 19s
❌ cifmw-molecule-shiftstack FAILURE in 21m 20s

@eshulman2

eshulman2 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302

tusharjadhav3302 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@bshewale

bshewale commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from e75f403 to fb296a0 Compare August 31, 2026 07:32
@tusharjadhav3302

tusharjadhav3302 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

/retest-required

2 similar comments
@tusharjadhav3302

tusharjadhav3302 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302

tusharjadhav3302 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from fb296a0 to 6df275c Compare August 31, 2026 12:31
@centosinfra-prod-github-app

centosinfra-prod-github-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/2260089efffe45f2ac95d8615f6248ac

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 11m 00s
❌ podified-multinode-edpm-deployment-crc NODE_FAILURE Node(set) request 099-0000186871 failed in 0s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 49m 00s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 50m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 16s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 22m 25s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 44m 37s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 22s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 16s

@tusharjadhav3302 @cursoragent
[shiftstack] Retry transient oc exec errors during shiftstack pod pol…
b18c45e 
…ling

Long-running osp_verification tests poll shiftstackclient via oc exec.
Transient API/admission failures (for example SCCExecRestrictions webhook
timeouts) were treated as pod loss and aborted post-deployment early.

Keep polling on known transient stderr patterns so the test can finish
when the background command is still running.

Related: OSPRH-36061
Signed-off-by: Tushar Jadhav <tjadhav@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 6df275c to b18c45e Compare September 1, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ok-to-test Ready For Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tusharjadhav3302 @eshulman2 @bshewale