CNTRLPLANE-3732: ci(hypershift): add OCP 5.0 MCE periodic integration tests - #81194
CNTRLPLANE-3732: ci(hypershift): add OCP 5.0 MCE periodic integration tests#81194mgencur wants to merge 4 commits into
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@mgencur: This pull request references CNTRLPLANE-3732 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/hold |
openshift-ci
Bot
added
the
do-not-merge/hold
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds Hypershift MCE periodics for release 5.0, reschedules release 4.20 jobs, and makes HCP CLI downloads retryable with archive validation. ChangesHypershift MCE CI
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested labels: Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics-mce.yaml`:
- Around line 241-328: The remaining periodic job definitions are still pinned
to 4.21/4.22 operator streams instead of the 5.0 release under test. Update the
kubevirt job entries in the periodic config so the relevant symbols like
ODF_OPERATOR_SUB_CHANNEL, ODF_OPERATOR_SUB_SOURCE, REDHAT_OPERATORS_INDEX_TAG,
MIRROR_OLM_REMOTE_INDEX, and CCS_OPERATOR_CHANNELS point to the 5.0 equivalents,
keeping the job names and workflows unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: bb21a935-18d3-409b-bbfa-098518f37c2f
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics-mce.yaml
|
/pj-rehearsee periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-agent-connected-ovn-ipv4-metal-conformance periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-kubevirt-metal-ovn-backuprestore |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml (1)
61-194: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winThese cron expressions still don’t limit the jobs to the second weekend. They run on every day 8-14 of the month, and cron ORs day-of-month with day-of-week here, so they also fire on every Saturday/Sunday. Use a schedule that matches the intended monthly window, or add an explicit weekday/date gate in the job.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml` around lines 61 - 194, The periodic schedules for the e2e jobs are still too broad because the cron in this config combines day-of-month and day-of-week, so the jobs can trigger beyond the intended second weekend. Update the schedules for the affected entries (for example the e2e-agent-connected-ovn-* and e2e-agent-disconnected-ovn-* periodic jobs) so they only run in the intended monthly window, or add an explicit date/weekday guard in the job definition rather than relying on the current cron expression alone.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml`:
- Around line 61-194: The periodic schedules for the e2e jobs are still too
broad because the cron in this config combines day-of-month and day-of-week, so
the jobs can trigger beyond the intended second weekend. Update the schedules
for the affected entries (for example the e2e-agent-connected-ovn-* and
e2e-agent-disconnected-ovn-* periodic jobs) so they only run in the intended
monthly window, or add an explicit date/weekday guard in the job definition
rather than relying on the current cron expression alone.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 1d6070f7-a59a-4c95-89a5-eeeb1cc972f9
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml
|
@mgencur: job(s): /pj-rehearsee either don't exist or were not found to be affected, and cannot be rehearsed |
|
@mgencur, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-aws-ovn-conformance |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/hypershift/mce/aws/conformance/hypershift-mce-aws-conformance-workflow.yaml`:
- Line 12: The workflow is still invoking the generic wait ref, which brings in
the default 3-hour timeout and delays post-job cleanup. Update the
hypershift-mce-aws-conformance-workflow step to avoid using the default wait
behavior, either by removing the wait ref or overriding its timeout explicitly
in the workflow so the post section proceeds immediately to dump/destroy. Use
the step entry in the workflow and the wait ref behavior from wait-ref.yaml to
confirm the correct override.
In `@ci-operator/step-registry/mce/must-gather/mce-must-gather-commands.sh`:
- Line 8: The must-gather version mapping is pointing LATEST_ACM_VERSION at an
unmapped 5.0 value, which will not resolve for the RHACM must-gather image.
Update the version constant in mce-must-gather-commands.sh so the MCE
must-gather path uses the latest published 2.x ACM tag instead, and make sure
any references tied to LATEST_ACM_VERSION resolve to a real
registry.redhat.io/rhacm2/acm-must-gather-rhel9 tag.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: cb80c464-b00c-476f-96db-490b5eb98b81
📒 Files selected for processing (2)
ci-operator/step-registry/hypershift/mce/aws/conformance/hypershift-mce-aws-conformance-workflow.yamlci-operator/step-registry/mce/must-gather/mce-must-gather-commands.sh
|
/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-agent-connected-ovn-ipv4-metal-conformance |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-kubevirt-metal-ovn-backuprestore periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-aws-ovn-conformance |
|
/unhold |
openshift-ci
Bot
removed
the
do-not-merge/hold
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-kubevirt-metal-ovn-backuprestore periodic-ci-openshift-hypershift-release-5.0-periodics-mce-e2e-aws-ovn-conformance |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh`:
- Around line 331-346: The HCP CLI download in the
`hypershift-mce-install-commands.sh` flow is using insecure TLS bypass via `curl
-k`. Update the download logic in the `downURL`/`_hcp_downloaded` retry loop to
verify the server certificate instead of disabling it, preferably by wiring in
the cluster CA and using `curl` with explicit CA trust. Keep the existing
retry/archive validation behavior intact, but remove the insecure verification
bypass from the CLI fetch path.
In
`@ci-operator/step-registry/operatorhub/subscribe/odf-operator/operatorhub-subscribe-odf-operator-commands.sh`:
- Around line 13-19: The ODF operator subscription step is still using the
default ODF_OPERATOR_SUB_CHANNEL value instead of deriving or overriding it
explicitly for callers that only set ODF_OPERATOR_SUB_SOURCE. Update
operatorhub-subscribe-odf-operator-commands.sh so the channel derivation in the
ODF_OPERATOR_SUB_CHANNEL empty-check is actually reached, and remove the stale
default from operatorhub-subscribe-odf-operator-ref.yaml or set the intended
channel in each caller such as the hypershift baremetalds workflows. Use the
ODF_OPERATOR_SUB_CHANNEL and CUSTOM_CATALOG_SOURCE handling in this script as
the main place to verify the fix.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 3b679307-f2a5-4c97-b8a1-ca59cefbfcc8
⛔ Files ignored due to path filters (2)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (5)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics-mce.yamlci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.shci-operator/step-registry/mce/must-gather/mce-must-gather-commands.shci-operator/step-registry/operatorhub/subscribe/odf-operator/operatorhub-subscribe-odf-operator-commands.sh
🚧 Files skipped from review as they are similar to previous changes (3)
- ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics-mce.yaml
- ci-operator/step-registry/mce/must-gather/mce-must-gather-commands.sh
- ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml
Add periodics-mce variant config for hypershift release-5.0 with 25 MCE integration tests covering AWS, agent, kubevirt, IBM Z, Power, calico, cilium conformance, backup/restore, and critical tests. Version updates from 4.22: - MCE_VERSION: 2.17 -> 5.0 - CLUSTER_VERSION: 4.22 -> 5.0 - LVM_OPERATOR_SUB_CHANNEL: stable-4.22 -> stable-5.0 - ODF_OPERATOR_SUB_CHANNEL: stable-4.21 -> stable-4.21 (n-2) - ODF_OPERATOR_SUB_SOURCE: redhat-operators-v4-21 -> redhat-operators-v4-22 - ODF_OPERATOR_CHANNEL: stable-4.20 -> stable-4.21 (n-2) - MIRROR_OLM_REMOTE_INDEX: v4.21 -> v4.22 - CCS_OPERATOR_CHANNELS: stable-4.21/4.20 -> stable-4.22/4.21 - Konflux FBC tags: ocp__4.22__ -> ocp__5.0__ - base_images/releases: 4.22 -> 5.0 - assisted-service-scripts: ocm-2.17 (unchanged, no ocm-2.18 yet) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change all 4.20 periodics-mce jobs to run monthly on the 2nd week (days 8-14) instead of weekly. Jobs are spread across Saturday and Sunday with staggered hours to avoid resource contention. Converted minimum_interval jobs (72h, 168h, 336h) and the daily IBM Z s390x job to the same monthly cron pattern. The Feb 29 e2e-aws-critical job is unchanged (effectively disabled). Scheduled on the 2nd week to avoid overlap with: - 4.19 periodics-mce (last week, days 22-28) - 4.17 periodics-mce (scattered specific dates) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@mgencur: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
The Full PR test history shows the following tests:
|
|
/pj-rehearse ack |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
/lgtm |
|
/approve |
|
@aleskandro @gparvin Could you please review the changes under ci-operator/step-registry/mce/must-gather |
|
@dfrazzette maybe you could review as well, please? see my previous message. Thanks |
|
/approve The owners list should be updated with people in the relevant team as both me and Liangquan are not there anymore. |
Fix OCP_VERSION_FOR_ACM typo causing unbound variable error and add missing OCP 5.0 → ACM 5.0 mapping with a fallback for unknown versions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add retry loop with 30s interval and 5min timeout for HCP CLI download. Validates gzip format on each attempt and prints response on failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
New changes are detected. LGTM label has been removed. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: aleskandro, celebdor, jparrill, mgencur The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
openshift-merge-bot
Bot
removed
the
rehearsals-ack
|
[REHEARSALNOTIFIER]
A total of 403 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse ack |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh`:
- Around line 342-344: Update the _hcp_downloaded failure branch to remove the
unbounded cat of /tmp/${HYPERSHIFT_NAME}.tar.gz. Log only bounded, non-sensitive
metadata such as the file size, and if an excerpt is retained, sanitize it and
strictly limit its length to prevent control characters, secret leakage, or CI
log flooding.
- Around line 334-341: Update the HCP CLI download loop around the curl and tar
validation so curl runs within the retry condition, allowing transient DNS,
connection, or transfer failures to continue retrying instead of terminating
under set -e. Preserve the archive validation and retry behavior, and replace
the final full-response cat with a concise error summary.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 1e3ebd26-6db0-4e94-8c25-1ef11df9019a
⛔ Files ignored due to path filters (2)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (3)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics-mce.yamlci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh
| curl -k --output /tmp/${HYPERSHIFT_NAME}.tar.gz "${downURL}" | ||
| if tar -tzf /tmp/${HYPERSHIFT_NAME}.tar.gz &>/dev/null; then | ||
| _hcp_downloaded=true | ||
| break | ||
| fi | ||
| echo "HCP CLI download is not a valid gzip archive, retrying in 30s... ($i/10)" | ||
| sleep 30 | ||
| done |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
set -euo pipefail
file='ci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh'
nl -ba "$file" | sed -n '300,360p'
printf '\n--- top ---\n'
nl -ba "$file" | sed -n '1,80p'Repository: openshift/release
Length of output: 195
🏁 Script executed:
set -euo pipefail
file='ci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh'
sed -n '1,50p' "$file" | cat -n
printf '\n--- loop ---\n'
sed -n '325,346p' "$file" | cat -nRepository: openshift/release
Length of output: 3169
Keep the download inside the retry check. A standalone curl will exit on transient DNS/connection/transfer failures under set -e, so the loop never reaches the retry path. Wrap it in the if condition, and replace the final cat with a short error summary instead of dumping the whole downloaded response.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh`
around lines 334 - 341, Update the HCP CLI download loop around the curl and tar
validation so curl runs within the retry condition, allowing transient DNS,
connection, or transfer failures to continue retrying instead of terminating
under set -e. Preserve the archive validation and retry behavior, and replace
the final full-response cat with a concise error summary.
Source: Coding guidelines
| if [ "$_hcp_downloaded" = false ]; then | ||
| echo "ERROR: Failed to download HCP CLI after 5 minutes. Response content:" | ||
| cat /tmp/${HYPERSHIFT_NAME}.tar.gz |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Do not dump the entire failed response into CI logs.
The response may be arbitrary HTML/binary data or an unbounded body. cat can flood logs, emit control characters, or expose response contents. Log bounded metadata such as the file size, and only a small sanitized excerpt if necessary.
As per coding guidelines, step-registry scripts must avoid leaking sensitive information into CI logs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/hypershift/mce/install/hypershift-mce-install-commands.sh`
around lines 342 - 344, Update the _hcp_downloaded failure branch to remove the
unbounded cat of /tmp/${HYPERSHIFT_NAME}.tar.gz. Log only bounded, non-sensitive
metadata such as the file size, and if an excerpt is retained, sanitize it and
strictly limit its length to prevent control characters, secret leakage, or CI
log flooding.
Source: Coding guidelines
5 participants
Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3732
Previously:
Version updates from 4.22:
Summary by CodeRabbit
release-5.0periodics-mceCI configuration with 25 MCE integration test jobs across AWS, agent, KubeVirt, IBM Z, Power, networking, backup/restore, and critical workflows.ocm-2.17.release-4.20periodic schedules and reduces olderrelease-4.209job frequency to approximately monthly.