Skip to content

OCPBUGS-126793: aws: enable service-type-load-balancer-availability test for private cluster - #31662

Open
tthvo wants to merge 1 commit into
openshift:mainfrom
tthvo:slb-availability-internal-lb
Open

tthvo wants to merge 1 commit into
openshift:mainfrom
tthvo:slb-availability-internal-lb

Conversation

@tthvo

@tthvo tthvo commented Sep 18, 2026

Copy link
Copy Markdown
Member

Descriptions

On AWS clusters with only private subnets, an internet-facing LB cannot be provisioned, so the test's external Service of type LoadBalancer never comes up. Detect the internal-scoped default IngressController and request an internal LB (aws-load-balancer-internal) for the test's service too.

For reference: openshift/release#65357 (review)

Summary by CodeRabbit

  • Bug Fixes
    • Network disruption testing on AWS now correctly configures load balancers as internal when the default ingress load balancer uses an internal scope.

…cluster

On AWS clusters with only private subnets, an internet-facing LB cannot be
provisioned, so the test's external Service of type LoadBalancer never comes
up. Detect the internal-scoped default IngressController and request an
internal LB (aws-load-balancer-internal) for the test's service too.
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Sep 18, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@tthvo: This pull request references Jira Issue OCPBUGS-126793, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.1.0) matches configured target version for branch (5.1.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Descriptions

On AWS clusters with only private subnets, an internet-facing LB cannot be provisioned, so the test's external Service of type LoadBalancer never comes up. Detect the internal-scoped default IngressController and request an internal LB (aws-load-balancer-internal) for the test's service too.

For reference: openshift/release#65357 (review)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 434f3fef-de9c-4374-9270-e7e7c33e9844

📥 Commits

Reviewing files that changed from the base of the PR and between 3c85f76 and 3f2ad77.

📒 Files selected for processing (1)
  • pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The test now checks the default IngressController on AWS. If its load-balancer scope is internal, the test applies the AWS internal-load-balancer annotation to the created service.

Changes

AWS ingress load balancer handling

Layer / File(s) Summary
Detect internal ingress configuration
pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go
The test uses the operator client to inspect the default IngressController. It retries the lookup and treats missing strategy data as external.
Apply AWS service annotation
pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go
AWS preparation creates the operator client and propagates client or lookup errors. The created service receives the internal load-balancer annotation when the ingress configuration is internal.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~15 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 3f2ad

The test correctly requests an internal AWS load balancer when ingress is private, with no identified merge-blocking risk.


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
Ote Binary Stdout Contract ❌ Error The pull request adds a fifth utility.RetryWithExponentialBackoff call in PrepareCollection through isInternalIngressLoadBalancer. That helper emits klog.Warningf on transient API errors. The … Configure klog to write to stderr before monitor setup runs, for example with klog.SetOutput(os.Stderr) or klog.LogToStderr(true) in the OTE process entry point. Alternatively, change the retry helper to route its transient-error warnin…
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (13 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The pull request changes only AWS load-balancer detection and service annotation setup. It adds no Ginkgo title calls and does not modify the existing static test names. The existing names are fixed s…
Test Structure And Quality ✅ Passed PASS. The pull request changes only the service-load-balancer monitor. The added IngressController read is wrapped in the repository's bounded RetryWithExponentialBackoff helper (5 steps, 16-second ca…
Microshift Test Compatibility ✅ Passed PASS — the pull request does not add a Ginkgo e2e test. The authoritative diff changes only the existing MonitorTest implementation and adds AWS IngressController detection plus a service annotati…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS — The pull request adds AWS IngressController detection and an internal load-balancer annotation. It does not add any Ginkgo It, Describe, Context, or When test. The existing service load…
Topology-Aware Scheduling Compatibility ✅ Passed The pull request changes only a monitor test. It adds an IngressController read and an AWS Service annotation for an internal load balancer. The diff adds no pod affinity, topology spread, replica, no…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The pull request does not add a Ginkgo e2e test. The authoritative diff changes an existing monitor test by adding an in-cluster IngressController lookup and an AWS internal load-balancer annota…
No-Weak-Crypto ✅ Passed PASS: The pull request changes only pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go. Added code reads the IngressController scope and conditionally adds the AWS internal-load-b…
Container-Privileges ✅ Passed PASS. The pull request changes only monitortest.go (+43 lines) and adds AWS IngressController detection plus a load-balancer annotation. The diff adds no Kubernetes manifest and no privileged, `ho…
No-Sensitive-Data-In-Logs ✅ Passed PASS. The pull request adds an operator API read and an AWS service annotation. The added lines contain no logging calls, formatted output, credentials, tokens, identifiers, or customer data. Existing…
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the AWS-specific change and explains that the service-type-load-balancer-availability test is enabled for private clusters.
Full details: Ote Binary Stdout Contract

Explanation

The pull request adds a fifth utility.RetryWithExponentialBackoff call in PrepareCollection through isInternalIngressLoadBalancer. That helper emits klog.Warningf on transient API errors. The new AWS ingress-controller GET can therefore write a non-JSON warning to the main process stdout. No klog stderr redirection exists in the entry point or repository setup. The existing retry calls are pre-existing, but this change adds a new process-level execution path and worsens the contract violation.

Resolution

Configure klog to write to stderr before monitor setup runs, for example with klog.SetOutput(os.Stderr) or klog.LogToStderr(true) in the OTE process entry point. Alternatively, change the retry helper to route its transient-error warning to stderr without relying on the default klog destination.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@openshift-ci-robot

Copy link
Copy Markdown

@tthvo: This pull request references Jira Issue OCPBUGS-126793, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.1.0) matches configured target version for branch (5.1.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Descriptions

On AWS clusters with only private subnets, an internet-facing LB cannot be provisioned, so the test's external Service of type LoadBalancer never comes up. Detect the internal-scoped default IngressController and request an internal LB (aws-load-balancer-internal) for the test's service too.

For reference: openshift/release#65357 (review)

Summary by CodeRabbit

  • Bug Fixes
  • Network disruption testing on AWS now correctly configures load balancers as internal when the default ingress load balancer uses an internal scope.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tthvo
Once this PR has been reviewed and has the lgtm label, please assign bertinatto for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-merge checks failed. Please resolve the failing checks before merging.

@tthvo

tthvo commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai This change won't break OTE contract. AFAICT, the openshift-tests entry point configures the default klog settings, which set logtostderr=true. Setting klog.LogToStderr(true) would be a no-op. Besides, this is an existing pattern for this monitoring test.

Thus, we can skip the reported failed pre-flight check. For docstring warning, I added a comment for the new function and would like not to touch other existing ones.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Locate openshift-tests entry point and klog configuration =='
rg -n -C 5 --type go 'LogToStderr\s*\(\s*true\s*\)|logtostderr|klog\.InitFlags|FlagSet' .

printf '%s\n' '== Locate monitoring-test uses of klog.LogToStderr =='
rg -n -C 6 --type go 'klog\.LogToStderr\s*\(\s*true\s*\)' pkg/monitortests

printf '%s\n' '== Map changed monitoring test file =='
ast-grep outline pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go --items all
sed -n '1,260p' pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

Length of output: 201295


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== openshift-tests candidates =='
fd -t f -e go 'openshift-tests|openshift_tests' . | head -50

printf '%s\n' '== Direct klog LogToStderr calls in monitoring sources =='
rg -n --type go -g '!**/testdata/**' -g '!**/vendor/**' \
  'klog\.LogToStderr\s*\(\s*true\s*\)' pkg/monitortests || true

printf '%s\n' '== Updated load-balancer monitoring test =='
sed -n '1,260p' pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

printf '%s\n' '== klog flag initialization in command sources =='
rg -n --type go -g '!**/testdata/**' -g '!**/vendor/**' -C 4 \
  'klog\.InitFlags|logtostderr' cmd test/extended 2>/dev/null || true

Length of output: 11297


@tthvo Thanks for the clarification.

The OpenShift Tests Extension entry point sets logtostderr=true. Therefore, klog.LogToStderr(true) does not change the configured logging behavior. The reported pre-flight concern does not apply.

The new isInternalIngressLoadBalancer comment documents the added function. Existing functions are outside the scope of this change.


📝 Learnings were identified and not saved (knowledge base disabled). Enable

You are interacting with an AI system.

@tthvo

tthvo commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai re-run your pre-merge check and update your review.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="openshift/origin"
pr="31662"

printf '%s\n' '== Pull request metadata and checks =='
gh pr view "$pr" --repo "$repo" --json number,title,state,mergeable,reviewDecision,statusCheckRollup,headRefOid,baseRefName \
  --jq '{number,title,state,mergeable,reviewDecision,baseRefName,headRefOid,statusCheckRollup:[.statusCheckRollup[]? | {name,conclusion,status,detailsUrl}]}'

printf '%s\n' '== Changed files =='
gh pr diff "$pr" --repo "$repo" --name-only

printf '%s\n' '== Pull request diff =='
gh pr diff "$pr" --repo "$repo" -- pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

printf '%s\n' '== Relevant source map and implementation =='
ast-grep outline pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go --items all
sed -n '1,300p' pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

printf '%s\n' '== Operator-client patterns for IngressController scope =='
rg -n --type go -C 4 'IngressController|LoadBalancerScope|aws-load-balancer-internal' pkg test/extended | head -500

Length of output: 3018


🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="openshift/origin"
pr="31662"

printf '%s\n' '== Pull request checks =='
gh pr checks "$pr" --repo "$repo" --json name,state,bucket,workflow,link \
  --jq '.[] | {name,state,bucket,workflow,link}' || true

printf '%s\n' '== Pull request patch =='
gh pr diff "$pr" --repo "$repo"

printf '%s\n' '== Relevant source map =='
ast-grep outline pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go --items all

printf '%s\n' '== Relevant source =='
sed -n '1,320p' pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

printf '%s\n' '== Scope and annotation usage patterns =='
rg -n --type go -C 5 'LoadBalancerScopeInternal|IngressController|aws-load-balancer-internal' \
  pkg/monitortests pkg/cli test/extended 2>/dev/null | head -500 || true

Length of output: 66953


@tthvo I re-ran the pre-merge check for #31662.

I found no remaining actionable findings.

The change only applies on AWS. It reads the default IngressController status and sets service.beta.kubernetes.io/aws-load-balancer-internal=true only when the load balancer scope is operatorv1.InternalLoadBalancer.

The completed agentic-images, dependency verification, image, lint, unit, and verification checks passed. The AWS, GCP, and metal end-to-end checks remain pending.

The prior klog.LogToStderr(true) concern does not apply and is withdrawn.

⚠️ Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6

@tthvo

tthvo commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai you still have a stale "Requested changes" in the PR. If you have other reviews, let me know. If not, remove it.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="openshift/origin"
pr="31662"

printf '%s\n' '== Pull request review decision =='
gh pr view "$pr" --repo "$repo" --json reviewDecision,reviews \
  --jq '{reviewDecision, reviews: [.reviews[] | {author: .author.login, state, submittedAt, body}]}'

printf '%s\n' '== CodeRabbit review comments and resolution state =='
gh api --paginate "repos/$repo/pulls/$pr/comments?per_page=100" \
  --jq '.[] | select(.user.login == "coderabbitai[bot]" or .user.login == "coderabbitai") | {id, path, line, side, body, created_at}' || true

printf '%s\n' '== CodeRabbit-authored issue comments =='
gh api --paginate "repos/$repo/issues/$pr/comments?per_page=100" \
  --jq '.[] | select(.user.login == "coderabbitai[bot]" or .user.login == "coderabbitai") | {id, created_at, body}' || true

Length of output: 27516


@tthvo There are no remaining CodeRabbit findings on #31662.

I will replace the stale requested-changes review with an approval.

✅ Action performed

Comments resolved and changes approved.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Sep 18, 2026
@tthvo

tthvo commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

/payload-job periodic-ci-openshift-release-main-nightly-5.1-e2e-aws-ovn-installer-dualstack-ipv6-primary
/payload-job periodic-ci-openshift-release-main-nightly-5.1-e2e-aws-ovn-installer-dualstack-ipv4-primary

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@tthvo: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.1-e2e-aws-ovn-installer-dualstack-ipv6-primary
  • periodic-ci-openshift-release-main-nightly-5.1-e2e-aws-ovn-installer-dualstack-ipv4-primary

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/225094e0-b3a8-11f1-9574-9e6675537c07-0

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/override-sticky ci/prow/e2e-gcp-ovn-upgrade

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit e2e/integration job. e2e-gcp-ovn-upgrade uses the openshift-upgrade-gcp workflow with the openshift-org-gcp cluster profile and OVN upgrade coverage.
Revision check: run 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; current PR HEAD 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; match.
Execution status: Tests executed. The upgrade test ran openshift-tests run-upgrade all with 1,270 selected specs; install and pre phases succeeded, the test phase ran for 42m36s, and the job failed on MonitorTest metrics-endpoints-down, including [Monitor:metrics-endpoints-down][sig-node] kubelet metrics endpoints should always be reachable and [Monitor:metrics-endpoints-down][Jira:"Test Framework"] monitor test metrics-endpoints-down interval construction.
Completed supporting jobs: ci/prow/unit, ci/prow/lint, ci/prow/verify, ci/prow/verify-deps, ci/prow/go-verify-deps, ci/prow/images, ci/prow/e2e-aws-ovn-microshift, and ci/prow/e2e-aws-ovn-microshift-serial succeeded.
Pending jobs: ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-serial-2of2, ci/prow/e2e-gcp-ovn, ci/prow/e2e-metal-ipi-ovn-ipv6, and tide.
Fleet-wide failure rate: The job pass rate is 70.83% over 14 days in Sippy and 74.2% in BigQuery. For [Monitor:metrics-endpoints-down][sig-node] kubelet metrics endpoints should always be reachable, the global clean-pass rate is 62.29% with 11,940 flakes; GCP clean-pass rate is 55.61% with 0 hard failures and 1,877 flakes. The monitor interval-construction test has a 99.12% global pass rate and fails when the parent monitor test fails.
Open regressions: None found for the failing test; Sippy reports open_bugs=0.
Linked bugs: Sippy bug_tests links the failing test to OCPBUGS-60105 (Closed, Won't Do), OCPBUGS-36744 (Closed, Obsolete), OCPBUGS-35371 (Closed, Done), and OCPBUGS-36263 (Closed, Done). No linked bug is currently open.
Overlap assessment: The PR changes pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go and gates its new behavior on configv1.AWSPlatformType, adding the AWS internal-load-balancer annotation. The failed job is GCP/OVN and the failing test is the kubelet metrics monitor; there is no direct or indirect overlap with the changed AWS-only path.
Missing-coverage risk: Low for accepting this PR without this failed job's signal: the failed coverage is the GCP kubelet-metrics monitor, while the PR's behavior is AWS-only service load-balancer handling. The successful unit, lint, verify, dependency, image, and auxiliary e2e checks provide additional signal; pending checks are not used as positive evidence.
Prior bot activity on this SHA: /test e2e-gcp-ovn-upgrade was already triggered at 2026-09-18T20:39:17Z; no prior override. No additional retest is warranted under the one-retest-per-job-per-SHA limit.
Rationale: This is an executed-test failure in a known high-flake monitor. The exact GCP data shows 0 hard failures and 1,877 flakes in the current period, and the PR does not touch the failing GCP kubelet-metrics surface.

If you disagree with this assessment, rerun the current job with /test e2e-gcp-ovn-upgrade.


AI-generated. Review for accuracy.

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-gcp-ovn-upgrade

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/e2e-gcp-ovn-upgrade

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit e2e/integration job. e2e-gcp-ovn-upgrade uses the openshift-upgrade-gcp workflow with the openshift-org-gcp cluster profile and OVN upgrade coverage.
Revision check: run 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; current PR HEAD 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; match.
Execution status: Tests executed. The upgrade test ran openshift-tests run-upgrade all with 1,270 selected specs; install and pre phases succeeded, the test phase ran for 42m36s, and the job failed on MonitorTest metrics-endpoints-down, including [Monitor:metrics-endpoints-down][sig-node] kubelet metrics endpoints should always be reachable and [Monitor:metrics-endpoints-down][Jira:"Test Framework"] monitor test metrics-endpoints-down interval construction.
Completed supporting jobs: ci/prow/unit, ci/prow/lint, ci/prow/verify, ci/prow/verify-deps, ci/prow/go-verify-deps, ci/prow/images, ci/prow/e2e-aws-ovn-microshift, and ci/prow/e2e-aws-ovn-microshift-serial succeeded.
Pending jobs: ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-serial-2of2, ci/prow/e2e-gcp-ovn, ci/prow/e2e-metal-ipi-ovn-ipv6, and tide.
Fleet-wide failure rate: The job pass rate is 70.83% over 14 days in Sippy and 74.2% in BigQuery. For [Monitor:metrics-endpoints-down][sig-node] kubelet metrics endpoints should always be reachable, the global clean-pass rate is 62.29% with 11,940 flakes; GCP clean-pass rate is 55.61% with 0 hard failures and 1,877 flakes. The monitor interval-construction test has a 99.12% global pass rate and fails when the parent monitor test fails.
Open regressions: None found for the failing test; Sippy reports open_bugs=0.
Linked bugs: Sippy bug_tests links the failing test to OCPBUGS-60105 (Closed, Won't Do), OCPBUGS-36744 (Closed, Obsolete), OCPBUGS-35371 (Closed, Done), and OCPBUGS-36263 (Closed, Done). No linked bug is currently open.
Overlap assessment: The PR changes pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go and gates its new behavior on configv1.AWSPlatformType, adding the AWS internal-load-balancer annotation. The failed job is GCP/OVN and the failing test is the kubelet metrics monitor; there is no direct or indirect overlap with the changed AWS-only path.
Missing-coverage risk: Low for accepting this PR without this failed job's signal: the failed coverage is the GCP kubelet-metrics monitor, while the PR's behavior is AWS-only service load-balancer handling. The successful unit, lint, verify, dependency, image, and auxiliary e2e checks provide additional signal; pending checks are not used as positive evidence.
Prior bot activity on this SHA: /test e2e-gcp-ovn-upgrade was already triggered at 2026-09-18T20:39:17Z; no prior override. No additional retest is warranted under the one-retest-per-job-per-SHA limit.
Rationale: This is an executed-test failure in a known high-flake monitor. The exact GCP data shows 0 hard failures and 1,877 flakes in the current period, and the PR does not touch the failing GCP kubelet-metrics surface.

If you disagree with this assessment, rerun the current job with /test e2e-gcp-ovn-upgrade.


AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/override-sticky ci/prow/e2e-metal-ipi-ovn-ipv6

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running generated presubmit e2e/integration job. The job uses the equinix-ocp-metal cluster profile and the baremetalds-e2e-ovn-ipv6 workflow; the run lasted 3h58m44s.

Revision check: Run SHA 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; current PR HEAD 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; match.

Execution status: Tests executed in baremetalds-e2e-test. The run executed 4,232 e2e tests, with 12 hard failures and 85 monitor failures/flakes. The failure pattern was broad cluster instability on the IPv6 metal environment, including container-startup failures, etcd leadership loss, and OVS/API instability; it was not an early/no-test-execution abort.

Completed supporting jobs: ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-microshift, ci/prow/e2e-aws-ovn-microshift-serial, ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-serial-2of2, ci/prow/e2e-gcp-ovn, and ci/prow/e2e-gcp-ovn-upgrade succeeded. ci/prow/agentic-images, ci/prow/go-verify-deps, ci/prow/images, ci/prow/lint, ci/prow/okd-scos-images, ci/prow/unit, ci/prow/verify, and ci/prow/verify-deps also succeeded. Pending: tide.

Fleet-wide failure rate: This exact job passed 73.6% over 14 days and 78.3% over 7 days. Representative exact failing-test rates for the current 5.1 view were: [sig-auth][Feature:SecurityPenetration] Security Penetration Tests TestNoUnexpectedPrivilegedPods [Suite:openshift/conformance/parallel] — 0.0% global, 0.0% IPv6, 0.0% metal; [sig-auth][Feature:SecurityPenetration] Security Penetration Tests TestAllRoutesUseTLS [apigroup:route.openshift.io] [Suite:openshift/conformance/parallel] — 13.2% global, 16.9% IPv6, 12.0% metal; [OTP][sig-baremetal][Level0] IPI BareMetal Author:jhajyahy-Medium-34195-Verify all pods replicas are running on workers only — 67.7% global, 0.0% IPv6, 67.7% metal; [OTP][sig-baremetal][Level0] IPI BareMetal Author:jhajyahy-Medium-32361-Verify that deployment exists and is not empty — 67.7% global, 0.0% IPv6, 67.7% metal; cluster-config-operator should have a healthy deployment in openshift-config-operator namespace [apigroup:apps][Operator][Parallel] — 78.4% global, 41.0% IPv6, 58.8% metal; [sig-network] Conntrack proxy implementation should not be vulnerable to the invalid conntrack state bug [Privileged] — 99.7% global, 98.7% IPv6, 99.5% metal; [sig-arch] Managed cluster should have operators on the cluster version [apigroup:config.openshift.io] [Suite:openshift/conformance/parallel] — 99.9% global, 100.0% IPv6, 100.0% metal. Exact individual rates were unavailable for the remaining five SecurityPenetration tests, so they are not inferred.

Open regressions: No separate Component Readiness regression record was retrieved for the remaining failure groups. The conntrack failure is explicitly covered by the open regression tracked in OCPBUGS-90520.

Linked bugs: OCPBUGS-90520 is currently ASSIGNED and explicitly linked via bug_tests to [sig-network] Conntrack proxy implementation should not be vulnerable to the invalid conntrack state bug [Privileged]. The Jira description also records this test as consistently failing on metal/techpreview variants.

Overlap assessment: The PR changes only pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go, adding AWS-only IngressController scope detection and the AWS internal-load-balancer annotation. The failing tests cover SecurityPenetration, bare-metal pod/deployment health, cluster-config-operator health, conntrack, and cluster-version operator health. There is no plausible direct overlap; the only indirect relationship is the shared origin test image. The failing run was on Equinix metal IPv6, so the PR's AWS-specific path was not the tested surface.

Missing-coverage risk: Low for this failure decision. The job's failed coverage is unrelated to the AWS-only code path, and the remaining completed AWS/GCP e2e jobs plus unit/lint/verify/images checks are green. The metal IPv6 job remains a useful platform signal, but its broad infrastructure failure does not provide meaningful evidence against this change.

Prior bot activity on this SHA: One /test e2e-metal-ipi-ovn-ipv6 was triggered at 2026-09-18T20:39:17Z; no prior override for this job was recorded.

Rationale: The job is an eligible long-running e2e/integration job, the revision matches, tests ran, and fleet/platform evidence shows the observed failures are pre-existing systemic IPv6-metal/infrastructure failures rather than regressions in this AWS-only monitor-test change. A Prow controller-manager Degraded outage also overlapped part of the run. Override is appropriate.

If you disagree with this assessment, rerun the current job with /test e2e-metal-ipi-ovn-ipv6.


AI-generated. Review for accuracy.

@openshift-ci

openshift-ci Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-metal-ipi-ovn-ipv6

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/e2e-metal-ipi-ovn-ipv6

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running generated presubmit e2e/integration job. The job uses the equinix-ocp-metal cluster profile and the baremetalds-e2e-ovn-ipv6 workflow; the run lasted 3h58m44s.

Revision check: Run SHA 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; current PR HEAD 3f2ad776b13dc9666ee241e5d64f60d13fa6b51f; match.

Execution status: Tests executed in baremetalds-e2e-test. The run executed 4,232 e2e tests, with 12 hard failures and 85 monitor failures/flakes. The failure pattern was broad cluster instability on the IPv6 metal environment, including container-startup failures, etcd leadership loss, and OVS/API instability; it was not an early/no-test-execution abort.

Completed supporting jobs: ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-microshift, ci/prow/e2e-aws-ovn-microshift-serial, ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-serial-2of2, ci/prow/e2e-gcp-ovn, and ci/prow/e2e-gcp-ovn-upgrade succeeded. ci/prow/agentic-images, ci/prow/go-verify-deps, ci/prow/images, ci/prow/lint, ci/prow/okd-scos-images, ci/prow/unit, ci/prow/verify, and ci/prow/verify-deps also succeeded. Pending: tide.

Fleet-wide failure rate: This exact job passed 73.6% over 14 days and 78.3% over 7 days. Representative exact failing-test rates for the current 5.1 view were: [sig-auth][Feature:SecurityPenetration] Security Penetration Tests TestNoUnexpectedPrivilegedPods [Suite:openshift/conformance/parallel] — 0.0% global, 0.0% IPv6, 0.0% metal; [sig-auth][Feature:SecurityPenetration] Security Penetration Tests TestAllRoutesUseTLS [apigroup:route.openshift.io] [Suite:openshift/conformance/parallel] — 13.2% global, 16.9% IPv6, 12.0% metal; [OTP][sig-baremetal][Level0] IPI BareMetal Author:jhajyahy-Medium-34195-Verify all pods replicas are running on workers only — 67.7% global, 0.0% IPv6, 67.7% metal; [OTP][sig-baremetal][Level0] IPI BareMetal Author:jhajyahy-Medium-32361-Verify that deployment exists and is not empty — 67.7% global, 0.0% IPv6, 67.7% metal; cluster-config-operator should have a healthy deployment in openshift-config-operator namespace [apigroup:apps][Operator][Parallel] — 78.4% global, 41.0% IPv6, 58.8% metal; [sig-network] Conntrack proxy implementation should not be vulnerable to the invalid conntrack state bug [Privileged] — 99.7% global, 98.7% IPv6, 99.5% metal; [sig-arch] Managed cluster should have operators on the cluster version [apigroup:config.openshift.io] [Suite:openshift/conformance/parallel] — 99.9% global, 100.0% IPv6, 100.0% metal. Exact individual rates were unavailable for the remaining five SecurityPenetration tests, so they are not inferred.

Open regressions: No separate Component Readiness regression record was retrieved for the remaining failure groups. The conntrack failure is explicitly covered by the open regression tracked in OCPBUGS-90520.

Linked bugs: OCPBUGS-90520 is currently ASSIGNED and explicitly linked via bug_tests to [sig-network] Conntrack proxy implementation should not be vulnerable to the invalid conntrack state bug [Privileged]. The Jira description also records this test as consistently failing on metal/techpreview variants.

Overlap assessment: The PR changes only pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go, adding AWS-only IngressController scope detection and the AWS internal-load-balancer annotation. The failing tests cover SecurityPenetration, bare-metal pod/deployment health, cluster-config-operator health, conntrack, and cluster-version operator health. There is no plausible direct overlap; the only indirect relationship is the shared origin test image. The failing run was on Equinix metal IPv6, so the PR's AWS-specific path was not the tested surface.

Missing-coverage risk: Low for this failure decision. The job's failed coverage is unrelated to the AWS-only code path, and the remaining completed AWS/GCP e2e jobs plus unit/lint/verify/images checks are green. The metal IPv6 job remains a useful platform signal, but its broad infrastructure failure does not provide meaningful evidence against this change.

Prior bot activity on this SHA: One /test e2e-metal-ipi-ovn-ipv6 was triggered at 2026-09-18T20:39:17Z; no prior override for this job was recorded.

Rationale: The job is an eligible long-running e2e/integration job, the revision matches, tests ran, and fleet/platform evidence shows the observed failures are pre-existing systemic IPv6-metal/infrastructure failures rather than regressions in this AWS-only monitor-test change. A Prow controller-manager Degraded outage also overlapped part of the run. Override is appropriate.

If you disagree with this assessment, rerun the current job with /test e2e-metal-ipi-ovn-ipv6.


AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

@tthvo: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants