Skip to content

OCPBUGS-54864: reduce verbosity of skipping metrics log messages - #1439

Open
savio87 wants to merge 1 commit into
openshift:mainfrom
savio87:OCPBUGS-54864
Open

OCPBUGS-54864: reduce verbosity of skipping metrics log messages#1439
savio87 wants to merge 1 commit into
openshift:mainfrom
savio87:OCPBUGS-54864

Conversation

@savio87

@savio87 savio87 commented Aug 5, 2026

Copy link
Copy Markdown

Bug

https://redhat.atlassian.net/browse/OCPBUGS-54864

The CVO pod logs excessive "skipping metrics" messages on healthy clusters. The "skipping metrics for ... ClusterOperator condition ...=Unknown (neither True nor False)" message fires on every Prometheus scrape (~30s) for every ClusterOperator condition that is Unknown, producing 61,919+ log lines per day on a fresh 4.17.z cluster.

Root Cause

The log uses klog.V(2), but CVO runs with --v=2 in production, so these messages always appear. Since Unknown is a normal condition state (not an error), this is purely diagnostic noise.

Fix

Change klog.V(2) to klog.V(4) for both "skipping metrics" log lines in pkg/cvo/metrics.go:

  • Line 665: ClusterVersion conditions
  • Line 728: ClusterOperator conditions

This keeps the log available for debugging (at --v=4) but silences it in production (--v=2).

Summary by CodeRabbit

  • Chores
    • Increased diagnostic logging detail for skipped non-boolean cluster conditions.
    • Metric behavior remains unchanged.

@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 5, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@savio87: This pull request references Jira Issue OCPBUGS-54864, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Bug

https://redhat.atlassian.net/browse/OCPBUGS-54864

The CVO pod logs excessive "skipping metrics" messages on healthy clusters. The "skipping metrics for ... ClusterOperator condition ...=Unknown (neither True nor False)" message fires on every Prometheus scrape (~30s) for every ClusterOperator condition that is Unknown, producing 61,919+ log lines per day on a fresh 4.17.z cluster.

Root Cause

The log uses klog.V(2), but CVO runs with --v=2 in production, so these messages always appear. Since Unknown is a normal condition state (not an error), this is purely diagnostic noise.

Fix

Change klog.V(2) to klog.V(4) for both "skipping metrics" log lines in pkg/cvo/metrics.go:

  • Line 665: ClusterVersion conditions
  • Line 728: ClusterOperator conditions

This keeps the log available for debugging (at --v=4) but silences it in production (--v=2).

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Walkthrough

The change raises log verbosity for skipped non-boolean ClusterVersion and ClusterOperator conditions. Metric behavior remains unchanged.

Changes

Condition logging

Layer / File(s) Summary
Raise skipped condition log verbosity
pkg/cvo/metrics.go
Skipped ClusterVersion and ClusterOperator condition logs now use verbosity 4 instead of verbosity 2.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: hongkailiu

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The patch changes only two klog verbosity calls in pkg/cvo/metrics.go and adds no Ginkgo test declarations or test titles.
Test Structure And Quality ✅ Passed The commit changes only two klog verbosity calls in pkg/cvo/metrics.go; it adds no Ginkgo tests, resource setup, waits, or assertions to review.
Microshift Test Compatibility ✅ Passed The patch changes only two klog verbosity calls in pkg/cvo/metrics.go and adds no Ginkgo tests or test declarations requiring MicroShift compatibility review.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR changes only two log verbosity calls in pkg/cvo/metrics.go and adds no Ginkgo e2e tests, so the SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The PR changes only two klog verbosity levels in pkg/cvo/metrics.go. It adds no deployment manifests, scheduling constraints, replica logic, selectors, affinity, tolerations, or PDBs.
Ote Binary Stdout Contract ✅ Passed The commit only changes two klog verbosity levels in metrics.go; the OTE main and suite have no process-level stdout writes, and OTE/Ginkgo and klog default output goes to stderr.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR modifies log verbosity in pkg/cvo/metrics.go production code, not Ginkgo e2e tests. The check for IPv6/disconnected test compatibility does not apply.
No-Weak-Crypto ✅ Passed The PR changes only two klog verbosity levels in pkg/cvo/metrics.go; no MD5, SHA1, DES, RC4, Blowfish, ECB, crypto implementation, or secret comparison was added.
Container-Privileges ✅ Passed The PR changes only two klog verbosity calls in pkg/cvo/metrics.go; it adds no container or Kubernetes manifest privilege setting.
No-Sensitive-Data-In-Logs ✅ Passed The two changed logs emit only ClusterVersion condition type/status and ClusterOperator name, condition type/status; no passwords, tokens, keys, PII, hostnames, or customer data.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: reducing verbosity for skipped metrics log messages.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: savio87
Once this PR has been reviewed and has the lgtm label, please assign davidhurta for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Hi @savio87. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 5, 2026
@savio87

savio87 commented Aug 5, 2026

Copy link
Copy Markdown
Author

Hi, this is my first contribution to this repo. Could a maintainer please:
Set the target version on the linked Jira issue OCPBUGS-54864
Run /ok-to-test to trigger CI
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants