TRT-2788: add scheduling Prometheus monitors for build-farm clusters and add missing clusters

[smg247]

Adding 4 new Degraded prometheus queries for all build-farm clusters:

1. Cluster autoscaler health — checks that cluster-autoscaler-default deployment has available replicas (detects autoscaler being down)
2. Failed scale-ups — checks for zero cluster_autoscaler_failed_scale_ups_total increase over 30m (detects cloud API errors, expander failures)
3. Unschedulable pods — checks cluster_autoscaler_unschedulable_pods_count <= 20 over 15m (detects slow provisioning, taint mismatches, general scheduling failures)
4. CI scheduling webhook health — checks ci-scheduling-admission-webhook deployment has available replicas

Also added the missing build12 and build13, and put build02 back since it is up again.

Summary by CodeRabbit

• Added Degraded Prometheus monitors for cluster-autoscaler health, failed scale-ups, unschedulable pods, and scheduling webhook availability across build-farm clusters.
• Kept the scheduling webhook monitor healthy when its metric is absent.
• Replaced the pending-pod count monitor with targeted scheduling probes.
• Activated build02 and added build12 and build13 to the build-farm configuration.
• Updated monitor generation and console URLs for builds build01 through build13.

[openshift-ci-robot]

@smg247: This pull request references TRT-2788 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Adding 2 new Degraded prometheus queries for all build-farm clusters:

1. Pending pods: (count(kube_pod_status_phase{...,phase="Pending",...} == 1) or vector(0)) <= 50
   Counts pods stuck in Pending state in CI namespaces (ci-op-*). Healthy when 50 or fewer. Fires Degraded after 10 minutes above that threshold. The failure_query returns the actual count for debugging.

2. ci-scheduling-webhook: kube_deployment_status_replicas_available{...,deployment="ci-scheduling-admission-webhook"} > 0 or absent(...)
   Checks that the scheduling webhook has at least one available replica. The absent() clause keeps it healthy if the metric doesn't exist (e.g., cluster doesn't have the webhook yet). Fires Degraded after 5 minutes with zero replicas. The failure_query returns replica counts for debugging.

Also added the missing build12 and build13, and put build02 back since it is up again.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Walkthrough

The build-farm monitor generator now covers build01 through build13. It creates cluster-local Prometheus checks for autoscaler and scheduling health. SHIP Status activates Build02, Build12, and Build13 dashboard components.

Changes

Build-farm monitoring

Layer / File(s)	Summary
Extend monitor generation hack/generate-build-farm-monitor-config.py	The generator adds build12 and build13 URLs and emits four cluster-local Prometheus checks with failure queries and Degraded severity.
Apply cluster monitor entries core-services/ship-status/component-monitor-config.yaml	The generated configuration adds autoscaler, failed scale-up, unschedulable pod, and scheduling webhook checks for build01 through build13.
Enable dashboard components core-services/ship-status/dashboard-config.yaml	The dashboard configuration activates Build02, Build12, and Build13 monitored sub-components.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BuildMonitorGenerator
  participant ComponentMonitorConfig
  participant DashboardConfig
  BuildMonitorGenerator->>ComponentMonitorConfig: Generate build01-build13 Prometheus monitors
  ComponentMonitorConfig->>DashboardConfig: Provide monitored build components
  DashboardConfig-->>DashboardConfig: Activate Build02, Build12, and Build13

Loading

Possibly related PRs

• openshift/release#82725: Adds Build12 and Build13 support to related SHIP Status monitoring configuration.

Suggested reviewers: danilo-gemoli, psalajova

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	This PR modifies only YAML configuration and Python code files. It contains no Go test files, Ginkgo tests, or test names of any kind. The custom check does not apply.
Test Structure And Quality	✅ Passed	This pull request modifies only monitoring configuration (YAML) and a Python automation script. It contains no Ginkgo test files, so the test structure review check is not applicable.
Microshift Test Compatibility	✅ Passed	The PR changes only YAML monitoring configuration and a Python generator; no Go or Ginkgo e2e tests were added, so MicroShift test compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR does not add any Ginkgo e2e tests. Changes are monitoring config files and a Python generation script. SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff changes only ship-status monitoring YAML and its generator; it adds no deployment manifest, operator/controller code, or topology scheduling constraint.
Ote Binary Stdout Contract	✅ Passed	The PR changes only YAML and a Python config generator; it adds no OTE binary, Go suite entry point, or process-level stdout write.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR changes only Prometheus YAML and a Python generator; no Go test files or added Ginkgo declarations are present, so this check is not applicable.
No-Weak-Crypto	✅ Passed	No weak cryptographic algorithms (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or non-constant-time secret comparisons detected. PR contains only monitoring configurati...
Container-Privileges	✅ Passed	PR modifies monitoring configuration files and a monitoring config generator script. No Kubernetes manifests or container specifications with privileged settings are present in the changes.
No-Sensitive-Data-In-Logs	✅ Passed	The PR adds Prometheus monitoring configurations with numeric metrics (replica counts, pod counts, scale-up failures) and availability checks. No logging, credentials, tokens, API keys, PII, or oth...
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: adding scheduling Prometheus monitors and restoring missing build-farm clusters.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

hack/generate-build-farm-monitor-config.py (1)

91-92: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Align the webhook failure_query metric with the health query metric.

The health query on Line 91 evaluates kube_deployment_status_replicas_available, but failure_query on Line 92 evaluates kube_deployment_status_replicas (desired/current replicas, not available replicas). If the check degrades because available replicas dropped to 0 while the deployment still reports 1 desired replica, the failure query surfaces a "1", which does not explain why the check failed. Use kube_deployment_status_replicas_available in the failure query so debugging output reflects the actual metric driving the alert.

This fix applies to the generator, so it propagates to all 13 generated webhook check pairs in core-services/ship-status/component-monitor-config.yaml.

♻️ Proposed fix

         - query: "kube_deployment_status_replicas_available{{namespace=\\"ci-scheduling-webhook\\",deployment=\\"ci-scheduling-admission-webhook\\"}} > 0 or absent(kube_deployment_status_replicas_available{{namespace=\\"ci-scheduling-webhook\\",deployment=\\"ci-scheduling-admission-webhook\\"}})"
-          failure_query: "kube_deployment_status_replicas{{namespace=\\"ci-scheduling-webhook\\",deployment=\\"ci-scheduling-admission-webhook\\"}}"
+          failure_query: "kube_deployment_status_replicas_available{{namespace=\\"ci-scheduling-webhook\\",deployment=\\"ci-scheduling-admission-webhook\\"}}"

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/generate-build-farm-monitor-config.py` around lines 91 - 92, Update the
webhook failure_query metric in the generator’s configuration template to use
kube_deployment_status_replicas_available, matching the health query’s metric
while preserving its existing namespace and deployment selectors. Ensure this
generator change propagates the aligned metric to all generated webhook check
pairs.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/generate-build-farm-monitor-config.py`:
- Around line 91-92: Update the webhook failure_query metric in the generator’s
configuration template to use kube_deployment_status_replicas_available,
matching the health query’s metric while preserving its existing namespace and
deployment selectors. Ensure this generator change propagates the aligned metric
to all generated webhook check pairs.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 9bbd4fef-bedc-4a74-a9f4-5f09ed5dae3e

📥 Commits

Reviewing files that changed from the base of the PR and between aa94289 and 59c2f40.

📒 Files selected for processing (3)

• core-services/ship-status/component-monitor-config.yaml
• core-services/ship-status/dashboard-config.yaml
• hack/generate-build-farm-monitor-config.py

[deepsm007]

We will need to work on improving the reasoning with outage/alert. Starting with these basic alerts help us zone in the actual problem area.

[smg247]

/hold

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@smg247: no rehearsable tests are affected by this change

Note: If this PR includes changes to step registry files (ci-operator/step-registry/) and you expected jobs to be found, try rebasing your PR onto the base branch. This helps pj-rehearse accurately detect changes when the base branch has moved forward.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

@smg247: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deepsm007, smg247

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [deepsm007,smg247]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment