-
Notifications
You must be signed in to change notification settings - Fork 6
HYPERFLEET-1409 - feat: report status conditions on HyperFleetConfig #18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # Status conditions | ||
|
|
||
| `HyperFleetConfig` reports its installation health via `status.conditions`. This | ||
| is deliberately a *different* vocabulary from the HyperFleet API's own | ||
| condition dialect — the two are separate layers, and both are documented here | ||
| so it's clear which one you're looking at. | ||
|
|
||
| ## Two layers | ||
|
|
||
| - **Operator layer** (this document): `status.conditions` on the | ||
| `HyperFleetConfig` CR itself. It describes whether the operator has | ||
| successfully installed and is maintaining a healthy deployment of the | ||
| operand(s) selected by `spec.bundle` — i.e. "is the operator doing its job." | ||
| This follows the OpenShift `ClusterOperator` convention (`Available`, | ||
| `Progressing`, `Degraded`), minus `Upgradeable` (see architecture ADR-0019). | ||
| - **API layer**: conditions reported by the HyperFleet API's own resources | ||
| (`Available`, `Ready`, `Reconciled`, `LastKnownReconciled`, per-adapter | ||
| `Successful`, etc. — see architecture ADR-0007 and ADR-0008). These describe | ||
| whether the API is successfully reconciling partner-managed resources | ||
| (clusters, node pools, etc.) — i.e. "is the API doing its job." They are | ||
| unrelated to and unaffected by the operator-layer conditions here. | ||
|
|
||
| A `HyperFleetConfig` can be `Available=True` (the API Deployment is healthy) | ||
| while individual API-layer resources are failing to reconcile, and vice versa: | ||
| the operator conditions say nothing about the health of resources managed | ||
| through the API. | ||
|
|
||
| ## Operator-layer condition types | ||
|
|
||
| | Type | Meaning | | ||
| |---|---| | ||
| | `Available` | The operand (the API) is deployed and healthy. | | ||
| | `Progressing` | The operator is actively rolling out a change to the operand. | | ||
| | `Degraded` | The operator cannot reach or maintain the desired state. | | ||
|
|
||
| Each condition's `observedGeneration` (and the top-level | ||
| `status.observedGeneration`) reflects the `.metadata.generation` the operator | ||
| had processed when the condition was last evaluated. `lastTransitionTime` | ||
| changes only when a condition's `status` actually flips — reconciles that | ||
| leave health unchanged do not touch it. | ||
|
|
||
| If a reconcile fails before any component's health could actually be checked | ||
| (e.g. a failed OIDC discovery, or a failed apply on the first operand), | ||
| `Available` and `Progressing` are left exactly as they were last recorded | ||
| rather than being guessed — only `Degraded` and `observedGeneration` are | ||
| updated. This avoids publishing a fabricated "healthy" value alongside | ||
| `Degraded=True` when the real state was never actually observed. | ||
|
|
||
| ## Reason strings | ||
|
|
||
| Every condition write uses one of the following reasons — no ad hoc strings. | ||
| Partners may read `status.conditions[].reason` as part of the published | ||
| contract, so this table is the source of truth; the Go constants live | ||
| alongside the condition types in `api/v1alpha1/hyperfleetconfig_types.go`. | ||
|
|
||
| | Condition | Status | Reason | Meaning | | ||
| |---|---|---|---| | ||
| | Available | True | `DeploymentAvailable` | The operand Deployment reports Available; all desired replicas are ready. | | ||
| | Available | False | `DeploymentUnavailable` | The operand Deployment is missing, or has zero available replicas. | | ||
| | Available | False | `DeploymentNotReady` | The operand Deployment exists with some, but not all, replicas ready. | | ||
| | Progressing | True | `RolloutInProgress` | The operand Deployment has not finished rolling out its current generation. | | ||
| | Progressing | False | `RolloutComplete` | The operand Deployment is fully rolled out and stable. | | ||
| | Degraded | False | `AsExpected` | No failure detected (the ClusterOperator convention's default). | | ||
| | Degraded | True | `ReferencedSecretMissing` | A Secret referenced by `spec.api` (database, TLS, or JWKS) does not exist in the operator's namespace. | | ||
| | Degraded | True | `ReconcileError` | Any other error during the most recent reconcile — JWKS discovery, reading referenced Secrets, resolving bundle components, or a component's render/apply. | | ||
|
|
||
| ## Notes on `Available` | ||
|
|
||
| The operand's readiness probe (`/readyz`) only succeeds once the API has | ||
| established a working database connection, so `Available=True` also implies | ||
| the API can reach its configured PostgreSQL database — not just that the pod | ||
| is running (`/healthz`, the liveness probe, does not check this). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,6 +20,7 @@ import ( | |
| "context" | ||
| "fmt" | ||
|
|
||
| appsv1 "k8s.io/api/apps/v1" | ||
| metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
| "sigs.k8s.io/controller-runtime/pkg/client" | ||
|
|
||
|
|
@@ -161,11 +162,117 @@ func (c *Component) resolveJWKSource(cr *hyperfleetv1alpha1.HyperFleetConfig) (u | |
| return c.ResolvedJWKSURL, "" | ||
| } | ||
|
|
||
| // Conditions reports the component's health as metav1.Conditions. The contract is | ||
| // defined now (HYPERFLEET-1407) so it need not be reopened next story, but its | ||
| // output is not yet rolled up into status.conditions — real health derivation and | ||
| // status wiring land in HYPERFLEET-1409. Returning nil until then keeps the | ||
| // reconciler from writing status prematurely. | ||
| func (c *Component) Conditions(_ context.Context, _ *hyperfleetv1alpha1.HyperFleetConfig) ([]metav1.Condition, error) { | ||
| return nil, nil | ||
| // Conditions reports the component's health as metav1.Conditions, derived from | ||
| // the live state of the Deployment Render produced (already updated in place by | ||
| // apply.Objects with the server's current status — see bundle.Component). It | ||
| // never reads the cluster itself. | ||
| // | ||
| // Available reflects whether the Deployment is up: True only once all desired | ||
| // replicas are ready, distinguishing "not present at all" (DeploymentUnavailable) | ||
| // from "up but partially ready" (DeploymentNotReady). Progressing is derived from | ||
| // replica-count/generation lag rather than trusting the Deployment's own | ||
| // "Progressing" condition Reason verbatim, since that built-in condition stays | ||
| // True/NewReplicaSetAvailable even at steady state, which is not what | ||
| // HyperFleetConfig's Progressing means. | ||
| func (c *Component) Conditions(_ context.Context, _ *hyperfleetv1alpha1.HyperFleetConfig, applied []client.Object) ([]metav1.Condition, error) { | ||
| dep := findDeployment(applied) | ||
| return []metav1.Condition{ | ||
| availableCondition(dep), | ||
| progressingCondition(dep), | ||
| }, nil | ||
| } | ||
|
|
||
| // findDeployment returns the API Deployment from applied, or nil if absent. | ||
| func findDeployment(applied []client.Object) *appsv1.Deployment { | ||
| for _, o := range applied { | ||
| if dep, ok := o.(*appsv1.Deployment); ok && dep.Name == ResourceName { | ||
| return dep | ||
| } | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // desiredReplicas returns dep.Spec.Replicas, defaulting to 1 to match | ||
| // render.go's current fixed baseline (Replicas is not yet configurable). | ||
| func desiredReplicas(dep *appsv1.Deployment) int32 { | ||
| if dep.Spec.Replicas != nil { | ||
| return *dep.Spec.Replicas | ||
| } | ||
| return 1 | ||
| } | ||
|
|
||
| // availableCondition derives Available from the Deployment's replica counts. | ||
| func availableCondition(dep *appsv1.Deployment) metav1.Condition { | ||
| if dep == nil { | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionAvailable, | ||
| Status: metav1.ConditionFalse, | ||
| Reason: hyperfleetv1alpha1.ReasonDeploymentUnavailable, | ||
| Message: "the API Deployment does not exist", | ||
| } | ||
| } | ||
|
|
||
| desired := desiredReplicas(dep) | ||
|
|
||
| switch { | ||
| case desired == 0: | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionAvailable, | ||
| Status: metav1.ConditionTrue, | ||
| Reason: hyperfleetv1alpha1.ReasonDeploymentAvailable, | ||
| Message: "the API Deployment is scaled to zero", | ||
| } | ||
| case dep.Status.AvailableReplicas == 0: | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionAvailable, | ||
| Status: metav1.ConditionFalse, | ||
| Reason: hyperfleetv1alpha1.ReasonDeploymentUnavailable, | ||
| Message: "the API Deployment has no available replicas", | ||
| } | ||
| case dep.Status.AvailableReplicas < desired: | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionAvailable, | ||
| Status: metav1.ConditionFalse, | ||
| Reason: hyperfleetv1alpha1.ReasonDeploymentNotReady, | ||
| Message: fmt.Sprintf("the API Deployment has %d/%d replicas available", dep.Status.AvailableReplicas, desired), | ||
| } | ||
| default: | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionAvailable, | ||
| Status: metav1.ConditionTrue, | ||
| Reason: hyperfleetv1alpha1.ReasonDeploymentAvailable, | ||
| Message: "the API Deployment is available", | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // progressingCondition derives Progressing from replica-count and generation | ||
| // lag: the Deployment has not caught up with its most recently observed | ||
| // generation, or not all replicas have been updated to the current template. | ||
| func progressingCondition(dep *appsv1.Deployment) metav1.Condition { | ||
| if dep == nil { | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionProgressing, | ||
| Status: metav1.ConditionTrue, | ||
| Reason: hyperfleetv1alpha1.ReasonRolloutInProgress, | ||
| Message: "the API Deployment does not exist yet", | ||
| } | ||
| } | ||
|
|
||
| desired := desiredReplicas(dep) | ||
|
|
||
| if dep.Status.ObservedGeneration < dep.Generation || dep.Status.UpdatedReplicas < desired { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
set -euo pipefail
# Confirm the declared Kubernetes dependency version and inspect whether the
# Deployment replica-surplus scenario is covered by API component tests.
rg -n 'k8s.io/(api|apimachinery)' go.mod
rg -n -C 8 'progressingCondition|UpdatedReplicas|Status\.Replicas|Replicas:' \
internal/component/api/api_test.go internal/component/api/api.goRepository: openshift-hyperfleet/hyperfleet-operator Length of output: 5755 🏁 Script executed: #!/bin/bash
set -euo pipefail
sed -n '249,285p' internal/component/api/api.go
sed -n '320,470p' internal/component/api/api_test.go
rg -n -C 5 'func desiredReplicas|desiredReplicas\(' internal/component/api
rg -n 'type DeploymentStatus struct|Replicas.*int32|UpdatedReplicas.*int32' "$(go env GOPATH 2>/dev/null)/pkg/mod/k8s.io/api@v0.33.0/apps/v1/types.go" 2>/dev/null || trueRepository: openshift-hyperfleet/hyperfleet-operator Length of output: 10070 🤖 get_repo_knowledge executed:
Length of output: 20528 Keep Line 264 checks only 🤖 Prompt for AI Agents
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Building on the CodeRabbit comment above, it's not just the surplus-replica case. The predicate Worth matching all four and adding a table case per clause in |
||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionProgressing, | ||
| Status: metav1.ConditionTrue, | ||
| Reason: hyperfleetv1alpha1.ReasonRolloutInProgress, | ||
| Message: "the API Deployment rollout has not completed", | ||
| } | ||
| } | ||
| return metav1.Condition{ | ||
| Type: hyperfleetv1alpha1.ConditionProgressing, | ||
| Status: metav1.ConditionFalse, | ||
| Reason: hyperfleetv1alpha1.ReasonRolloutComplete, | ||
| Message: "the API Deployment rollout is complete", | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not for this PR, but worth a ticket: a bad image tag leaves the Deployment with
ProgressDeadlineExceededforever, and with this logic we'd sit atProgressing=True, Degraded=Falseindefinitely. ClusterOperator convention would flipDegradedthere. The Deployment's ownProgressingcondition with reasonProgressDeadlineExceededis the signal to key off. Is there a follow up for this?