csi-node: add optional startupProbe for slow-starting drivers#197
Merged
Conversation
CSI drivers that perform device cleanup at startup (e.g. Huawei eSDK garbage collector) can block the gRPC server for minutes. The existing livenessProbe (initialDelaySeconds=5, default failureThreshold=3) kills the pod before it becomes ready, creating a crash loop. Add a `startupProbeFailureThreshold` config parameter: when set to a positive integer, a Kubernetes startupProbe is rendered on the node container. While the startup probe is active the kubelet suspends liveness checks, giving the driver enough time to finish initialisation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
duckhawk
force-pushed
the
fix/csi-node-startup-probe
branch
from
a8f7be2 to
235a895
Compare
ldmonster
approved these changes
Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a
startupProbeFailureThresholdconfig parameter to the CSI nodeDaemonSet template (
_csi_node.tpl). When set to a positive integer, aKubernetes
startupProbeis rendered on thenodecontainer using thesame
/healthzHTTP endpoint as the existing liveness probe.While the startup probe is active the kubelet suspends liveness checks,
giving the driver enough time to finish initialisation before the pod is
considered unhealthy.
Why do we need it, and what problem does it solve?
CSI drivers that perform device cleanup at startup (e.g. Huawei eSDK
garbage collector,
triggerGarbageCollector()) can block the gRPCserver for up to 240 seconds. The existing
livenessProbe(
initialDelaySeconds: 5, defaultfailureThreshold: 3,periodSeconds: 10) kills the pod after ~35 s, well before startupcompletes. This creates an infinite crash loop where stale multipath
maps are never cleaned and trigger the next restart.
What is the expected result?
Modules that set
startupProbeFailureThresholdin their CSI node config(e.g.
30→ 300 s max startup time) will have astartupProberenderedin the DaemonSet manifest. Modules that do not set the parameter are
unaffected (backward compatible, defaults to
0/ disabled).Checklist