Self-hosted worker: don't fail tasks on transient Kubernetes re-watch and FailedMount errors

[warp-agent-staging]

Fixes REMOTE-2701.

Symptom

Three self-hosted worker task failures in one aggregated error report, all cluster-side and transient:

Failed to execute task: failed to re-watch Pods for Job oz-task-...-exec-4150627: Get "https://100.100.0.1:443/api/v1/namespaces/warp-oz/pods?...&watch=true": dial tcp 100.100.0.1:443: connect: connection refused
Failed to execute task: failed to re-watch Job oz-task-...-exec-4150558: ... connect: connection refused
Failed to execute task: pod oz-task-...-5f9sv failed to mount a volume. Kubernetes event reason: FailedMount. Kubernetes event message: MountVolume.SetUp failed for volume "istiod-ca-cert" : failed to sync configmap cache: timed out waiting for the condition

Cause

In internal/worker/kubernetes.go:

• A brief API server outage failed the task twice over. When a Job or Pod watch closed or emitted watch.Error, the loop re-established it exactly once and failed the whole task if that single call errored. The 30s safety poll that was supposed to be the fallback was no better: it returned a task failure on any error from Jobs().Get() or listTaskPods(). So a momentary connection refused killed a running task through whichever path hit it first.
• FailedMount failed on the first event. Any FailedMount event (or any message containing MountVolume.SetUp failed) failed the task immediately, including kubelet conditions that Kubernetes retries on its own.

What changed

Watches survive a blip. A watchStream helper owns each watch and its reconnect policy. A failed reopen detaches the stream — resultChan() returns a nil channel, which is never ready in the select — instead of failing the task, and reconnects from a dedicated 1s ticker with exponential backoff (1s, doubling, capped at 30s, reset on success). Reconnect attempts are rate-limited to one per second whatever the outcome, so a watch that is re-established and then immediately closed can no longer spin the loop against a struggling API server. open runs under a watchdog that cancels only while the call is in flight, so a blackholed API server cannot stall the task loop on the dialer; the established watch keeps its own context, which client-go needs for the watch's lifetime. The initial watch failure at task start still fails the task, as before.

The safety poll survives the same blip. Consecutive poll failures are tolerated for 3 minutes before the task fails, so the poll can actually carry a task while the watches are down — which is what the whole design rests on. A Job that is genuinely gone (IsNotFound) and a cancelled context still fail immediately, and a single successful poll clears the streak.

FailedMount gets a bounded grace window, aged by the events themselves. New backend.kubernetes.volume_mount_timeout (default 2m, 0s disables), wired through the config file, mergeConfig, the Helm chart, and the README. The window is measured across the mount-failure events rather than from pod creation: a mount is only attempted once the pod is scheduled and its images are available, so pod age would already be spent before the first attempt on a slow-to-schedule pod, and would also fail pods on an event the kubelet resolved long ago. The failures must have started at least volume_mount_timeout ago and must still be current; a pod that has stopped reporting them is treated as recovered.

Two callers explicitly opt out of the grace window, because they have nothing left to wait for and the event is the most useful diagnostic: the startup preflight (which runs under its own 15s timeout and exists to surface cluster incompatibilities) and pod inspection for a Job that has already failed terminally. This is a named volumeMountGrace argument rather than an implicit rule.

Nothing else from the error report is touched; the other rows are handled separately.

Validation

• gofmt -s -l ., go build ./..., go vet ./..., golangci-lint run — all clean
• go test ./... — all packages pass; go test ./internal/worker -race also passes

New tests, each of which fails without its corresponding fix:

• TestExecuteTaskCompletesThroughSafetyPollWhileWatchesAreDown — both watches are dropped and never come back, the first poll fails with connection refused, and the task still completes off the next poll. With the poll tolerance removed this fails with failed to get Job ... connect: connection refused, which is the original incident.
• TestExecuteTaskSurvivesFailedPodRewatch — a pod watch that closes and cannot be re-established no longer fails the task.
• TestWatchStreamRateLimitsReopenAttempts — 1000 back-to-back reopens produce a single attempt.
• TestWatchStreamDetachesAndReconnectsAfterFailedReopen, TestNextWatchReopenBackoffIsBounded, TestSafetyPollFailuresAreOnlyTerminalOncePersistent — reconnect and tolerance policy in isolation.
• TestInspectPodFailureRespectsVolumeMountTimeout — a just-started mount failure on an old pod is tolerated, a persistent one fails within the bounded window, a failure the kubelet stopped reporting is ignored, 0s disables it, and the fail-fast callers still fail on the first event.
• Config coverage for volume_mount_timeout parsing and for an invalid value.

The existing TestRunStartupPreflightFailsOnFailedMountEvent still passes unchanged, so preflight diagnostics are unaffected.

Conversation: https://staging.warp.dev/conversation/09c693c8-299b-40e3-a08e-529eb6a75d8e
Run: https://oz.staging.warp.dev/runs/019ff348-9c2a-77a4-a7a7-3b6e28b633ab

This PR was generated with Oz.

[warp-agent-staging]

Overview

Makes the self-hosted worker tolerate transient Kubernetes watch and mount failures instead of failing the task on first sight (REMOTE-2701). The mechanical findings are going back to the author; these two are judgment calls that need a human.

Concerns

• The preflight still fails fast on a mount error (internal/worker/kubernetes.go:940-946). The transient in the report is istiod-ca-cert failing to mount — an Istio-injected volume that lands on every pod in the namespace, including the preflight pod — so with failFastOnMountFailures there, the same blip makes NewKubernetesBackend return an error and the worker refuse to start, moving the amplification this issue is about to startup. The preflight already runs under a 15s deadline that bounds the wait, so honoring the grace window and folding the event into the timeout message would keep the cluster-incompatibility diagnostic while surviving the blip — but that trades a crisp startup error for a slower one, so it is your call.
• volume_mount_timeout: "0s" disables mount failure detection entirely. With backoffLimit: 0, a pod that can never mount then hangs until the Job's activeDeadlineSeconds (chart default 8h), or indefinitely if that is unset. Worth deciding whether operators should be able to turn this off at all, or whether the floor should be a long timeout rather than "never".

Verdict

Checks: build pass, tests pass, CI green, visual proof n/a

Found: 0 critical, 0 important, 0 suggestions, 0 nits, 2 questions

[warp-agent-staging]

Revised in af00ab3. The review's two questions are deliberately untouched and left for the requester: the preflight failFastOnMountFailures carve-out, and whether volume_mount_timeout: "0s" should be able to disable mount-failure detection at all.

Everything else raised in review is addressed:

• The safety poll now tolerates the same outage the watch does. This was the real gap — the poll returned a task failure on any error from Jobs().Get() or listTaskPods(), so the connection refused that the re-watch survived still failed the task at the next tick, and the task only lived if the outage fell between ticks. Consecutive poll failures are now terminal only after 3 minutes; a missing Job (IsNotFound) and a cancelled context still fail immediately, and a fully successful poll clears the streak.
• The mount grace window is aged by the events, not the pod. shouldFailVolumeMount now takes the pod's mount-failure events: they must have started at least volume_mount_timeout ago and must still be current. That fixes both directions of the old rule — a pod slow to schedule no longer arrives at its first mount attempt with the grace already spent, and a stale event the kubelet resolved long ago no longer fails a pod that is pending for an unrelated reason.
• Reconnects are rate-limited whatever the outcome, so the success-then-immediate-close path can no longer spin against a struggling API server (the 1000-reopens-in-a-loop case now makes exactly one attempt).
• The backoff is real now. Retries are driven by a dedicated 1s ticker instead of the 30s safety ticker, so the 1s first step actually means something; the PR description has been corrected to match.
• open runs under a watchdog that cancels only while the call is in flight, so a blackholed API server cannot stall the task loop on the dialer. The established watch keeps its own context, which client-go needs for the watch's lifetime.
• The chart omits volume_mount_timeout when unset rather than rendering an empty value the worker rejects at startup.

New tests for the two gaps that were untested: TestExecuteTaskCompletesThroughSafetyPollWhileWatchesAreDown drops both watches, fails the first poll, and asserts the task still completes off the next one — it fails with failed to get Job ... connect: connection refused when the tolerance is removed, which is the original incident. The mount test now drives the window through event timestamps and covers the failure the kubelet stopped reporting. watchSafetyInterval is injectable through an unexported field so the poll can be observed without waiting on the production cadence.

CI is green.