Collect self-hosted worker logs for debug archives

[warp-agent-staging]

The problem

Warp assembles a HAR-style debug archive for a cloud-agent run, but for self-hosted executions the logs live inside customer infrastructure that warp-server cannot reach. It cannot query a customer's Docker daemon, Kubernetes cluster, or direct child process.

Worse, the worker removed its ownership record and destroyed the backend resource as soon as an execution ended. The ANY_FAILURE case — the one an operator most wants an archive for — was therefore unrecoverable by construction: by the time the server reacted to task_failed, the container was already gone.

The change

This implements the worker half of the REMOTE-2516 protocol, per the approved spec committed on this branch at .agents/specs/REMOTE-2516-debug-archive-worker-logs.md. The server half is warp-server#13839.

New internal/debuglog package

• Protocol-v1 request validation: identifiers, format, 30-minute expiry ceiling, content-transformer descriptor, byte bound, and upload-target method/scheme/header rules.
• Schema-v1 NDJSON encoder: 32 KiB chunking on UTF-8 boundaries, utf8/base64 encoding selection, source_error records carrying only a stable warning code, and first/last truncation that always cuts on a record boundary.
• Versioned ContentTransformer hook applied while encoding, so the first object uploaded to cloud storage already carries the transformed bytes. V1 ships only the byte-preserving noop@1; an unsupported descriptor uploads nothing.
• Secure capture store: 0700 root, 0600 files, non-user-derived names, O_EXCL creation, startup orphan removal, and a shared process-local disk budget.
• TaskLogCapture: bounded, disk-backed, non-blocking. A full queue drops archive bytes and marks the capture partial rather than back-pressuring the subprocess pipe.
• Immutable request snapshots with CRC32C and SHA-256 over the exact uploaded bytes, so every retry replays identical content.
• Upload client: streaming PUT and multipart POST, redirects refused, bounded exponential backoff that never starts a retry past expiry, and response bodies drained but never logged.
• DebugLogCoordinator: runs entirely off the WebSocket read loop behind an upload semaphore and a per-execution mutex, with a 1,024-entry request cache making duplicate delivery idempotent.

Exact ownership and cleanup grace

• TaskRegistry resolves ownership by the exact (run_id, execution_id) pair. Only the instance that executed the assignment answers; every other instance receiving the Pub/Sub fan-out is completely silent — no acknowledgement, no log, no ID-bearing metric, no cache entry. Ownership is checked before validation so a non-owner stays silent even for a request it would otherwise reject.
• Ownership moves to cleanup grace before the terminal lifecycle message is enqueued, so a request the server triggers off task_failed always finds the grace entry instead of racing registry deletion.
• Retention reuses the execution's already-resolved idle-on-complete grace (task idle_timeout_minutes → worker idle_on_complete → Oz default) rather than adding a second cleanup clock. A request never extends it.

Backend adapters

• Docker registers the exact container before start, no longer force-removes it at terminal return, and demultiplexes the framed log stream into true stdout/stderr records with per-line provider timestamps. The unbounded io.ReadAll diagnostic path is now capped.
• Kubernetes selects pods by exact execution/task/worker label hashes and re-verifies the returned labels, walks pods and containers in deterministic order, attempts previous logs for a restarted container, and turns one unreadable container into a partial upload rather than discarding readable siblings. Successful Jobs are deleted at the grace deadline instead of at task completion; failed Jobs still go to the TTL controller.
• Direct tees setup, agent, and teardown stdout/stderr into the bounded capture while still writing to the worker console.
• Command reports unavailable/backend_not_supported; its dispatch stdout is not the remote agent's log.

Worker build provenance
main.Version now travels as X-Warp-Worker-Version on every authenticated dial and reconnect, so the server can snapshot the exact build that claims an execution. An empty, overlong, or control-character-bearing value is omitted (never logged raw) and the connection still executes tasks.

Non-negotiable: collection is best effort

A capture that cannot be allocated, a provider that cannot be read, a transform that fails, or an upload that is rejected never changes a task's claim, execution result, terminal message, cleanup deadline, or reconnect behavior. Invalid capture bounds or an unwritable capture root disable archive capture and log it; they do not fail assigned task execution.

Verification

gofmt -s, go vet ./..., golangci-lint run (0 issues), go test ./..., go build -v ./..., helm lint, and helm template all pass. The internal/debuglog suite additionally passes under -race -count=2.

Two real defects were caught by the new tests and fixed before this landed:

• TaskLogCapture.Finalize returned once the queue drained, but the background encoder could still be holding the last record — so a terminal snapshot could miss the execution's final output. It now tracks in-flight records.
• The uploader classified a refused redirect as a transport error and retried it for the full backoff window instead of failing immediately as upload_rejected.

Regression tests added:

• Encoder — schema-v1 field shape, chunk bounds and reassembly, base64 round-trip for binary output, empty streams producing no record, first/last truncation with exactly one truncation record and retained records on both sides, and a fake transformer proving only decoded data changes while timestamps, sequence, and identity stay structural.
• Validation — a golden protocol-v1 fixture plus 16 rejection cases each asserting its stable reason code, and the lower-of-two effective byte bound.
• Capture store — 0700/0600 modes, orphan removal, symlink refusal, budget exhaustion and release, per-execution isolation, watermark snapshots that exclude later output, and Write always reporting the child's byte count even when the queue is saturated.
• Upload — PUT body/header fidelity, streaming multipart POST, retry on 408/429/5xx/transport errors with byte-identical replay, terminal 4xx, redirect refusal with the destination never contacted, and no retry started past expiry.
• Coordinator — non-owner silence (including a process owning a different execution of the same run), full acknowledgement field round-trip, partial capture warnings, empty capture as unavailable, command-backend unsupported, duplicate request replay with exactly one upload, reused request ID with different content rejected, expiry and grace lapse while queued, and semaphore-bounded concurrency.
• Worker — version-header acceptance/omission over a real WebSocket upgrade, cleanup-grace transition before terminal reporting for both success and failure, idempotent grace expiry, exact-execution registry lookups, and archive requests dispatched off the read loop.
• Backends — Docker frame demultiplexing and bounded memory over a large log; Kubernetes label scoping, deterministic ordering, and previous-logs attempts; and a real direct execution whose six phase/stream sentinels each appear exactly once in a snapshot taken after workspace cleanup, with no workspace file content captured.
• Config/chart — the exact block the chart renders parses, an unknown retention key is rejected so no second cleanup clock can be configured, and cleanup-grace precedence is proven to match the emitted --idle-on-complete flag.

This is a headless backend change with no rendered UI, so computer-use visual verification does not apply.

Operator-facing notes

README.md documents the sensitive-data implication, the supported backends, sizing the cleanup grace for ANY_FAILURE, Kubernetes ttl_seconds_after_finished alignment, the ephemeral worker-replacement limitation, capture bounds, and version compatibility. The chart gains a bounded 1 GiB ephemeral capture volume and renders the matching config; RBAC is unchanged — the existing namespace-scoped get pods/log grant is sufficient.

Behavior changes worth a close look

• A Docker task container and a successful Kubernetes Job now survive until the cleanup-grace deadline instead of being removed at terminal return. This is the deliberate trade the spec makes for post-failure collection, and it means provider resources are held longer than before. Docker's Shutdown sweeps any still-retained containers so nothing leaks across a worker restart.
• Operators wanting reliable ANY_FAILURE archives may need to lengthen idle_on_complete and raise ttl_seconds_after_finished.

Rework changes

Code review cycle 1 returned two findings, both fixed here (commit 0aea651). A third defect surfaced while validating them.

1. max_bytes was not a hard output limit — internal/debuglog/spool.go
The budget was a retention target, not a ceiling. A valid request could set max_bytes=1 and still receive a complete NDJSON line, because the spool retained any record too large for a tail segment rather than dropping it, and the truncation record was appended on top of an already-full budget.

The spool now reserves room for the truncation record up front, caps head plus both tail segments against the remainder, and drops a record that cannot fit a segment instead of overrunning — counting it in the omitted byte total so the loss is still reported. Finalize skips the truncation record when even that would not fit, leaving an empty object the coordinator already classifies as unavailable, and asserts the ceiling before returning. Retention is still whole records at both ends, so the object stays valid NDJSON at every bound.

Added TestEncoderNeverExceedsItsBound, a sweep over bounds 1, 2, 16, 64, maxTruncationLineBytes±1, 256, 512, 1024, 4096, 65536 crossed with tiny, line-sized, and full-chunk payloads, asserting the finalized size never exceeds the bound and the output still parses; plus targeted tests for a tight-but-usable bound (whole records at both ends, exactly one truncation record), a bound too small to hold any record (empty object, truncated=false), and an oversized single record (dropped and accounted for).

2. Shutdown left terminal cleanup-grace resources behind — internal/worker/worker.go
Shutdown drained cleanup-grace entries without running backend cleanup. Those executions have already reported terminal state and are retained only so their logs stay readable, but ownership and the expiry timer are process-local — a replacement worker cannot inherit them. On Kubernetes, whose shutdown deliberately preserves Jobs, a successful terminal Job therefore outlived its resolved cleanup grace and survived to the 24h TTL default. This turned the deliberate retention change into a leak across worker restarts.

Shutdown now performs the same cleanup the expiry timer would have, under one bounded budget for the whole sweep so it cannot stall on an unresponsive backend. Active executions are untouched: they are not in cleanup grace, so each backend's own shutdown contract still decides whether their task units may outlive the process.

Added tests covering the restart path (terminal grace entry released), the preservation path (an active execution is never cleaned up at shutdown), and direct-capture disk budget release.

3. Data race on the cleanup-grace timer (found while validating the above)
Running the race detector over internal/worker — the previous pass had only raced internal/debuglog, which I should have said rather than implying full coverage — surfaced a real race. MoveToCleanupGrace assigned entry.cleanupTimer outside the registry mutex, but a zero grace fires onExpiry before that assignment completes, and the callback reads the field under the mutex. The timer is now armed and stopped only under that mutex.

Verification
Both new test groups were confirmed to fail before their fix and pass after. gofmt -s, go vet ./..., golangci-lint run (0 issues), go test ./..., go build ./..., helm lint, and helm template all pass; internal/debuglog, internal/worker, internal/common, and internal/config are additionally green under -race -count=2.

On the reviewer's golangci-lint note: I hit the same Go 1.25-vs-1.26.5 wall with the preinstalled binary and resolved it by installing github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest, which targets the repo's Go version. Lint was run locally, not skipped.

Directed follow-up changes

The terminal re-review verified all three rework fixes and found one more real issue, plus stale docs. Both addressed in 8e32ae1.

1. Shutdown cleanup could lose resources it failed to delete — internal/worker/{docker,kubernetes,worker}.go
The previous fix made shutdown attempt cleanup; the attempt was not durable. Both backends removed the registry entry before deletion was confirmed, and Docker additionally returned nil after a failed remove, so a transient API error left nothing for the backend's own shutdown to retry — a stopped container or successful Job then outlived its grace across a worker restart.

CleanupTaskResources now looks the resource up, deletes it, and drops the identifier only once deletion is confirmed; an already-absent container or Job counts as deleted. removeContainer surfaces every failure that is not a not-found, and Docker's Shutdown warns by task name for anything it still could not remove. An unreleasable entry stays registered, which is what gives Shutdown something to retry.

The sweep also shared one 10-second budget across every pending entry, so one slow call could consume it and starve the rest — worst on a busy worker, which is exactly when there is the most to release. Each entry now runs concurrently under its own BackendShutdownTimeout with a bounded 3-attempt retry, keeping the whole sweep within one timeout regardless of entry count. The local capture is still released unconditionally: a replacement worker's startup sweep removes any file left behind, so unlike a provider resource it cannot accumulate.

New internal/worker/cleanup_durability_test.go covers a failed Kubernetes delete retaining the Job and a retry both deleting it and releasing the entry, an absent Job counting as deleted, a failed Job released without deletion, a failed Docker removal against an unreachable daemon surfacing the error and keeping the container registered, and the sweep retrying a transient failure, giving up after its budget, and attempting all 8 entries when each call stalls 300 ms. The two retention tests were confirmed to fail under the previous forget-before-confirm ordering.

2. Stale docs — charts/oz-agent-worker/values.yaml and the taskJobTTLSecondsAfterFinished comment
Both still said successful Jobs are deleted immediately. They now describe cleanup-grace retention and state plainly that a ttl_seconds_after_finished shorter than the effective grace makes a debug archive partial, matching the README.

Verification. gofmt -s, go vet ./..., golangci-lint run (0 issues), go test ./..., go build ./..., helm lint, helm template all pass, plus -race -count=2 across every internal/... package.

Spec deviation

The backend contract is SnapshotTaskLogs(ctx, *SnapshotParams) carrying a debuglog.Sink rather than the spec's (ctx, taskID, executionID string, writer io.Writer). A bare io.Writer cannot work: the content transformer must be applied to each record's decoded data while leaving timestamps, sequence, and identity structural, so the transformer has to live inside the encoder rather than in each backend. The Sink keeps framing, chunk bounds, transformation, encoding selection, sequencing, and truncation in one place while backends supply only provider output and the identity the provider actually reports. Every behavioral requirement of the spec's contract — streaming, cancellation, bounded output, no lifecycle change, no resource removal, safe during ExecuteTask, deterministic source ordering, typed partial errors — is preserved.

Originating thread: https://warpdev.slack.com/archives/C0BDQDW8V5E/p1785881958158819

Co-Authored-By: Warp Agent agent@warp.dev

Conversation: https://staging.warp.dev/conversation/12a2ee47-a3be-4337-a627-e55424ca1a82
Run: https://oz.staging.warp.dev/runs/019fd69b-ebf9-71c8-a33f-28546567f042
This PR was generated with Oz.

[warp-agent-staging]

Overview

This re-review confirms the hard snapshot bound, terminal cleanup-grace shutdown, and zero-grace timer race fixes at 0aea651. A shutdown failure path can still leave retained provider resources past the configured grace, so the worker PR is not ready to accept.

Concerns

The operator-facing ttlSecondsAfterFinished comment in the Helm values and the matching Kubernetes helper comment still say successful Jobs are deleted immediately, despite the new grace-period retention. This implementation-addressable documentation correction is included in the foreman relay.

Verdict

Checks: CI ✅ · build ✅ · tests ✅ · race ✅ · visual proof n/a (headless backend change); local golangci-lint/Helm unavailable because the installed linter targets Go 1.25 and Helm is absent, while their current CI jobs pass.
Found: 0 critical, 1 important, 0 suggestions, 1 question
Prior concerns: the prior snapshot-bound and restart-cleanup findings are addressed; the shutdown failure path above remains.

Request changes

Review run

https://oz.staging.warp.dev/runs/019fd6cb-ac2b-784b-ad28-72b55a74fd2f