fix(daemon): contain zombie generations from abandoned requests, name mute endpoint holders - #1920
Open
OhOkThisIsFine wants to merge 1 commit into
Open
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
OhOkThisIsFine
force-pushed
the
claude/focused-herschel-ee8e1c
branch
from
August 29, 2026 21:44
f10d216 to
fc1b1ee
Compare
… mute endpoint holders Fixes the 2026-08-29 daemon zombie class: a client disconnected while its application request was in flight, the request never observed cancellation, and runtime_worker_finish joined the request thread with no deadline. The disconnect path wedged forever, the generation stayed formally RUNNING, and a dead process held the endpoint pipes and UI port for nine hours while every new client timed out with no diagnostic. Four changes: - runtime: the disconnect-path reap of an in-flight application request now waits at most RUNTIME_ABANDONED_REQUEST_JOIN_TIMEOUT_MS (30 s), then logs daemon.application_request_unresponsive (peer pid, request token) and fail-stops the process. The kernel releases every native claim and the next client starts a fresh generation. Test seams expose the ceiling and replace the terminal stop with a recordable hook. - diagnostics: a transport connect that reaches a live process but gets no valid answer now resolves the holder's kernel pid (GetNamedPipeServerProcessId / SO_PEERCRED). `daemon start` classifies a mute holder as RESERVED (no doomed competitor spawns) and names the pid in its timeout message; `daemon status` prints "not responding (endpoint held by pid N)" with recovery guidance instead of "not running". - frontend: the maintenance monitor's quiet-state poll drops from 10 ms to 250 ms. Each probe revalidates the whole ancestor directory chain on Windows; at 100 Hz that burned ~20% of a core for the life of every MCP session (7762 CPU-seconds on one 9.5 h session). Detection latency spends 0.25 s of the 15 s activation drain budget. - version_cohort: the healthy fresh-claim record is renamed from "claimed_unheld" (misread as a stale-claim anomaly during the incident diagnosis) to "claimed_fresh" with prior_holder=none. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: OhOkThisIsFine <102485413+OhOkThisIsFine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a daemon zombie failure observed in production on 2026-08-29 (v0.10.8, win32): after the last busy client disconnected, the daemon process stayed alive for 9+ hours with a dead runtime — holding both
\\.\pipe\cbm-daemon-*named pipes and the UI port (HTTP still answered) whiledaemon statussaid "not running",daemon starttimed out after 30 s with no diagnostic, and every MCP client got CONNECTION_CLOSED.Root cause
The zombie generation's log shows no
daemon.runtime_stoppingrecord and a healthy log sink throughout, so the service never left RUNNING. The wedge: a client disconnected while its application request was in flight and the request never observed cancellation.runtime_worker_finish→runtime_worker_reap_application(wait=true)joined that request thread viacbm_thread_joinwith no deadline (runtime.c). The disconnect path blocked forever, the worker slot never released, later sessions wedged behind the poisoned application state, and no host timeout could fire because the service still reported RUNNING.Changes
RUNTIME_ABANDONED_REQUEST_JOIN_TIMEOUT_MS(30 s;session_cancelhas already run, so a compliant handler returns in milliseconds), then logsdaemon.application_request_unresponsive(peer pid, request token) and fail-stops the process (daemon.forced_shutdown component=application_request_join). The kernel releases pipes/claims and the next client starts a fresh generation. Seam-gated test:daemon_runtime_abandoned_request_join_reaches_containment_in_bounded_time.cbm_daemon_ipc_connection_peer_pid(GetNamedPipeServerProcessId/SO_PEERCRED; previously unused client-side).cbm_daemon_bootstrap_classify_failed_connecttreats a mute holder as RESERVED — never absence — so the starter stops spawning doomed competitors, and thedaemon starttimeout message names the pid with recovery guidance.daemon statusprintsdaemon: not responding (endpoint held by pid N)instead of "not running" (also for wrong-op reject frames from a capacity-wedged generation). Tests:daemon_bootstrap_mute_endpoint_holder_is_reserved_and_never_unavailable,daemon_runtime_mute_endpoint_holder_pid_is_reported.FRONTEND_MAINTENANCE_IDLE_POLL_MS). Each probe try-acquires the maintenance marker lock, which on Windows revalidates the entire ancestor directory chain; at 100 Hz this burned ~20% of a core for the life of every MCP session (7762 CPU-seconds on one 9.5 h session during the incident). The 250 ms detection latency spends under 2% of the 15 s activation drain budget; the post-detection grace loop keeps its 10 ms pacing.version_cohort.claimed_unheld→version_cohort.claimed_fresh prior_holder=none. The old key reads as a stale-claim anomaly and derailed the incident diagnosis; the record marks the healthy fresh-claim path every normal start logs. No test or script references the old key.Testing
daemon_bootstrap(19/19) anddaemon_runtimesuites pass locally (WSL Ubuntu, ASan+UBSan).daemon_runtime_process_fingerprint_never_hashes_replacement_pathcopies/bin/catto a file namedimage, which cannot run on uutils/Rust-coreutils systems (Ubuntu 25.10+ multi-call binary; fails identically without this PR). CI runners with GNU coreutils are unaffected.🤖 Generated with Claude Code