Skip to content

fix(daemon): contain zombie generations from abandoned requests, name mute endpoint holders - #1920

Open
OhOkThisIsFine wants to merge 1 commit into
DeusData:mainfrom
OhOkThisIsFine:claude/focused-herschel-ee8e1c
Open

fix(daemon): contain zombie generations from abandoned requests, name mute endpoint holders#1920
OhOkThisIsFine wants to merge 1 commit into
DeusData:mainfrom
OhOkThisIsFine:claude/focused-herschel-ee8e1c

Conversation

@OhOkThisIsFine

Copy link
Copy Markdown

Summary

Fixes a daemon zombie failure observed in production on 2026-08-29 (v0.10.8, win32): after the last busy client disconnected, the daemon process stayed alive for 9+ hours with a dead runtime — holding both \\.\pipe\cbm-daemon-* named pipes and the UI port (HTTP still answered) while daemon status said "not running", daemon start timed out after 30 s with no diagnostic, and every MCP client got CONNECTION_CLOSED.

Root cause

The zombie generation's log shows no daemon.runtime_stopping record and a healthy log sink throughout, so the service never left RUNNING. The wedge: a client disconnected while its application request was in flight and the request never observed cancellation. runtime_worker_finishruntime_worker_reap_application(wait=true) joined that request thread via cbm_thread_join with no deadline (runtime.c). The disconnect path blocked forever, the worker slot never released, later sessions wedged behind the poisoned application state, and no host timeout could fire because the service still reported RUNNING.

Changes

  1. Containment (runtime.c) — the disconnect-path reap now waits at most RUNTIME_ABANDONED_REQUEST_JOIN_TIMEOUT_MS (30 s; session_cancel has already run, so a compliant handler returns in milliseconds), then logs daemon.application_request_unresponsive (peer pid, request token) and fail-stops the process (daemon.forced_shutdown component=application_request_join). The kernel releases pipes/claims and the next client starts a fresh generation. Seam-gated test: daemon_runtime_abandoned_request_join_reaches_containment_in_bounded_time.
  2. Mute-holder diagnostics (runtime.c/h, bootstrap.c, main.c) — when a transport connect reaches a live process but no valid answer arrives, the holder's kernel pid is resolved via the existing cbm_daemon_ipc_connection_peer_pid (GetNamedPipeServerProcessId / SO_PEERCRED; previously unused client-side). cbm_daemon_bootstrap_classify_failed_connect treats a mute holder as RESERVED — never absence — so the starter stops spawning doomed competitors, and the daemon start timeout message names the pid with recovery guidance. daemon status prints daemon: not responding (endpoint held by pid N) instead of "not running" (also for wrong-op reject frames from a capacity-wedged generation). Tests: daemon_bootstrap_mute_endpoint_holder_is_reserved_and_never_unavailable, daemon_runtime_mute_endpoint_holder_pid_is_reported.
  3. Frontend maintenance-monitor cost (frontend.c) — the quiet-state presence poll drops 10 ms → 250 ms (FRONTEND_MAINTENANCE_IDLE_POLL_MS). Each probe try-acquires the maintenance marker lock, which on Windows revalidates the entire ancestor directory chain; at 100 Hz this burned ~20% of a core for the life of every MCP session (7762 CPU-seconds on one 9.5 h session during the incident). The 250 ms detection latency spends under 2% of the 15 s activation drain budget; the post-detection grace loop keeps its 10 ms pacing.
  4. Log rewording (version_cohort.c)version_cohort.claimed_unheldversion_cohort.claimed_fresh prior_holder=none. The old key reads as a stale-claim anomaly and derailed the incident diagnosis; the record marks the healthy fresh-claim path every normal start logs. No test or script references the old key.

Testing

  • New regression tests listed above; full daemon_bootstrap (19/19) and daemon_runtime suites pass locally (WSL Ubuntu, ASan+UBSan).
  • One pre-existing test fails in the local venue only: daemon_runtime_process_fingerprint_never_hashes_replacement_path copies /bin/cat to a file named image, which cannot run on uutils/Rust-coreutils systems (Ubuntu 25.10+ multi-call binary; fails identically without this PR). CI runners with GNU coreutils are unaffected.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@OhOkThisIsFine
OhOkThisIsFine force-pushed the claude/focused-herschel-ee8e1c branch from f10d216 to fc1b1ee Compare August 29, 2026 21:44
… mute endpoint holders

Fixes the 2026-08-29 daemon zombie class: a client disconnected while its
application request was in flight, the request never observed cancellation,
and runtime_worker_finish joined the request thread with no deadline. The
disconnect path wedged forever, the generation stayed formally RUNNING, and
a dead process held the endpoint pipes and UI port for nine hours while
every new client timed out with no diagnostic.

Four changes:

- runtime: the disconnect-path reap of an in-flight application request now
  waits at most RUNTIME_ABANDONED_REQUEST_JOIN_TIMEOUT_MS (30 s), then logs
  daemon.application_request_unresponsive (peer pid, request token) and
  fail-stops the process. The kernel releases every native claim and the
  next client starts a fresh generation. Test seams expose the ceiling and
  replace the terminal stop with a recordable hook.
- diagnostics: a transport connect that reaches a live process but gets no
  valid answer now resolves the holder's kernel pid
  (GetNamedPipeServerProcessId / SO_PEERCRED). `daemon start` classifies a
  mute holder as RESERVED (no doomed competitor spawns) and names the pid
  in its timeout message; `daemon status` prints "not responding (endpoint
  held by pid N)" with recovery guidance instead of "not running".
- frontend: the maintenance monitor's quiet-state poll drops from 10 ms to
  250 ms. Each probe revalidates the whole ancestor directory chain on
  Windows; at 100 Hz that burned ~20% of a core for the life of every MCP
  session (7762 CPU-seconds on one 9.5 h session). Detection latency spends
  0.25 s of the 15 s activation drain budget.
- version_cohort: the healthy fresh-claim record is renamed from
  "claimed_unheld" (misread as a stale-claim anomaly during the incident
  diagnosis) to "claimed_fresh" with prior_holder=none.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: OhOkThisIsFine <102485413+OhOkThisIsFine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant