Watchdog kill->respawn loop under memory exhaustion exhausts system commit and crashes unrelated apps (Windows, v1.2.0) #3137

Summary

On Windows, the codegraph serve --mcp long-running worker (the --liftoff-only child) repeatedly ballooned to 38–41 GB of commit charge, wedged, was killed by the watchdog added for #850, and was respawned — then ballooned again. The result was a kill→respawn→re-blowup loop (9 different node.exe PIDs in ~22 minutes) that exhausted the system commit limit (49.7 GB) and crashed unrelated applications. The watchdog fixed the "wedged forever" CPU failure mode of #850, but it exposed a worse one under memory pressure: the respawn loop itself takes the whole machine down, because every fresh worker re-attempts the same work that blew it up in the first place.

This happened on v1.2.0; we have not yet reproduced on current versions. Filed because the loop dynamic (no respawn backoff, no memory guard, no client-visible error) is version-independent as long as the watchdog design stays as-is, and none of the open memory issues (#1553/#1583, #3027) cover it.

Environment

What happened (timeline)

1. Five client sessions were started within ~25 minutes while the target repo was being actively edited; each session start spawned a fresh codegraph worker.

2. Windows Resource-Exhaustion-Detector (System log, event 2004) fired 11 times between 16:46:07 and 17:05:42. Every event names node.exe as the top committer, each time a different PID except the last, at 38–41 GB commit each:

   Time	PID	Commit (bytes)
   16:46:07	38796	38,849,732,608
   16:47:50	22568	40,931,942,400
   16:50:03	13300	40,550,477,824
   16:52:47	33408	40,462,024,704
   16:56:39	31140	39,835,422,720
   16:58:35	20260	40,462,966,784
   17:01:45 / 17:02:29 / 17:03:13 / 17:05:42	37132	~38–41 GB (same PID across 4 minutes, 4 events)

3. The worker's watchdog child (node -e, the codegraph cause 100% CPU #850 mitigation) SIGKILLs the main process once the stdin heartbeat stops for 60 s, "so a fresh one can start". The PID churn above is consistent with that kill→respawn cycle repeating. (Process-creation auditing was not enabled, so PID→binary mapping is inferred, not logged — codegraph was, however, the only node.exe that matches all three constraints: freshly spawned inside the window, equipped with the kill-and-respawn watchdog, and holding a large in-memory index.)

4. Commit exhaustion took unrelated applications down with it: the MCP client itself crashed twice (16:47:51, 16:50:04), Microsoft Edge crashed twice, dwm.exe crashed once (desktop-wide UI stalls), PowerPoint hung, and even the Windows Error Reporting Service terminated with an error. To the user this looked like "the whole PC is extremely slow with fans pegged".

5. The storm stopped only when the client-side retry loop stopped respawning sessions. Surviving codegraph instances then idled at 33–86 MB, so the blow-up appears to happen during active (re)indexing, not at rest.

Notably this incident did not look like the #850 CPU-peg (system CPU stayed modest throughout); it manifested as unbounded commit growth instead.

Why the watchdog alone can't cover this failure mode

The watchdog turns "wedged forever" into "wedged, killed, respawned" — strictly better for a CPU wedge. But for a memory-blowup wedge it produces a loop: every fresh worker starts (re)indexing from scratch, balloons again within ~1–4 minutes, gets killed again. When the MCP client auto-respawns the server (or the user retries), this repeats indefinitely, and every cycle degrades the whole OS — commit pressure makes unrelated processes fail allocations and crash. A single misbehaving index target ends up crashing the user's browser, shell, and window manager.

Suggestions

1. Memory guard in the worker: cap the worker's heap (e.g. --max-old-space-size for the --liftoff-only child) and monitor external/buffer memory; abort indexing with a clear error to the client instead of growing until the OS commit limit is hit.
2. Respawn backoff / circuit breaker: after N watchdog kills within M minutes, stop respawning and surface an error to the client, instead of looping. (This is the piece that would have capped yesterday's incident at one crash instead of nine.)
3. Incremental index persistence: if a fresh worker rebuilds from scratch, persist enough state that a respawn doesn't redo the full pass that triggered the blow-up.
4. Surface watchdog kills: emit the kill reason (and the last exception, if any) to stderr so MCP clients can log it. We had no local trace of the root cause — only the Resource-Exhaustion events.
5. Possibly: documented ignore/scoping globs for the indexer, so users can exclude pathological inputs.

Happy to enable process auditing and reproduce with a profiler attached (or retest on 1.6.2 first and report back) — just tell me what data would help.

Related: #850 (the watchdog origin), #2111 (OOM during index rebuild, closed), #900 (large-repo indexing OOMs, closed), #1553/#1583 (in-flight memory bounding), #3027 (worker pool growth / every worker opens every project).