Skip to content

fix(kernel-cli): log fatal exits from daemon-entry before terminating#966

Open
FUDCo wants to merge 2 commits into
mainfrom
chip/daemon-entry-fatal-logging
Open

fix(kernel-cli): log fatal exits from daemon-entry before terminating#966
FUDCo wants to merge 2 commits into
mainfrom
chip/daemon-entry-fatal-logging

Conversation

@FUDCo

@FUDCo FUDCo commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add process-level handlers in daemon-entry.ts for uncaughtException, unhandledRejection, SIGHUP, and exit.
  • Each handler synchronously appends a line to daemon.log before the process exits.
  • Handlers install at module load, before main() runs, so early kernel-init failures also leave a fingerprint.

Why

daemon-entry runs with stdio: 'ignore' under the CLI spawner (see daemon-spawn.ts). Node's default behaviour on uncaughtException / unhandledRejection (print stack to stderr, exit 1) therefore writes to nowhere, and the operator sees only that the daemon vanished — no trace in ~/.ocap*/daemon.log.

I've hit two silent daemon deaths in the last few weeks (matcher daemon disappearing mid-rehearsal, services daemon disappearing between registration and the next call), and in both cases the log ended cleanly on the last successful message with no shutdown line, no error, no stack trace. Debugging cost real time each time.

With this change, every terminating path now leaves at least one line:

[timestamp] [error] Uncaught exception (about to exit): <stack>
[timestamp] [error] Process exiting (code=1).

or

[timestamp] [error] SIGHUP received; exiting.
[timestamp] [error] Process exiting (code=0).

Log-write itself is wrapped in try/catch so a broken log file doesn't mask the original exit cause.

Test plan

  • yarn workspace @metamask/kernel-cli test:dev:quiet --run — all package tests pass.
  • yarn workspace @metamask/kernel-cli lint — clean.
  • CI green.
  • Manual (in a downstream branch): induce an uncaught exception in a scratch daemon under a temp $OCAP_HOME, confirm the log line lands and the process exits 1.

🤖 Generated with Claude Code


Note

Low Risk
Observability-only change to daemon exit handling; no auth, data, or RPC behavior changes beyond explicit SIGHUP logging and exit codes.

Overview
Daemon processes spawned with stdio: 'ignore' no longer die without a trace in daemon.log.

daemon-entry now registers fatal handlers at module load (before main()), so early init failures are covered too. On uncaughtException, unhandledRejection, or SIGHUP, it synchronously appends an [error] line to daemon.log (with stack/detail when available), then exits; an exit handler adds a final Process exiting (code=…) fingerprint. Writes are best-effort (try/catch) so a broken log file cannot mask the real failure.

The file logger transport is switched to a direct appendFileSync import instead of a sync require of node:fs. The changelog documents the fix under Fixed.

Reviewed by Cursor Bugbot for commit 9b9d82b. Bugbot is set up for automated code reviews on this repo. Configure here.

daemon-entry runs with `stdio: 'ignore'` under the CLI spawner, so
Node's default behaviour on uncaughtException / unhandledRejection
(print stack to stderr, exit 1) writes to nowhere and the operator
sees only that the daemon vanished. Two recent debugging sessions
were consumed by silent daemon deaths that left no trace.

Install process-level handlers that append a synchronous log line
before the process exits:

  - uncaughtException — captures stack, exits(1)
  - unhandledRejection — captures reason, exits(1)
  - SIGHUP — logs, exits(0) (default was silent terminate)
  - exit — last-ditch record; fires on every exit path

Handlers are installed at module load, before main() runs, so
early kernel-init failures also leave a fingerprint. Each handler
uses only synchronous fs and the fs write is wrapped in try/catch
so a log-write failure never masks the original exit cause.
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 71.27%
⬇️ -0.10%
8862 / 12433
🔵 Statements 71.1%
⬇️ -0.10%
9012 / 12674
🔵 Functions 72.42%
⬇️ -0.15%
2138 / 2952
🔵 Branches 64.76%
⬇️ -0.12%
3579 / 5526
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
packages/kernel-cli/src/commands/daemon-entry.ts 0%
🟰 ±0%
0%
🟰 ±0%
0%
🟰 ±0%
0%
🟰 ±0%
23-228
Generated in workflow #4494 for commit 9b9d82b by the Vitest Coverage Report Action

@FUDCo FUDCo marked this pull request as ready for review July 9, 2026 00:43
@FUDCo FUDCo requested a review from a team as a code owner July 9, 2026 00:43

@grypez grypez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would suffice to create the logger at file root scope instead of in the main header. File-scoped logger for entrypoints is already standard convention across the monorepo; seems we just missed this one because it is a command.

Comment on lines +155 to +181
/**
* Append a fatal-path entry to `daemon.log` synchronously. Used from
* `process.on('uncaughtException' | 'unhandledRejection' | 'SIGHUP')`
* handlers where the async logger pipeline can't be trusted to
* flush before the process exits. Best-effort: if the log file is
* unwritable we swallow the error rather than throw from a fatal
* handler.
*
* @param logPath - The daemon-log file path.
* @param message - Short label for the entry.
* @param detail - Optional extra data (stack, error, etc.) — coerced
* to string.
*/
function logFatalSync(
logPath: string,
message: string,
detail?: string | number,
): void {
try {
const tail = detail === undefined ? '' : ` ${detail}`;
const line = `[${new Date().toISOString()}] [error] ${message}${tail}\n`;
// eslint-disable-next-line n/no-sync -- fatal handler must flush before exit
appendFileSync(logPath, line);
} catch {
// Best-effort — the daemon is dying either way.
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/**
* Append a fatal-path entry to `daemon.log` synchronously. Used from
* `process.on('uncaughtException' | 'unhandledRejection' | 'SIGHUP')`
* handlers where the async logger pipeline can't be trusted to
* flush before the process exits. Best-effort: if the log file is
* unwritable we swallow the error rather than throw from a fatal
* handler.
*
* @param logPath - The daemon-log file path.
* @param message - Short label for the entry.
* @param detail - Optional extra data (stack, error, etc.) coerced
* to string.
*/
function logFatalSync(
logPath: string,
message: string,
detail?: string | number,
): void {
try {
const tail = detail === undefined ? '' : ` ${detail}`;
const line = `[${new Date().toISOString()}] [error] ${message}${tail}\n`;
// eslint-disable-next-line n/no-sync -- fatal handler must flush before exit
appendFileSync(logPath, line);
} catch {
// Best-effort — the daemon is dying either way.
}
}

The logger dispatch routine is already synchronous; async methods follow 'spray and pray' semantics, i.e. best effort.

I think the process.on handlers are doing the work here, not this function.

process.on('uncaughtException', (error: unknown) => {
const detail =
error instanceof Error ? (error.stack ?? error.message) : String(error);
logFatalSync(logPath, 'Uncaught exception (about to exit):', detail);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logFatalSync(logPath, 'Uncaught exception (about to exit):', detail);
logger.error('Uncaught exception', detail);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants