diff --git a/.github/workflows/ios.yml b/.github/workflows/ios.yml index 2841edd259..c138a600d9 100644 --- a/.github/workflows/ios.yml +++ b/.github/workflows/ios.yml @@ -169,6 +169,14 @@ jobs: - name: Preflight iOS runner through public CLI run: | pnpm clean:daemon + # The public-SDK record is the cold-startup evidence; the CLI timer only wraps attach. + node --input-type=module -e ' + const { performance } = await import("node:perf_hooks"); + const { ensureDaemon } = await import("./dist/src/index.js"); + const t0 = performance.now(); + const c = await ensureDaemon({ stateDir: process.env.AGENT_DEVICE_STATE_DIR }); + console.log(JSON.stringify({ phase: "daemon_cold_startup", started: c.started, pid: c.daemon.pid, durationMs: Math.round(performance.now() - t0) })); + ' node bin/agent-device.mjs prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json # The replay reuses the prepared runner in the same daemon. --debug keeps every step's request diff --git a/CHANGELOG.md b/CHANGELOG.md index c0b3ecaa40..adb4e8a75f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - The PNG worker no longer calls `process.emitWarning` when it is unavailable, and it starts with an empty environment (`env: {}`). The daemon registers a sink when it pre-warms the worker, so every transition to unavailable (spawn failure, missing entry, load error, exit) leaves exactly one `png_worker_unavailable` event, with a typed `reason` (`entry-not-found`, `spawn-failed`, `worker-error`, `worker-exited`) and its `detail`, in `daemon.log`, with or without `--debug`. Requests then continue on the in-process fallback without repeating the event. - Behavior change: `resolveHubAppReference`, shipped through `agent-device/plugins/webdriver`, now needs `cwd` to resolve a relative app path. Without it the call fails with `INVALID_ARGS` and `details.reason: provider_app_cwd_missing`; it used to resolve against the process's working directory. Absolute paths, URLs and provider references are unaffected. The daemon fills it from the client's working directory (or its host's) for every provider lease request, so `connect` and `lease_allocate` are unchanged. +- A client starting a local daemon now waits while the daemon process stays alive and keeps publishing startup progress, so an overloaded host makes startup slower instead of failing it. The wait stops at a ceiling of 120 s by default; set `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` to a positive integer number of milliseconds to override it for the CLI, MCP, and Node clients alike. The variable is read only when the client must start a daemon; a value that is not a positive integer fails with `INVALID_ARGS` before the daemon spawns, and a client that only attaches to or reuses a running daemon never reads it. A failed startup names the outcome in `error.details.reason`, and you should branch on it, never on the message: `startup-exited` (the process ended before registering), `startup-stalled` (alive but no startup progress for a fixed 60 s window, which the ceiling variable does not change; the daemon is stopped), or `startup-ceiling` (a startup was still live — this client's daemon or another client's — when the ceiling passed; the client stops its own launch and never signals another client's live daemon). `details.phases` carries the startup phase timings the client observed and `details.startupTimeoutMs` the effective ceiling. - The `daemon-not-running` hint now always names `--state-dir `, and a daemon force-retired for outlasting its drain is reported as `Replaced unreachable daemon` instead of `Replaced idle daemon`. `daemon-drain-timeout` and the post-drain force retire are documented in the client API and `help`. - Fix `ensureDaemon()`, the CLI and MCP killing a daemon they had just asked to leave. After the daemon answers the quiesce request with `draining`, the client now waits up to 60 s for it to exit by itself and clean up its registration, so plugin disposal and lease finalization finish (on Windows a signal ends the process outright, and on POSIX a drain longer than 3 s was killed). A daemon still running at the deadline that fails its liveness probes is force-retired; one that still answers is left running and the call fails with `details.reason: 'daemon-drain-timeout'`, with a hint naming `agent-device daemon stop --state-dir `. - `@agent-device/testmu` is now a version 2 provider plugin. `import testmu from '@agent-device/testmu'` gives a factory, `testmu({ credentials: { username, accessKey } })`, that reads those credentials instead of `LT_USERNAME` and `LT_ACCESS_KEY`; without options it reads the variables as before. The package still declares the version 1 plugin entry, so `agent-device plugins add @agent-device/testmu` keeps working on hosts that predate the version 2 contract. diff --git a/packages/daemon-contracts/package.json b/packages/daemon-contracts/package.json index bc4d0cf0af..56faa64f4f 100644 --- a/packages/daemon-contracts/package.json +++ b/packages/daemon-contracts/package.json @@ -44,6 +44,10 @@ "types": "./src/daemon-shutdown-report.ts", "default": "./src/daemon-shutdown-report.ts" }, + "./daemon-startup-phases": { + "types": "./src/daemon-startup-phases.ts", + "default": "./src/daemon-startup-phases.ts" + }, "./daemon-stop": { "types": "./src/daemon-stop.ts", "default": "./src/daemon-stop.ts" diff --git a/packages/daemon-contracts/src/daemon-registration-owner.ts b/packages/daemon-contracts/src/daemon-registration-owner.ts index 5e7090da51..2e452699ad 100644 --- a/packages/daemon-contracts/src/daemon-registration-owner.ts +++ b/packages/daemon-contracts/src/daemon-registration-owner.ts @@ -44,6 +44,7 @@ import { resolveDaemonShutdownReportPath, type DaemonShutdownOutcome, } from './daemon-shutdown-report.ts'; +import { sweepDeadDaemonStartupPhasesFiles } from './daemon-startup-phases.ts'; export const DAEMON_STARTUP_EXIT_CODES = Object.freeze({ busy: 75, unproven: 78 }); export type DaemonRegistrationFields = Readonly<{ @@ -85,6 +86,7 @@ export async function tryAcquireDaemonRegistration( try { acquisition.assertHeld(); fs.rmSync(resolveDaemonShutdownReportPath(boundPaths.baseDir), { force: true }); + sweepDeadDaemonStartupPhasesFiles(boundPaths.baseDir); } catch (error) { await releaseRegistrationAfterFailure(acquisition, error, boundPaths.logPath); } diff --git a/packages/daemon-contracts/src/daemon-startup-phases.test.ts b/packages/daemon-contracts/src/daemon-startup-phases.test.ts new file mode 100644 index 0000000000..513567f921 --- /dev/null +++ b/packages/daemon-contracts/src/daemon-startup-phases.test.ts @@ -0,0 +1,144 @@ +import { appendFile, readFile, writeFile } from 'node:fs/promises'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { expect, test } from 'vitest'; +import { tryAcquireDaemonRegistration } from '@agent-device/daemon-contracts/daemon-registration-owner'; +import { + readDaemonStartupPhaseMarkers, + recordDaemonStartupMarker, + removeDaemonStartupPhasesFile, + resolveDaemonStartupPhasesPath, +} from '@agent-device/daemon-contracts/daemon-startup-phases'; +import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution'; +import { mkdtempForTest } from '@agent-device/test-kit'; + +test('per-launch marker files: a fresh launch never reads an older launch phases', async () => { + const stateDir = await mkdtempForTest('agent-device-markers-'); + const first = resolveDaemonStartupPhasesPath(stateDir, 1111); + const second = resolveDaemonStartupPhasesPath(stateDir, 2222); + recordDaemonStartupMarker(first, 'lock-acquired'); + expect(second).not.toBe(first); + expect(existsSync(second)).toBe(false); + recordDaemonStartupMarker(second, 'modules-loaded'); + const firstPhases = readDaemonStartupPhaseMarkers(readFileSync(first, 'utf8')); + const secondPhases = readDaemonStartupPhaseMarkers(readFileSync(second, 'utf8')); + expect(Object.keys(firstPhases)).toEqual(['lock-acquired']); + expect(Object.keys(secondPhases)).toEqual(['modules-loaded']); +}); + +test('marker reading keeps the earliest record when a phase replays after a later one', () => { + const text = [ + '{"data":{"startupPhase":"listening","timestampMs":500}}', + '{"data":{"startupPhase":"lock-acquired","timestampMs":20}}', + '{"data":{"startupPhase":"lock-acquired","timestampMs":35}}', + ].join('\n'); + const phases = readDaemonStartupPhaseMarkers(text); + expect(phases['lock-acquired']).toBe(20); + expect(phases.listening).toBe(500); +}); + +test('marker reading ignores unknown phases and unparsable lines', () => { + const text = [ + '{"data":{"startupPhase":"heartbeat","timestampMs":10}}', + '{"data":{"startupPhase":"registered","timestampMs":"7"}}', + 'not json', + '{"data":{"startupPhase":"listening","timestampMs":30}}', + ].join('\n'); + expect(readDaemonStartupPhaseMarkers(text)).toEqual({ listening: 30 }); +}); + +test('the producer lifetime claim empties a same-pid leftover', async () => { + // Pid reuse after both the old client and daemon died before either cleanup: the entry removes + // its record before its first marker; this child repeats that producer sequence itself. + const stateDir = await mkdtempForTest('agent-device-markers-'); + writeFileSync(path.join(stateDir, 'package.json'), '{"type":"module"}'); + const entry = path.join(stateDir, 'claim-producer.js'); + const registrationUrl = new URL('./daemon-startup-phases.ts', import.meta.url).href; + writeFileSync( + entry, + `import { removeDaemonStartupPhasesFile, recordDaemonStartupMarker, resolveDaemonStartupPhasesPath } from ${JSON.stringify(registrationUrl)}; +const file = resolveDaemonStartupPhasesPath(${JSON.stringify(stateDir)}, process.pid); +recordDaemonStartupMarker(file, 'listening'); +recordDaemonStartupMarker(file, 'registered'); +removeDaemonStartupPhasesFile(file); +recordDaemonStartupMarker(file, 'modules-loaded'); +`, + ); + const child = spawnSync(process.execPath, ['--experimental-strip-types', entry], { + encoding: 'utf8', + }); + if (child.status !== 0) throw new Error(child.stderr); + const phases = readDaemonStartupPhaseMarkers( + readFileSync(resolveDaemonStartupPhasesPath(stateDir, child.pid!), 'utf8'), + ); + expect(phases).toEqual({ 'modules-loaded': expect.any(Number) }); +}); + +test.each([ + ['remove', (p: string, cb: (e: unknown) => void) => removeDaemonStartupPhasesFile(p, cb)], + [ + 'record', + (p: string, cb: (e: unknown) => void) => recordDaemonStartupMarker(p, 'modules-loaded', cb), + ], +] as const)('a refused %s reports through the callback instead of throwing', async (_name, run) => { + const stateDir = await mkdtempForTest('agent-device-markers-'); + const markerPath = resolveDaemonStartupPhasesPath(stateDir, 777); + mkdirSync(markerPath, { recursive: true }); + const reported: unknown[] = []; + expect(() => + run(markerPath, (error) => { + reported.push(error); + }), + ).not.toThrow(); + expect(reported).toHaveLength(1); + expect(reported[0]).toHaveProperty('message'); +}); + +test('acquiring the registration retires files of dead or unowned pids only', async () => { + const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-')); + const dead = resolveDaemonStartupPhasesPath(paths.baseDir, spawnDeadPid()); + const own = resolveDaemonStartupPhasesPath(paths.baseDir, process.pid); + const unparseable = path.join(paths.baseDir, 'daemon-startup-phases-.jsonl'); + const legacy = path.join(paths.baseDir, 'daemon-startup-phases.jsonl'); + await Promise.all([ + writeFile(own, 'x\n', 'utf8'), + writeFile(dead, 'x\n', 'utf8'), + writeFile(unparseable, 'x\n', 'utf8'), + writeFile(legacy, 'x\n', 'utf8'), + ]); + const attempt = await tryAcquireDaemonRegistration(paths); + expect(attempt.status).toBe('acquired'); + if (attempt.status !== 'acquired') throw new Error('registration refused'); + expect(existsSync(dead)).toBe(false); + expect(existsSync(unparseable)).toBe(false); + expect(existsSync(own)).toBe(true); + expect(existsSync(legacy)).toBe(true); + await attempt.owner.finish(); +}); + +test('removeDaemonStartupPhasesFile removes a file and tolerates one already gone', async () => { + const stateDir = await mkdtempForTest('agent-device-markers-'); + const file = resolveDaemonStartupPhasesPath(stateDir, 4242); + await writeFile(file, 'x\n', 'utf8'); + removeDaemonStartupPhasesFile(file); + expect(existsSync(file)).toBe(false); + expect(() => removeDaemonStartupPhasesFile(file)).not.toThrow(); +}); + +test('a crash-replay append keeps the file valid JSONL', async () => { + const stateDir = await mkdtempForTest('agent-device-markers-'); + const markerPath = resolveDaemonStartupPhasesPath(stateDir, 999); + await appendFile(markerPath, '{"data":{"startupPhase":"modules-lo', 'utf8'); + recordDaemonStartupMarker(markerPath, 'composition-loaded'); + const tail = await readFile(markerPath, 'utf8'); + expect(Object.keys(readDaemonStartupPhaseMarkers(tail))).toEqual(['composition-loaded']); +}); + +function spawnDeadPid(): number { + const child = spawnSync(process.execPath, ['-e', ''], { encoding: 'utf8' }); + if (child.error) throw child.error; + const pid = child.pid; + if (pid === undefined) throw new Error('spawnSync returned no pid'); + return pid; +} diff --git a/packages/daemon-contracts/src/daemon-startup-phases.ts b/packages/daemon-contracts/src/daemon-startup-phases.ts new file mode 100644 index 0000000000..129fb4b963 --- /dev/null +++ b/packages/daemon-contracts/src/daemon-startup-phases.ts @@ -0,0 +1,137 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { normalizeError, type NormalizedError } from '@agent-device/kernel/errors'; +import { isProcessAlive } from '@agent-device/host-kit/process'; + +/** + * Startup phase markers: a per-launch record, named for the writing pid, of how far that startup + * got, so a waiting client tells "slow but progressing" from "stuck" without a heartbeat. The + * writer removes it on completion, the client after a joined exit, and each new registration + * owner sweeps the files of provably dead pids. + */ +const DAEMON_STARTUP_PHASES = Object.freeze([ + 'modules-loaded', + 'composition-loaded', + 'lock-acquired', + 'recovery-finished', + 'listening', + 'registered', +] as const); + +export type DaemonStartupPhase = (typeof DAEMON_STARTUP_PHASES)[number]; + +const DAEMON_STARTUP_PHASE_SET: ReadonlySet = new Set(DAEMON_STARTUP_PHASES); + +const DAEMON_STARTUP_PHASES_ENTRY = /^daemon-startup-phases-(\d*)\.jsonl$/; + +/** The marker file the daemon process `pid` writes in `stateDir`; an absent file is no evidence. */ +export function resolveDaemonStartupPhasesPath(stateDir: string, pid: number): string { + return path.join(stateDir, `daemon-startup-phases-${pid}.jsonl`); +} + +/** Appends one phase marker; a refused append goes to `reportFailure`, never fatal to a startup. */ +export function recordDaemonStartupMarker( + startupPhasesPath: string, + phase: DaemonStartupPhase, + reportFailure?: (error: NormalizedError) => void, +): void { + try { + const timestampMs = Date.now(); + fs.mkdirSync(path.dirname(startupPhasesPath), { recursive: true }); + // A crash mid-append can leave no trailing newline; glue two appends and both are lost. + const separator = endsWithoutNewline(startupPhasesPath) ? '\n' : ''; + fs.appendFileSync( + startupPhasesPath, + `${separator}${JSON.stringify({ data: { startupPhase: phase, timestampMs } })}\n`, + ); + } catch (error) { + reportFailure?.(normalizeError(error)); + } +} + +function endsWithoutNewline(file: string): boolean { + let fd: number | undefined; + try { + const stats = fs.statSync(file); + if (stats.size === 0) return false; + const buffer = Buffer.alloc(1); + fd = fs.openSync(file, 'r'); + if (fs.readSync(fd, buffer, 0, 1, stats.size - 1) !== 1) return false; + return buffer[0] !== 0x0a; + } catch { + return false; + } finally { + if (fd !== undefined) fs.closeSync(fd); + } +} + +/** + * Removes this launch's record: before its first marker (a per-pid name identifies a writer, not + * a lifetime, so the entry claims its file first), and after a finished startup or joined exit. + * A refused removal only reports through `reportFailure`; it never stops a startup. + */ +export function removeDaemonStartupPhasesFile( + startupPhasesPath: string, + reportFailure?: (error: NormalizedError) => void, +): void { + try { + fs.rmSync(startupPhasesPath, { force: true }); + } catch (error) { + reportFailure?.(normalizeError(error)); + } +} + +/** Sweeps marker files of provably dead (or unnameable) pids; a live pid's file is kept. */ +export function sweepDeadDaemonStartupPhasesFiles(stateDir: string): void { + let entries: string[]; + try { + entries = fs.readdirSync(stateDir); + } catch { + return; + } + for (const entry of entries) { + const match = DAEMON_STARTUP_PHASES_ENTRY.exec(entry); + if (match === null) continue; + const deadPid = Number(match[1]); + if (deadPid === process.pid) continue; + if (!Number.isSafeInteger(deadPid) || deadPid <= 0 || !isProcessAlive(deadPid)) { + removeDaemonStartupPhasesFile(path.join(stateDir, entry)); + } + } +} + +/** + * The phases observed in a marker tail, keeping each phase's earliest occurrence. Unknown and + * unparsable lines are ignored: a client never fails a startup over a line it does not know. + */ +export function readDaemonStartupPhaseMarkers( + markerTail: string, +): Partial> { + const phases: Partial> = {}; + for (const line of markerTail.split('\n')) { + if (!line.includes('"startupPhase"')) continue; + let record: DaemonStartupMarkerRecord; + try { + record = JSON.parse(line) as DaemonStartupMarkerRecord; + } catch { + continue; + } + const phase = record.data?.startupPhase; + const timestampMs = record.data?.timestampMs; + if ( + typeof phase !== 'string' || + !DAEMON_STARTUP_PHASE_SET.has(phase) || + typeof timestampMs !== 'number' || + !Number.isFinite(timestampMs) + ) { + continue; + } + const seen = phases[phase as DaemonStartupPhase]; + if (seen === undefined || timestampMs < seen) phases[phase as DaemonStartupPhase] = timestampMs; + } + return phases; +} + +type DaemonStartupMarkerRecord = { + data?: { startupPhase?: unknown; timestampMs?: unknown }; +}; diff --git a/packages/daemon/src/server/__tests__/daemon-runtime-app-log.test.ts b/packages/daemon/src/server/__tests__/daemon-runtime-app-log.test.ts index e6230f3a9d..18f5f902cb 100644 --- a/packages/daemon/src/server/__tests__/daemon-runtime-app-log.test.ts +++ b/packages/daemon/src/server/__tests__/daemon-runtime-app-log.test.ts @@ -10,7 +10,7 @@ import type { ResourceDiagnostic } from '@agent-device/host-kit/diagnostics'; import { createDurableResourceEnvelope } from '@agent-device/capture-kit'; import { createTestAppLogLiveHandle, mkdtempForTestSync } from '@agent-device/test-kit'; import { makeSessionStore } from '../../__tests__/support/store-factory.ts'; -import { flushDaemonStartupDiagnostics } from '../http/daemon-runtime.ts'; +import { flushDaemonStartupDiagnostics } from '../http/daemon-startup-diagnostics.ts'; import type { SessionState } from '../../core/session-state.ts'; import { unavailableDeviceRuntimeGateway } from '../../__tests__/support/unavailable-device-runtime-gateway.ts'; diff --git a/packages/daemon/src/server/http/daemon-runtime.ts b/packages/daemon/src/server/http/daemon-runtime.ts index eae6f66584..aa869d6721 100644 --- a/packages/daemon/src/server/http/daemon-runtime.ts +++ b/packages/daemon/src/server/http/daemon-runtime.ts @@ -6,7 +6,7 @@ import { type ClaimRecoveryGatewayFactory, } from '@agent-device/daemon-contracts/device-claim-recovery'; import crypto from 'node:crypto'; -import { asAppError, AppError, normalizeError } from '@agent-device/kernel/errors'; +import { asAppError, AppError } from '@agent-device/kernel/errors'; import { SessionStore } from '../../core/session-store.ts'; import { DAEMON_CAPABILITIES, @@ -76,7 +76,6 @@ import { readCurrentOwnerIdentity, reapOwnedProcessRecordsAtStartup, type OwnedProcessRecordStore, - type OwnerIdentity, } from '@agent-device/host-kit/process'; import { isEnvTruthy, sleep } from '@agent-device/host-kit/retry'; import { @@ -88,9 +87,12 @@ import { import { tryAcquireDaemonRegistration, DAEMON_STARTUP_EXIT_CODES, - type DaemonRegistrationOwner, } from '@agent-device/daemon-contracts/daemon-registration-owner'; -import { watchDaemonMetadataLoss } from './daemon-metadata-loss.ts'; +import { + recordDaemonStartupMarker, + removeDaemonStartupPhasesFile, + resolveDaemonStartupPhasesPath, +} from '@agent-device/daemon-contracts/daemon-startup-phases'; import { retainPngWorker } from '@agent-device/capture-kit/png-worker-client'; import { openWebSessionNames } from '../../sessions/web-session-names.ts'; import { createAppLogAdmissionLedger } from '../../resources/app-log-admission-ledger.ts'; @@ -242,83 +244,6 @@ export function daemonStartupExitCode(error: unknown): number { return typeof exitCode === 'number' ? exitCode : 1; } -export async function flushDaemonStartupDiagnostics( - logPath: string, - diagnostics: readonly ResourceDiagnostic[], -): Promise { - if (diagnostics.length === 0) return; - await withDaemonDiagnosticsScope({ logPath, command: 'daemon-startup', debug: false }, () => { - for (const diagnostic of diagnostics) { - emitDiagnostic({ - level: 'warn', - phase: diagnostic.phase, - data: { resourcePath: diagnostic.resourcePath, ...diagnostic.data }, - }); - } - }); -} - -/** - * Records one daemon-level event. These run outside any request, so there is no request scope and no - * resolved debug level to inherit; debug is forced on for the same reason the #2681 handoff forces it — - * the event is the point of the record and must not be dropped by a level that was never set for it. - */ -async function emitDaemonDiagnostic( - logPath: string, - phase: string, - data: Record, -): Promise { - await withDaemonDiagnosticsScope({ logPath }, () => { - emitDiagnostic({ level: 'warn', phase, data }); - }); -} - -async function finishDaemonRegistration(params: { - registration: DaemonRegistrationOwner; - infoPath: string; - logPath: string; - outcome?: Parameters[0]; -}): Promise { - try { - const removal = await params.registration.finish(params.outcome); - if (removal.state !== 'removed' && removal.state !== 'absent') { - await emitDaemonDiagnostic(params.logPath, 'daemon_info_removal_declined', { - infoPath: params.infoPath, - removed: false, - reason: removal.state, - ...(removal.state === 'replaced' ? { registeredPid: removal.identity.pid } : {}), - }); - } - } catch (error) { - await emitDaemonDiagnostic(params.logPath, 'daemon_registration_finish_failed', { - error: normalizeError(error), - }); - } -} - -/** - * Starts the watch that reports this daemon's registration being taken over, and returns the handle - * that stops it. It is armed only once this process has published its own record: before publication - * the file legitimately describes a predecessor, and losing that is not this daemon's event. - */ -function armDaemonMetadataLossWatch( - stateDir: string, - infoPath: string, - logPath: string, - owner: OwnerIdentity, -): () => void { - return watchDaemonMetadataLoss({ - infoPath, - stateDir, - owner, - onLoss: (loss) => - void emitDaemonDiagnostic(logPath, 'daemon_metadata_lost', { - infoPath, - ...loss, - }).catch(() => {}), - }); -} - export async function startDaemonRuntime( options: DaemonRuntimeOptions, ): Promise { @@ -347,6 +272,13 @@ async function startDaemonRuntimeOnHost( const { env } = host; const stdout = options.stdout ?? process.stdout; const stderr = options.stderr ?? process.stderr; + // Function-scoped import: these helpers stay out of the daemon entry's eager closure (ADR 0019). + const { + armDaemonMetadataLossWatch, + emitDaemonDiagnostic, + finishDaemonRegistration, + flushDaemonStartupDiagnostics, + } = await import('./daemon-startup-diagnostics.ts'); const daemonPaths = resolveDaemonPaths(env.AGENT_DEVICE_STATE_DIR, { env, cwd: host.cwd }); const { baseDir, infoPath, logPath, sessionsDir } = daemonPaths; const daemonServerMode = resolveDaemonServerMode(env.AGENT_DEVICE_DAEMON_SERVER_MODE); @@ -660,6 +592,13 @@ async function startDaemonRuntimeOnHost( rollback.onRollback(async () => { await finishDaemonRegistration({ registration, infoPath, logPath }); }); + const startupPhasesPath = resolveDaemonStartupPhasesPath(baseDir, process.pid); + const recordStartupPhase = (phase: Parameters[1]): void => + recordDaemonStartupMarker(startupPhasesPath, phase, (error) => { + stderr.write(`Daemon startup marker write failed: ${error.message}\n`); + }); + recordStartupPhase('lock-acquired'); + rollback.onRollback(() => { stopMetadataLossWatch(); }); @@ -713,6 +652,7 @@ async function startDaemonRuntimeOnHost( data: { error: error instanceof Error ? error.message : String(error) }, }); }); + recordStartupPhase('recovery-finished'); const { openDaemonServers, reconcileDeviceClaimsForDaemonStartup } = await import('./server-startup.ts'); const { servers, socketPort, httpPort } = await openDaemonServers( @@ -752,7 +692,10 @@ async function startDaemonRuntimeOnHost( }, rollback, ); + recordStartupPhase('listening'); publishDaemonInfo(socketPort, httpPort); + recordStartupPhase('registered'); + removeDaemonStartupPhasesFile(startupPhasesPath); stopMetadataLossWatch = armDaemonMetadataLossWatch(baseDir, infoPath, logPath, daemonIdentity); await flushDaemonStartupDiagnostics(logPath, startupDiagnostics); for (const plugin of deps.pluginInventory.filter(({ state }) => state === 'failed')) { diff --git a/packages/daemon/src/server/http/daemon-startup-diagnostics.ts b/packages/daemon/src/server/http/daemon-startup-diagnostics.ts new file mode 100644 index 0000000000..ba5ccbc2d9 --- /dev/null +++ b/packages/daemon/src/server/http/daemon-startup-diagnostics.ts @@ -0,0 +1,75 @@ +import { normalizeError } from '@agent-device/kernel/errors'; +import { emitDiagnostic, type ResourceDiagnostic } from '@agent-device/host-kit/diagnostics'; +import type { OwnerIdentity } from '@agent-device/host-kit/process'; +import type { DaemonRegistrationOwner } from '@agent-device/daemon-contracts/daemon-registration-owner'; +import { withDaemonDiagnosticsScope } from '@agent-device/daemon-contracts/daemon-diagnostics-scope'; +import { watchDaemonMetadataLoss } from './daemon-metadata-loss.ts'; + +export async function flushDaemonStartupDiagnostics( + logPath: string, + diagnostics: readonly ResourceDiagnostic[], +): Promise { + if (diagnostics.length === 0) return; + await withDaemonDiagnosticsScope({ logPath, command: 'daemon-startup', debug: false }, () => { + for (const diagnostic of diagnostics) { + emitDiagnostic({ + level: 'warn', + phase: diagnostic.phase, + data: { resourcePath: diagnostic.resourcePath, ...diagnostic.data }, + }); + } + }); +} + +/** Records one daemon-level event outside any request scope, with debug forced on. */ +export async function emitDaemonDiagnostic( + logPath: string, + phase: string, + data: Record, +): Promise { + await withDaemonDiagnosticsScope({ logPath }, () => { + emitDiagnostic({ level: 'warn', phase, data }); + }); +} + +export async function finishDaemonRegistration(params: { + registration: DaemonRegistrationOwner; + infoPath: string; + logPath: string; + outcome?: Parameters[0]; +}): Promise { + try { + const removal = await params.registration.finish(params.outcome); + if (removal.state !== 'removed' && removal.state !== 'absent') { + await emitDaemonDiagnostic(params.logPath, 'daemon_info_removal_declined', { + infoPath: params.infoPath, + removed: false, + reason: removal.state, + ...(removal.state === 'replaced' ? { registeredPid: removal.identity.pid } : {}), + }); + } + } catch (error) { + await emitDaemonDiagnostic(params.logPath, 'daemon_registration_finish_failed', { + error: normalizeError(error), + }); + } +} + +/** Watches this registration being taken over; arm only after this process published. */ +export function armDaemonMetadataLossWatch( + stateDir: string, + infoPath: string, + logPath: string, + owner: OwnerIdentity, +): () => void { + return watchDaemonMetadataLoss({ + infoPath, + stateDir, + owner, + onLoss: (loss) => + void emitDaemonDiagnostic(logPath, 'daemon_metadata_lost', { + infoPath, + ...loss, + }).catch(() => {}), + }); +} diff --git a/scripts/__tests__/eager-closure-budgets.ts b/scripts/__tests__/eager-closure-budgets.ts index 73851ada96..403ec79032 100644 --- a/scripts/__tests__/eager-closure-budgets.ts +++ b/scripts/__tests__/eager-closure-budgets.ts @@ -152,6 +152,13 @@ export const APPROVED_CLOSURE_ADDITIONS: readonly Readonly<{ 'The one owner of per-runtime async-local state: the store, its standalone fallback and its disposal, so Apple, request-cancel and the root declare only slot keys instead of each keeping a copy.', owner: 'thymikee', }, + { + module: 'packages/daemon-contracts/src/daemon-startup-phases.ts', + issue: '#3782', + reason: + 'The startup phase record owner the #3791 review asked for. The daemon entry records its earliest `modules-loaded` marker at module load, so the owner must be eager there; the entry pays for the record it writes first.', + owner: 'thymikee', + }, ]); /** The category is a function of the path, never a hand-written column. */ diff --git a/src/__tests__/test-utils/registered-daemon-fixture.test.ts b/src/__tests__/test-utils/registered-daemon-fixture.test.ts index 53e4105d83..d179429605 100644 --- a/src/__tests__/test-utils/registered-daemon-fixture.test.ts +++ b/src/__tests__/test-utils/registered-daemon-fixture.test.ts @@ -1,15 +1,53 @@ import assert from 'node:assert/strict'; import fs from 'node:fs'; +import path from 'node:path'; import { test } from 'vitest'; import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution'; import { stopDaemonProcess } from '@agent-device/daemon-contracts/daemon-process'; +import { DAEMON_STARTUP_EXIT_CODES } from '@agent-device/daemon-contracts/daemon-registration-owner'; +import { resolveDaemonStartupPhasesPath } from '@agent-device/daemon-contracts/daemon-startup-phases'; import { mkdtempForTestSync } from '@agent-device/test-kit'; import { + launchRegisteredDaemonFixtureArgs, + registeredDaemonFixtureArgs, spawnRegisteredDaemonFixture, + waitForFixtureFile, waitForRegisteredDaemonFixture, finishRegisteredDaemonFixture, } from './registered-daemon-fixture.ts'; +test('args built before any launch keep their own entry script under a concurrent second build', async () => { + // Under a shared entry path child 1 would run child 2's switches and carry the wrong marker. + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-fixture-entry-isolation-')); + fs.mkdirSync(paths.baseDir, { recursive: true }); + fs.writeFileSync(path.join(paths.baseDir, 'defer-publication'), 'wait'); + const fields = { + httpPort: 4211, + token: 'fixture', + version: 'test', + codeOrigin: 'checkout' as const, + codeSignature: 'fixture', + }; + const firstArgs = registeredDaemonFixtureArgs(paths, fields, undefined, { + emitStartupMarkerBeforeAcquisition: true, + }); + const secondArgs = registeredDaemonFixtureArgs(paths, fields, undefined, { + emitStartupMarker: true, + }); + assert.notEqual(firstArgs[1], secondArgs[1]); + const first = launchRegisteredDaemonFixtureArgs(paths, firstArgs, { stdio: 'ignore' }); + try { + await waitForFixtureFile(path.join(paths.baseDir, `registration-held-${first.pid}`)); + const second = launchRegisteredDaemonFixtureArgs(paths, secondArgs, { stdio: 'ignore' }); + assert.equal((await second.exited).exitCode, DAEMON_STARTUP_EXIT_CODES.busy); + const text = fs.readFileSync(resolveDaemonStartupPhasesPath(paths.baseDir, first.pid), 'utf8'); + assert.match(text, /modules-loaded/); + assert.doesNotMatch(text, /lock-acquired/, 'child 1 must run the script built for it'); + } finally { + await finishRegisteredDaemonFixture(paths.baseDir); + } +}); + test('a joined fixture exit refuses its remaining registration metadata', async () => { const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-fixture-exited-publication-')); const child = spawnRegisteredDaemonFixture( diff --git a/src/__tests__/test-utils/registered-daemon-fixture.ts b/src/__tests__/test-utils/registered-daemon-fixture.ts index a4240ce539..295a78ae51 100644 --- a/src/__tests__/test-utils/registered-daemon-fixture.ts +++ b/src/__tests__/test-utils/registered-daemon-fixture.ts @@ -1,4 +1,5 @@ import assert from 'node:assert/strict'; +import crypto from 'node:crypto'; import fs from 'node:fs'; import path from 'node:path'; import { vi } from 'vitest'; @@ -17,32 +18,54 @@ const children = new Map< Array<{ launch: ReturnType; startTime: string | null }> >(); +/** Per-child behavior switches, bound into this child's entry script at write time. */ +export type RegisteredDaemonFixtureOptions = Readonly<{ + /** Record `lock-acquired` after acquisition, as the daemon runtime does, for this child only. */ + emitStartupMarker?: boolean; + /** Record `modules-loaded` before acquisition, as the daemon entry does, for this child only. */ + emitStartupMarkerBeforeAcquisition?: boolean; +}>; + /** A real registration owner, advertising the caller's HTTP fixture and joining before deletion. */ export function registeredDaemonFixtureArgs( paths: DaemonPaths, fields: DaemonRegistrationFields, acquisitionBarrier?: string, + fixtureOptions?: RegisteredDaemonFixtureOptions, ): string[] { - const entry = path.join(paths.baseDir, 'dist', 'src', 'internal', 'daemon.js'); + // One directory per launch so a later build cannot overwrite a script an earlier child has + // not loaded; the path still ends in the real daemon entry name for the identity checks. + const launchDir = path.join(paths.baseDir, 'fixture-entries', crypto.randomUUID()); + const entry = path.join(launchDir, 'dist', 'src', 'internal', 'daemon.js'); fs.mkdirSync(path.dirname(entry), { recursive: true }); - fs.writeFileSync(path.join(paths.baseDir, 'package.json'), '{"type":"module"}'); + fs.writeFileSync(path.join(launchDir, 'package.json'), '{"type":"module"}'); const registrationUrl = new URL( '../../../packages/daemon-contracts/src/daemon-registration-owner.ts', import.meta.url, ).href; + const phasesUrl = new URL( + '../../../packages/daemon-contracts/src/daemon-startup-phases.ts', + import.meta.url, + ).href; fs.writeFileSync( entry, `import fs from 'node:fs'; import path from 'node:path'; import { DAEMON_STARTUP_EXIT_CODES, tryAcquireDaemonRegistration } from ${JSON.stringify(registrationUrl)}; +import { removeDaemonStartupPhasesFile, recordDaemonStartupMarker, resolveDaemonStartupPhasesPath } from ${JSON.stringify(phasesUrl)}; const paths = ${JSON.stringify(paths)}; const barrier = ${JSON.stringify(acquisitionBarrier)}; +const emitBefore = ${fixtureOptions?.emitStartupMarkerBeforeAcquisition ? 'true' : 'false'}; +const emitAfter = ${fixtureOptions?.emitStartupMarker ? 'true' : 'false'}; if (barrier) { fs.writeFileSync(barrier + '.ready-' + process.pid, 'ready'); while (!fs.existsSync(barrier)) await new Promise(resolve => setTimeout(resolve, 10)); } +if (emitBefore) { const own = resolveDaemonStartupPhasesPath(paths.baseDir, process.pid); removeDaemonStartupPhasesFile(own); recordDaemonStartupMarker(own, 'modules-loaded'); } const acquired = await tryAcquireDaemonRegistration(paths); if (acquired.status !== 'acquired') process.exit(DAEMON_STARTUP_EXIT_CODES[acquired.status]); +if (emitAfter) + recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(paths.baseDir, process.pid), 'lock-acquired'); process.on('SIGTERM', async () => { if (fs.existsSync(path.join(paths.baseDir, 'ignore-sigterm'))) return; const deferred = path.join(paths.baseDir, 'repair-on-shutdown.json'); @@ -55,6 +78,7 @@ process.on('SIGTERM', async () => { process.exit(0); }); fs.writeFileSync(path.join(paths.baseDir, 'registration-held'), 'ready'); +fs.writeFileSync(path.join(paths.baseDir, 'registration-held-' + process.pid), 'ready'); while (fs.existsSync(path.join(paths.baseDir, 'defer-publication'))) await new Promise(resolve => setTimeout(resolve, 10)); acquired.owner.publish(${JSON.stringify(fields)}); setInterval(() => {}, 1000); @@ -63,21 +87,38 @@ setInterval(() => {}, 1000); return ['--experimental-strip-types', entry]; } +export function launchRegisteredDaemonFixtureArgs( + paths: DaemonPaths, + args: string[], + options: Parameters[2], +): ReturnType { + const child = actualCommand.runCmdDetachedMonitored(process.execPath, args, options); + const owned = children.get(paths.baseDir) ?? []; + owned.push({ launch: child, startTime: readProcessStartTime(child.pid) }); + children.set(paths.baseDir, owned); + return child; +} + export function spawnRegisteredDaemonFixture( paths: DaemonPaths, fields: DaemonRegistrationFields, options: Parameters[2], acquisitionBarrier?: string, + fixtureOptions?: RegisteredDaemonFixtureOptions, ): ReturnType { - const child = actualCommand.runCmdDetachedMonitored( - process.execPath, - registeredDaemonFixtureArgs(paths, fields, acquisitionBarrier), + return launchRegisteredDaemonFixtureArgs( + paths, + registeredDaemonFixtureArgs(paths, fields, acquisitionBarrier, fixtureOptions), options, ); - const owned = children.get(paths.baseDir) ?? []; - owned.push({ launch: child, startTime: readProcessStartTime(child.pid) }); - children.set(paths.baseDir, owned); - return child; +} + +/** Waits up to 4 s for a fixture child to publish `file`; one shared wait for every probe. */ +export async function waitForFixtureFile(file: string): Promise { + for (let attempt = 0; !fs.existsSync(file); attempt += 1) { + assert.ok(attempt < 400, `fixture did not publish ${file}`); + await new Promise((resolve) => setTimeout(resolve, 10)); + } } export async function waitForRegisteredDaemonFixture( diff --git a/src/commands/schema/cli-help.ts b/src/commands/schema/cli-help.ts index 2619c82eb5..12fbdb0894 100644 --- a/src/commands/schema/cli-help.ts +++ b/src/commands/schema/cli-help.ts @@ -41,6 +41,10 @@ const ENVIRONMENT_LINES = [ label: 'AGENT_DEVICE_DAEMON_AUTH_TOKEN', description: 'Remote daemon service/API token', }, + { + label: 'AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', + description: 'Local daemon startup ceiling (default 120000)', + }, { label: 'AGENT_DEVICE_CLOUD_BASE_URL', description: 'Bridge/control-plane API origin for cloud auth and /api-keys', @@ -508,6 +512,7 @@ Runner and daemon lifecycle (applies to simulators too): No runner read launches a session app that is not running: snapshot, wait, is, get, a reading find, and an interaction's leading reads (a gesture's viewport read, the capture that resolves a selector click/fill) answer the retriable APP_NOT_RUNNING instead of bare-launching over a launch SpringBoard still holds behind its deep-link confirmation. Only open, activate, and a command that mutates without a leading read bring a stopped app up. close keeps a healthy iOS simulator XCTest runner warm by default so the next open on that simulator (same udid in the same simulator set) skips the runner build, unless --shutdown was requested, the session was recording, or the session held a device lease. A retained runner auto-stops after an idle window (default 5 minutes); set AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS to override, or 0 to disable idle stop and retain until daemon exit. Only a ready runner with an observed listener is retained; close retires unfinished prewarm recovery. Losing the retained listener stops that generation, and daemon shutdown stops idle retained runners rather than handing them off without supervision. Each AGENT_DEVICE_STATE_DIR runs its own daemon. It self-exits after an idle window (default 5 minutes, matching the runner idle-stop default) once it has no open sessions, no in-flight requests, and no active recording; set AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS to override, or 0 to disable idle reap. + A client starting a local daemon waits while the daemon process stays alive and keeps publishing startup progress, so an overloaded host makes startup slower, not failed. It stops at a ceiling (default 120 s; set AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS to a positive integer number of milliseconds to override) and names the outcome in error details.reason: startup-exited (the process ended before registering), startup-stalled (alive, no startup progress for a fixed 60 s window the ceiling variable does not change), or startup-ceiling (a startup was still live when the ceiling passed — this client's daemon or another client's; the client stops its own launch and never signals another client's live daemon. Only a progressing own launch outlasts the 60 s stall window, so a ceiling set below 60 s also ends a silent own launch this way). details.phases carries the startup phase timings it observed and details.startupTimeoutMs the effective ceiling. Branch on details.reason, never on the message. The variable is read only when a client must start a daemon: a value that is not a positive integer fails with INVALID_ARGS before the daemon spawns, and a client that only attaches to or reuses a running daemon never reads it. A command that needs the shared local daemon starts it for its state dir when none runs (device status and device release --stale never contact one). When the running daemon comes from another version or build, the command replaces it only if the daemon reports no work: no in-flight request, session, repair session, or live lease (a streaming resumable upload and a host human-control hold that names no lease are not counted); otherwise it fails with details.reason daemon-busy and leaves the daemon alone. A daemon that agreed to leave is waited for up to 60 s and never signaled while it drains; one still running then fails with details.reason daemon-drain-timeout unless it answers nothing and is provably a daemon process, which is force-retired. A daemon from a release that predates this check is never replaced and fails with daemon-incompatible. End a busy or older daemon deliberately with agent-device daemon stop --state-dir . Node clients pick daemonLifecycle 'attach' (never starts or replaces a daemon; fails with daemon-not-running or daemon-incompatible) or 'managed' (the default until 0.22), and ensureDaemon() starts or replaces one explicitly. On a machine shared by several agents, set AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS to also expire an individual session that has taken a host-global device claim and then received no commands for that long; the claim is released and the next command on that session answers SESSION_NOT_FOUND with details.reason SESSION_IDLE_EXPIRED naming the window and the device. Off by default, because on a shared host "idle" and "thinking" are indistinguishable: only set it where an abandoned claim blocking every other agent is the worse failure. Sessions holding a remote lease or any active capture (recording, logs, audio, performance, or trace) are never expired this way. A stale iOS runner lease — its owner process dead, or its AGENT_DEVICE_STATE_DIR deleted — is reclaimed automatically instead of failing with "is already owned by another agent-device daemon". A live owner's runner is also reclaimed when the requesting daemon holds the host-global device claim for that device: claims are exclusive, so holding one proves the runner's owner released the device and merely kept the runner warm. The error remains only for owners outside claim arbitration (a pre-claims build, or daemons pointed at different claim stores). diff --git a/src/daemon-client/__tests__/daemon-client-startup-budget.test.ts b/src/daemon-client/__tests__/daemon-client-startup-budget.test.ts new file mode 100644 index 0000000000..577aedb236 --- /dev/null +++ b/src/daemon-client/__tests__/daemon-client-startup-budget.test.ts @@ -0,0 +1,478 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { afterEach, test, vi } from 'vitest'; +import { AppError } from '@agent-device/kernel/errors'; +import type { ExecDetachedExit } from '@agent-device/host-kit/command'; +import { DAEMON_STARTUP_EXIT_CODES } from '@agent-device/daemon-contracts/daemon-registration-owner'; +import { + readDaemonStartupPhaseMarkers, + recordDaemonStartupMarker, + resolveDaemonStartupPhasesPath, + type DaemonStartupPhase, +} from '@agent-device/daemon-contracts/daemon-startup-phases'; +import { + resolveDaemonPaths, + type DaemonPaths, +} from '@agent-device/daemon-contracts/daemon-resolution'; +import { + closeLoopbackServer, + mkdtempForTestSync, + supportsLoopbackBind, +} from '@agent-device/test-kit'; +import { isProcessAlive } from '@agent-device/host-kit/process'; +import { readVersion } from '@agent-device/host-kit/version'; +import { + spawnRegisteredDaemonFixture, + finishRegisteredDaemonFixtures, + waitForFixtureFile, +} from '../../__tests__/test-utils/registered-daemon-fixture.ts'; +import { + startHttpDaemonFixture, + currentDaemonCodeSignature, +} from '../../__tests__/test-utils/daemon-http-fixture.ts'; + +vi.mock('@agent-device/host-kit/command', async (importOriginal) => ({ + ...(await importOriginal()), + runCmdDetachedMonitored: vi.fn(), +})); +vi.mock('@agent-device/host-kit/retry', async (importOriginal) => ({ + ...(await importOriginal()), + sleep: vi.fn(), +})); +// A pinned fabricated pid answers the signal-0 probe from this map; afterEach drops the map. +let livenessOverrides: Map | undefined; +vi.mock('@agent-device/host-kit/process', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + isProcessAlive: (pid: number) => livenessOverrides?.get(pid) ?? actual.isProcessAlive(pid), + }; +}); +import { runCmdDetachedMonitored } from '@agent-device/host-kit/command'; +import { sleep } from '@agent-device/host-kit/retry'; +import { sendToDaemon } from '../daemon-client.ts'; + +const actualRetry = await vi.importActual( + '@agent-device/host-kit/retry', +); +const spawn = vi.mocked(runCmdDetachedMonitored); +const pause = vi.mocked(sleep); + +afterEach(async () => { + livenessOverrides = undefined; + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + await finishRegisteredDaemonFixtures(); + spawn.mockReset(); + pause.mockReset(); +}); + +function request(paths: DaemonPaths, command = 'devices') { + return { + session: 'default', + command, + positionals: [], + flags: { stateDir: paths.baseDir, daemonTransport: 'http' as const }, + }; +} + +function fields(httpPort: number) { + return { + httpPort, + token: 'secret', + version: readVersion(), + codeOrigin: 'checkout' as const, + codeSignature: currentDaemonCodeSignature(), + }; +} + +/** A deferred-publication fixture that never exits on its own; retirement can still stop it. */ +function spawnUnregisteredDaemonChild( + paths: DaemonPaths, + options: Parameters[2], + emitStartupMarker = false, +): ReturnType { + fs.mkdirSync(paths.baseDir, { recursive: true }); + fs.writeFileSync(path.join(paths.baseDir, 'defer-publication'), 'wait'); + return spawnRegisteredDaemonFixture(paths, fields(1), options, undefined, { emitStartupMarker }); +} + +function registrationHeldFor(paths: DaemonPaths, pid: number): string { + return path.join(paths.baseDir, `registration-held-${pid}`); +} + +async function expectStartupFailure(pending: Promise): Promise { + let thrown: AppError | undefined; + try { + await pending; + } catch (error) { + if (!(error instanceof AppError)) throw error; + thrown = error; + } + assert.ok(thrown, 'expected the startup to reject'); + assert.equal(thrown.message, 'Failed to start daemon'); + assert.equal(thrown.details?.kind, 'daemon_startup_failed'); + return thrown; +} + +function startupPhases(failure: AppError): Record { + assert.ok( + failure.details?.phases && typeof failure.details.phases === 'object', + 'startup failure must record the phase timings it observed', + ); + return failure.details.phases as Record; +} + +async function realPause(ms = 5) { + await actualRetry.sleep(ms); +} + +/** Pins `Date.now` to a mutable value: advancing the wait costs no real time. */ +function virtualClock(): { value: number } { + const now = { value: Date.now() }; + vi.spyOn(Date, 'now').mockImplementation(() => now.value); + return now; +} + +/** The startup phase records left in this state dir, by file name. */ +function phaseRecordEntries(paths: DaemonPaths): string[] { + return fs.readdirSync(paths.baseDir).filter((entry) => entry.startsWith('daemon-startup-phases')); +} + +function registrationBooted(paths: DaemonPaths): () => boolean { + let booted = false; + return () => (booted ||= fs.existsSync(path.join(paths.baseDir, 'registration-held'))); +} + +test('a daemon slower than the retired 15 s budget starts successfully while it keeps progressing', async (t) => { + if (!(await supportsLoopbackBind())) return t.skip('loopback unavailable'); + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '150000'); + const http = await startHttpDaemonFixture({ devices: [] }); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-slow-success-')); + // The success must rest on the markers: four successive phases 20 s apart across 80 s of virtual + // time under a 150 s ceiling keep progress inside the 60 s stall window; the same span with no + // readable markers fails as startup-stalled, one rewritten phase never advances progress. + const phases: readonly DaemonStartupPhase[] = [ + 'modules-loaded', + 'composition-loaded', + 'lock-acquired', + 'recovery-finished', + ]; + const now = virtualClock(); + const spawned = { at: 0, child: undefined as { pid: number } | undefined }; + const booted = registrationBooted(paths); + spawn.mockImplementation((_command, _args, options) => { + const child = spawnRegisteredDaemonFixture(paths, fields(http.port), options); + spawned.at = now.value; + spawned.child = child; + return child; + }); + fs.writeFileSync(path.join(paths.baseDir, 'defer-publication'), 'wait'); + let nextPhase = 0; + pause.mockImplementation(async () => { + if (!booted()) return await realPause(); + if (spawned.child && now.value - spawned.at < 80_000) { + now.value += 20_000; + if (nextPhase < phases.length) { + recordDaemonStartupMarker( + resolveDaemonStartupPhasesPath(paths.baseDir, spawned.child.pid), + phases[nextPhase]!, + ); + nextPhase += 1; + } + if (nextPhase === phases.length) + fs.rmSync(path.join(paths.baseDir, 'defer-publication'), { force: true }); + return await realPause(); + } + await realPause(20); + }); + try { + const response = await sendToDaemon(request(paths)); + assert.equal(response.ok, true); + assert.ok(now.value - spawned.at >= 60_000, 'this startup must outlast the stall window'); + assert.equal(spawn.mock.calls.length, 1); + assert.equal(http.rpcRequests.length, 1); + assert.ok(spawned.child && isProcessAlive(spawned.child.pid), 'the client must not retire it'); + // The wait survived on four distinct phases, not one repeated marker. + const markers = readDaemonStartupPhaseMarkers( + fs.readFileSync(resolveDaemonStartupPhasesPath(paths.baseDir, spawned.child!.pid), 'utf8'), + ); + for (const phase of phases) { + assert.equal(typeof markers[phase], 'number', `the ${phase} phase was never observed`); + } + } finally { + await closeLoopbackServer(http.server); + } +}); + +test('a daemon that exits during startup fails with startup-exited within one second of the exit', async () => { + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-exit-fast-')); + let resolveFirstExit: ((exit: ExecDetachedExit) => void) | undefined; + let exitObservedAt = 0; + spawn.mockImplementation(() => { + const pid = 998_231 + spawn.mock.calls.length; + if (spawn.mock.calls.length === 1) { + const exited = new Promise((resolve) => { + resolveFirstExit = resolve; + }); + void exited.then(() => { + exitObservedAt = Date.now(); + }); + setTimeout(() => resolveFirstExit?.({ pid, exitCode: 1 }), 400); + return { pid, exited }; + } + return { pid, exited: Promise.resolve({ pid, exitCode: 1 }) }; + }); + pause.mockImplementation(async (ms) => { + await realPause(Math.min(Number(ms), 50)); + }); + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-exited'); + assert.ok(exitObservedAt > 0, 'the fixture must have delivered the exit'); + assert.ok(Date.now() - exitObservedAt <= 1_000, 'the failure must follow the exit within 1 s'); +}); + +test('a live daemon that publishes no startup marker is retired as stalled before the ceiling', async () => { + // Unsets the variable, even when the host exports it: the resolved ceiling must be the default. + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', undefined); + const now = virtualClock(); + const started = now.value; + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-no-marker-')); + let child: ReturnType | undefined; + spawn.mockImplementation((_command, _args, options) => { + child = spawnUnregisteredDaemonChild(paths, options); + return child; + }); + const booted = registrationBooted(paths); + pause.mockImplementation(async () => { + if (!booted()) return await realPause(); + if (now.value - started < 70_000) now.value += 5_000; + await realPause(); + }); + try { + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + const elapsed = now.value - started; + assert.equal(failure.details?.reason, 'startup-stalled'); + assert.ok( + elapsed >= 60_000 && elapsed < 75_000, + `a stall must retire the daemon at the stall window, not the ceiling (${elapsed}ms)`, + ); + assert.equal(failure.details?.startupTimeoutMs, 120_000); + assert.equal(startupPhases(failure).spawned, 0); + assert.match(String(failure.details?.hint), /AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS/); + assert.equal(spawn.mock.calls.length, 1, 'a stalled daemon is not retried'); + await (child?.exited ?? Promise.resolve()); + assert.deepEqual( + phaseRecordEntries(paths), + [], + 'a joined startup must sweep its own phase-marker file', + ); + } finally { + await (child?.exited ?? Promise.resolve()); + } +}); + +test('a live daemon that keeps progressing but never registers fails at the env ceiling with phases', async () => { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '30000'); + const now = virtualClock(); + const started = now.value; + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-ceiling-')); + let child: ReturnType | undefined; + const booted = registrationBooted(paths); + spawn.mockImplementation((_command, _args, options) => { + child = spawnUnregisteredDaemonChild(paths, options, true); + return child; + }); + pause.mockImplementation(async () => { + if (!booted()) return await realPause(); + if (now.value - started < 30_000) now.value += 1_000; + await realPause(); + }); + try { + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-ceiling'); + assert.equal(failure.details?.startupTimeoutMs, 30_000); + const phases = startupPhases(failure); + assert.equal(phases.spawned, 0); + assert.ok( + typeof phases['lock-acquired'] === 'number' && + phases['lock-acquired'] >= 0 && + phases['lock-acquired'] < 30_000, + 'the observed marker must be reported relative to the spawn', + ); + assert.match(String(failure.details?.hint), /AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS/); + } finally { + await (child?.exited ?? Promise.resolve()); + } +}); + +test("a startup progressed by a fresh launch is not judged stalled by an earlier launch's leftover markers", async () => { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '30000'); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-stale-history-')); + fs.mkdirSync(paths.baseDir, { recursive: true }); + fs.writeFileSync(path.join(paths.baseDir, 'defer-publication'), 'wait'); + const leftover = spawnRegisteredDaemonFixture(paths, fields(1), { stdio: 'ignore' }, undefined, { + emitStartupMarker: true, + }); + await waitForFixtureFile(registrationHeldFor(paths, leftover.pid)); + process.kill(leftover.pid, 'SIGTERM'); + await leftover.exited; + assert.ok( + phaseRecordEntries(paths).length > 0, + 'the leftover launch must have left a real daemon-written marker record', + ); + const now = virtualClock(); + const started = now.value; + let child: ReturnType | undefined; + let booted = false; + spawn.mockImplementation((_command, _args, options) => { + child = spawnUnregisteredDaemonChild(paths, options, true); + return child; + }); + pause.mockImplementation(async () => { + if (child && !booted) { + booted = fs.existsSync(registrationHeldFor(paths, child.pid)); + if (!booted) return await realPause(); + } + if (now.value - started < 30_000) now.value += 1_000; + await realPause(); + }); + try { + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-ceiling'); + const phases = startupPhases(failure); + assert.equal( + typeof phases['lock-acquired'], + 'number', + 'the fresh launch must observe its own marker despite the older record', + ); + await (child?.exited ?? Promise.resolve()); + assert.ok( + !phaseRecordEntries(paths).some((entry) => entry.includes(String(child?.pid))), + "the joined exit must sweep this launch's own marker file", + ); + } finally { + await (child?.exited ?? Promise.resolve()); + } +}); + +test('a busy exit while a live lock holder defers publication reaches the ceiling, not startup-exited', async () => { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '2000'); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-holder-ceiling-')); + const deferred = path.join(paths.baseDir, 'defer-publication'); + fs.writeFileSync(deferred, 'wait'); + const winner = spawnRegisteredDaemonFixture(paths, fields(1), { stdio: 'ignore' }, undefined, { + emitStartupMarker: true, + }); + await waitForFixtureFile(registrationHeldFor(paths, winner.pid)); + let contender: ReturnType | undefined; + spawn.mockImplementation((_command, _args, options) => { + contender = spawnRegisteredDaemonFixture(paths, fields(1), options, undefined, { + emitStartupMarkerBeforeAcquisition: true, + }); + return contender; + }); + pause.mockImplementation(async () => { + await realPause(20); + }); + const signal = vi.spyOn(process, 'kill'); + try { + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-ceiling'); + assert.equal(failure.details?.startupTimeoutMs, 2_000); + assert.match(String(failure.details?.hint), /AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS/); + assert.equal(typeof startupPhases(failure)['modules-loaded'], 'number'); + assert.equal(isProcessAlive(winner.pid), true); + assert.equal( + signal.mock.calls.some(([pid, kind]) => pid === winner.pid && kind !== 0), + false, + ); + assert.ok(contender); + assert.deepEqual( + phaseRecordEntries(paths).sort(), + [path.basename(resolveDaemonStartupPhasesPath(paths.baseDir, winner.pid))], + 'the exited contender record must be swept; only the live winner record may remain', + ); + } finally { + signal.mockRestore(); + fs.rmSync(deferred, { force: true }); + await (contender?.exited ?? Promise.resolve()); + } +}); + +test('a child gone at the deadline without an observed exit and no live startup fails exited', async () => { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '2000'); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-gone-at-deadline-')); + livenessOverrides = new Map([[997_842, false]]); + spawn.mockImplementation(() => ({ + pid: 997_842, + exited: new Promise(() => {}), + readIdentity: () => ({ pid: 997_842, startTime: null }), + })); + pause.mockImplementation(async () => { + await realPause(50); + }); + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-exited'); + assert.equal(failure.details?.startupTimeoutMs, 2_000); +}); + +test('an observed exit on a reused launch pid is not a live startup at the deadline', async () => { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '2000'); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-reused-pid-exit-')); + const reusedPid = 997_603; + livenessOverrides = new Map([[reusedPid, true]]); + let resolveExit: ((exit: ExecDetachedExit) => void) | undefined; + spawn.mockImplementation(() => ({ + pid: reusedPid, + exited: new Promise((resolve) => { + resolveExit = resolve; + }), + readIdentity: () => ({ pid: reusedPid, startTime: null }), + })); + const now = virtualClock(); + let advanced = false; + pause.mockImplementation(async () => { + if (!advanced) { + advanced = true; + now.value += 2_000; + resolveExit?.({ pid: reusedPid, exitCode: DAEMON_STARTUP_EXIT_CODES.busy }); + } + await realPause(20); + }); + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-exited'); + assert.equal(failure.details?.startupTimeoutMs, 2_000); + assert.equal(spawn.mock.calls.length, 1, 'the deadline had passed, so no second attempt started'); +}); + +test('an unproven registration exit reports startup-exited even under a short env ceiling', async () => { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '2000'); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-exit-under-ceiling-')); + spawn.mockImplementation(() => ({ + pid: 998_444, + exited: Promise.resolve({ pid: 998_444, exitCode: DAEMON_STARTUP_EXIT_CODES.unproven }), + })); + pause.mockImplementation(async () => { + await realPause(); + }); + const failure = await expectStartupFailure(sendToDaemon(request(paths))); + assert.equal(failure.details?.reason, 'startup-exited'); + assert.equal(failure.details?.startupTimeoutMs, 2_000); +}); + +test('a startup ceiling that is not a positive integer is rejected before any daemon spawns', async () => { + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-bad-ceiling-')); + for (const raw of ['0', '-5', 'abc', '12.5']) { + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', raw); + await assert.rejects( + sendToDaemon(request(paths)), + (error: unknown) => + error instanceof AppError && + error.code === 'INVALID_ARGS' && + error.message.includes('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS'), + ); + } + assert.equal(spawn.mock.calls.length, 0); +}); diff --git a/src/daemon-client/__tests__/daemon-client-startup-race.test.ts b/src/daemon-client/__tests__/daemon-client-startup-race.test.ts index 726c45031a..d1c1a26430 100644 --- a/src/daemon-client/__tests__/daemon-client-startup-race.test.ts +++ b/src/daemon-client/__tests__/daemon-client-startup-race.test.ts @@ -205,6 +205,7 @@ test.for(['early exit', 'timeout'] as const)( let child: ReturnType | undefined; let now = Date.now(); let advanced = false; + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '15000'); vi.spyOn(Date, 'now').mockImplementation(() => now); spawn.mockImplementation((_command, _args, options) => { paths = resolveDaemonPaths(String(options?.env?.AGENT_DEVICE_STATE_DIR)); @@ -384,7 +385,55 @@ for (const exit of [ }); } +test('a busy contender that exits sweeps its own pre-lock marker before the retried attempt', async (t) => { + if (!(await supportsLoopbackBind())) return t.skip('loopback unavailable'); + const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-busy-retry-marker-')); + const http = await startHttpDaemonFixture({ devices: [] }); + const claim = tryAcquireProcessLock({ + lockDirPath: paths.lockPath, + owner: { ...readCurrentOwnerIdentity(), acquiredAtMs: Date.now() }, + }); + assert.equal(claim.status, 'acquired'); + if (claim.status !== 'acquired') throw new Error('fixture claim refused'); + let released = false; + spawn.mockImplementation((_command, _args, options) => { + const child = spawnRegisteredDaemonFixture(paths, fields(http.port), options, undefined, { + emitStartupMarkerBeforeAcquisition: true, + }); + if (spawn.mock.calls.length === 1) { + void child.exited.then((exit) => { + assert.equal(exit.exitCode, DAEMON_STARTUP_EXIT_CODES.busy); + if (!released) { + void claim.acquisition.release().then(() => { + released = true; + }); + } + }); + return child; + } + throw new Error('no second launch: the retry boundary must be observable without a sweeper'); + }); + pause.mockImplementation((ms) => actualRetry.sleep(Math.min(Number(ms), 50))); + try { + await assert.rejects( + sendToDaemon(request(paths)), + (error: unknown) => + error instanceof AppError && error.details?.kind === 'daemon_startup_failed', + ); + assert.deepEqual( + fs.readdirSync(paths.baseDir).filter((e) => e.startsWith('daemon-startup-phases')), + [], + "the retry result must not leave the joined busy contender's record behind", + ); + assert.equal(released, true, 'the contender must have exited busy against the held claim'); + } finally { + if (!released) await claim.acquisition.release(); + await closeLoopbackServer(http.server); + } +}); + test('a joined busy contender waits for a published winner to become ready without signaling it', async (t) => { + // The contender records its pre-lock phase through the real protocol; the handoff must sweep it. if (!(await supportsLoopbackBind())) return t.skip('loopback unavailable'); const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-delayed-ready-')); let probes = 0; @@ -395,7 +444,9 @@ test('a joined busy contender waits for a published winner to become ready witho await awaitFile(path.join(paths.baseDir, 'registration-held')); let contenderExit: ExecDetachedExit | undefined; spawn.mockImplementation((_command, _args, options) => { - const child = spawnRegisteredDaemonFixture(paths, fields(http.port), options); + const child = spawnRegisteredDaemonFixture(paths, fields(http.port), options, undefined, { + emitStartupMarkerBeforeAcquisition: true, + }); void child.exited.then((exit) => { contenderExit = exit; }); @@ -417,6 +468,11 @@ test('a joined busy contender waits for a published winner to become ready witho signal.mock.calls.some(([pid, kind]) => pid === winner.pid && kind !== 0), false, ); + assert.deepEqual( + fs.readdirSync(paths.baseDir).filter((e) => e.startsWith('daemon-startup-phases')), + [], + "the winner handoff must not leave the exited contender's record behind", + ); } finally { signal.mockRestore(); await closeLoopbackServer(http.server); @@ -426,6 +482,7 @@ test('a joined busy contender waits for a published winner to become ready witho for (const held of [true, false]) { test(`startup uses one deadline when the claim is ${held ? 'held' : 'released for relaunch'}`, async () => { const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-budget-')); + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '15000'); const claim = tryAcquireProcessLock({ lockDirPath: paths.lockPath, owner: { ...readCurrentOwnerIdentity(), acquiredAtMs: Date.now() }, @@ -484,6 +541,8 @@ test.for([ ])('an older winner is replaced only with enough startup time ($budget)', async (expected, t) => { if (!(await supportsLoopbackBind())) return t.skip('loopback unavailable'); const paths = resolveDaemonPaths(mkdtempForTestSync('daemon-start-older-winner-')); + // The replacement-window behavior is budget-independent; this pins the original 15 s shape. + vi.stubEnv('AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS', '15000'); const http = await startHttpDaemonFixture({ devices: [] }); const deferred = path.join(paths.baseDir, 'defer-publication'); fs.writeFileSync(deferred, 'wait'); diff --git a/src/daemon-client/daemon-client-attach.ts b/src/daemon-client/daemon-client-attach.ts index 9597a83f1a..975c80eb95 100644 --- a/src/daemon-client/daemon-client-attach.ts +++ b/src/daemon-client/daemon-client-attach.ts @@ -1,10 +1,6 @@ -import { - assertDaemonPolicyMatches, - ensureManagedDaemon, - observeLocalDaemon, - type DaemonClientSettings, - type EnsuredDaemon, -} from './daemon-client-lifecycle.ts'; +import { ensureManagedDaemon } from './daemon-client-lifecycle.ts'; +import { assertDaemonPolicyMatches, observeLocalDaemon } from './daemon-client-startup.ts'; +import type { DaemonClientSettings, EnsuredDaemon } from './daemon-client-lifecycle.ts'; import { daemonIncompatibleError, daemonNotRunningError } from './daemon-lifecycle-errors.ts'; /** diff --git a/src/daemon-client/daemon-client-ensure.ts b/src/daemon-client/daemon-client-ensure.ts index d2d6696b81..3db48b3902 100644 --- a/src/daemon-client/daemon-client-ensure.ts +++ b/src/daemon-client/daemon-client-ensure.ts @@ -9,16 +9,15 @@ import { waitForDaemonExit } from '@agent-device/daemon-contracts/daemon-process import { recoverAbandonedDaemonRegistration } from '@agent-device/daemon-contracts/daemon-registration-owner'; import { sleep } from '@agent-device/host-kit/retry'; import { AppError } from '@agent-device/kernel/errors'; +import { resolveLocalDaemonSettings } from './daemon-client-lifecycle.ts'; import { assertDaemonPolicyMatches, observeLocalDaemon, - resolveLocalDaemonSettings, retireDaemonForTakeover, - stopDaemonForTakeover, startLocalDaemon, - type DaemonClientSettings, - type EnsuredDaemon, -} from './daemon-client-lifecycle.ts'; + stopDaemonForTakeover, +} from './daemon-client-startup.ts'; +import type { DaemonClientSettings, EnsuredDaemon } from './daemon-client-lifecycle.ts'; import { requestDaemonQuiesce } from './daemon-client-quiesce.ts'; import { probeDaemonResponsive } from './daemon-client-liveness-probe.ts'; import type { DaemonInfo } from './daemon-client-metadata.ts'; diff --git a/src/daemon-client/daemon-client-host.test.ts b/src/daemon-client/daemon-client-host.test.ts index 3e3f7578c4..0902267605 100644 --- a/src/daemon-client/daemon-client-host.test.ts +++ b/src/daemon-client/daemon-client-host.test.ts @@ -9,6 +9,9 @@ vi.mock('./daemon-client-lifecycle.ts', async (importOriginal) => ({ ensured.settings = settings; throw new Error('stop before talking to a daemon'); }, +})); +vi.mock('./daemon-client-startup.ts', async (importOriginal) => ({ + ...(await importOriginal()), observeLocalDaemon: async (settings: unknown) => { ensured.settings = settings; throw new Error('stop before talking to a daemon'); diff --git a/src/daemon-client/daemon-client-lifecycle.ts b/src/daemon-client/daemon-client-lifecycle.ts index 47ccda9d53..a30fd19bcc 100644 --- a/src/daemon-client/daemon-client-lifecycle.ts +++ b/src/daemon-client/daemon-client-lifecycle.ts @@ -1,34 +1,15 @@ import type { DaemonWireRequest } from '@agent-device/contracts/command'; import type { DaemonLifecycle, EnsureDaemonOptions } from '@agent-device/contracts/client'; import type { DaemonResponse } from '@agent-device/kernel/contracts'; -import { - daemonContendedError, - daemonRetirementUnconfirmedError, - daemonStopCommand, -} from './daemon-lifecycle-errors.ts'; -import { attachActiveSessionAddressHint } from './daemon-client-address-hints.ts'; -import fs from 'node:fs'; -import { processLockHoldsDaemonIdentity } from '@agent-device/daemon-contracts/daemon-registration'; import net from 'node:net'; import { AppError, normalizeError, type NormalizedError } from '@agent-device/kernel/errors'; -import { readReplayDivergenceResume } from '@agent-device/ad-replay/divergence'; -import { type ExecDetachedExit } from '@agent-device/host-kit/command'; import type { EnvMap, HostEnvironment } from '@agent-device/kernel/source-value'; +import { readReplayDivergenceResume } from '@agent-device/ad-replay/divergence'; import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; -import { isProcessAlive } from '@agent-device/host-kit/process'; -import { sleep } from '@agent-device/host-kit/retry'; -import { inspectProcessLock, type ProcessLockInspection } from '@agent-device/host-kit/file'; - -import type { findUnrecoveredRepairCommitFailure } from '@agent-device/daemon-contracts/session-repair-tombstone'; import { - DAEMON_STARTUP_EXIT_CODES, createOwnedReplayStateDir, - recoverAbandonedDaemonRegistration, - type DaemonRetirementResult, - launchDaemonProcess, stopAndRetireDaemon, type OwnedReplayStateDir, - type DaemonStartupLaunch, } from '@agent-device/daemon-contracts/daemon-registration-owner'; import { resolveDaemonPaths, @@ -38,25 +19,18 @@ import { type DaemonServerMode, type DaemonTransportPreference, } from '@agent-device/daemon-contracts/daemon-resolution'; -import { - resolveDaemonLaunchSpec, - resolveDaemonTakeover, - type DaemonTakeoverDecision, -} from './daemon-launch-spec.ts'; import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog'; +import type { findUnrecoveredRepairCommitFailure } from '@agent-device/daemon-contracts/session-repair-tombstone'; +import type { DaemonInfo } from './daemon-client-metadata.ts'; +import { attachActiveSessionAddressHint } from './daemon-client-address-hints.ts'; +import { isRemoteDaemon } from './daemon-client-metadata.ts'; +import { cachedRemoteDaemonHealth } from './daemon-client-transport.ts'; import { - getDaemonMetadataState, - isRemoteDaemon, - readDaemonInfo, - resolveDaemonStartupHint, - type DaemonInfo, -} from './daemon-client-metadata.ts'; -import { - canConnect, - cachedRemoteDaemonHealth, - isDaemonTransportUnavailableError, -} from './daemon-client-transport.ts'; + assertDaemonPolicyMatches, + readReusableLocalDaemon, + startLocalDaemon, +} from './daemon-client-startup.ts'; export type DaemonClientSettings = { /** The environment and working directory this client reads, for every setting below and the launch. */ @@ -81,18 +55,6 @@ export type EnsuredDaemon = { startedByClient: boolean; }; -type DaemonStartupWaitResult = - | { kind: 'ready'; daemon: EnsuredDaemon } - | { kind: 'early_exit'; exit: ExecDetachedExit } - | { kind: 'unproven'; error: AppError } - | { kind: 'retry' | 'timeout' }; - -const DAEMON_STARTUP_TIMEOUT_MS = 15_000; -const MINIMUM_DAEMON_TAKEOVER_BUDGET_MS = 5_000; -const LIVE_DAEMON_PROBE_RETRIES = 3; -const LIVE_DAEMON_PROBE_RETRY_DELAY_MS = 200; -const DAEMON_STARTUP_ATTEMPTS = 2; -const DAEMON_STARTUP_LOG_TAIL_BYTES = 64_000; const LOOPBACK_BLOCK_LIST = new net.BlockList(); LOOPBACK_BLOCK_LIST.addSubnet('127.0.0.0', 8, 'ipv4'); LOOPBACK_BLOCK_LIST.addAddress('::1', 'ipv6'); @@ -266,379 +228,6 @@ async function ensureRemoteDaemon(settings: DaemonClientSettings): Promise { - const { deadline } = options; - const inspection = inspectProcessLock(settings.paths.lockPath); - if (inspection.state === 'unproven') throw daemonRegistrationUnprovenError(settings, inspection); - const existing = readDaemonInfo(settings.paths.infoPath); - if (!existing) return null; - if (!registrationAllowsDaemonObservation(inspection, existing)) return null; - if (options.waitForLiveStartup && (await isAwaitingDaemonStartup(existing, deadline))) - return null; - - const decision = await resolveDaemonTakeover(existing, { - onClientTransport: () => - canReachReusableDaemon(existing, settings.transportPreference, deadline), - onAnyAdvertisedTransport: () => canReachReusableDaemon(existing, 'auto', deadline), - }); - return { info: existing, decision: await refineReplacement(existing, decision, options) }; -} - -/** - * Tells apart what the ladder lumps together, for callers that must not kill a daemon on a guess: a - * registration whose process is gone is unreachable whatever its version, and one that answers on - * another transport is alive. - */ -async function refineReplacement( - info: DaemonInfo, - decision: DaemonTakeoverDecision, - options: { deadline?: number; refineUnreachable?: boolean }, -): Promise { - if (!options.refineUnreachable || decision.kind !== 'replace') return decision; - if (!isProcessAlive(info.pid)) return { ...decision, cause: 'unreachable' }; - if ( - decision.cause === 'unreachable' && - (await canReachReusableDaemon(info, 'auto', options.deadline)) - ) { - return { ...decision, cause: 'transport' }; - } - return decision; -} - -async function readReusableLocalDaemon( - settings: DaemonClientSettings, - options: { deadline?: number; waitForLiveStartup?: boolean } = {}, -): Promise { - const { deadline } = options; - const observed = await observeLocalDaemon(settings, options); - if (!observed) return null; - const { info: existing, decision } = observed; - if (decision.kind === 'reuse') return existing; - if (decision.kind === 'refuseNewer') { - throw newerDaemonRefusedError(existing, decision, settings.paths.baseDir); - } - - if (!settings.retireIncompatibleContender) throw daemonContendedError(); - if (remainingStartupBudget(deadline) < MINIMUM_DAEMON_TAKEOVER_BUDGET_MS) return null; - emitDaemonTakeoverNotice(existing, decision.reason, settings.paths.baseDir); - await retireDaemonForTakeover(existing, settings.paths); - return null; -} - -async function isAwaitingDaemonStartup( - info: DaemonInfo, - deadline: number | undefined, -): Promise { - return ( - isProcessAlive(info.pid) && !(await canConnect(info, 'auto', remainingStartupBudget(deadline))) - ); -} - -function registrationAllowsDaemonObservation( - inspection: ProcessLockInspection, - info: DaemonInfo, -): boolean { - return ( - inspection.state === 'absent' || - processLockHoldsDaemonIdentity(inspection, { - pid: info.pid, - startTime: info.processStartTime ?? null, - }) - ); -} - -function daemonRegistrationUnprovenError( - settings: DaemonClientSettings, - inspection?: ProcessLockInspection, -): AppError { - return new AppError('COMMAND_FAILED', 'Daemon registration ownership could not be verified.', { - reason: 'daemon_registration_unproven', - inspection, - stateDir: settings.paths.baseDir, - hint: resolveDaemonStartupHint(getDaemonMetadataState(settings.paths), settings.paths), - }); -} - -export async function stopDaemonForTakeover( - existing: DaemonInfo, - paths: DaemonPaths, - mode: 'graceful' | 'force' = 'graceful', -): Promise { - return await stopAndRetireDaemon({ - paths: paths, - observed: { pid: existing.pid, startTime: existing.processStartTime ?? null }, - mode, - }); -} - -export async function retireDaemonForTakeover( - existing: DaemonInfo, - paths: DaemonPaths, - mode: 'graceful' | 'force' = 'graceful', -): Promise { - const retirement = await stopDaemonForTakeover(existing, paths, mode); - if (retirement.status === 'retained') throw daemonRetirementUnconfirmedError(retirement, paths); -} - -/** - * A daemon whose pid is still alive is probed again before it can be judged unreachable. A probe's - * budget is wall-clock time on this client's event loop, so a client that stalls past it (a large - * synchronous parse, a GC pause on a loaded host) reads a listening daemon as unreachable, and - * replacing it ends every session the daemon holds. Liveness is the signal-0 check, not the `ps` - * identity read: under the load that stalls the probe, `ps` misses its deadline too, and the - * takeover still proves identity before it signals anything. - */ -async function canReachReusableDaemon( - info: DaemonInfo, - preference: DaemonTransportPreference, - deadline?: number, -): Promise { - if (await canConnectReusableDaemon(info, preference, deadline)) return true; - for (let retry = 1; retry <= LIVE_DAEMON_PROBE_RETRIES; retry += 1) { - if (!isProcessAlive(info.pid) || (deadline !== undefined && Date.now() >= deadline)) - return false; - await sleep(Math.min(LIVE_DAEMON_PROBE_RETRY_DELAY_MS, remainingStartupBudget(deadline))); - if (await canConnectReusableDaemon(info, preference, deadline)) { - emitDiagnostic({ - level: 'warn', - phase: 'daemon_probe_recovered', - data: { pid: info.pid, retry }, - }); - return true; - } - } - return false; -} - -/** - * ADR 0029: a caller that names a daemon policy must not silently use a daemon that enforces a - * different one (or none). A caller that names no policy uses whatever the daemon enforces. - */ -export async function assertDaemonPolicyMatches( - existing: DaemonInfo, - stateDir: string, - host: HostEnvironment, -): Promise { - const { env } = host; - if (!env.AGENT_DEVICE_DAEMON_POLICY?.trim()) return; - const { loadDaemonPolicy } = await import('@agent-device/daemon-contracts/daemon-policy-file'); - const expected = loadDaemonPolicy(env, host.cwd)?.digest; - if (expected === existing.policyDigest) return; - throw new AppError( - 'COMMAND_FAILED', - 'The running daemon does not enforce the daemon policy named by AGENT_DEVICE_DAEMON_POLICY.', - { - reason: 'DAEMON_POLICY_MISMATCH', - expectedPolicyDigest: expected, - daemonPolicyDigest: existing.policyDigest ?? null, - hint: `Stop the running daemon (${daemonStopCommand(stateDir)}), then retry so a daemon starts with this policy.`, - }, - ); -} - -async function canConnectReusableDaemon( - info: DaemonInfo, - preference: DaemonTransportPreference, - deadline?: number, -): Promise { - try { - return await canConnect(info, preference, remainingStartupBudget(deadline)); - } catch (error) { - if (isDaemonTransportUnavailableError(error)) return false; - throw error; - } -} - -function newerDaemonRefusedError( - info: DaemonInfo, - decision: Extract, - stateDir: string, -): AppError { - const { daemonVersion, clientVersion } = decision; - return new AppError( - 'COMMAND_FAILED', - `Daemon (pid ${info.pid}, v${daemonVersion}) is newer than this client (v${clientVersion}); refusing to replace it.`, - { - daemonPid: info.pid, - daemonVersion, - clientVersion, - hint: `Use the agent-device v${daemonVersion} CLI that started it, or stop it deliberately: ${daemonStopCommand(stateDir)}`, - }, - ); -} - -function emitDaemonTakeoverNotice(info: DaemonInfo, reason: string, stateDir: string): void { - try { - const identity = info.version ? `pid ${info.pid}, v${info.version}` : `pid ${info.pid}`; - process.stderr.write(`Replacing daemon (${identity}) in ${stateDir}: ${reason}\n`); - } catch { - // The takeover notice is best effort; never fail the command on stderr issues. - } -} - -type FailedDaemonStartup = { - cleanup?: DaemonRetirementResult; - startError?: string; - startupError?: NormalizedError; - daemonProcess?: ExecDetachedExit | { pid: number }; - retry: boolean; -}; - -export async function startLocalDaemon(settings: DaemonClientSettings): Promise { - const deadline = Date.now() + DAEMON_STARTUP_TIMEOUT_MS; - const cleanupResults: DaemonRetirementResult[] = []; - let failure: FailedDaemonStartup | undefined; - let attempts = 0; - while (attempts < DAEMON_STARTUP_ATTEMPTS && Date.now() < deadline) { - attempts += 1; - const result = await attemptLocalDaemonStartup(settings, deadline); - if ('daemon' in result) return result.daemon; - failure = result; - if (result.cleanup) cleanupResults.push(result.cleanup); - if (!result.retry) break; - await sleep(Math.min(150, Math.max(0, deadline - Date.now()))); - } - const { startError, startupError, daemonProcess }: Partial = failure ?? {}; - const state = getDaemonMetadataState(settings.paths); - const daemonLogTail = readRecentLogTail(settings.paths.logPath); - throw new AppError('COMMAND_FAILED', 'Failed to start daemon', { - kind: 'daemon_startup_failed', - stateDir: settings.paths.baseDir, - infoPath: settings.paths.infoPath, - lockPath: settings.paths.lockPath, - logPath: settings.paths.logPath, - startupTimeoutMs: DAEMON_STARTUP_TIMEOUT_MS, - startupAttempts: attempts, - cleanupResults, - startError, - startupError, - daemonProcess, - ...(daemonLogTail ? { daemonLogTail } : {}), - metadataState: state, - hint: resolveDaemonStartupHint(state, settings.paths), - }); -} - -async function attemptLocalDaemonStartup( - settings: DaemonClientSettings, - deadline: number, -): Promise<{ daemon: EnsuredDaemon } | FailedDaemonStartup> { - let launch: DaemonStartupLaunch; - try { - launch = startDaemon(settings); - } catch (error) { - return await recoverFailedDaemonLaunch(settings, error); - } - const startup = await waitForDaemonStartup(deadline, settings, launch); - if (startup.kind === 'ready') return { daemon: startup.daemon }; - if (startup.kind === 'retry') return { retry: true }; - if (startup.kind === 'unproven') { - const startupError = normalizeError(startup.error); - return { - retry: false, - startError: startupError.message, - startupError, - daemonProcess: { pid: launch.pid }, - }; - } - const { cleanup, joined } = await retireStartupAttempt(settings, launch, deadline); - const available = isRegistrationAvailable(inspectProcessLock(settings.paths.lockPath)); - return { - cleanup, - retry: joined && startup.kind === 'early_exit' && available, - startError: startup.kind === 'early_exit' ? describeDaemonEarlyExit(startup.exit) : undefined, - daemonProcess: startup.kind === 'early_exit' ? startup.exit : { pid: launch.pid }, - }; -} - -async function recoverFailedDaemonLaunch( - settings: DaemonClientSettings, - error: unknown, -): Promise { - const cleanup = settings.ownedStateDir - ? await stopAndRetireDaemon({ - paths: settings.paths, - observed: null, - mode: 'graceful', - ownedStateDir: settings.ownedStateDir, - lockTimeoutMs: 0, - }) - : await recoverAbandonedDaemonRegistration({ - paths: settings.paths, - observed: null, - lockTimeoutMs: 0, - }); - return { - cleanup, - startError: normalizeError(error).message, - retry: !settings.ownedStateDir && cleanup.status !== 'retained', - }; -} - -async function retireStartupAttempt( - settings: DaemonClientSettings, - launch: DaemonStartupLaunch, - deadline: number, -): Promise<{ cleanup: DaemonRetirementResult; joined: boolean }> { - const cleanup = await stopAndRetireDaemon({ - paths: settings.paths, - observed: launch.readIdentity(), - mode: 'graceful', - ownedStateDir: settings.ownedStateDir, - termTimeoutMs: Math.min(3_000, remainingStartupBudget(deadline)), - killTimeoutMs: 1_000, - lockTimeoutMs: 0, - }); - const joined = await joinStartup(launch); - return { cleanup, joined }; -} - -async function joinStartup(launch: DaemonStartupLaunch): Promise { - let timer: ReturnType | undefined; - try { - return await Promise.race([ - launch.exited.then(() => true), - new Promise((resolve) => { - timer = setTimeout(() => resolve(false), 1_000); - }), - ]); - } finally { - clearTimeout(timer); - } -} - -function remainingStartupBudget(deadline?: number): number { - return deadline === undefined ? Number.POSITIVE_INFINITY : Math.max(0, deadline - Date.now()); -} - -function isRegistrationAvailable(inspection: ProcessLockInspection): boolean { - return ( - inspection.state === 'absent' || - (inspection.state === 'held' && - (inspection.liveness === 'owner-process-dead' || - inspection.liveness === 'owner-process-reused')) - ); -} - /** * ADR 0012 decision 6 (BLOCKER 2, third follow-up): a one-shot repair * (`replay --save-script`) that COMPLETES without diverging returns SUCCESS @@ -842,149 +431,6 @@ function isActiveReplaySessionResponse( return response.data?.sessionActive === true; } -async function waitForDaemonStartup( - deadline: number, - settings: DaemonClientSettings, - launch: DaemonStartupLaunch, -): Promise { - let earlyExit: ExecDetachedExit | undefined; - void launch.exited.then((exit) => { - earlyExit = exit; - }); - while (Date.now() < deadline) { - if (earlyExit) { - const kind = classifyDaemonStartupExit(earlyExit); - if (kind === 'unproven') - return { kind: 'unproven', error: daemonRegistrationUnprovenError(settings) }; - if (kind === 'failed') return { kind: 'early_exit', exit: earlyExit }; - const contender = await observeContendingDaemon(settings, deadline); - if (contender) return contender; - } else { - const info = await readReadyLaunchedDaemon(settings, launch, deadline); - if (info && !earlyExit) return { kind: 'ready', daemon: { info, startedByClient: true } }; - } - await sleep(Math.min(100, remainingStartupBudget(deadline))); - } - return { kind: 'timeout' }; -} - -function classifyDaemonStartupExit(exit: ExecDetachedExit): 'busy' | 'unproven' | 'failed' { - if (exit.error || exit.signal) return 'failed'; - switch (exit.exitCode) { - case DAEMON_STARTUP_EXIT_CODES.busy: - return 'busy'; - case DAEMON_STARTUP_EXIT_CODES.unproven: - return 'unproven'; - default: - return 'failed'; - } -} - -async function observeContendingDaemon( - settings: DaemonClientSettings, - deadline: number, -): Promise { - let winner: DaemonInfo | null; - try { - winner = await readReusableLocalDaemon(settings, { deadline, waitForLiveStartup: true }); - } catch (error) { - if (error instanceof AppError && error.details?.reason === 'daemon_registration_unproven') - return { kind: 'unproven', error }; - throw error; - } - if (Date.now() >= deadline) return null; - if (winner) return { kind: 'ready', daemon: { info: winner, startedByClient: false } }; - const inspection = inspectProcessLock(settings.paths.lockPath); - if (inspection.state === 'unproven') - return { kind: 'unproven', error: daemonRegistrationUnprovenError(settings, inspection) }; - return isRegistrationAvailable(inspection) ? { kind: 'retry' } : null; -} - -async function readReadyLaunchedDaemon( - settings: DaemonClientSettings, - launch: DaemonStartupLaunch, - deadline: number, -): Promise { - const info = readDaemonInfo(settings.paths.infoPath); - if (!info || !isLaunchedDaemon(info, launch)) return null; - try { - return (await canConnect( - info, - settings.transportPreference, - remainingStartupBudget(deadline), - )) && Date.now() < deadline - ? info - : null; - } catch (error) { - const { cleanup, joined } = await retireStartupAttempt(settings, launch, deadline); - emitDiagnostic({ - level: 'warn', - phase: 'daemon_startup_observation_failed', - data: { stateDir: settings.paths.baseDir, cleanup, joined, error: normalizeError(error) }, - }); - if (error instanceof AppError) { - error.details = { - ...error.details, - stateDir: settings.paths.baseDir, - cleanupResults: [cleanup], - startupJoined: joined, - }; - } - throw error; - } -} - -/** Whether `info` names the daemon process this client launched: same pid and start time. */ -function isLaunchedDaemon(info: DaemonInfo, launch: DaemonStartupLaunch): boolean { - const identity = launch.readIdentity(); - return ( - info.pid === identity.pid && - identity.startTime !== null && - info.processStartTime === identity.startTime - ); -} - -function startDaemon(settings: DaemonClientSettings): DaemonStartupLaunch { - const launchSpec = resolveDaemonLaunchSpec(); - return launchDaemonProcess({ - paths: settings.paths, - serverMode: settings.serverMode, - env: settings.host.env, - cwd: settings.host.cwd, - ownedStateDir: settings.ownedStateDir, - args: launchSpec.useSrc - ? ['--experimental-strip-types', launchSpec.srcPath] - : [launchSpec.distPath], - }); -} - -function describeDaemonEarlyExit(exit: ExecDetachedExit): string { - if (exit.error) return `daemon process ${exit.pid} failed to start: ${exit.error}`; - if (exit.signal) - return `daemon process ${exit.pid} exited before readiness with signal ${exit.signal}`; - return `daemon process ${exit.pid} exited before readiness with code ${exit.exitCode ?? 0}`; -} - -function readRecentLogTail(logPath: string): string | undefined { - try { - if (!fs.existsSync(logPath)) return undefined; - const stats = fs.statSync(logPath); - if (stats.size <= 0) return undefined; - const length = Math.min(stats.size, DAEMON_STARTUP_LOG_TAIL_BYTES); - const fd = fs.openSync(logPath, 'r'); - try { - const buffer = Buffer.alloc(length); - fs.readSync(fd, buffer, 0, length, stats.size - length); - const text = buffer.toString('utf8').trim(); - return text.length > 0 ? text : undefined; - } finally { - fs.closeSync(fd); - } - } catch { - return undefined; - } -} - function resolveRemoteDaemonBaseUrl(raw: string | undefined): string | undefined { if (!raw) return undefined; let parsed: URL; diff --git a/src/daemon-client/daemon-client-startup.ts b/src/daemon-client/daemon-client-startup.ts new file mode 100644 index 0000000000..b02220d85c --- /dev/null +++ b/src/daemon-client/daemon-client-startup.ts @@ -0,0 +1,739 @@ +import fs from 'node:fs'; +import { AppError, normalizeError, type NormalizedError } from '@agent-device/kernel/errors'; +import type { EnvMap, HostEnvironment } from '@agent-device/kernel/source-value'; +import { processLockHoldsDaemonIdentity } from '@agent-device/daemon-contracts/daemon-registration'; +import { inspectProcessLock, type ProcessLockInspection } from '@agent-device/host-kit/file'; +import { classifyOwnerLiveness, isProcessAlive } from '@agent-device/host-kit/process'; +import { sleep } from '@agent-device/host-kit/retry'; +import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; +import { type ExecDetachedExit } from '@agent-device/host-kit/command'; +import { + DAEMON_STARTUP_EXIT_CODES, + recoverAbandonedDaemonRegistration, + type DaemonRetirementResult, + launchDaemonProcess, + stopAndRetireDaemon, + type DaemonStartupLaunch, +} from '@agent-device/daemon-contracts/daemon-registration-owner'; +import { + readDaemonStartupPhaseMarkers, + removeDaemonStartupPhasesFile, + resolveDaemonStartupPhasesPath, + type DaemonStartupPhase, +} from '@agent-device/daemon-contracts/daemon-startup-phases'; +import type { + DaemonPaths, + DaemonTransportPreference, +} from '@agent-device/daemon-contracts/daemon-resolution'; +import { + resolveDaemonLaunchSpec, + resolveDaemonTakeover, + type DaemonTakeoverDecision, +} from './daemon-launch-spec.ts'; +import { + getDaemonMetadataState, + readDaemonInfo, + resolveDaemonStartupHint, + type DaemonInfo, +} from './daemon-client-metadata.ts'; +import { canConnect, isDaemonTransportUnavailableError } from './daemon-client-transport.ts'; +import { + daemonContendedError, + daemonRetirementUnconfirmedError, + daemonStopCommand, +} from './daemon-lifecycle-errors.ts'; +import type { DaemonClientSettings, EnsuredDaemon } from './daemon-client-lifecycle.ts'; + +type LocalDaemonObservation = { + info: DaemonInfo; + decision: DaemonTakeoverDecision; +}; + +type DaemonStartupWaitResult = + | { kind: 'ready'; daemon: EnsuredDaemon } + | { kind: 'early_exit'; exit: ExecDetachedExit } + | { kind: 'unproven'; error: AppError } + | { kind: 'retry' } + | { kind: 'exit_unobserved' } + | { kind: 'stalled' } + | { kind: 'ceiling' }; + +type StartupPhaseObservation = { + spawnedAtMs: number; + markers: Partial>; +}; + +/** The ceiling on one local daemon startup; `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` overrides it. */ +const DAEMON_STARTUP_CEILING_MS = 120_000; +/** Silence this long ends the wait early; an env ceiling below it makes the ceiling the stop. */ +const DAEMON_STARTUP_STALL_WINDOW_MS = 60_000; +const MINIMUM_DAEMON_TAKEOVER_BUDGET_MS = 5_000; + +const DAEMON_STARTUP_TIMEOUT_ENV = 'AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS'; + +type DaemonStartupFailureReason = 'startup-exited' | 'startup-stalled' | 'startup-ceiling'; + +function resolveDaemonStartupTimeoutMs(env: Readonly): number { + const raw = env[DAEMON_STARTUP_TIMEOUT_ENV]?.trim(); + if (raw === undefined || raw === '') return DAEMON_STARTUP_CEILING_MS; + const parsed = Number(raw); + if (!Number.isSafeInteger(parsed) || parsed <= 0) { + throw new AppError( + 'INVALID_ARGS', + `${DAEMON_STARTUP_TIMEOUT_ENV} must be a positive integer number of milliseconds`, + { value: raw }, + ); + } + return parsed; +} + +const LIVE_DAEMON_PROBE_RETRIES = 3; +const LIVE_DAEMON_PROBE_RETRY_DELAY_MS = 200; +const DAEMON_STARTUP_ATTEMPTS = 2; +const DAEMON_STARTUP_LOG_TAIL_BYTES = 64_000; +const DAEMON_STARTUP_MARKER_TAIL_BYTES = 16_000; + +/** Judges the registered daemon against this client, changing nothing. */ +export async function observeLocalDaemon( + settings: DaemonClientSettings, + options: { + deadline?: number; + waitForLiveStartup?: boolean; + refineUnreachable?: boolean; + } = {}, +): Promise { + const { deadline } = options; + const inspection = inspectProcessLock(settings.paths.lockPath); + if (inspection.state === 'unproven') throw daemonRegistrationUnprovenError(settings, inspection); + const existing = readDaemonInfo(settings.paths.infoPath); + if (!existing) return null; + if (!registrationAllowsDaemonObservation(inspection, existing)) return null; + if (options.waitForLiveStartup && (await isAwaitingDaemonStartup(existing, deadline))) + return null; + + const decision = await resolveDaemonTakeover(existing, { + onClientTransport: () => + canReachReusableDaemon(existing, settings.transportPreference, deadline), + onAnyAdvertisedTransport: () => canReachReusableDaemon(existing, 'auto', deadline), + }); + return { info: existing, decision: await refineReplacement(existing, decision, options) }; +} + +async function refineReplacement( + info: DaemonInfo, + decision: DaemonTakeoverDecision, + options: { deadline?: number; refineUnreachable?: boolean }, +): Promise { + if (!options.refineUnreachable || decision.kind !== 'replace') return decision; + if (!isProcessAlive(info.pid)) return { ...decision, cause: 'unreachable' }; + if ( + decision.cause === 'unreachable' && + (await canReachReusableDaemon(info, 'auto', options.deadline)) + ) { + return { ...decision, cause: 'transport' }; + } + return decision; +} + +export async function readReusableLocalDaemon( + settings: DaemonClientSettings, + options: { deadline?: number; waitForLiveStartup?: boolean } = {}, +): Promise { + const { deadline } = options; + const observed = await observeLocalDaemon(settings, options); + if (!observed) return null; + const { info: existing, decision } = observed; + if (decision.kind === 'reuse') return existing; + if (decision.kind === 'refuseNewer') { + throw newerDaemonRefusedError(existing, decision, settings.paths.baseDir); + } + + if (!settings.retireIncompatibleContender) throw daemonContendedError(); + if (remainingStartupBudget(deadline) < MINIMUM_DAEMON_TAKEOVER_BUDGET_MS) return null; + emitDaemonTakeoverNotice(existing, decision.reason, settings.paths.baseDir); + await retireDaemonForTakeover(existing, settings.paths); + return null; +} + +async function isAwaitingDaemonStartup( + info: DaemonInfo, + deadline: number | undefined, +): Promise { + return ( + isProcessAlive(info.pid) && !(await canConnect(info, 'auto', remainingStartupBudget(deadline))) + ); +} + +function registrationAllowsDaemonObservation( + inspection: ProcessLockInspection, + info: DaemonInfo, +): boolean { + return ( + inspection.state === 'absent' || + processLockHoldsDaemonIdentity(inspection, { + pid: info.pid, + startTime: info.processStartTime ?? null, + }) + ); +} + +function daemonRegistrationUnprovenError( + settings: DaemonClientSettings, + inspection?: ProcessLockInspection, +): AppError { + return new AppError('COMMAND_FAILED', 'Daemon registration ownership could not be verified.', { + reason: 'daemon_registration_unproven', + inspection, + stateDir: settings.paths.baseDir, + hint: resolveDaemonStartupHint(getDaemonMetadataState(settings.paths), settings.paths), + }); +} + +export async function stopDaemonForTakeover( + existing: DaemonInfo, + paths: DaemonPaths, + mode: 'graceful' | 'force' = 'graceful', +): Promise { + return await stopAndRetireDaemon({ + paths: paths, + observed: { pid: existing.pid, startTime: existing.processStartTime ?? null }, + mode, + }); +} + +export async function retireDaemonForTakeover( + existing: DaemonInfo, + paths: DaemonPaths, + mode: 'graceful' | 'force' = 'graceful', +): Promise { + const retirement = await stopDaemonForTakeover(existing, paths, mode); + if (retirement.status === 'retained') throw daemonRetirementUnconfirmedError(retirement, paths); +} + +/** A live pid is probed again before being judged unreachable; liveness is signal 0, not `ps`. */ +async function canReachReusableDaemon( + info: DaemonInfo, + preference: DaemonTransportPreference, + deadline?: number, +): Promise { + if (await canConnectReusableDaemon(info, preference, deadline)) return true; + for (let retry = 1; retry <= LIVE_DAEMON_PROBE_RETRIES; retry += 1) { + if (!isProcessAlive(info.pid) || (deadline !== undefined && Date.now() >= deadline)) + return false; + await sleep(Math.min(LIVE_DAEMON_PROBE_RETRY_DELAY_MS, remainingStartupBudget(deadline))); + if (await canConnectReusableDaemon(info, preference, deadline)) { + emitDiagnostic({ + level: 'warn', + phase: 'daemon_probe_recovered', + data: { pid: info.pid, retry }, + }); + return true; + } + } + return false; +} + +/** + * ADR 0029: a caller that names a daemon policy must not silently use a daemon that enforces a + * different one (or none). A caller that names no policy uses whatever the daemon enforces. + */ +export async function assertDaemonPolicyMatches( + existing: DaemonInfo, + stateDir: string, + host: HostEnvironment, +): Promise { + const { env } = host; + if (!env.AGENT_DEVICE_DAEMON_POLICY?.trim()) return; + const { loadDaemonPolicy } = await import('@agent-device/daemon-contracts/daemon-policy-file'); + const expected = loadDaemonPolicy(env, host.cwd)?.digest; + if (expected === existing.policyDigest) return; + throw new AppError( + 'COMMAND_FAILED', + 'The running daemon does not enforce the daemon policy named by AGENT_DEVICE_DAEMON_POLICY.', + { + reason: 'DAEMON_POLICY_MISMATCH', + expectedPolicyDigest: expected, + daemonPolicyDigest: existing.policyDigest ?? null, + hint: `Stop the running daemon (${daemonStopCommand(stateDir)}), then retry so a daemon starts with this policy.`, + }, + ); +} + +async function canConnectReusableDaemon( + info: DaemonInfo, + preference: DaemonTransportPreference, + deadline?: number, +): Promise { + try { + return await canConnect(info, preference, remainingStartupBudget(deadline)); + } catch (error) { + if (isDaemonTransportUnavailableError(error)) return false; + throw error; + } +} + +function newerDaemonRefusedError( + info: DaemonInfo, + decision: Extract, + stateDir: string, +): AppError { + const { daemonVersion, clientVersion } = decision; + return new AppError( + 'COMMAND_FAILED', + `Daemon (pid ${info.pid}, v${daemonVersion}) is newer than this client (v${clientVersion}); refusing to replace it.`, + { + daemonPid: info.pid, + daemonVersion, + clientVersion, + hint: `Use the agent-device v${daemonVersion} CLI that started it, or stop it deliberately: ${daemonStopCommand(stateDir)}`, + }, + ); +} + +function emitDaemonTakeoverNotice(info: DaemonInfo, reason: string, stateDir: string): void { + try { + const identity = info.version ? `pid ${info.pid}, v${info.version}` : `pid ${info.pid}`; + process.stderr.write(`Replacing daemon (${identity}) in ${stateDir}: ${reason}\n`); + } catch {} +} + +type FailedDaemonStartup = { + cleanup?: DaemonRetirementResult; + startError?: string; + startupError?: NormalizedError; + daemonProcess?: ExecDetachedExit | { pid: number }; + retry: boolean; + reason?: DaemonStartupFailureReason; + phases?: StartupPhaseObservation; +}; + +export async function startLocalDaemon(settings: DaemonClientSettings): Promise { + const startupTimeoutMs = resolveDaemonStartupTimeoutMs(settings.host.env); + const deadline = Date.now() + startupTimeoutMs; + const cleanupResults: DaemonRetirementResult[] = []; + let failure: FailedDaemonStartup | undefined; + let attempts = 0; + while (attempts < DAEMON_STARTUP_ATTEMPTS && Date.now() < deadline) { + attempts += 1; + const result = await attemptLocalDaemonStartup(settings, deadline); + if ('daemon' in result) return result.daemon; + failure = result; + if (result.cleanup) cleanupResults.push(result.cleanup); + if (!result.retry) break; + await sleep(Math.min(150, Math.max(0, deadline - Date.now()))); + } + const { startError, startupError, daemonProcess, reason, phases }: Partial = + failure ?? {}; + const effectiveReason: DaemonStartupFailureReason = reason ?? 'startup-exited'; + const state = getDaemonMetadataState(settings.paths); + const daemonLogTail = readRecentLogTail(settings.paths.logPath); + throw new AppError('COMMAND_FAILED', 'Failed to start daemon', { + kind: 'daemon_startup_failed', + reason: effectiveReason, + stateDir: settings.paths.baseDir, + infoPath: settings.paths.infoPath, + lockPath: settings.paths.lockPath, + logPath: settings.paths.logPath, + startupTimeoutMs, + startupAttempts: attempts, + cleanupResults, + startError, + startupError, + daemonProcess, + ...(phases ? { phases: describeStartupPhaseTimings(phases) } : {}), + ...(daemonLogTail ? { daemonLogTail } : {}), + metadataState: state, + hint: + reason === 'startup-stalled' || reason === 'startup-ceiling' + ? resolveStartupProgressHint(reason, startupTimeoutMs) + : resolveDaemonStartupHint(state, settings.paths), + }); +} + +async function attemptLocalDaemonStartup( + settings: DaemonClientSettings, + deadline: number, +): Promise<{ daemon: EnsuredDaemon } | FailedDaemonStartup> { + let launch: DaemonStartupLaunch; + const spawnedAtMs = Date.now(); + try { + launch = startDaemon(settings); + } catch (error) { + return await recoverFailedDaemonLaunch(settings, error); + } + const observation: StartupPhaseObservation = { spawnedAtMs, markers: {} }; + const startup = await waitForDaemonStartup(deadline, settings, launch, observation); + if (startup.kind === 'ready') { + // Winner readiness (startedByClient false) is only reachable after this launch exited busy: + // its own record is this client's to sweep before the handoff. + if (!startup.daemon.startedByClient) await discardExitedLaunchRecord(settings, launch); + return { daemon: startup.daemon }; + } + refreshStartupPhaseMarkers(settings, launch.pid, observation); + if (startup.kind === 'retry') { + await discardExitedLaunchRecord(settings, launch); + return { retry: true, phases: observation }; + } + if (startup.kind === 'unproven') { + await discardExitedLaunchRecord(settings, launch); + const startupError = normalizeError(startup.error); + return { + retry: false, + startError: startupError.message, + startupError, + daemonProcess: { pid: launch.pid }, + phases: observation, + }; + } + const { cleanup, joined } = await retireStartupAttempt(settings, launch, deadline); + const reason: DaemonStartupFailureReason = + startup.kind === 'stalled' + ? 'startup-stalled' + : startup.kind === 'ceiling' + ? 'startup-ceiling' + : 'startup-exited'; + const available = isRegistrationAvailable(inspectProcessLock(settings.paths.lockPath)); + return { + cleanup, + retry: joined && startup.kind === 'early_exit' && available, + reason, + startError: startup.kind === 'early_exit' ? describeDaemonEarlyExit(startup.exit) : undefined, + daemonProcess: startup.kind === 'early_exit' ? startup.exit : { pid: launch.pid }, + phases: observation, + }; +} + +async function recoverFailedDaemonLaunch( + settings: DaemonClientSettings, + error: unknown, +): Promise { + const cleanup = settings.ownedStateDir + ? await stopAndRetireDaemon({ + paths: settings.paths, + observed: null, + mode: 'graceful', + ownedStateDir: settings.ownedStateDir, + lockTimeoutMs: 0, + }) + : await recoverAbandonedDaemonRegistration({ + paths: settings.paths, + observed: null, + lockTimeoutMs: 0, + }); + return { + cleanup, + startError: normalizeError(error).message, + retry: !settings.ownedStateDir && cleanup.status !== 'retained', + }; +} + +async function retireStartupAttempt( + settings: DaemonClientSettings, + launch: DaemonStartupLaunch, + deadline: number, +): Promise<{ cleanup: DaemonRetirementResult; joined: boolean }> { + const cleanup = await stopAndRetireDaemon({ + paths: settings.paths, + observed: launch.readIdentity(), + mode: 'graceful', + ownedStateDir: settings.ownedStateDir, + termTimeoutMs: Math.min(3_000, remainingStartupBudget(deadline)), + killTimeoutMs: 1_000, + lockTimeoutMs: 0, + }); + const joined = await joinStartup(launch); + // A joined exit is what licenses removing the record; without it the file stays. + if (joined) + removeDaemonStartupPhasesFile( + resolveDaemonStartupPhasesPath(settings.paths.baseDir, launch.pid), + ); + return { cleanup, joined }; +} + +async function joinStartup(launch: DaemonStartupLaunch): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + launch.exited.then(() => true), + new Promise((resolve) => { + timer = setTimeout(() => resolve(false), 1_000); + }), + ]); + } finally { + clearTimeout(timer); + } +} + +/** Sweeps a proven-exited launch's own record after joining that exact child. */ +async function discardExitedLaunchRecord( + settings: DaemonClientSettings, + launch: DaemonStartupLaunch, +): Promise { + if (await joinStartup(launch)) { + removeDaemonStartupPhasesFile( + resolveDaemonStartupPhasesPath(settings.paths.baseDir, launch.pid), + ); + } +} + +function remainingStartupBudget(deadline?: number): number { + return deadline === undefined ? Number.POSITIVE_INFINITY : Math.max(0, deadline - Date.now()); +} + +function isRegistrationAvailable(inspection: ProcessLockInspection): boolean { + return ( + inspection.state === 'absent' || + (inspection.state === 'held' && + (inspection.liveness === 'owner-process-dead' || + inspection.liveness === 'owner-process-reused')) + ); +} + +async function waitForDaemonStartup( + deadline: number, + settings: DaemonClientSettings, + launch: DaemonStartupLaunch, + observation: StartupPhaseObservation, +): Promise { + let earlyExit: ExecDetachedExit | undefined; + void launch.exited.then((exit) => { + earlyExit = exit; + }); + while (Date.now() < deadline) { + if (earlyExit) { + const settled = await judgeExitedDaemonStartup(settings, earlyExit, deadline); + if (settled) return settled; + } else { + const settled = await observeLiveDaemonStartup(settings, launch, deadline, observation); + // An in-flight exit outranks a `ready` read: the publisher is gone. + if (settled && !(settled.kind === 'ready' && earlyExit)) return settled; + } + await sleep(Math.min(100, remainingStartupBudget(deadline))); + } + return classifyExpiredDaemonStartup(settings, launch, earlyExit); +} + +/** Judges the wait at an expired deadline; never signals anything. */ +function classifyExpiredDaemonStartup( + settings: DaemonClientSettings, + launch: DaemonStartupLaunch, + earlyExit: ExecDetachedExit | undefined, +): DaemonStartupWaitResult { + // A live lock owner outranks an observed exit; a delivered exit is authoritative for its lifetime. + const liveLockOwner = isLiveLockOwner(inspectProcessLock(settings.paths.lockPath)); + if (liveLockOwner) return { kind: 'ceiling' }; + if (earlyExit) return { kind: 'early_exit', exit: earlyExit }; + return isLiveLaunchedStartup(launch) ? { kind: 'ceiling' } : { kind: 'exit_unobserved' }; +} + +function isLiveLaunchedStartup(launch: DaemonStartupLaunch): boolean { + if (!isProcessAlive(launch.pid)) return false; + const identity = launch.readIdentity(); + if (!identity.startTime) return true; + return classifyOwnerLiveness({ owner: identity }) !== 'owner-process-reused'; +} + +/** A lock owner is a pending startup unless provably dead or reused. */ +function isLiveLockOwner(inspection: ProcessLockInspection): boolean { + return ( + inspection.state === 'held' && + inspection.liveness !== 'owner-process-dead' && + inspection.liveness !== 'owner-process-reused' + ); +} + +async function judgeExitedDaemonStartup( + settings: DaemonClientSettings, + exit: ExecDetachedExit, + deadline: number, +): Promise { + const kind = classifyDaemonStartupExit(exit); + if (kind === 'unproven') + return { kind: 'unproven', error: daemonRegistrationUnprovenError(settings) }; + if (kind === 'failed') return { kind: 'early_exit', exit }; + return await observeContendingDaemon(settings, deadline); +} + +async function observeLiveDaemonStartup( + settings: DaemonClientSettings, + launch: DaemonStartupLaunch, + deadline: number, + observation: StartupPhaseObservation, +): Promise { + const info = await readReadyLaunchedDaemon(settings, launch, deadline); + if (info) return { kind: 'ready', daemon: { info, startedByClient: true } }; + refreshStartupPhaseMarkers(settings, launch.pid, observation); + if ( + isProcessAlive(launch.pid) && + Date.now() - lastStartupProgressMs(observation) >= DAEMON_STARTUP_STALL_WINDOW_MS + ) { + return { kind: 'stalled' }; + } + return null; +} + +function refreshStartupPhaseMarkers( + settings: DaemonClientSettings, + launchPid: number, + observation: StartupPhaseObservation, +): void { + const tail = readRecentFileTail( + resolveDaemonStartupPhasesPath(settings.paths.baseDir, launchPid), + DAEMON_STARTUP_MARKER_TAIL_BYTES, + ); + if (!tail) return; + for (const [phase, timestampMs] of Object.entries(readDaemonStartupPhaseMarkers(tail)) as Array< + [DaemonStartupPhase, number] + >) { + observation.markers[phase] = timestampMs; + } +} + +function lastStartupProgressMs(observation: StartupPhaseObservation): number { + return Object.values(observation.markers).reduce( + (latest, timestampMs) => Math.max(latest, timestampMs), + observation.spawnedAtMs, + ); +} + +function describeStartupPhaseTimings(observation: StartupPhaseObservation): Record { + const timings: Record = { spawned: 0 }; + for (const [phase, timestampMs] of Object.entries(observation.markers)) { + timings[phase] = Math.max(0, timestampMs - observation.spawnedAtMs); + } + return timings; +} + +function resolveStartupProgressHint(reason: DaemonStartupFailureReason, ceilingMs: number): string { + if (reason === 'startup-stalled') { + return ( + `The daemon is alive but made no startup progress for ${DAEMON_STARTUP_STALL_WINDOW_MS / 1000} s. ` + + `The host can be overloaded. Retry once the host load drops; ${DAEMON_STARTUP_TIMEOUT_ENV} ` + + `raises only the registration ceiling and does not change this stall window.` + ); + } + return ( + `The daemon is alive but did not register within ${Math.round(ceilingMs / 1000)} s. ` + + `The host can be overloaded. Retry, or raise ${DAEMON_STARTUP_TIMEOUT_ENV}.` + ); +} + +function classifyDaemonStartupExit(exit: ExecDetachedExit): 'busy' | 'unproven' | 'failed' { + if (exit.error || exit.signal) return 'failed'; + switch (exit.exitCode) { + case DAEMON_STARTUP_EXIT_CODES.busy: + return 'busy'; + case DAEMON_STARTUP_EXIT_CODES.unproven: + return 'unproven'; + default: + return 'failed'; + } +} + +async function observeContendingDaemon( + settings: DaemonClientSettings, + deadline: number, +): Promise { + let winner: DaemonInfo | null; + try { + winner = await readReusableLocalDaemon(settings, { deadline, waitForLiveStartup: true }); + } catch (error) { + if (error instanceof AppError && error.details?.reason === 'daemon_registration_unproven') + return { kind: 'unproven', error }; + throw error; + } + if (Date.now() >= deadline) return null; + if (winner) return { kind: 'ready', daemon: { info: winner, startedByClient: false } }; + const inspection = inspectProcessLock(settings.paths.lockPath); + if (inspection.state === 'unproven') + return { kind: 'unproven', error: daemonRegistrationUnprovenError(settings, inspection) }; + return isRegistrationAvailable(inspection) ? { kind: 'retry' } : null; +} + +async function readReadyLaunchedDaemon( + settings: DaemonClientSettings, + launch: DaemonStartupLaunch, + deadline: number, +): Promise { + const info = readDaemonInfo(settings.paths.infoPath); + if (!info || !isLaunchedDaemon(info, launch)) return null; + try { + return (await canConnect( + info, + settings.transportPreference, + remainingStartupBudget(deadline), + )) && Date.now() < deadline + ? info + : null; + } catch (error) { + const { cleanup, joined } = await retireStartupAttempt(settings, launch, deadline); + emitDiagnostic({ + level: 'warn', + phase: 'daemon_startup_observation_failed', + data: { stateDir: settings.paths.baseDir, cleanup, joined, error: normalizeError(error) }, + }); + if (error instanceof AppError) { + error.details = { + ...error.details, + stateDir: settings.paths.baseDir, + cleanupResults: [cleanup], + startupJoined: joined, + }; + } + throw error; + } +} + +function isLaunchedDaemon(info: DaemonInfo, launch: DaemonStartupLaunch): boolean { + const identity = launch.readIdentity(); + return ( + info.pid === identity.pid && + identity.startTime !== null && + info.processStartTime === identity.startTime + ); +} + +function startDaemon(settings: DaemonClientSettings): DaemonStartupLaunch { + const launchSpec = resolveDaemonLaunchSpec(); + return launchDaemonProcess({ + paths: settings.paths, + serverMode: settings.serverMode, + env: settings.host.env, + cwd: settings.host.cwd, + ownedStateDir: settings.ownedStateDir, + args: launchSpec.useSrc + ? ['--experimental-strip-types', launchSpec.srcPath] + : [launchSpec.distPath], + }); +} + +function describeDaemonEarlyExit(exit: ExecDetachedExit): string { + if (exit.error) return `daemon process ${exit.pid} failed to start: ${exit.error}`; + if (exit.signal) + return `daemon process ${exit.pid} exited before readiness with signal ${exit.signal}`; + return `daemon process ${exit.pid} exited before readiness with code ${exit.exitCode ?? 0}`; +} + +function readRecentLogTail(logPath: string): string | undefined { + return readRecentFileTail(logPath, DAEMON_STARTUP_LOG_TAIL_BYTES); +} + +function readRecentFileTail(filePath: string, maxBytes: number): string | undefined { + try { + if (!fs.existsSync(filePath)) return undefined; + const stats = fs.statSync(filePath); + if (stats.size <= 0) return undefined; + const length = Math.min(stats.size, maxBytes); + const fd = fs.openSync(filePath, 'r'); + try { + const buffer = Buffer.alloc(length); + fs.readSync(fd, buffer, 0, length, stats.size - length); + const text = buffer.toString('utf8').trim(); + return text.length > 0 ? text : undefined; + } finally { + fs.closeSync(fd); + } + } catch { + return undefined; + } +} diff --git a/src/daemon.ts b/src/daemon.ts index 5f34d5f54e..471280fd5e 100644 --- a/src/daemon.ts +++ b/src/daemon.ts @@ -1,6 +1,28 @@ -import { asAppError } from '@agent-device/kernel/errors'; +import { asAppError, normalizeError } from '@agent-device/kernel/errors'; import { daemonStartupExitCode, reportDaemonClose, startDaemonRuntime } from '@agent-device/daemon'; import { captureHostEnvironment } from '@agent-device/host-kit/process'; +import { + removeDaemonStartupPhasesFile, + recordDaemonStartupMarker, + resolveDaemonStartupPhasesPath, +} from '@agent-device/daemon-contracts/daemon-startup-phases'; +import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution'; + +// A refused marker append goes to stderr, which the launcher redirects into the daemon log. +const reportMarkerFailure = (error: unknown): void => { + process.stderr.write(`Daemon startup marker write failed: ${normalizeError(error).message}\n`); +}; + +// Captured once: the earliest marker and the runtime resolve the state dir from this one host, +// so a relative AGENT_DEVICE_STATE_DIR cannot land the markers beside another directory's lock. +const host = captureHostEnvironment(process); +const ownPhasesPath = resolveDaemonStartupPhasesPath( + resolveDaemonPaths(host.env.AGENT_DEVICE_STATE_DIR, { env: host.env, cwd: host.cwd }).baseDir, + process.pid, +); +// The claim first: after pid reuse it is the only defense against a dead lifetime's phases. +removeDaemonStartupPhasesFile(ownPhasesPath, reportMarkerFailure); +recordDaemonStartupMarker(ownPhasesPath, 'modules-loaded', reportMarkerFailure); // The executable owns the process: signals, fatal-error handlers and the exit code. The runtime // only reports when it has closed, whoever closed it. @@ -9,9 +31,16 @@ import { captureHostEnvironment } from '@agent-device/host-kit/process'; // closure of this entry stays exactly the operational server's (ADR 0019). async function runDaemon(): Promise { const runtime = await startDaemonRuntime({ - host: captureHostEnvironment(process), - composeDeps: async (input) => - (await import('./daemon-composition.ts')).createDaemonRuntimeDeps(input), + host, + composeDeps: async (input) => { + const deps = await (await import('./daemon-composition.ts')).createDaemonRuntimeDeps(input); + recordDaemonStartupMarker( + resolveDaemonStartupPhasesPath(input.baseDir, process.pid), + 'composition-loaded', + reportMarkerFailure, + ); + return deps; + }, }); let exitCode = 0; const closeAndExit = (code: number, cause?: unknown): void => { diff --git a/website/docs/docs/configuration.md b/website/docs/docs/configuration.md index eb2896d7b0..b326fa917d 100644 --- a/website/docs/docs/configuration.md +++ b/website/docs/docs/configuration.md @@ -122,7 +122,7 @@ Only the variables below, and those documented on command pages, are supported. | Category | Env vars | Decision | | --- | --- | --- | -| CLI defaults and config | `AGENT_DEVICE_HOME`, `AGENT_DEVICE_CONFIG`, `AGENT_DEVICE_SESSION`, `AGENT_DEVICE_PLATFORM`, `AGENT_DEVICE_SCREENSHOT_SCALE`, `AGENT_DEVICE_SESSION_LOCK`, `AGENT_DEVICE_DAEMON_BASE_URL`, `AGENT_DEVICE_DAEMON_AUTH_TOKEN`, `AGENT_DEVICE_CLOUD_BASE_URL` | Public | +| CLI defaults and config | `AGENT_DEVICE_HOME`, `AGENT_DEVICE_CONFIG`, `AGENT_DEVICE_SESSION`, `AGENT_DEVICE_PLATFORM`, `AGENT_DEVICE_SCREENSHOT_SCALE`, `AGENT_DEVICE_SESSION_LOCK`, `AGENT_DEVICE_DAEMON_BASE_URL`, `AGENT_DEVICE_DAEMON_AUTH_TOKEN`, `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS`, `AGENT_DEVICE_CLOUD_BASE_URL` | Public. `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` is the ceiling (a positive integer number of milliseconds, default `120000`; any other value fails with `INVALID_ARGS` before a daemon spawns, and only when the client must start a daemon — a client that attaches to or reuses a running daemon never reads it) a client waits for a local daemon it starts to register; CLI, MCP, and Node clients share it. A live daemon is waited for while it records startup progress, so an overloaded host slows startup instead of failing it. A failed startup names the outcome in `details.reason`: `startup-exited` (the process ended before registering), `startup-stalled` (alive, no startup progress for a fixed 60 s window this variable does not change), or `startup-ceiling` (a startup was still live when the ceiling passed — this client's daemon or another client's; the client stops its own launch and never signals another client's live daemon. Only a progressing own launch outlasts the fixed stall window). Branch on `details.reason`, never on the message. | | Device scoping | `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST`, `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` | Public. `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` is read by the daemon: every iOS simulator request that names no `--ios-simulator-device-set` uses that set, so iOS physical devices are not listed, and a request naming a different path string fails with `INVALID_ARGS` (`details.reason: "ios_simulator_device_set_outside_daemon_set"`). macOS and non-iOS requests ignore it. A daemon started by the CLI inherits the CLI's value; restart the daemon after changing it. | | Android test IME | `AGENT_DEVICE_TEST_IME` | Public. Same setting as `--test-ime` / `--no-test-ime` on `open`, `test`, and `replay`; see known limitations. | | Local daemon storage | `AGENT_DEVICE_STATE_DIR` | Public |