From 46441da9c421d433025580434322f58a0eafd655 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 12:08:22 -0700 Subject: [PATCH 001/265] [rush-client-core] Reclaim provably stale daemon artifacts; make `daemon stop`/`restart` idempotent; add `daemon stop --force` (#6080) * [rush-client-core] Reclaim provably stale daemon artifacts; make daemon stop/restart idempotent Fixes #6061 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-client-core] Update API report and test expectation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-client-core] Address review: force stop clears leftovers after shutdown, validate records before waiting, reset hint on unresolved handoff Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-client-core] Update API report Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- apps/rush-cli-client/README.md | 30 +- apps/rush-cli-client/src/daemonCommands.ts | 79 +++++- .../src/test/launchClient.test.ts | 81 +++++- .../daemon-recovery_2026-09-23.json | 10 + .../daemon-recovery_2026-09-23.json | 10 + common/reviews/api/rush-client-core.api.md | 13 + libraries/rush-client-core/README.md | 11 +- .../rush-client-core/src/DaemonOwnership.ts | 258 ++++++++++++++++++ .../rush-client-core/src/ProcessStartTime.ts | 67 +++++ .../src/connectOrStartDaemon.ts | 90 ++---- libraries/rush-client-core/src/index.ts | 5 + .../src/test/ProcessStartTime.test.ts | 31 +++ .../src/test/connectOrStartDaemon.test.ts | 147 +++++++++- 13 files changed, 740 insertions(+), 92 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/daemon-recovery_2026-09-23.json create mode 100644 common/changes/@rushstack/rush-client-core/daemon-recovery_2026-09-23.json create mode 100644 libraries/rush-client-core/src/DaemonOwnership.ts create mode 100644 libraries/rush-client-core/src/ProcessStartTime.ts create mode 100644 libraries/rush-client-core/src/test/ProcessStartTime.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 483d671312..bb93cf69f5 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -257,19 +257,39 @@ attests a restart request, not completion of successor startup or success of a c `rush-client daemon stop` requires protocol >= 0.6 and waits for `shutdownAck` followed by EOF. It reports `state: "shutdownAccepted"` with exit code 0; this -does not assert successful workspace disposal. An absent/unreachable daemon, -unsupported protocol, missing acknowledgement, or timeout returns exit code 1. -It does not auto-start anything. +does not assert successful workspace disposal. Stop is idempotent: when nothing +listens at the endpoint it reports `state: "notRunning"` with exit code 0. An +unsupported protocol, missing acknowledgement, handshake failure, or timeout +returns exit code 1. It does not auto-start anything. + +`rush-client daemon stop --force` stops a running daemon the same way, then waits +(up to 15 seconds) for it to release its listener and ownership record and removes +any remaining artifacts, such as an abandoned startup reservation, reporting them in +`removedPaths`. When none is listening, it removes this workspace's leftover ownership record +(`.pid.json`), socket, and startup reservation (`.starting`), then reports +`state: "reset"` and the `removedPaths` (or `state: "notRunning"` if nothing was +left behind). It holds the start mutex, proves that no listener is bound, and +refuses (exit 1) while the recorded owner PID still exists and cannot be shown to +be a reused PID. It never kills a process. Automatic startup already reclaims +the common leftovers on its own (see below); this is the documented escape hatch +that every fail-closed startup message points to. `rush-client daemon restart` first verifies that the selected Rush version has a launcher and captures the original lock's PID/start timestamp, checking that it matches pong's positive PID and the selected endpoint, then performs acknowledged shutdown. It waits for original ownership release or a demonstrably dead owner -before calling the existing locked starter. A live/reused owner fails closed at +before calling the existing locked starter. A live owner fails closed at the startup deadline; no PID is killed and no live ownership record is deleted. A newly started/reused successor must pass hello/ping before reporting `state: "ready"`. -An absent daemon must be started explicitly with `daemon start`. +When nothing listens at the endpoint, restart starts a daemon exactly like `daemon start`. + +Automatic and explicit startup reclaim stale artifacts only when that is provably +safe: while holding the start mutex with no `.starting` reservation, a socket +without an ownership record, or an unreadable/corrupt record, is removed only after +a connection attempt is refused (so no listener exists). On Linux, a record whose +PID now belongs to a process that started after the record's `startedAt` (PID reuse) +is treated as dead; other platforms fail closed and point to `daemon stop --force`. Restart is explicit even when automatic startup or CI execution routing is disabled, but conflicts with `--no-daemon`. The two-phase host retains ownership diff --git a/apps/rush-cli-client/src/daemonCommands.ts b/apps/rush-cli-client/src/daemonCommands.ts index d2c3e592b1..b6ad07750d 100644 --- a/apps/rush-cli-client/src/daemonCommands.ts +++ b/apps/rush-cli-client/src/daemonCommands.ts @@ -7,9 +7,14 @@ import { DaemonClient, connectOrStartDaemonAsync, requestDaemonShutdownAsync, + resetDaemonArtifactsAsync, type IConnectOrStartDaemonOptions } from '@rushstack/rush-client-core'; -import type { IDaemonLockfile } from '@rushstack/rush-daemon-transport'; +import { + DaemonTransportError, + DaemonTransportErrorCode, + type IDaemonLockfile +} from '@rushstack/rush-daemon-transport'; import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; @@ -17,6 +22,8 @@ import { printDaemonLogAsync } from './daemonLogs'; import { executeDaemonGraphCommandAsync } from './daemonGraph'; import { writeStreamAsync } from './writeStreamAsync'; +const FORCE_STOP_WAIT_MS: number = 15000; + export interface IDaemonCommandOptions { readonly argv: ReadonlyArray; readonly environment: Readonly; @@ -38,14 +45,18 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): } if ( (options.argv.length !== 1 && - !(command === 'logs' && options.argv.length === 2 && options.argv[1] === '--follow')) || + !( + options.argv.length === 2 && + ((command === 'logs' && options.argv[1] === '--follow') || + (command === 'stop' && options.argv[1] === '--force')) + )) || (command !== 'start' && command !== 'status' && command !== 'stop' && command !== 'restart' && command !== 'logs') ) { - throw new Error('Usage: rush-client daemon start|status|stop|restart|logs [--follow]'); + throw new Error('Usage: rush-client daemon start|status|stop [--force]|restart|logs [--follow]'); } if (!options.rushJsonPath) throw new Error('Daemon management requires a repository containing rush.json.'); const mayStart: boolean = command === 'start' || command === 'restart'; @@ -76,13 +87,52 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): } // Status observes the selected endpoint, including a compatible daemon from a different client version. // It never starts a process or trusts a PID file as evidence of readiness. - const client: DaemonClient = + const client: DaemonClient | undefined = command === 'start' ? await connectOrStartDaemonAsync(connectionOptions) - : await DaemonClient.connectAsync({ socketPath: connectionOptions.paths.socketPath }); + : await connectExistingAsync(connectionOptions, command !== 'status'); + if (!client) { + if (command === 'restart') { + // Nothing to shut down: restart behaves like start. + const started: DaemonClient = await connectOrStartDaemonAsync(connectionOptions); + try { + await writeStatusAsync({ + state: 'ready', + socketPath: connectionOptions.paths.socketPath, + ...(await started.status) + }); + } finally { + await started.closeAsync(); + } + return; + } + const { removedPaths } = + options.argv[1] === '--force' + ? await resetDaemonArtifactsAsync(connectionOptions.paths) + : { removedPaths: [] }; + await writeStatusAsync({ + state: removedPaths.length > 0 ? 'reset' : 'notRunning', + socketPath: connectionOptions.paths.socketPath, + ...(options.argv[1] === '--force' ? { removedPaths } : {}) + }); + return; + } try { if (command === 'stop') { await client.shutdownAsync(); + if (options.argv[1] === '--force') { + // Wait for the acknowledged daemon to release its listener and record, then clear leftovers + // such as an abandoned startup reservation in the same invocation. + const { removedPaths } = await resetDaemonArtifactsAsync(connectionOptions.paths, { + waitTimeoutMs: FORCE_STOP_WAIT_MS + }); + await writeStatusAsync({ + state: 'shutdownAccepted', + socketPath: connectionOptions.paths.socketPath, + removedPaths + }); + return; + } await writeStatusAsync({ state: 'shutdownAccepted', socketPath: connectionOptions.paths.socketPath @@ -105,6 +155,25 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): } } +/** Returns undefined when nothing listens at the endpoint and `allowAbsent` is set; other failures propagate. */ +async function connectExistingAsync( + options: IConnectOrStartDaemonOptions, + allowAbsent: boolean +): Promise { + try { + return await DaemonClient.connectAsync({ socketPath: options.paths.socketPath }); + } catch (error) { + if ( + allowAbsent && + error instanceof DaemonTransportError && + error.code === DaemonTransportErrorCode.connectionRefused + ) { + return undefined; + } + throw error; + } +} + async function restartDaemonAsync( client: DaemonClient, options: IConnectOrStartDaemonOptions diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index 14c8d888d9..b3ec694fca 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -248,13 +248,82 @@ describe('standalone rushx fallback', () => { expect((await invokeAsync(true, false, false, ['daemon', 'logs', '--follow', 'extra'])).code).toBe(1); }); - it('does not start an absent daemon when stop or restart cannot be acknowledged', async () => { - for (const verb of ['stop', 'restart']) { - const result: IInvocationResult = await invokeAsync(true, false, false, ['daemon', verb]); - expect(result.code).toBe(1); - expect(result.stderr).toContain('Could not connect to daemon'); + it('treats stop as idempotent and restart as start when no daemon is running', async () => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + for (const args of [ + ['daemon', 'stop'], + ['daemon', 'stop', '--force'] + ]) { + const result: IInvocationResult = await invokeAsync(true, false, false, args); + expect(result).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(result.stdout)).toEqual({ + state: 'notRunning', + socketPath: paths.socketPath, + ...(args[2] ? { removedPaths: [] } : {}) + }); } - }); + try { + const restarted: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'restart']); + expect(restarted.stderr).toBe(''); + expect(restarted.code).toBe(0); + expect(JSON.parse(restarted.stdout)).toMatchObject({ state: 'ready', socketPath: paths.socketPath }); + const stopped: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop']); + expect(stopped.code).toBe(0); + expect(JSON.parse(stopped.stdout)).toMatchObject({ state: 'shutdownAccepted' }); + } finally { + const deadline: number = Date.now() + 7000; + while (fs.existsSync(paths.lockfilePath) && Date.now() < deadline) await delayAsync(50); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + } + }, 30000); + + it('stop --force clears an abandoned startup reservation next to a running daemon', async () => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + const reservation: string = `${paths.lockfilePath}.starting`; + try { + expect((await invokeAsync(true, false, false, ['daemon', 'start'])).code).toBe(0); + fs.writeFileSync(reservation, 'abandoned'); + const result: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop', '--force']); + expect(result).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(result.stdout)).toEqual({ + state: 'shutdownAccepted', + socketPath: paths.socketPath, + removedPaths: [reservation] + }); + expect(fs.existsSync(reservation)).toBe(false); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + } finally { + const deadline: number = Date.now() + 7000; + while (fs.existsSync(paths.lockfilePath) && Date.now() < deadline) await delayAsync(50); + } + }, 30000); + + (process.platform === 'win32' ? it.skip : it)( + 'stop --force removes stale artifacts left by a killed daemon', + async () => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + fs.mkdirSync(path.dirname(paths.lockfilePath), { recursive: true, mode: 0o700 }); + const listener: ChildProcess = spawn( + process.execPath, + [ + '-e', + `require('net').createServer().listen(${JSON.stringify(paths.socketPath)}, () => process.kill(process.pid, 'SIGKILL'))` + ], + { stdio: 'ignore' } + ); + await once(listener, 'close'); + fs.writeFileSync(paths.lockfilePath, 'garbage{'); + const result: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop', '--force']); + expect(result).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(result.stdout)).toEqual({ + state: 'reset', + socketPath: paths.socketPath, + removedPaths: [paths.lockfilePath, paths.socketPath] + }); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + expect(fs.existsSync(paths.socketPath)).toBe(false); + } + ); it.each([false, true])( 'restarts after ownership release and stops the successor (embedded: %s)', diff --git a/common/changes/@rushstack/rush-cli-client/daemon-recovery_2026-09-23.json b/common/changes/@rushstack/rush-cli-client/daemon-recovery_2026-09-23.json new file mode 100644 index 0000000000..ddc56a8ade --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/daemon-recovery_2026-09-23.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Make `rush-client daemon stop` idempotent (state notRunning, exit 0), make `daemon restart` start a daemon when none is running, and add `daemon stop --force` to remove stale workspace daemon artifacts.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-client-core/daemon-recovery_2026-09-23.json b/common/changes/@rushstack/rush-client-core/daemon-recovery_2026-09-23.json new file mode 100644 index 0000000000..4dbf5f28e0 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/daemon-recovery_2026-09-23.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Reclaim provably stale daemon artifacts (a socket without an ownership record, a corrupt record, or a Linux PID reused since the record was written) instead of permanently disabling the daemon, and add resetDaemonArtifactsAsync() as the explicit recovery path referenced by fail-closed messages.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 103df5d039..a731a4a1d2 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -81,6 +81,16 @@ export interface IConnectOrStartDaemonOptions extends Omit; +} + // @beta export interface IDaemonClientConnectOptions { // (undocumented) @@ -131,4 +141,7 @@ export interface IDaemonStartCommand { // @beta export function requestDaemonShutdownAsync(client: DaemonClient, paths: IDaemonPaths, timeoutMs?: number): Promise>; +// @beta +export function resetDaemonArtifactsAsync(paths: IDaemonPaths, options?: IDaemonArtifactResetOptions): Promise; + ``` diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 6c1afe75e6..86142f10ee 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -48,7 +48,13 @@ handing the explicit command to a detached startup helper. The helper spawns wit a shell and retains that reservation until the daemon completes hello/ping readiness, independently of whether the requesting client survives. Clients still await hello/pong under bounded backoff. Stdout/stderr go to `.log`. No PID -is killed; a live (possibly reused) PID with an unreachable socket fails closed. +is killed. While holding the mutex with no startup reservation, stale leftovers are +reclaimed only when provably safe: a socket without an ownership record, or a corrupt +record, once a connection attempt is refused (no listener exists); and, on Linux, a +record whose PID now belongs to a process that started after the record's `startedAt` +(PID reuse, detected from `/proc`). Any other live PID with an unreachable socket fails +closed, pointing to `resetDaemonArtifactsAsync()` (`rush-client daemon stop --force`), +which removes the record, socket and reservation after the same no-listener/no-live-owner checks. The helper uses a stable tool cwd, and the starting client awaits its exit after readiness. The explicit launcher's cwd is unchanged. @@ -90,7 +96,8 @@ identifies the original ownership record by its `pid` and `startedAt`, captured before sending shutdown. Startup waits until that record disappears, another owner replaces it, or its owner is demonstrably dead. A new owner is checked by hello/ping; it is never blindly reclaimed. Signal 0 is only a liveness probe; no -process is killed. A live/reused owner times out conservatively, while corrupt or +process is killed. A live owner times out conservatively (a Linux PID provably reused +since `startedAt` counts as dead), while corrupt or unreadable metadata fails closed. During a captured predecessor handoff, transient Windows sharing-denied reads stay unknown and are retried only within the existing startup deadline. They never diff --git a/libraries/rush-client-core/src/DaemonOwnership.ts b/libraries/rush-client-core/src/DaemonOwnership.ts new file mode 100644 index 0000000000..a7226d0ee9 --- /dev/null +++ b/libraries/rush-client-core/src/DaemonOwnership.ts @@ -0,0 +1,258 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as net from 'node:net'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonLockfile, IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { DaemonClientError } from './DaemonClientError'; +import { getDaemonStartupFilePath } from './DaemonStartup'; +import { isProcessStartedAfter } from './ProcessStartTime'; +import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; + +const PROBE_TIMEOUT_MS: number = 1000; +const RESET_RETRY_MS: number = 100; + +/** Printed wherever automatic recovery fails closed. */ +export const DAEMON_RESET_HINT: string = + 'If no daemon is running for this workspace, run "rush-client daemon stop --force" to remove its stale files.'; + +export type DaemonOwnership = Pick; + +type OwnershipState = + | { readonly kind: 'absent' } + | { readonly kind: 'corrupt'; readonly raw: string } + | { readonly kind: 'owned'; readonly raw: string; readonly owner: DaemonOwnership }; + +/** Options for {@link resetDaemonArtifactsAsync}. @beta */ +export interface IDaemonArtifactResetOptions { + /** How long to keep re-checking a bound listener, live owner, or held start mutex. Defaults to 0. */ + readonly waitTimeoutMs?: number; +} + +/** The result of {@link resetDaemonArtifactsAsync}. @beta */ +export interface IDaemonArtifactResetResult { + /** The stale files that were removed; empty when nothing was left behind. */ + readonly removedPaths: ReadonlyArray; +} + +export function isDaemonOwnership(record: unknown): record is DaemonOwnership { + return ( + typeof record === 'object' && + record !== null && + 'pid' in record && + typeof record.pid === 'number' && + Number.isSafeInteger(record.pid) && + record.pid > 0 && + 'startedAt' in record && + typeof record.startedAt === 'string' && + Number.isFinite(Date.parse(record.startedAt)) + ); +} + +export function readDaemonOwnership(lockfilePath: string): DaemonOwnership | undefined { + let record: unknown; + try { + record = JSON.parse(fs.readFileSync(lockfilePath, 'utf8')); + } catch (error) { + if (hasErrorCode(error, 'ENOENT')) return undefined; + throw new DaemonClientError( + 'startupFailed', + `Cannot safely read ${lockfilePath}; refusing automatic reclaim. ${DAEMON_RESET_HINT}`, + { cause: error } + ); + } + if (!isDaemonOwnership(record)) { + throw new DaemonClientError( + 'startupFailed', + `Invalid daemon ownership record in ${lockfilePath}; refusing automatic reclaim. ${DAEMON_RESET_HINT}` + ); + } + return { pid: record.pid, startedAt: record.startedAt }; +} + +export function isProcessAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + if (hasErrorCode(error, 'ESRCH')) return false; + throw error; + } +} + +/** + * True unless the recorded owner is demonstrably gone: its PID does not exist, or the process now using + * that PID started after the record was written (PID reuse). + */ +export function isOwnerProcessAlive(owner: { readonly pid: number; readonly startedAt?: unknown }): boolean { + if (!isProcessAlive(owner.pid)) return false; + return !(typeof owner.startedAt === 'string' && isProcessStartedAfter(owner.pid, owner.startedAt)); +} + +/** + * Makes stale ownership reclaimable when that is provably safe. The caller must hold the start mutex and + * have observed no startup reservation, so no legitimate daemon can be between bind and record publication; + * a refused connection then proves no listener exists. + */ +export async function reclaimAbandonedOwnershipAsync(paths: IDaemonPaths): Promise { + const state: OwnershipState = inspectOwnership(paths.lockfilePath); + if (state.kind === 'owned' && !isProcessAlive(state.owner.pid)) return; + if (state.kind === 'owned' && !isProcessStartedAfter(state.owner.pid, state.owner.startedAt)) { + throw new DaemonClientError( + 'startupFailed', + `PID ${state.owner.pid} still exists but the daemon is not ready. It may be a daemon that is still shutting down, or a reused PID; refusing to kill it or remove ${paths.lockfilePath}. ${DAEMON_RESET_HINT}` + ); + } + if (state.kind === 'absent' && (process.platform === 'win32' || !fs.existsSync(paths.socketPath))) return; + if (!(await isEndpointUnboundAsync(paths.socketPath))) { + throw new DaemonClientError( + 'startupFailed', + `${describeOwnership(state, paths)}, but ${paths.socketPath} did not refuse a connection; refusing automatic reclaim. ${DAEMON_RESET_HINT}` + ); + } + // The transport reclaim then removes the unbound socket under its own two-factor checks. + if (state.kind !== 'absent') removeIfUnchanged(paths.lockfilePath, state.raw); +} + +/** + * Removes this workspace's leftover daemon files (ownership record, socket, and startup reservation) + * after verifying that no listener is bound and that the recorded owner, if any, is gone. + * @remarks Never kills a process. Fails when another client holds the start mutex, a listener is bound, + * or the recorded owner is alive; with `waitTimeoutMs`, those conditions are re-checked until the deadline + * (for example, while a daemon that just acknowledged shutdown finishes its cleanup). + * @beta + */ +export async function resetDaemonArtifactsAsync( + paths: IDaemonPaths, + options?: IDaemonArtifactResetOptions +): Promise { + const deadline: number = Date.now() + (options?.waitTimeoutMs ?? 0); + while (true) { + const outcome: IDaemonArtifactResetResult | DaemonClientError = await tryResetDaemonArtifactsAsync(paths); + if (!(outcome instanceof DaemonClientError)) return outcome; + if (Date.now() >= deadline) throw outcome; + await delayAsync(Math.min(RESET_RETRY_MS, Math.max(1, deadline - Date.now()))); + } +} + +/** Returns a (not thrown) error for conditions that may clear on their own. */ +async function tryResetDaemonArtifactsAsync( + paths: IDaemonPaths +): Promise { + if (!fs.existsSync(path.dirname(paths.lockfilePath))) return { removedPaths: [] }; + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + if (!lock) { + return new DaemonClientError( + 'startupFailed', + `Another client is starting the daemon for ${paths.lockfilePath}; retry after it finishes.` + ); + } + try { + if (!(await isEndpointUnboundAsync(paths.socketPath))) { + return new DaemonClientError( + 'startupFailed', + `A process is still listening at ${paths.socketPath}; use "rush-client daemon stop" to stop it.` + ); + } + const state: OwnershipState = inspectOwnership(paths.lockfilePath); + if (state.kind === 'owned' && isOwnerProcessAlive(state.owner)) { + return new DaemonClientError( + 'startupFailed', + `PID ${state.owner.pid} still owns ${paths.lockfilePath}; it may be a daemon that is shutting down. Wait for it to exit (or stop that process yourself), then retry. No process was killed.` + ); + } + const removedPaths: string[] = []; + if (state.kind !== 'absent' && removeIfUnchanged(paths.lockfilePath, state.raw)) { + removedPaths.push(paths.lockfilePath); + } + const others: string[] = [getDaemonStartupFilePath(paths)]; + if (process.platform !== 'win32') others.push(paths.socketPath); + for (const filePath of others) { + if (tryUnlink(filePath)) removedPaths.push(filePath); + } + return { removedPaths }; + } finally { + await lock.releaseAsync(); + } +} + +/** Resolves true only when a connection attempt proves that nothing listens at the endpoint. */ +export function isEndpointUnboundAsync(socketPath: string): Promise { + return new Promise((resolve) => { + const socket: net.Socket = net.createConnection(socketPath); + const settle = (unbound: boolean): void => { + socket.destroy(); + resolve(unbound); + }; + socket.setTimeout(PROBE_TIMEOUT_MS); + socket.once('connect', () => settle(false)); + socket.once('timeout', () => settle(false)); + socket.once('error', (error) => settle(hasErrorCode(error, 'ECONNREFUSED') || hasErrorCode(error, 'ENOENT'))); + }); +} + +export function hasErrorCode(error: unknown, code: string): boolean { + return typeof error === 'object' && error !== null && 'code' in error && error.code === code; +} + +function inspectOwnership(lockfilePath: string): OwnershipState { + let raw: string; + try { + raw = fs.readFileSync(lockfilePath, 'utf8'); + } catch (error) { + if (hasErrorCode(error, 'ENOENT')) return { kind: 'absent' }; + throw new DaemonClientError( + 'startupFailed', + `Cannot safely read ${lockfilePath}; refusing automatic reclaim. ${DAEMON_RESET_HINT}`, + { cause: error } + ); + } + let record: unknown; + try { + record = JSON.parse(raw); + } catch { + return { kind: 'corrupt', raw }; + } + return isDaemonOwnership(record) + ? { kind: 'owned', raw, owner: { pid: record.pid, startedAt: record.startedAt } } + : { kind: 'corrupt', raw }; +} + +function describeOwnership(state: OwnershipState, paths: IDaemonPaths): string { + switch (state.kind) { + case 'absent': + return `Socket ${paths.socketPath} has no ownership record`; + case 'corrupt': + return `Invalid daemon ownership record in ${paths.lockfilePath}`; + default: + return `PID ${state.owner.pid} was reused by a process that started after ${paths.lockfilePath} was written`; + } +} + +function removeIfUnchanged(filePath: string, expected: string): boolean { + let current: string; + try { + current = fs.readFileSync(filePath, 'utf8'); + } catch (error) { + if (hasErrorCode(error, 'ENOENT')) return false; + throw error; + } + if (current !== expected) { + throw new DaemonClientError('startupFailed', `${filePath} changed during reclaim; refusing to remove it.`); + } + return tryUnlink(filePath); +} + +function tryUnlink(filePath: string): boolean { + try { + fs.unlinkSync(filePath); + return true; + } catch (error) { + if (hasErrorCode(error, 'ENOENT')) return false; + throw error; + } +} diff --git a/libraries/rush-client-core/src/ProcessStartTime.ts b/libraries/rush-client-core/src/ProcessStartTime.ts new file mode 100644 index 0000000000..a258752e5d --- /dev/null +++ b/libraries/rush-client-core/src/ProcessStartTime.ts @@ -0,0 +1,67 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import { performance } from 'node:perf_hooks'; + +// USER_HZ is fixed at 100 on mainstream Linux ABIs; it is verified against this process before use. +const USER_HZ: number = 100; +const PROC_STAT_START_TIME_FIELD: number = 22; +const MAX_CALIBRATION_ERROR_MS: number = 1000; +/** A process that began this long after a record was written cannot be the record's writer. */ +const MIN_REUSE_MARGIN_MS: number = 2000; + +/** + * Returns the wall-clock start time of `pid`, or `undefined` when it cannot be determined reliably. + * Only Linux `/proc` is supported; other platforms always return `undefined` (unknown). + */ +export function tryGetProcessStartTimeMs(pid: number): number | undefined { + if (process.platform !== 'linux') return undefined; + const uptimeSeconds: number | undefined = readUptimeSeconds(); + const selfStartSeconds: number | undefined = readStartSeconds('self'); + if (uptimeSeconds === undefined || selfStartSeconds === undefined) return undefined; + const now: number = Date.now(); + const selfStartMs: number = now - (uptimeSeconds - selfStartSeconds) * 1000; + // Guards against an unexpected USER_HZ, a non-standard /proc, or a wall-clock jump since this process began. + if (Math.abs(selfStartMs - performance.timeOrigin) > MAX_CALIBRATION_ERROR_MS) return undefined; + const startSeconds: number | undefined = readStartSeconds(pid); + return startSeconds === undefined ? undefined : now - (uptimeSeconds - startSeconds) * 1000; +} + +/** + * True only when `pid` provably started after `recordedAt`, so it cannot be the process that wrote a record + * at that time (the PID was reused). Unknown start times return false. + */ +export function isProcessStartedAfter(pid: number, recordedAt: string): boolean { + const recordedMs: number = Date.parse(recordedAt); + if (!Number.isFinite(recordedMs)) return false; + const startMs: number | undefined = tryGetProcessStartTimeMs(pid); + return startMs !== undefined && startMs > recordedMs + MIN_REUSE_MARGIN_MS; +} + +function readUptimeSeconds(): number | undefined { + try { + const uptime: number = Number.parseFloat(fs.readFileSync('/proc/uptime', 'utf8').split(' ')[0]); + return Number.isFinite(uptime) ? uptime : undefined; + } catch { + return undefined; + } +} + +function readStartSeconds(pid: number | 'self'): number | undefined { + let stat: string; + try { + stat = fs.readFileSync(`/proc/${pid}/stat`, 'utf8'); + } catch { + return undefined; + } + // The command name (field 2) may contain spaces and parentheses; fields after it never do. + const commandEnd: number = stat.lastIndexOf(')'); + if (commandEnd < 0) return undefined; + const fields: string[] = stat + .slice(commandEnd + 1) + .trim() + .split(' '); + const jiffies: number = Number(fields[PROC_STAT_START_TIME_FIELD - 3]); + return Number.isSafeInteger(jiffies) && jiffies >= 0 ? jiffies / USER_HZ : undefined; +} diff --git a/libraries/rush-client-core/src/connectOrStartDaemon.ts b/libraries/rush-client-core/src/connectOrStartDaemon.ts index 866bde97c4..83cc47ddee 100644 --- a/libraries/rush-client-core/src/connectOrStartDaemon.ts +++ b/libraries/rush-client-core/src/connectOrStartDaemon.ts @@ -21,6 +21,15 @@ import { import { DaemonClient, type IDaemonClientConnectOptions } from './DaemonClient'; import { DaemonClientError } from './DaemonClientError'; import { getDaemonLogFilePath } from './DaemonLogFile'; +import { + DAEMON_RESET_HINT, + hasErrorCode, + isDaemonOwnership, + isOwnerProcessAlive, + readDaemonOwnership, + reclaimAbandonedOwnershipAsync, + type DaemonOwnership +} from './DaemonOwnership'; import { getDaemonStartupFilePath, reserveDaemonStartup, @@ -127,7 +136,7 @@ async function startDaemonAsync( if (Date.now() >= deadline) { throw startupError( options, - `has an unresolved startup handoff at ${getDaemonStartupFilePath(options.paths)}; refusing another launch` + `has an unresolved startup handoff at ${getDaemonStartupFilePath(options.paths)}; refusing another launch. ${DAEMON_RESET_HINT}` ); } await delayAsync(Math.min(100, Math.max(1, deadline - Date.now())), undefined, { @@ -141,7 +150,7 @@ async function startDaemonAsync( const handoff: DaemonClient | undefined = await waitForHandoffAsync(options, deadline); if (handoff) return handoff; if (Date.now() >= deadline) throw startupError(options, 'exceeded its deadline before reclaim'); - assertNoLiveOwner(options.paths); + await reclaimAbandonedOwnershipAsync(options.paths); await reclaimStaleDaemonAsync(options.paths); if (Date.now() >= deadline) throw startupError(options, 'exceeded its deadline before spawn'); options.abortSignal?.throwIfAborted(); @@ -290,8 +299,13 @@ async function waitForHandoffAsync( let backoffMs: number = 50; while (Date.now() < deadline) { const owner: IDaemonLockfile | undefined = readDaemonLockfile(options.paths.lockfilePath); - // Only wait on a fully published endpoint; malformed or ambiguous ownership still fails closed. - if (!owner || owner.socketPath !== options.paths.socketPath || !isProcessAlive(owner.pid)) + // Only wait on a fully published endpoint; malformed or ambiguous ownership is resolved by reclaim. + if ( + !owner || + !isDaemonOwnership(owner) || + owner.socketPath !== options.paths.socketPath || + !isOwnerProcessAlive(owner) + ) return undefined; await delayAsync(Math.min(backoffMs, Math.max(1, deadline - Date.now())), undefined, { signal: options.abortSignal @@ -305,57 +319,11 @@ async function waitForHandoffAsync( } function assertNoLiveOwner(paths: IDaemonPaths): void { - const owner: Pick | undefined = readDaemonOwnership( - paths.lockfilePath - ); - if (!owner) { - if (process.platform !== 'win32' && fs.existsSync(paths.socketPath)) { - throw new DaemonClientError( - 'startupFailed', - `Socket ${paths.socketPath} has no ownership record; refusing automatic reclaim.` - ); - } - return; - } - if (!isProcessAlive(owner.pid)) return; + const owner: DaemonOwnership | undefined = readDaemonOwnership(paths.lockfilePath); + if (!owner || !isOwnerProcessAlive(owner)) return; throw new DaemonClientError( 'startupFailed', - `PID ${owner.pid} still exists but the daemon is not ready. It may be a reused PID; refusing to kill it or remove ${paths.lockfilePath}.` - ); -} - -function readDaemonOwnership(lockfilePath: string): Pick | undefined { - let record: unknown; - try { - record = JSON.parse(fs.readFileSync(lockfilePath, 'utf8')); - } catch (error) { - if (hasErrorCode(error, 'ENOENT')) return undefined; - throw new DaemonClientError( - 'startupFailed', - `Cannot safely read ${lockfilePath}; refusing automatic reclaim.`, - { cause: error } - ); - } - if (!isDaemonOwnership(record)) { - throw new DaemonClientError( - 'startupFailed', - `Invalid daemon ownership record in ${lockfilePath}; refusing automatic reclaim.` - ); - } - return { pid: record.pid, startedAt: record.startedAt }; -} - -function isDaemonOwnership(record: unknown): record is Pick { - return ( - typeof record === 'object' && - record !== null && - 'pid' in record && - typeof record.pid === 'number' && - Number.isSafeInteger(record.pid) && - record.pid > 0 && - 'startedAt' in record && - typeof record.startedAt === 'string' && - Number.isFinite(Date.parse(record.startedAt)) + `PID ${owner.pid} still exists but the daemon is not ready. It may be a daemon that is still shutting down, or a reused PID; refusing to kill it or remove ${paths.lockfilePath}. ${DAEMON_RESET_HINT}` ); } @@ -408,7 +376,7 @@ async function waitForPreviousDaemonAsync( deadline, abortSignal ); - if (!owner || owner.pid !== pid || owner.startedAt !== startedAt || !isProcessAlive(owner.pid)) return; + if (!owner || owner.pid !== pid || owner.startedAt !== startedAt || !isOwnerProcessAlive(owner)) return; if (Date.now() >= deadline) { throw new DaemonClientError( 'timeout', @@ -422,16 +390,6 @@ async function waitForPreviousDaemonAsync( } } -function isProcessAlive(pid: number): boolean { - try { - process.kill(pid, 0); - return true; - } catch (error) { - if (hasErrorCode(error, 'ESRCH')) return false; - throw error; - } -} - async function spawnDetachedAsync( options: IConnectOrStartDaemonOptions, deadline: number @@ -540,7 +498,3 @@ function startupError(options: IConnectOrStartDaemonOptions, reason: string): Da `Daemon startup ${reason}. Inspect ${getDaemonLogFilePath(options.paths)} and retry, or use --no-daemon.` ); } - -function hasErrorCode(error: unknown, code: string): boolean { - return typeof error === 'object' && error !== null && 'code' in error && error.code === code; -} diff --git a/libraries/rush-client-core/src/index.ts b/libraries/rush-client-core/src/index.ts index b2962188b7..7597de72ab 100644 --- a/libraries/rush-client-core/src/index.ts +++ b/libraries/rush-client-core/src/index.ts @@ -15,6 +15,11 @@ export { } from './DaemonClient'; export { DaemonClientError, type DaemonClientErrorCode } from './DaemonClientError'; export { getDaemonLogFilePath } from './DaemonLogFile'; +export { + resetDaemonArtifactsAsync, + type IDaemonArtifactResetOptions, + type IDaemonArtifactResetResult +} from './DaemonOwnership'; export { executeWithDaemonRestartAsync } from './executeWithDaemonRestart'; export { connectOrStartDaemonAsync, diff --git a/libraries/rush-client-core/src/test/ProcessStartTime.test.ts b/libraries/rush-client-core/src/test/ProcessStartTime.test.ts new file mode 100644 index 0000000000..a7a9ff4418 --- /dev/null +++ b/libraries/rush-client-core/src/test/ProcessStartTime.test.ts @@ -0,0 +1,31 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import { performance } from 'node:perf_hooks'; + +import { isProcessStartedAfter, tryGetProcessStartTimeMs } from '../ProcessStartTime'; + +const linuxIt: typeof it = process.platform === 'linux' ? it : it.skip; + +describe('process start time', () => { + linuxIt('estimates this process start time from /proc', () => { + const startMs: number | undefined = tryGetProcessStartTimeMs(process.pid); + expect(startMs).toEqual(expect.any(Number)); + expect(Math.abs(startMs! - performance.timeOrigin)).toBeLessThan(1000); + }); + + linuxIt('detects a PID whose process started after a record was written', () => { + expect(isProcessStartedAfter(process.pid, new Date(Date.now() - 3600000).toISOString())).toBe(true); + expect(isProcessStartedAfter(process.pid, new Date().toISOString())).toBe(false); + }); + + it('treats unknown start times and invalid timestamps as not provably reused', async () => { + const exited: ChildProcess = spawn(process.execPath, ['-e', ''], { stdio: 'ignore' }); + await once(exited, 'close'); + expect(tryGetProcessStartTimeMs(exited.pid!)).toBeUndefined(); + expect(isProcessStartedAfter(exited.pid!, new Date(0).toISOString())).toBe(false); + expect(isProcessStartedAfter(process.pid, 'not a timestamp')).toBe(false); + }); +}); diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index dc4275cd74..fc623f9fb3 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -4,16 +4,22 @@ import { spawn, type ChildProcess } from 'node:child_process'; import { once } from 'node:events'; import * as fs from 'node:fs'; +import * as net from 'node:net'; import * as os from 'node:os'; import * as path from 'node:path'; import { setTimeout as delayAsync } from 'node:timers/promises'; import { FileSystem } from '@rushstack/node-core-library'; -import type { IDaemonLockfile, IDaemonPaths } from '@rushstack/rush-daemon-transport'; +import { + readDaemonLockfile, + type IDaemonLockfile, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; import { DaemonClient } from '../DaemonClient'; import { captureDaemonRequest } from '../captureDaemonRequest'; import { getDaemonLogFilePath } from '../DaemonLogFile'; +import { resetDaemonArtifactsAsync } from '../DaemonOwnership'; import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from '../connectOrStartDaemon'; import { executeWithDaemonRestartAsync } from '../executeWithDaemonRestart'; import { getDaemonStartupFilePath } from '../DaemonStartup'; @@ -98,6 +104,19 @@ describe('detached daemon startup', () => { }; } + async function leaveStaleSocketAsync(): Promise { + // A listener killed without cleanup leaves a bound-nowhere socket file behind. + const child: ChildProcess = spawn( + process.execPath, + [ + '-e', + `require('net').createServer().listen(${JSON.stringify(paths.socketPath)}, () => process.kill(process.pid, 'SIGKILL'))` + ], + { stdio: 'ignore' } + ); + await once(child, 'close'); + } + async function killStarterBeforeBindAsync(): Promise { fs.writeFileSync(path.join(folder, 'hold-prebind'), ''); const starter = startClient(); @@ -156,6 +175,7 @@ describe('detached daemon startup', () => { code: 1, stderr: expect.stringContaining('unresolved startup handoff') }); + expect((await result).stderr).toContain('daemon stop --force'); expect(fs.readFileSync(startupPath, 'utf8')).toBe(contents); expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); }); @@ -603,19 +623,134 @@ describe('detached daemon startup', () => { } ); - it('never reclaims a live or reused PID', async () => { + it('reclaims a parseable record with an invalid timestamp without waiting out the deadline', async () => { + const record: string = JSON.stringify({ + pid: process.pid, + protocolVersion: { major: 0, minor: 6 }, + startedAt: 'invalid', + socketPath: paths.socketPath + }); + fs.writeFileSync(paths.lockfilePath, record); + const started: number = Date.now(); + const client = await connectOrStartDaemonAsync(options); + await client.closeAsync(); + expect(Date.now() - started).toBeLessThan(options.startupTimeoutMs! - 2000); + expect(readDaemonLockfile(paths.lockfilePath)?.pid).not.toBe(process.pid); + }); + + (process.platform === 'win32' ? it.skip : it)( + 'force reset waits for a listener to release the endpoint', + async () => { + fs.writeFileSync(getDaemonStartupFilePath(paths), 'abandoned'); + const listener: net.Server = net.createServer((socket) => socket.destroy()); + await new Promise((resolve) => listener.listen(paths.socketPath, resolve)); + await expect(resetDaemonArtifactsAsync(paths)).rejects.toThrow('still listening'); + const closing: NodeJS.Timeout = setTimeout(() => listener.close(), 300); + try { + expect(await resetDaemonArtifactsAsync(paths, { waitTimeoutMs: 5000 })).toEqual({ + removedPaths: [getDaemonStartupFilePath(paths)] + }); + } finally { + clearTimeout(closing); + listener.close(); + } + } + ); + + it('never reclaims a live PID that may still own the record', async () => { const record: string = JSON.stringify({ pid: process.pid, startedAt: new Date().toISOString() }); fs.writeFileSync(paths.lockfilePath, record); - await expect(connectOrStartDaemonAsync(options)).rejects.toThrow('may be a reused PID'); + await expect(connectOrStartDaemonAsync(options)).rejects.toThrow('or a reused PID'); + await expect(connectOrStartDaemonAsync(options)).rejects.toThrow('daemon stop --force'); + expect(fs.readFileSync(paths.lockfilePath, 'utf8')).toBe(record); + await expect(resetDaemonArtifactsAsync(paths)).rejects.toThrow(`PID ${process.pid} still owns`); expect(fs.readFileSync(paths.lockfilePath, 'utf8')).toBe(record); }); - it('fails closed on corrupt ownership records', async () => { + it('reclaims a corrupt ownership record once the endpoint refuses connections', async () => { fs.writeFileSync(paths.lockfilePath, 'not json'); - await expect(connectOrStartDaemonAsync(options)).rejects.toThrow('refusing automatic reclaim'); - expect(fs.readFileSync(paths.lockfilePath, 'utf8')).toBe('not json'); + const client = await connectOrStartDaemonAsync(options); + await client.closeAsync(); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + expect(readDaemonLockfile(paths.lockfilePath)?.pid).toBe( + Number(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim()) + ); }); + it('fails closed on a corrupt ownership record while something still listens', async () => { + fs.writeFileSync(paths.lockfilePath, 'not json'); + const listener: net.Server = net.createServer((socket) => socket.destroy()); + await new Promise((resolve) => listener.listen(paths.socketPath, resolve)); + try { + await expect(connectOrStartDaemonAsync({ ...options, startupTimeoutMs: 2000 })).rejects.toThrow( + 'did not refuse a connection' + ); + await expect(resetDaemonArtifactsAsync(paths)).rejects.toThrow('still listening'); + expect(fs.readFileSync(paths.lockfilePath, 'utf8')).toBe('not json'); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + } finally { + await new Promise((resolve) => listener.close(() => resolve())); + } + }); + + (process.platform === 'win32' ? it.skip : it)( + 'reclaims a socket without an ownership record once it refuses connections', + async () => { + await leaveStaleSocketAsync(); + expect(fs.existsSync(paths.socketPath)).toBe(true); + const client = await connectOrStartDaemonAsync(options); + await client.closeAsync(); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + } + ); + + (process.platform === 'linux' ? it : it.skip)( + 'reclaims a record whose live PID started after the record was written', + async () => { + const unrelated: ChildProcess = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { + stdio: 'ignore' + }); + await once(unrelated, 'spawn'); + try { + await leaveStaleSocketAsync(); + fs.writeFileSync( + paths.lockfilePath, + JSON.stringify({ + pid: unrelated.pid, + protocolVersion: { major: 0, minor: 6 }, + startedAt: new Date(Date.now() - 3600000).toISOString(), + socketPath: paths.socketPath + }) + ); + const started: number = Date.now(); + const client = await connectOrStartDaemonAsync(options); + await client.closeAsync(); + // Not the full startup deadline spent polling the unrelated process. + expect(Date.now() - started).toBeLessThan(options.startupTimeoutMs! - 2000); + expect(unrelated.exitCode).toBeNull(); + expect(readDaemonLockfile(paths.lockfilePath)?.pid).not.toBe(unrelated.pid); + } finally { + const closed: Promise = once(unrelated, 'close'); + unrelated.kill('SIGKILL'); + await closed; + } + } + ); + + (process.platform === 'win32' ? it.skip : it)( + 'force reset removes stale artifacts without starting a daemon', + async () => { + await leaveStaleSocketAsync(); + fs.writeFileSync(paths.lockfilePath, 'garbage{'); + fs.writeFileSync(getDaemonStartupFilePath(paths), 'abandoned'); + expect(await resetDaemonArtifactsAsync(paths)).toEqual({ + removedPaths: [paths.lockfilePath, getDaemonStartupFilePath(paths), paths.socketPath] + }); + expect(await resetDaemonArtifactsAsync(paths)).toEqual({ removedPaths: [] }); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + } + ); + it('starts after ownership release even while the original process remains alive', async () => { const client = await connectOrStartDaemonAsync({ ...options, From 582a57442cfd392584e91a3158f367e71c0c521e Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:13:42 -0700 Subject: [PATCH 002/265] [rush-daemon] Arbitrate environment restarts and bound client restart retries (#6079) * [rush-daemon] Arbitrate environment restarts and bound client restart retries Fixes #6057 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Report restart-drain admission errors independently of scheduler error mapping Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-client-core] Bound restart hand-offs by the admission deadline; arbitrate graph-control restarts; document bounded retries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-client-core] Allow the admission deadline to expire before resubmitting to the last successor in the restart-always test Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- ...-restart-arbitration_2026-09-24-01-45.json | 10 + ...-restart-arbitration_2026-09-24-18-10.json | 10 + ...-restart-arbitration_2026-09-24-01-45.json | 10 + common/reviews/api/rush-client-core.api.md | 2 +- libraries/rush-client-core/README.md | 17 +- .../rush-client-core/src/DaemonClient.ts | 6 +- .../src/executeWithDaemonRestart.ts | 173 ++++++++++++------ .../src/test/connectOrStartDaemon.test.ts | 75 ++++++-- .../src/test/fixtures/daemon.ts | 8 +- .../src/DaemonCommandResult.ts | 3 +- .../src/WorkspaceRequestAdmission.ts | 14 ++ .../src/WorkspaceRequestLifecycle.ts | 20 +- .../src/WorkspaceRestartArbiter.ts | 125 +++++++++++++ .../src/test/WorkspaceRestartArbiter.test.ts | 75 ++++++++ .../test/WorkspaceRestartArbitration.test.ts | 82 +++++++++ 15 files changed, 547 insertions(+), 83 deletions(-) create mode 100644 common/changes/@rushstack/rush-client-core/fix-restart-arbitration_2026-09-24-01-45.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/fix-restart-arbitration_2026-09-24-18-10.json create mode 100644 common/changes/@rushstack/rush-daemon/fix-restart-arbitration_2026-09-24-01-45.json create mode 100644 libraries/rush-daemon/src/WorkspaceRestartArbiter.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts diff --git a/common/changes/@rushstack/rush-client-core/fix-restart-arbitration_2026-09-24-01-45.json b/common/changes/@rushstack/rush-client-core/fix-restart-arbitration_2026-09-24-01-45.json new file mode 100644 index 0000000000..06c96fc736 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/fix-restart-arbitration_2026-09-24-01-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Retry daemon restarts with jittered backoff inside the admission deadline and return a `restartRetriesExhausted` fallback outcome instead of failing when successors keep restarting for other environments.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/fix-restart-arbitration_2026-09-24-18-10.json b/common/changes/@rushstack/rush-daemon-protocol/fix-restart-arbitration_2026-09-24-18-10.json new file mode 100644 index 0000000000..a61beef311 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/fix-restart-arbitration_2026-09-24-18-10.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Document that clients may retry `retryAfterRestart` a bounded number of times within the admission deadline.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon-protocol" +} diff --git a/common/changes/@rushstack/rush-daemon/fix-restart-arbitration_2026-09-24-01-45.json b/common/changes/@rushstack/rush-daemon/fix-restart-arbitration_2026-09-24-01-45.json new file mode 100644 index 0000000000..69802e647c --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/fix-restart-arbitration_2026-09-24-01-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Queue a request whose environment requires a process restart until every queued or in-flight request that matches the running process has drained, so concurrent clients with different environments no longer preempt each other.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index a731a4a1d2..ed0f2f1813 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -49,7 +49,7 @@ export type DaemonClientOutcome = { readonly result: IDaemonCommandResult; } | { readonly kind: 'fallback'; - readonly reason: 'unsupported' | 'controllingTerminalRequired' | 'stdinEndUnsupported'; + readonly reason: 'unsupported' | 'controllingTerminalRequired' | 'stdinEndUnsupported' | 'restartRetriesExhausted'; readonly message?: string; } | { readonly kind: 'rejected'; diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 86142f10ee..a91c95b331 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -24,19 +24,22 @@ or admitting stdin: that is a protocol error, not permission to replay the comma Abort signals send `requestCancel`, then wait for the result; cancellation has a bounded grace period. Disconnects, protocol errors and sink failures are errors, never reasons to replay possibly executed work. Only pre-execution `unsupported`, -`controllingTerminalRequired`, and `stdinEndUnsupported` outcomes permit fallback. Raw-mode changes are +`controllingTerminalRequired`, `stdinEndUnsupported`, and `restartRetriesExhausted` outcomes permit fallback. Raw-mode changes are acknowledged only after applying them. Input listeners and raw state are restored on success, cancellation, disconnect and failure. No resize messages are sent. `executeWithDaemonRestartAsync(readyClient, connectionOptions, executionOptions)` -adds one bounded retry for an explicit `retryAfterRestart: true` result. It captures -the endpoint's PID/start identity before sending, requires protocol 0.10, waits for -that ownership to be released, and reconnects through the same startup mutex. +retries an explicit `retryAfterRestart: true` result a bounded number of times. Before +each hand-off it captures the endpoint's PID/start identity, requires protocol 0.10, +waits for that ownership to be released, and reconnects through the same startup mutex. +Retries after the first use jittered backoff, and the backoff, the successor hand-off +and the resubmitted request all share the request's admission deadline. The original immutable request and unread input are preserved. Output, events, terminal control, or stdin admission forbid retry, as do connection loss and plain -error messages. A second restart result fails explicitly. Cancellation stops waiting -without killing a daemon. Disabling auto-start still permits waiting for a -host-started successor, but never lets the client spawn one. +error messages. When the retry bound or the admission deadline is exhausted, it +returns a `restartRetriesExhausted` fallback outcome so the caller can run in-process. +Cancellation stops waiting without killing a daemon. Disabling auto-start still +permits waiting for a host-started successor, but never lets the client spawn one. `connectOrStartDaemonAsync()` accepts an **explicit, version-selected** executable, arguments, environment and cwd. It does not discover or install a Rush version. diff --git a/libraries/rush-client-core/src/DaemonClient.ts b/libraries/rush-client-core/src/DaemonClient.ts index 012b215cf3..911021c50c 100644 --- a/libraries/rush-client-core/src/DaemonClient.ts +++ b/libraries/rush-client-core/src/DaemonClient.ts @@ -67,7 +67,11 @@ export type DaemonClientOutcome = | { readonly kind: 'result'; readonly result: IDaemonCommandResult } | { readonly kind: 'fallback'; - readonly reason: 'unsupported' | 'controllingTerminalRequired' | 'stdinEndUnsupported'; + readonly reason: + | 'unsupported' + | 'controllingTerminalRequired' + | 'stdinEndUnsupported' + | 'restartRetriesExhausted'; readonly message?: string; } | { readonly kind: 'rejected'; readonly rejection: IDaemonRequestRejectedMessage['payload'] }; diff --git a/libraries/rush-client-core/src/executeWithDaemonRestart.ts b/libraries/rush-client-core/src/executeWithDaemonRestart.ts index 2299d55e27..e0d8563648 100644 --- a/libraries/rush-client-core/src/executeWithDaemonRestart.ts +++ b/libraries/rush-client-core/src/executeWithDaemonRestart.ts @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import { setTimeout as delayAsync } from 'node:timers/promises'; + import { DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR } from '@rushstack/rush-daemon-protocol'; import { readDaemonLockfile, type IDaemonLockfile } from '@rushstack/rush-daemon-transport'; @@ -10,8 +12,20 @@ import type { DaemonClient, DaemonClientOutcome, IDaemonClientExecuteOptions } f import { DaemonClientError } from './DaemonClientError'; /** - * Executes on a ready client, retrying once only for a typed pre-execution restart. + * The maximum number of successors a single request follows. Each restart serves at least one other + * environment first, so this bounds the wait when several environments share one workspace daemon. + */ +const MAX_RESTART_RETRIES: number = 6; +const RETRY_JITTER_BASE_MS: number = 50; +const RETRY_JITTER_MAX_MS: number = 1000; +/** Matches the default of {@link IConnectOrStartDaemonOptions.startupTimeoutMs}. */ +const DEFAULT_STARTUP_TIMEOUT_MS: number = 15000; + +/** + * Executes on a ready client, retrying only for a typed pre-execution restart. * Preserves the original request, callbacks and unread input; never retries connection loss. + * Restarts are retried with jittered backoff inside the request's admission deadline; once the + * retries or the deadline are exhausted, a `fallback` outcome lets the caller run in-process instead. * The connection options must select the request's expected daemon and startup environment. * @beta */ @@ -25,64 +39,117 @@ export async function executeWithDaemonRestartAsync( execution.abortSignal && connection.abortSignal ? AbortSignal.any([execution.abortSignal, connection.abortSignal]) : (execution.abortSignal ?? connection.abortSignal); + const waitTimeoutMs: number | undefined = execution.request.admission?.waitTimeoutMs; + let owner: IDaemonLockfile | undefined = await attestOwnerAsync(client, connection); + let outcome: DaemonClientOutcome = await client.executeAsync({ ...execution, abortSignal }); + let previous: DaemonClient | undefined; + try { + for (let retry: number = 1; outcome.kind === 'result' && outcome.result.retryAfterRestart; retry++) { + if (!owner) { + throw new DaemonClientError( + 'startupFailed', + 'Cannot attest the restarting daemon ownership; the request was not retried.' + ); + } + if (abortSignal?.aborted) return abortedOutcome(execution); + const getRemainingMs = (): number | undefined => + waitTimeoutMs === undefined ? undefined : waitTimeoutMs - (Date.now() - startedAt); + if (retry > MAX_RESTART_RETRIES || isExpired(getRemainingMs())) { + return restartExhaustedOutcome(retry - 1); + } + let successor: DaemonClient; + let boundedByAdmission: boolean = false; + try { + // The first retry follows the planned successor immediately; later ones back off with jitter so + // clients whose environments differ do not reach each new successor in lockstep. + if (retry > 1) { + await delayAsync(getRetryDelayMs(retry, getRemainingMs()), undefined, { signal: abortSignal }); + } + const remainingMs: number | undefined = getRemainingMs(); + if (isExpired(remainingMs)) return restartExhaustedOutcome(retry - 1); + const startupTimeoutMs: number = connection.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS; + // The successor handoff shares the request's admission deadline rather than starting a fresh one. + boundedByAdmission = remainingMs !== undefined && remainingMs < startupTimeoutMs; + successor = await connectOrStartDaemonAsync({ + ...connection, + startupTimeoutMs: boundedByAdmission ? Math.max(1, Math.ceil(remainingMs!)) : startupTimeoutMs, + previousDaemon: { pid: owner.pid, startedAt: owner.startedAt }, + abortSignal + }); + } catch (error) { + if ( + abortSignal?.aborted && + (error === abortSignal.reason || + (typeof error === 'object' && error !== null && 'code' in error && error.code === 'ABORT_ERR')) + ) { + return abortedOutcome(execution); + } + // A startup error after the admission deadline expired is the deadline, not a new failure mode. + if (boundedByAdmission && error instanceof DaemonClientError && isExpired(getRemainingMs())) { + return restartExhaustedOutcome(retry); + } + throw error; + } finally { + await previous?.closeAsync().catch(() => undefined); + previous = undefined; + } + previous = successor; + const remainingMs: number | undefined = getRemainingMs(); + if (isExpired(remainingMs)) return restartExhaustedOutcome(retry); + owner = await attestOwnerAsync(successor, connection); + outcome = await successor.executeAsync({ + ...execution, + abortSignal, + request: + remainingMs === undefined + ? execution.request + : captureDaemonRequest({ + ...execution.request, + admission: { ...execution.request.admission, waitTimeoutMs: Math.floor(remainingMs) } + }) + }); + } + return outcome; + } finally { + await previous?.closeAsync().catch(() => undefined); + } +} + +function isExpired(remainingMs: number | undefined): boolean { + return remainingMs !== undefined && remainingMs <= 0; +} +/** Returns the published ownership record only when it names the connected, restart-capable process. */ +async function attestOwnerAsync( + client: DaemonClient, + connection: IConnectOrStartDaemonOptions +): Promise { const owner: IDaemonLockfile | undefined = readDaemonLockfile(connection.paths.lockfilePath); const { pid } = await client.status; - const attested: boolean = - client.protocolVersion.minor >= DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR && + return client.protocolVersion.minor >= DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR && owner !== undefined && owner.pid === pid && owner.socketPath === connection.paths.socketPath && Number.isSafeInteger(owner.pid) && owner.pid > 0 && - Number.isFinite(Date.parse(owner.startedAt)); - const outcome: DaemonClientOutcome = await client.executeAsync({ ...execution, abortSignal }); - if (outcome.kind !== 'result' || !outcome.result.retryAfterRestart) return outcome; - if (!attested || !owner) { - throw new DaemonClientError( - 'startupFailed', - 'Cannot attest the restarting daemon ownership; the request was not retried.' - ); - } - if (abortSignal?.aborted) return abortedOutcome(execution); - let successor: DaemonClient; - try { - successor = await connectOrStartDaemonAsync({ - ...connection, - previousDaemon: { pid: owner.pid, startedAt: owner.startedAt }, - abortSignal - }); - } catch (error) { - if ( - abortSignal?.aborted && - (error === abortSignal.reason || - (typeof error === 'object' && error !== null && 'code' in error && error.code === 'ABORT_ERR')) - ) { - return abortedOutcome(execution); - } - throw error; - } - const waitTimeoutMs: number | undefined = execution.request.admission?.waitTimeoutMs; - const result: DaemonClientOutcome = await successor.executeAsync({ - ...execution, - abortSignal, - request: - waitTimeoutMs === undefined - ? execution.request - : captureDaemonRequest({ - ...execution.request, - admission: { - ...execution.request.admission, - waitTimeoutMs: Math.max(0, waitTimeoutMs - (Date.now() - startedAt)) - } - }) - }); - if (result.kind === 'result' && result.result.retryAfterRestart) { - throw new DaemonClientError( - 'startupFailed', - 'The successor requested another restart; the single safe retry was exhausted.' - ); - } - return result; + Number.isFinite(Date.parse(owner.startedAt)) + ? owner + : undefined; +} + +function getRetryDelayMs(retry: number, remainingMs: number | undefined): number { + const ceilingMs: number = Math.min(RETRY_JITTER_MAX_MS, RETRY_JITTER_BASE_MS * 2 ** (retry - 1)); + const delayMs: number = Math.floor(ceilingMs / 2 + (Math.random() * ceilingMs) / 2); + return remainingMs === undefined ? delayMs : Math.max(0, Math.min(delayMs, remainingMs - 1)); +} + +function restartExhaustedOutcome(restarts: number): DaemonClientOutcome { + return { + kind: 'fallback', + reason: 'restartRetriesExhausted', + message: `The daemon was still restarting for other environments after ${restarts} ${ + restarts === 1 ? 'restart' : 'restarts' + }; no operation was started by the daemon` + }; } function abortedOutcome(execution: IDaemonClientExecuteOptions): DaemonClientOutcome { @@ -90,4 +157,4 @@ function abortedOutcome(execution: IDaemonClientExecuteOptions): DaemonClientOut kind: 'result', result: { requestId: execution.request.requestId, exitCode: 130, outcome: 'aborted', aborted: true } }; -} +} \ No newline at end of file diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index fc623f9fb3..e49f287909 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -400,9 +400,11 @@ describe('detached daemon startup', () => { expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(2); }); - it.each(['restart-once', 'restart-always', 'restart-held'])( + it.each(['restart-once', 'restart-twice', 'restart-always', 'restart-held'])( 'retries only the typed pre-execution result for %s after ownership release', async (mode) => { + // restart-always exhausts the deadline; the others need room for loaded CI machines. + const waitTimeoutMs: number = mode === 'restart-always' ? 1000 : 10000; const connection: IConnectOrStartDaemonOptions = { ...options, startCommand: { ...options.startCommand!, args: [...options.startCommand!.args, 'fixture', mode] } @@ -415,7 +417,7 @@ describe('detached daemon startup', () => { cwd: folder, environment: {}, terminal: { isTTY: false, supportsColor: false }, - admission: { waitTimeoutMs: 1000 } + admission: { waitTimeoutMs } }); const pending = executeWithDaemonRestartAsync( client, @@ -425,24 +427,36 @@ describe('detached daemon startup', () => { }, { request } ); - if (mode === 'restart-once') { + if (mode === 'restart-once' || mode === 'restart-twice') { + const restarts: number = mode === 'restart-once' ? 1 : 2; expect(await pending).toMatchObject({ kind: 'result', result: { exitCode: 0 } }); const waits = fs.readFileSync(path.join(folder, 'waits'), 'utf8').trim().split('\n').map(Number); - expect(waits[0]).toBe(1000); - expect(waits[1]).toBeLessThan(1000); - expect(waits[1]).toBeGreaterThanOrEqual(0); - expect(request.admission?.waitTimeoutMs).toBe(1000); - } else { - await expect(pending).rejects.toThrow( - mode === 'restart-held' ? 'previous daemon still owns' : 'single safe retry was exhausted' + expect(waits).toHaveLength(restarts + 1); + expect(waits[0]).toBe(waitTimeoutMs); + for (let index: number = 1; index <= restarts; index++) { + expect(waits[index]).toBeLessThanOrEqual(waits[index - 1]); + expect(waits[index]).toBeLessThan(waitTimeoutMs); + expect(waits[index]).toBeGreaterThanOrEqual(0); + } + expect(request.admission?.waitTimeoutMs).toBe(waitTimeoutMs); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength( + restarts + 1 ); + } else if (mode === 'restart-always') { + // Every successor asks again: bounded retries inside the admission deadline, then a fallback. + expect(await pending).toMatchObject({ kind: 'fallback', reason: 'restartRetriesExhausted' }); + const starts: number = fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n').length; + expect(starts).toBeGreaterThanOrEqual(2); + expect(starts).toBeLessThanOrEqual(7); + // The deadline may expire after the last successor started but before the request was resubmitted. + const requests: number = fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n').length; + expect(requests).toBeGreaterThanOrEqual(starts - 1); + expect(requests).toBeLessThanOrEqual(starts); + } else { + await expect(pending).rejects.toThrow('previous daemon still owns'); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n')).toHaveLength(1); } - expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength( - mode === 'restart-held' ? 1 : 2 - ); - expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n')).toHaveLength( - mode === 'restart-held' ? 1 : 2 - ); }, 15000 ); @@ -483,6 +497,35 @@ describe('detached daemon startup', () => { } ); + it('bounds the successor hand-off by the admission deadline instead of a fresh startup timeout', async () => { + const connection: IConnectOrStartDaemonOptions = { + ...options, + startupTimeoutMs: 7000, + startCommand: { + ...options.startCommand!, + args: [...options.startCommand!.args, 'fixture', 'restart-held'] + } + }; + const client = await connectOrStartDaemonAsync(connection); + const request = captureDaemonRequest({ + argv: ['test'], + commandName: 'test', + commandOrigin: 'custom', + cwd: folder, + environment: {}, + terminal: { isTTY: false, supportsColor: false }, + admission: { waitTimeoutMs: 500 } + }); + const startedAt: number = Date.now(); + expect(await executeWithDaemonRestartAsync(client, connection, { request })).toMatchObject({ + kind: 'fallback', + reason: 'restartRetriesExhausted' + }); + expect(Date.now() - startedAt).toBeLessThan(5000); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n')).toHaveLength(1); + }); + it('refuses restart retry if ownership was not attested before submitting', async () => { const connection: IConnectOrStartDaemonOptions = { ...options, diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index 6c00f3b323..e03e1cc12e 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -68,9 +68,13 @@ async function mainAsync(): Promise { if (message.payload.admission?.waitTimeoutMs !== undefined) { fs.appendFileSync(path.join(folder, 'waits'), `${message.payload.admission.waitTimeoutMs}\n`); } + const restartCount: number = fs.existsSync(path.join(folder, 'restarted')) + ? fs.readFileSync(path.join(folder, 'restarted'), 'utf8').length + : 0; const restart: boolean = restartMode !== undefined && - (restartMode !== 'restart-once' || !fs.existsSync(path.join(folder, 'restarted'))); + (restartMode !== 'restart-once' || restartCount < 1) && + (restartMode !== 'restart-twice' || restartCount < 2); await connection.sendFrameAsync({ kind: DaemonFrameType.controlJson, payload: encodeDaemonControlMessage({ @@ -85,7 +89,7 @@ async function mainAsync(): Promise { }) }); if (restart && restartMode !== 'restart-held') { - fs.writeFileSync(path.join(folder, 'restarted'), ''); + fs.appendFileSync(path.join(folder, 'restarted'), 'r'); await stopAsync(); } } else if (message.kind === 'shutdown') { diff --git a/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts b/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts index c3726e2b70..293776da32 100644 --- a/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts +++ b/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts @@ -18,7 +18,8 @@ export type DaemonCommandOutcome = 'success' | 'success-with-warning' | 'failure export interface IDaemonCommandResult { /** * Protocol 0.10: no execution or request IO occurred, and a successor has been selected. - * Retry at most once, after attested predecessor ownership release. Never infer this from an error. + * Retry only after attested predecessor ownership release, within the request's admission deadline and a + * small client-defined retry bound; then fall back instead of retrying. Never infer this from an error. */ readonly retryAfterRestart?: true; /** Whether cancellation or disconnect was observed, even if a cleanup failure determines the outcome. */ diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index 0863216263..a541976266 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -18,6 +18,7 @@ import { } from './RequestScheduler'; import type { IWorkspaceSession } from './WorkspaceSession'; import { assertWorkspaceRequestResourcesHealthy } from './WorkspaceRequestResources'; +import type { IWorkspaceRestartTicket, WorkspaceRestartArbiter } from './WorkspaceRestartArbiter'; export interface IRequestAdmissionClient { readonly abortSignal: AbortSignal; @@ -191,6 +192,19 @@ export class RequestAdmissionController { } } + /** Waits, within the same admission budget, until a restart would not preempt another request. */ + public async waitForRestartDrainAsync( + arbiter: WorkspaceRestartArbiter, + ticket: IWorkspaceRestartTicket + ): Promise { + // The arbiter reports its own admission errors, so this does not depend on the scheduler error mapping. + await arbiter.waitForDrainAsync(ticket, { + abortSignal: this.#abortController.signal, + noWait: this.#admission?.noWait, + waitTimeoutMs: this.#getRemainingWaitTimeoutMs() + }); + } + public dispose(): void { this.#client.abortSignal.removeEventListener('abort', this.#abortFromClient); } diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index da5ca8230a..adb9753ce6 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -49,6 +49,7 @@ import type { IWorkspaceProcessRestartPlan, IWorkspaceSuccessorLaunch } from './WorkspaceProcessRestart'; +import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from './WorkspaceRestartArbiter'; interface IExecutionState { began: boolean; @@ -106,6 +107,7 @@ class RestartPendingBeforeExecution extends Error { export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { readonly #options: IWorkspaceRequestLifecycleOptions; readonly #gate: RequestScheduler = new RequestScheduler(); + readonly #restartArbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); readonly #abortController: AbortController = new AbortController(); readonly #observers: Set = new Set(); readonly #terminal: Terminal = new Terminal(new NoOpTerminalProvider()); @@ -195,11 +197,13 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { client, requestId: envelope.requestId }); + // Long-lived observers are cancelled by a transition, so they never delay a restart. + const ticket: IWorkspaceRestartTicket | undefined = observer ? undefined : this.#restartArbiter.enter(); let generation: IPreparedGeneration | undefined; try { for (let attempt: number = 0; ; attempt++) { try { - generation = await this.#prepareAsync(envelope, client, admission); + generation = await this.#prepareAsync(envelope, client, admission, ticket); const requestEnvelope: IDaemonRequestEnvelope = { ...envelope, admission: admission.remainingAdmission @@ -272,6 +276,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } } } finally { + if (ticket) this.#restartArbiter.leave(ticket); admission.dispose(); if (observer) this.#observers.delete(observer); } @@ -281,6 +286,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { envelope: IDaemonRequestEnvelope, client: IDaemonRequestDispatchClient, admission: RequestAdmissionController, + ticket: IWorkspaceRestartTicket | undefined, admittedLease?: IRequestLease ): Promise { let lease: IRequestLease = @@ -316,6 +322,11 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { const currentTier: WorkspaceInputChangeTier = this.#classify(current, false); if (currentTier === WorkspaceInputChangeTier.Restart) { lease.release(); + if (ticket) { + // Like build requests, a graph-control restart must not preempt requests this process can serve. + await admission.waitForRestartDrainAsync(this.#restartArbiter, ticket); + if (this.#restartPending) throw new RestartPendingBeforeExecution(); + } this.#cancelObservers(); lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); session = await this.#options.provider.getSessionAsync(); @@ -408,12 +419,17 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } lease.release(); + if (tier === WorkspaceInputChangeTier.Restart && ticket) { + // Serve every queued or in-flight request that matches this process before restarting for another one. + await admission.waitForRestartDrainAsync(this.#restartArbiter, ticket); + if (this.#restartPending) throw new RestartPendingBeforeExecution(); + } if (this.#transitioning) { const shared: IRequestLease = await admission.acquireAsync( this.#gate, RequestExclusivityClass.SharedBuild ); - return await this.#prepareAsync(envelope, client, admission, shared); + return await this.#prepareAsync(envelope, client, admission, ticket, shared); } this.#transitioning = ownsTransition = true; this.#cancelObservers(); diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts new file mode 100644 index 0000000000..9fb26f6455 --- /dev/null +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -0,0 +1,125 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { RequestSchedulerError, RequestSchedulerErrorCode } from './RequestScheduler'; + +const MAX_TIMER_DELAY_MS: number = 0x7fffffff; + +/** One dispatched request tracked by a {@link WorkspaceRestartArbiter}. */ +export interface IWorkspaceRestartTicket { + readonly waitingForDrain: boolean; +} + +interface IMutableTicket { + waitingForDrain: boolean; + left: boolean; +} + +/** Options for {@link WorkspaceRestartArbiter.waitForDrainAsync}, supplied by request admission. */ +export interface IWorkspaceRestartDrainOptions { + readonly abortSignal: AbortSignal; + readonly noWait: boolean | undefined; + readonly waitTimeoutMs: number | undefined; +} + +/** + * Arbitrates process restarts between requests whose environments differ from the running daemon. + * A request that needs a restart waits until every other request this process can serve has finished, + * so a mismatched environment never preempts queued or in-flight work that matches the running process. + */ +export class WorkspaceRestartArbiter { + readonly #listeners: Set<() => void> = new Set(); + #servingCount: number = 0; + + /** The number of tracked requests that are not waiting for a restart. */ + public get servingCount(): number { + return this.#servingCount; + } + + public enter(): IWorkspaceRestartTicket { + this.#servingCount++; + const ticket: IMutableTicket = { waitingForDrain: false, left: false }; + return ticket; + } + + public leave(ticket: IWorkspaceRestartTicket): void { + const state: IMutableTicket = ticket as IMutableTicket; + if (state.left) return; + state.left = true; + if (!state.waitingForDrain) this.#decrement(); + } + + /** + * Waits until no other tracked request is still being served by this process, then counts the ticket + * as served again so concurrent restart candidates proceed one at a time. + */ + public async waitForDrainAsync( + ticket: IWorkspaceRestartTicket, + options: IWorkspaceRestartDrainOptions + ): Promise { + const state: IMutableTicket = ticket as IMutableTicket; + if (state.left || state.waitingForDrain) throw new Error('The restart ticket is not being served.'); + state.waitingForDrain = true; + this.#decrement(); + try { + const deadline: number | undefined = + options.waitTimeoutMs === undefined ? undefined : Date.now() + options.waitTimeoutMs; + while (this.#servingCount > 0) { + if (options.noWait) { + throw new RequestSchedulerError( + RequestSchedulerErrorCode.NoWait, + 'Another environment is still being served; the request did not wait for a restart.' + ); + } + await this.#waitForChangeAsync(options.abortSignal, deadline); + } + } finally { + state.waitingForDrain = false; + this.#servingCount++; + } + } + + #decrement(): void { + this.#servingCount--; + if (this.#servingCount === 0) { + for (const listener of Array.from(this.#listeners)) listener(); + } + } + + #waitForChangeAsync(abortSignal: AbortSignal, deadline: number | undefined): Promise { + return new Promise((resolve, reject) => { + let timer: ReturnType | undefined; + const unsubscribe: AbortController = new AbortController(); + const settle = (error?: RequestSchedulerError): void => { + this.#listeners.delete(settle); + unsubscribe.abort(); + if (timer) clearTimeout(timer); + if (error) reject(error); + else resolve(); + }; + const settleAborted = (): void => + settle( + new RequestSchedulerError(RequestSchedulerErrorCode.Aborted, 'The request was aborted before execution.') + ); + if (abortSignal.aborted) { + settleAborted(); + return; + } + this.#listeners.add(settle); + abortSignal.addEventListener('abort', settleAborted, { once: true, signal: unsubscribe.signal }); + if (deadline !== undefined) { + timer = setTimeout( + () => + settle( + new RequestSchedulerError( + RequestSchedulerErrorCode.WaitTimeout, + 'The request was not admitted before the daemon could restart for its environment. ' + + 'Use --wait-timeout or RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS to wait longer.' + ) + ), + Math.min(MAX_TIMER_DELAY_MS, Math.max(0, deadline - Date.now())) + ); + } + }); + } +} diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts new file mode 100644 index 0000000000..8ee1001739 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts @@ -0,0 +1,75 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestScheduler'; +import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from '../WorkspaceRestartArbiter'; + +const WAIT: { abortSignal: AbortSignal; noWait: undefined; waitTimeoutMs: undefined } = { + abortSignal: new AbortController().signal, + noWait: undefined, + waitTimeoutMs: undefined +}; + +async function isSettledAsync(promise: Promise): Promise { + let settled: boolean = false; + void promise.then( + () => (settled = true), + () => (settled = true) + ); + await new Promise((resolve) => setImmediate(resolve)); + return settled; +} + +describe(WorkspaceRestartArbiter.name, () => { + it('proceeds immediately when no other request is being served', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const ticket: IWorkspaceRestartTicket = arbiter.enter(); + await arbiter.waitForDrainAsync(ticket, WAIT); + expect(arbiter.servingCount).toBe(1); + arbiter.leave(ticket); + expect(arbiter.servingCount).toBe(0); + }); + + it('waits for served requests, including ones that arrive later, and admits candidates one at a time', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const first: IWorkspaceRestartTicket = arbiter.enter(); + const second: IWorkspaceRestartTicket = arbiter.enter(); + const firstWait: Promise = arbiter.waitForDrainAsync(first, WAIT); + const secondWait: Promise = arbiter.waitForDrainAsync(second, WAIT); + const late: IWorkspaceRestartTicket = arbiter.enter(); + arbiter.leave(serving); + expect(await isSettledAsync(firstWait)).toBe(false); + arbiter.leave(late); + await firstWait; + expect(await isSettledAsync(secondWait)).toBe(false); + arbiter.leave(first); + await secondWait; + arbiter.leave(second); + expect(arbiter.servingCount).toBe(0); + }); + + it.each([ + ['no-wait', RequestSchedulerErrorCode.NoWait], + ['timeout', RequestSchedulerErrorCode.WaitTimeout], + ['abort', RequestSchedulerErrorCode.Aborted] + ])('reports %s as an admission failure and restores its accounting', async (mode, code) => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const ticket: IWorkspaceRestartTicket = arbiter.enter(); + const abort: AbortController = new AbortController(); + const waiting: Promise = arbiter.waitForDrainAsync(ticket, { + abortSignal: abort.signal, + noWait: mode === 'no-wait' ? true : undefined, + waitTimeoutMs: mode === 'timeout' ? 10 : undefined + }); + if (mode === 'abort') abort.abort(); + const error: unknown = await waiting.catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(RequestSchedulerError); + expect((error as RequestSchedulerError).code).toBe(code); + expect(arbiter.servingCount).toBe(2); + arbiter.leave(ticket); + arbiter.leave(serving); + expect(arbiter.servingCount).toBe(0); + }); +}); \ No newline at end of file diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts new file mode 100644 index 0000000000..557c54ddc5 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts @@ -0,0 +1,82 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; + +import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import type { ITerminalExchange } from './DaemonRequestWireTestUtilities'; +import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; +import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; + +jest.setTimeout(60_000); + +it('queues a mismatched-environment restart until matching queued and in-flight requests drain', async () => { + const fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + // Project c holds its build open until the test removes the marker. + created.write('hold', ''); + created.write( + 'c/build.cjs', + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','c\\n');" + + "const t=setInterval(()=>{if(!fs.existsSync('../hold')){clearInterval(t);console.log('finished-c');}},20);" + ); + }); + const order: string[] = []; + const track = (name: string, exchange: Promise): Promise => + exchange.then((result) => { + order.push(name); + return result; + }); + try { + const before = await pongAsync(fixture); + const matching = track('matching', fixture.runAsync(['build', '--to', 'c', '--parallelism', '3'])); + const deadline: number = Date.now() + 30_000; + while (!fixture.runs().includes('c') && Date.now() < deadline) await delayAsync(20); + expect(fixture.runs()).toContain('c'); + + const mismatched = track( + 'mismatched', + fixture.runAsync(['build', '--to', 'b', '--parallelism', '3'], { + environment: { ...fixture.environment, RUSHD_RELOAD_TIER_TEST: 'changed' } + }) + ); + await delayAsync(1000); + // Arrives after the mismatched request; it must still be served by this process. + const lateMatching = track('late', fixture.runAsync(['build', '--to', 'c', '--parallelism', '3'])); + await delayAsync(1000); + expect(order).toEqual([]); + expect(fixture.host.workspaceStatus.lastReloadTier).not.toBe(WorkspaceInputChangeTier.Restart); + + fs.rmSync(path.join(fixture.folder, 'hold')); + const [held, late, restart] = await Promise.all([matching, lateMatching, mismatched]); + expect(held.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(held.terminal.payload).not.toHaveProperty('retryAfterRestart'); + expect(late.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(late.terminal.payload).not.toHaveProperty('retryAfterRestart'); + expect(restart.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true } + }); + expect(order[order.length - 1]).toBe('mismatched'); + + const restarted = await fixture.host.restartCompleted; + expect(restarted?.pid).not.toBe(before.pid); + expect((await pongAsync(fixture)).pid).toBe(restarted?.pid); + expect(fixture.runs()).not.toContain('b'); + } finally { + fs.rmSync(path.join(fixture.folder, 'hold'), { force: true }); + try { + await fixture.host.closeAsync(); + await fixture.host.restartCompleted; + } finally { + await stopSuccessorAsync(fixture.host.paths); + await fixture[Symbol.asyncDispose](); + } + } +}); \ No newline at end of file From 74d0cb4f27f1683bbd3e8105abbe198ecb537fb2 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:16:14 -0700 Subject: [PATCH 003/265] [rush-daemon] Use directory-level async watchers on Linux (#6089) * [rush-daemon] Use directory-level async watchers on Linux Fixes #6078 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Fix test lint warnings Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Address Linux tree watcher review feedback Forward unknown filenames, re-check exclusions after they load, re-register replaced directories, fail on non-transient walk errors, and keep the .rush/temp exclusion when rush-project.json cannot be loaded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../linux-tree-watcher_2026-09-24.json | 10 + libraries/rush-daemon/src/LinuxTreeWatcher.ts | 269 ++++++++++++++++ .../src/WorkspaceSessionFileWatcher.ts | 96 +++++- .../src/test/LinuxTreeWatcher.test.ts | 295 ++++++++++++++++++ .../test/WorkspaceSessionFileWatcher.test.ts | 25 +- 5 files changed, 678 insertions(+), 17 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/linux-tree-watcher_2026-09-24.json create mode 100644 libraries/rush-daemon/src/LinuxTreeWatcher.ts create mode 100644 libraries/rush-daemon/src/test/LinuxTreeWatcher.test.ts diff --git a/common/changes/@rushstack/rush-daemon/linux-tree-watcher_2026-09-24.json b/common/changes/@rushstack/rush-daemon/linux-tree-watcher_2026-09-24.json new file mode 100644 index 0000000000..c658db784c --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/linux-tree-watcher_2026-09-24.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "On Linux, observe watched projects with one non-recursive watch per directory created by an asynchronous walk instead of Node's recursive fs.watch emulation (one inotify watch per file, synchronous tree walk). Skip node_modules, .git, .rush/temp and declared output folders, follow directories created or removed later, tolerate directories that vanish during registration, and report inotify watch-limit exhaustion (ENOSPC) explicitly.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/LinuxTreeWatcher.ts b/libraries/rush-daemon/src/LinuxTreeWatcher.ts new file mode 100644 index 0000000000..062a330daa --- /dev/null +++ b/libraries/rush-daemon/src/LinuxTreeWatcher.ts @@ -0,0 +1,269 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { EventEmitter } from 'node:events'; + +/** Watches one directory without recursion. */ +export type DirectoryWatchFunction = ( + folderPath: string, + listener: fs.WatchListener +) => fs.FSWatcher; + +/** Options for {@link LinuxTreeWatcher}. */ +export interface ILinuxTreeWatcherOptions { + /** Absolute directory paths that are never observed, such as declared build output folders. */ + readonly getExcludedFolderPathsAsync?: () => Promise>; + /** + * Reports the root once the initial walk finishes, covering changes made to directories before they + * were registered. Callers that await {@link LinuxTreeWatcher.initialWalk} instead can leave this off. + */ + readonly reportInitialWalkCompletion?: boolean; + /** Test hook; defaults to a non-recursive `fs.watch`. */ + readonly watchDirectory?: DirectoryWatchFunction; +} + +/** Directory names that are never observed at any depth. */ +export const PRUNED_DIRECTORY_NAMES: ReadonlySet = new Set(['node_modules', '.git']); + +const TRANSIENT_ERROR_CODES: ReadonlySet = new Set(['ENOENT', 'ENOTDIR', 'EACCES', 'EPERM']); +const EMPTY_SET: ReadonlySet = new Set(); + +const defaultWatchDirectory: DirectoryWatchFunction = (folderPath, listener) => + fs.watch(folderPath, { encoding: 'utf8', persistent: true }, listener); + +/** + * A recursive watcher for Linux built from one non-recursive inotify watch per directory. + * + * @remarks + * Node's Linux emulation of `fs.watch(..., { recursive: true })` walks the tree synchronously on the event + * loop and registers one inotify watch per file, including `node_modules` and build outputs. A directory-level + * inotify watch already reports changes to the files it contains, so this watcher registers directories only, + * walks asynchronously, prunes `node_modules`, `.git` and caller-provided folders, follows directories that are + * created or removed later, and tolerates directories that disappear while they are being registered. + * + * The root is registered synchronously so a missing root fails like `fs.watch`. Each directory watch is created + * before the directory is listed, so a child created during the walk is reported by its parent and then walked. + * The object is `fs.FSWatcher`-compatible: it emits `error` and `close` and supports `ref`/`unref`. + */ +export class LinuxTreeWatcher extends EventEmitter { + readonly #root: string; + readonly #listener: fs.WatchListener; + readonly #watchDirectory: DirectoryWatchFunction; + readonly #watchers: Map = new Map(); + readonly #excludedFolderPaths: Promise>; + #resolvedExcludedFolderPaths: ReadonlySet = EMPTY_SET; + #closed: boolean = false; + #failed: boolean = false; + #isUnref: boolean = false; + + /** Resolves when the initial asynchronous walk has finished (or stopped). */ + public readonly initialWalk: Promise; + + public constructor(root: string, listener: fs.WatchListener, options: ILinuxTreeWatcherOptions = {}) { + super(); + this.#root = path.resolve(root); + this.#listener = listener; + this.#watchDirectory = options.watchDirectory ?? defaultWatchDirectory; + this.#excludedFolderPaths = loadExcludedFolderPathsAsync(options.getExcludedFolderPathsAsync); + this.#addDirectory(this.#root, true); + this.initialWalk = this.#walkAsync(this.#root).then(() => { + if (options.reportInitialWalkCompletion) this.#reportCoverage(this.#root); + }); + } + + /** The directories that currently hold an inotify watch. */ + public get watchedFolderPaths(): ReadonlySet { + return new Set(this.#watchers.keys()); + } + + public close(): void { + if (this.#closed) return; + this.#closed = true; + for (const watcher of this.#watchers.values()) watcher.close(); + this.#watchers.clear(); + process.nextTick(() => this.emit('close')); + } + + public ref(): this { + this.#isUnref = false; + for (const watcher of this.#watchers.values()) watcher.ref(); + return this; + } + + public unref(): this { + this.#isUnref = true; + for (const watcher of this.#watchers.values()) watcher.unref(); + return this; + } + + #isPruned(folderPath: string): boolean { + return ( + PRUNED_DIRECTORY_NAMES.has(path.basename(folderPath)) || this.#resolvedExcludedFolderPaths.has(folderPath) + ); + } + + /** Returns false when the directory vanished before it could be watched. */ + #addDirectory(folderPath: string, isRoot: boolean = false): boolean { + if (this.#closed || this.#failed) return false; + if (this.#watchers.has(folderPath)) return true; + let watcher: fs.FSWatcher; + try { + watcher = this.#watchDirectory(folderPath, (eventType, filename) => + this.#onDirectoryEvent(folderPath, eventType, filename) + ); + } catch (error) { + if (isRoot) throw toWatchError(error, folderPath); + if (TRANSIENT_ERROR_CODES.has(getErrorCode(error))) return false; + this.#fail(error, folderPath); + return false; + } + watcher.on('error', (error: Error) => this.#onDirectoryError(folderPath, error)); + if (this.#isUnref) watcher.unref(); + this.#watchers.set(folderPath, watcher); + return true; + } + + #onDirectoryEvent(folderPath: string, eventType: fs.WatchEventType, filename: string | null): void { + if (this.#closed) return; + if (!filename) { + // An unknown entry must stay unknown so the consumer performs a full invalidation. + this.#listener(eventType, null); + return; + } + const changedPath: string = path.join(folderPath, filename); + if (this.#isPruned(changedPath)) return; + this.#listener(eventType, path.relative(this.#root, changedPath)); + if (eventType === 'rename') { + void this.#reconcileEntryAsync(changedPath); + } + } + + #onDirectoryError(folderPath: string, error: Error): void { + if (TRANSIENT_ERROR_CODES.has(getErrorCode(error)) && folderPath !== this.#root) { + // The directory was removed; its parent's `rename` event already reported the change. + this.#removeDirectory(folderPath); + return; + } + this.#fail(error, folderPath); + } + + async #reconcileEntryAsync(changedPath: string): Promise { + // Exclusions may still be loading when the first events arrive; never register an excluded folder. + this.#resolvedExcludedFolderPaths = await this.#excludedFolderPaths; + if (this.#closed || this.#failed || this.#isPruned(changedPath)) return; + let isDirectory: boolean; + try { + isDirectory = (await fs.promises.lstat(changedPath)).isDirectory(); + } catch { + isDirectory = false; + } + // A `rename` for a directory that still exists may be a delete-and-recreate or an atomic replacement. + // inotify stays attached to the old inode, so always drop existing watches and register the current one. + this.#removeDirectory(changedPath); + if (isDirectory && this.#addDirectory(changedPath)) { + await this.#walkAsync(changedPath); + // Files written into the new directory before its watch existed were not reported. + this.#reportCoverage(changedPath); + } + } + + #reportCoverage(folderPath: string): void { + if (!this.#closed && !this.#failed) this.#listener('rename', path.relative(this.#root, folderPath)); + } + + #removeDirectory(folderPath: string): void { + const prefix: string = folderPath + path.sep; + for (const [key, watcher] of this.#watchers) { + if (key === folderPath || key.startsWith(prefix)) { + watcher.close(); + this.#watchers.delete(key); + } + } + } + + async #walkAsync(folderPath: string): Promise { + this.#resolvedExcludedFolderPaths = await this.#excludedFolderPaths; + let directory: fs.Dir; + try { + directory = await fs.promises.opendir(folderPath); + } catch (error) { + this.#onWalkError(folderPath, error); + return; + } + const children: string[] = []; + try { + for await (const entry of directory) { + if (this.#closed || this.#failed) break; + const childPath: string = path.join(folderPath, entry.name); + if (entry.isDirectory() && !this.#isPruned(childPath) && this.#addDirectory(childPath)) { + children.push(childPath); + } + } + } catch (error) { + this.#onWalkError(folderPath, error); + } + for (const childPath of children) { + if (this.#closed || this.#failed) return; + await this.#walkAsync(childPath); + } + } + + #onWalkError(folderPath: string, error: unknown): void { + if (folderPath !== this.#root && TRANSIENT_ERROR_CODES.has(getErrorCode(error))) { + // Removed or replaced while walking; the parent's `rename` event already reported the change. + this.#removeDirectory(folderPath); + return; + } + // Any other failure (for example EMFILE or EIO) leaves part of the tree unobserved. + this.#fail(error, folderPath); + } + + #fail(error: unknown, folderPath: string): void { + if (this.#failed || this.#closed) return; + this.#failed = true; + this.emit('error', toWatchError(error, folderPath)); + } +} + +/** Creates a {@link LinuxTreeWatcher} typed as an `fs.FSWatcher`. */ +export function createLinuxTreeWatcher( + root: string, + listener: fs.WatchListener, + options?: ILinuxTreeWatcherOptions +): fs.FSWatcher { + return new LinuxTreeWatcher(root, listener, options) as unknown as fs.FSWatcher; +} + +async function loadExcludedFolderPathsAsync( + getExcludedFolderPathsAsync: (() => Promise>) | undefined +): Promise> { + if (!getExcludedFolderPathsAsync) return EMPTY_SET; + try { + const folders: ReadonlySet = await getExcludedFolderPathsAsync(); + return new Set(Array.from(folders, (folder: string) => path.resolve(folder))); + } catch { + // Pruning is an optimization; observing a folder that could have been skipped is always safe. + return EMPTY_SET; + } +} + +function getErrorCode(error: unknown): string | undefined { + return (error as NodeJS.ErrnoException | undefined)?.code; +} + +/** Makes inotify exhaustion (`ENOSPC`) explicit instead of surfacing a bare libuv error. */ +export function toWatchError(error: unknown, folderPath: string): Error { + const cause: Error = error instanceof Error ? error : new Error(String(error)); + if (getErrorCode(error) !== 'ENOSPC') return cause; + const limitError: NodeJS.ErrnoException = new Error( + `The Linux inotify watch limit was reached while watching "${folderPath}" ` + + `(fs.inotify.max_user_watches). Rush daemon change detection is incomplete. ` + + `Increase the limit (for example "sudo sysctl fs.inotify.max_user_watches=524288") ` + + `or unset RUSH_DAEMON_WATCH.`, + { cause } + ); + limitError.code = 'ENOSPC'; + return limitError; +} diff --git a/libraries/rush-daemon/src/WorkspaceSessionFileWatcher.ts b/libraries/rush-daemon/src/WorkspaceSessionFileWatcher.ts index 05cb910190..095021b7d2 100644 --- a/libraries/rush-daemon/src/WorkspaceSessionFileWatcher.ts +++ b/libraries/rush-daemon/src/WorkspaceSessionFileWatcher.ts @@ -4,9 +4,15 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; -import type { RushConfiguration } from '@microsoft/rush-lib'; +import { + RushProjectConfiguration, + type RushConfiguration, + type RushConfigurationProject +} from '@microsoft/rush-lib'; +import { NoOpTerminalProvider, Terminal } from '@rushstack/terminal'; import type { IWorkspaceInvalidationWatcher } from './WorkspaceSession'; +import { createLinuxTreeWatcher, type LinuxTreeWatcher } from './LinuxTreeWatcher'; /** Options for the generation-owned workspace watcher. @beta */ export interface IWorkspaceSessionFileWatcherOptions { @@ -20,6 +26,7 @@ export interface IWorkspaceSessionFileWatcherOptions { interface IWatchPath { readonly folderPath: string; readonly recursive: boolean; + readonly project?: RushConfigurationProject; } /** Creates an individual filesystem watcher. @beta */ @@ -38,9 +45,10 @@ const PATH_SEGMENT_SEPARATOR_REGEXP: RegExp = /[\\/]/; */ export class WorkspaceSessionFileWatcher implements IWorkspaceInvalidationWatcher { readonly #onError: ((error: Error) => void) | undefined; - readonly #watchFactory: WorkspaceWatchFactory; + readonly #watchFactory: WorkspaceWatchFactory | undefined; readonly #permanentPaths: ReadonlyArray; readonly #projectFolders: ReadonlyMap; + readonly #projects: ReadonlyMap; readonly #initialProjectNames: ReadonlyArray; readonly #watchers: Map = new Map(); readonly #closing: Map> = new Map(); @@ -49,8 +57,11 @@ export class WorkspaceSessionFileWatcher implements IWorkspaceInvalidationWatche public constructor(options: IWorkspaceSessionFileWatcherOptions) { this.#onError = options.onError; - this.#watchFactory = options.watchFactory ?? fs.watch; + this.#watchFactory = options.watchFactory; this.#permanentPaths = getPermanentWatchPaths(options.rushConfiguration); + this.#projects = new Map( + options.rushConfiguration.projects.map((project) => [project.packageName, project]) + ); this.#projectFolders = new Map( options.rushConfiguration.projects.map((project) => [project.packageName, project.projectFolder]) ); @@ -77,6 +88,10 @@ export class WorkspaceSessionFileWatcher implements IWorkspaceInvalidationWatche for (const watchPath of this.#permanentPaths) { this.#watchers.set(watchPath.folderPath, this.#createWatcher(watchPath)); } + // Permanent config folders are small; finish registering them before initialization is acknowledged. + await Promise.all( + Array.from(this.#watchers.values(), (watcher) => (watcher as Partial).initialWalk) + ); this.watchProjects(this.#initialProjectNames); } @@ -94,7 +109,10 @@ export class WorkspaceSessionFileWatcher implements IWorkspaceInvalidationWatche throw new Error(`Project watcher is still closing: ${name}`); } if (!existing) { - this.#watchers.set(folderPath, this.#createWatcher({ folderPath, recursive: true })); + this.#watchers.set( + folderPath, + this.#createWatcher({ folderPath, recursive: true, project: this.#projects.get(name) }) + ); // Cover the observation gap without claiming an unknown graph/configuration mutation. this.#onInvalidation(folderPath); } @@ -167,19 +185,22 @@ export class WorkspaceSessionFileWatcher implements IWorkspaceInvalidationWatche } #createWatcher(watchPath: IWatchPath): fs.FSWatcher { - const watcher: fs.FSWatcher = this.#watchFactory( - watchPath.folderPath, - { encoding: 'utf8', recursive: watchPath.recursive }, - (eventType: string, filename: string | null) => { - void eventType; - const changedFilename: string | undefined = filename ?? undefined; - if (!isIgnoredPath(changedFilename)) { - this.#onInvalidation?.( - changedFilename === undefined ? undefined : path.resolve(watchPath.folderPath, changedFilename) - ); - } + const listener: fs.WatchListener = (eventType: string, filename: string | null) => { + void eventType; + const changedFilename: string | undefined = filename ?? undefined; + if (!isIgnoredPath(changedFilename)) { + this.#onInvalidation?.( + changedFilename === undefined ? undefined : path.resolve(watchPath.folderPath, changedFilename) + ); } - ); + }; + const watchOptions: { encoding: 'utf8'; recursive: boolean } = { + encoding: 'utf8', + recursive: watchPath.recursive + }; + const watcher: fs.FSWatcher = this.#watchFactory + ? this.#watchFactory(watchPath.folderPath, watchOptions, listener) + : createDefaultWatcher(watchPath, watchOptions, listener); watcher.on('error', (error: Error) => { this.#onInvalidation?.(); if (this.#onError) this.#onError(error); @@ -195,6 +216,49 @@ export class WorkspaceSessionFileWatcher implements IWorkspaceInvalidationWatche } } +/** + * Node's recursive `fs.watch` on Linux walks the tree synchronously and adds one inotify watch per file, + * including build outputs. On Linux, recursive observation uses per-directory watches from an async walk instead. + */ +function createDefaultWatcher( + watchPath: IWatchPath, + watchOptions: { encoding: 'utf8'; recursive: boolean }, + listener: fs.WatchListener +): fs.FSWatcher { + if (process.platform !== 'linux' || !watchPath.recursive) { + return fs.watch(watchPath.folderPath, watchOptions, listener); + } + const project: RushConfigurationProject | undefined = watchPath.project; + return createLinuxTreeWatcher(watchPath.folderPath, listener, { + getExcludedFolderPathsAsync: project ? () => getProjectExcludedFolderPathsAsync(project) : undefined, + reportInitialWalkCompletion: project !== undefined + }); +} + +/** The project's `.rush/temp` folder plus every declared operation output folder. @internal */ +export async function getProjectExcludedFolderPathsAsync( + project: RushConfigurationProject +): Promise> { + const excluded: Set = new Set([path.resolve(project.projectRushTempFolder)]); + const terminal: Terminal = new Terminal(new NoOpTerminalProvider()); + let configuration: RushProjectConfiguration | undefined; + try { + configuration = await RushProjectConfiguration.tryLoadForProjectAsync(project, terminal); + } catch { + // Output-folder pruning is an optimization; an unreadable configuration only loses that part. + return excluded; + } + const projectFolder: string = path.resolve(project.projectFolder); + for (const settings of configuration?.operationSettingsByOperationName.values() ?? []) { + for (const outputFolderName of settings.outputFolderNames ?? []) { + const outputFolder: string = path.resolve(projectFolder, outputFolderName); + // Never prune the project folder itself or anything outside it. + if (outputFolder.startsWith(projectFolder + path.sep)) excluded.add(outputFolder); + } + } + return excluded; +} + function getPermanentWatchPaths(rushConfiguration: RushConfiguration): ReadonlyArray { const recursiveFolders: Set = new Set([rushConfiguration.commonRushConfigFolder]); for (const subspace of rushConfiguration.subspaces) { diff --git a/libraries/rush-daemon/src/test/LinuxTreeWatcher.test.ts b/libraries/rush-daemon/src/test/LinuxTreeWatcher.test.ts new file mode 100644 index 0000000000..3d4c18874a --- /dev/null +++ b/libraries/rush-daemon/src/test/LinuxTreeWatcher.test.ts @@ -0,0 +1,295 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { EventEmitter } from 'node:events'; + +import { + LinuxTreeWatcher, + toWatchError, + type DirectoryWatchFunction, + type ILinuxTreeWatcherOptions +} from '../LinuxTreeWatcher'; + +class FakeDirectoryWatcher extends EventEmitter { + public closed: boolean = false; + public readonly listener: fs.WatchListener; + public constructor(listener: fs.WatchListener) { + super(); + this.listener = listener; + } + public close(): void { + this.closed = true; + } + public ref(): this { + return this; + } + public unref(): this { + return this; + } +} + +interface IHarness { + readonly root: string; + readonly events: string[]; + readonly errors: Error[]; + readonly fakes: Map; + readonly watcher: LinuxTreeWatcher; + fire(folder: string, eventType: fs.WatchEventType, filename: string): void; +} + +const roots: string[] = []; + +function makeTree(files: string[]): string { + const root: string = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-tree-'))); + roots.push(root); + for (const file of files) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), ''); + } + return root; +} + +function createHarness( + root: string, + options: Omit = {}, + watchOverride?: (folder: string) => void +): IHarness { + const events: string[] = []; + const errors: Error[] = []; + const fakes: Map = new Map(); + const watchDirectory: DirectoryWatchFunction = (folder, listener) => { + watchOverride?.(folder); + const fake: FakeDirectoryWatcher = new FakeDirectoryWatcher(listener); + fakes.set(folder, fake); + return fake as unknown as fs.FSWatcher; + }; + const watcher: LinuxTreeWatcher = new LinuxTreeWatcher( + root, + (eventType, filename) => events.push(`${eventType}:${filename}`), + { ...options, watchDirectory } + ); + watcher.on('error', (error: Error) => errors.push(error)); + return { + root, + events, + errors, + fakes, + watcher, + fire: (folder, eventType, filename) => fakes.get(path.join(root, folder))!.listener(eventType, filename) + }; +} + +function relativeWatched(harness: IHarness): string[] { + return [...harness.watcher.watchedFolderPaths].map((folder) => path.relative(harness.root, folder)).sort(); +} + +async function waitForAsync(condition: () => boolean): Promise { + for (let attempt: number = 0; attempt < 200 && !condition(); attempt++) { + await new Promise((resolve) => setTimeout(resolve, 5)); + } + expect(condition()).toBe(true); +} + +afterAll(() => { + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe(LinuxTreeWatcher.name, () => { + it('watches directories only and prunes node_modules, .git, .rush/temp and output folders', async () => { + const root: string = makeTree([ + 'package.json', + 'src/a.ts', + 'src/nested/b.ts', + 'lib/a.js', + 'lib/nested/b.js', + 'node_modules/x/index.js', + '.git/HEAD', + '.rush/temp/shrinkwrap.yaml', + '.rush/other.json', + 'config/rush-project.json' + ]); + const harness: IHarness = createHarness(root, { + getExcludedFolderPathsAsync: async () => + new Set([path.join(root, 'lib'), path.join(root, '.rush', 'temp')]) + }); + await harness.watcher.initialWalk; + expect(relativeWatched(harness)).toEqual(['', '.rush', 'config', 'src', path.join('src', 'nested')]); + expect(harness.errors).toEqual([]); + harness.watcher.close(); + }); + + it('suppresses events for pruned paths and reports others relative to the root', async () => { + const root: string = makeTree(['src/a.ts']); + const harness: IHarness = createHarness(root, { + getExcludedFolderPathsAsync: async () => new Set([path.join(root, 'lib')]) + }); + await harness.watcher.initialWalk; + harness.fire('', 'rename', 'lib'); + harness.fire('', 'rename', 'node_modules'); + harness.fire('src', 'change', 'a.ts'); + expect(harness.events).toEqual([`change:${path.join('src', 'a.ts')}`]); + harness.watcher.close(); + }); + + it('follows directories created and removed after the initial walk', async () => { + const root: string = makeTree(['src/a.ts']); + const harness: IHarness = createHarness(root); + await harness.watcher.initialWalk; + fs.mkdirSync(path.join(root, 'src', 'added', 'deep'), { recursive: true }); + harness.fire('src', 'rename', 'added'); + const added: string = path.join('src', 'added'); + await waitForAsync(() => harness.events.includes(`rename:${added}`) && harness.events.length === 2); + expect(relativeWatched(harness)).toEqual(['', 'src', added, path.join(added, 'deep')]); + + const deepWatcher: FakeDirectoryWatcher = harness.fakes.get(path.join(root, added, 'deep'))!; + fs.rmSync(path.join(root, added), { recursive: true }); + harness.fire('src', 'rename', 'added'); + await waitForAsync(() => relativeWatched(harness).length === 2); + expect(deepWatcher.closed).toBe(true); + expect(harness.errors).toEqual([]); + harness.watcher.close(); + }); + + it('tolerates directories that disappear during registration without reporting an error', async () => { + const root: string = makeTree(['a/x.ts', 'b/y.ts']); + const harness: IHarness = createHarness(root, {}, (folder) => { + if (path.basename(folder) === 'a') { + throw Object.assign(new Error('ENOENT: no such file or directory'), { code: 'ENOENT' }); + } + }); + await harness.watcher.initialWalk; + expect(relativeWatched(harness)).toEqual(['', 'b']); + harness.fakes.get(path.join(root, 'b'))!.emit('error', Object.assign(new Error('gone'), { code: 'ENOENT' })); + expect(relativeWatched(harness)).toEqual(['']); + expect(harness.errors).toEqual([]); + harness.watcher.close(); + }); + + it('reports ENOSPC once with an explicit inotify limit error and stops registering', async () => { + const root: string = makeTree(['a/x.ts', 'b/y.ts', 'c/z.ts']); + const harness: IHarness = createHarness(root, {}, (folder) => { + if (folder !== root) { + throw Object.assign(new Error('ENOSPC: System limit for number of file watchers reached'), { + code: 'ENOSPC' + }); + } + }); + await harness.watcher.initialWalk; + expect(harness.errors).toHaveLength(1); + expect((harness.errors[0] as NodeJS.ErrnoException).code).toBe('ENOSPC'); + expect(harness.errors[0].message).toContain('fs.inotify.max_user_watches'); + harness.watcher.close(); + }); + + it('throws synchronously when the root cannot be watched, like fs.watch', () => { + const missing: string = path.join(makeTree([]), 'missing'); + expect( + () => + new LinuxTreeWatcher(missing, () => {}, { + watchDirectory: (folder) => fs.watch(folder) + }) + ).toThrow(/ENOENT/); + }); + + it('reports the root after the initial walk when requested and emits close asynchronously', async () => { + const root: string = makeTree(['src/a.ts']); + const harness: IHarness = createHarness(root, { reportInitialWalkCompletion: true }); + await harness.watcher.initialWalk; + expect(harness.events).toEqual(['rename:']); + const closed: Promise = new Promise((resolve) => harness.watcher.once('close', resolve)); + harness.watcher.close(); + await closed; + expect([...harness.fakes.values()].every((fake) => fake.closed)).toBe(true); + expect(harness.watcher.watchedFolderPaths.size).toBe(0); + }); + + it('forwards an unknown (null) filename unchanged so the consumer invalidates fully', async () => { + const root: string = makeTree(['src/a.ts']); + const harness: IHarness = createHarness(root); + await harness.watcher.initialWalk; + harness.fakes.get(path.join(root, 'src'))!.listener('change', null); + expect(harness.events).toEqual(['change:null']); + harness.watcher.close(); + }); + + it('does not register a folder that becomes excluded after exclusions finish loading', async () => { + const root: string = makeTree(['src/a.ts', 'lib/a.js']); + let resolveExclusions!: (folders: ReadonlySet) => void; + const exclusions: Promise> = new Promise((resolve) => (resolveExclusions = resolve)); + const harness: IHarness = createHarness(root, { getExcludedFolderPathsAsync: () => exclusions }); + harness.fire('', 'rename', 'lib'); + await new Promise((resolve) => setTimeout(resolve, 10)); + resolveExclusions(new Set([path.join(root, 'lib')])); + await harness.watcher.initialWalk; + await new Promise((resolve) => setTimeout(resolve, 10)); + expect(relativeWatched(harness)).toEqual(['', 'src']); + harness.watcher.close(); + }); + + it('re-registers a directory that was replaced in place', async () => { + const root: string = makeTree(['src/nested/a.ts']); + const harness: IHarness = createHarness(root); + await harness.watcher.initialWalk; + const oldSrc: FakeDirectoryWatcher = harness.fakes.get(path.join(root, 'src'))!; + const oldNested: FakeDirectoryWatcher = harness.fakes.get(path.join(root, 'src', 'nested'))!; + fs.rmSync(path.join(root, 'src'), { recursive: true }); + fs.mkdirSync(path.join(root, 'src', 'nested'), { recursive: true }); + harness.fire('', 'rename', 'src'); + await waitForAsync(() => harness.fakes.get(path.join(root, 'src', 'nested')) !== oldNested); + expect(oldSrc.closed).toBe(true); + expect(oldNested.closed).toBe(true); + expect(harness.fakes.get(path.join(root, 'src'))).not.toBe(oldSrc); + expect(relativeWatched(harness)).toEqual(['', 'src', path.join('src', 'nested')]); + expect(harness.errors).toEqual([]); + harness.watcher.close(); + }); + + it('reports non-transient walk failures instead of treating them as a vanished directory', async () => { + const root: string = makeTree(['a/x.ts']); + const opendir: typeof fs.promises.opendir = fs.promises.opendir; + const spy: jest.SpyInstance = jest + .spyOn(fs.promises, 'opendir') + .mockImplementation(async (folder: fs.PathLike, options?: fs.OpenDirOptions) => { + if (path.basename(String(folder)) === 'a') { + throw Object.assign(new Error('EMFILE: too many open files'), { code: 'EMFILE' }); + } + return await opendir(folder, options); + }); + try { + const harness: IHarness = createHarness(root); + await harness.watcher.initialWalk; + expect(harness.errors.map((error) => (error as NodeJS.ErrnoException).code)).toEqual(['EMFILE']); + harness.watcher.close(); + } finally { + spy.mockRestore(); + } + }); + + it('leaves non-ENOSPC errors unchanged', () => { + const error: Error = Object.assign(new Error('EMFILE'), { code: 'EMFILE' }); + expect(toWatchError(error, '/x')).toBe(error); + }); + + (process.platform === 'linux' ? it : it.skip)('observes real inotify events in new directories', async () => { + const root: string = makeTree(['src/a.ts', 'lib/a.js']); + const events: string[] = []; + const watcher: LinuxTreeWatcher = new LinuxTreeWatcher(root, (...args) => events.push(args[1]!), { + getExcludedFolderPathsAsync: async () => new Set([path.join(root, 'lib')]) + }); + try { + await watcher.initialWalk; + expect(watcher.watchedFolderPaths.size).toBe(2); + fs.mkdirSync(path.join(root, 'src', 'new')); + await waitForAsync(() => watcher.watchedFolderPaths.size === 3); + fs.writeFileSync(path.join(root, 'src', 'new', 'b.ts'), 'x'); + fs.writeFileSync(path.join(root, 'lib', 'b.js'), 'x'); + await waitForAsync(() => events.includes(path.join('src', 'new', 'b.ts'))); + expect(events.some((event) => event.startsWith('lib'))).toBe(false); + } finally { + watcher.close(); + } + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceSessionFileWatcher.test.ts b/libraries/rush-daemon/src/test/WorkspaceSessionFileWatcher.test.ts index 7596e49d0d..59cf271ed5 100644 --- a/libraries/rush-daemon/src/test/WorkspaceSessionFileWatcher.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceSessionFileWatcher.test.ts @@ -2,9 +2,12 @@ // See LICENSE in the project root for license information. import type * as fs from 'node:fs'; +import * as path from 'node:path'; import { EventEmitter } from 'node:events'; -import { WorkspaceSessionFileWatcher } from '../WorkspaceSessionFileWatcher'; +import { RushProjectConfiguration } from '@microsoft/rush-lib'; + +import { getProjectExcludedFolderPathsAsync, WorkspaceSessionFileWatcher } from '../WorkspaceSessionFileWatcher'; import { TEST_RUSH_CONFIGURATION } from './TestWorkspaceSession'; class TestFsWatcher extends EventEmitter { @@ -22,6 +25,26 @@ class TestFsWatcher extends EventEmitter { } describe(WorkspaceSessionFileWatcher.name, () => { + it('excludes the project .rush/temp folder from Linux tree observation', async () => { + const project = TEST_RUSH_CONFIGURATION.projects[0]; + const excluded: ReadonlySet = await getProjectExcludedFolderPathsAsync(project); + expect(excluded.has(path.resolve(project.projectRushTempFolder))).toBe(true); + expect(excluded.has(path.resolve(project.projectFolder))).toBe(false); + }); + + it('keeps the .rush/temp exclusion when rush-project.json cannot be loaded', async () => { + const project = TEST_RUSH_CONFIGURATION.projects[0]; + const spy: jest.SpyInstance = jest + .spyOn(RushProjectConfiguration, 'tryLoadForProjectAsync') + .mockRejectedValue(new Error('invalid rush-project.json')); + try { + const excluded: ReadonlySet = await getProjectExcludedFolderPathsAsync(project); + expect([...excluded]).toEqual([path.resolve(project.projectRushTempFolder)]); + } finally { + spy.mockRestore(); + } + }); + it('watches every configured subspace config folder', async () => { const watchedPaths: string[] = []; const watcher: WorkspaceSessionFileWatcher = new WorkspaceSessionFileWatcher({ From 18fc2b67edf69f894a39a5db577ac61231b38beb Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:16:27 -0700 Subject: [PATCH 004/265] [rush-daemon-transport] Reap a dead daemon's orphaned operation processes on reclaim (#6088) * [rush-daemon-transport] Reap a dead daemon's orphaned operation processes on reclaim Fixes #6055 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon-transport] Address review: fail closed on probe errors, unknown own group, and SIGKILL survivors Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../rushd-reap-orphans_2026-09-24-03-00.json | 11 ++ .../src/DaemonOrphanReaper.ts | 100 ++++++++++++++++++ .../src/DaemonProcessGroup.ts | 78 ++++++++++++++ .../src/DaemonReclaim.ts | 9 +- .../src/test/DaemonOrphanReaper.test.ts | 64 +++++++++++ .../src/test/DaemonProcessGroup.test.ts | 32 ++++++ .../src/test/OrphanReapOnReclaim.test.ts | 70 ++++++++++++ .../src/test/OrphanReaperFixture.ts | 50 +++++++++ 8 files changed, 412 insertions(+), 2 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon-transport/rushd-reap-orphans_2026-09-24-03-00.json create mode 100644 libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonProcessGroup.ts create mode 100644 libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/DaemonProcessGroup.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/OrphanReapOnReclaim.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts diff --git a/common/changes/@rushstack/rush-daemon-transport/rushd-reap-orphans_2026-09-24-03-00.json b/common/changes/@rushstack/rush-daemon-transport/rushd-reap-orphans_2026-09-24-03-00.json new file mode 100644 index 0000000000..89a8bf208b --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-transport/rushd-reap-orphans_2026-09-24-03-00.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-transport", + "comment": "When reclaiming the socket of a daemon that died uncleanly (SIGKILL/OOM), terminate the operation processes still running in its process group (SIGTERM, then SIGKILL after a grace period) so a successor daemon does not re-run them concurrently.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-transport", + "email": "TheLarkInn@users.noreply.github.com" +} diff --git a/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts b/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts new file mode 100644 index 0000000000..bd4c61534b --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts @@ -0,0 +1,100 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonLockfile } from './DaemonLockfile'; +import type { IDaemonProcessGroupOps } from './DaemonProcessGroup'; +import { POSIX_PROCESS_GROUP_OPS } from './DaemonProcessGroup'; + +const WINDOWS_PLATFORM: NodeJS.Platform = 'win32'; +// 0 and 1 are never daemons, and kill(-0)/kill(-1) would signal our own group or every process. +const FIRST_USER_PID: number = 2; +const POLL_INTERVAL_MS: number = 20; +const DEFAULT_GRACE_MS: number = 2000; + +/** Outcome of reaping a dead daemon's process group. */ +export type DaemonOrphanReapOutcome = 'none' | 'terminated' | 'killed'; + +/** Options for {@link reapDeadDaemonProcessGroupAsync}; every field defaults to the real process. */ +export interface IDaemonOrphanReaperOptions { + readonly ops?: IDaemonProcessGroupOps; + readonly platform?: NodeJS.Platform; + readonly selfPid?: number; + /** How long SIGTERM'd (and then SIGKILL'd) processes get to exit. */ + readonly graceMs?: number; +} + +interface IReapContext { + readonly ops: IDaemonProcessGroupOps; + readonly deadPid: number; + readonly graceMs: number; +} +type OrphanCheck = (pid: number) => boolean; + +function isNeitherSelfNorOwnGroup(pid: number, selfPid: number, ops: IDaemonProcessGroupOps): boolean { + // Fail closed: an unknown own group might be `pid` (e.g. rush-client run by an operation). + const ownGroupId: number | undefined = ops.ownGroupId(); + return pid !== selfPid && ownGroupId !== undefined && pid !== ownGroupId; +} + +function orphanChecks(options: IDaemonOrphanReaperOptions, ops: IDaemonProcessGroupOps): OrphanCheck[] { + return [ + () => (options.platform ?? process.platform) !== WINDOWS_PLATFORM, + (pid: number) => Number.isSafeInteger(pid) && pid >= FIRST_USER_PID, + (pid: number) => isNeitherSelfNorOwnGroup(pid, options.selfPid ?? process.pid, ops), + (pid: number) => !ops.isProcessAlive(pid), + (pid: number) => ops.groupExists(pid) + ]; +} + +async function waitForGroupExitAsync(context: IReapContext): Promise { + const deadline: number = context.ops.now() + context.graceMs; + while (context.ops.now() < deadline) { + if (!context.ops.groupExists(context.deadPid)) return true; + await context.ops.delayAsync(POLL_INTERVAL_MS); + } + return !context.ops.groupExists(context.deadPid); +} + +async function terminateGroupAsync(context: IReapContext): Promise { + context.ops.signalGroup(context.deadPid, 'SIGTERM'); + if (await waitForGroupExitAsync(context)) return 'terminated'; + context.ops.signalGroup(context.deadPid, 'SIGKILL'); + if (await waitForGroupExitAsync(context)) return 'killed'; + throw new Error(`Processes of dead daemon ${context.deadPid} survived SIGKILL; not reclaiming its socket.`); +} + +/** + * Terminates operation processes left behind by a daemon that died without joining them (SIGKILL, OOM). + * + * @remarks + * The daemon is spawned detached, so its pid is its process group id, and phased operation children inherit + * that group (children spawned with their own detached group are out of scope). Sends SIGTERM, then SIGKILL + * after `graceMs`, and throws if the group still has not exited after a further `graceMs`. + * PID-reuse guard: only group `deadPid` is signaled, and only once `deadPid` is proven dead while the group + * still exists; POSIX never reuses a pid still in use as a process group id, so every remaining member + * belongs to the dead daemon. Never signals the caller's pid or group, and does nothing when the caller's + * group is unknown (no `/proc`). Call only under the reclaim mutex. + */ +export async function reapDeadDaemonProcessGroupAsync( + deadPid: number, + options: IDaemonOrphanReaperOptions = {} +): Promise { + const context: IReapContext = createReapContext(deadPid, options); + if (!orphanChecks(options, context.ops).every((check: OrphanCheck) => check(deadPid))) return 'none'; + const outcome: DaemonOrphanReapOutcome = await terminateGroupAsync(context); + context.ops.log(`Reclaimed dead daemon ${deadPid}: its orphaned operation process group was ${outcome}.`); + return outcome; +} + +function createReapContext(deadPid: number, options: IDaemonOrphanReaperOptions): IReapContext { + return { + ops: options.ops ?? POSIX_PROCESS_GROUP_OPS, + deadPid, + graceMs: options.graceMs ?? DEFAULT_GRACE_MS + }; +} + +/** Reaps the orphaned operations of a reclaimed daemon's recorded owner, if there is one. */ +export async function reapOrphansOfDeadOwnerAsync(owner: IDaemonLockfile | undefined): Promise { + if (owner) await reapDeadDaemonProcessGroupAsync(owner.pid); +} diff --git a/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts b/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts new file mode 100644 index 0000000000..a8eb871383 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts @@ -0,0 +1,78 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { isDaemonProcessAlive } from './DaemonLockfile'; + +const NO_SIGNAL: number = 0; +const NO_SUCH_PROCESS: string = 'ESRCH'; +const PROC_SELF_STAT: string = '/proc/self/stat'; +const UTF8: BufferEncoding = 'utf8'; +const COMM_END: string = ')'; +const FIELD_SEPARATOR: string = ' '; +// After the ")" that ends the command name come: " ...". +const PGRP_FIELD_INDEX: number = 3; + +/** Process probing/signaling used to reap a dead daemon's process group; injectable for tests. */ +export interface IDaemonProcessGroupOps { + readonly isProcessAlive: (pid: number) => boolean; + readonly groupExists: (groupId: number) => boolean; + readonly signalGroup: (groupId: number, signal: NodeJS.Signals) => void; + /** The caller's own process group id, or `undefined` when the platform cannot report it. */ + readonly ownGroupId: () => number | undefined; + readonly delayAsync: (ms: number) => Promise; + readonly now: () => number; + readonly log: (message: string) => void; +} + +// Only ESRCH proves the group is gone; EPERM and other failures propagate so reclaim fails closed. +function rethrowUnlessNoSuchProcess(error: unknown): void { + if ((error as NodeJS.ErrnoException | undefined)?.code !== NO_SUCH_PROCESS) throw error; +} + +function groupExists(groupId: number): boolean { + try { + process.kill(-groupId, NO_SIGNAL); + return true; + } catch (error) { + rethrowUnlessNoSuchProcess(error); + return false; + } +} + +function signalGroup(groupId: number, signal: NodeJS.Signals): void { + try { + process.kill(-groupId, signal); + } catch (error) { + rethrowUnlessNoSuchProcess(error); + } +} + +function ownGroupId(): number | undefined { + try { + const stat: string = fs.readFileSync(PROC_SELF_STAT, UTF8); + const fields: string[] = stat.slice(stat.lastIndexOf(COMM_END)).split(FIELD_SEPARATOR); + return Number(fields[PGRP_FIELD_INDEX]); + } catch { + return undefined; + } +} + +function log(message: string): void { + process.emitWarning(message, { code: 'RUSH_DAEMON_ORPHANS_REAPED' }); +} + +/** The real POSIX implementation of {@link IDaemonProcessGroupOps}. */ +export const POSIX_PROCESS_GROUP_OPS: IDaemonProcessGroupOps = { + isProcessAlive: isDaemonProcessAlive, + groupExists, + signalGroup, + ownGroupId, + delayAsync: async (ms: number) => { + await delayAsync(ms); + }, + now: Date.now, + log +}; diff --git a/libraries/rush-daemon-transport/src/DaemonReclaim.ts b/libraries/rush-daemon-transport/src/DaemonReclaim.ts index e70d12f0f2..92f0ed9323 100644 --- a/libraries/rush-daemon-transport/src/DaemonReclaim.ts +++ b/libraries/rush-daemon-transport/src/DaemonReclaim.ts @@ -10,6 +10,7 @@ import { readDaemonLockfile, removeDaemonArtifacts } from './DaemonLockfile'; +import { reapOrphansOfDeadOwnerAsync } from './DaemonOrphanReaper'; import type { IDaemonPaths } from './DaemonPaths'; import { tryAcquireReclaimLock } from './DaemonReclaimLock'; import type { DaemonReclaimLockOutcome } from './DaemonReclaimLock'; @@ -24,7 +25,8 @@ import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTranspor * never reclaimed underneath itself. Reclaims are serialized through the * lockfile mutex ({@link tryAcquireReclaimLock}): only the mutex holder may * unlink the socket path, so a concurrent starter cannot delete a socket that - * another process just bound. + * another process just bound. Operation processes still running in the dead + * daemon's process group are terminated first (see `DaemonOrphanReaper`). * * @throws {@link DaemonTransportError} with code `daemonAlreadyRunning` when a * live (or plausibly live) daemon owns the path, or when another starter holds @@ -57,13 +59,16 @@ function reclaimLockPath(paths: IDaemonPaths): string { } async function reclaimUnderLockAsync(paths: IDaemonPaths): Promise { - if (isLockfilePidAlive(readDaemonLockfile(paths.lockfilePath))) { + const owner: ReturnType = readDaemonLockfile(paths.lockfilePath); + if (isLockfilePidAlive(owner)) { throwAlreadyRunning(paths, 'its lockfile PID is alive'); } const probeFailed: boolean = await probeConnectionFailsAsync(paths.socketPath); if (!probeFailed) { throwAlreadyRunning(paths, 'it answers a connect probe'); } + // A daemon that died uncleanly leaves its operations running; stop them before a successor re-runs them. + await reapOrphansOfDeadOwnerAsync(owner); removeDaemonArtifacts(paths.lockfilePath, paths.socketPath); } diff --git a/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts b/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts new file mode 100644 index 0000000000..a775495413 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts @@ -0,0 +1,64 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { reapDeadDaemonProcessGroupAsync } from '../DaemonOrphanReaper'; +import type { IDaemonOrphanReaperOptions } from '../DaemonOrphanReaper'; + +import { DEAD_PID, SELF_PID, createFakeGroup } from './OrphanReaperFixture'; +import type { IFakeGroup } from './OrphanReaperFixture'; + +const INIT_PID: number = 1; + +it('stops at SIGTERM when the orphaned group exits within the grace period', async () => { + const fake: IFakeGroup = createFakeGroup({ exitsOn: 'SIGTERM' }); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, fake.options)).resolves.toBe('terminated'); + expect(fake.signals).toEqual(['SIGTERM']); + expect(fake.logs).toEqual([expect.stringContaining(`dead daemon ${DEAD_PID}`)]); +}); + +it('escalates to SIGKILL when the group outlives the grace period', async () => { + const fake: IFakeGroup = createFakeGroup({ exitsOn: 'SIGKILL' }); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, fake.options)).resolves.toBe('killed'); + expect(fake.signals).toEqual(['SIGTERM', 'SIGKILL']); + expect(fake.logs).toEqual([expect.stringContaining('killed')]); +}); + +it('fails the reclaim when the group is still present after SIGKILL', async () => { + const fake: IFakeGroup = createFakeGroup({}); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, fake.options)).rejects.toThrow(/survived SIGKILL/); + expect(fake.signals).toEqual(['SIGTERM', 'SIGKILL']); +}); + +it('never signals a live daemon (its pid is not proven dead)', async () => { + const fake: IFakeGroup = createFakeGroup({ daemonAlive: true }); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, fake.options)).resolves.toBe('none'); + expect(fake.signals).toEqual([]); +}); + +it('never signals when no group with the dead pid remains', async () => { + const fake: IFakeGroup = createFakeGroup({}); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID + INIT_PID, fake.options)).resolves.toBe('none'); + expect(fake.signals).toEqual([]); + expect(fake.logs).toEqual([]); +}); + +it.each([INIT_PID, SELF_PID, Number.NaN])('never signals the unsafe group id %p', async (pid: number) => { + const fake: IFakeGroup = createFakeGroup({ anyGroupExists: true, ownGroupId: DEAD_PID + INIT_PID }); + await expect(reapDeadDaemonProcessGroupAsync(pid, fake.options)).resolves.toBe('none'); + expect(fake.signals).toEqual([]); +}); + +it('never signals the caller own process group, or when that group is unknown', async () => { + const own: IFakeGroup = createFakeGroup({ ownGroupId: DEAD_PID }); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, own.options)).resolves.toBe('none'); + const unknown: IFakeGroup = createFakeGroup({ unknownOwnGroup: true }); + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, unknown.options)).resolves.toBe('none'); + expect([...own.signals, ...unknown.signals]).toEqual([]); +}); + +it('never signals anything on Windows', async () => { + const win: IFakeGroup = createFakeGroup({}); + const winOptions: IDaemonOrphanReaperOptions = { ...win.options, platform: 'win32' }; + await expect(reapDeadDaemonProcessGroupAsync(DEAD_PID, winOptions)).resolves.toBe('none'); + expect(win.signals).toEqual([]); +}); diff --git a/libraries/rush-daemon-transport/src/test/DaemonProcessGroup.test.ts b/libraries/rush-daemon-transport/src/test/DaemonProcessGroup.test.ts new file mode 100644 index 0000000000..8ab89515a3 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/DaemonProcessGroup.test.ts @@ -0,0 +1,32 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { POSIX_PROCESS_GROUP_OPS } from '../DaemonProcessGroup'; + +const GROUP_ID: number = 4242; + +function throwErrno(code: string): never { + throw Object.assign(new Error(code), { code }); +} + +afterEach(() => { + jest.restoreAllMocks(); +}); + +it('treats only ESRCH as proof that a process group is gone', () => { + jest.spyOn(process, 'kill').mockImplementation(() => throwErrno('ESRCH')); + expect(POSIX_PROCESS_GROUP_OPS.groupExists(GROUP_ID)).toBe(false); + expect(() => POSIX_PROCESS_GROUP_OPS.signalGroup(GROUP_ID, 'SIGTERM')).not.toThrow(); +}); + +it('propagates EPERM instead of assuming the group is gone', () => { + jest.spyOn(process, 'kill').mockImplementation(() => throwErrno('EPERM')); + expect(() => POSIX_PROCESS_GROUP_OPS.groupExists(GROUP_ID)).toThrow('EPERM'); + expect(() => POSIX_PROCESS_GROUP_OPS.signalGroup(GROUP_ID, 'SIGKILL')).toThrow('EPERM'); +}); + +it('signals the negated group id', () => { + const kill: jest.SpyInstance = jest.spyOn(process, 'kill').mockImplementation(() => true); + POSIX_PROCESS_GROUP_OPS.signalGroup(GROUP_ID, 'SIGTERM'); + expect(kill).toHaveBeenCalledWith(-GROUP_ID, 'SIGTERM'); +}); diff --git a/libraries/rush-daemon-transport/src/test/OrphanReapOnReclaim.test.ts b/libraries/rush-daemon-transport/src/test/OrphanReapOnReclaim.test.ts new file mode 100644 index 0000000000..2e4a96a4cf --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/OrphanReapOnReclaim.test.ts @@ -0,0 +1,70 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn } from 'node:child_process'; +import type { ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import type { Readable } from 'node:stream'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import { isDaemonProcessAlive, writeDaemonLockfile } from '../DaemonLockfile'; +import type { IDaemonPaths } from '../DaemonPaths'; +import { reclaimStaleDaemonAsync } from '../DaemonReclaim'; + +import { createTestDaemonPaths } from './TestDaemonFixture'; + +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +const FIRST_ATTEMPT: number = 0; +const POLL_ATTEMPTS: number = 100; +const POLL_INTERVAL_MS: number = 20; +// A stand-in daemon: spawns one operation child and prints the child pid. Like a phased operation (spawned +// without `detached`), the child inherits the daemon's process group. +const FAKE_DAEMON_SCRIPT: string = + "const c=require('node:child_process').spawn(process.execPath,['-e','setInterval(()=>{},1000)'],{stdio:'ignore'});" + + "process.stdout.write(String(c.pid)+'\\n');setInterval(()=>{},1000);"; + +interface IOrphanedGroup { + readonly daemonPid: number; + readonly orphanPid: number; +} + +/** Spawns a detached fake daemon with one child in its group, then SIGKILLs only the daemon. */ +async function createOrphanedGroupAsync(): Promise { + const daemon: ChildProcess = spawn(process.execPath, ['-e', FAKE_DAEMON_SCRIPT], { + detached: true, + stdio: ['ignore', 'pipe', 'ignore'] + }); + const stdout: Readable = daemon.stdout as Readable; + const [chunk] = (await once(stdout, 'data')) as [Buffer]; + daemon.kill('SIGKILL'); + await once(daemon, 'exit'); + stdout.destroy(); + return { daemonPid: Number(daemon.pid), orphanPid: Number(chunk.toString().trim()) }; +} + +async function waitUntilDeadAsync(pid: number): Promise { + for (let attempt: number = FIRST_ATTEMPT; attempt < POLL_ATTEMPTS; attempt++) { + if (!isDaemonProcessAlive(pid)) return true; + await delayAsync(POLL_INTERVAL_MS); + } + return false; +} + +posixIt('reaps operation processes orphaned by a SIGKILLed daemon before reclaiming', async () => { + const warning: jest.SpyInstance = jest.spyOn(process, 'emitWarning').mockImplementation(() => undefined); + const { daemonPid, orphanPid } = await createOrphanedGroupAsync(); + expect(isDaemonProcessAlive(orphanPid)).toBe(true); + const paths: IDaemonPaths = createTestDaemonPaths(); + writeDaemonLockfile(paths.lockfilePath, { + pid: daemonPid, + protocolVersion: DAEMON_PROTOCOL_VERSION, + startedAt: new Date().toISOString(), + socketPath: paths.socketPath + }); + await reclaimStaleDaemonAsync(paths); + expect(await waitUntilDeadAsync(orphanPid)).toBe(true); + expect(warning).toHaveBeenCalledWith(expect.stringContaining(`dead daemon ${daemonPid}`), expect.anything()); + warning.mockRestore(); +}); diff --git a/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts b/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts new file mode 100644 index 0000000000..c0e1f8bede --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts @@ -0,0 +1,50 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonOrphanReaperOptions } from '../DaemonOrphanReaper'; +import type { IDaemonProcessGroupOps } from '../DaemonProcessGroup'; + +/** The pid of the fake dead daemon, which is also its process group id. */ +export const DEAD_PID: number = 4242; +/** The fake caller's own pid (and, by default, its process group id). */ +export const SELF_PID: number = 1000; +const GRACE_MS: number = 100; +const CLOCK_START: number = 0; + +/** A fake process table recording every signal sent and every message logged. */ +export interface IFakeGroup { + readonly signals: NodeJS.Signals[]; + readonly logs: string[]; + readonly options: IDaemonOrphanReaperOptions; +} + +/** Describes the fake process table. */ +export interface IFakeGroupSpec { + readonly daemonAlive?: boolean; + /** The signal after which the group is gone; omitted means it never exits. */ + readonly exitsOn?: NodeJS.Signals; + readonly ownGroupId?: number; + readonly unknownOwnGroup?: boolean; + readonly anyGroupExists?: boolean; +} + +/** Creates a fake process table with a virtual clock. */ +export function createFakeGroup(spec: IFakeGroupSpec): IFakeGroup { + const signals: NodeJS.Signals[] = []; + const logs: string[] = []; + let clock: number = CLOCK_START; + const ops: IDaemonProcessGroupOps = { + isProcessAlive: () => spec.daemonAlive === true, + groupExists: (groupId: number) => + (spec.anyGroupExists === true || groupId === DEAD_PID) && + !signals.some((signal: NodeJS.Signals) => signal === spec.exitsOn), + signalGroup: (groupId: number, signal: NodeJS.Signals) => signals.push(signal), + ownGroupId: () => (spec.unknownOwnGroup === true ? undefined : (spec.ownGroupId ?? SELF_PID)), + delayAsync: async (ms: number) => { + clock += ms; + }, + now: () => clock, + log: (message: string) => logs.push(message) + }; + return { signals, logs, options: { ops, platform: 'linux', selfPid: SELF_PID, graceMs: GRACE_MS } }; +} From 7aecafacdc79cfad5110694b743ebcaa06a05589 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:16:46 -0700 Subject: [PATCH 005/265] [rush-daemon] Stop --verbose / --parallelism / --timeline from reloading the warm graph (#6063) * [rush-daemon] Apply --verbose/--parallelism per request instead of reloading the warm graph Presentation and scheduling flags (--verbose, --parallelism, --timeline) are no longer part of the engine parameter identity, so they no longer force a tier-1 reload that discards all retained results. rushd applies quietMode/parallelism to the shared graph before each iteration and only coalesces requests with equal settings. Fixes #6048 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-lib] Update API reports for per-request engine settings Fixes #6048 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Test that differing request settings are not batched together Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../rush/flag-reload_2026-09-24.json | 11 ++ .../rush-daemon/flag-reload_2026-09-24.json | 11 ++ common/reviews/api/rush-daemon.api.md | 4 +- common/reviews/api/rush-lib.api.md | 9 ++ .../src/DaemonRequestDispatcher.ts | 6 +- .../rush-daemon/src/PhasedRequestRouter.ts | 30 ++++- .../src/ProductionDaemonRequestResolver.ts | 5 +- .../src/test/PhasedRequestBatching.test.ts | 40 +++++++ .../WorkspaceRequestScopedParameters.test.ts | 52 +++++++++ .../rush-lib/src/api/PhasedCommandEngine.ts | 17 +++ ...asedCommandEngineParameterIdentity.test.ts | 110 ++++++++++++++++++ .../cli/scriptActions/PhasedScriptAction.ts | 31 ++++- libraries/rush-lib/src/index.ts | 1 + 13 files changed, 319 insertions(+), 8 deletions(-) create mode 100644 common/changes/@microsoft/rush/flag-reload_2026-09-24.json create mode 100644 common/changes/@rushstack/rush-daemon/flag-reload_2026-09-24.json create mode 100644 libraries/rush-daemon/src/test/WorkspaceRequestScopedParameters.test.ts create mode 100644 libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts diff --git a/common/changes/@microsoft/rush/flag-reload_2026-09-24.json b/common/changes/@microsoft/rush/flag-reload_2026-09-24.json new file mode 100644 index 0000000000..705c6b3397 --- /dev/null +++ b/common/changes/@microsoft/rush/flag-reload_2026-09-24.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Exclude `--verbose`, `--parallelism` and `--timeline` from the daemon engine parameter identity and expose them as per-request settings, so these flags no longer force the Rush daemon to reload its warm operation graph.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/flag-reload_2026-09-24.json b/common/changes/@rushstack/rush-daemon/flag-reload_2026-09-24.json new file mode 100644 index 0000000000..4c774a0912 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/flag-reload_2026-09-24.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Apply `--verbose` and `--parallelism` to the warm graph per iteration instead of reloading the graph, and only coalesce requests that share these settings.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index 99cc2ca6a5..d823a0545e 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -27,6 +27,7 @@ import type { IDaemonWarmSetStatus } from '@rushstack/rush-daemon-protocol'; import type { IDaemonWorkspaceStatus } from '@rushstack/rush-daemon-protocol'; import type { IInputsSnapshot } from '@microsoft/rush-lib'; import { IOperationGraph } from '@microsoft/rush-lib'; +import type { IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; import type { ITerminal } from '@rushstack/terminal'; import type { LockFile } from '@rushstack/node-core-library'; import { Operation } from '@microsoft/rush-lib'; @@ -399,6 +400,7 @@ export interface IResolvedDaemonPhasedRequest { readonly kind: 'phased'; // (undocumented) readonly request: IDaemonPhasedRequest; + readonly requestSettings?: IPhasedCommandEngineRequestSettings; } // @beta @@ -666,7 +668,7 @@ export type MapWorkspaceInvalidationsToOperationsAsync = (options: IMapWorkspace // @beta export class PhasedRequestRouter { constructor(workspaceSession: IWorkspaceSession); - executeAsync(request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection?: boolean, onExecutionStarting?: () => void): Promise; + executeAsync(request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection?: boolean, onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings): Promise; } // @beta diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 2157f656c6..c553ceacb8 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -953,6 +953,14 @@ export interface IPhasedCommandEngine extends AsyncDisposable { readonly rushSession: RushSession; } +// @alpha +export interface IPhasedCommandEngineRequestSettings { + // (undocumented) + readonly parallelism: Parallelism; + // (undocumented) + readonly quietMode: boolean; +} + // @alpha export interface IPhasedCommandPlugin { apply(hooks: PhasedCommandHooks): void; @@ -1512,6 +1520,7 @@ export class PhasedCommandEngine { readonly parameterIdentity: string; // (undocumented) static parseAsync(options: IParsePhasedCommandOptions): Promise; + get requestSettings(): IPhasedCommandEngineRequestSettings; selectOperationsAsync(graph: IOperationGraph): Promise>; } diff --git a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts index 38f8d53e7f..e78e631731 100644 --- a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts +++ b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; import type { IDaemonCommandResult, IDaemonEventEnvelope, @@ -32,6 +33,8 @@ export interface IResolvedDaemonPhasedRequest { readonly request: IDaemonPhasedRequest; /** Native selection has already resolved all required project/phase dependencies. */ readonly exactSelection?: boolean; + /** Verbosity and parallelism for this request; applied to the shared graph before its iteration. */ + readonly requestSettings?: IPhasedCommandEngineRequestSettings; } /** A resolver outcome that uses the existing isolated global executor contract. @beta */ @@ -187,7 +190,8 @@ async function dispatchWorkspaceRequestAsync( resolved.request, createPhasedClient(client), resolved.exactSelection, - onExecutionStarting + onExecutionStarting, + resolved.requestSettings ); } const globalRouter: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(workspaceSession); diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index e6a8416e45..f9cd596123 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -4,6 +4,7 @@ import type { IOperationExecutionResult, IOperationGraph, + IPhasedCommandEngineRequestSettings, Operation, _IOperationGraphEventSink } from '@microsoft/rush-lib'; @@ -65,6 +66,9 @@ interface IPreparedPhasedRequest { readonly exclusivityClass: RequestExclusivityClass; readonly interactiveSession: IInteractiveRequestSession | undefined; readonly request: IDaemonPhasedRequest; + /** Only requests with the same settings share one graph iteration. */ + readonly requestSettings: IPhasedCommandEngineRequestSettings | undefined; + readonly requestSettingsKey: string; readonly selection: IResolvedSelection; readonly warningsAllowedByEnvironment: boolean; } @@ -116,7 +120,8 @@ export class PhasedRequestRouter { request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection: boolean = false, - onExecutionStarting?: () => void + onExecutionStarting?: () => void, + requestSettings?: IPhasedCommandEngineRequestSettings ): Promise { validateRequestIdentity(request); const interactiveSession: IInteractiveRequestSession | undefined = validateInteractiveSession( @@ -194,6 +199,8 @@ export class PhasedRequestRouter { exclusivityClass, interactiveSession, request, + requestSettings, + requestSettingsKey: JSON.stringify(requestSettings ?? null), selection, warningsAllowedByEnvironment, onExecutionStarting @@ -342,14 +349,19 @@ class PhasedRequestBatchCoordinator { } return ( this.#acceptingCurrentBatch && - this.#currentBatch?.[0]?.exclusivityClass === RequestExclusivityClass.SharedBuild + this.#currentBatch?.[0]?.exclusivityClass === RequestExclusivityClass.SharedBuild && + this.#currentBatch[0].requestSettingsKey === request.requestSettingsKey ); } #takeCompatiblePending(batch: IBatchEntry[]): void { + const { requestSettingsKey } = batch[0]; for (let index: number = 0; index < this.#pending.length; ) { const entry: IBatchEntry = this.#pending[index]; - if (entry.exclusivityClass === RequestExclusivityClass.SharedBuild) { + if ( + entry.exclusivityClass === RequestExclusivityClass.SharedBuild && + entry.requestSettingsKey === requestSettingsKey + ) { this.#pending.splice(index, 1); entry.executionStarted = true; batch.push(entry); @@ -399,6 +411,7 @@ class PhasedRequestBatchCoordinator { return; } + applyRequestSettings(this.#graph, participants[0].requestSettings); applySelections( this.#graph, participants.map((entry: IBatchEntry) => entry.selection) @@ -885,6 +898,17 @@ function collectSelectionClosure( return Array.from(activeOperations); } +/** Presentation/scheduling settings are request-scoped, so they are applied per iteration, not per graph. */ +function applyRequestSettings( + graph: IOperationGraph, + settings: IPhasedCommandEngineRequestSettings | undefined +): void { + if (settings) { + graph.quietMode = settings.quietMode; + graph.parallelism = settings.parallelism; + } +} + function applySelections(graph: IOperationGraph, selections: ReadonlyArray): void { const enabledClosureBySelection: ReadonlyArray> = selections.map( (selection: IResolvedSelection) => diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 4bc77ac193..6c503f46ee 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -38,7 +38,9 @@ import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; * Binds the standalone host to a real native build/rebuild graph on its first request. * * @remarks - * A host is pinned to its first command and non-selection parameters. Incompatible parameters, + * A host is pinned to its first command and graph-affecting, non-selection parameters. Presentation and + * scheduling parameters (`--verbose`, `--parallelism`, `--timeline`) are applied per request instead. + * Incompatible parameters, * environments, or graph inputs are rejected before scheduling; no request is retried automatically. * The initial supported surface excludes external plugins, .env initialization, install/watch, * event-hook scripts, and rushx/global commands. Use the unchanged native CLI for those surfaces. @@ -118,6 +120,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { return { kind: 'phased', exactSelection: true, + requestSettings: command.requestSettings, request: { admission: envelope.admission, commandName: envelope.commandName, diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index affc97c12c..3d67e07eb5 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -10,6 +10,7 @@ import type { } from '@rushstack/rush-daemon-protocol'; import { RUSHD_OPERATION_HEADER, RUSHD_OPERATION_STREAM_CLOSED } from '@rushstack/rush-daemon-protocol'; import { OperationStatus } from '@microsoft/rush-lib'; +import type { IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; import { @@ -140,6 +141,45 @@ function eventOperationId(event: IDaemonEventEnvelope): string | undefined { } describe('shared phased request batching', () => { + it('schedules separate iterations for overlapping requests with different request settings', async () => { + const fixture: ITestRoutingFixture = createFixture(); + const graph: ITestRoutingFixture['graph'] = fixture.graph; + const scheduledSettings: IPhasedCommandEngineRequestSettings[] = []; + const originalScheduleAsync: typeof graph.scheduleIterationAsync = + graph.scheduleIterationAsync.bind(graph); + const scheduleSpy: jest.SpyInstance = jest + .spyOn(graph, 'scheduleIterationAsync') + .mockImplementation((...args: Parameters) => { + scheduledSettings.push({ parallelism: graph.parallelism, quietMode: graph.quietMode }); + return originalScheduleAsync(...args); + }); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const defaultSettings: IPhasedCommandEngineRequestSettings = { parallelism: 4, quietMode: true }; + const verboseSerialSettings: IPhasedCommandEngineRequestSettings = { parallelism: 1, quietMode: false }; + + const [first, second] = await Promise.all([ + router.executeAsync( + createRequest('default', OPERATION_A), + new TestPhasedRequestClient('one'), + false, + undefined, + defaultSettings + ), + router.executeAsync( + createRequest('verbose-serial', OPERATION_B), + new TestPhasedRequestClient('two'), + false, + undefined, + verboseSerialSettings + ) + ]); + + expect(scheduleSpy).toHaveBeenCalledTimes(2); + expect(scheduledSettings).toEqual([defaultSettings, verboseSerialSettings]); + expect(first).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(second).toMatchObject({ exitCode: 0, outcome: 'success' }); + }); + it('merges overlapping selections into one real graph iteration and executes shared operations once', async () => { const fixture: ITestRoutingFixture = createFixture(); const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); diff --git a/libraries/rush-daemon/src/test/WorkspaceRequestScopedParameters.test.ts b/libraries/rush-daemon/src/test/WorkspaceRequestScopedParameters.test.ts new file mode 100644 index 0000000000..dddefa02d3 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceRequestScopedParameters.test.ts @@ -0,0 +1,52 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { WorkspaceInputChangeTier, type IOperationGraph } from '@microsoft/rush-lib'; + +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { setDaemonPolicy } from './WarmGenerationTestUtilities'; + +jest.setTimeout(30_000); + +it('applies --verbose and --parallelism per request without reloading the warm graph', async () => { + const fixture = await DaemonGraphTestFixture.createAsync((created) => setDaemonPolicy(created, {})); + try { + const buildAsync = async (...extra: string[]): Promise => { + const result = await fixture.runAsync(['build', '--to', 'b', ...extra]); + expect(result.terminal).toMatchObject({ payload: { exitCode: 0 } }); + }; + await buildAsync(); + expect(fixture.runs()).toEqual(['a', 'b']); + const generation: number = fixture.host.workspaceGeneration; + const graph: IOperationGraph | undefined = fixture.session.operationGraph; + const defaultParallelism: number = graph!.parallelism; + const expectWarm = (): void => { + expect(fixture.host.workspaceGeneration).toBe(generation); + expect(fixture.session.operationGraph).toBe(graph); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(WorkspaceInputChangeTier.Reuse); + expect(fixture.runs()).toEqual(['a', 'b']); + }; + + await buildAsync(); + expectWarm(); + await buildAsync('--verbose'); + expectWarm(); + expect(graph!.quietMode).toBe(false); + await buildAsync(); + expectWarm(); + expect(graph!.quietMode).toBe(true); + await buildAsync('-p', '1', '--timeline'); + expectWarm(); + expect(graph!.parallelism).toBe(1); + await buildAsync(); + expectWarm(); + expect(graph!.parallelism).toBe(defaultParallelism); + + fixture.write('a/input.txt', 'two'); + await buildAsync('--verbose', '-p', '1'); + expect(fixture.host.workspaceGeneration).toBe(generation); + expect(fixture.runs()).toEqual(['a', 'b', 'a', 'b']); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); diff --git a/libraries/rush-lib/src/api/PhasedCommandEngine.ts b/libraries/rush-lib/src/api/PhasedCommandEngine.ts index 0fefc3fddf..e5246fbfa1 100644 --- a/libraries/rush-lib/src/api/PhasedCommandEngine.ts +++ b/libraries/rush-lib/src/api/PhasedCommandEngine.ts @@ -12,6 +12,7 @@ import { PhasedScriptAction } from '../cli/scriptActions/PhasedScriptAction'; import type { GetInputsSnapshotAsyncFn, IInputsSnapshot } from '../logic/incremental/InputsSnapshot'; import type { IOperationGraph } from '../logic/operations/IOperationGraph'; import type { Operation, OperationEnabledState } from '../logic/operations/Operation'; +import type { Parallelism } from '../logic/operations/ParseParallelism'; import { PhasedCommandEngineExecution } from '../logic/operations/PhasedCommandEngineExecution'; import type { RushSession } from '../pluginFramework/RushSession'; import type { RushConfiguration } from './RushConfiguration'; @@ -44,6 +45,17 @@ export interface IParsePhasedCommandOptions { readonly terminalProvider: ITerminalProvider; } +/** + * Presentation and scheduling settings of one parsed command. They do not affect the operation graph or any + * operation hash, so they are not part of `PhasedCommandEngine.parameterIdentity`; hosts apply them to the + * shared graph (`IOperationGraph.quietMode` / `IOperationGraph.parallelism`) before each iteration. + * @alpha + */ +export interface IPhasedCommandEngineRequestSettings { + readonly quietMode: boolean; + readonly parallelism: Parallelism; +} + /** * A parsed native build/rebuild command. Parsing never runs scripts or changes cwd/process.env. * @@ -164,4 +176,9 @@ export class PhasedCommandEngine { ): Promise> { return await this._action.selectEngineOperationsAsync(graph); } + + /** Presentation and scheduling settings requested by this command; not part of `parameterIdentity`. */ + public get requestSettings(): IPhasedCommandEngineRequestSettings { + return this._action.getEngineRequestSettings(); + } } diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts new file mode 100644 index 0000000000..6aebbe7d35 --- /dev/null +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts @@ -0,0 +1,110 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { NoOpTerminalProvider } from '@rushstack/terminal'; + +import { PhasedCommandEngine } from '../PhasedCommandEngine'; +import { RushConfiguration } from '../RushConfiguration'; +import { Rush } from '../Rush'; +import { parseParallelism } from '../../logic/operations/ParseParallelism'; + +describe(`${PhasedCommandEngine.name} parameter identity`, () => { + let folder: string; + let rushConfiguration: RushConfiguration; + + function write(name: string, value: unknown): void { + const filename: string = path.join(folder, name); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, JSON.stringify(value)); + } + + async function parseAsync(...argv: string[]): Promise { + return await PhasedCommandEngine.parseAsync({ + argv, + cwd: folder, + rushConfiguration, + terminalProvider: new NoOpTerminalProvider() + }); + } + + beforeAll(() => { + folder = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-engine-identity-'))); + write('rush.json', { + rushVersion: Rush.version, + npmVersion: '10.0.0', + projectFolderMinDepth: 1, + projects: [{ packageName: 'a', projectFolder: 'a' }] + }); + write('a/package.json', { name: 'a', version: '1.0.0', scripts: { '_phase:compile': 'node -v' } }); + write('common/config/rush/command-line.json', { + phases: [{ name: '_phase:compile', dependencies: { upstream: ['_phase:compile'] } }], + commands: [ + { + commandKind: 'phased', + name: 'build', + summary: 'Build', + phases: ['_phase:compile'], + incremental: true, + enableParallelism: true + } + ], + parameters: [ + { + parameterKind: 'flag', + longName: '--production', + description: 'A graph-affecting custom parameter', + associatedCommands: ['build'], + associatedPhases: ['_phase:compile'] + } + ] + }); + rushConfiguration = RushConfiguration.loadFromConfigurationFile(path.join(folder, 'rush.json')); + }); + + afterAll(() => { + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it('excludes presentation and scheduling parameters from the identity', async () => { + const baseline: string = (await parseAsync('build')).parameterIdentity; + for (const argv of [ + ['build', '--verbose'], + ['build', '-v'], + ['build', '--parallelism', '2'], + ['build', '-p', 'max'], + ['build', '--timeline'], + ['build', '--to', 'a'], + ['build', '--verbose', '-p', '1', '--timeline', '--only', 'a'] + ]) { + expect((await parseAsync(...argv)).parameterIdentity).toBe(baseline); + } + }); + + it('includes graph-affecting parameters and the command name in the identity', async () => { + const baseline: string = (await parseAsync('build')).parameterIdentity; + expect((await parseAsync('build', '--production')).parameterIdentity).not.toBe(baseline); + expect((await parseAsync('build', '--production', '--verbose')).parameterIdentity).toBe( + (await parseAsync('build', '--production')).parameterIdentity + ); + expect((await parseAsync('rebuild')).parameterIdentity).not.toBe(baseline); + }); + + it('reports the excluded settings per request', async () => { + expect((await parseAsync('build')).requestSettings).toEqual({ + quietMode: true, + parallelism: parseParallelism(undefined) + }); + expect((await parseAsync('build', '--verbose', '-p', '2')).requestSettings).toEqual({ + quietMode: false, + parallelism: 2 + }); + expect((await parseAsync('build', '-p', '50%')).requestSettings).toEqual({ + quietMode: true, + parallelism: { scalar: 0.5 } + }); + }); +}); diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index 2818bca1f7..89a0dbd3f7 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -32,7 +32,10 @@ import type { IOperationGraph, IOperationGraphIterationOptions } from '../../logic/operations/IOperationGraph'; -import type { IPhasedCommandEngine } from '../../api/PhasedCommandEngine'; +import type { + IPhasedCommandEngine, + IPhasedCommandEngineRequestSettings +} from '../../api/PhasedCommandEngine'; import { PhasedCommandEngineConfigurationChangedError } from '../../api/PhasedCommandEngineConfigurationChangedError'; import { getDaemonIpcImplementationIdentityAsync } from '../../logic/operations/DaemonIpcConfiguration'; import { SetupChecks } from '../../logic/SetupChecks'; @@ -82,6 +85,17 @@ import { _isRushSessionOperationStreamEnabled } from '../../pluginFramework/Rush const PERF_PREFIX: 'rush:phasedScriptAction' = 'rush:phasedScriptAction'; +/** + * Parameters that change neither the operation graph nor any operation hash. A long-lived engine applies + * them per request (see `getEngineRequestSettings`), so they are excluded from the engine parameter identity. + * `--timeline` only adds a presentation plugin whose output is discarded by engine hosts. + */ +const ENGINE_REQUEST_SCOPED_PARAMETER_NAMES: ReadonlySet = new Set([ + '--verbose', + '--parallelism', + '--timeline' +]); + /** * The set of overall execution statuses that mean the command did what was asked of it and should * exit with code 0. @@ -370,10 +384,23 @@ export class PhasedScriptAction extends BaseScriptAction i return JSON.stringify([ this.actionName, this.parser.getParameterStringMap(), - Object.entries(this.getParameterStringMap()).filter(([name]) => !selectionNames.has(name)) + Object.entries(this.getParameterStringMap()).filter( + ([name]) => !selectionNames.has(name) && !ENGINE_REQUEST_SCOPED_PARAMETER_NAMES.has(name) + ) ]); } + /** + * Output verbosity and scheduling settings for one engine request. These are excluded from + * `getEngineParameterIdentity` and must be applied to the shared graph before each iteration. + */ + public getEngineRequestSettings(): IPhasedCommandEngineRequestSettings { + return { + quietMode: !this.#verboseParameter.value, + parallelism: this.#enableParallelism ? parseParallelism(this.#parallelismParameter?.value) : 1 + }; + } + public async selectEngineOperationsAsync( graph: IOperationGraph ): Promise> { diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index ebd267247c..8af91cfeca 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -177,6 +177,7 @@ export { OperationStatus } from './logic/operations/OperationStatus'; export { PhasedCommandEngine, type IPhasedCommandEngine, + type IPhasedCommandEngineRequestSettings, type IParsePhasedCommandOptions } from './api/PhasedCommandEngine'; export { PhasedCommandEngineConfigurationChangedError } from './api/PhasedCommandEngineConfigurationChangedError'; From 570fe51e31e82dd75bc202938da95e4bb4239bce Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:16:55 -0700 Subject: [PATCH 006/265] [rush-daemon] Surface warm snapshot and daemon shutdown errors to the right client (#6070) * [rush-daemon] Surface warm snapshot and daemon shutdown errors to the right client Fixes #6059 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Update API reports Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Report the shutdown reason for requests aborted before engine initialization Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Move request-scoped reconcile diagnostics into EngineTerminalProvider Keeps the ProductionDaemonRequestResolver change to a one-line call so it composes with other open daemon PRs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Address review: shutdown reason for queued and raw-mode requests, protocol minor 11 - Report the shutdown reason for requests aborted while waiting for admission, and prefer it on the client. - Keep the shutdown reason when restoring raw mode fails with it. - Begin shutdown before awaiting the acknowledgement write so the reported count matches the aborted set. - Advertise protocol minor 11 for the shutdownAck activeRequests field. - Keep the daemon stop note accurate for every request kind. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Report the shutdown reason from global and graph request aborts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Format with prettier and update the protocol API report Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Update the protocol change file for minor 11 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-cli-client] Expect the cancelled request count from daemon stop --force Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- apps/rush-cli-client/src/daemonCommands.ts | 16 +++- apps/rush-cli-client/src/launchClient.ts | 6 +- apps/rush-cli-client/src/resultDiagnostics.ts | 25 +++++++ .../src/test/launchClient.test.ts | 1 + .../src/test/resultDiagnostics.test.ts | 36 +++++++++ ...ushd-error-surfacing_2026-09-24-01-50.json | 11 +++ ...ushd-error-surfacing_2026-09-24-01-50.json | 11 +++ ...ushd-error-surfacing_2026-09-24-01-50.json | 11 +++ ...ushd-error-surfacing_2026-09-24-01-50.json | 11 +++ common/reviews/api/rush-client-core.api.md | 3 +- .../reviews/api/rush-daemon-protocol.api.md | 7 +- common/reviews/api/rush-daemon.api.md | 21 +++++- .../rush-client-core/src/DaemonClient.ts | 10 ++- .../src/test/DaemonClient.test.ts | 7 +- .../src/ControlMessageValidation.ts | 3 +- .../src/DaemonLifecycleControl.ts | 8 +- .../src/DaemonProtocolVersion.ts | 5 +- .../src/ShutdownAckValidation.ts | 17 +++++ libraries/rush-daemon-protocol/src/index.ts | 1 + .../src/test/LifecycleControl.test.ts | 10 +++ .../rush-daemon/src/DaemonControlSession.ts | 24 ++++-- .../src/DaemonGraphRequestRouter.ts | 8 +- .../rush-daemon/src/DaemonShutdownError.ts | 73 +++++++++++++++++++ .../rush-daemon/src/EngineTerminalProvider.ts | 46 +++++++++++- .../src/GlobalCommandRequestRouter.ts | 14 +++- .../rush-daemon/src/PhasedRequestRouter.ts | 16 +++- .../src/ProductionDaemonRequestResolver.ts | 8 +- libraries/rush-daemon/src/RushDaemonHost.ts | 30 +++++--- .../src/WorkspaceRequestLifecycle.ts | 6 +- libraries/rush-daemon/src/index.ts | 5 ++ libraries/rush-daemon/src/serveRushDaemon.ts | 13 +++- .../src/test/DaemonRequestWireGlobal.test.ts | 46 ++++++++++++ .../src/test/DaemonShutdown.test.ts | 2 +- .../src/test/EngineTerminalProvider.test.ts | 57 +++++++++++++++ .../src/test/PhasedRequestInteractive.test.ts | 30 ++++++++ .../ProductionDaemonRequestResolver.test.ts | 64 ++++++++++++++++ 36 files changed, 620 insertions(+), 42 deletions(-) create mode 100644 apps/rush-cli-client/src/resultDiagnostics.ts create mode 100644 apps/rush-cli-client/src/test/resultDiagnostics.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/fix-rushd-error-surfacing_2026-09-24-01-50.json create mode 100644 common/changes/@rushstack/rush-client-core/fix-rushd-error-surfacing_2026-09-24-01-50.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/fix-rushd-error-surfacing_2026-09-24-01-50.json create mode 100644 common/changes/@rushstack/rush-daemon/fix-rushd-error-surfacing_2026-09-24-01-50.json create mode 100644 libraries/rush-daemon-protocol/src/ShutdownAckValidation.ts create mode 100644 libraries/rush-daemon/src/DaemonShutdownError.ts create mode 100644 libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts diff --git a/apps/rush-cli-client/src/daemonCommands.ts b/apps/rush-cli-client/src/daemonCommands.ts index b6ad07750d..fd19088343 100644 --- a/apps/rush-cli-client/src/daemonCommands.ts +++ b/apps/rush-cli-client/src/daemonCommands.ts @@ -119,7 +119,17 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): } try { if (command === 'stop') { - await client.shutdownAsync(); + const { activeRequests } = await client.shutdownAsync(); + if (activeRequests) { + await writeStreamAsync( + process.stderr, + Buffer.from( + `rush-client: the daemon was running ${activeRequests} request(s); they were cancelled.\n` + ) + ); + } + const cancelled: { cancelledRequests?: number } = + activeRequests === undefined ? {} : { cancelledRequests: activeRequests }; if (options.argv[1] === '--force') { // Wait for the acknowledged daemon to release its listener and record, then clear leftovers // such as an abandoned startup reservation in the same invocation. @@ -129,13 +139,15 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): await writeStatusAsync({ state: 'shutdownAccepted', socketPath: connectionOptions.paths.socketPath, + ...cancelled, removedPaths }); return; } await writeStatusAsync({ state: 'shutdownAccepted', - socketPath: connectionOptions.paths.socketPath + socketPath: connectionOptions.paths.socketPath, + ...cancelled }); return; } diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index af3cadb9c1..102c9b732b 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -27,6 +27,7 @@ import { formatAdmissionFailure, getConfiguredAdmission } from './ClientAdmissio import { ClientOperationRenderer } from './ClientOperationRenderer'; import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; import { selectClientRoute, type IClientRoute } from './routing'; +import { getResultDiagnostic } from './resultDiagnostics'; import { writeStreamAsync } from './writeStreamAsync'; import { getBundledRushVersion, @@ -206,7 +207,10 @@ export async function launchClientAsync(rushx: boolean): Promise { } if (outcome.kind === 'result') { process.exitCode = outcome.result.exitCode; - if (outcome.result.admissionErrorCode) { + const diagnostic: string | undefined = getResultDiagnostic(outcome.result); + if (diagnostic) { + await writeStreamAsync(process.stderr, Buffer.from(diagnostic)); + } else if (outcome.result.admissionErrorCode) { await writeStreamAsync( process.stderr, Buffer.from(formatAdmissionFailure(outcome.result.admissionErrorCode, request.admission)) diff --git a/apps/rush-cli-client/src/resultDiagnostics.ts b/apps/rush-cli-client/src/resultDiagnostics.ts new file mode 100644 index 0000000000..e65be24364 --- /dev/null +++ b/apps/rush-cli-client/src/resultDiagnostics.ts @@ -0,0 +1,25 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonCommandResult } from '@rushstack/rush-daemon-protocol'; + +/** + * Returns the stderr line that explains a failed daemon result, if any. + * + * @remarks + * A non-zero result's error message (for example, a daemon shutdown that aborted the request) is the only + * place the daemon reports failures that are not attributed to an operation, so it must not be dropped. + * Returns `undefined` for `no-wait` and `wait-timeout` admission failures, which `formatAdmissionFailure` + * explains. + */ +export function getResultDiagnostic( + result: Pick +): string | undefined { + // A request aborted while waiting for admission carries the reason (such as a daemon shutdown) in its + // error message; other admission failures are explained by `formatAdmissionFailure`. + if (result.admissionErrorCode !== undefined && result.admissionErrorCode !== 'aborted') return undefined; + if (result.exitCode !== 0 && result.errorMessage) { + return `rush-client: ${result.errorMessage}\n`; + } + return undefined; +} diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index b3ec694fca..d454c99c4e 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -288,6 +288,7 @@ describe('standalone rushx fallback', () => { expect(JSON.parse(result.stdout)).toEqual({ state: 'shutdownAccepted', socketPath: paths.socketPath, + cancelledRequests: 0, removedPaths: [reservation] }); expect(fs.existsSync(reservation)).toBe(false); diff --git a/apps/rush-cli-client/src/test/resultDiagnostics.test.ts b/apps/rush-cli-client/src/test/resultDiagnostics.test.ts new file mode 100644 index 0000000000..46c0ad5814 --- /dev/null +++ b/apps/rush-cli-client/src/test/resultDiagnostics.test.ts @@ -0,0 +1,36 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { getResultDiagnostic } from '../resultDiagnostics'; + +describe(getResultDiagnostic.name, () => { + it('prints the error message of a failed result', () => { + expect( + getResultDiagnostic({ + exitCode: 1, + errorMessage: 'The Rush daemon was shut down (idle timeout) while this request was running.' + }) + ).toBe('rush-client: The Rush daemon was shut down (idle timeout) while this request was running.\n'); + }); + + it('prefers the reason of a request aborted while waiting for admission', () => { + expect( + getResultDiagnostic({ exitCode: 1, admissionErrorCode: 'aborted', errorMessage: 'daemon shut down' }) + ).toBe('rush-client: daemon shut down\n'); + }); + + it('leaves no-wait and wait-timeout admission failures to the admission formatter', () => { + expect( + getResultDiagnostic({ exitCode: 1, admissionErrorCode: 'wait-timeout', errorMessage: 'x' }) + ).toBeUndefined(); + expect( + getResultDiagnostic({ exitCode: 1, admissionErrorCode: 'no-wait', errorMessage: 'x' }) + ).toBeUndefined(); + expect(getResultDiagnostic({ exitCode: 1, admissionErrorCode: 'aborted' })).toBeUndefined(); + }); + + it('stays silent for successful results and failures without a message', () => { + expect(getResultDiagnostic({ exitCode: 0, errorMessage: 'ignored' })).toBeUndefined(); + expect(getResultDiagnostic({ exitCode: 1 })).toBeUndefined(); + }); +}); diff --git a/common/changes/@rushstack/rush-cli-client/fix-rushd-error-surfacing_2026-09-24-01-50.json b/common/changes/@rushstack/rush-cli-client/fix-rushd-error-surfacing_2026-09-24-01-50.json new file mode 100644 index 0000000000..925deeffab --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/fix-rushd-error-surfacing_2026-09-24-01-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Print the error message of a failed daemon result (for example, a daemon shutdown that cancelled the build), and report cancelled requests from \"rush-client daemon stop\".", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/fix-rushd-error-surfacing_2026-09-24-01-50.json b/common/changes/@rushstack/rush-client-core/fix-rushd-error-surfacing_2026-09-24-01-50.json new file mode 100644 index 0000000000..fad92bb8f4 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/fix-rushd-error-surfacing_2026-09-24-01-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Return the daemon shutdown acknowledgement, including its optional active request count, from DaemonClient.shutdownAsync().", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/fix-rushd-error-surfacing_2026-09-24-01-50.json b/common/changes/@rushstack/rush-daemon-protocol/fix-rushd-error-surfacing_2026-09-24-01-50.json new file mode 100644 index 0000000000..5aebbe2a85 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/fix-rushd-error-surfacing_2026-09-24-01-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add an optional \"activeRequests\" count to the shutdownAck control message; advertised as protocol minor 11 (`DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR`); older peers omit or ignore it.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/fix-rushd-error-surfacing_2026-09-24-01-50.json b/common/changes/@rushstack/rush-daemon/fix-rushd-error-surfacing_2026-09-24-01-50.json new file mode 100644 index 0000000000..6af207cd00 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/fix-rushd-error-surfacing_2026-09-24-01-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Report warm input-snapshot failures (with the engine diagnostics) to the failing request instead of replaying them into the next request, abort requests interrupted by a daemon shutdown with a typed DaemonShutdownError that names its initiator, and report running requests in the shutdown acknowledgement.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index ed0f2f1813..465a2ce471 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -13,6 +13,7 @@ import { IDaemonPongMessage } from '@rushstack/rush-daemon-protocol'; import { IDaemonProtocolVersion } from '@rushstack/rush-daemon-protocol'; import { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; import { IDaemonRequestRejectedMessage } from '@rushstack/rush-daemon-protocol'; +import { IDaemonShutdownAckMessage } from '@rushstack/rush-daemon-protocol'; import type { Readable } from 'node:stream'; // @beta @@ -29,7 +30,7 @@ export class DaemonClient { static connectAsync(options: IDaemonClientConnectOptions): Promise; executeAsync(options: IDaemonClientExecuteOptions): Promise; get protocolVersion(): IDaemonProtocolVersion; - shutdownAsync(timeoutMs?: number): Promise; + shutdownAsync(timeoutMs?: number): Promise; get status(): Promise; } diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index 8ca51ef110..0584bcb366 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -79,6 +79,9 @@ export const DAEMON_REQUEST_ADMISSION_PROTOCOL_MINOR: number; // @beta export const DAEMON_REQUEST_LIFECYCLE_PROTOCOL_MINOR: number; +// @beta +export const DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR: number; + // @beta export const DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR: number; @@ -592,7 +595,9 @@ export interface IDaemonShutdownAckMessage { // (undocumented) readonly kind: 'shutdownAck'; // (undocumented) - readonly payload: Record; + readonly payload: { + readonly activeRequests?: number; + }; } // @beta diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index d823a0545e..dfb68a7841 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -67,6 +67,18 @@ export class DaemonRequiresInProcessError extends Error { readonly policy: IDaemonTerminalPolicyResult; } +// @beta +export class DaemonShutdownError extends Error { + constructor(options: IDaemonShutdownErrorOptions); + // (undocumented) + readonly initiator: DaemonShutdownInitiator; + // (undocumented) + readonly signal: string | undefined; +} + +// @beta +export type DaemonShutdownInitiator = 'controlClient' | 'signal' | 'idleTimeout' | 'restart' | 'host'; + // @beta export type DispatchWorkspaceRequestAsync = (options: IDispatchWorkspaceRequestOptions) => Promise; @@ -178,6 +190,13 @@ export interface IDaemonRequestResolver { readonly workspaceLifecycle?: IWorkspaceResolverLifecycle; } +// @beta +export interface IDaemonShutdownErrorOptions { + // (undocumented) + readonly initiator: DaemonShutdownInitiator; + readonly signal?: string; +} + // @beta export interface IDispatchWorkspaceRequestOptions { // (undocumented) @@ -725,7 +744,7 @@ export type ResolvedDaemonRequest = IResolvedDaemonPhasedRequest | IResolvedDaem // @beta export class RushDaemonHost { - closeAsync(): Promise; + closeAsync(reason?: DaemonShutdownError): Promise; readonly closed: Promise; getWorkspaceSessionAsync(): Promise; // (undocumented) diff --git a/libraries/rush-client-core/src/DaemonClient.ts b/libraries/rush-client-core/src/DaemonClient.ts index 911021c50c..9cdc0e5b21 100644 --- a/libraries/rush-client-core/src/DaemonClient.ts +++ b/libraries/rush-client-core/src/DaemonClient.ts @@ -25,7 +25,8 @@ import { type IDaemonPongMessage, type IDaemonProtocolVersion, type IDaemonRequestEnvelope, - type IDaemonRequestRejectedMessage + type IDaemonRequestRejectedMessage, + type IDaemonShutdownAckMessage } from '@rushstack/rush-daemon-protocol'; import { connectDaemonAsync, type DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; @@ -106,6 +107,7 @@ export class DaemonClient { #result: IDeferred | undefined; #shutdown: IDeferred | undefined; #shutdownAcknowledged: boolean = false; + #shutdownAck: IDaemonShutdownAckMessage['payload'] = {}; #execution: IDaemonClientExecuteOptions | undefined; #finished: boolean = false; #inputStarted: boolean = false; @@ -190,8 +192,10 @@ export class DaemonClient { * Requests shutdown on a fresh connection and waits for acknowledgement followed by EOF. * @remarks This confirms acceptance and connection closure, not successful workspace cleanup. * Requires protocol 0.6. The timeout defaults to 15000 milliseconds. + * @returns The acknowledgement, including the number of running requests the shutdown aborts when the + * daemon reports it. */ - public async shutdownAsync(timeoutMs: number = 15000): Promise { + public async shutdownAsync(timeoutMs: number = 15000): Promise { if (this.#used) throw new Error('Create a fresh DaemonClient for shutdown.'); this.#used = true; let timer: ReturnType | undefined; @@ -210,6 +214,7 @@ export class DaemonClient { ); }, timeoutMs); await Promise.all([this.#shutdown.promise, this.#sendControlAsync({ kind: 'shutdown', payload: {} })]); + return this.#shutdownAck; } finally { clearTimeout(timer); await this.closeAsync(); @@ -379,6 +384,7 @@ export class DaemonClient { throw new DaemonProtocolError('malformedControlMessage', 'Unexpected shutdown acknowledgement.'); } this.#shutdownAcknowledged = true; + this.#shutdownAck = message.payload; return; } const execution: IDaemonClientExecuteOptions = this.#requireExecution(); diff --git a/libraries/rush-client-core/src/test/DaemonClient.test.ts b/libraries/rush-client-core/src/test/DaemonClient.test.ts index d5aaf43094..9a2ccc6af8 100644 --- a/libraries/rush-client-core/src/test/DaemonClient.test.ts +++ b/libraries/rush-client-core/src/test/DaemonClient.test.ts @@ -521,21 +521,22 @@ describe('DaemonClient', () => { }); onRequest = async (message) => { if (message.kind === 'shutdown') { - await sendAsync({ kind: 'shutdownAck', payload: {} }); + await sendAsync({ kind: 'shutdownAck', payload: { activeRequests: 2 } }); acknowledged(); } }; const client = await DaemonClient.connectAsync({ socketPath: address }); expect(client.protocolVersion.minor).toBeGreaterThanOrEqual(6); let completed: boolean = false; - const shutdown: Promise = client.shutdownAsync().then(() => { + const shutdown: Promise = client.shutdownAsync().then((payload) => { completed = true; + return payload; }); await ack; await new Promise((resolve) => setTimeout(resolve, 20)); expect(completed).toBe(false); await connection!.closeAsync(); - await shutdown; + await expect(shutdown).resolves.toEqual({ activeRequests: 2 }); expect(completed).toBe(true); expect(controls.filter((message) => message.kind === 'shutdown')).toHaveLength(1); }); diff --git a/libraries/rush-daemon-protocol/src/ControlMessageValidation.ts b/libraries/rush-daemon-protocol/src/ControlMessageValidation.ts index 8022db1d85..2be3cf95f9 100644 --- a/libraries/rush-daemon-protocol/src/ControlMessageValidation.ts +++ b/libraries/rush-daemon-protocol/src/ControlMessageValidation.ts @@ -13,6 +13,7 @@ import { validateRequestResultControl, validateRequestStartControl } from './RequestControlValidation'; +import { validateShutdownAck } from './ShutdownAckValidation'; import { validateSubscribeControl } from './SubscribeControlValidation'; function fail(reason: string): never { throw new DaemonProtocolError('malformedControlMessage', reason); @@ -67,7 +68,7 @@ const VALIDATORS_BY_KIND: Record = { requestRejected: validateRequestRejectedControl, requestResult: validateRequestResultControl, shutdown: noopValidator, - shutdownAck: noopValidator, + shutdownAck: validateShutdownAck, stdinReady: validateRequestCancelControl, stdinEnd: validateRequestCancelControl }; diff --git a/libraries/rush-daemon-protocol/src/DaemonLifecycleControl.ts b/libraries/rush-daemon-protocol/src/DaemonLifecycleControl.ts index 31921111a2..a7c084ed11 100644 --- a/libraries/rush-daemon-protocol/src/DaemonLifecycleControl.ts +++ b/libraries/rush-daemon-protocol/src/DaemonLifecycleControl.ts @@ -10,5 +10,11 @@ export interface IDaemonShutdownMessage { /** Acknowledges shutdown before the host closes connections and releases its endpoint. @beta */ export interface IDaemonShutdownAckMessage { readonly kind: 'shutdownAck'; - readonly payload: Record; + readonly payload: { + /** + * Requests that were still running and will be aborted by this shutdown. Daemons older than + * `DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR` omit it. + */ + readonly activeRequests?: number; + }; } diff --git a/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts b/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts index a687dae853..b35d1d6883 100644 --- a/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts +++ b/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts @@ -25,6 +25,9 @@ export const DAEMON_INVOCATION_KIND_PROTOCOL_MINOR: number = 8; /** The first minor supporting native mutations and guaranteed pre-execution restart outcomes. @beta */ export const DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR: number = 10; +/** The first additive protocol minor whose shutdown acknowledgement reports the active request count. @beta */ +export const DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR: number = 11; + /** * A rushd wire protocol version. * @@ -58,7 +61,7 @@ export interface IDaemonProtocolVersion { */ export const DAEMON_PROTOCOL_VERSION: IDaemonProtocolVersion = { major: 0, - minor: DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR + minor: DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR }; /** diff --git a/libraries/rush-daemon-protocol/src/ShutdownAckValidation.ts b/libraries/rush-daemon-protocol/src/ShutdownAckValidation.ts new file mode 100644 index 0000000000..5bd6c80cbc --- /dev/null +++ b/libraries/rush-daemon-protocol/src/ShutdownAckValidation.ts @@ -0,0 +1,17 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { DaemonProtocolError } from './DaemonProtocolError'; + +const ZERO: number = 0; + +/** Validates the optional active request count of a shutdown acknowledgement. @internal */ +export function validateShutdownAck(payload: Record): void { + const value: unknown = payload.activeRequests; + if (value === undefined || isNonnegativeInteger(value)) return; + throw new DaemonProtocolError('malformedControlMessage', 'Invalid shutdownAck field "activeRequests".'); +} + +function isNonnegativeInteger(value: unknown): boolean { + return typeof value === 'number' && Number.isSafeInteger(value) && value >= ZERO; +} diff --git a/libraries/rush-daemon-protocol/src/index.ts b/libraries/rush-daemon-protocol/src/index.ts index b170c7e899..1537163377 100644 --- a/libraries/rush-daemon-protocol/src/index.ts +++ b/libraries/rush-daemon-protocol/src/index.ts @@ -29,6 +29,7 @@ export { DAEMON_INVOCATION_KIND_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_LIFECYCLE_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_REQUEST_ADMISSION_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_REQUEST_LIFECYCLE_PROTOCOL_MINOR, DAEMON_PROTOCOL_VERSION } from './DaemonProtocolVersion'; +export { DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { isDaemonProtocolCompatible } from './DaemonProtocolVersion'; export type { IDaemonProtocolVersion } from './DaemonProtocolVersion'; diff --git a/libraries/rush-daemon-protocol/src/test/LifecycleControl.test.ts b/libraries/rush-daemon-protocol/src/test/LifecycleControl.test.ts index 00c5363e2d..b6487b245a 100644 --- a/libraries/rush-daemon-protocol/src/test/LifecycleControl.test.ts +++ b/libraries/rush-daemon-protocol/src/test/LifecycleControl.test.ts @@ -14,6 +14,8 @@ const FRACTION: number = 1.5; const MESSAGES: readonly DaemonControlMessage[] = [ { kind: 'shutdown', payload: {} }, { kind: 'shutdownAck', payload: {} }, + { kind: 'shutdownAck', payload: { activeRequests: ZERO } }, + { kind: 'shutdownAck', payload: { activeRequests: PID } }, { kind: 'pong', payload: { pid: PID, residentMemoryBytes: MEMORY_BYTES, uptimeMs: UPTIME_MS } } ]; @@ -21,6 +23,14 @@ it.each(MESSAGES)('round-trips lifecycle message $kind', (message: DaemonControl expect(decodeDaemonControlMessage(encodeDaemonControlMessage(message))).toEqual(message); }); +it.each([NEGATIVE, FRACTION, '1'])( + 'rejects an invalid shutdown active request count %s', + (value: unknown) => { + const json: string = JSON.stringify({ kind: 'shutdownAck', payload: { activeRequests: value } }); + expect(() => decodeDaemonControlMessage(new TextEncoder().encode(json))).toThrow('activeRequests'); + } +); + it.each([ZERO, NEGATIVE, FRACTION, '42'])('rejects an invalid daemon PID %s', (pid: unknown) => { const json: string = JSON.stringify({ kind: 'pong', payload: { pid, uptimeMs: UPTIME_MS } }); expect(() => decodeDaemonControlMessage(new TextEncoder().encode(json))).toThrow('pid'); diff --git a/libraries/rush-daemon/src/DaemonControlSession.ts b/libraries/rush-daemon/src/DaemonControlSession.ts index 5189f6ffec..51f9bff38a 100644 --- a/libraries/rush-daemon/src/DaemonControlSession.ts +++ b/libraries/rush-daemon/src/DaemonControlSession.ts @@ -32,6 +32,7 @@ import type { IDaemonInteractiveConnection } from './DaemonInteractiveConnection import { MAX_REQUESTS_PER_CONNECTION } from './DaemonConnectionLimits'; import { DaemonRequestDispatchError } from './DaemonRequestDispatcher'; import type { DaemonRequestDispatcher } from './DaemonRequestDispatcher'; +import type { DaemonShutdownError } from './DaemonShutdownError'; import { DaemonWireRequestClient } from './DaemonWireRequestClient'; import { InteractiveInputRoutingError, @@ -49,6 +50,8 @@ export interface IDaemonControlSessionOptions { readonly onError: (error: Error) => void; readonly onRequestStarted?: () => () => void; readonly onShutdownRequested: () => void; + /** Counts requests running on every connection, reported in the shutdown acknowledgement. */ + readonly getActiveRequestCount?: () => number; readonly getWorkspaceStatus?: () => IDaemonWorkspaceStatus; } @@ -103,11 +106,15 @@ export class DaemonControlSession { options.onInteractiveConnection?.(this.#interactiveConnection); } - public closeAsync(drainRequests: boolean = false): Promise { - this.#closePromise ??= this.#closeOnceAsync(drainRequests); + public closeAsync(drainRequests: boolean = false, reason?: DaemonShutdownError): Promise { + this.#closePromise ??= this.#closeOnceAsync(drainRequests, reason); return this.#closePromise; } + public get activeRequestCount(): number { + return this.#requestById.size; + } + async #handleFrameSafelyAsync(frame: IDaemonFrame): Promise { try { await this.#onFrameAsync(frame); @@ -240,8 +247,15 @@ export class DaemonControlSession { 'Daemon shutdown requires a lifecycle-capable protocol version.' ); } - await this.#enqueueControlAsync({ kind: 'shutdownAck', payload: {} }); + const activeRequests: number | undefined = this.#options.getActiveRequestCount?.(); + // Queue the acknowledgement, then begin shutdown synchronously so the reported count is the set that + // shutdown aborts; closing drains the send queue, so the acknowledgement is still delivered first. + const ackPromise: Promise = this.#enqueueControlAsync({ + kind: 'shutdownAck', + payload: activeRequests === undefined ? {} : { activeRequests } + }); this.#options.onShutdownRequested(); + await ackPromise; } #startRequest(envelope: IDaemonRequestEnvelope): void { @@ -443,8 +457,8 @@ export class DaemonControlSession { } } - async #closeOnceAsync(drainRequests: boolean = false): Promise { - const closeReason: Error = new Error('The daemon control session is closing.'); + async #closeOnceAsync(drainRequests: boolean = false, reason?: DaemonShutdownError): Promise { + const closeReason: Error = reason ?? new Error('The daemon control session is closing.'); if (drainRequests) { const pending: Promise[]> = Promise.allSettled( Array.from(this.#requestById.values(), (state: IRequestState) => state.completion) diff --git a/libraries/rush-daemon/src/DaemonGraphRequestRouter.ts b/libraries/rush-daemon/src/DaemonGraphRequestRouter.ts index 567ac94026..10b8f818f3 100644 --- a/libraries/rush-daemon/src/DaemonGraphRequestRouter.ts +++ b/libraries/rush-daemon/src/DaemonGraphRequestRouter.ts @@ -37,6 +37,7 @@ import { import type { IWorkspaceSession } from './WorkspaceSession'; import { setPauseNextIteration } from './PhasedRequestRouter'; import { getWorkspaceGenerationToken } from './WorkspaceGeneration'; +import { type DaemonShutdownError, getDaemonShutdownReason } from './DaemonShutdownError'; const DAEMON_PACKAGE_VERSION: string = PackageJsonLookup.loadOwnPackageJson(__dirname).version; @@ -86,7 +87,12 @@ export class DaemonGraphRequestRouter { admissionErrorCode: getRequestAdmissionErrorCode(error) }; } - await client.writeResultAsync(result); + const shutdownReason: DaemonShutdownError | undefined = result.aborted + ? getDaemonShutdownReason(client.abortSignal) + : undefined; + await client.writeResultAsync( + shutdownReason ? { ...result, errorMessage: shutdownReason.message } : result + ); } private async _mutateAsync( diff --git a/libraries/rush-daemon/src/DaemonShutdownError.ts b/libraries/rush-daemon/src/DaemonShutdownError.ts new file mode 100644 index 0000000000..f50a30252f --- /dev/null +++ b/libraries/rush-daemon/src/DaemonShutdownError.ts @@ -0,0 +1,73 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * What initiated a daemon shutdown. + * + * @beta + */ +export type DaemonShutdownInitiator = 'controlClient' | 'signal' | 'idleTimeout' | 'restart' | 'host'; + +/** + * Options for {@link DaemonShutdownError}. + * + * @beta + */ +export interface IDaemonShutdownErrorOptions { + readonly initiator: DaemonShutdownInitiator; + /** The process signal name, when the initiator is `signal`. */ + readonly signal?: string; +} + +/** + * The typed reason used to abort requests that were still running when the daemon shut down. + * + * @remarks + * Its message is delivered to the affected clients as the request's error message. + * + * @beta + */ +export class DaemonShutdownError extends Error { + public readonly initiator: DaemonShutdownInitiator; + public readonly signal: string | undefined; + + public constructor(options: IDaemonShutdownErrorOptions) { + super( + `The Rush daemon was shut down (${describeInitiator(options)}) while this request was running; ` + + 're-run the command.' + ); + this.name = 'DaemonShutdownError'; + this.initiator = options.initiator; + this.signal = options.signal; + } +} + +function describeInitiator(options: IDaemonShutdownErrorOptions): string { + switch (options.initiator) { + case 'controlClient': + return 'requested by "rush-client daemon stop" or "daemon restart"'; + case 'signal': + return `the daemon process received ${options.signal ?? 'a termination signal'}`; + case 'idleTimeout': + return 'idle timeout'; + case 'restart': + return 'the daemon restarted to apply workspace changes'; + case 'host': + return 'the daemon host was closed'; + } +} + +/** Returns the shutdown reason if the signal was aborted because the daemon shut down. */ +export function getDaemonShutdownReason(signal: AbortSignal): DaemonShutdownError | undefined { + return signal.aborted && signal.reason instanceof DaemonShutdownError ? signal.reason : undefined; +} + +/** + * Returns the cleanup error unless it repeats a shutdown reason that is already the primary error, for example when + * restoring raw mode fails because shutdown closed the connection. + */ +export function withoutRepeatedShutdownReason(primary: unknown, cleanupError: unknown): unknown { + return primary instanceof DaemonShutdownError && cleanupError instanceof DaemonShutdownError + ? undefined + : cleanupError; +} diff --git a/libraries/rush-daemon/src/EngineTerminalProvider.ts b/libraries/rush-daemon/src/EngineTerminalProvider.ts index 13c8b48d5e..88345d0372 100644 --- a/libraries/rush-daemon/src/EngineTerminalProvider.ts +++ b/libraries/rush-daemon/src/EngineTerminalProvider.ts @@ -4,25 +4,69 @@ import type { IOperationGraph, _IOperationGraphEventSink } from '@microsoft/rush-lib'; import { TerminalProviderSeverity, type ITerminalProvider } from '@rushstack/terminal'; +import { WorkspaceEngineRecreationRequiredError } from './WorkspaceEngineComponentFactory'; + export class EngineTerminalProvider implements ITerminalProvider { public readonly supportsColor: boolean = false; public readonly eolCharacter: string = '\n'; readonly #messages: Array<{ text: string; severity: TerminalProviderSeverity }> = []; #graph: (IOperationGraph & { eventSink?: _IOperationGraphEventSink }) | undefined; #executing: boolean = false; + #hasReconciled: boolean = false; public write(text: string, severity: TerminalProviderSeverity): void { if (this.#executing) this.#emit(text, severity); else this.#messages.push({ text, severity }); } + /** + * Drains buffered diagnostics into the failure description, so that they belong to the failing request + * and are never replayed into a later request. + */ public describeError(error: unknown): string { return [ - ...this.#messages.map(({ text }) => text), + ...this.#messages.splice(0).map(({ text }) => text), error instanceof Error ? error.message : String(error) ].join('\n'); } + public get hasBufferedMessages(): boolean { + return this.#messages.length > 0; + } + + /** Discards diagnostics buffered by an earlier request before a new request starts using this terminal. */ + public discardBufferedMessages(): void { + this.#messages.length = 0; + } + + /** + * Runs a warm reconcile with request-scoped diagnostics. The graph keeps the binding request's terminal, so + * diagnostics buffered before a later request's reconcile belong to an earlier request and are discarded; the + * binding request's own diagnostics are kept. A failure carries the diagnostics buffered while reconciling. + */ + public async reconcileWithRequestDiagnosticsAsync(reconcileAsync: () => Promise): Promise { + if (this.#hasReconciled) this.discardBufferedMessages(); + this.#hasReconciled = true; + try { + return await reconcileAsync(); + } catch (error) { + throw this.#attachBufferedDiagnostics(error); + } + } + + #attachBufferedDiagnostics(error: unknown): unknown { + if (error instanceof WorkspaceEngineRecreationRequiredError) { + // The replacement engine gets a fresh terminal; the stale diagnostics must not reach a later request. + this.discardBufferedMessages(); + return error; + } + if (!this.hasBufferedMessages) return error; + if (!(error instanceof Error)) return new Error(this.describeError(error), { cause: error }); + // Keep the error's identity and type, which callers use for classification. + error.message = this.describeError(error); + return error; + } + public attach(graph: IOperationGraph): void { if (!('eventSink' in graph)) throw new Error('The native graph does not expose its operation event sink.'); diff --git a/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts b/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts index 256564c846..8ce2d4a61f 100644 --- a/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts +++ b/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts @@ -24,6 +24,7 @@ import { import { type IRequestLease, RequestSchedulerError, RequestSchedulerErrorCode } from './RequestScheduler'; import type { IWorkspaceSession } from './WorkspaceSession'; import { assertWorkspaceRequestResourcesHealthy } from './WorkspaceRequestResources'; +import { getDaemonShutdownReason, withoutRepeatedShutdownReason } from './DaemonShutdownError'; /** * Executes caller-resolved global command logic. @@ -179,7 +180,12 @@ async function executeAdmittedAsync( cleanupError = combineExecutionAndCleanupErrors(cleanupError, error); } aborted ||= context.requestAborted; - const combinedError: unknown = combineExecutionAndCleanupErrors(executionError, cleanupError); + const primaryError: unknown = + executionError ?? (aborted ? getDaemonShutdownReason(client.abortSignal) : undefined); + const combinedError: unknown = combineExecutionAndCleanupErrors( + primaryError, + withoutRepeatedShutdownReason(primaryError, cleanupError) + ); let result: IDaemonCommandResult; try { result = createGlobalCommandResult({ @@ -259,8 +265,12 @@ async function finishAfterAdmissionErrorAsync( throw combineExecutionAndCleanupErrors(admissionError, cleanupError); } const aborted: boolean = admissionError.code === RequestSchedulerErrorCode.Aborted; + const shutdownReason: unknown = aborted ? getDaemonShutdownReason(client.abortSignal) : undefined; const error: unknown = aborted - ? cleanupError + ? combineExecutionAndCleanupErrors( + shutdownReason, + withoutRepeatedShutdownReason(shutdownReason, cleanupError) + ) : combineExecutionAndCleanupErrors(admissionError, cleanupError); const result: IDaemonCommandResult = { ...createGlobalCommandResult({ diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index f9cd596123..430751653c 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -22,6 +22,7 @@ import { PhasedRequestEventSink } from './PhasedRequestEventSink'; import { PhasedRequestEventMultiplexer } from './PhasedRequestEventMultiplexer'; import type { IPhasedRequestClient } from './PhasedRequestClient'; import { DaemonRequiresInProcessError, evaluateDaemonTerminalPolicy } from './DaemonTerminalPolicy'; +import { DaemonShutdownError, getDaemonShutdownReason } from './DaemonShutdownError'; import type { IInteractiveRequestSession } from './InteractiveRequestInputRouter'; import { classifyRushCommand } from './RushCommandRequestPolicy'; import { @@ -641,7 +642,10 @@ class PhasedRequestBatchCoordinator { : []; const result: IDaemonPhasedRequestResult = createPhasedCommandResult({ aborted, - error: combineErrors(executionError, cleanupErrors), + error: combineErrors( + executionError ?? getDaemonShutdownReason(entry.client.abortSignal), + cleanupErrors + ), graphStatus: getClientGraphStatus(aborted, operationOutcomes), operationOutcomes, requestId: entry.request.requestId, @@ -1029,7 +1033,7 @@ async function writeAbortedResultAsync( const result: IDaemonPhasedRequestResult = { ...createPhasedCommandResult({ aborted: true, - error: combineErrors(undefined, cleanupErrors), + error: combineErrors(getDaemonShutdownReason(client.abortSignal), cleanupErrors), graphStatus: OperationStatus.Aborted, operationOutcomes: [], requestId, @@ -1131,7 +1135,13 @@ async function finishAfterAdmissionErrorAsync( return result; } -function combineErrors(executionError: unknown, cleanupErrors: unknown[]): unknown { +function combineErrors(executionError: unknown, allCleanupErrors: unknown[]): unknown { + // Cleanup that fails with the same daemon shutdown reason (for example, restoring raw mode after the + // interactive connection closed) must not hide that reason from the client. + const cleanupErrors: unknown[] = + executionError instanceof DaemonShutdownError + ? allCleanupErrors.filter((error: unknown) => !(error instanceof DaemonShutdownError)) + : allCleanupErrors; if (executionError !== undefined && cleanupErrors.length > 0) { return new AggregateError( [executionError, ...cleanupErrors], diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 6c503f46ee..6467828ef5 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -32,6 +32,7 @@ import { } from './WorkspaceEngineComponentFactory'; import type { IWorkspaceSession, IWorkspaceSessionComponents } from './WorkspaceSession'; import { EngineTerminalProvider } from './EngineTerminalProvider'; +import { getDaemonShutdownReason } from './DaemonShutdownError'; import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; /** @@ -174,7 +175,8 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { if (abortSignal.aborted) throw new DaemonRequestDispatchError( 'routingFailed', - 'The request was cancelled before engine initialization.' + getDaemonShutdownReason(abortSignal)?.message ?? + 'The request was cancelled before engine initialization.' ); return command; } @@ -224,7 +226,9 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { ...components, reconcileInvalidationsAsync: async () => { const result: IWorkspaceInvalidationReconciliation = - await components.reconcileInvalidationsAsync!(); + await terminal.reconcileWithRequestDiagnosticsAsync(() => + components.reconcileInvalidationsAsync!() + ); if (!engine.isIncremental) engine.operationGraph.invalidateOperations(undefined, 'rebuild'); return result; } diff --git a/libraries/rush-daemon/src/RushDaemonHost.ts b/libraries/rush-daemon/src/RushDaemonHost.ts index 6aa4989ac3..17ff4a7bed 100644 --- a/libraries/rush-daemon/src/RushDaemonHost.ts +++ b/libraries/rush-daemon/src/RushDaemonHost.ts @@ -18,6 +18,7 @@ import { DaemonIdleTimer } from './DaemonIdleTimer'; import type { IDaemonInteractiveConnection } from './DaemonInteractiveConnection'; import { DaemonRequestDispatcher } from './DaemonRequestDispatcher'; import type { IDaemonRequestResolver } from './DaemonRequestDispatcher'; +import { DaemonShutdownError, type DaemonShutdownInitiator } from './DaemonShutdownError'; import { WorkspaceSession } from './WorkspaceSession'; import type { IWorkspaceSession, WorkspaceSessionFactory } from './WorkspaceSession'; import { WorkspaceSessionProvider } from './WorkspaceSessionProvider'; @@ -180,7 +181,12 @@ export class RushDaemonHost { }, onError: (error: Error) => options.onError?.(error), onRequestStarted: () => idleTimer.acquire(), - onShutdownRequested: requestShutdown + onShutdownRequested: () => requestShutdown('controlClient'), + getActiveRequestCount: () => { + let count: number = 0; + for (const activeSession of sessions) count += activeSession.activeRequestCount; + return count; + } }); sessions.add(session); if (lifecycle.closing) { @@ -223,13 +229,13 @@ export class RushDaemonHost { function requestRestart(plan: IWorkspaceProcessRestartPlan): void { host.#requestRestart(plan); } - function requestShutdown(): void { - void host.closeAsync().catch((error: Error) => { + function requestShutdown(initiator: DaemonShutdownInitiator): void { + void host.closeAsync(new DaemonShutdownError({ initiator })).catch((error: Error) => { if (options.onError) options.onError(error); else process.emitWarning(error); }); } - idleTimer.start(requestShutdown); + idleTimer.start(() => requestShutdown('idleTimeout')); return host; } @@ -248,9 +254,13 @@ export class RushDaemonHost { return this.#readWorkspaceStatus(); } - /** Closes active connections, stops listening, and removes transport artifacts. */ - public closeAsync(): Promise { - this.#closePromise ??= this.#closeOnceAsync().finally(() => { + /** + * Closes active connections, stops listening, and removes transport artifacts. + * + * @param reason - Delivered to requests that are still running; only the first close call's reason is used. + */ + public closeAsync(reason?: DaemonShutdownError): Promise { + this.#closePromise ??= this.#closeOnceAsync(reason).finally(() => { this.#notifyClosed?.(); if (!this.#restartPromise) this.#resolveRestart?.(undefined); }); @@ -272,7 +282,7 @@ export class RushDaemonHost { } async #restartOnceAsync(plan: IWorkspaceProcessRestartPlan): Promise { - await this.closeAsync(); + await this.closeAsync(new DaemonShutdownError({ initiator: 'restart' })); if (plan.failure) throw plan.failure; if (!plan.launch) throw new Error('A successor was not selected.'); const paths: IDaemonPaths = resolveDaemonPathsFromProcess( @@ -297,7 +307,7 @@ export class RushDaemonHost { } } - async #closeOnceAsync(): Promise { + async #closeOnceAsync(reason: DaemonShutdownError | undefined): Promise { this.#idleTimer[Symbol.dispose](); this.#lifecycle.closing = true; const errors: unknown[] = []; @@ -305,7 +315,7 @@ export class RushDaemonHost { // A failed standalone host must not exit naturally and become reclaimable over unjoined children. const sessionSettlements: PromiseSettledResult[] = await Promise.allSettled( Array.from(this.#sessions, (session: DaemonControlSession) => - session.closeAsync(!!this.#restartPromise) + session.closeAsync(!!this.#restartPromise, reason ?? new DaemonShutdownError({ initiator: 'host' })) ) ); for (const settlement of sessionSettlements) { diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index adb9753ce6..2ee815aad2 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -40,6 +40,7 @@ import { getWorkspaceRequestScheduler } from './WorkspaceRequestAdmission'; import { WorkspaceEngineRecreationRequiredError } from './WorkspaceEngineComponentFactory'; +import { getDaemonShutdownReason } from './DaemonShutdownError'; import type { IWorkspaceSession } from './WorkspaceSession'; import type { WorkspaceSessionProvider } from './WorkspaceSessionProvider'; import { assertWorkspaceRequestResourcesHealthy } from './WorkspaceRequestResources'; @@ -263,7 +264,10 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { if (error instanceof RequestSchedulerError && !state.began && !state.terminalAttempted) { await client.interactiveSession.finishAsync(); await client.writeResultAsync({ - ...preExecutionFailure(envelope.requestId, error), + ...preExecutionFailure( + envelope.requestId, + getDaemonShutdownReason(client.abortSignal) ?? error + ), aborted: client.abortSignal.aborted, admissionErrorCode: getRequestAdmissionErrorCode(error) }); diff --git a/libraries/rush-daemon/src/index.ts b/libraries/rush-daemon/src/index.ts index 89039a0c43..b895226872 100644 --- a/libraries/rush-daemon/src/index.ts +++ b/libraries/rush-daemon/src/index.ts @@ -57,6 +57,11 @@ export { type IGlobalCommandRequestResult } from './GlobalCommandRequestRouter'; export { RushDaemonHost, type IRushDaemonHostOptions } from './RushDaemonHost'; +export { + DaemonShutdownError, + type DaemonShutdownInitiator, + type IDaemonShutdownErrorOptions +} from './DaemonShutdownError'; export { serveRushDaemonAsync, type IRushDaemonServeOptions } from './serveRushDaemon'; export { WorkspaceEngineComponentFactory, diff --git a/libraries/rush-daemon/src/serveRushDaemon.ts b/libraries/rush-daemon/src/serveRushDaemon.ts index 929b28bbf0..1baec925df 100644 --- a/libraries/rush-daemon/src/serveRushDaemon.ts +++ b/libraries/rush-daemon/src/serveRushDaemon.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import { DaemonShutdownError } from './DaemonShutdownError'; import { RushDaemonHost } from './RushDaemonHost'; import type { IRushDaemonHostOptions } from './RushDaemonHost'; import { getInstalledWorkspaceSuccessorLaunchAsync } from './WorkspaceProcessRestart'; @@ -41,7 +42,7 @@ export async function serveRushDaemonAsync(options: IRushDaemonServeOptions): Pr }); await options.onReady?.(host); await waitForShutdownAsync(host, signalRegistration.signal); - await host.closeAsync(); + await host.closeAsync(getShutdownReason(signalRegistration.signal)); await host.restartCompleted; } finally { signalRegistration.dispose(); @@ -49,6 +50,13 @@ export async function serveRushDaemonAsync(options: IRushDaemonServeOptions): Pr } } +function getShutdownReason(signal: AbortSignal): DaemonShutdownError | undefined { + if (!signal.aborted) return undefined; + return signal.reason instanceof DaemonShutdownError + ? signal.reason + : new DaemonShutdownError({ initiator: 'host' }); +} + interface IShutdownSignalRegistration { readonly signal: AbortSignal; readonly dispose: () => void; @@ -56,7 +64,8 @@ interface IShutdownSignalRegistration { function createProcessShutdownSignal(): IShutdownSignalRegistration { const controller: AbortController = new AbortController(); - const onSignal: () => void = () => controller.abort(); + const onSignal: (signal: NodeJS.Signals) => void = (signal: NodeJS.Signals) => + controller.abort(new DaemonShutdownError({ initiator: 'signal', signal })); process.once('SIGINT', onSignal); process.once('SIGTERM', onSignal); return { diff --git a/libraries/rush-daemon/src/test/DaemonRequestWireGlobal.test.ts b/libraries/rush-daemon/src/test/DaemonRequestWireGlobal.test.ts index 86d984c61b..e570988009 100644 --- a/libraries/rush-daemon/src/test/DaemonRequestWireGlobal.test.ts +++ b/libraries/rush-daemon/src/test/DaemonRequestWireGlobal.test.ts @@ -10,6 +10,7 @@ import type { DaemonControlMessage, IDaemonRequestEnvelope } from '@rushstack/ru import type { GlobalCommandExecutor, IDaemonRequestResolver } from '../index'; import { MAX_REQUESTS_PER_CONNECTION } from '../DaemonConnectionLimits'; +import { DaemonShutdownError } from '../DaemonShutdownError'; import { RushDaemonHost } from '../RushDaemonHost'; import type { IRushDaemonHostOptions } from '../RushDaemonHost'; import { TestWorkspaceSession } from './TestWorkspaceSession'; @@ -303,6 +304,51 @@ describe('daemon global request wire integration', () => { } }); + it('tells a request queued for admission that the daemon shut down', async () => { + const repoRoot: string = createRepoRoot(); + const holderStarted: IDeferred = createDeferred(); + const releaseHolder: IDeferred = createDeferred(); + const resolver: IDaemonRequestResolver = new CallbackDaemonRequestResolver(async ({ envelope }) => { + const executorAsync: GlobalCommandExecutor = async () => { + if (envelope.requestId === 'holder') { + holderStarted.resolve(); + await releaseHolder.promise; + } + return { exitCode: 0 }; + }; + return { executor: executorAsync, kind: 'global' }; + }); + const host: RushDaemonHost = await RushDaemonHost.startAsync(createHostOptions(repoRoot, resolver)); + const clients: DaemonRequestWireClient[] = await Promise.all([connectAsync(host), connectAsync(host)]); + const shutdown: DaemonShutdownError = new DaemonShutdownError({ initiator: 'controlClient' }); + try { + await clients[0].sendControlAsync({ + kind: 'requestStart', + payload: createWireEnvelope('holder', 'custom', repoRoot) + }); + await holderStarted.promise; + await clients[1].sendControlAsync({ + kind: 'requestStart', + payload: createWireEnvelope('queued', 'custom', repoRoot) + }); + expect(await clients[1].readControlAsync()).toMatchObject({ + kind: 'queuePosition', + payload: { requestId: 'queued' } + }); + const closePromise: Promise = host.closeAsync(shutdown); + releaseHolder.resolve(); + expect((await clients[1].readTerminalAsync('queued')).terminal).toMatchObject({ + kind: 'requestResult', + payload: { aborted: true, admissionErrorCode: 'aborted', errorMessage: shutdown.message } + }); + await closePromise; + } finally { + releaseHolder.resolve(); + await Promise.all(clients.map((client: DaemonRequestWireClient) => client.closeAsync())); + await host.closeAsync(); + } + }); + it('rejects a second active request on one connection without cancelling the first', async () => { const repoRoot: string = createRepoRoot(); const started: IDeferred = createDeferred(); diff --git a/libraries/rush-daemon/src/test/DaemonShutdown.test.ts b/libraries/rush-daemon/src/test/DaemonShutdown.test.ts index 284d26a697..c2c8c91e3d 100644 --- a/libraries/rush-daemon/src/test/DaemonShutdown.test.ts +++ b/libraries/rush-daemon/src/test/DaemonShutdown.test.ts @@ -38,7 +38,7 @@ describe('daemon management shutdown', () => { await client.sendControlAsync(createDaemonHello(DAEMON_PROTOCOL_VERSION)); expect((await client.readControlAsync()).kind).toBe('helloAck'); await client.sendControlAsync({ kind: 'shutdown', payload: {} }); - expect(await client.readControlAsync()).toEqual({ kind: 'shutdownAck', payload: {} }); + expect(await client.readControlAsync()).toEqual({ kind: 'shutdownAck', payload: { activeRequests: 0 } }); await client.closed; await host.closed; expect(readDaemonLockfile(host.paths.lockfilePath)).toBeUndefined(); diff --git a/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts b/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts new file mode 100644 index 0000000000..2c6fcbfdf7 --- /dev/null +++ b/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts @@ -0,0 +1,57 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { TerminalProviderSeverity } from '@rushstack/terminal'; + +import { EngineTerminalProvider } from '../EngineTerminalProvider'; +import { WorkspaceEngineRecreationRequiredError } from '../WorkspaceEngineComponentFactory'; + +describe(EngineTerminalProvider.name, () => { + it('drains buffered diagnostics into the failure description so a later request cannot replay them', () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + terminal.write('Permission denied', TerminalProviderSeverity.error); + expect(terminal.hasBufferedMessages).toBe(true); + expect(terminal.describeError(new Error('snapshot failed'))).toBe('Permission denied\nsnapshot failed'); + expect(terminal.hasBufferedMessages).toBe(false); + expect(terminal.describeError(new Error('next request'))).toBe('next request'); + }); + + it('discards diagnostics buffered by an earlier request', () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + terminal.write('stale', TerminalProviderSeverity.warning); + terminal.discardBufferedMessages(); + expect(terminal.hasBufferedMessages).toBe(false); + expect(terminal.describeError('failure')).toBe('failure'); + }); + + it('scopes reconcile diagnostics to the request whose reconcile produced them', async () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + terminal.write('binding request diagnostic', TerminalProviderSeverity.warning); + const failure: RangeError = new RangeError('could not capture'); + await expect( + terminal.reconcileWithRequestDiagnosticsAsync(async () => { + terminal.write('Permission denied', TerminalProviderSeverity.error); + throw failure; + }) + ).rejects.toBe(failure); + expect(failure.message).toBe('binding request diagnostic\nPermission denied\ncould not capture'); + + terminal.write('stale', TerminalProviderSeverity.warning); + await expect(terminal.reconcileWithRequestDiagnosticsAsync(async () => 'ok')).resolves.toBe('ok'); + expect(terminal.hasBufferedMessages).toBe(false); + }); + + it('drops diagnostics when the engine must be recreated', async () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + const recreate: WorkspaceEngineRecreationRequiredError = new WorkspaceEngineRecreationRequiredError(); + const message: string = recreate.message; + await expect( + terminal.reconcileWithRequestDiagnosticsAsync(async () => { + terminal.write('stale', TerminalProviderSeverity.error); + throw recreate; + }) + ).rejects.toBe(recreate); + expect(recreate.message).toBe(message); + expect(terminal.hasBufferedMessages).toBe(false); + }); +}); diff --git a/libraries/rush-daemon/src/test/PhasedRequestInteractive.test.ts b/libraries/rush-daemon/src/test/PhasedRequestInteractive.test.ts index 7cc2ab8b51..6c915f8b5c 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestInteractive.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestInteractive.test.ts @@ -3,9 +3,11 @@ import type { IDaemonPhasedRequest, + IDaemonPhasedRequestResult, IDaemonSetRawModeMessage } from '@rushstack/rush-daemon-protocol'; +import { DaemonShutdownError } from '../DaemonShutdownError'; import { DaemonRequiresInProcessError } from '../DaemonTerminalPolicy'; import { InteractiveRequestInputRouter } from '../InteractiveRequestInputRouter'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; @@ -68,6 +70,34 @@ it('restores phased-request raw mode before publishing the command result', asyn expect(lifecycleOrder).toEqual(['raw:true', 'raw:false', 'result']); }); +it('keeps the daemon shutdown reason when restoring raw mode fails with it', async () => { + const fixture: ITestRoutingFixture = createFixture(); + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + const shutdown: DaemonShutdownError = new DaemonShutdownError({ initiator: 'signal', signal: 'SIGTERM' }); + client.interactiveSession = new InteractiveRequestInputRouter().register({ + acceptsStdin: true, + client: { + abortSignal: client.abortSignal, + writeRawModeControlAsync: (message: IDaemonSetRawModeMessage): Promise => + message.payload.enabled ? Promise.resolve() : Promise.reject(shutdown) + }, + onFailure: (error: Error) => client.abortController.abort(error), + requestId: 'interactive-request' + }); + client.interactiveInputSink = { + writeInputAsync: (): Promise => Promise.resolve() + }; + await client.interactiveSession.setRawModeAsync(true); + client.abortController.abort(shutdown); + + await new PhasedRequestRouter(fixture.session) + .executeAsync(createRequest({ acceptsStdin: true, terminalRequirement: 'interactiveInput' }), client) + .catch(() => undefined); + + const result: IDaemonPhasedRequestResult | undefined = client.writes.find((write) => write.result)?.result; + expect(result).toMatchObject({ aborted: true, errorMessage: shutdown.message }); +}); + it('signals requiresInProcess without scheduling a PTY-only phased request', async () => { const fixture: ITestRoutingFixture = createFixture(); const client: TestPhasedRequestClient = new TestPhasedRequestClient(); diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index b0d02fd298..01f38bfd80 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -34,6 +34,7 @@ import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; import { removeTestFolderAsync } from './TestProcessExit'; import { readDaemonLockfile } from '@rushstack/rush-daemon-transport'; import { EngineTerminalProvider } from '../EngineTerminalProvider'; +import { DaemonShutdownError } from '../DaemonShutdownError'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; import type { GetWorkspaceSuccessorLaunchAsync, @@ -1436,4 +1437,67 @@ process.exit(23); await fixture[Symbol.asyncDispose](); } }); + + const canRevokeReadAccess: boolean = process.platform !== 'win32' && process.getuid?.() !== 0; + (canRevokeReadAccess ? it : it.skip)( + 'reports a warm snapshot failure to the failing request and never replays it into the next request', + async () => { + const fixture: IFixture = await createFixtureAsync(); + const inputPath: string = path.join(fixture.repoRoot, 'projects/a/input.txt'); + try { + await runAsync(fixture, 'initial', ['build', '--only', 'a']); + fs.writeFileSync(inputPath, 'unreadable'); + fs.chmodSync(inputPath, 0); + const failed: ITerminalExchange = await runAsync(fixture, 'unreadable', ['build', '--only', 'a']); + expect(failed.terminal).toMatchObject({ + kind: 'requestRejected', + payload: { + message: expect.stringMatching( + /Permission denied[\s\S]*Rush could not capture the next workspace inputs snapshot\./ + ) + } + }); + fs.chmodSync(inputPath, 0o644); + const recovered: ITerminalExchange = await runAsync(fixture, 'recovered', ['build', '--only', 'a']); + expect(recovered.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + const output: string = [ + logText(recovered), + ...recovered.frames + .filter((frame) => frame.kind === DaemonFrameType.event) + .map((frame) => JSON.stringify(decodeDaemonEventFrame(frame.payload))) + ].join('\n'); + expect(output).not.toContain('Permission denied'); + expect(output).not.toContain('state of the repo'); + } finally { + if (fs.existsSync(inputPath)) fs.chmodSync(inputPath, 0o644); + await fixture[Symbol.asyncDispose](); + } + } + ); + + it('aborts an in-flight build with the typed daemon shutdown reason', async () => { + const fixture: IFixture = await createFixtureAsync(); + const gate: INativeScriptGate = await createNativeScriptGateAsync(fixture.repoRoot, 'a'); + try { + const victim: Promise = runAsync(fixture, 'victim', ['build', '--only', 'a']); + await gate.entered; + const closing: Promise = fixture.host.closeAsync( + new DaemonShutdownError({ initiator: 'signal', signal: 'SIGTERM' }) + ); + await gate.releaseAsync(); + expect((await victim).terminal).toMatchObject({ + kind: 'requestResult', + payload: { + aborted: true, + errorMessage: expect.stringMatching( + /^The Rush daemon was shut down \(the daemon process received SIGTERM\) while this request was running; re-run the command\.$/ + ) + } + }); + await closing; + } finally { + await gate.releaseAsync(); + await fixture[Symbol.asyncDispose](); + } + }); }); From 23cb48819646c2672cc0e24bd9d19274a873da46 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:17:36 -0700 Subject: [PATCH 007/265] [rush-cli-client] Add agent output mode and honor useRushReporter on the daemon path (#6082) * [rush-cli-client] Add agent output mode and honor useRushReporter on the daemon path Fixes #6076 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-cli-client] Keep explicit reporter controls on the native path; agent mode only when no reporter is chosen Addresses review: --reporter=ai (with or without --no-daemon, and on fallback) and RUSH_REPORTER=ai keep the native AI reporter JSON; agent mode writes nothing ahead of native reporter output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-cli-client] Address review: ABORTED counts, per-operation failure tails, final line on rejection, admission-control command name, normalized RUSH_REPORTER, queue dedup Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- apps/rush-cli-client/README.md | 34 ++- .../src/AgentProgressRenderer.ts | 288 ++++++++++++++++++ apps/rush-cli-client/src/launchClient.ts | 33 +- apps/rush-cli-client/src/outputSelection.ts | 143 +++++++++ apps/rush-cli-client/src/routing.ts | 8 +- apps/rush-cli-client/src/start.ts | 35 ++- .../src/test/AgentProgressRenderer.test.ts | 199 ++++++++++++ .../src/test/outputSelection.test.ts | 92 ++++++ apps/rush-cli-client/src/test/routing.test.ts | 22 +- .../agent-reporter_2026-09-24-02-35.json | 11 + 10 files changed, 852 insertions(+), 13 deletions(-) create mode 100644 apps/rush-cli-client/src/AgentProgressRenderer.ts create mode 100644 apps/rush-cli-client/src/outputSelection.ts create mode 100644 apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts create mode 100644 apps/rush-cli-client/src/test/outputSelection.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/agent-reporter_2026-09-24-02-35.json diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index bb93cf69f5..dda32fb6ef 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -61,9 +61,37 @@ Admission controls also apply to experimental graph requests, but not retains native command behavior. Waiting positions are shown on interactive stderr, and admission failures report their typed reason and a nonzero exit code. -Explicit reporter/output/log-level controls retain the native frontend reporter path. -The current daemon client renders the legacy operation stream; it does not silently -reinterpret requests for JSON, AI, file, or other reporter formats. +Explicit reporter/output/log-level controls (`--reporter`, `--output`, `--log-level`, +`RUSH_REPORTER` other than `legacy`, or `RUSH_LOG_LEVEL`) retain the native frontend +reporter path, with or without `--no-daemon`, including `--reporter=ai`. The daemon client +does not silently reinterpret requests for JSON, AI, file, or other reporter formats. + +A repository that opts into the native reporter with `"useRushReporter": true` in +`common/config/rush/experiments.json` also stays on the native (in-process) path, so +that its reporter output is honored rather than silently replaced by the daemon +stream. Native reporter rendering over the daemon protocol is a follow-up. + +### Output modes + +The `rush-client` daemon path has two output modes (`rushx-client` always uses `legacy`). +Requests that use the native reporter path (see above) always get native output, and +agent mode writes nothing ahead of it. Otherwise, selection precedence is: + +1. `RUSHD_OUTPUT=agent` or `RUSHD_OUTPUT=legacy`. +2. An active `COPILOT_CLI` agent marker selects `agent`, matching `detectAgent()` in + `@rushstack/reporter` (a value is inactive when empty, `0`, `false`, `no` or `off`). + Other agents can opt in with `RUSHD_OUTPUT=agent`. +3. Otherwise `legacy`: the unchanged collated operation stream. + +Agent mode is plain text for humans and agents, not the AI reporter's JSON record format; +use `--reporter=ai` for machine-parsed records. It writes a first status line before +`@microsoft/rush-lib` is loaded, then at most three live rows on a TTY (append-only lines +throttled to one per 2 seconds on a pipe), the queue position when waiting for admission, +and always one final summary line (`rush build: SUCCESS 12/12 operations (...) in 3.1s`, or +`up to date (no operations needed)`). On failure, it lists failed operations and a +bounded tail (10 lines) of their stderr, or of their stdout when they wrote no stderr. +Operation logs are otherwise not printed; use `RUSHD_OUTPUT=legacy` for full logs. When +a request falls back to in-process Rush, agent mode stops and native output follows. Positively identified built-in `install` and `update` follow the same opt-in routing precedence as workspace builds and require protocol **0.10** diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts new file mode 100644 index 0000000000..08fa108796 --- /dev/null +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -0,0 +1,288 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// Keep this module free of heavy imports: start.ts loads it before @microsoft/rush-lib +// so that the first line can be written within a few milliseconds. + +import type { IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; + +const SPINNER_FRAMES: readonly string[] = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']; +const TERMINAL_STATUSES: ReadonlySet = new Set([ + 'SUCCESS', + 'SUCCESS WITH WARNINGS', + 'SKIPPED', + 'FROM CACHE', + 'FAILURE', + 'BLOCKED', + 'NO OP', + 'ABORTED' +]); +const MAX_ERROR_LINES: number = 10; +const PIPE_MIN_INTERVAL_MS: number = 2000; +const PIPE_HEARTBEAT_MS: number = 10000; +const TTY_INTERVAL_MS: number = 100; + +export interface IAgentProgressRendererOptions { + readonly commandName: string; + readonly isTTY: boolean; + readonly columns: number; + readonly write: (text: string) => void; + readonly now?: () => number; + readonly startTimeMs?: number; +} + +interface IAgentFinalResult { + readonly exitCode: number; + readonly errorMessage?: string; +} + +/** + * Compact progress for agents on the daemon path: an immediate first line, at most + * three live rows (TTY) or throttled append-only lines (pipes), and a guaranteed + * bounded final summary line, even when no operation ran. + */ +export class AgentProgressRenderer { + readonly #options: IAgentProgressRendererOptions; + readonly #now: () => number; + readonly #startTimeMs: number; + readonly #registered: Set = new Set(); + readonly #statuses: Map = new Map(); + readonly #running: Set = new Set(); + readonly #counts: Map = new Map(); + readonly #failed: string[] = []; + readonly #stderrTails: Map = new Map(); + readonly #stdoutTails: Map = new Map(); + #total: number = 0; + #done: number = 0; + #lastActivity: string = ''; + #phase: string = 'connecting to rushd (auto-starts if needed)'; + #painted: number = 0; + #frame: number = 0; + #lastLineKey: string = ''; + #lastLineAtMs: number = -Infinity; + #timer: ReturnType | undefined; + #stopped: boolean = false; + + public constructor(options: IAgentProgressRendererOptions) { + this.#options = options; + this.#now = options.now ?? Date.now; + this.#startTimeMs = options.startTimeMs ?? this.#now(); + } + + /** Writes the first line and starts the spinner / heartbeat. */ + public start(): void { + this.#render(true); + this.#timer = setInterval( + () => this.#render(false), + this.#options.isTTY ? TTY_INTERVAL_MS : PIPE_MIN_INTERVAL_MS + ); + this.#timer.unref?.(); + } + + public setPhase(phase: string): void { + if (phase === this.#phase) { + return; + } + this.#phase = phase; + this.#render(true); + } + + public onQueuePosition(position: number): void { + this.setPhase(`queued behind another request (position ${position})`); + } + + public onEvent(event: IDaemonEventEnvelope): void { + const payload: Record = (event.payload ?? {}) as Record; + switch (event.type) { + case 'operationRegistered': { + if (!payload.silent && typeof payload.operationId === 'string') { + this.#registered.add(payload.operationId); + } + break; + } + case 'operationStatusChanged': { + const operationId: unknown = payload.operationId; + const status: unknown = payload.status; + if (typeof operationId !== 'string' || typeof status !== 'string') { + break; + } + this.#phase = 'running'; + const previous: string | undefined = this.#statuses.get(operationId); + this.#statuses.set(operationId, status); + if (status === 'EXECUTING') { + this.#running.add(operationId); + } + if (TERMINAL_STATUSES.has(status) && (previous === undefined || !TERMINAL_STATUSES.has(previous))) { + this.#running.delete(operationId); + this.#done++; + this.#counts.set(status, (this.#counts.get(status) ?? 0) + 1); + if (status === 'FAILURE') { + this.#failed.push(operationId); + } else { + this.#stdoutTails.delete(operationId); + this.#stderrTails.delete(operationId); + } + } + break; + } + case 'extension': { + const data: { totalOperations?: unknown } | undefined = payload.data as + | { totalOperations?: unknown } + | undefined; + if (data && typeof data.totalOperations === 'number') { + this.#total = Math.max(this.#total, data.totalOperations); + } + break; + } + case 'activityChanged': { + if (typeof payload.text === 'string' && payload.text.trim()) { + this.#lastActivity = payload.text.trim().split('\n')[0]; + if (this.#phase !== 'running') { + this.#phase = 'running'; + } + } + break; + } + } + this.#render(false); + } + + /** + * Keeps bounded per-operation stderr and stdout tails (the last lines of each). They are only + * printed for failed operations; stdout is used when an operation reported its diagnostics + * there (tsc, eslint, jest) and wrote nothing to stderr. + */ + public onLog(bytes: Uint8Array, operationId: string, stream: 'stdout' | 'stderr'): void { + const status: string | undefined = this.#statuses.get(operationId); + if (status !== undefined && TERMINAL_STATUSES.has(status) && status !== 'FAILURE') { + return; + } + const tails: Map = stream === 'stderr' ? this.#stderrTails : this.#stdoutTails; + for (const line of Buffer.from(bytes).toString('utf8').split('\n')) { + if (!line.trim()) { + continue; + } + let tail: string[] | undefined = tails.get(operationId); + if (!tail) { + tail = []; + tails.set(operationId, tail); + } + tail.push(line.trim()); + if (tail.length > MAX_ERROR_LINES) { + tail.shift(); + } + } + } + + /** Stops rendering without a summary (e.g. the request is handed to in-process Rush). */ + public dispose(): void { + this.#stop(); + } + + /** Stops the live region and writes the final summary line, at most once. */ + public finish(result: IAgentFinalResult | undefined): void { + if (!this.#stop()) { + return; + } + const succeeded: boolean = result !== undefined && result.exitCode === 0; + const total: number = this.#getTotal(); + const parts: string[] = [...this.#counts].map(([status, count]) => `${count} ${status.toLowerCase()}`); + const scope: string = + total === 0 && succeeded + ? 'up to date (no operations needed)' + : `${this.#done}/${total} operations${parts.length ? ` (${parts.join(', ')})` : ''}`; + let line: string = `rush ${this.#options.commandName}: ${succeeded ? 'SUCCESS' : 'FAILURE'} ${scope} in ${this.#elapsed()}`; + if (this.#failed.length) { + line += ` · failed: ${this.#failed.join(', ')}`; + } + if (result?.errorMessage) { + line += ` · ${result.errorMessage}`; + } + this.#options.write(`${line}\n`); + if (!succeeded) { + for (const errorLine of this.#getFailureLines().slice(0, MAX_ERROR_LINES)) { + this.#options.write(` ${errorLine}\n`); + } + } + } + + /** Failed operations' tails, or every operation's stderr tail when no operation failed. */ + #getFailureLines(): string[] { + const operationIds: Iterable = this.#failed.length ? this.#failed : this.#stderrTails.keys(); + const lines: string[] = []; + for (const operationId of operationIds) { + const tail: string[] = this.#stderrTails.get(operationId) ?? this.#stdoutTails.get(operationId) ?? []; + for (const line of tail) { + lines.push(`${operationId}: ${line}`); + } + } + return lines; + } + + /** Returns false if rendering had already stopped; after stopping, nothing more is written. */ + #stop(): boolean { + if (this.#stopped) { + return false; + } + this.#stopped = true; + if (this.#timer) { + clearInterval(this.#timer); + this.#timer = undefined; + } + this.#clear(); + return true; + } + + #getTotal(): number { + return Math.max(this.#total, this.#registered.size, this.#done); + } + + #elapsed(): string { + return `${((this.#now() - this.#startTimeMs) / 1000).toFixed(1)}s`; + } + + #rows(): [string, string, string] { + const total: number = this.#getTotal(); + const running: string[] = [...this.#running]; + const shown: string = + running.slice(0, 3).join(', ') + (running.length > 3 ? ` +${running.length - 3} more` : ''); + const counter: string = total ? ` ${this.#done}/${total}` : ''; + return [ + `rush ${this.#options.commandName}${counter} · ${this.#elapsed()} · ${this.#phase}`, + running.length ? `running: ${shown}` : '', + this.#lastActivity + ]; + } + + #render(force: boolean): void { + if (this.#stopped) { + return; + } + const rows: [string, string, string] = this.#rows(); + if (this.#options.isTTY) { + const width: number = Math.max(20, this.#options.columns || 80) - 1; + const clip = (row: string): string => (row.length > width ? `${row.slice(0, width - 1)}…` : row); + const frame: string = SPINNER_FRAMES[this.#frame++ % SPINNER_FRAMES.length]; + const text: string = [`${frame} ${rows[0]}`, rows[1], rows[2]].map(clip).join('\n'); + this.#options.write(`${this.#painted ? `\x1b[${this.#painted}A\x1b[0J` : '\x1b[?25l'}${text}\n`); + this.#painted = 3; + return; + } + const key: string = `${this.#phase}|${this.#done}`; + const nowMs: number = this.#now(); + const sinceLast: number = nowMs - this.#lastLineAtMs; + if (!force && (sinceLast < PIPE_MIN_INTERVAL_MS || (key === this.#lastLineKey && sinceLast < PIPE_HEARTBEAT_MS))) { + return; + } + this.#lastLineKey = key; + this.#lastLineAtMs = nowMs; + this.#options.write(`${rows[0]}${rows[1] ? ` · ${rows[1]}` : ''}\n`); + } + + #clear(): void { + if (this.#options.isTTY && this.#painted) { + this.#options.write(`\x1b[${this.#painted}A\x1b[0J\x1b[?25h`); + this.#painted = 0; + } + } +} diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 102c9b732b..1145164c4f 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -25,7 +25,9 @@ import { ConsoleTerminalProvider } from '@rushstack/terminal'; import { executeDaemonCommandAsync } from './daemonCommands'; import { formatAdmissionFailure, getConfiguredAdmission } from './ClientAdmissionControls'; import { ClientOperationRenderer } from './ClientOperationRenderer'; +import type { AgentProgressRenderer } from './AgentProgressRenderer'; import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; +import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; import { getResultDiagnostic } from './resultDiagnostics'; import { writeStreamAsync } from './writeStreamAsync'; @@ -41,7 +43,10 @@ interface IWorkspaceJson { readonly daemon?: IDaemonConfigurationJson; } -export async function launchClientAsync(rushx: boolean): Promise { +export async function launchClientAsync( + rushx: boolean, + agentRenderer?: AgentProgressRenderer +): Promise { const cwd: string = process.cwd(); const environment: Readonly = Object.freeze({ ...process.env }); const rushJsonPath: string | undefined = tryFindRushJsonLocation(cwd); @@ -55,11 +60,13 @@ export async function launchClientAsync(rushx: boolean): Promise { environment, enabled: config.enabled, rushx, - hasTerminal: !!(process.stdin.isTTY || process.stdout.isTTY || process.stderr.isTTY) + hasTerminal: !!(process.stdin.isTTY || process.stdout.isTTY || process.stderr.isTTY), + useRushReporter: !rushx && !!rushJsonPath && readUseRushReporter(rushJsonPath) }); const selectedVersion: string = environment.RUSH_PREVIEW_VERSION ?? workspace?.rushVersion ?? getBundledRushVersion(); if (!rushx && route.commandName === 'daemon') { + agentRenderer?.dispose(); if ((route.argv[1] === 'start' || route.argv[1] === 'restart') && process.argv.includes('--no-daemon')) { throw new Error(`--no-daemon cannot be combined with daemon ${route.argv[1]}.`); } @@ -76,12 +83,17 @@ export async function launchClientAsync(rushx: boolean): Promise { return; } if (!route.daemon || !rushJsonPath || route.commandName === undefined) { + agentRenderer?.dispose(); launchInProcess(route.argv, rushx, selectedVersion); return; } const terminal: ConsoleTerminalProvider = new ConsoleTerminalProvider(); const verbosity: DaemonVerbosity = - route.argv.includes('--verbose') || route.argv.includes('-v') ? 'verbose' : 'quiet'; + route.argv.includes('--verbose') || route.argv.includes('-v') + ? 'verbose' + : agentRenderer + ? 'normal' + : 'quiet'; const request: IDaemonRequestEnvelope = captureDaemonRequest({ argv: route.argv, commandName: route.commandName, @@ -130,6 +142,7 @@ export async function launchClientAsync(rushx: boolean): Promise { !(error instanceof loadVersionSelectedDaemonLauncher().DaemonLauncherUnavailableError) ) throw error; + agentRenderer?.dispose(); process.stderr.write(`rush-client: ${error.message} Using in-process Rush.\n`); launchInProcess(route.argv, rushx, selectedVersion); return; @@ -167,19 +180,25 @@ export async function launchClientAsync(rushx: boolean): Promise { ); } await renderer.initializeAsync(); + agentRenderer?.setPhase('request submitted; preparing the workspace graph'); outcome = await executeWithDaemonRestartAsync(client, connection, { request, abortSignal: abort.signal, onStdoutAsync: async (bytes, operationId) => { + if (agentRenderer) return agentRenderer.onLog(bytes, operationId, 'stdout'); await writeDiscoveryAsync(); await renderer.writeLogAsync(bytes, operationId, 'stdout'); }, onStderrAsync: async (bytes, operationId) => { + if (agentRenderer) return agentRenderer.onLog(bytes, operationId, 'stderr'); await writeDiscoveryAsync(); await renderer.writeLogAsync(bytes, operationId, 'stderr'); }, - onEventAsync: (event) => renderer.writeEventAsync(event), - onQueuePositionAsync: process.stderr.isTTY + onEventAsync: async (event) => + agentRenderer ? agentRenderer.onEvent(event) : renderer.writeEventAsync(event), + onQueuePositionAsync: agentRenderer + ? async (position) => agentRenderer.onQueuePosition(position) + : process.stderr.isTTY ? (position) => writeStreamAsync( process.stderr, @@ -206,6 +225,7 @@ export async function launchClientAsync(rushx: boolean): Promise { } } if (outcome.kind === 'result') { + agentRenderer?.finish(outcome.result); process.exitCode = outcome.result.exitCode; const diagnostic: string | undefined = getResultDiagnostic(outcome.result); if (diagnostic) { @@ -217,10 +237,13 @@ export async function launchClientAsync(rushx: boolean): Promise { ); } } else if (outcome.kind === 'rejected') { + agentRenderer?.finish({ exitCode: 1, errorMessage: `daemon rejected the request (${outcome.rejection.code})` }); throw new Error(`Daemon rejected the request (${outcome.rejection.code}): ${outcome.rejection.message}`); } else if (abort.signal.aborted) { + agentRenderer?.finish({ exitCode: 130, errorMessage: 'cancelled' }); process.exitCode = 130; } else { + agentRenderer?.dispose(); process.stderr.write(`rush-client: ${outcome.message ?? outcome.reason}; using in-process Rush.\n`); launchInProcess(route.argv, rushx, selectedVersion); } diff --git a/apps/rush-cli-client/src/outputSelection.ts b/apps/rush-cli-client/src/outputSelection.ts new file mode 100644 index 0000000000..cc5bce489c --- /dev/null +++ b/apps/rush-cli-client/src/outputSelection.ts @@ -0,0 +1,143 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// Keep this module free of heavy imports: start.ts loads it before @microsoft/rush-lib. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +/** Environment variable that selects the rush-client output mode: `agent` or `legacy`. */ +export const RUSHD_OUTPUT_ENV_VAR: 'RUSHD_OUTPUT' = 'RUSHD_OUTPUT'; + +/** + * Agent markers that auto-select agent output. This intentionally matches the default of + * `detectAgent()` in `@rushstack/reporter` (libraries/reporter/src/config/AgentDetection.ts). + */ +const AGENT_MARKERS: readonly string[] = ['COPILOT_CLI']; +const INACTIVE_VALUES: ReadonlySet = new Set(['', '0', 'false', 'no', 'off']); +const NATIVE_REPORTER_FLAGS: readonly string[] = ['--reporter', '--output', '--log-level']; + +export type ClientOutputMode = 'agent' | 'legacy'; + +function isActive(value: string | undefined): boolean { + return value !== undefined && !INACTIVE_VALUES.has(value.trim().toLowerCase()); +} + +/** + * Returns true when `RUSH_REPORTER` requests a native reporter, i.e. it is set to anything other than + * the `legacy` escape hatch. Normalized like `isLegacyEmergencyFallbackRequested()` in `@rushstack/reporter`. + */ +export function isNativeReporterEnvironmentRequested(value: string | undefined): boolean { + return value !== undefined && value.trim().toLowerCase() !== 'legacy'; +} + +/** + * Returns the command name for early agent output, or undefined when the invocation has no plain + * command (a leading option, `--help`/`-h`, or `daemon`). Daemon admission controls (`--no-wait`, + * `--wait-timeout SECONDS`) are skipped like `parseClientAdmissionControls()`, which is not imported + * here to avoid loading `@rushstack/rush-daemon-protocol` before the first line; invalid controls + * are reported later by the full parser. + */ +export function getAgentCommandName(argv: ReadonlyArray): string | undefined { + const remaining: string[] = []; + for (let index: number = 0; index < argv.length && argv[index] !== '--'; index++) { + const arg: string = argv[index]; + if (arg === '--wait-timeout') { + index++; + } else if (arg !== '--no-wait' && !arg.startsWith('--wait-timeout=')) { + remaining.push(arg); + } + } + const commandName: string | undefined = remaining[0]; + if ( + commandName === undefined || + commandName.startsWith('-') || + commandName === 'daemon' || + remaining.includes('--help') || + remaining.includes('-h') + ) { + return undefined; + } + return commandName; +} + +/** + * Returns true when the invocation explicitly selects a reporter, output or log level + * (`--reporter`, `--output`, `--log-level` before `--`, `RUSH_REPORTER` other than `legacy`, + * or `RUSH_LOG_LEVEL`). Such requests always use the native reporter path. + */ +export function hasExplicitReporterControls( + argv: ReadonlyArray, + environment: Readonly> +): boolean { + if (environment.RUSH_LOG_LEVEL !== undefined) { + return true; + } + if (isNativeReporterEnvironmentRequested(environment.RUSH_REPORTER)) { + return true; + } + const separator: number = argv.indexOf('--'); + const prefix: ReadonlyArray = separator < 0 ? argv : argv.slice(0, separator); + return prefix.some((arg) => NATIVE_REPORTER_FLAGS.some((name) => arg === name || arg.startsWith(`${name}=`))); +} + +/** + * Reads the `useRushReporter` opt-in from `common/config/rush/experiments.json` next to `rush.json`. + * A missing or unreadable file means the opt-in is absent; in-process Rush reports invalid files. + */ +export function readUseRushReporter(rushJsonPath: string): boolean { + const experimentsPath: string = path.join(path.dirname(rushJsonPath), 'common', 'config', 'rush', 'experiments.json'); + let contents: string; + try { + contents = fs.readFileSync(experimentsPath, 'utf8'); + } catch { + return false; + } + const uncommented: string = contents.replace(/\/\*[\s\S]*?\*\//g, '').replace(/^\s*\/\/.*$/gm, ''); + return /"useRushReporter"\s*:\s*true\b/.test(uncommented); +} + +/** Finds `rush.json` in `startingFolder` or an ancestor without loading `@microsoft/rush-lib`. */ +export function findRushJsonPath(startingFolder: string): string | undefined { + let folder: string = path.resolve(startingFolder); + for (;;) { + const candidate: string = path.join(folder, 'rush.json'); + if (fs.existsSync(candidate)) { + return candidate; + } + const parent: string = path.dirname(folder); + if (parent === folder) { + return undefined; + } + folder = parent; + } +} + +export interface IClientOutputModeOptions { + readonly argv: ReadonlyArray; + readonly environment: Readonly>; + /** Whether the repository opted into the native reporter (experiments.json `useRushReporter`). */ + readonly useRushReporter?: boolean; +} + +/** + * Selects the rush-client output mode. Requests that will use the native reporter path + * (explicit reporter controls, `--no-daemon`, or a `useRushReporter` repository) always use `legacy`, + * so that nothing is written ahead of native reporter output. Otherwise: + * 1. `RUSHD_OUTPUT=agent|legacy` + * 2. an active agent marker (`COPILOT_CLI`) selects `agent` + * 3. otherwise `legacy` (the unchanged default output) + */ +export function selectClientOutputMode(options: IClientOutputModeOptions): ClientOutputMode { + const { argv, environment } = options; + const separator: number = argv.indexOf('--'); + const prefix: ReadonlyArray = separator < 0 ? argv : argv.slice(0, separator); + if (options.useRushReporter || prefix.includes('--no-daemon') || hasExplicitReporterControls(argv, environment)) { + return 'legacy'; + } + const explicit: string | undefined = environment[RUSHD_OUTPUT_ENV_VAR]?.trim().toLowerCase(); + if (explicit === 'agent' || explicit === 'legacy') { + return explicit; + } + return AGENT_MARKERS.some((name) => isActive(environment[name])) ? 'agent' : 'legacy'; +} \ No newline at end of file diff --git a/apps/rush-cli-client/src/routing.ts b/apps/rush-cli-client/src/routing.ts index 33b6c9418a..b9f8c52f84 100644 --- a/apps/rush-cli-client/src/routing.ts +++ b/apps/rush-cli-client/src/routing.ts @@ -6,6 +6,7 @@ import type { IRushXCommandLineArguments } from '@microsoft/rush-lib'; import { loadRushLib } from './lazyRushModules'; import { parseClientAdmissionControls, type IClientAdmissionControls } from './ClientAdmissionControls'; +import { isNativeReporterEnvironmentRequested } from './outputSelection'; const neverDaemonize: ReadonlySet = new Set([ 'add', @@ -34,6 +35,8 @@ export interface IClientRouteOptions { readonly enabled: boolean; readonly rushx: boolean; readonly hasTerminal?: boolean; + /** The repository's experiments.json `useRushReporter` opt-in; such requests use the native reporter. */ + readonly useRushReporter?: boolean; } export interface IClientRoute { @@ -59,11 +62,12 @@ export function selectClientRoute(options: IClientRouteOptions): IClientRoute { const commandName: string | undefined = rushxArguments ? rushxArguments.commandName || undefined : argv[0]; const reporterControls: boolean = options.environment.RUSH_LOG_LEVEL !== undefined || - (options.environment.RUSH_REPORTER !== undefined && options.environment.RUSH_REPORTER !== 'legacy') || + isNativeReporterEnvironmentRequested(options.environment.RUSH_REPORTER) || (!options.rushx && prefix.some((arg) => ['--reporter', '--output', '--log-level'].some((name) => arg === name || arg.startsWith(`${name}=`)) - )); + )) || + (!options.rushx && !!options.useRushReporter); const ci: boolean = ['CI', 'TF_BUILD', 'GITHUB_ACTIONS', 'JENKINS_URL', 'TEAMCITY_VERSION'].some((key) => { const value: string | undefined = options.environment[key]; return value !== undefined && value !== '' && value !== '0' && value !== 'false'; diff --git a/apps/rush-cli-client/src/start.ts b/apps/rush-cli-client/src/start.ts index d940f3f82a..989adb4e9f 100644 --- a/apps/rush-cli-client/src/start.ts +++ b/apps/rush-cli-client/src/start.ts @@ -1,9 +1,40 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { launchClientAsync } from './launchClient'; +import { AgentProgressRenderer } from './AgentProgressRenderer'; +import { + findRushJsonPath, + getAgentCommandName, + readUseRushReporter, + selectClientOutputMode +} from './outputSelection'; -launchClientAsync(false).catch((error: Error) => { +const startTimeMs: number = Date.now(); +const argv: string[] = process.argv.slice(2); +const commandName: string | undefined = getAgentCommandName(argv); +const rushJsonPath: string | undefined = findRushJsonPath(process.cwd()); +// Write the agent status line before loading @microsoft/rush-lib (hundreds of milliseconds). +const agentRenderer: AgentProgressRenderer | undefined = + selectClientOutputMode({ + argv, + environment: process.env, + useRushReporter: !!rushJsonPath && readUseRushReporter(rushJsonPath) + }) === 'agent' && + commandName !== undefined + ? new AgentProgressRenderer({ + commandName, + isTTY: !!process.stdout.isTTY && process.env.TERM !== 'dumb', + columns: process.stdout.columns || 80, + write: (text: string) => process.stdout.write(text), + startTimeMs + }) + : undefined; +agentRenderer?.start(); + +const { launchClientAsync } = require('./launchClient') as typeof import('./launchClient'); + +launchClientAsync(false, agentRenderer).catch((error: Error) => { + agentRenderer?.finish({ exitCode: 1, errorMessage: error.message }); process.stderr.write(`rush-client: ${error.message}\n`); process.exitCode = 1; }); diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts new file mode 100644 index 0000000000..6a23fc2c5c --- /dev/null +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -0,0 +1,199 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { + DAEMON_PROTOCOL_VERSION, + type DaemonEventType, + type IDaemonEventEnvelope +} from '@rushstack/rush-daemon-protocol'; + +import { AgentProgressRenderer } from '../AgentProgressRenderer'; + +function event(type: DaemonEventType, payload: unknown): IDaemonEventEnvelope { + return { + eventId: 'event', + sessionId: 'session', + sequence: 1, + timestamp: new Date().toISOString(), + protocolVersion: DAEMON_PROTOCOL_VERSION, + source: { packageName: 'test', packageVersion: '1.0.0' }, + privacy: 'public', + required: false, + type, + payload + }; +} + +const ANSI_ESCAPE: RegExp = new RegExp(`${String.fromCharCode(27)}\\[[0-9;?]*[A-Za-z]`, 'g'); + +function createRenderer(isTTY: boolean): { renderer: AgentProgressRenderer; output: string[]; clock: { ms: number } } { + const output: string[] = []; + const clock: { ms: number } = { ms: 0 }; + const renderer: AgentProgressRenderer = new AgentProgressRenderer({ + commandName: 'build', + isTTY, + columns: 60, + write: (text: string) => output.push(text), + now: () => clock.ms, + startTimeMs: 0 + }); + return { renderer, output, clock }; +} + +function status(operationId: string, value: string): IDaemonEventEnvelope { + return event('operationStatusChanged', { operationId, previousStatus: 'READY', status: value }); +} + +describe(AgentProgressRenderer.name, () => { + it('writes a first line immediately and a bounded summary for a successful build (pipe)', () => { + const { renderer, output, clock } = createRenderer(false); + renderer.start(); + expect(output).toEqual(['rush build · 0.0s · connecting to rushd (auto-starts if needed)\n']); + renderer.onEvent(event('operationRegistered', { operationId: 'a (build)', silent: false })); + renderer.onEvent(event('operationRegistered', { operationId: 'b (build)', silent: false })); + renderer.onEvent(event('operationRegistered', { operationId: 'hidden', silent: true })); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('noise\n'), 'a (build)', 'stdout'); + clock.ms = 2500; + renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.onEvent(status('b (build)', 'SKIPPED')); + clock.ms = 3000; + renderer.finish({ exitCode: 0 }); + renderer.dispose(); + expect(output.join('')).not.toContain('noise'); + expect(output[output.length - 1]).toBe( + 'rush build: SUCCESS 2/2 operations (1 success, 1 skipped) in 3.0s\n' + ); + expect(output.length).toBeLessThanOrEqual(4); + }); + + it('reports an up-to-date request instead of printing nothing', () => { + const { renderer, output } = createRenderer(false); + renderer.finish({ exitCode: 0 }); + expect(output).toEqual(['rush build: SUCCESS up to date (no operations needed) in 0.0s\n']); + }); + + it('lists failed operations and a bounded stderr tail on failure', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(status('p05 (build)', 'EXECUTING')); + for (let i = 0; i < 20; i++) { + renderer.onLog(Buffer.from(`error ${i}\n`), 'p05 (build)', 'stderr'); + } + renderer.onEvent(status('p05 (build)', 'FAILURE')); + renderer.onEvent(status('p06 (build)', 'BLOCKED')); + renderer.finish({ exitCode: 1 }); + const text: string = output.join(''); + expect(text).toContain('rush build: FAILURE 2/2 operations (1 failure, 1 blocked) in 0.0s · failed: p05 (build)\n'); + expect(text).toContain(' p05 (build): error 10\n'); + expect(text).toContain(' p05 (build): error 19\n'); + expect(text).not.toContain('error 9\n'); + }); + + it('keeps failure diagnostics when successful operations wrote stderr first', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(status('noisy (build)', 'EXECUTING')); + for (let i = 0; i < 20; i++) { + renderer.onLog(Buffer.from(`warning ${i}\n`), 'noisy (build)', 'stderr'); + } + renderer.onEvent(status('noisy (build)', 'SUCCESS WITH WARNINGS')); + renderer.onEvent(status('broken (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('the real error\n'), 'broken (build)', 'stderr'); + renderer.onEvent(status('broken (build)', 'FAILURE')); + renderer.finish({ exitCode: 1 }); + const text: string = output.join(''); + expect(text).toContain(' broken (build): the real error\n'); + expect(text).not.toContain('noisy (build): warning'); + }); + + it('counts ABORTED operations as finished', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(event('operationRegistered', { operationId: 'a (build)', silent: false })); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onEvent(status('a (build)', 'ABORTED')); + renderer.finish({ exitCode: 1 }); + expect(output[output.length - 1]).toBe('rush build: FAILURE 1/1 operations (1 aborted) in 0.0s\n'); + }); + + it('writes the final line at most once and nothing after it', () => { + const { renderer, output } = createRenderer(false); + renderer.finish({ exitCode: 1, errorMessage: 'daemon rejected the request (x)' }); + renderer.finish({ exitCode: 1, errorMessage: 'again' }); + renderer.onQueuePosition(3); + renderer.dispose(); + expect(output).toEqual(['rush build: FAILURE 0/0 operations in 0.0s · daemon rejected the request (x)\n']); + }); + + it('does not repeat an unchanged queue position', () => { + const { renderer, output } = createRenderer(false); + renderer.onQueuePosition(2); + renderer.onQueuePosition(2); + renderer.onQueuePosition(2); + expect(output).toHaveLength(1); + renderer.onQueuePosition(1); + expect(output).toHaveLength(2); + renderer.dispose(); + }); + + it('shows the stdout tail of a failed operation that reported errors on stdout', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(status('ok (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('ok noise\n'), 'ok (build)', 'stdout'); + renderer.onEvent(status('ok (build)', 'SUCCESS')); + renderer.onEvent(status('tsc (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('src/x.ts(1,1): error TS1005: stdout-error\n'), 'tsc (build)', 'stdout'); + renderer.onEvent(status('tsc (build)', 'FAILURE')); + renderer.finish({ exitCode: 1 }); + const text: string = output.join(''); + expect(text).toContain(' tsc (build): src/x.ts(1,1): error TS1005: stdout-error\n'); + expect(text).not.toContain('ok noise'); + }); + + it('shows queue position immediately', () => { + const { renderer, output } = createRenderer(false); + renderer.onQueuePosition(2); + expect(output[0]).toContain('queued behind another request (position 2)'); + }); + + it('writes a final summary line after a queued request completes', () => { + const { renderer, output, clock } = createRenderer(false); + renderer.start(); + renderer.onQueuePosition(1); + clock.ms = 4000; + renderer.finish({ exitCode: 0 }); + expect(output[output.length - 1]).toBe('rush build: SUCCESS up to date (no operations needed) in 4.0s\n'); + }); + + it('throttles progress lines on a pipe', () => { + const { renderer, output, clock } = createRenderer(false); + renderer.start(); + for (let i = 0; i < 50; i++) { + clock.ms = i * 10; + renderer.onEvent(status(`p${i} (build)`, 'EXECUTING')); + } + expect(output).toHaveLength(1); + clock.ms = 2500; + renderer.onEvent(status('p0 (build)', 'SUCCESS')); + expect(output).toHaveLength(2); + clock.ms = 3000; + renderer.onEvent(status('p1 (build)', 'SUCCESS')); + expect(output).toHaveLength(2); + renderer.dispose(); + }); + + it('renders at most three live rows on a TTY and clears them before the summary', () => { + const { renderer, output } = createRenderer(true); + renderer.start(); + renderer.onEvent(status('a-very-long-project-name-that-will-not-fit (build)', 'EXECUTING')); + renderer.finish({ exitCode: 0 }); + const frames: string[] = output.slice(0, -2); + for (const frame of frames) { + const rows: string[] = frame.replace(ANSI_ESCAPE, '').split('\n'); + expect(rows.length).toBe(4); // three rows plus the trailing newline + for (const row of rows) { + expect(row.length).toBeLessThanOrEqual(59); + } + } + expect(output[output.length - 2]).toBe('\x1b[3A\x1b[0J\x1b[?25h'); + expect(output[output.length - 1]).toContain('rush build: SUCCESS'); + }); +}); diff --git a/apps/rush-cli-client/src/test/outputSelection.test.ts b/apps/rush-cli-client/src/test/outputSelection.test.ts new file mode 100644 index 0000000000..23f11f467d --- /dev/null +++ b/apps/rush-cli-client/src/test/outputSelection.test.ts @@ -0,0 +1,92 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + findRushJsonPath, + getAgentCommandName, + readUseRushReporter, + selectClientOutputMode +} from '../outputSelection'; + +describe(selectClientOutputMode.name, () => { + it.each([ + { argv: ['build'], environment: {}, mode: 'legacy' }, + { argv: ['build'], environment: { RUSHD_OUTPUT: 'agent' }, mode: 'agent' }, + { argv: ['build'], environment: { RUSHD_OUTPUT: ' AGENT ' }, mode: 'agent' }, + { argv: ['build'], environment: { COPILOT_CLI: '1' }, mode: 'agent' }, + { argv: ['build'], environment: { COPILOT_CLI: 'false' }, mode: 'legacy' }, + { argv: ['build'], environment: { COPILOT_CLI: '1', RUSHD_OUTPUT: 'legacy' }, mode: 'legacy' }, + { argv: ['build'], environment: { CLAUDECODE: '1' }, mode: 'legacy' }, + { argv: ['build'], environment: { COPILOT_CLI: '1', RUSH_REPORTER: 'legacy' }, mode: 'agent' }, + { argv: ['build'], environment: { COPILOT_CLI: '1', RUSH_REPORTER: ' LEGACY ' }, mode: 'agent' }, + // Explicit reporter controls always keep the native reporter path, and nothing is written ahead of it. + { argv: ['build', '--reporter=ai'], environment: { RUSHD_OUTPUT: 'agent' }, mode: 'legacy' }, + { argv: ['build', '--reporter', 'ai'], environment: { COPILOT_CLI: '1' }, mode: 'legacy' }, + { argv: ['build', '--reporter=ai', '--no-daemon'], environment: { COPILOT_CLI: '1' }, mode: 'legacy' }, + { argv: ['build', '--no-daemon'], environment: { RUSHD_OUTPUT: 'agent' }, mode: 'legacy' }, + { argv: ['build', '--output', 'x.log'], environment: { RUSHD_OUTPUT: 'agent' }, mode: 'legacy' }, + { argv: ['build'], environment: { RUSHD_OUTPUT: 'agent', RUSH_REPORTER: 'ai' }, mode: 'legacy' }, + { argv: ['build'], environment: { RUSHD_OUTPUT: 'agent', RUSH_LOG_LEVEL: 'debug' }, mode: 'legacy' }, + { argv: ['build', '--', '--reporter=ai'], environment: { RUSHD_OUTPUT: 'agent' }, mode: 'agent' } + ])('selects $mode for $argv with $environment', ({ argv, environment, mode }) => { + expect(selectClientOutputMode({ argv, environment })).toBe(mode); + }); + + it('keeps useRushReporter repositories on the native reporter output', () => { + expect( + selectClientOutputMode({ argv: ['build'], environment: { COPILOT_CLI: '1' }, useRushReporter: true }) + ).toBe('legacy'); + }); +}); + +describe(getAgentCommandName.name, () => { + it.each([ + { argv: ['build'], commandName: 'build' }, + { argv: ['--wait-timeout', '1.25', 'build'], commandName: 'build' }, + { argv: ['--wait-timeout=2', 'build', '-t', 'a'], commandName: 'build' }, + { argv: ['--no-wait', 'rebuild'], commandName: 'rebuild' }, + { argv: ['--', 'build'], commandName: undefined }, + { argv: ['-q', 'build'], commandName: undefined }, + { argv: ['daemon', 'status'], commandName: undefined }, + { argv: ['build', '--help'], commandName: undefined }, + { argv: ['build', '-h'], commandName: undefined }, + { argv: ['build', '--', '--help'], commandName: 'build' }, + { argv: [], commandName: undefined } + ])('returns $commandName for $argv', ({ argv, commandName }) => { + expect(getAgentCommandName(argv)).toBe(commandName); + }); +}); + +describe(readUseRushReporter.name, () => { + let folder: string; + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-output-selection-')); + fs.writeFileSync(path.join(folder, 'rush.json'), '{}'); + fs.mkdirSync(path.join(folder, 'common', 'config', 'rush'), { recursive: true }); + }); + afterEach(() => fs.rmSync(folder, { recursive: true, force: true })); + + function write(contents: string): void { + fs.writeFileSync(path.join(folder, 'common', 'config', 'rush', 'experiments.json'), contents); + } + + it('reads the opt-in and ignores commented-out settings', () => { + const rushJsonPath: string = path.join(folder, 'rush.json'); + expect(readUseRushReporter(rushJsonPath)).toBe(false); + write('{\n // "useRushReporter": true,\n /* "useRushReporter": true */\n}\n'); + expect(readUseRushReporter(rushJsonPath)).toBe(false); + write('{ "useRushReporter": false }'); + expect(readUseRushReporter(rushJsonPath)).toBe(false); + write('{\n "useRushReporter": true\n}\n'); + expect(readUseRushReporter(rushJsonPath)).toBe(true); + }); + + it('finds rush.json from a nested folder', () => { + const nested: string = path.join(folder, 'common', 'config'); + expect(findRushJsonPath(nested)).toBe(path.join(folder, 'rush.json')); + }); +}); \ No newline at end of file diff --git a/apps/rush-cli-client/src/test/routing.test.ts b/apps/rush-cli-client/src/test/routing.test.ts index b71a98a4e6..0bac92c254 100644 --- a/apps/rush-cli-client/src/test/routing.test.ts +++ b/apps/rush-cli-client/src/test/routing.test.ts @@ -16,6 +16,7 @@ describe('opt-in routing', () => { { argv: ['build', '--log-level=debug'], enabled: true, environment: {}, daemon: false }, { argv: ['build'], enabled: true, environment: { RUSH_REPORTER: 'json' }, daemon: false }, { argv: ['build'], enabled: true, environment: { RUSH_REPORTER: 'legacy' }, daemon: true }, + { argv: ['build'], enabled: true, environment: { RUSH_REPORTER: ' LEGACY ' }, daemon: true }, { argv: ['install'], enabled: true, environment: { RUSH_DAEMON: '1' }, daemon: true }, { argv: ['update'], enabled: true, environment: { RUSH_DAEMON: '1' }, daemon: true }, { argv: ['install', '--no-daemon'], enabled: true, environment: { RUSH_DAEMON: '1' }, daemon: false }, @@ -23,11 +24,30 @@ describe('opt-in routing', () => { { argv: ['publish'], enabled: true, environment: { RUSH_DAEMON: '1' }, daemon: false }, { argv: ['daemon', 'status'], enabled: true, environment: {}, daemon: false }, { argv: ['--help'], enabled: true, environment: {}, daemon: false }, - { argv: ['build', '--help'], enabled: true, environment: {}, daemon: false } + { argv: ['build', '--help'], enabled: true, environment: {}, daemon: false }, + { argv: ['build', '--reporter=ai'], enabled: true, environment: {}, daemon: false }, + { argv: ['build', '--reporter', 'ai'], enabled: true, environment: {}, daemon: false }, + { argv: ['build', '--reporter=ai', '--no-daemon'], enabled: true, environment: {}, daemon: false }, + { argv: ['build'], enabled: true, environment: { RUSH_REPORTER: 'ai' }, daemon: false }, + { argv: ['build'], enabled: true, environment: {}, useRushReporter: true, daemon: false }, + { argv: ['build'], enabled: true, environment: {}, useRushReporter: false, daemon: true } ])('selects $daemon for $argv', ({ daemon, ...options }) => { expect(selectClientRoute({ ...options, rushx: false }).daemon).toBe(daemon); }); + it('forwards an explicit AI reporter flag unchanged to the native path', () => { + expect( + selectClientRoute({ argv: ['build', '--reporter', 'ai'], enabled: true, environment: {}, rushx: false }) + ).toMatchObject({ argv: ['build', '--reporter', 'ai'], daemon: false }); + }); + + it('ignores useRushReporter for rushx scripts', () => { + expect( + selectClientRoute({ argv: ['build'], enabled: true, environment: {}, rushx: true, useRushReporter: true }) + .daemon + ).toBe(true); + }); + it('preserves script arguments after -- and permits scripts named like built-ins', () => { expect( selectClientRoute({ diff --git a/common/changes/@rushstack/rush-cli-client/agent-reporter_2026-09-24-02-35.json b/common/changes/@rushstack/rush-cli-client/agent-reporter_2026-09-24-02-35.json new file mode 100644 index 0000000000..97bc13b49c --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-reporter_2026-09-24-02-35.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Add an opt-in agent output mode on the daemon path (RUSHD_OUTPUT=agent or COPILOT_CLI) with an immediate first line, bounded live progress, and a guaranteed final summary; keep repositories that opt into useRushReporter on the native reporter path.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} From 8c34d131a3271c6dceb74b0537d7f807115c8c21 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:17:54 -0700 Subject: [PATCH 008/265] [rush-daemon] Run rushx scripts without exclusive admission and complete on child exit (#6091) * [rush-daemon] Run rushx scripts without exclusive admission and complete on child exit Fixes #6085 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-cli-client] Update rushx tests for admission-free script execution Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Keep close-based child completion on Windows Windows cannot recover a child's descendants after it exits, so closing pipes remain the only signal that native install/update workers have finished. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Scope post-exit output drain progress to each child Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../src/test/RushXDaemonAlias.test.ts | 29 +-- .../src/test/RushXDaemonBoundaries.test.ts | 25 +-- .../src/test/pipedInput.test.ts | 14 +- .../rushx-admission-6085_2026-09-24.json | 10 + common/reviews/api/rush-daemon.api.md | 4 + .../src/DaemonRequestDispatcher.ts | 1 + .../src/GlobalCommandExecutionContext.ts | 138 +++++++++++--- .../rush-daemon/src/GlobalCommandRequest.ts | 5 + .../src/GlobalCommandRequestRouter.ts | 35 ++-- .../test/GlobalCommandRequestRouter.test.ts | 177 ++++++++++++++++++ 10 files changed, 354 insertions(+), 84 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/rushx-admission-6085_2026-09-24.json diff --git a/apps/rush-cli-client/src/test/RushXDaemonAlias.test.ts b/apps/rush-cli-client/src/test/RushXDaemonAlias.test.ts index 7c6a4361be..b8f81115f6 100644 --- a/apps/rush-cli-client/src/test/RushXDaemonAlias.test.ts +++ b/apps/rush-cli-client/src/test/RushXDaemonAlias.test.ts @@ -128,7 +128,7 @@ console.log(JSON.stringify({ expect(result.stdout.length).toBe(0); }); - it('pins physical request identity when an alias changes while waiting for admission', async () => { + it('runs through an alias without waiting for admission behind an exclusive request', async () => { fixture.write( 'retargeted/projects/a/package.json', JSON.stringify({ @@ -163,33 +163,16 @@ console.log(JSON.stringify({ invocationKind: 'rush' }); await holding; - const input: PassThrough = new PassThrough(); - input.end('not consumed before execution'); + const onQueuePositionAsync = jest.fn(async () => undefined); try { const result = await fixture.runAsync( fixture.request(['-q', 'build'], path.join(aliasRoot, 'projects/a')), undefined, - { - stdin: input, - onQueuePositionAsync: async () => { - fs.unlinkSync(aliasRoot); - fs.symlinkSync( - path.join(physicalRoot, 'retargeted'), - aliasRoot, - process.platform === 'win32' ? 'junction' : 'dir' - ); - release(); - } - } + { onQueuePositionAsync } ); - if (process.platform === 'win32') { - expect(result.outcome).toMatchObject({ - kind: 'result', - result: { exitCode: 1, errorMessage: expect.stringContaining('invocation directory changed') } - }); - expect(input.read().toString()).toBe('not consumed before execution'); - } else { - expect(result.exitCode).toBe(0); + expect(result.exitCode).toBe(0); + expect(onQueuePositionAsync).not.toHaveBeenCalled(); + if (process.platform !== 'win32') { expect(JSON.parse(result.stdout.toString()).cwd).toBe(path.join(physicalRoot, 'projects/a')); } expect(fs.existsSync(path.join(physicalRoot, 'retargeted/projects/a/executed'))).toBe(false); diff --git a/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts b/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts index 451373461d..ecb58c4df2 100644 --- a/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts +++ b/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts @@ -156,7 +156,7 @@ describe('native Rushx execution boundaries', () => { } ); - it('fails visibly without running or reading when hooks change during queue admission', async () => { + it('runs a script without queueing behind an exclusive workspace request', async () => { const cwd: string = await startAsync(); let release: () => void = () => {}; let started: () => void = () => {}; @@ -177,25 +177,14 @@ describe('native Rushx execution boundaries', () => { }); const holder = fixture.runAsync({ ...fixture.request(['hold'], cwd), invocationKind: 'rush' }); await holding; - const stdin: PassThrough = new PassThrough(); - stdin.end('not-consumed'); + const onQueuePositionAsync = jest.fn(async () => undefined); try { - const result = await fixture.runAsync(fixture.request(['build'], cwd), undefined, { - stdin, - onQueuePositionAsync: async () => { - const file: string = path.join(fixture.folder, 'rush.json'); - const config = JSON.parse(fs.readFileSync(file, 'utf8')); - fixture.write( - 'rush.json', - JSON.stringify({ ...config, eventHooks: { preRushx: ['node hook.cjs'] } }) - ); - release(); - } + const result = await fixture.runAsync(fixture.request(['-q', 'build'], cwd), undefined, { + onQueuePositionAsync }); - expect(result.outcome).toMatchObject({ kind: 'result', result: { exitCode: 1, outcome: 'failure' } }); - expect(result.stderr.toString()).toContain('Rush configuration changed'); - expect(stdin.read().toString()).toBe('not-consumed'); - expect(fs.existsSync(path.join(cwd, 'runs.txt'))).toBe(false); + expect(result.outcome).toMatchObject({ kind: 'result', result: { exitCode: 0, outcome: 'success' } }); + expect(onQueuePositionAsync).not.toHaveBeenCalled(); + expect(fs.readFileSync(path.join(cwd, 'runs.txt'), 'utf8')).toBe('ran\n'); } finally { release(); await holder; diff --git a/apps/rush-cli-client/src/test/pipedInput.test.ts b/apps/rush-cli-client/src/test/pipedInput.test.ts index aac77051f4..0de06c1cc3 100644 --- a/apps/rush-cli-client/src/test/pipedInput.test.ts +++ b/apps/rush-cli-client/src/test/pipedInput.test.ts @@ -132,11 +132,11 @@ describe('standalone client piped input', () => { ); it.each([ - { args: ['--no-wait'], admission: { noWait: true }, reason: 'no-wait' }, - { args: ['--wait-timeout=0.01'], admission: { waitTimeoutMs: 10 }, reason: 'wait-timeout' } + { args: ['--no-wait'], admission: { noWait: true } }, + { args: ['--wait-timeout=0.01'], admission: { waitTimeoutMs: 10 } } ])( - 'forwards $args without leaking queue flags to scripts', - async ({ args, admission, reason }) => { + 'forwards $args without leaking queue flags to scripts, which do not queue for admission', + async ({ args, admission }) => { let started: () => void = () => {}; let release: () => void = () => {}; const running: Promise = new Promise((resolve) => { @@ -178,9 +178,9 @@ describe('standalone client piped input', () => { }) ]); const result: IPipedResult = await invokeAsync(Buffer.alloc(0), true, args); - expect(result.code).toBe(1); - expect(result.stderr.toString()).toContain(`daemon admission failed (${reason})`); - expect(runScriptAsync).not.toHaveBeenCalled(); + expect(result.code).toBe(0); + expect(result.stderr.toString()).not.toContain('daemon admission failed'); + expect(runScriptAsync).toHaveBeenCalledTimes(1); } finally { release(); await holding; diff --git a/common/changes/@rushstack/rush-daemon/rushx-admission-6085_2026-09-24.json b/common/changes/@rushstack/rush-daemon/rushx-admission-6085_2026-09-24.json new file mode 100644 index 0000000000..413e48926e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushx-admission-6085_2026-09-24.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Run Rushx package scripts without exclusive workspace admission, complete global command requests on child exit plus a bounded output drain, and terminate an exited child's process group on cancellation.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index dfb68a7841..fd40da63ca 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -7,6 +7,7 @@ /// import * as childProcess from 'node:child_process'; +import type { DaemonInvocationKind } from '@rushstack/rush-daemon-protocol'; import type { DaemonRushCommandOrigin } from '@rushstack/rush-daemon-protocol'; import type { DaemonTerminalRequirement } from '@rushstack/rush-daemon-protocol'; import * as fs from 'node:fs'; @@ -435,6 +436,8 @@ export interface IResolvedGlobalCommandRequest { // (undocumented) readonly environment: IGlobalCommandEnvironment; // (undocumented) + readonly invocationKind?: DaemonInvocationKind; + // (undocumented) readonly requestId: string; // (undocumented) readonly terminal: IGlobalCommandTerminalProperties; @@ -452,6 +455,7 @@ export interface IResolveGlobalCommandRequestOptions { readonly cwd: string; // (undocumented) readonly environment: Readonly; + readonly invocationKind?: DaemonInvocationKind; // (undocumented) readonly requestId: string; // (undocumented) diff --git a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts index e78e631731..d984b44f91 100644 --- a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts +++ b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts @@ -201,6 +201,7 @@ async function dispatchWorkspaceRequestAsync( commandOrigin: isRushxInvocation(envelope) ? 'custom' : envelope.commandOrigin, cwd: envelope.cwd, environment: envelope.environment, + invocationKind: isRushxInvocation(envelope) ? 'rushx' : 'rush', requestId: envelope.requestId, terminal: { ...envelope.terminal, diff --git a/libraries/rush-daemon/src/GlobalCommandExecutionContext.ts b/libraries/rush-daemon/src/GlobalCommandExecutionContext.ts index bbdfa7ef93..c2cffbb35a 100644 --- a/libraries/rush-daemon/src/GlobalCommandExecutionContext.ts +++ b/libraries/rush-daemon/src/GlobalCommandExecutionContext.ts @@ -27,6 +27,11 @@ import { waitForLinuxProcessGroupExitAsync } from './LinuxProcessGroupExit'; import { recordWorkspaceRequestCleanupFailure } from './WorkspaceRequestResources'; const MAX_PENDING_TERMINAL_BYTES: number = 1024 * 1024; +/** + * How long an exited child's output pipes may stay idle before they are destroyed. A pipe that is still held open by + * a descendant must not keep the request (and its client) waiting indefinitely. + */ +const CHILD_OUTPUT_DRAIN_IDLE_TIMEOUT_MS: number = 250; /** * Options for a request-scoped child process. @@ -160,6 +165,12 @@ interface ITrackedChild { readonly completion: Promise; } +/** Forwarding activity for one child's output, used to bound its post-exit drain. */ +interface IChildOutputProgress { + events: number; + pendingWrites: number; +} + export class GlobalCommandExecutionContext implements IGlobalCommandExecutionContext, AsyncDisposable { readonly #abortController: AbortController = new AbortController(); readonly #client: IGlobalCommandRequestClient; @@ -260,18 +271,22 @@ export class GlobalCommandExecutionContext implements IGlobalCommandExecutionCon windowsHide: options.windowsHide }); SubprocessTerminator.killProcessTreeOnExit(child, SubprocessTerminator.RECOMMENDED_OPTIONS); - const completion: Promise = this.#trackChildAsync(child).catch((error: unknown) => { - this.#childCompletionErrors.push(error); - }); + const outputProgress: IChildOutputProgress | undefined = + options.forwardOutput !== false ? { events: 0, pendingWrites: 0 } : undefined; + const completion: Promise = this.#trackChildAsync(child, outputProgress).catch( + (error: unknown) => { + this.#childCompletionErrors.push(error); + } + ); const trackedChild: ITrackedChild = { completion }; this.#trackedChildren.add(trackedChild); void completion.then(() => this.#trackedChildren.delete(trackedChild)); if (options.forwardInput === true) { this.#attachChildInput(child); } - if (options.forwardOutput !== false) { - this.#forwardChildOutput(child.stdout, 'stdout'); - this.#forwardChildOutput(child.stderr, 'stderr'); + if (outputProgress) { + this.#forwardChildOutput(child.stdout, 'stdout', outputProgress); + this.#forwardChildOutput(child.stderr, 'stderr', outputProgress); } return child; } @@ -339,33 +354,34 @@ export class GlobalCommandExecutionContext implements IGlobalCommandExecutionCon this.#abortController.abort(reason); } - async #trackChildAsync(child: childProcess.ChildProcessWithoutNullStreams): Promise { - const terminateChild = (): void => { - try { - SubprocessTerminator.killProcessTree(child, SubprocessTerminator.RECOMMENDED_OPTIONS); - } catch (error) { - this.#childTerminationErrors.push(this.#recordResourceCleanupFailure(error)); - try { - child.kill('SIGKILL'); - } catch (fallbackError) { - this.#childTerminationErrors.push(this.#recordResourceCleanupFailure(fallbackError)); - } - } - }; + async #trackChildAsync( + child: childProcess.ChildProcessWithoutNullStreams, + outputProgress: IChildOutputProgress | undefined + ): Promise { + const terminateChild = (): void => this.#terminateChild(child); this.abortSignal.addEventListener('abort', terminateChild, { once: true }); let childError: Error | undefined; try { + // On POSIX, complete on 'exit' rather than 'close': a background descendant may inherit and hold the output + // pipes open, and the child's process group is killed below. Windows cannot recover descendants after the child + // exits, so the pipes closing remains the only signal that the child's descendants have exited. await new Promise((resolve) => { child.once('error', (error: Error) => { childError = error; }); + if (process.platform !== 'win32') { + child.once('exit', () => resolve()); + } child.once('close', () => resolve()); }); + terminateExitedChildProcessGroup(child); + await this.#drainChildOutputAsync(child, outputProgress); + } catch (error) { + throw this.#recordResourceCleanupFailure(error); } finally { this.abortSignal.removeEventListener('abort', terminateChild); } try { - terminateExitedChildProcessGroup(child); if (process.platform === 'linux' && child.pid !== undefined) { await waitForLinuxProcessGroupExitAsync(child.pid); } @@ -375,18 +391,94 @@ export class GlobalCommandExecutionContext implements IGlobalCommandExecutionCon if (childError) throw childError; } + /** + * Terminates the child's whole process tree. + * + * @remarks + * `SubprocessTerminator.killProcessTree` is a no-op once the direct child has exited, which would leave its + * surviving process group running, so an exited child's group is signalled directly. + */ + #terminateChild(child: childProcess.ChildProcessWithoutNullStreams): void { + try { + if (hasChildExited(child)) { + terminateExitedChildProcessGroup(child); + } else { + SubprocessTerminator.killProcessTree(child, SubprocessTerminator.RECOMMENDED_OPTIONS); + } + } catch (error) { + this.#childTerminationErrors.push(this.#recordResourceCleanupFailure(error)); + try { + child.kill('SIGKILL'); + } catch (fallbackError) { + this.#childTerminationErrors.push(this.#recordResourceCleanupFailure(fallbackError)); + } + } + } + + /** + * Waits for output that the exited child left in its pipes. + * + * @remarks + * Forwarded output is drained until its pipes close or stay idle for a bounded time; the deadline is extended only + * while this child's own output is still being forwarded, so a slow client does not truncate it and unrelated output + * cannot keep the drain open. An idle pipe, for example one held by a descendant outside the child's process group, + * is then destroyed. Output that the caller consumes itself is not forwarded, so its progress cannot be observed; + * those pipes are awaited until they close or the request aborts. + */ + async #drainChildOutputAsync( + child: childProcess.ChildProcessWithoutNullStreams, + progress: IChildOutputProgress | undefined + ): Promise { + if (child.stdout.closed && child.stderr.closed) { + return; + } + let lastEvents: number | undefined = progress?.events; + await new Promise((resolve) => { + let timer: NodeJS.Timeout | undefined; + const finish = (): void => { + clearTimeout(timer); + this.abortSignal.removeEventListener('abort', finish); + resolve(); + }; + const check = (): void => { + const events: number | undefined = progress?.events; + const progressed: boolean = events !== lastEvents || (progress?.pendingWrites ?? 0) > 0; + if (!progressed || this.abortSignal.aborted) { + finish(); + } else { + lastEvents = events; + timer = setTimeout(check, CHILD_OUTPUT_DRAIN_IDLE_TIMEOUT_MS); + } + }; + child.once('close', finish); + this.abortSignal.addEventListener('abort', finish, { once: true }); + if (this.abortSignal.aborted) { + finish(); + } else if (progress) { + timer = setTimeout(check, CHILD_OUTPUT_DRAIN_IDLE_TIMEOUT_MS); + } + }); + child.stdout.destroy(); + child.stderr.destroy(); + } + #recordResourceCleanupFailure(error: unknown): Error { return recordWorkspaceRequestCleanupFailure(this.workspaceSession, this.#request.requestId, error); } #forwardChildOutput( source: NodeJS.ReadableStream & { pause(): unknown; resume(): unknown }, - stream: 'stdout' | 'stderr' + stream: 'stdout' | 'stderr', + progress: IChildOutputProgress ): void { source.on('data', (chunk: Buffer | string) => { source.pause(); const bytes: Uint8Array = typeof chunk === 'string' ? Buffer.from(chunk) : chunk; + progress.events++; + progress.pendingWrites++; void this.#writer.writeAsync(stream, bytes).then(() => { + progress.events++; + progress.pendingWrites--; if (!this.abortSignal.aborted) { source.resume(); } @@ -402,6 +494,10 @@ export class GlobalCommandExecutionContext implements IGlobalCommandExecutionCon } } +function hasChildExited(child: childProcess.ChildProcess): boolean { + return child.exitCode !== null || child.signalCode !== null; +} + function terminateExitedChildProcessGroup(child: childProcess.ChildProcessWithoutNullStreams): void { if (process.platform === 'win32' || child.pid === undefined) { return; diff --git a/libraries/rush-daemon/src/GlobalCommandRequest.ts b/libraries/rush-daemon/src/GlobalCommandRequest.ts index ed396b7dfd..0b41a05f27 100644 --- a/libraries/rush-daemon/src/GlobalCommandRequest.ts +++ b/libraries/rush-daemon/src/GlobalCommandRequest.ts @@ -7,6 +7,7 @@ import * as path from 'node:path'; import { EnvironmentMap } from '@rushstack/node-core-library'; import { validateDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; import type { + DaemonInvocationKind, DaemonRushCommandOrigin, DaemonTerminalRequirement, IDaemonRequestAdmissionOptions @@ -48,6 +49,8 @@ export interface IResolveGlobalCommandRequestOptions { readonly commandOrigin: DaemonRushCommandOrigin; readonly cwd: string; readonly environment: Readonly; + /** Whether the request runs a Rush command or a Rushx package script. Defaults to `rush`. */ + readonly invocationKind?: DaemonInvocationKind; readonly requestId: string; readonly terminal: IGlobalCommandTerminalProperties; } @@ -63,6 +66,7 @@ export interface IResolvedGlobalCommandRequest { readonly commandOrigin: DaemonRushCommandOrigin; readonly cwd: string; readonly environment: IGlobalCommandEnvironment; + readonly invocationKind?: DaemonInvocationKind; readonly requestId: string; readonly terminal: IGlobalCommandTerminalProperties; } @@ -101,6 +105,7 @@ export function resolveGlobalCommandRequest( commandOrigin: options.commandOrigin, cwd, environment: new GlobalCommandEnvironment(options.environment), + invocationKind: options.invocationKind === 'rushx' ? 'rushx' : 'rush', requestId: options.requestId, terminal: resolveTerminalProperties(options.terminal) }); diff --git a/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts b/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts index 8ce2d4a61f..95cbb69c35 100644 --- a/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts +++ b/libraries/rush-daemon/src/GlobalCommandRequestRouter.ts @@ -97,20 +97,25 @@ export class GlobalCommandRequestRouter { throw new DaemonRequiresInProcessError(policy); } let admissionController: RequestAdmissionController | undefined; - let lease: IRequestLease; + let lease: IRequestLease | undefined; try { - admissionController = new RequestAdmissionController({ - admission: request.admission, - client, - requestId: request.requestId - }); - lease = await admissionController.acquireAsync( - getWorkspaceRequestScheduler(this.#workspaceSession), - classifyRushCommand({ - commandName: request.commandName, - commandOrigin: request.commandOrigin - }) - ); + // Rushx package scripts do not read or mutate daemon-owned workspace state after resolution, so they bypass + // workspace admission. Native rushx takes no workspace lock either, and holding a scheduler lease for the + // script's whole lifetime would serialize concurrent scripts and block unrelated builds (issue #6085). + if (request.invocationKind !== 'rushx') { + admissionController = new RequestAdmissionController({ + admission: request.admission, + client, + requestId: request.requestId + }); + lease = await admissionController.acquireAsync( + getWorkspaceRequestScheduler(this.#workspaceSession), + classifyRushCommand({ + commandName: request.commandName, + commandOrigin: request.commandOrigin + }) + ); + } } catch (error) { admissionController?.dispose(); return await finishAfterAdmissionErrorAsync(request.requestId, client, interactiveSession, error); @@ -126,10 +131,10 @@ export class GlobalCommandRequestRouter { this.#workspaceSession ); } finally { - lease.release(); + lease?.release(); } } finally { - admissionController.dispose(); + admissionController?.dispose(); } } } diff --git a/libraries/rush-daemon/src/test/GlobalCommandRequestRouter.test.ts b/libraries/rush-daemon/src/test/GlobalCommandRequestRouter.test.ts index dc549b31f1..4fe768425a 100644 --- a/libraries/rush-daemon/src/test/GlobalCommandRequestRouter.test.ts +++ b/libraries/rush-daemon/src/test/GlobalCommandRequestRouter.test.ts @@ -466,6 +466,183 @@ describe(GlobalCommandRequestRouter.name, () => { } ); + it('does not hold workspace admission for Rushx package scripts', async () => { + const session: TestWorkspaceSession = new TestWorkspaceSession(TEST_REPO_ROOT); + const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(session); + const exclusiveLease = await getWorkspaceRequestScheduler(session).acquireAsync({ + exclusivityClass: RequestExclusivityClass.Exclusive, + noWait: true + }); + const bothStarted = createDeferred(); + let startedCount: number = 0; + const runScriptAsync = (requestId: string): Promise => + router.executeAsync( + router.resolveRequest({ + ...createRequestOptions(requestId, FIRST_CWD, {}, 80), + invocationKind: 'rushx' + }), + async () => { + if (++startedCount === 2) bothStarted.resolve(); + await bothStarted.promise; + return { exitCode: 0 }; + }, + new TestGlobalCommandClient() + ); + try { + const results: IGlobalCommandRequestResult[] = await Promise.all([ + runScriptAsync('script-1'), + runScriptAsync('script-2') + ]); + expect(results.map(({ outcome }) => outcome)).toEqual(['success', 'success']); + await expect( + router.executeAsync( + router.resolveRequest({ + ...createRequestOptions('rush-custom', FIRST_CWD, {}, 80), + admission: { noWait: true } + }), + async () => ({ exitCode: 0 }), + new TestGlobalCommandClient() + ) + ).resolves.toMatchObject({ outcome: 'failure' }); + } finally { + exclusiveLease.release(); + } + }); + + (process.platform === 'win32' ? it.skip : it)( + 'completes on child exit when a background descendant holds the output pipes', + async () => { + const session: TestWorkspaceSession = new TestWorkspaceSession(TEST_REPO_ROOT); + const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(session); + const client: TestGlobalCommandClient = new TestGlobalCommandClient(); + // The detached grandchild leaves the child's process group, so only the bounded pipe drain can release it. + const script: string = [ + "const { spawn } = require('node:child_process');", + "const grandchild = spawn(process.execPath, ['-e', 'setTimeout(() => {}, 60000)'],", + " { detached: true, stdio: 'inherit' });", + 'grandchild.unref();', + "process.stdout.write('grandchild=' + grandchild.pid + '\\n');" + ].join('\n'); + const startTime: number = Date.now(); + const result: IGlobalCommandRequestResult = await router.executeAsync( + router.resolveRequest(createRequestOptions('background-descendant', FIRST_CWD, {}, 80)), + async (context) => { + const child = context.spawnChild(process.execPath, ['-e', script]); + const exitCode: number | null = await new Promise((resolve) => + child.once('close', (code: number | null) => resolve(code)) + ); + return { exitCode: exitCode ?? 1 }; + }, + client + ); + const output: string = client.chunks.map(({ text }) => text).join(''); + const grandchildPid: number = Number(/grandchild=(\d+)/.exec(output)?.[1]); + try { + expect(grandchildPid).toBeGreaterThan(0); + expect(result).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(Date.now() - startTime).toBeLessThan(30000); + } finally { + if (grandchildPid > 0) { + process.kill(grandchildPid, 'SIGKILL'); + } + } + } + ); + + (process.platform === 'win32' ? it.skip : it)( + "does not extend an exited child's drain with another child's output", + async () => { + const session: TestWorkspaceSession = new TestWorkspaceSession(TEST_REPO_ROOT); + const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(session); + const client: TestGlobalCommandClient = new TestGlobalCommandClient(); + const heldScript: string = [ + "const { spawn } = require('node:child_process');", + "const grandchild = spawn(process.execPath, ['-e', 'setTimeout(() => {}, 60000)'],", + " { detached: true, stdio: 'inherit' });", + 'grandchild.unref();', + "process.stdout.write('grandchild=' + grandchild.pid + '\\n');" + ].join('\n'); + let heldCloseMs: number | undefined; + const result: IGlobalCommandRequestResult = await router.executeAsync( + router.resolveRequest(createRequestOptions('unrelated-output', FIRST_CWD, {}, 80)), + async (context) => { + const chatty = context.spawnChild(process.execPath, [ + '-e', + "setInterval(() => process.stdout.write('.'), 20)" + ]); + const startTime: number = Date.now(); + const held = context.spawnChild(process.execPath, ['-e', heldScript]); + await new Promise((resolve) => held.once('close', () => resolve())); + heldCloseMs = Date.now() - startTime; + chatty.kill('SIGKILL'); + await new Promise((resolve) => chatty.once('close', () => resolve())); + return { exitCode: 0 }; + }, + client + ); + const output: string = client.chunks.map(({ text }) => text).join(''); + const grandchildPid: number = Number(/grandchild=(\d+)/.exec(output)?.[1]); + try { + expect(grandchildPid).toBeGreaterThan(0); + expect(result).toMatchObject({ outcome: 'success' }); + expect(heldCloseMs).toBeLessThan(10000); + } finally { + if (grandchildPid > 0) { + process.kill(grandchildPid, 'SIGKILL'); + } + } + }, + 30000 + ); + + (process.platform === 'win32' ? it.skip : it)( + 'terminates the process group on cancellation after the direct child exited', + async () => { + const session: TestWorkspaceSession = new TestWorkspaceSession(TEST_REPO_ROOT); + const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(session); + const client: TestGlobalCommandClient = new TestGlobalCommandClient(); + const killProcessTreeSpy: jest.SpyInstance = jest.spyOn(SubprocessTerminator, 'killProcessTree'); + const processKillSpy: jest.SpyInstance = jest.spyOn(process, 'kill'); + const script: string = [ + "const { spawn } = require('node:child_process');", + "const grandchild = spawn(process.execPath, ['-e', 'setTimeout(() => {}, 60000)'],", + " { detached: true, stdio: 'inherit' });", + 'grandchild.unref();', + "process.stdout.write('grandchild=' + grandchild.pid + '\\n');" + ].join('\n'); + let childPid: number | undefined; + let output: string = ''; + try { + const result: IGlobalCommandRequestResult = await router.executeAsync( + router.resolveRequest(createRequestOptions('cancel-after-exit', FIRST_CWD, {}, 80)), + async (context) => { + const child = context.spawnChild(process.execPath, ['-e', script], { forwardOutput: false }); + childPid = child.pid; + child.stdout.on('data', (chunk: Buffer) => { + output += chunk.toString(); + }); + await new Promise((resolve) => child.once('exit', () => resolve())); + processKillSpy.mockClear(); + client.abortController.abort(new Error('client cancelled')); + await new Promise((resolve) => child.once('close', () => resolve())); + return { exitCode: 0 }; + }, + client + ); + expect(result).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(killProcessTreeSpy).not.toHaveBeenCalled(); + expect(processKillSpy).toHaveBeenCalledWith(-(childPid ?? 0), 'SIGKILL'); + } finally { + killProcessTreeSpy.mockRestore(); + processKillSpy.mockRestore(); + const grandchildPid: number = Number(/grandchild=(\d+)/.exec(output)?.[1]); + if (grandchildPid > 0) { + process.kill(grandchildPid, 'SIGKILL'); + } + } + } + ); + it('reports child spawn failures during request cleanup', async () => { const session: TestWorkspaceSession = new TestWorkspaceSession(TEST_REPO_ROOT); const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(session); From 83922cad193b660b1487c33fede50a52451fb8d7 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:18:06 -0700 Subject: [PATCH 009/265] [rush-daemon] Read Linux process group state from procfs instead of ps (#6087) * [rush-daemon] Read Linux process group state from procfs instead of ps Fixes #6072 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Bound procfs scans and never trust a scan with unreadable entries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- ...process-group-checks_2026-09-24-03-00.json | 10 + .../rush-daemon/src/LinuxProcessGroupExit.ts | 108 ++++++++++- .../src/test/LinuxProcessGroupExit.test.ts | 182 +++++++++++++++++- .../test/NativeMutationCleanupFailure.test.ts | 6 +- 4 files changed, 291 insertions(+), 15 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/fix-rushd-procfs-process-group-checks_2026-09-24-03-00.json diff --git a/common/changes/@rushstack/rush-daemon/fix-rushd-procfs-process-group-checks_2026-09-24-03-00.json b/common/changes/@rushstack/rush-daemon/fix-rushd-procfs-process-group-checks_2026-09-24-03-00.json new file mode 100644 index 0000000000..11fbf2beaf --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/fix-rushd-procfs-process-group-checks_2026-09-24-03-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "On Linux, verify that a rushx or global command's process group has exited by reading /proc instead of running `ps --sid`, so cleanup no longer fails (and retires the workspace session) on images without procps `ps` or with busybox `ps`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/LinuxProcessGroupExit.ts b/libraries/rush-daemon/src/LinuxProcessGroupExit.ts index 5aa60f1c32..67b2181360 100644 --- a/libraries/rush-daemon/src/LinuxProcessGroupExit.ts +++ b/libraries/rush-daemon/src/LinuxProcessGroupExit.ts @@ -2,16 +2,42 @@ // See LICENSE in the project root for license information. import { execFile } from 'node:child_process'; +import * as fs from 'node:fs'; import { setTimeout as delayAsync } from 'node:timers/promises'; +const PROC_ROOT: string = '/proc'; +const PID_ENTRY_REGEXP: RegExp = /^\d+$/; +// Indices into the fields that follow "(comm) " in /proc//stat. +const PROC_STAT_STATE_INDEX: number = 0; +const PROC_STAT_SESSION_INDEX: number = 3; +const PROC_STAT_READ_CONCURRENCY: number = 32; +// Sentinel (never a real one-letter state) for a procfs entry that exists but cannot be read. +const UNREADABLE_STATE: string = ''; const DEFAULT_EXIT_TIMEOUT_MS: number = 5_000; const EXIT_POLL_INTERVAL_MS: number = 10; const MAX_TIMEOUT_MS: number = 0x7fffffff; -/** Waits for a captured detached Linux group/session to disappear or contain only zombies. */ +/** Reads Linux procfs. Injectable so tests can simulate process tables or a missing procfs. */ +export interface ILinuxProcfsReader { + /** Lists the entries of the procfs root; rejects when procfs is unavailable. */ + readonly listEntriesAsync: () => Promise; + /** Reads `/proc//stat`; rejects when the process has exited. */ + readonly readStatAsync: (pid: string) => Promise; +} + +export const NODE_PROCFS_READER: ILinuxProcfsReader = { + listEntriesAsync: () => fs.promises.readdir(PROC_ROOT), + readStatAsync: (pid: string) => fs.promises.readFile(`${PROC_ROOT}/${pid}/stat`, 'utf8') +}; + +/** + * Waits for a captured detached Linux group/session to disappear or contain only zombies. + * Member states come from procfs; `ps --sid` is used only when procfs is unavailable. + */ export async function waitForLinuxProcessGroupExitAsync( groupId: number, - timeoutMs: number = DEFAULT_EXIT_TIMEOUT_MS + timeoutMs: number = DEFAULT_EXIT_TIMEOUT_MS, + procfs: ILinuxProcfsReader = NODE_PROCFS_READER ): Promise { if (!Number.isSafeInteger(groupId) || groupId <= 0 || groupId === process.pid) { throw new RangeError('Expected an owned child process group ID.'); @@ -25,7 +51,7 @@ export async function waitForLinuxProcessGroupExitAsync( if (remaining <= 0) { throw new Error(`Owned Linux process group ${groupId} did not exit within ${timeoutMs}ms.`); } - const states: string[] = await readSessionStatesAsync(groupId, remaining); + const states: string[] = await readSessionStatesAsync(groupId, remaining, procfs); if (states.length > 0 && states.every((state) => state.startsWith('Z'))) return; await delayAsync(Math.min(EXIT_POLL_INTERVAL_MS, remaining)); } @@ -44,7 +70,81 @@ function processGroupExists(groupId: number): boolean { } } -function readSessionStatesAsync(groupId: number, timeoutMs: number): Promise { +async function readSessionStatesAsync( + groupId: number, + timeoutMs: number, + procfs: ILinuxProcfsReader +): Promise { + const states: string[] | undefined = await tryReadSessionStatesFromProcAsync(groupId, procfs); + return states ?? (await readSessionStatesFromPsAsync(groupId, timeoutMs)); +} + +/** + * Reads member states from procfs, which every Linux system has; `ps` is missing from slim/distroless images + * and busybox `ps` does not support `--sid`. Returns `undefined` when procfs is unavailable or an entry cannot + * be read, so the caller falls back to `ps`. + */ +async function tryReadSessionStatesFromProcAsync( + groupId: number, + procfs: ILinuxProcfsReader +): Promise { + let entries: string[]; + try { + entries = await procfs.listEntriesAsync(); + } catch { + return undefined; + } + const pids: string[] = entries.filter((entry: string) => PID_ENTRY_REGEXP.test(entry)); + const states: string[] = []; + // Bound concurrent reads so a large process table cannot flood the shared libuv thread pool. + for (let start: number = 0; start < pids.length; start += PROC_STAT_READ_CONCURRENCY) { + const batch: (string | undefined)[] = await Promise.all( + pids + .slice(start, start + PROC_STAT_READ_CONCURRENCY) + .map((pid: string) => readSessionMemberStateAsync(pid, groupId, procfs)) + ); + for (const state of batch) { + if (state === undefined) continue; + // An unreadable entry could hide a live member, so procfs cannot prove the group exited. + if (state === UNREADABLE_STATE) return undefined; + // One live member already means "not exited"; only a zombies-only result needs a full scan. + if (!state.startsWith('Z')) return [state]; + states.push(state); + } + } + return states; +} + +/** + * Returns the member's state, `undefined` for a vanished PID or another session, or `UNREADABLE_STATE` + * when the entry cannot be read (for example `hidepid` or `EIO`) and so might hide a live member. + */ +async function readSessionMemberStateAsync( + pid: string, + groupId: number, + procfs: ILinuxProcfsReader +): Promise { + let stat: string; + try { + stat = await procfs.readStatAsync(pid); + } catch (error) { + return isProcessGoneError(error) ? undefined : UNREADABLE_STATE; + } + // Format: "pid (comm) state ppid pgrp session ..."; comm may contain spaces and parentheses. + const fields: string[] = stat.slice(stat.lastIndexOf(')') + 2).split(' '); + return Number(fields[PROC_STAT_SESSION_INDEX]) === groupId ? fields[PROC_STAT_STATE_INDEX] : undefined; +} + +function isProcessGoneError(error: unknown): boolean { + return ( + typeof error === 'object' && + error !== null && + 'code' in error && + (error.code === 'ENOENT' || error.code === 'ESRCH') + ); +} + +function readSessionStatesFromPsAsync(groupId: number, timeoutMs: number): Promise { return new Promise((resolve, reject) => { // detached=true creates a new process group and session with the child's PID. execFile( diff --git a/libraries/rush-daemon/src/test/LinuxProcessGroupExit.test.ts b/libraries/rush-daemon/src/test/LinuxProcessGroupExit.test.ts index 5d4af13f54..237854b90d 100644 --- a/libraries/rush-daemon/src/test/LinuxProcessGroupExit.test.ts +++ b/libraries/rush-daemon/src/test/LinuxProcessGroupExit.test.ts @@ -8,13 +8,17 @@ jest.mock('node:child_process', () => ({ import * as childProcess from 'node:child_process'; -import { waitForLinuxProcessGroupExitAsync } from '../LinuxProcessGroupExit'; +import { type ILinuxProcfsReader, waitForLinuxProcessGroupExitAsync } from '../LinuxProcessGroupExit'; const GROUP_ID: number = 123_456_789; const execFileMock = jest.mocked(childProcess.execFile); const gone = (): never => { throw Object.assign(new Error('No such process group'), { code: 'ESRCH' }); }; +const NO_PROCFS: ILinuxProcfsReader = { + listEntriesAsync: () => Promise.reject(Object.assign(new Error('No procfs'), { code: 'ENOENT' })), + readStatAsync: () => Promise.reject(new Error('Unexpected procfs read')) +}; function reportPs( args: Parameters, @@ -28,7 +32,10 @@ function reportPs( return new childProcess.ChildProcess(); } -describe('Linux subprocess group completion', () => { +describe('Linux subprocess group completion (ps fallback without procfs)', () => { + const waitAsync = (groupId: number, timeoutMs?: number): Promise => + waitForLinuxProcessGroupExitAsync(groupId, timeoutMs, NO_PROCFS); + beforeEach(() => { execFileMock.mockReset(); jest.spyOn(process, 'kill').mockReturnValue(true); @@ -40,14 +47,14 @@ describe('Linux subprocess group completion', () => { it('avoids process inspection when the captured group has disappeared', async () => { jest.mocked(process.kill).mockImplementation(gone); - await waitForLinuxProcessGroupExitAsync(GROUP_ID); + await waitAsync(GROUP_ID); expect(execFileMock).not.toHaveBeenCalled(); }); it('waits for live members rather than treating signal delivery as completion', async () => { let queries: number = 0; execFileMock.mockImplementation((...args) => reportPs(args, ++queries === 1 ? 'R\nZ\n' : 'Z\nZ\n')); - await waitForLinuxProcessGroupExitAsync(GROUP_ID); + await waitAsync(GROUP_ID); expect(queries).toBe(2); expect( jest.mocked(process.kill).mock.calls.every(([pid, signal]) => pid === -GROUP_ID && signal === 0) @@ -59,19 +66,19 @@ describe('Linux subprocess group completion', () => { execFileMock.mockImplementation((...args) => reportPs(args, '', Object.assign(new Error('No matching processes'), { code: 1 })) ); - await waitForLinuxProcessGroupExitAsync(GROUP_ID); + await waitAsync(GROUP_ID); expect(process.kill).toHaveBeenCalledTimes(2); }); it('surfaces inspection failures instead of completing cleanup', async () => { const failure = Object.assign(new Error('Cannot execute ps'), { code: 'ENOENT' }); execFileMock.mockImplementation((...args) => reportPs(args, '', failure)); - await expect(waitForLinuxProcessGroupExitAsync(GROUP_ID)).rejects.toBe(failure); + await expect(waitAsync(GROUP_ID)).rejects.toBe(failure); }); it('rejects inspection diagnostics rather than trusting incomplete process state', async () => { execFileMock.mockImplementation((...args) => reportPs(args, 'Z\n', undefined, 'inspection warning')); - await expect(waitForLinuxProcessGroupExitAsync(GROUP_ID)).rejects.toThrow('inspection warning'); + await expect(waitAsync(GROUP_ID)).rejects.toThrow('inspection warning'); }); it('does not mistake permission denial for a released group', async () => { @@ -79,16 +86,171 @@ describe('Linux subprocess group completion', () => { jest.mocked(process.kill).mockImplementation(() => { throw failure; }); - await expect(waitForLinuxProcessGroupExitAsync(GROUP_ID)).rejects.toBe(failure); + await expect(waitAsync(GROUP_ID)).rejects.toBe(failure); }); it('bounds the wait for members that remain live', async () => { execFileMock.mockImplementation((...args) => reportPs(args, 'S\n')); - await expect(waitForLinuxProcessGroupExitAsync(GROUP_ID, 25)).rejects.toThrow('did not exit within 25ms'); + await expect(waitAsync(GROUP_ID, 25)).rejects.toThrow('did not exit within 25ms'); }); it.each([0, -1, 1.5, process.pid])('rejects invalid or unowned group ID %s', async (pid) => { - await expect(waitForLinuxProcessGroupExitAsync(pid)).rejects.toThrow('owned child process group'); + await expect(waitAsync(pid)).rejects.toThrow('owned child process group'); expect(process.kill).not.toHaveBeenCalled(); }); }); + +describe('Linux subprocess group completion (procfs)', () => { + const procStat = (pid: number, state: string, session: number): string => + `${pid} (sh -c (x) y) ${state} 1 ${session} ${session} 0 -1 4194560`; + let table: Map; + const fakeProcfs: ILinuxProcfsReader = { + listEntriesAsync: async () => ['self', 'sys', ...table.keys()], + readStatAsync: async (pid: string) => { + const stat: string | undefined = table.get(pid); + if (stat === undefined) throw Object.assign(new Error('gone'), { code: 'ENOENT' }); + return stat; + } + }; + const waitAsync = (timeoutMs?: number): Promise => + waitForLinuxProcessGroupExitAsync(GROUP_ID, timeoutMs, fakeProcfs); + + beforeEach(() => { + execFileMock.mockReset(); + jest.spyOn(process, 'kill').mockReturnValue(true); + table = new Map(); + }); + afterEach(() => { + jest.restoreAllMocks(); + }); + + it('completes from procfs without executing ps when only zombies remain', async () => { + table = new Map([ + ['10', procStat(10, 'Z', GROUP_ID)], + ['11', procStat(11, 'S', 42)] + ]); + await waitAsync(); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it('ignores members of other sessions whose command names look like session fields', async () => { + table = new Map([ + ['10', procStat(10, 'Z', GROUP_ID)], + ['11', `11 (x) S 1 ${GROUP_ID} ${GROUP_ID}) S 1 42 42 0 -1 4194560`] + ]); + await waitAsync(); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it('ignores members that exit between listing and reading', async () => { + const reads: string[] = []; + const racingProcfs: ILinuxProcfsReader = { + listEntriesAsync: async () => ['10', '11'], + readStatAsync: async (pid: string) => { + reads.push(pid); + if (pid === '11') throw Object.assign(new Error('gone'), { code: 'ENOENT' }); + return procStat(10, 'Z', GROUP_ID); + } + }; + await waitForLinuxProcessGroupExitAsync(GROUP_ID, undefined, racingProcfs); + expect(reads.sort()).toEqual(['10', '11']); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it('waits for live session members found in procfs', async () => { + table = new Map([['10', procStat(10, 'R', GROUP_ID)]]); + setTimeout(() => { + table = new Map([['10', procStat(10, 'Z', GROUP_ID)]]); + }, 30); + await waitAsync(); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it('bounds the wait for live procfs members', async () => { + table = new Map([['10', procStat(10, 'S', GROUP_ID)]]); + await expect(waitAsync(25)).rejects.toThrow('did not exit within 25ms'); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it.each(['EACCES', 'EIO'])( + 'does not trust a zombie when another entry is unreadable (%s); falls back to ps', + async (code) => { + table = new Map([['10', procStat(10, 'Z', GROUP_ID)]]); + const unreadableProcfs: ILinuxProcfsReader = { + listEntriesAsync: async () => ['10', '11'], + readStatAsync: async (pid: string) => { + if (pid === '11') throw Object.assign(new Error('unreadable'), { code }); + return fakeProcfs.readStatAsync(pid); + } + }; + let queries: number = 0; + execFileMock.mockImplementation((...args) => reportPs(args, ++queries === 1 ? 'S\nZ\n' : 'Z\nZ\n')); + await waitForLinuxProcessGroupExitAsync(GROUP_ID, undefined, unreadableProcfs); + expect(queries).toBe(2); + } + ); + + it('surfaces the ps failure when an unreadable entry forces the fallback and ps is missing', async () => { + const unreadableProcfs: ILinuxProcfsReader = { + listEntriesAsync: async () => ['10', '11'], + readStatAsync: async (pid: string) => { + if (pid === '11') throw Object.assign(new Error('unreadable'), { code: 'EACCES' }); + return procStat(10, 'Z', GROUP_ID); + } + }; + const failure = Object.assign(new Error('spawn ps ENOENT'), { code: 'ENOENT' }); + execFileMock.mockImplementation((...args) => reportPs(args, '', failure)); + await expect(waitForLinuxProcessGroupExitAsync(GROUP_ID, undefined, unreadableProcfs)).rejects.toBe( + failure + ); + }); + + it('treats ESRCH from a stat read as a vanished process', async () => { + const racingProcfs: ILinuxProcfsReader = { + listEntriesAsync: async () => ['10', '11'], + readStatAsync: async (pid: string) => { + if (pid === '11') throw Object.assign(new Error('gone'), { code: 'ESRCH' }); + return procStat(10, 'Z', GROUP_ID); + } + }; + await waitForLinuxProcessGroupExitAsync(GROUP_ID, undefined, racingProcfs); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it('bounds concurrent stat reads and stops scanning after a live member', async () => { + const pids: string[] = Array.from({ length: 200 }, (unused, index) => String(index + 1)); + let inFlight: number = 0; + let maxInFlight: number = 0; + let reads: number = 0; + let live: boolean = true; + const largeProcfs: ILinuxProcfsReader = { + listEntriesAsync: async () => pids, + readStatAsync: async (pid: string) => { + reads++; + maxInFlight = Math.max(maxInFlight, ++inFlight); + await new Promise((resolve) => setImmediate(resolve)); + inFlight--; + return pid === '1' ? procStat(1, live ? 'R' : 'Z', GROUP_ID) : procStat(Number(pid), 'S', 42); + } + }; + setTimeout(() => { + live = false; + }, 30); + await waitForLinuxProcessGroupExitAsync(GROUP_ID, undefined, largeProcfs); + expect(maxInFlight).toBeLessThanOrEqual(32); + const scans: number = jest.mocked(process.kill).mock.calls.length; + // Each scan while the member is live stops after the first batch; only the final scan reads everything. + expect(scans).toBeGreaterThan(1); + expect(reads).toBeLessThanOrEqual((scans - 1) * 32 + pids.length); + expect(execFileMock).not.toHaveBeenCalled(); + }); + + it('does not depend on ps being installed while procfs is readable', async () => { + execFileMock.mockImplementation((...args) => + reportPs(args, '', Object.assign(new Error('spawn ps ENOENT'), { code: 'ENOENT' })) + ); + table = new Map([['10', procStat(10, 'Z', GROUP_ID)]]); + await waitAsync(); + expect(execFileMock).not.toHaveBeenCalled(); + }); +}); diff --git a/libraries/rush-daemon/src/test/NativeMutationCleanupFailure.test.ts b/libraries/rush-daemon/src/test/NativeMutationCleanupFailure.test.ts index 873a07b5f9..c64a429c1b 100644 --- a/libraries/rush-daemon/src/test/NativeMutationCleanupFailure.test.ts +++ b/libraries/rush-daemon/src/test/NativeMutationCleanupFailure.test.ts @@ -68,7 +68,11 @@ jest.setTimeout(30_000); .spyOn(linuxProcessGroupExit, 'waitForLinuxProcessGroupExitAsync') .mockImplementation(async (pid) => { workerPid = pid; - await originalWait(pid, 25); + // Hide procfs so the injected `ps` inspection outcome is what the join observes. + await originalWait(pid, 25, { + listEntriesAsync: () => Promise.reject(new Error('procfs hidden by test')), + readStatAsync: () => Promise.reject(new Error('procfs hidden by test')) + }); }); jest .spyOn(process, 'kill') From 061f27d30026e8f1c6977b955e9945a7f8bd01db Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:18:25 -0700 Subject: [PATCH 010/265] [rush-lib] Skip build cache writes when an operation's inputs changed during execution (#6086) * [rush-lib] Skip build cache writes when an operation's inputs changed during execution The cache key is derived from the iteration's inputs snapshot, but the outputs were written under that key without re-verifying the inputs. Record a stat signature of each cacheable operation's tracked input files right after the snapshot and refuse the cache write if it changed. Fixes #6073 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-lib] Also detect new input files, block downstream cache writes, and resolve absolute input paths Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- ...build-cache-poisoning-race_2026-09-24.json | 9 ++ .../operations/CacheableOperationPlugin.ts | 69 +++++++- .../operations/InputFilesStatSignature.ts | 151 ++++++++++++++++++ .../test/InputFilesStatSignature.test.ts | 147 +++++++++++++++++ 4 files changed, 374 insertions(+), 2 deletions(-) create mode 100644 common/changes/@microsoft/rush/fix-build-cache-poisoning-race_2026-09-24.json create mode 100644 libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts diff --git a/common/changes/@microsoft/rush/fix-build-cache-poisoning-race_2026-09-24.json b/common/changes/@microsoft/rush/fix-build-cache-poisoning-race_2026-09-24.json new file mode 100644 index 0000000000..87048f4639 --- /dev/null +++ b/common/changes/@microsoft/rush/fix-build-cache-poisoning-race_2026-09-24.json @@ -0,0 +1,9 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Fix a build cache poisoning race: skip writing a build cache entry when an operation's tracked input files changed while it was executing.", + "type": "patch" + } + ] +} diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index c915abf8ca..6213db3ede 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -2,8 +2,9 @@ // See LICENSE in the project root for license information. import * as crypto from 'node:crypto'; +import * as path from 'node:path'; -import { InternalError, NewlineKind, Sort } from '@rushstack/node-core-library'; +import { InternalError, NewlineKind, Sort, Executable } from '@rushstack/node-core-library'; import { CollatedTerminal, type CollatedWriter } from '@rushstack/stream-collator'; import { DiscardStdoutTransform, @@ -28,6 +29,13 @@ import { import type { CobuildConfiguration } from '../../api/CobuildConfiguration'; import { DisjointSet } from '../cobuild/DisjointSet'; import { PeriodicCallback } from './PeriodicCallback'; +import { + captureInputFilesState, + haveInputFilesChanged, + hasUntrackedGitFiles, + type IInputFilesState +} from './InputFilesStatSignature'; +import { EnvironmentConfiguration } from '../../api/EnvironmentConfiguration'; import { NullTerminalProvider } from '../../utilities/NullTerminalProvider'; import type { Operation } from './Operation'; import type { IOperationRunnerContext } from './IOperationRunner'; @@ -70,6 +78,11 @@ export interface IOperationBuildCacheContext { periodicCallback: PeriodicCallback; cacheRestored: boolean; isCacheReadAttempted: boolean; + + // The on-disk state of the tracked input files whose hashes produced the cache key, captured right after + // the iteration's inputs snapshot. Used to refuse cache writes if the inputs changed while the operation + // was executing. + inputFilesState?: IInputFilesState; } export interface ICacheableOperationPluginOptions { @@ -102,10 +115,33 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { readonly #options: ICacheableOperationPluginOptions; + #gitPathResolved: boolean = false; + #gitPath: string | undefined; + public constructor(options: ICacheableOperationPluginOptions) { this.#options = options; } + #isNewInput( + newEntryPaths: ReadonlyArray, + rootDirectory: string, + projectFolder: string, + outputFolderNames: ReadonlyArray + ): boolean { + if (!this.#gitPathResolved) { + this.#gitPath = EnvironmentConfiguration.gitBinaryPath || Executable.tryResolve('git'); + this.#gitPathResolved = true; + } + if (!this.#gitPath) { + // Without Git we cannot tell whether the new entries are ignored, so assume they are inputs. + return true; + } + const outputFolderPaths: string[] = outputFolderNames.map((folderName: string) => + path.resolve(projectFolder, folderName) + ); + return hasUntrackedGitFiles(this.#gitPath, rootDirectory, newEntryPaths, outputFolderPaths); + } + public apply(hooks: PhasedCommandHooks): void { const { allowWarningsInSuccessfulBuild, @@ -184,6 +220,11 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { disjointSet?.add(operation); + const inputFilesState: IInputFilesState | undefined = + cacheWriteEnabled && !cacheDisabledReason && record.enabled + ? captureInputFilesState(inputsSnapshot.rootDirectory, fileHashes.keys()) + : undefined; + const buildCacheContext: IOperationBuildCacheContext = { // Supports cache writes by default for initial operations. // Don't write during watch runs for performance reasons (and to avoid flooding the cache) @@ -200,7 +241,8 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { interval: PERIODIC_CALLBACK_INTERVAL_IN_SECONDS * 1000 }), cacheRestored: false, - isCacheReadAttempted: false + isCacheReadAttempted: false, + inputFilesState }; // Upstream runners may mutate the property of build cache context for downstream runners this.#buildCacheContextByOperation.set(operation, buildCacheContext); @@ -545,6 +587,29 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { if (!setCacheEntryPromise && taskIsSuccessful && isCacheWriteAllowed && operationBuildCache) { setCacheEntryPromise = () => operationBuildCache.trySetCacheEntryAsync(buildCacheTerminal); } + const { inputFilesState } = buildCacheContext; + if ( + !cacheRestored && + isCacheWriteAllowed && + inputFilesState && + haveInputFilesChanged(inputFilesState, (newEntryPaths: ReadonlyArray) => + this.#isNewInput( + newEntryPaths, + inputFilesState.rootDirectory, + project.projectFolder, + buildCacheContext.outputFolderNames + ) + ) + ) { + // The cache key was derived from the iteration's inputs snapshot. Storing outputs produced from + // edited inputs under that key would poison the cache for every consumer of the entry. + // Consumers' cache keys also embed this operation's pre-edit state, so block their writes too. + buildCacheTerminal.writeLine( + 'Input files changed while this operation was executing; not writing a build cache entry.' + ); + buildCacheContext.isCacheWriteAllowed = false; + setCacheEntryPromise = undefined; + } if (!cacheRestored) { const cacheWriteSuccess: boolean | undefined = await setCacheEntryPromise?.(); await setCompletedStatePromiseFunction?.(); diff --git a/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts b/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts new file mode 100644 index 0000000000..aef3eaff54 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts @@ -0,0 +1,151 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as crypto from 'node:crypto'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { Executable } from '@rushstack/node-core-library'; + +/** + * The on-disk state of an operation's tracked input files, captured right after the inputs snapshot + * (from which the operation's build cache key is derived) was taken. + */ +export interface IInputFilesState { + /** + * The repository root that relative input file paths were resolved against. + */ + readonly rootDirectory: string; + /** + * Absolute paths of the tracked input files. + */ + readonly filePaths: ReadonlyArray; + /** + * Signature of the size, modification time, and inode of each tracked input file. + */ + readonly statSignature: string; + /** + * For each folder inside the repository that contains a tracked input file, the names of its entries. + * Used to detect files (or folders) that were created after the snapshot was taken. + */ + readonly folderEntries: ReadonlyMap>; +} + +/** + * Given the absolute paths of entries that appeared in input folders after the snapshot was taken, + * returns true if any of them is a potential input of the operation (e.g. an untracked, non-ignored file). + */ +export type IsNewInputCallback = (newEntryPaths: ReadonlyArray) => boolean; + +/** + * Computes a cheap signature of the on-disk identity (size, mtime, inode) of the specified files. + * Missing files are included in the signature, so deleting or creating a listed file also changes it. + */ +export function getInputFilesStatSignature(filePaths: Iterable): string { + const hasher: crypto.Hash = crypto.createHash('sha1'); + for (const filePath of filePaths) { + const stats: fs.BigIntStats | undefined = fs.statSync(filePath, { bigint: true, throwIfNoEntry: false }); + if (stats) { + hasher.update(`${filePath}\0${stats.size}\0${stats.mtimeNs}\0${stats.ino}\n`); + } else { + hasher.update(`${filePath}\0missing\n`); + } + } + return hasher.digest('hex'); +} + +function tryReadFolderEntries(folderPath: string): Set | undefined { + try { + return new Set(fs.readdirSync(folderPath)); + } catch { + return undefined; + } +} + +/** + * Captures the on-disk state of an operation's tracked input files. + * + * @param rootDirectory - The repository root that relative input file paths are resolved against + * @param inputFilePaths - The tracked input file paths. Relative paths are resolved against `rootDirectory`; + * absolute paths (e.g. `dependsOnAdditionalFiles` outside of the repository) are stat'ed but their folders + * are not watched for new entries. + */ +export function captureInputFilesState( + rootDirectory: string, + inputFilePaths: Iterable +): IInputFilesState { + const filePaths: string[] = []; + const folderEntries: Map> = new Map(); + for (const inputFilePath of inputFilePaths) { + const absolutePath: string = path.resolve(rootDirectory, inputFilePath); + filePaths.push(absolutePath); + if (!path.isAbsolute(inputFilePath)) { + const folderPath: string = path.dirname(absolutePath); + if (!folderEntries.has(folderPath)) { + folderEntries.set(folderPath, tryReadFolderEntries(folderPath) ?? new Set()); + } + } + } + return { rootDirectory, filePaths, statSignature: getInputFilesStatSignature(filePaths), folderEntries }; +} + +/** + * Returns the absolute paths of entries that exist now but did not exist when the folder entries were captured. + */ +export function getNewFolderEntries(folderEntries: ReadonlyMap>): string[] { + const newEntryPaths: string[] = []; + for (const [folderPath, originalEntries] of folderEntries) { + const currentEntries: Set | undefined = tryReadFolderEntries(folderPath); + if (currentEntries) { + for (const entry of currentEntries) { + if (!originalEntries.has(entry)) { + newEntryPaths.push(path.join(folderPath, entry)); + } + } + } + } + return newEntryPaths; +} + +/** + * Returns true if any of the operation's tracked input files was modified, deleted, or replaced, or if a + * potential new input file was created in one of the input folders, since the state was captured. + */ +export function haveInputFilesChanged(state: IInputFilesState, isNewInput: IsNewInputCallback): boolean { + if (getInputFilesStatSignature(state.filePaths) !== state.statSignature) { + return true; + } + const newEntryPaths: string[] = getNewFolderEntries(state.folderEntries); + return newEntryPaths.length > 0 && isNewInput(newEntryPaths); +} + +function toGitPathspec(rootDirectory: string, absolutePath: string): string { + return path.relative(rootDirectory, absolutePath).split(path.sep).join('/'); +} + +/** + * Uses Git to determine whether any of the specified paths is, or contains, an untracked file that is not + * ignored by `.gitignore`, excluding the specified folders (typically the operation's output folders). + * If Git fails, conservatively returns true. + */ +export function hasUntrackedGitFiles( + gitPath: string, + rootDirectory: string, + candidatePaths: ReadonlyArray, + excludedFolderPaths: ReadonlyArray +): boolean { + const args: string[] = ['ls-files', '--others', '--exclude-standard', '-z', '--']; + for (const candidatePath of candidatePaths) { + args.push(`:(literal)${toGitPathspec(rootDirectory, candidatePath)}`); + } + for (const excludedFolderPath of excludedFolderPaths) { + args.push(`:(exclude,literal)${toGitPathspec(rootDirectory, excludedFolderPath)}`); + } + const result: ReturnType = Executable.spawnSync(gitPath, args, { + currentWorkingDirectory: rootDirectory + }); + if (result.status !== 0) { + return true; + } + return result.stdout.length > 0; +} \ No newline at end of file diff --git a/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts b/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts new file mode 100644 index 0000000000..0e0839c269 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts @@ -0,0 +1,147 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as child_process from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + captureInputFilesState, + getNewFolderEntries, + hasUntrackedGitFiles, + haveInputFilesChanged, + type IInputFilesState +} from '../InputFilesStatSignature'; + +describe('InputFilesStatSignature', () => { + let tempFolder: string; + let srcFolder: string; + let fileA: string; + let fileB: string; + let noNewInputs: jest.Mock]>; + + function capture(...absolutePaths: string[]): IInputFilesState { + return captureInputFilesState( + tempFolder, + absolutePaths.map((filePath: string) => path.relative(tempFolder, filePath)) + ); + } + + beforeEach(() => { + tempFolder = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-input-stat-'))); + srcFolder = path.join(tempFolder, 'src'); + fs.mkdirSync(srcFolder); + fileA = path.join(srcFolder, 'a.ts'); + fileB = path.join(srcFolder, 'b.ts'); + fs.writeFileSync(fileA, 'export const a = 1;'); + fs.writeFileSync(fileB, 'export const b = 1;'); + noNewInputs = jest.fn().mockReturnValue(false); + }); + + afterEach(() => { + fs.rmSync(tempFolder, { recursive: true, force: true }); + }); + + describe(haveInputFilesChanged.name, () => { + it('reports no change when the input files are unchanged', () => { + const state: IInputFilesState = capture(fileA, fileB); + expect(state.filePaths).toEqual([fileA, fileB]); + expect(haveInputFilesChanged(state, noNewInputs)).toBe(false); + expect(noNewInputs).not.toHaveBeenCalled(); + }); + + it('detects a modified input file', () => { + const state: IInputFilesState = capture(fileA, fileB); + fs.writeFileSync(fileB, 'export const b = 2; // edited during the build'); + expect(haveInputFilesChanged(state, noNewInputs)).toBe(true); + }); + + it('detects a same-size edit with a different modification time', () => { + const state: IInputFilesState = capture(fileA); + fs.writeFileSync(fileA, 'export const a = 2;'); + const future: Date = new Date(Date.now() + 60 * 1000); + fs.utimesSync(fileA, future, future); + expect(haveInputFilesChanged(state, noNewInputs)).toBe(true); + }); + + it('detects a deleted input file', () => { + const state: IInputFilesState = capture(fileA, fileB); + fs.unlinkSync(fileA); + expect(haveInputFilesChanged(state, noNewInputs)).toBe(true); + }); + + it('asks whether files created in an input folder are inputs', () => { + const state: IInputFilesState = capture(fileA, fileB); + const fileC: string = path.join(srcFolder, 'c.ts'); + fs.writeFileSync(fileC, 'export const c = 1;'); + + expect(haveInputFilesChanged(state, noNewInputs)).toBe(false); + expect(noNewInputs).toHaveBeenCalledWith([fileC]); + + expect(haveInputFilesChanged(state, () => true)).toBe(true); + }); + + it('reports a folder created in an input folder', () => { + const state: IInputFilesState = capture(fileA); + fs.mkdirSync(path.join(srcFolder, 'nested')); + fs.writeFileSync(path.join(srcFolder, 'nested', 'd.ts'), 'export const d = 1;'); + expect(getNewFolderEntries(state.folderEntries)).toEqual([path.join(srcFolder, 'nested')]); + }); + + it('ignores files created outside of the input folders', () => { + const state: IInputFilesState = capture(fileA); + fs.writeFileSync(path.join(tempFolder, 'unrelated.txt'), 'not an input'); + expect(haveInputFilesChanged(state, noNewInputs)).toBe(false); + expect(noNewInputs).not.toHaveBeenCalled(); + }); + + it('resolves absolute input paths as-is and does not watch their folders', () => { + const state: IInputFilesState = captureInputFilesState(path.join(tempFolder, 'other-root'), [fileA]); + expect(state.filePaths).toEqual([fileA]); + expect(state.folderEntries.size).toBe(0); + fs.writeFileSync(fileA, 'export const a = 3; // edited during the build'); + expect(haveInputFilesChanged(state, noNewInputs)).toBe(true); + }); + }); + + describe(hasUntrackedGitFiles.name, () => { + const gitPath: string = 'git'; + + function git(...args: string[]): void { + child_process.execFileSync(gitPath, args, { cwd: tempFolder, stdio: 'ignore' }); + } + + beforeEach(() => { + git('init', '-q'); + fs.writeFileSync(path.join(tempFolder, '.gitignore'), 'temp/\n*.log\n'); + git('add', '-A'); + }); + + it('returns false for ignored files and excluded folders', () => { + fs.mkdirSync(path.join(srcFolder, 'temp')); + fs.writeFileSync(path.join(srcFolder, 'temp', 'x.ts'), ''); + fs.writeFileSync(path.join(srcFolder, 'build.log'), ''); + fs.mkdirSync(path.join(srcFolder, 'lib')); + fs.writeFileSync(path.join(srcFolder, 'lib', 'a.js'), ''); + + expect( + hasUntrackedGitFiles( + gitPath, + tempFolder, + [path.join(srcFolder, 'temp'), path.join(srcFolder, 'build.log'), path.join(srcFolder, 'lib')], + [path.join(srcFolder, 'lib')] + ) + ).toBe(false); + }); + + it('returns true for new untracked files and folders that are not ignored', () => { + fs.writeFileSync(path.join(srcFolder, 'c.ts'), ''); + fs.mkdirSync(path.join(srcFolder, 'nested')); + fs.writeFileSync(path.join(srcFolder, 'nested', 'd.ts'), ''); + + expect(hasUntrackedGitFiles(gitPath, tempFolder, [path.join(srcFolder, 'c.ts')], [])).toBe(true); + expect(hasUntrackedGitFiles(gitPath, tempFolder, [path.join(srcFolder, 'nested')], [])).toBe(true); + }); + }); +}); \ No newline at end of file From 560efefd6202d08c97d28a7194634096f41f7539 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:18:44 -0700 Subject: [PATCH 011/265] [rush-daemon] Only count resource-holding projects toward warmSetMaxProjects (#6081) * [rush-daemon] Only count resource-holding projects toward warmSetMaxProjects The warm-set project cap deleted retained results of resource-free (shell/null runner) projects beyond rank 20, so every no-op build of a >20-project workspace re-restored those projects from the build cache. The cap now only counts and evicts projects holding an active runner or watcher, and equal-recency ties keep the explicitly requested target. Fixes #6049 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Update watch-policy test for requested-target retention order Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Count closable runners without isActive as warm resource holders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- apps/rush-cli-client/README.md | 4 +- .../fix-warmset-cap_2026-09-24-01-00.json | 10 +++ .../fix-warmset-cap_2026-09-24-01-00.json | 10 +++ .../fix-warmset-cap_2026-09-24-01-00.json | 10 +++ libraries/rush-daemon/README.md | 10 ++- libraries/rush-daemon/src/WarmSetRanking.ts | 4 + libraries/rush-daemon/src/WorkspaceWarmSet.ts | 36 ++++++-- .../src/test/WarmSetRanking.test.ts | 11 +++ .../src/test/WarmSetTestFixture.ts | 24 +++++ .../src/test/WorkspaceWarmSet.test.ts | 87 +++++++++++++++++++ .../src/test/WorkspaceWatchPolicy.test.ts | 3 +- .../rush-lib/src/api/DaemonConfiguration.ts | 5 +- .../rush-lib/src/schemas/rush.schema.json | 2 +- 13 files changed, 202 insertions(+), 14 deletions(-) create mode 100644 common/changes/@microsoft/rush/fix-warmset-cap_2026-09-24-01-00.json create mode 100644 common/changes/@rushstack/rush-cli-client/fix-warmset-cap_2026-09-24-01-00.json create mode 100644 common/changes/@rushstack/rush-daemon/fix-warmset-cap_2026-09-24-01-00.json diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index dda32fb6ef..66d54e7f3b 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -211,8 +211,8 @@ keys and unknown `RUSH_DAEMON*` variables fail validation. | `watch` | `RUSH_DAEMON_WATCH` | false | Persistent host observation of requested warm projects; false keeps root/config guards only. Never schedules builds | | `usePersistentIpcRunners` | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | false | Enables explicit per-operation `daemonIpc` Node launchers for unsharded incremental daemon builds | | `warmIdleTimeoutSeconds` | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | 300 | Idle runner, project-watcher and retained-result eviction | -| `warmMemoryBudgetMB` | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | 512 | Best-effort sampled RSS budget in MiB, not a hard ceiling | -| `warmSetMaxProjects` | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | 20 | Best-effort retained-project limit; never trims requested execution | +| `warmMemoryBudgetMB` | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | 512 | Best-effort sampled RSS budget in MiB, not a hard ceiling. Compared against whole-daemon RSS plus measured child RSS, so keep it above the daemon baseline (~130-190 MiB) | +| `warmSetMaxProjects` | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | 20 | Best-effort limit on projects holding warm resources (active runners, watchers); retained results of resource-free projects do not count. Never trims requested execution | | `autoWarmByTelemetry` | `RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY` | false | Measured retention ranking with conservative LRU fallback; no speculative scripts | For genuine persistent Node execution, enable `usePersistentIpcRunners` and add diff --git a/common/changes/@microsoft/rush/fix-warmset-cap_2026-09-24-01-00.json b/common/changes/@microsoft/rush/fix-warmset-cap_2026-09-24-01-00.json new file mode 100644 index 0000000000..95f657e41d --- /dev/null +++ b/common/changes/@microsoft/rush/fix-warmset-cap_2026-09-24-01-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Document that the daemon warmSetMaxProjects limit only counts projects holding warm resources (active runners or file watchers).", + "type": "patch" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-cli-client/fix-warmset-cap_2026-09-24-01-00.json b/common/changes/@rushstack/rush-cli-client/fix-warmset-cap_2026-09-24-01-00.json new file mode 100644 index 0000000000..9eec39ac08 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/fix-warmset-cap_2026-09-24-01-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Document the warmSetMaxProjects and warmMemoryBudgetMB daemon policy semantics.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-daemon/fix-warmset-cap_2026-09-24-01-00.json b/common/changes/@rushstack/rush-daemon/fix-warmset-cap_2026-09-24-01-00.json new file mode 100644 index 0000000000..98fe35f1aa --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/fix-warmset-cap_2026-09-24-01-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Fix warmSetMaxProjects evicting the retained results of resource-free projects, which made every no-op build of a workspace with more than 20 projects re-restore the excess projects from the build cache. The cap now only counts and releases projects that hold a live runner or watcher, and ties keep the explicitly requested target over its dependencies.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 7979146a41..4ee695eba8 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -300,13 +300,15 @@ preserves its records and diagnostics without failing an otherwise successful bu | --- | --- | | `watch` | Retains host observation of requested warm projects between requests when true. False (the default) keeps root/config guards only. Never schedules builds. | | `warmIdleTimeoutSeconds` | Expires unused project runners, watchers and retained results after requests finish. Unchanged requests refresh recency too. | -| `warmSetMaxProjects` | Retains the highest-ranked idle projects within the limit; executing/prepared and explicitly protected work is exempt. | -| `warmMemoryBudgetMB` | Attempts idle eviction under sampled daemon-plus-measured-child RSS pressure. Never treats cache files as memory or claims a hard RSS ceiling. | +| `warmSetMaxProjects` | Limits the projects that hold warm **resources** (an active runner such as a persistent IPC child, or a `watch: true` file watcher). The lowest-ranked holders are released (runners closed, watchers removed, records deleted); executing/prepared and explicitly protected work is exempt. Projects whose only retained state is operation results from resource-free (shell/null) runners neither count toward nor are evicted for this limit, so no-op re-requests of large workspaces stay skipped. | +| `warmMemoryBudgetMB` | Attempts idle eviction under sampled daemon-plus-measured-child RSS pressure. The comparison uses the **whole daemon process RSS** (graph, Node heap and retained records, typically 130-190 MiB) plus measured child RSS, so a budget below the daemon's baseline evicts every idle project on each pass and disables warm skipping. Never treats cache files as memory or claims a hard RSS ceiling. | | `autoWarmByTelemetry` | Promotes already-requested high-value work instead of pure LRU. Never schedules or executes speculative scripts. | One deterministic best-first comparator is shared by retention and reverse-order eviction. With complete -measurements it uses `(timeSavedMs * requestFrequency) / residentMemoryBytes`, then recency, then ordinal project -name. Measured entries precede the missing-data bucket; that bucket uses LRU and the same name tie-break. +measurements it uses `(timeSavedMs * requestFrequency) / residentMemoryBytes`, then recency, then whether the +project owned an explicitly requested target (an enabled operation with no enabled consumer, so `--to x` keeps +`x` over its same-request dependencies), then ordinal project name. Measured entries precede the missing-data +bucket; that bucket uses LRU and the same tie-breaks. Without telemetry mode the entire order is LRU. Savings compare actual cold and reused execution stopwatches (or native non-cached duration versus cache-restoration duration); no startup cost or RSS is invented. `operation-graph`'s existing `WatchLoop` now reports its own measured RSS in an optional IPC completion field. diff --git a/libraries/rush-daemon/src/WarmSetRanking.ts b/libraries/rush-daemon/src/WarmSetRanking.ts index cc50659d62..e4b069d970 100644 --- a/libraries/rush-daemon/src/WarmSetRanking.ts +++ b/libraries/rush-daemon/src/WarmSetRanking.ts @@ -8,6 +8,8 @@ export interface IWarmSetRank { readonly frequency: number; readonly timeSavedMs: number | undefined; readonly residentMemoryBytes: number | undefined; + /** The project owned a selection root (an enabled operation with no enabled consumer) when last requested. */ + readonly requestedTarget?: boolean; } export function getWarmSetScore(entry: IWarmSetRank): number | undefined { @@ -38,5 +40,7 @@ export function compareWarmSetRanks(a: IWarmSetRank, b: IWarmSetRank, telemetry: if (aScore !== undefined && bScore !== undefined && aScore !== bScore) return bScore - aScore; } if (a.lastUsed !== b.lastUsed) return b.lastUsed - a.lastUsed; + // A request stamps its whole closure with one timestamp; keep the explicit target over its dependencies. + if (!a.requestedTarget !== !b.requestedTarget) return a.requestedTarget ? -1 : 1; return a.key === b.key ? 0 : a.key < b.key ? -1 : 1; } diff --git a/libraries/rush-daemon/src/WorkspaceWarmSet.ts b/libraries/rush-daemon/src/WorkspaceWarmSet.ts index 17461ea2da..3b7e5aa5f5 100644 --- a/libraries/rush-daemon/src/WorkspaceWarmSet.ts +++ b/libraries/rush-daemon/src/WorkspaceWarmSet.ts @@ -53,6 +53,7 @@ interface IProjectHistory { readonly operations: Operation[]; lastUsed: number; frequency: number; + requestedTarget: boolean; } interface IOperationTiming { @@ -63,6 +64,8 @@ interface IOperationTiming { interface IWarmProject extends IWarmSetRank { readonly operations: ReadonlyArray; readonly protected: boolean; + /** Owns a live runner or a file watcher. Only these count toward, and are evicted for, `warmSetMaxProjects`. */ + readonly holdsResources: boolean; } const PLUGIN_NAME: string = 'WorkspaceWarmSet'; @@ -99,7 +102,7 @@ export class WorkspaceWarmSet implements AsyncDisposable { const name: string = operation.associatedProject.packageName; let project: IProjectHistory | undefined = this.#projects.get(name); if (!project) { - project = { operations: [], lastUsed: now, frequency: 0 }; + project = { operations: [], lastUsed: now, frequency: 0, requestedTarget: false }; this.#projects.set(name, project); } project.operations.push(operation); @@ -110,9 +113,11 @@ export class WorkspaceWarmSet implements AsyncDisposable { const requested: string[] = []; const requestedAt: number = performance.now(); for (const [name, project] of this.#projects) { - if (!project.operations.some((operation) => operation.enabled !== false)) continue; + const enabled: Operation[] = project.operations.filter((operation) => operation.enabled !== false); + if (!enabled.length) continue; project.lastUsed = requestedAt; project.frequency++; + project.requestedTarget = enabled.some((operation) => !hasEnabledConsumer(operation)); requested.push(name); } try { @@ -202,7 +207,9 @@ export class WorkspaceWarmSet implements AsyncDisposable { overMemoryBudget: daemonResidentMemoryBytes + measuredRunnerMemoryBytes > this.#configuration.warmMemoryBudgetMB * BYTES_PER_MB, - overProjectLimit: projects.length > this.#configuration.warmSetMaxProjects, + // Retained results of resource-free projects are cheap and are what makes a warm no-op skip possible. + overProjectLimit: + projects.filter((project) => project.holdsResources).length > this.#configuration.warmSetMaxProjects, deferredReason: this.#deferredReason, cleanupFailures: [ ...this.#cleanupFailures.values(), @@ -358,7 +365,8 @@ export class WorkspaceWarmSet implements AsyncDisposable { project.operations.every( (operation) => !graph.resultByOperation.has(operation) && !operation.runner?.isActive ); - if (!unrequested && !expired && !status.overMemoryBudget && !status.overProjectLimit) continue; + const overProjectLimit: boolean = status.overProjectLimit && project.holdsResources; + if (!unrequested && !expired && !status.overMemoryBudget && !overProjectLimit) continue; try { await graph.closeRunnersAsync(project.operations); if (project.operations.some((operation) => operation.runner?.isActive)) { @@ -409,7 +417,9 @@ export class WorkspaceWarmSet implements AsyncDisposable { ...history, residentMemoryBytes, timeSavedMs, - protected: history.operations.some((operation) => protectedOperations?.has(operation)) + protected: history.operations.some((operation) => protectedOperations?.has(operation)), + holdsResources: + watched.has(key) || resident.some((operation) => mayHoldRunnerResources(operation.runner)) }); } return projects.sort((a, b) => compareWarmSetRanks(a, b, this.#configuration.autoWarmByTelemetry)); @@ -499,6 +509,22 @@ function isMeasuredMemory(bytes: number | undefined): bytes is number { return bytes !== undefined && Number.isSafeInteger(bytes) && bytes > 0; } +/** + * `isActive` is optional for backward compatibility; a retained runner that leaves it undefined but can be closed + * may own background resources, matching the conservative accounting in `getStatus()`. + */ +function mayHoldRunnerResources(runner: IOperationRunner | undefined): boolean { + if (!runner) return false; + return runner.isActive === undefined ? !!runner.closeAsync : runner.isActive; +} + +function hasEnabledConsumer(operation: Operation): boolean { + for (const consumer of operation.consumers) { + if (consumer.enabled !== false) return true; + } + return false; +} + function isGraphBusy(graph: IOperationGraph): boolean { return ( graph.hasScheduledIteration || diff --git a/libraries/rush-daemon/src/test/WarmSetRanking.test.ts b/libraries/rush-daemon/src/test/WarmSetRanking.test.ts index b400bd1770..3ca6c70dca 100644 --- a/libraries/rush-daemon/src/test/WarmSetRanking.test.ts +++ b/libraries/rush-daemon/src/test/WarmSetRanking.test.ts @@ -49,6 +49,17 @@ describe('warm retention and eviction ranking', () => { ).toEqual(rank(true)); }); + it('prefers the explicitly requested target over same-request dependencies before the key tie-break', () => { + const entries: IWarmSetRank[] = [ + entry('a-dependency', { lastUsed: 2 }), + entry('z-target', { lastUsed: 2, requestedTarget: true }), + entry('older-target', { lastUsed: 1, requestedTarget: true }) + ]; + expect( + [...entries].sort((a, b) => compareWarmSetRanks(a, b, false)).map((item) => item.key) + ).toEqual(['z-target', 'a-dependency', 'older-target']); + }); + it('has a transitive missing-data order rather than a pair-dependent score/LRU comparison', () => { const entries = [ entry('high-old', { timeSavedMs: 400, lastUsed: 1 }), diff --git a/libraries/rush-daemon/src/test/WarmSetTestFixture.ts b/libraries/rush-daemon/src/test/WarmSetTestFixture.ts index b534fb498c..6588f22566 100644 --- a/libraries/rush-daemon/src/test/WarmSetTestFixture.ts +++ b/libraries/rush-daemon/src/test/WarmSetTestFixture.ts @@ -32,6 +32,8 @@ export interface IWarmFixtureOptions { readonly ipc?: boolean; readonly cache?: boolean; readonly configurationKind?: 'direct' | 'rig' | 'inherited'; + /** Adds independent shell-runner projects p01..pNN beside a, b and c. */ + readonly extraProjectCount?: number; } export const GENEROUS_WARM_CONFIGURATION: WorkspaceWarmSetConfiguration = { @@ -139,6 +141,7 @@ export class WarmSetTestFixture implements AsyncDisposable { ); fixture.write('a/config/rush-project.json', '{"extends":"../../common/temp/inherited.json"}'); } + if (options.extraProjectCount) addExtraProjects(fixture, options.extraProjectCount); }, false); }); result._attachOnInitialization(); @@ -223,6 +226,27 @@ export class WarmSetTestFixture implements AsyncDisposable { } } +export function getExtraProjectNames(count: number): string[] { + return Array.from({ length: count }, (unused, index) => `p${String(index + 1).padStart(2, '0')}`); +} + +function addExtraProjects(fixture: DaemonGraphTestFixture, count: number): void { + const names: string[] = getExtraProjectNames(count); + const rushJson: { projects: object[] } = JSON.parse( + fs.readFileSync(path.join(fixture.folder, 'rush.json'), 'utf8') + ); + rushJson.projects.push(...names.map((name) => ({ packageName: name, projectFolder: name }))); + fixture.write('rush.json', JSON.stringify(rushJson)); + for (const name of names) { + fixture.write( + `${name}/package.json`, + JSON.stringify({ name, version: '1.0.0', scripts: { '_phase:compile': 'node build.cjs' } }) + ); + fixture.write(`${name}/input.txt`, 'one'); + fixture.write(`${name}/build.cjs`, `require('node:fs').appendFileSync('../runs.txt', '${name}\\n');`); + } +} + /** Test engines may supply real IPC runners; the production non-watch resolver still selects shells. */ export function useNativeIpcRunners(graph: IOperationGraph): void { for (const operation of graph.operations) { diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts index 8eaddb9ce7..f4193dbd27 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts @@ -16,6 +16,8 @@ import { createDeferred } from './DaemonRequestWireTestUtilities'; import { createNativeScriptGateAsync, runNativeCommandAsync } from './NativeEngineTestCommands'; import { captureWarmRankingDurations, + GENEROUS_WARM_CONFIGURATION, + getExtraProjectNames, getMeasuredFixtureRetentionOrder, WarmSetTestFixture, type IWarmFixtureOptions @@ -93,6 +95,91 @@ describe('warm policies attached to native graphs and real filesystem watchers', expect(fixture.runs()).toEqual(runs); }); + it('keeps the explicitly requested target over its same-request dependencies at the project cap', async () => { + const { warm, graph } = await startAsync(); + test!.update({ warmSetMaxProjects: 1 }); + expect((await warm.maintainAsync()).retainedProjectNames).toEqual(['b']); + expect(graph.resultByOperation.has(test!.operation('a'))).toBe(false); + }); + + it('does not count or evict resource-free retained results for the project cap, but still evicts resource holders', async () => { + test = await WarmSetTestFixture.createAsync({ extraProjectCount: 40 }); + test.configuration = { ...GENEROUS_WARM_CONFIGURATION, watch: false, warmSetMaxProjects: 20 }; + const { fixture } = test; + const build = async (): Promise => { + const exchange = await fixture.runAsync(['build', '--parallelism', '8']); + expect(exchange.terminal).toMatchObject({ payload: { exitCode: 0 } }); + }; + await build(); + const { warm, graph, watcher } = test; + const projectCount: number = 43; + const built = await warm.maintainAsync(); + expect(built.overProjectLimit).toBe(false); + expect(built.retainedProjectNames).toHaveLength(projectCount); + expect(graph.resultByOperation.size).toBe(projectCount); + expect(watcher.watchedProjectNames.size).toBe(0); + const runs: string[] = fixture.runs(); + expect(runs).toHaveLength(projectCount); + + // A repeated no-op build skips every retained project instead of re-running or restoring it. + await build(); + expect((await warm.maintainAsync()).retainedProjectNames).toHaveLength(projectCount); + expect(fixture.runs()).toEqual(runs); + expect(graph.resultByOperation.size).toBe(projectCount); + + // 21 real resource holders exceed the cap of 20: exactly the lowest-ranked holder is released. + const closed: string[] = []; + for (const name of getExtraProjectNames(21)) { + let active: boolean = true; + test.operation(name).runner = { + name: `resident-${name}`, + isNoOp: false, + cacheable: false, + reportTiming: false, + silent: false, + warningsAreAllowed: false, + get isActive() { + return active; + }, + residentMemoryBytes: 1024, + getConfigHash: () => '', + executeAsync: async () => OperationStatus.Success, + closeAsync: async () => { + active = false; + closed.push(name); + } + }; + } + expect(warm.getStatus().overProjectLimit).toBe(true); + const capped = await warm.maintainAsync(); + expect(closed).toEqual(['p21']); + expect(graph.resultByOperation.has(test.operation('p21'))).toBe(false); + expect(graph.resultByOperation.size).toBe(projectCount - 1); + expect(capped.overProjectLimit).toBe(false); + expect(capped.measuredRunnerMemoryBytes).toBe(20 * 1024); + + // A closable legacy runner that omits the optional isActive flag is conservatively a resource holder. + const legacyClosed: string[] = []; + test.operation('p22').runner = { + name: 'legacy-p22', + isNoOp: false, + cacheable: false, + reportTiming: false, + silent: false, + warningsAreAllowed: false, + getConfigHash: () => '', + executeAsync: async () => OperationStatus.Success, + closeAsync: async () => { + legacyClosed.push('p22'); + } + }; + expect(warm.getStatus().overProjectLimit).toBe(true); + expect((await warm.maintainAsync()).overProjectLimit).toBe(false); + expect(legacyClosed).toEqual(['p22']); + expect(graph.resultByOperation.has(test.operation('p22'))).toBe(false); + expect(graph.resultByOperation.size).toBe(projectCount - 2); + }); + it('lets autoWarmByTelemetry change actual retention using real cold/reused durations and IPC RSS', async () => { const { fixture, warm, graph } = await startAsync({ ipc: true }); const coldDurations = captureWarmRankingDurations(graph); diff --git a/libraries/rush-daemon/src/test/WorkspaceWatchPolicy.test.ts b/libraries/rush-daemon/src/test/WorkspaceWatchPolicy.test.ts index fcef21ab5c..7f37422b26 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWatchPolicy.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWatchPolicy.test.ts @@ -40,7 +40,8 @@ describe('daemon.watch observation-only policy', () => { expect((await pongAsync(fixture)).workspace?.warmSet).toMatchObject({ configuration: { watch: false }, watchedProjectNames: [], - retainedProjectNames: ['a', 'b'] + // Ranked best-first: the requested target b precedes its same-request dependency a. + retainedProjectNames: ['b', 'a'] }); fixture.write('a/input.txt', 'changed-unwatched'); await fixture.buildSuccessfullyAsync(); diff --git a/libraries/rush-lib/src/api/DaemonConfiguration.ts b/libraries/rush-lib/src/api/DaemonConfiguration.ts index 582c57f83e..695dcaa548 100644 --- a/libraries/rush-lib/src/api/DaemonConfiguration.ts +++ b/libraries/rush-lib/src/api/DaemonConfiguration.ts @@ -19,7 +19,10 @@ export interface IDaemonConfigurationJson { readonly warmIdleTimeoutSeconds?: number; /** Best-effort sampled RSS budget for an attached warm set, not a hard ceiling. Defaults to 512 MiB. */ readonly warmMemoryBudgetMB?: number; - /** Best-effort retained project limit; active/protected work is exempt. Defaults to 20 projects. */ + /** + * Best-effort limit on projects holding warm resources (active runners or file watchers); active/protected work + * is exempt, and retained results of resource-free projects do not count. Defaults to 20 projects. + */ readonly warmSetMaxProjects?: number; /** Prefer measured time-saved * frequency / resident-memory retention over LRU. Never starts scripts. Defaults to false. */ readonly autoWarmByTelemetry?: boolean; diff --git a/libraries/rush-lib/src/schemas/rush.schema.json b/libraries/rush-lib/src/schemas/rush.schema.json index 9e6e6328cf..801ccdafc3 100644 --- a/libraries/rush-lib/src/schemas/rush.schema.json +++ b/libraries/rush-lib/src/schemas/rush.schema.json @@ -305,7 +305,7 @@ "minimum": 1, "maximum": 9007199254740991, "default": 20, - "description": "Best-effort retained project limit for an attached warm set, at most 9007199254740991 (Number.MAX_SAFE_INTEGER). RUSH_DAEMON_WARM_SET_MAX_PROJECTS overrides." + "description": "Best-effort limit on projects holding warm resources (active runners or file watchers) in an attached warm set, at most 9007199254740991 (Number.MAX_SAFE_INTEGER). Retained results of resource-free projects do not count and are not evicted for this limit. RUSH_DAEMON_WARM_SET_MAX_PROJECTS overrides." }, "autoWarmByTelemetry": { "type": "boolean", From 7a0348d7c0505f739cda036ac255b1c506eabd6b Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Thu, 24 Sep 2026 14:18:58 -0700 Subject: [PATCH 012/265] [rush-daemon] Print the native operation summary and duration line on the daemon path (#6068) * [rush-daemon] Print the native operation summary and duration line for phased requests Fixes #6053 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Fix summary test typing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-sdk] Update the export snapshot for _printOperationStatus Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Honor the request warnings policy in the operation summary Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- ...rushd-summary-banner_2026-09-24-01-30.json | 11 ++ ...rushd-summary-banner_2026-09-24-01-30.json | 11 ++ common/reviews/api/rush-lib.api.md | 3 + .../rush-daemon/src/PhasedRequestRouter.ts | 16 ++ .../rush-daemon/src/PhasedRequestSummary.ts | 182 +++++++++++++++++ .../src/test/PhasedRequestSummary.test.ts | 185 ++++++++++++++++++ libraries/rush-lib/src/index.ts | 1 + .../test/__snapshots__/script.test.ts.snap | 1 + 8 files changed, 410 insertions(+) create mode 100644 common/changes/@microsoft/rush/rushd-summary-banner_2026-09-24-01-30.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-summary-banner_2026-09-24-01-30.json create mode 100644 libraries/rush-daemon/src/PhasedRequestSummary.ts create mode 100644 libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts diff --git a/common/changes/@microsoft/rush/rushd-summary-banner_2026-09-24-01-30.json b/common/changes/@microsoft/rush/rushd-summary-banner_2026-09-24-01-30.json new file mode 100644 index 0000000000..c7e8ba2f44 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-summary-banner_2026-09-24-01-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Expose the internal operation summary printer so the Rush daemon can print the native end-of-run summary.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "TheLarkInn@users.noreply.github.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-summary-banner_2026-09-24-01-30.json b/common/changes/@rushstack/rush-daemon/rushd-summary-banner_2026-09-24-01-30.json new file mode 100644 index 0000000000..89b47051e2 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-summary-banner_2026-09-24-01-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Print the native operation summary tables and the \"rush ()\" line for each phased request, including warm no-op builds.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "TheLarkInn@users.noreply.github.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index c553ceacb8..1b712f18dd 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -1599,6 +1599,9 @@ export type PnpmStoreOptions = PnpmStoreLocation; // @public export type PnpmTrustPolicy = 'no-downgrade' | 'off'; +// @internal +export function _printOperationStatus(terminal: ITerminal, result: IExecutionResult): void; + // @beta (undocumented) export class ProjectChangeAnalyzer { constructor(rushConfiguration: RushConfiguration); diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index 430751653c..1a6e9bd0a7 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -20,6 +20,7 @@ import type { import { PhasedRequestEventSink } from './PhasedRequestEventSink'; import { PhasedRequestEventMultiplexer } from './PhasedRequestEventMultiplexer'; +import { writePhasedRequestSummary } from './PhasedRequestSummary'; import type { IPhasedRequestClient } from './PhasedRequestClient'; import { DaemonRequiresInProcessError, evaluateDaemonTerminalPolicy } from './DaemonTerminalPolicy'; import { DaemonShutdownError, getDaemonShutdownReason } from './DaemonShutdownError'; @@ -71,6 +72,8 @@ interface IPreparedPhasedRequest { readonly requestSettings: IPhasedCommandEngineRequestSettings | undefined; readonly requestSettingsKey: string; readonly selection: IResolvedSelection; + /** The `performance.now()` timestamp at which the router received the request. */ + readonly startTimeMs: number; readonly warningsAllowedByEnvironment: boolean; } @@ -124,6 +127,7 @@ export class PhasedRequestRouter { onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings ): Promise { + const startTimeMs: number = performance.now(); validateRequestIdentity(request); const interactiveSession: IInteractiveRequestSession | undefined = validateInteractiveSession( request, @@ -203,6 +207,7 @@ export class PhasedRequestRouter { requestSettings, requestSettingsKey: JSON.stringify(requestSettings ?? null), selection, + startTimeMs, warningsAllowedByEnvironment, onExecutionStarting }, @@ -618,6 +623,17 @@ class PhasedRequestBatchCoordinator { } const cleanupErrors: unknown[] = [...batchCleanupErrors]; if (entry.requestSink) { + if (entry.participated && this.#isEntryLive(entry)) { + writePhasedRequestSummary({ + activeOperations: entry.selection.activeOperations, + commandName: entry.request.commandName, + elapsedMs: performance.now() - entry.startTimeMs, + executionError, + graph: this.#graph, + sink: entry.requestSink, + warningsAllowedByEnvironment: entry.warningsAllowedByEnvironment + }); + } try { await entry.requestSink.flushAsync(); } catch (error) { diff --git a/libraries/rush-daemon/src/PhasedRequestSummary.ts b/libraries/rush-daemon/src/PhasedRequestSummary.ts new file mode 100644 index 0000000000..ed0628d60b --- /dev/null +++ b/libraries/rush-daemon/src/PhasedRequestSummary.ts @@ -0,0 +1,182 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { + IOperationExecutionResult, + IOperationGraph, + Operation, + _IOperationActivityOptions +} from '@microsoft/rush-lib'; +import { OperationStatus, _printOperationStatus } from '@microsoft/rush-lib'; +import { Terminal, TerminalProviderSeverity, type ITerminalProvider } from '@rushstack/terminal'; + +const SECONDS_PER_MINUTE: number = 60; +const MILLISECONDS_PER_SECOND: number = 1000; +const SUMMARIZED_STATUSES: ReadonlySet = new Set([ + OperationStatus.Aborted, + OperationStatus.Blocked, + OperationStatus.Failure, + OperationStatus.FromCache, + OperationStatus.NoOp, + OperationStatus.Skipped, + OperationStatus.Success, + OperationStatus.SuccessWithWarning +]); + +/** The subset of a request event sink used to render a request's end-of-run summary. */ +export interface IPhasedRequestSummarySink { + getObservedResult(operation: Operation): { readonly executionResult: IOperationExecutionResult } | undefined; + onActivity(text: string, options?: _IOperationActivityOptions): void; +} + +export interface IWritePhasedRequestSummaryOptions { + readonly activeOperations: ReadonlyArray; + readonly commandName: string; + readonly elapsedMs: number; + readonly executionError: unknown; + readonly graph: IOperationGraph; + readonly sink: IPhasedRequestSummarySink; + /** Whether the request environment allows warnings in a successful build (`RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD`). */ + readonly warningsAllowedByEnvironment: boolean; +} + +/** + * Buffers terminal output into request-scoped activity events, one event per contiguous stream run. + */ +class RequestActivityTerminalProvider implements ITerminalProvider { + public readonly supportsColor: boolean = false; + public readonly eolCharacter: string = '\n'; + readonly #sink: IPhasedRequestSummarySink; + #buffer: string = ''; + #stderr: boolean = false; + + public constructor(sink: IPhasedRequestSummarySink) { + this.#sink = sink; + } + + public write(text: string, severity: TerminalProviderSeverity): void { + if (severity === TerminalProviderSeverity.verbose || severity === TerminalProviderSeverity.debug) { + return; + } + const stderr: boolean = + severity === TerminalProviderSeverity.error || severity === TerminalProviderSeverity.warning; + if (stderr !== this.#stderr) { + this.flush(); + this.#stderr = stderr; + } + this.#buffer += text; + } + + public flush(): void { + if (this.#buffer.length > 0) { + this.#sink.onActivity(this.#buffer, { stderr: this.#stderr }); + this.#buffer = ''; + } + } +} + +/** + * Writes the native end-of-run summary (the status tables and the `rush ()` line) for one + * phased request into that request's own event sink. + * + * @remarks + * Coalesced requests share one graph iteration, so the summary is computed per request from the request's own + * selection rather than from the whole iteration. Selected operations that the warm graph did not need to run are + * reported as already up to date, so a warm no-op still reports what it checked. + */ +export function writePhasedRequestSummary(options: IWritePhasedRequestSummaryOptions): void { + const { commandName, elapsedMs, executionError, sink } = options; + const provider: RequestActivityTerminalProvider = new RequestActivityTerminalProvider(sink); + const terminal: Terminal = new Terminal(provider); + const duration: string = formatDuration(elapsedMs); + if (executionError === undefined) { + const operationResults: ReadonlyMap = + collectSummaryResults(options); + _printOperationStatus(terminal, { + operationResults, + status: getSummaryStatus(operationResults, options.warningsAllowedByEnvironment) + }); + terminal.writeLine(`rush ${commandName} (${duration})`); + } else { + terminal.writeErrorLine(`rush ${commandName} - Errors! (${duration})`); + } + provider.flush(); +} + +function collectSummaryResults( + options: IWritePhasedRequestSummaryOptions +): ReadonlyMap { + const { activeOperations, graph, sink } = options; + const active: ReadonlySet = new Set(activeOperations); + const results: Map = new Map(); + // Iterate the graph so the summary lists operations in the same order as the native summary. + for (const operation of graph.operations) { + if (!active.has(operation) || operation.runner?.silent !== false) { + continue; + } + const observed: IOperationExecutionResult | undefined = + sink.getObservedResult(operation)?.executionResult; + if (observed && !observed.silent) { + if (SUMMARIZED_STATUSES.has(observed.status)) { + results.set(operation, observed); + } + continue; + } + // A silent observed record belongs to an operation the graph disabled because it was already up to date. + const previous: IOperationExecutionResult | undefined = + observed ?? graph.resultByOperation.get(operation); + if (previous) { + results.set(operation, createUpToDateResult(previous)); + } + } + return results; +} + +function createUpToDateResult(previous: IOperationExecutionResult): IOperationExecutionResult { + // The summary only reads these members for skipped operations; the shared record itself must not change. + const upToDate: Pick = { + operation: previous.operation, + silent: false, + status: OperationStatus.Skipped, + stopwatch: previous.stopwatch + }; + return upToDate as IOperationExecutionResult; +} + +function getSummaryStatus( + results: ReadonlyMap, + warningsAllowedByEnvironment: boolean +): OperationStatus { + let status: OperationStatus = OperationStatus.Success; + for (const [operation, result] of results) { + switch (result.status) { + case OperationStatus.Failure: + case OperationStatus.Blocked: + return OperationStatus.Failure; + case OperationStatus.Aborted: + status = OperationStatus.Aborted; + break; + case OperationStatus.SuccessWithWarning: + if ( + status === OperationStatus.Success && + !warningsAllowedByEnvironment && + !operation.runner?.warningsAreAllowed + ) { + status = OperationStatus.SuccessWithWarning; + } + break; + } + } + return status; +} + +/** Matches the native Rush stopwatch format, for example `1.23 seconds` or `2 minutes 3.4 seconds`. */ +function formatDuration(elapsedMs: number): string { + const totalSeconds: number = elapsedMs / MILLISECONDS_PER_SECOND; + if (totalSeconds > SECONDS_PER_MINUTE) { + const minutes: number = Math.floor(totalSeconds / SECONDS_PER_MINUTE); + const seconds: number = totalSeconds % SECONDS_PER_MINUTE; + return `${minutes.toFixed(0)} minute${minutes === 1 ? '' : 's'} ${seconds.toFixed(1)} seconds`; + } + return `${totalSeconds.toFixed(2)} seconds`; +} diff --git a/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts b/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts new file mode 100644 index 0000000000..a2f05f65af --- /dev/null +++ b/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts @@ -0,0 +1,185 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { OperationStatus } from '@microsoft/rush-lib'; +import type { IDaemonPhasedOperationSelection, IDaemonPhasedRequest } from '@rushstack/rush-daemon-protocol'; + +import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import { + TEST_ENGINE_SHAPE, + TestOperationRunner, + TestPhasedRequestClient, + createRoutingFixture +} from './PhasedRequestRouterTestUtilities'; +import type { ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; + +const OPERATION_A: string = 'project-a (_phase:test)'; +const OPERATION_B: string = 'project-b (_phase:test)'; +const OPERATION_C: string = 'project-c (_phase:test)'; +const DURATION_LINE: RegExp = /^rush build \(\d+\.\d\d seconds\)$/m; + +function createRequest(requestId: string, ...operationIds: string[]): IDaemonPhasedRequest { + return { + commandName: 'build', + commandOrigin: 'built-in', + engineShape: TEST_ENGINE_SHAPE, + environment: {}, + operationSelection: operationIds.map( + (operationId: string): IDaemonPhasedOperationSelection => ({ enabledState: true, operationId }) + ), + requestId + }; +} + +function createFixture(statusA: OperationStatus = OperationStatus.Success): ITestRoutingFixture { + return createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A, statusA)], + [OPERATION_B, new TestOperationRunner(OPERATION_B)], + [OPERATION_C, new TestOperationRunner(OPERATION_C)] + ]), + [[OPERATION_B, OPERATION_A]] + ); +} + +function getActivity(client: TestPhasedRequestClient, stream: 'stdout' | 'stderr'): string { + let text: string = ''; + for (const { event } of client.writes) { + const payload: { stream?: string; text?: string } | undefined = + event?.type === 'activityChanged' ? (event.payload as { stream?: string; text?: string }) : undefined; + if (payload?.stream === stream) { + text += payload.text; + } + } + return text; +} + +function getSummary(stdout: string): string { + // Operation headers are structured events, so the first activity banner starts the end-of-run summary. + return stdout.slice(stdout.indexOf('==[ ')); +} + +describe('phased request summary', () => { + it('reports the native summary tables and duration line after a cold build', async () => { + const fixture: ITestRoutingFixture = createFixture(); + try { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + await new PhasedRequestRouter(fixture.session).executeAsync( + createRequest('cold', OPERATION_B), + client + ); + const stdout: string = getActivity(client, 'stdout'); + const summary: string = getSummary(stdout); + expect(summary).toContain('==[ SUCCESS: 2 operations ]=='); + expect(summary).toContain('These operations completed successfully:'); + expect(summary).toContain(` ${OPERATION_A}`); + expect(summary).toContain(` ${OPERATION_B}`); + expect(summary).not.toContain(OPERATION_C); + expect(stdout).toMatch(DURATION_LINE); + expect(stdout.indexOf('==[ SUCCESS')).toBeLessThan(stdout.search(DURATION_LINE)); + expect(client.writes[client.writes.length - 1].result).toBeDefined(); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('reports up-to-date operations and the duration line for a warm no-op', async () => { + const fixture: ITestRoutingFixture = createFixture(); + try { + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + await router.executeAsync(createRequest('cold', OPERATION_B), new TestPhasedRequestClient()); + // Simulate the incremental plugin disabling every operation whose inputs did not change. + fixture.graph.hooks.configureIteration.tap('test', (records) => { + for (const record of records.values()) { + record.enabled = false; + } + }); + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + const result = await router.executeAsync(createRequest('warm', OPERATION_B), client); + expect(result.scheduled).toBe(false); + const stdout: string = getActivity(client, 'stdout'); + expect(stdout).toContain('==[ SKIPPED: 2 operations ]=='); + expect(stdout).toContain('These operations were already up to date:'); + expect(stdout).toMatch(DURATION_LINE); + expect(getActivity(client, 'stderr')).toBe(''); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('reports failed and blocked operations and the duration line for a failing build', async () => { + const fixture: ITestRoutingFixture = createFixture(OperationStatus.Failure); + try { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + const result = await new PhasedRequestRouter(fixture.session).executeAsync( + createRequest('failing', OPERATION_B), + client + ); + expect(result.exitCode).not.toBe(0); + const stdout: string = getActivity(client, 'stdout'); + expect(stdout).toContain('==[ BLOCKED: 1 operation ]=='); + expect(stdout).toContain('==[ FAILURE: 1 operation ]=='); + expect(stdout).toContain(`--[ FAILURE: ${OPERATION_A} ]--`); + expect(stdout).toMatch(DURATION_LINE); + expect(getActivity(client, 'stderr')).toContain('Operations failed.'); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('honors the request warnings policy in the summary verdict', async () => { + for (const [allowWarnings, expectedVerdict] of [ + ['0', 'Operations succeeded with warnings.'], + ['1', ''] + ] as const) { + const fixture: ITestRoutingFixture = createFixture(OperationStatus.SuccessWithWarning); + try { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + await new PhasedRequestRouter(fixture.session).executeAsync( + { + ...createRequest('warning', OPERATION_A), + environment: { RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD: allowWarnings } + }, + client + ); + expect(getActivity(client, 'stdout')).toContain('==[ SUCCESS WITH WARNINGS: 1 operation ]=='); + expect(getActivity(client, 'stdout')).toMatch(DURATION_LINE); + const stderr: string = getActivity(client, 'stderr'); + if (expectedVerdict) { + expect(stderr).toContain(expectedVerdict); + } else { + expect(stderr).not.toContain('Operations succeeded with warnings.'); + } + } finally { + await fixture.session[Symbol.asyncDispose](); + } + } + }); + + it('gives each coalesced request a summary of only its own selection', async () => { + const fixture: ITestRoutingFixture = createFixture(); + try { + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const clientA: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const clientC: TestPhasedRequestClient = new TestPhasedRequestClient('two'); + await Promise.all([ + router.executeAsync(createRequest('a', OPERATION_A), clientA), + router.executeAsync(createRequest('c', OPERATION_C), clientC) + ]); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); + const stdoutA: string = getSummary(getActivity(clientA, 'stdout')); + const stdoutC: string = getSummary(getActivity(clientC, 'stdout')); + expect(stdoutA).toContain('==[ SUCCESS: 1 operation ]=='); + expect(stdoutA).toContain(` ${OPERATION_A}`); + expect(stdoutA).not.toContain(OPERATION_C); + expect(stdoutC).toContain('==[ SUCCESS: 1 operation ]=='); + expect(stdoutC).toContain(` ${OPERATION_C}`); + expect(stdoutC).not.toContain(OPERATION_A); + expect(stdoutA).toMatch(DURATION_LINE); + expect(stdoutC).toMatch(DURATION_LINE); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); +}); diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index 8af91cfeca..a2346d9e99 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -174,6 +174,7 @@ export type { export { type IOperationOptions, type OperationEnabledState, Operation } from './logic/operations/Operation'; export { type IParallelismScalar, type Parallelism } from './logic/operations/ParseParallelism'; export { OperationStatus } from './logic/operations/OperationStatus'; +export { _printOperationStatus } from './logic/operations/OperationResultSummarizerPlugin'; export { PhasedCommandEngine, type IPhasedCommandEngine, diff --git a/libraries/rush-sdk/src/test/__snapshots__/script.test.ts.snap b/libraries/rush-sdk/src/test/__snapshots__/script.test.ts.snap index 799804d99f..e6cb906e49 100644 --- a/libraries/rush-sdk/src/test/__snapshots__/script.test.ts.snap +++ b/libraries/rush-sdk/src/test/__snapshots__/script.test.ts.snap @@ -69,6 +69,7 @@ Loaded @microsoft/rush-lib from process.env._RUSH_LIB_PATH '_OperationStateFile', '_RushGlobalFolder', '_RushInternals', + '_printOperationStatus', '_rushSdk_loadInternalModule', 'captureProjectConfigurationFingerprintAsync', 'captureWorkspaceInputFingerprintAsync', From 3d690ab03afea0a9e455874d89faa3b68689e524 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Fri, 25 Sep 2026 13:26:42 -0700 Subject: [PATCH 013/265] [rush-daemon] Fix summary writer racing early coalesced results (#6095) The early per-client result path (#6092) runs from onOperationCompleted, which the record's finalizeOperation() invokes synchronously before closing its StdioSummarizer. The summary writer (#6068) reads the failure tail from that summarizer, so it threw. Yield once before producing the early result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- ...ix-main-summary-race_2026-09-24-21-45.json | 11 ++++ .../rush-daemon/src/PhasedRequestRouter.ts | 22 +++++--- .../src/test/PhasedRequestSummary.test.ts | 51 +++++++++++++++++++ 3 files changed, 76 insertions(+), 8 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/fix-main-summary-race_2026-09-24-21-45.json diff --git a/common/changes/@rushstack/rush-daemon/fix-main-summary-race_2026-09-24-21-45.json b/common/changes/@rushstack/rush-daemon/fix-main-summary-race_2026-09-24-21-45.json new file mode 100644 index 0000000000..8c97b33d05 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/fix-main-summary-race_2026-09-24-21-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Fix an early coalesced phased result writing its summary before the failed operation's output summarizer was closed.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "TheLarkInn@users.noreply.github.com" +} diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index 1a6e9bd0a7..c943eee794 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -572,15 +572,21 @@ class PhasedRequestBatchCoordinator { } entry.unsubscribe?.(); entry.unsubscribe = undefined; - entry.finishPromise = this.#produceResultAsync(entry, true, undefined, [], undefined, true).catch( - (error: unknown) => { - // Unlike a batch-wide failure, an early result's failure concerns only this client. - if (!entry.completed) { - this.#completeEntry(entry); - entry.reject(error); - } + entry.finishPromise = this.#produceEarlyResultAsync(entry).catch((error: unknown) => { + // Unlike a batch-wide failure, an early result's failure concerns only this client. + if (!entry.completed) { + this.#completeEntry(entry); + entry.reject(error); } - ); + }); + } + + async #produceEarlyResultAsync(entry: IBatchEntry): Promise { + // The sink is notified from the record's `finalizeOperation()`, which synchronously precedes the close of + // the record's StdioSummarizer and ProblemCollector. The summary reads the failure tail from the closed + // summarizer, so yield once to let the notifying record finish closing before the summary is written. + await Promise.resolve(); + await this.#produceResultAsync(entry, true, undefined, [], undefined, true); } #requestIterationAbort(): void { diff --git a/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts b/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts index a2f05f65af..5c3063d1ba 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestSummary.test.ts @@ -156,6 +156,57 @@ describe('phased request summary', () => { } }); + it('writes the failure summary before an early result while the coalesced batch continues', async () => { + let releaseC: () => void = () => undefined; + const cHeld: Promise = new Promise((resolve) => { + releaseC = resolve; + }); + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [ + OPERATION_A, + new TestOperationRunner(OPERATION_A, OperationStatus.Failure, async (terminal) => + terminal.writeErrorLine('a-failure-detail') + ) + ], + [OPERATION_B, new TestOperationRunner(OPERATION_B)], + [OPERATION_C, new TestOperationRunner(OPERATION_C, OperationStatus.Success, () => cHeld)] + ]), + [[OPERATION_B, OPERATION_A]] + ); + fixture.graph.parallelism = 2; + try { + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const clientA: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const clientC: TestPhasedRequestClient = new TestPhasedRequestClient('two'); + const resultAPromise = router.executeAsync(createRequest('a', OPERATION_A), clientA); + const resultCPromise = router.executeAsync(createRequest('c', OPERATION_C), clientC); + + const resultA = await resultAPromise; + // The early result is published while the shared iteration still runs the other client's selection. + expect(fixture.graph.status).toBe(OperationStatus.Executing); + expect(resultA).toMatchObject({ exitCode: 1, outcome: 'failure' }); + const stdoutA: string = getActivity(clientA, 'stdout'); + const summaryA: string = getSummary(stdoutA); + expect(summaryA).toContain('==[ FAILURE: 1 operation ]=='); + expect(summaryA).toContain(`--[ FAILURE: ${OPERATION_A} ]--`); + expect(summaryA).toContain('a-failure-detail'); + expect(summaryA).not.toContain(OPERATION_C); + expect(stdoutA).toMatch(DURATION_LINE); + expect(getActivity(clientA, 'stderr')).toContain('Operations failed.'); + expect(clientA.writes[clientA.writes.length - 1].result).toBe(resultA); + + releaseC(); + const resultC = await resultCPromise; + expect(resultC).toMatchObject({ exitCode: 0, outcome: 'success' }); + const summaryC: string = getSummary(getActivity(clientC, 'stdout')); + expect(summaryC).toContain('==[ SUCCESS: 1 operation ]=='); + expect(summaryC).not.toContain(OPERATION_A); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + it('gives each coalesced request a summary of only its own selection', async () => { const fixture: ITestRoutingFixture = createFixture(); try { From c026f9e004d59e055e324b8133da523b0084602e Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Fri, 25 Sep 2026 16:19:07 -0700 Subject: [PATCH 014/265] [rush-daemon] Terminate running operations when a daemon client cancels (#6065) * [rush-daemon] Terminate running operations when a daemon client cancels Fixes #6060 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Address review: kill exited-shell process groups, honor aborted flag, detach pre-execution cancellations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Tolerate gated scripts being terminated by a hard abort in native engine tests Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../rush-cli-client/src/clientCancellation.ts | 40 ++++ apps/rush-cli-client/src/launchClient.ts | 38 +++- .../src/test/launchClient.test.ts | 51 +++++ .../test/persistentIpcCancellation.test.ts | 5 +- .../rushd-cancel-hard-abort_2026-09-23.json | 11 + .../rushd-cancel-hard-abort_2026-09-23.json | 11 + .../rushd-cancel-hard-abort_2026-09-23.json | 11 + common/reviews/api/rush-lib.api.md | 5 +- .../rush-daemon/src/PhasedRequestRouter.ts | 74 ++++++- .../src/test/NativeEngineTestCommands.ts | 2 + .../src/test/PhasedRequestBatching.test.ts | 20 +- .../test/PhasedRequestCancellation.test.ts | 189 +++++++++++++++++ .../test/PhasedRequestRouterTestUtilities.ts | 16 +- .../cli/scriptActions/PhasedScriptAction.ts | 3 +- .../src/logic/operations/IOperationGraph.ts | 5 +- .../src/logic/operations/IOperationRunner.ts | 8 + .../operations/OperationExecutionRecord.ts | 6 + .../src/logic/operations/OperationGraph.ts | 21 +- .../logic/operations/ShellOperationRunner.ts | 46 +++- .../test/ShellOperationRunnerAbort.test.ts | 199 ++++++++++++++++++ 20 files changed, 720 insertions(+), 41 deletions(-) create mode 100644 apps/rush-cli-client/src/clientCancellation.ts create mode 100644 common/changes/@microsoft/rush/rushd-cancel-hard-abort_2026-09-23.json create mode 100644 common/changes/@rushstack/rush-cli-client/rushd-cancel-hard-abort_2026-09-23.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-cancel-hard-abort_2026-09-23.json create mode 100644 libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerAbort.test.ts diff --git a/apps/rush-cli-client/src/clientCancellation.ts b/apps/rush-cli-client/src/clientCancellation.ts new file mode 100644 index 0000000000..4c60a964eb --- /dev/null +++ b/apps/rush-cli-client/src/clientCancellation.ts @@ -0,0 +1,40 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as os from 'node:os'; + +import type { DaemonClientOutcome } from '@rushstack/rush-client-core'; + +/** Signals that cancel a daemon-routed command. */ +export const CANCELLATION_SIGNALS: ReadonlyArray = ['SIGINT', 'SIGTERM', 'SIGHUP']; + +const SIGNAL_EXIT_CODE_BASE: number = 128; + +/** + * Returns the conventional shell exit code for a process terminated by `signal` (128 + signal number), + * e.g. 130 for SIGINT and 143 for SIGTERM. + */ +export function getSignalExitCode(signal: NodeJS.Signals): number { + return SIGNAL_EXIT_CODE_BASE + (os.constants.signals[signal] ?? os.constants.signals.SIGINT); +} + +/** Formats the notice printed when a daemon-routed command is cancelled. */ +export function formatCancellationMessage(commandName: string): string { + return `rush-client: ${commandName} cancelled.\n`; +} + +/** + * Returns whether a daemon outcome represents a cancelled command. A result is cancelled when the daemon reports it + * as aborted, even if an operation failure determines its semantic outcome. A completed (non-aborted) result wins + * over a late signal, and a rejection is never reported as a cancellation. + */ +export function isCancelledOutcome(outcome: DaemonClientOutcome, signalled: boolean): boolean { + switch (outcome.kind) { + case 'result': + return outcome.result.aborted; + case 'rejected': + return false; + default: + return signalled; + } +} diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 1145164c4f..6ae500af55 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -26,6 +26,12 @@ import { executeDaemonCommandAsync } from './daemonCommands'; import { formatAdmissionFailure, getConfiguredAdmission } from './ClientAdmissionControls'; import { ClientOperationRenderer } from './ClientOperationRenderer'; import type { AgentProgressRenderer } from './AgentProgressRenderer'; +import { + CANCELLATION_SIGNALS, + formatCancellationMessage, + getSignalExitCode, + isCancelledOutcome +} from './clientCancellation'; import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; @@ -148,9 +154,13 @@ export async function launchClientAsync( return; } const abort: AbortController = new AbortController(); - const onSignal = (): void => abort.abort(); - process.on('SIGINT', onSignal); - process.on('SIGTERM', onSignal); + let cancellationSignal: NodeJS.Signals | undefined; + // Windows test harnesses emit signals without a name; treat those as Ctrl+C. + const onSignal = (signal?: NodeJS.Signals): void => { + cancellationSignal ??= signal ?? 'SIGINT'; + abort.abort(); + }; + for (const signal of CANCELLATION_SIGNALS) process.on(signal, onSignal); const renderer: ClientOperationRenderer = new ClientOperationRenderer({ requestId: request.requestId, colorLevel: terminal.supportsColor ? 1 : 0, @@ -166,7 +176,7 @@ export async function launchClientAsync( writeAsync: (bytes, stream) => writeStreamAsync(stream === 'stderr' ? process.stderr : process.stdout, bytes) }); - let outcome: DaemonClientOutcome; + let outcome: DaemonClientOutcome | undefined; const discoveryLines: string[] = []; const writeDiscoveryAsync = async (): Promise => { if (discoveryLines.length > 0) { @@ -215,16 +225,27 @@ export async function launchClientAsync( } : undefined }); + } catch (error) { + // After cancellation, a transport failure (e.g. the cancellation deadline) still means "cancelled". + if (!abort.signal.aborted || !(error instanceof DaemonClientError)) throw error; + outcome = undefined; } finally { - process.removeListener('SIGINT', onSignal); - process.removeListener('SIGTERM', onSignal); + for (const signal of CANCELLATION_SIGNALS) process.removeListener(signal, onSignal); try { await renderer.closeAsync(); } finally { await client.closeAsync(); } } - if (outcome.kind === 'result') { + if (outcome === undefined || isCancelledOutcome(outcome, abort.signal.aborted)) { + const exitCode: number = getSignalExitCode(cancellationSignal ?? 'SIGINT'); + agentRenderer?.finish({ exitCode, errorMessage: 'cancelled' }); + process.exitCode = exitCode; + // After SIGHUP the terminal may be gone; the exit code is what matters. + await writeStreamAsync(process.stderr, Buffer.from(formatCancellationMessage(route.commandName))).catch( + () => undefined + ); + } else if (outcome.kind === 'result') { agentRenderer?.finish(outcome.result); process.exitCode = outcome.result.exitCode; const diagnostic: string | undefined = getResultDiagnostic(outcome.result); @@ -239,9 +260,6 @@ export async function launchClientAsync( } else if (outcome.kind === 'rejected') { agentRenderer?.finish({ exitCode: 1, errorMessage: `daemon rejected the request (${outcome.rejection.code})` }); throw new Error(`Daemon rejected the request (${outcome.rejection.code}): ${outcome.rejection.message}`); - } else if (abort.signal.aborted) { - agentRenderer?.finish({ exitCode: 130, errorMessage: 'cancelled' }); - process.exitCode = 130; } else { agentRenderer?.dispose(); process.stderr.write(`rush-client: ${outcome.message ?? outcome.reason}; using in-process Rush.\n`); diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index d454c99c4e..39b091f31f 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -11,6 +11,7 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; import { Rush } from '@microsoft/rush-lib'; import { DaemonClient, connectOrStartDaemonAsync, getDaemonLogFilePath } from '@rushstack/rush-client-core'; +import type { DaemonClientOutcome } from '@rushstack/rush-client-core'; import { RushDaemonHost, WorkspaceSession } from '@rushstack/rush-daemon'; import { removeTestFolderAsync, @@ -20,6 +21,12 @@ import { captureTestDaemonListenerAsync } from '@rushstack/rush-daemon/lib/test/ import { readDaemonLockfile } from '@rushstack/rush-daemon-transport'; import { getDaemonConnectionOptions } from '../daemonConnectionOptions'; +import { + CANCELLATION_SIGNALS, + formatCancellationMessage, + getSignalExitCode, + isCancelledOutcome +} from '../clientCancellation'; interface IInvocationResult { readonly code: number | undefined; @@ -493,3 +500,47 @@ describe('standalone rushx fallback', () => { expect(result.stderr).toContain('--no-daemon cannot be combined with daemon start'); }); }); + +describe('daemon client cancellation exit codes', () => { + const abortedResult: DaemonClientOutcome = { + kind: 'result', + result: { requestId: 'r', outcome: 'aborted', exitCode: 1, aborted: true } + }; + + it('maps cancellation signals to 128 + signal number', () => { + expect(getSignalExitCode('SIGINT')).toBe(130); + expect(getSignalExitCode('SIGTERM')).toBe(143); + expect(getSignalExitCode('SIGHUP')).toBe(129); + expect(CANCELLATION_SIGNALS).toEqual(['SIGINT', 'SIGTERM', 'SIGHUP']); + }); + + it('treats an aborted daemon result as cancelled instead of copying its exit code', () => { + expect(isCancelledOutcome(abortedResult, true)).toBe(true); + // Ctrl+C read from a raw-mode TTY cancels without a process signal. + expect(isCancelledOutcome(abortedResult, false)).toBe(true); + expect(formatCancellationMessage('build')).toBe('rush-client: build cancelled.\n'); + }); + + it('treats a cancelled result as cancelled even when a failure outcome takes precedence', () => { + const cancelledWithFailure: DaemonClientOutcome = { + kind: 'result', + result: { requestId: 'r', outcome: 'failure', exitCode: 1, aborted: true } + }; + expect(isCancelledOutcome(cancelledWithFailure, true)).toBe(true); + }); + + it('keeps completed results and rejections when a signal arrives late', () => { + const succeeded: DaemonClientOutcome = { + kind: 'result', + result: { requestId: 'r', outcome: 'success', exitCode: 0, aborted: false } + }; + const rejected: DaemonClientOutcome = { + kind: 'rejected', + rejection: { requestId: 'r', code: 'unsupportedProtocolVersion', message: 'no' } + } as unknown as DaemonClientOutcome; + expect(isCancelledOutcome(succeeded, true)).toBe(false); + expect(isCancelledOutcome(rejected, true)).toBe(false); + expect(isCancelledOutcome({ kind: 'fallback', reason: 'unsupported' }, true)).toBe(true); + expect(isCancelledOutcome({ kind: 'fallback', reason: 'unsupported' }, false)).toBe(false); + }); +}); diff --git a/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts b/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts index ddc6c5eb5d..883d2f83ab 100644 --- a/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts +++ b/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts @@ -55,8 +55,9 @@ describe('public-client cancellation of an admitted Node operation', () => { } if (process.platform === 'win32') client.send('SIGINT'); else client.kill('SIGINT'); - // Phased cancellation preserves the daemon's existing aborted-result exit code. - expect(await closed).toEqual([1, null]); + // Cancellation terminates the client like a native signal (128 + SIGINT). + expect(await closed).toEqual([130, null]); + expect(stderr).toContain('rush-client: build cancelled.'); expect(stderr).not.toMatch(/using in-process|not retried|timed out/i); expect(fixture.events().filter((event) => event.kind === 'ready')).toHaveLength(1); expect(fixture.events().filter((event) => event.kind === 'complete')).toHaveLength(2); diff --git a/common/changes/@microsoft/rush/rushd-cancel-hard-abort_2026-09-23.json b/common/changes/@microsoft/rush/rushd-cancel-hard-abort_2026-09-23.json new file mode 100644 index 0000000000..9cddb07fcf --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-cancel-hard-abort_2026-09-23.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add an opt-in hard abort for operation graphs (`abortCurrentIterationAsync({ terminateRunning: true })`) that terminates running shell operation process trees and reports them as Aborted; daemon engine graphs enable it.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/rushd-cancel-hard-abort_2026-09-23.json b/common/changes/@rushstack/rush-cli-client/rushd-cancel-hard-abort_2026-09-23.json new file mode 100644 index 0000000000..8487bcf692 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/rushd-cancel-hard-abort_2026-09-23.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Exit with 128+signal (130/143/129) and print a cancellation notice when a daemon-routed command is cancelled; handle SIGHUP like SIGTERM.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-cancel-hard-abort_2026-09-23.json b/common/changes/@rushstack/rush-daemon/rushd-cancel-hard-abort_2026-09-23.json new file mode 100644 index 0000000000..0fc8c070a3 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-cancel-hard-abort_2026-09-23.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Terminate running operations when the last live client cancels a phased request, and answer a cancelling client immediately when other clients still need the shared work.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 1b712f18dd..7b82e2af5d 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -721,7 +721,9 @@ export interface IOperationExecutionResult extends IBaseOperationExecutionResult // @alpha export interface IOperationGraph { readonly abortController: AbortController; - abortCurrentIterationAsync(): Promise; + abortCurrentIterationAsync(options?: { + terminateRunning?: boolean; + }): Promise; addTerminalDestination(destination: TerminalWritable): void; allowOversubscription: boolean; closeRunnersAsync(operations?: Iterable): Promise; @@ -823,6 +825,7 @@ export interface IOperationRunner { // @beta export interface IOperationRunnerContext { + readonly abortSignal?: AbortSignal; collatedWriter: CollatedWriter; // @internal createChildProcessReporter(): _IOperationChildProcessReporter | undefined; diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index c943eee794..dda743a498 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -101,6 +101,12 @@ const OBSERVED_STATUS_OVERRIDES_RETAINED: ReadonlySet = new Set OperationStatus.Blocked, OperationStatus.Skipped ]); +const IN_PROGRESS_STATUSES: ReadonlySet = new Set([ + OperationStatus.Waiting, + OperationStatus.Ready, + OperationStatus.Queued, + OperationStatus.Executing +]); /** * Routes one caller-resolved phased request through a real warm workspace operation graph. @@ -422,6 +428,12 @@ class PhasedRequestBatchCoordinator { this.#graph, participants.map((entry: IBatchEntry) => entry.selection) ); + for (const entry of batch) { + if (!participants.includes(entry)) { + // Clients that cancelled before execution must not wait for the participants' work. + this.#finishDetachedEntry(entry); + } + } for (const entry of participants) { entry.participated = true; const activeOperationIds: ReadonlySet = new Set( @@ -531,6 +543,16 @@ class PhasedRequestBatchCoordinator { } } + if ( + entry.executionStarted && + this.#currentBatch?.includes(entry) && + this.#hasLiveBatchParticipant() + ) { + // Other live participants still need the shared work: detach this client and answer it now. + this.#finishDetachedEntry(entry); + return; + } + if ( entry.executionStarted && this.#currentBatch && @@ -541,10 +563,43 @@ class PhasedRequestBatchCoordinator { } } + #finishDetachedEntry(entry: IBatchEntry): void { + entry.finishPromise ??= this.#produceResultAsync( + entry, + entry.participated, + undefined, + [], + undefined, + true + ).catch((error: unknown) => { + if (!entry.completed) { + this.#completeEntry(entry); + entry.reject(error); + } + }); + } + #isEntryLive(entry: IBatchEntry): boolean { return !entry.abortRequested && !entry.client.abortSignal.aborted && entry.outputError === undefined; } + /** + * Whether a live client still needs the current batch's iteration, including compatible requests that were + * accepted into the pending queue and will join the batch once the execution lease is acquired. + */ + #hasLiveBatchParticipant(): boolean { + if (this.#currentBatch?.some((candidate: IBatchEntry) => this.#needsIteration(candidate))) { + return true; + } + return ( + this.#acceptingCurrentBatch && + this.#pending.some( + (candidate: IBatchEntry) => + candidate.exclusivityClass === RequestExclusivityClass.SharedBuild && this.#isEntryLive(candidate) + ) + ); + } + /** Whether a live participant still waits for the running iteration to produce its result. */ #needsIteration(entry: IBatchEntry): boolean { return entry.finishPromise === undefined && this.#isEntryLive(entry); @@ -590,7 +645,8 @@ class PhasedRequestBatchCoordinator { } #requestIterationAbort(): void { - const abortPromise: Promise = this.#graph.abortCurrentIterationAsync(); + // Nobody needs the running work any more, so terminate in-flight operations instead of awaiting them. + const abortPromise: Promise = this.#graph.abortCurrentIterationAsync({ terminateRunning: true }); this.#abortTail = Promise.all([this.#abortTail, abortPromise]) .then(() => undefined) .catch((error: unknown) => { @@ -984,12 +1040,14 @@ function collectOperationOutcomes( const retained: IOperationExecutionResult | undefined = graph.resultByOperation.get(operation); let status: string | undefined; let errorMessage: string | undefined; - if ( + if (iterationInProgress && observed !== undefined) { + // While the iteration still runs, retained results may predate this iteration, and work this client + // stopped observing before it finished (a detached cancellation) was abandoned. + status = IN_PROGRESS_STATUSES.has(observed.status) ? OperationStatus.Aborted : observed.status; + errorMessage = observed.executionResult.error?.message; + } else if ( observed !== undefined && - // While the iteration still runs, retained results may predate this iteration. - (iterationInProgress || - retained === undefined || - OBSERVED_STATUS_OVERRIDES_RETAINED.has(observed.status)) + (retained === undefined || OBSERVED_STATUS_OVERRIDES_RETAINED.has(observed.status)) ) { status = observed.status; errorMessage = observed.executionResult.error?.message; @@ -998,6 +1056,10 @@ function collectOperationOutcomes( errorMessage = retained?.error?.message ?? observed?.executionResult.error?.message; } status ??= fillMissingAsAborted ? OperationStatus.Aborted : undefined; + if (fillMissingAsAborted && status !== undefined && IN_PROGRESS_STATUSES.has(status)) { + // The client stopped observing before this operation finished, e.g. because it was terminated. + status = OperationStatus.Aborted; + } if (status === undefined) { continue; } diff --git a/libraries/rush-daemon/src/test/NativeEngineTestCommands.ts b/libraries/rush-daemon/src/test/NativeEngineTestCommands.ts index b334e12e96..324916efbe 100644 --- a/libraries/rush-daemon/src/test/NativeEngineTestCommands.ts +++ b/libraries/rush-daemon/src/test/NativeEngineTestCommands.ts @@ -61,6 +61,8 @@ export async function createNativeScriptGateAsync( const server: net.Server = net.createServer((socket) => { sockets.add(socket); socket.once('close', () => sockets.delete(socket)); + // A daemon shutdown or cancellation may terminate the gated script, which resets its connection. + socket.on('error', () => undefined); entered.resolve(); }); await new Promise((resolve, reject) => { diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index 3d67e07eb5..1cde7c35a6 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -431,7 +431,7 @@ describe('shared phased request batching', () => { expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); }); - it('reports authoritative retained status when a client cancels during a shared operation', async () => { + it('answers a client that cancels during a shared operation immediately, without waiting for the batch', async () => { const operationStarted: IDeferred = createDeferred(); const releaseOperation: IDeferred = createDeferred(); const fixture: ITestRoutingFixture = createFixture({ @@ -448,12 +448,14 @@ describe('shared phased request batching', () => { await operationStarted.promise; cancelledClient.abortController.abort(); + // The shared operation is still running for the other client. + const cancelledResult: IDaemonPhasedRequestResult = await cancelled; releaseOperation.resolve(); - const [cancelledResult, continuingResult] = await Promise.all([cancelled, continuing]); + const continuingResult: IDaemonPhasedRequestResult = await continuing; expect(cancelledResult).toMatchObject({ aborted: true, outcome: 'aborted' }); expect(cancelledResult.operationResults).toEqual([ - expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Success }) + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Aborted }) ]); expect(continuingResult).toMatchObject({ exitCode: 0, outcome: 'success' }); expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); @@ -486,7 +488,7 @@ describe('shared phased request batching', () => { ]); }); - it('preserves failure precedence when a client cancels during a failing shared operation', async () => { + it('keeps failure for the continuing client when another client cancels during a failing shared operation', async () => { const operationStarted: IDeferred = createDeferred(); const releaseOperation: IDeferred = createDeferred(); const fixture: ITestRoutingFixture = createFixture({ @@ -506,14 +508,16 @@ describe('shared phased request batching', () => { await operationStarted.promise; cancelledClient.abortController.abort(); + const cancelledResult: IDaemonPhasedRequestResult = await cancelled; releaseOperation.resolve(); - const [cancelledResult, continuingResult] = await Promise.all([cancelled, continuing]); + const continuingResult: IDaemonPhasedRequestResult = await continuing; - expect(cancelledResult).toMatchObject({ aborted: true, exitCode: 1, outcome: 'failure' }); - expect(cancelledResult.operationResults).toEqual([ + // The cancelled client detached before the shared operation failed. + expect(cancelledResult).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(continuingResult).toMatchObject({ aborted: false, exitCode: 1, outcome: 'failure' }); + expect(continuingResult.operationResults).toEqual([ expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Failure }) ]); - expect(continuingResult).toMatchObject({ aborted: false, exitCode: 1, outcome: 'failure' }); expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); }); diff --git a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts new file mode 100644 index 0000000000..7642e99cb4 --- /dev/null +++ b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts @@ -0,0 +1,189 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { ITerminal } from '@rushstack/terminal'; +import type { IDaemonPhasedRequest, IDaemonPhasedRequestResult } from '@rushstack/rush-daemon-protocol'; +import { type IOperationRunnerContext, OperationStatus } from '@microsoft/rush-lib'; + +import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import { + TEST_ENGINE_SHAPE, + TestOperationRunner, + TestPhasedRequestClient, + createRoutingFixture +} from './PhasedRequestRouterTestUtilities'; +import type { ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; + +const OPERATION_A: string = 'project-a (_phase:test)'; +const OPERATION_B: string = 'project-b (_phase:test)'; +const PROMPT_CANCELLATION_MS: number = 1000; + +function createRequest(requestId: string, operationId: string): IDaemonPhasedRequest { + return { + commandName: 'build', + commandOrigin: 'built-in', + engineShape: TEST_ENGINE_SHAPE, + environment: {}, + operationSelection: [{ enabledState: true, operationId }], + requestId + }; +} + +interface IHangingOperation { + readonly started: Promise; + readonly terminated: Promise; + readonly signals: AbortSignal[]; + readonly release: () => void; +} + +/** An operation that only finishes when released, or when its hard-abort signal fires (like a killed process). */ +function createHangingOperation(): { + hanging: IHangingOperation; + actionAsync: (terminal: ITerminal, context: IOperationRunnerContext) => Promise; +} { + let onStarted: () => void = () => undefined; + let onTerminated: () => void = () => undefined; + let release: () => void = () => undefined; + const released: Promise = new Promise((resolve) => (release = resolve)); + const signals: AbortSignal[] = []; + const hanging: IHangingOperation = { + started: new Promise((resolve) => (onStarted = resolve)), + terminated: new Promise((resolve) => (onTerminated = resolve)), + signals, + release: () => release() + }; + const actionAsync = async ( + terminal: ITerminal, + context: IOperationRunnerContext + ): Promise => { + const { abortSignal } = context; + if (!abortSignal) throw new Error('Expected a hard-abort signal for daemon operations.'); + signals.push(abortSignal); + onStarted(); + const aborted: Promise = new Promise((resolve) => + abortSignal.addEventListener('abort', () => resolve(), { once: true }) + ); + await Promise.race([aborted, released]); + if (abortSignal.aborted) { + onTerminated(); + return OperationStatus.Aborted; + } + }; + return { hanging, actionAsync }; +} + +function createFixture( + actionAsync: (terminal: ITerminal, context: IOperationRunnerContext) => Promise +): ITestRoutingFixture { + return createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A, OperationStatus.Success, actionAsync)], + [OPERATION_B, new TestOperationRunner(OPERATION_B)] + ]), + [], + { supportsTerminateRunning: true } + ); +} + +describe('phased request client cancellation', () => { + it('terminates running operations when the last client cancels and releases the graph promptly', async () => { + const { hanging, actionAsync } = createHangingOperation(); + const fixture: ITestRoutingFixture = createFixture(actionAsync); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const client: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const cancelled: Promise = router.executeAsync( + createRequest('cancelled', OPERATION_A), + client + ); + await hanging.started; + + const cancelledAt: number = Date.now(); + client.abortController.abort(); + const result: IDaemonPhasedRequestResult = await cancelled; + + expect(Date.now() - cancelledAt).toBeLessThan(PROMPT_CANCELLATION_MS); + await hanging.terminated; + expect(abortSpy).toHaveBeenCalledWith({ terminateRunning: true }); + expect(result).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(result.operationResults).toEqual([ + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Aborted }) + ]); + // The aborted operation is not retained, and the next client is admitted immediately. + expect(fixture.graph.resultByOperation.size).toBe(0); + const next: IDaemonPhasedRequestResult = await router.executeAsync( + createRequest('next', OPERATION_B), + new TestPhasedRequestClient('two') + ); + expect(next).toMatchObject({ exitCode: 0, outcome: 'success' }); + }); + + it('only detaches a cancelling client while another live client still needs the running work', async () => { + const { hanging, actionAsync } = createHangingOperation(); + const fixture: ITestRoutingFixture = createFixture(actionAsync); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const cancelledClient: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const cancelled: Promise = router.executeAsync( + createRequest('cancelled', OPERATION_A), + cancelledClient + ); + const continuing: Promise = router.executeAsync( + createRequest('continuing', OPERATION_A), + new TestPhasedRequestClient('two') + ); + await hanging.started; + const abortCallsBeforeCancellation: number = abortSpy.mock.calls.length; + + cancelledClient.abortController.abort(); + const cancelledResult: IDaemonPhasedRequestResult = await cancelled; + + expect(cancelledResult).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(abortSpy).toHaveBeenCalledTimes(abortCallsBeforeCancellation); + expect(hanging.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([false]); + + hanging.release(); + expect(await continuing).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + }); + + it('answers a cancelling client at once when an accepted pending client will join the batch', async () => { + const { hanging, actionAsync } = createHangingOperation(); + const fixture: ITestRoutingFixture = createFixture(actionAsync); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + let onReconciling: () => void = () => undefined; + const reconciling: Promise = new Promise((resolve) => (onReconciling = resolve)); + let releaseReconcile: () => void = () => undefined; + const reconcileReleased: Promise = new Promise((resolve) => (releaseReconcile = resolve)); + fixture.session.onReconcileAsync = async () => { + onReconciling(); + await reconcileReleased; + }; + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const cancelledClient: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const cancelled: Promise = router.executeAsync( + createRequest('cancelled', OPERATION_A), + cancelledClient + ); + await reconciling; + // Accepted while the batch is still being prepared, so it joins once preparation finishes. + const continuing: Promise = router.executeAsync( + createRequest('continuing', OPERATION_A), + new TestPhasedRequestClient('two') + ); + // Let the continuing request finish preparation and enter the pending queue. + for (let tick: number = 0; tick < 20; tick++) { + await new Promise((resolve) => setImmediate(resolve)); + } + + cancelledClient.abortController.abort(); + expect(await cancelled).toMatchObject({ aborted: true, outcome: 'aborted' }); + + releaseReconcile(); + await hanging.started; + expect(hanging.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([false]); + expect(abortSpy).not.toHaveBeenCalledWith({ terminateRunning: true }); + hanging.release(); + expect(await continuing).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); + }); +}); diff --git a/libraries/rush-daemon/src/test/PhasedRequestRouterTestUtilities.ts b/libraries/rush-daemon/src/test/PhasedRequestRouterTestUtilities.ts index 41f17440d7..58a52c478d 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestRouterTestUtilities.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestRouterTestUtilities.ts @@ -137,14 +137,16 @@ export class TestOperationRunner implements IOperationRunner { public closeCount: number = 0; public runCount: number = 0; - readonly #actionAsync: ((terminal: ITerminal) => Promise) | undefined; + readonly #actionAsync: + | ((terminal: ITerminal, context: IOperationRunnerContext) => Promise) + | undefined; readonly #status: OperationStatus; public readonly name: string; public constructor( name: string, status: OperationStatus = OperationStatus.Success, - actionAsync?: (terminal: ITerminal) => Promise + actionAsync?: (terminal: ITerminal, context: IOperationRunnerContext) => Promise ) { this.name = name; this.#status = status; @@ -160,8 +162,8 @@ export class TestOperationRunner implements IOperationRunner { this.runCount++; return context.runWithTerminalAsync( async (terminal: ITerminal): Promise => { - await this.#actionAsync?.(terminal); - return this.#status; + const status: void | OperationStatus = await this.#actionAsync?.(terminal, context); + return status ?? this.#status; }, { createLogFile: false, logFileSuffix: '' } ); @@ -216,7 +218,8 @@ export class TestRoutingWorkspaceSession implements IWorkspaceSession { export function createRoutingFixture( runnerById: ReadonlyMap, - dependencies: ReadonlyArray = [] + dependencies: ReadonlyArray = [], + graphOptionOverrides: Partial = {} ): ITestRoutingFixture { const operations: Map = new Map(); const runners: Map = new Map(runnerById); @@ -252,7 +255,8 @@ export function createRoutingFixture( destinations: [new MockWritable()], parallelism: 1, pauseNextIteration: false, - quietMode: false + quietMode: false, + ...graphOptionOverrides }; // The package's bundled public declarations and deep-import declarations describe the same runtime classes, // but TypeScript assigns them distinct recursive identities. diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index 89a0dbd3f7..b04cb177b4 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -815,6 +815,7 @@ export class PhasedScriptAction extends BaseScriptAction i getInputsSnapshotAsync: getGraphInputsSnapshotAsync, abortController: this.sessionAbortController, closeRunnersOnAbort: !onEngine, + supportsTerminateRunning: !!onEngine, telemetry: executionTelemetryHandler }; @@ -1066,7 +1067,7 @@ async function disposeEngineGraphAsync( graph.abortController.abort(); const errors: unknown[] = []; for (const cleanupAsync of [ - () => graph.abortCurrentIterationAsync(), + () => graph.abortCurrentIterationAsync({ terminateRunning: true }), () => graph.closeRunnersAsync(), async () => { await cobuildConfiguration?.destroyLockProviderAsync(); diff --git a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts index 5575bf77ab..e6b4316fbe 100644 --- a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts @@ -106,8 +106,11 @@ export interface IOperationGraph { /** * Abort the current execution iteration, if any. Operations that have already started * will run to completion; only operations that have not yet begun will be aborted. + * + * If `options.terminateRunning` is true and the graph supports it, operations that are already running are also + * signaled to terminate (via `IOperationRunnerContext.abortSignal`) and are reported as `Aborted`. */ - abortCurrentIterationAsync(): Promise; + abortCurrentIterationAsync(options?: { terminateRunning?: boolean }): Promise; /** * Cleans up any resources used by the operation runners, if applicable. diff --git a/libraries/rush-lib/src/logic/operations/IOperationRunner.ts b/libraries/rush-lib/src/logic/operations/IOperationRunner.ts index 1e820a0af4..f422083d85 100644 --- a/libraries/rush-lib/src/logic/operations/IOperationRunner.ts +++ b/libraries/rush-lib/src/logic/operations/IOperationRunner.ts @@ -63,6 +63,14 @@ export interface IOperationRunnerContext { */ readonly shouldRunnerPersist: boolean; + /** + * When defined, this signal is aborted if the host requests termination of in-progress work + * (see `IOperationGraph.abortCurrentIterationAsync` with `terminateRunning: true`). Runners that observe it + * should promptly stop any work they started (for example, kill their child process tree) and return + * `OperationStatus.Aborted`. It is `undefined` when the graph does not support terminating running operations. + */ + readonly abortSignal?: AbortSignal; + /** * The environment in which the operation is being executed. * A return value of `undefined` indicates that it should inherit the environment from the parent process. diff --git a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts index ed7c448268..129d5c04c5 100644 --- a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts +++ b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts @@ -51,6 +51,8 @@ export interface IOperationExecutionRecordContext { invalidate?: (operations: Iterable, reason: string) => void; inputsSnapshot: IInputsSnapshot | undefined; maxParallelism: number; + /** Aborted when the host requests termination of running operations in this iteration. */ + terminateSignal?: AbortSignal; /** * Optional structured event sink for dual-emit. When present, every status @@ -245,6 +247,10 @@ export class OperationExecutionRecord implements IOperationRunnerContext, IOpera return this.#context.createEnvironment?.(this); } + public get abortSignal(): AbortSignal | undefined { + return this.#context.terminateSignal; + } + public getInvalidateCallback(): (reason: string) => void { const invalidateFn: ((operations: Iterable, reason: string) => void) | undefined = this.#context.invalidate; diff --git a/libraries/rush-lib/src/logic/operations/OperationGraph.ts b/libraries/rush-lib/src/logic/operations/OperationGraph.ts index 17a57096d0..a3ed25bb1c 100644 --- a/libraries/rush-lib/src/logic/operations/OperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/OperationGraph.ts @@ -54,6 +54,12 @@ export interface IOperationGraphOptions { abortController: AbortController; /** Hosts with awaited lifetime cleanup can disable the legacy fire-and-forget abort cleanup. */ closeRunnersOnAbort?: boolean; + /** + * If true, runners receive `IOperationRunnerContext.abortSignal`, and + * `abortCurrentIterationAsync({ terminateRunning: true })` terminates operations that are already running. + * Runners may isolate their child processes (e.g. in a separate process group) to support this. + */ + supportsTerminateRunning?: boolean; isWatch?: boolean; pauseNextIteration?: boolean; @@ -82,6 +88,7 @@ interface IStatefulExecutionContext { */ interface IExecutionIterationContext extends IOperationExecutionRecordContext { abortController: AbortController; + terminateController: AbortController | undefined; terminal: CollatedTerminal; records: Map; @@ -176,6 +183,7 @@ export class OperationGraph implements IOperationGraph { // Immutable properties from options readonly #isWatch: boolean; + readonly #supportsTerminateRunning: boolean; readonly #telemetry: IOperationGraphTelemetry | undefined; readonly #getInputsSnapshotAsync: (() => Promise) | undefined; @@ -220,11 +228,13 @@ export class OperationGraph implements IOperationGraph { abortController, isWatch = false, pauseNextIteration = false, + supportsTerminateRunning = false, telemetry, getInputsSnapshotAsync } = options; this.operations = operations; + this.#supportsTerminateRunning = supportsTerminateRunning; this.#maxParallelism = maxParallelism; this.#parallelism = coerceParallelism(parallelism, maxParallelism, 1); @@ -627,9 +637,12 @@ export class OperationGraph implements IOperationGraph { return true; } - public async abortCurrentIterationAsync(): Promise { + public async abortCurrentIterationAsync(options?: { terminateRunning?: boolean }): Promise { const iteration: IExecutionIterationContext | undefined = this.#currentIteration; if (iteration) { + if (options?.terminateRunning) { + iteration.terminateController?.abort(); + } iteration.abortController.abort(); try { await iteration.promise; @@ -709,10 +722,16 @@ export class OperationGraph implements IOperationGraph { return hooks.createEnvironmentForOperation.call({ ...process.env }, record); } + const terminateController: AbortController | undefined = this.#supportsTerminateRunning + ? new AbortController() + : undefined; + // Convert the developer graph to the mutable execution graph const iterationContext: IExecutionIterationContext = { iterationId: this.#nextIterationId++, abortController, + terminateController, + terminateSignal: terminateController?.signal, startTime, streamCollator, terminal, diff --git a/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts b/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts index c76fce6931..627236e23b 100644 --- a/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts +++ b/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts @@ -4,7 +4,7 @@ import type * as child_process from 'node:child_process'; import * as path from 'node:path'; -import { Path } from '@rushstack/node-core-library'; +import { Path, SubprocessTerminator } from '@rushstack/node-core-library'; import { type ITerminal, type ITerminalProvider, TerminalProviderSeverity } from '@rushstack/terminal'; import type { IPhase } from '../../api/CommandLineConfiguration'; @@ -103,7 +103,7 @@ export class ShellOperationRunner implements IOperationRunner { const { rushConfiguration, projectFolder } = this.#rushProject; - const { environment: initialEnvironment } = context; + const { environment: initialEnvironment, abortSignal } = context; const childProcessReporter: IOperationChildProcessReporter | undefined = !IS_WINDOWS && isHeftCommand(commandToRun) ? context.createChildProcessReporter() : undefined; @@ -117,8 +117,28 @@ export class ShellOperationRunner implements IOperationRunner { }, initialEnvironment, additionalEnvironment: childProcessReporter?.environment, - stdio: childProcessReporter?.stdio + stdio: childProcessReporter?.stdio, + // Isolate the process tree so that a hard abort can terminate it. + connectSubprocessTerminator: abortSignal !== undefined }); + const terminateProcessTree: () => void = () => { + try { + if (!IS_WINDOWS && subProcess.pid !== undefined && typeof subProcess.exitCode === 'number') { + // The shell already exited, but descendants in its process group may still hold its stdio open. + // killProcessTree() is a no-op in that state, so signal the process group directly. + killExitedProcessGroup(subProcess.pid); + } else { + SubprocessTerminator.killProcessTree(subProcess, SubprocessTerminator.RECOMMENDED_OPTIONS); + } + } catch (error) { + terminal.writeErrorLine(`Failed to terminate the operation process tree: ${error}`); + } + }; + if (abortSignal?.aborted) { + terminateProcessTree(); + } else { + abortSignal?.addEventListener('abort', terminateProcessTree, { once: true }); + } let reporterError: Error | undefined; const reporterDrainPromise: Promise = childProcessReporter ? childProcessReporter @@ -164,9 +184,14 @@ export class ShellOperationRunner implements IOperationRunner { const [{ exitCode, signal }]: [ { readonly exitCode: number | null; readonly signal: NodeJS.Signals | null }, void - ] = await Promise.all([closePromise, reporterDrainPromise]); + ] = await Promise.all([closePromise, reporterDrainPromise]).finally(() => { + abortSignal?.removeEventListener('abort', terminateProcessTree); + }); - if (signal) { + if (abortSignal?.aborted) { + terminal.writeLine('Terminated because the operation was aborted.'); + return OperationStatus.Aborted; + } else if (signal) { // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. context.error = new OperationError('error', `Terminated by signal: ${signal}`); return OperationStatus.Failure; @@ -195,6 +220,17 @@ export class ShellOperationRunner implements IOperationRunner { } } +function killExitedProcessGroup(pid: number): void { + try { + // The process group ID cannot be reused while any member of the group is still alive. + process.kill(-pid, 'SIGKILL'); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ESRCH') { + throw error; + } + } +} + /** * Returns whether a lifecycle command directly launches Heft. * diff --git a/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerAbort.test.ts b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerAbort.test.ts new file mode 100644 index 0000000000..930f63170d --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerAbort.test.ts @@ -0,0 +1,199 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('../OperationStateFile'); +jest.mock('../ProjectLogWritable', () => { + const actual = jest.requireActual('../ProjectLogWritable'); + const { MockWritable } = jest.requireActual('@rushstack/terminal'); + return { ...actual, initializeProjectLogFilesAsync: jest.fn(async () => new MockWritable()) }; +}); + +import { spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { SubprocessTerminator } from '@rushstack/node-core-library'; +import { MockWritable } from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import { type ILifecycleCommandOptions, Utilities } from '../../../utilities/Utilities'; +import type { IExecutionResult } from '../IOperationExecutionResult'; +import { Operation } from '../Operation'; +import { OperationGraph } from '../OperationGraph'; +import { OperationStatus } from '../OperationStatus'; +import { ShellOperationRunner } from '../ShellOperationRunner'; + +// Spawns a grandchild that never exits, reports its PID, then waits forever itself. +const NEVER_ENDING_TREE_SCRIPT: string = ` + const grandchild = require('node:child_process').spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' }); + process.stdout.write('grandchild=' + grandchild.pid + '\\n'); + setInterval(() => {}, 1000); +`; + +// Spawns a grandchild that inherits stdout and never exits, reports its PID, then exits itself. +const EXITED_PARENT_TREE_SCRIPT: string = ` + const grandchild = require('node:child_process').spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: ['ignore', 'inherit', 'ignore'] }); + process.stdout.write('grandchild=' + grandchild.pid + '\\n', () => process.exit(0)); +`; + +function isAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +async function waitForExitAsync(pid: number, timeoutMs: number): Promise { + const deadline: number = Date.now() + timeoutMs; + while (isAlive(pid)) { + if (Date.now() >= deadline) return false; + await delayAsync(20); + } + return true; +} + +describe('ShellOperationRunner hard abort', () => { + let child: ChildProcess | undefined; + let grandchildPid: number | undefined; + let spawnOptions: ILifecycleCommandOptions | undefined; + + function createGraph( + supportsTerminateRunning: boolean, + script: string = NEVER_ENDING_TREE_SCRIPT + ): { + graph: OperationGraph; + grandchildStarted: Promise; + } { + let onGrandchild: (pid: number) => void = () => undefined; + const grandchildStarted: Promise = new Promise((resolve) => (onGrandchild = resolve)); + jest.spyOn(Utilities, 'executeLifecycleCommandAsync').mockImplementation((command, options) => { + spawnOptions = options; + child = spawn(process.execPath, ['-e', script], { + stdio: ['ignore', 'pipe', 'pipe'], + detached: !!options.connectSubprocessTerminator && SubprocessTerminator.RECOMMENDED_OPTIONS.detached + }); + child.stdout!.on('data', (chunk: Buffer) => { + const match: RegExpMatchArray | null = chunk.toString().match(/grandchild=(\d+)/); + if (match) { + grandchildPid = Number(match[1]); + onGrandchild(grandchildPid); + } + }); + return child; + }); + const phase: IPhase = { + name: 'build', + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + dependencies: { self: new Set(), upstream: new Set() }, + isSynthetic: false, + logFilenameIdentifier: 'build', + missingScriptBehavior: 'silent' + }; + const project: RushConfigurationProject = { + packageName: 'sleeper', + projectFolder: __dirname, + rushConfiguration: { commonTempFolder: __dirname } + } as RushConfigurationProject; + const runner: ShellOperationRunner = new ShellOperationRunner({ + phase, + rushProject: project, + displayName: 'sleeper', + initialCommand: 'node sleeper.js', + incrementalCommand: undefined, + commandForHash: 'node sleeper.js', + ignoredParameterValues: [] + }); + const operation: Operation = new Operation({ phase, project, runner, logFilenameIdentifier: 'sleeper' }); + const graph: OperationGraph = new OperationGraph(new Set([operation]), { + quietMode: true, + debugMode: false, + parallelism: 1, + allowOversubscription: true, + destinations: [new MockWritable()], + abortController: new AbortController(), + supportsTerminateRunning + }); + return { graph, grandchildStarted }; + } + + afterEach(async () => { + jest.restoreAllMocks(); + const lastChild: ChildProcess | undefined = child; + const lastGrandchildPid: number | undefined = grandchildPid; + child = undefined; + grandchildPid = undefined; + spawnOptions = undefined; + if (lastGrandchildPid !== undefined && isAlive(lastGrandchildPid)) { + process.kill(lastGrandchildPid, 'SIGKILL'); + } + if (lastChild && lastChild.exitCode === null && lastChild.signalCode === null) { + const closed: Promise = once(lastChild, 'close'); + lastChild.kill('SIGKILL'); + await closed; + } + }); + + it('kills the running process tree and reports Aborted without waiting for it to finish', async () => { + const { graph, grandchildStarted } = createGraph(true); + const execution: Promise = graph.executeAsync({}); + const pid: number = await grandchildStarted; + expect(spawnOptions?.connectSubprocessTerminator).toBe(true); + + const abortStart: number = Date.now(); + await graph.abortCurrentIterationAsync({ terminateRunning: true }); + const result: IExecutionResult = await execution; + + expect(Date.now() - abortStart).toBeLessThan(5000); + expect(result.status).toBe(OperationStatus.Aborted); + const [record] = [...result.operationResults.values()]; + expect(record.status).toBe(OperationStatus.Aborted); + expect(record.error).toBeUndefined(); + expect(child!.exitCode !== null || child!.signalCode !== null).toBe(true); + expect(await waitForExitAsync(pid, 5000)).toBe(true); + // Aborted operations are not retained as the last execution result. + expect(graph.resultByOperation.size).toBe(0); + }, 20000); + + // Process groups are POSIX-only; Windows terminates the tree via TaskKill while the parent is alive. + (process.platform === 'win32' ? it.skip : it)( + 'kills descendants that keep the output open after the shell itself has exited', + async () => { + const { graph, grandchildStarted } = createGraph(true, EXITED_PARENT_TREE_SCRIPT); + const execution: Promise = graph.executeAsync({}); + const pid: number = await grandchildStarted; + const parent: ChildProcess = child!; + if (parent.exitCode === null) { + await once(parent, 'exit'); + } + expect(isAlive(pid)).toBe(true); + + await graph.abortCurrentIterationAsync({ terminateRunning: true }); + const result: IExecutionResult = await execution; + + expect(result.status).toBe(OperationStatus.Aborted); + expect(await waitForExitAsync(pid, 5000)).toBe(true); + }, + 20000 + ); + + it('does not isolate or terminate processes when the graph does not support it', async () => { + const { graph, grandchildStarted } = createGraph(false); + const execution: Promise = graph.executeAsync({}); + await grandchildStarted; + expect(spawnOptions?.connectSubprocessTerminator).toBe(false); + + // A soft abort only prevents unstarted work; the running process keeps going. + const abortPromise: Promise = graph.abortCurrentIterationAsync({ terminateRunning: true }); + await delayAsync(300); + expect(child!.exitCode).toBeNull(); + expect(child!.signalCode).toBeNull(); + + child!.kill('SIGKILL'); + await abortPromise; + expect((await execution).status).toBe(OperationStatus.Failure); + }, 20000); +}); From 982b33b16c9571b518747c68cfd1ebe1a1ccdd14 Mon Sep 17 00:00:00 2001 From: Sean Larkin Date: Fri, 25 Sep 2026 17:42:18 -0700 Subject: [PATCH 015/265] [rush-daemon] Invalidate warm operations whose declared outputs were deleted or changed (#6069) * [rush-daemon] Invalidate warm operations whose declared outputs were deleted or changed Fixes #6058 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Force re-execution of invalidated outputs under legacy skip detection Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush-daemon] Keep output fingerprints until legacy skip cleanup succeeds Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../output-invalidation_2026-09-23.json | 10 ++ libraries/rush-daemon/README.md | 6 +- .../src/OperationOutputFingerprints.ts | 128 ++++++++++++++++++ .../src/ProductionDaemonRequestResolver.ts | 11 +- .../ProductionDaemonRequestResolver.test.ts | 60 ++++++++ 5 files changed, 212 insertions(+), 3 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/output-invalidation_2026-09-23.json create mode 100644 libraries/rush-daemon/src/OperationOutputFingerprints.ts diff --git a/common/changes/@rushstack/rush-daemon/output-invalidation_2026-09-23.json b/common/changes/@rushstack/rush-daemon/output-invalidation_2026-09-23.json new file mode 100644 index 0000000000..217487259d --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/output-invalidation_2026-09-23.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Re-run or restore a warm operation when its declared output folders were deleted or changed outside the daemon, instead of reporting it as up to date.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 4ee695eba8..8f745e2993 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -135,7 +135,11 @@ fallback can run immediately after a completed single-client warm request withou A real native command holding the lock causes preparation or execution to be refused; there is no lock bypass or automatic retry. A later explicit request can retry after contention ends, including contention during the first engine initialization. A dirty native lock left by another command invalidates retained successes so the native -incremental/cache pipeline can reconcile possibly changed ignored outputs. Installation validity is also checked on +incremental/cache pipeline can reconcile possibly changed ignored outputs. Declared `outputFolderNames` are also +fingerprinted (one `stat` per folder: existence, identity and modification time) when an operation succeeds or is +restored from cache; a request whose reconciliation finds a missing or changed output folder (for example after +`rm -rf lib`, `git clean -xdf` or `heft clean`) invalidates only that operation, so it is re-executed or restored from +the build cache. In-place edits of nested output files are not detected. Installation validity is also checked on every snapshot refresh. Disposal stops new leases, awaits an outstanding lease, then aborts the graph lifetime and awaits runner/provider cleanup. The existing operation-completion cleanup is unchanged. diff --git a/libraries/rush-daemon/src/OperationOutputFingerprints.ts b/libraries/rush-daemon/src/OperationOutputFingerprints.ts new file mode 100644 index 0000000000..7536718113 --- /dev/null +++ b/libraries/rush-daemon/src/OperationOutputFingerprints.ts @@ -0,0 +1,128 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { + OperationStatus, + type IOperationExecutionResult, + type IOperationGraph, + type Operation +} from '@microsoft/rush-lib'; + +const PLUGIN_NAME: 'DaemonOperationOutputFingerprints' = 'DaemonOperationOutputFingerprints'; + +/** + * Retained results that allow the warm graph to skip an operation. Other statuses always re-run. + */ +const TRACKED_STATUSES: ReadonlySet = new Set([ + OperationStatus.Success, + OperationStatus.FromCache +]); + +interface IOutputFingerprint { + readonly record: IOperationExecutionResult; + readonly fingerprint: string; +} + +/** + * Detects retained successful operations whose declared output folders were changed outside the daemon. + * + * @remarks + * Build outputs are normally git-ignored, so they do not contribute to any operation state hash. Without + * this check, deleting an output folder (`rm -rf lib`, `git clean -xdf`, `heft clean`) leaves a warm graph + * that reports the operation as up to date. The fingerprint is deliberately cheap: one `stat` per declared + * output folder, capturing existence, identity and modification time. This detects deletion, recreation, + * and adding, removing or renaming direct children; it does not detect in-place edits of nested files. + */ +export class OperationOutputFingerprints { + readonly #fingerprints: Map = new Map(); + readonly #graph: IOperationGraph; + + public constructor(graph: IOperationGraph) { + this.#graph = graph; + graph.hooks.afterExecuteIterationAsync.tap( + PLUGIN_NAME, + (status: OperationStatus, records: ReadonlyMap) => { + this.#recordIteration(records); + return status; + } + ); + } + + /** + * Returns retained successful operations whose output folders no longer match the recorded fingerprint. + * + * @remarks + * Fingerprints of changed operations are forgotten only after all cleanup succeeded, so a failed + * reconciliation retries the output check on the next request instead of trusting the stale result. + */ + public getOperationsWithChangedOutputs(): Operation[] { + const changed: Operation[] = []; + for (const [operation, { record, fingerprint }] of this.#fingerprints) { + if (!this.#isRetained(operation, record)) { + this.#fingerprints.delete(operation); + } else if (getOutputFingerprint(operation) !== fingerprint) { + changed.push(operation); + } + } + for (const operation of changed) { + forgetLegacySkipState(operation); + } + for (const operation of changed) { + this.#fingerprints.delete(operation); + } + return changed; + } + + #recordIteration(records: ReadonlyMap): void { + for (const [operation, record] of records) { + // Only records produced by this iteration become the retained result; disabled operations keep + // their earlier record and fingerprint. + if (this.#isRetained(operation, record)) { + const fingerprint: string | undefined = getOutputFingerprint(operation); + if (fingerprint === undefined) { + this.#fingerprints.delete(operation); + } else { + this.#fingerprints.set(operation, { record, fingerprint }); + } + } + } + } + + #isRetained(operation: Operation, record: IOperationExecutionResult): boolean { + return this.#graph.resultByOperation.get(operation) === record && TRACKED_STATUSES.has(record.status); + } +} + +/** + * Without a build cache, Rush's legacy skip detection reports an operation as skipped when its recorded + * input state is unchanged, even though its outputs are gone. Remove that record (as the legacy skip logic + * does itself before executing) so the invalidated operation is executed instead. + */ +function forgetLegacySkipState(operation: Operation): void { + fs.rmSync( + path.join( + operation.associatedProject.projectRushTempFolder, + `package-deps_${operation.logFilenameIdentifier}.json` + ), + { force: true } + ); +} + +function getOutputFingerprint(operation: Operation): string | undefined { + const folderNames: ReadonlyArray | undefined = operation.settings?.outputFolderNames; + if (!folderNames?.length) { + return undefined; + } + const { projectFolder } = operation.associatedProject; + return folderNames + .map((folderName: string) => { + const stats: fs.Stats | undefined = fs.statSync(path.resolve(projectFolder, folderName), { + throwIfNoEntry: false + }); + return stats ? `${folderName}:${stats.ino}:${stats.mtimeMs}` : `${folderName}:missing`; + }) + .join('|'); +} diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 6467828ef5..69518fef3a 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -32,6 +32,7 @@ import { } from './WorkspaceEngineComponentFactory'; import type { IWorkspaceSession, IWorkspaceSessionComponents } from './WorkspaceSession'; import { EngineTerminalProvider } from './EngineTerminalProvider'; +import { OperationOutputFingerprints } from './OperationOutputFingerprints'; import { getDaemonShutdownReason } from './DaemonShutdownError'; import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; @@ -197,6 +198,9 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { } try { terminal.attach(engine.operationGraph); + const outputFingerprints: OperationOutputFingerprints = new OperationOutputFingerprints( + engine.operationGraph + ); const factory: WorkspaceEngineComponentFactory = new WorkspaceEngineComponentFactory({ createEngineComponentsAsync: async () => ({ ...engine, @@ -218,8 +222,11 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { shape: engine, refreshInputsOnEveryRequest: true, validateGraphInputsAsync: this.#validateGraphInputsAsync, - mapInvalidationsToOperationsAsync: async (invalidationOptions) => - getChangedOperations(invalidationOptions) + mapInvalidationsToOperationsAsync: async (invalidationOptions) => [ + ...getChangedOperations(invalidationOptions), + // Outputs are git-ignored and absent from state hashes, so check them separately. + ...outputFingerprints.getOperationsWithChangedOutputs() + ] }); const components: IWorkspaceSessionComponents = await factory.createAsync(options); return { diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 01f38bfd80..bdf1634744 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -1330,6 +1330,66 @@ process.exit(23); } }); + it('re-runs only the operation whose declared outputs were deleted after a warm build', async () => { + const fixture: IFixture = await createFixtureAsync(); + try { + await runAsync(fixture, 'initial', ['build']); + expect(runs(fixture)).toEqual(expect.arrayContaining(['a:one:', 'b:one:', 'c:one:'])); + expect((await runAsync(fixture, 'warm', ['build'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: false } + }); + const graph: IOperationGraph | undefined = fixture.session.operationGraph; + fs.rmSync(path.join(fixture.repoRoot, 'projects/a/lib'), { recursive: true }); + expect((await runAsync(fixture, 'unrelated', ['build', '--only', 'c'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: false } + }); + expect((await runAsync(fixture, 'deleted', ['build', '--to', 'b'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: true } + }); + expect(runs(fixture).slice(3)).toEqual(['a:one:']); + expect(fs.readFileSync(path.join(fixture.repoRoot, 'projects/a/lib/output.txt'), 'utf8')).toBe('one'); + fs.writeFileSync(path.join(fixture.repoRoot, 'projects/c/lib/extra.txt'), 'stray'); + expect((await runAsync(fixture, 'changed', ['build'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: true } + }); + expect(runs(fixture).slice(4)).toEqual(['c:one:']); + expect((await runAsync(fixture, 'unchanged', ['build'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: false } + }); + expect(runs(fixture)).toHaveLength(5); + expect(fixture.session.operationGraph).toBe(graph); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + + it('restores deleted outputs of a warm operation from the native build cache', async () => { + const fixture: IFixture = await createFixtureAsync(true); + try { + await runAsync(fixture, 'initial', ['build', '--to', 'b']); + fs.rmSync(path.join(fixture.repoRoot, 'projects/a/lib'), { recursive: true }); + const deleted: ITerminalExchange = await runAsync(fixture, 'deleted', ['build', '--to', 'b']); + expect(deleted.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + const { operationResults } = (deleted.terminal as { payload: IDaemonPhasedRequestResult }).payload; + expect(operationResults.filter((result) => result.status !== 'SKIPPED')).toEqual([ + expect.objectContaining({ operationId: 'a (compile)', status: 'FROM CACHE' }) + ]); + expect(runs(fixture)).toEqual(['a:one:', 'b:one:']); + expect(fs.readFileSync(path.join(fixture.repoRoot, 'projects/a/lib/output.txt'), 'utf8')).toBe('one'); + expect((await runAsync(fixture, 'restored', ['build', '--to', 'b'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: false } + }); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('reconciles changes made without a connected client and preserves an empty native selection', async () => { const fixture: IFixture = await createFixtureAsync(); let reconnected: DaemonRequestWireClient | undefined; From 9bba9ef639d3e2223940627030c62a5af767cad4 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 13:31:54 +0000 Subject: [PATCH 016/265] [rush-lib] Workspace fingerprint: ignore build state of projects nested under common/config (and 3 more) Swarm integration step 1; original commit 433c87f24f (merge of swarm/r05 at 190d77ea17). Commits folded into this step (4): - 40b5b24d72 [rush-lib] Workspace fingerprint: ignore build state of projects nested under common/config - 1880e7b2b7 [rush-daemon] Warm set: keep resource-free retained results under memory pressure and idle expiry - 07e908d32d [rush-daemon] Admission: don't spend a default wait budget behind a graph load or across an admitted request's execution - 190d77ea17 [rush-lib] Move the nested common/config fingerprint test to the end of its suite Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...ested-config-project_2026-09-28-12-25.json | 9 + ...5-warm-set-retention_2026-09-28-12-26.json | 9 + ...transition-admission_2026-09-28-12-45.json | 10 ++ ...5-warm-set-retention_2026-09-28-12-26.json | 10 ++ docs/rush/environment-variables.md | 4 +- libraries/rush-daemon/README.md | 16 +- .../src/WorkspaceRequestAdmission.ts | 140 ++++++++++++++- .../src/WorkspaceRequestLifecycle.ts | 51 +++--- libraries/rush-daemon/src/WorkspaceWarmSet.ts | 33 ++-- .../src/test/DaemonGraphTestFixture.ts | 3 + .../test/WorkspaceRequestAdmission.test.ts | 159 ++++++++++++++++++ .../test/WorkspaceTransitionAdmission.test.ts | 111 ++++++++++++ .../src/test/WorkspaceWarmSet.test.ts | 31 ++++ .../rush-lib/src/api/DaemonConfiguration.ts | 10 +- .../src/api/WorkspaceInputFingerprint.ts | 64 ++++++- .../test/WorkspaceInputFingerprint.test.ts | 61 +++++++ .../rush-lib/src/schemas/rush.schema.json | 4 +- 17 files changed, 671 insertions(+), 54 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r05-nested-config-project_2026-09-28-12-25.json create mode 100644 common/changes/@microsoft/rush/swarm-r05-warm-set-retention_2026-09-28-12-26.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-transition-admission_2026-09-28-12-45.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-warm-set-retention_2026-09-28-12-26.json create mode 100644 libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts diff --git a/common/changes/@microsoft/rush/swarm-r05-nested-config-project_2026-09-28-12-25.json b/common/changes/@microsoft/rush/swarm-r05-nested-config-project_2026-09-28-12-25.json new file mode 100644 index 0000000000..9ae1f6188c --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r05-nested-config-project_2026-09-28-12-25.json @@ -0,0 +1,9 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Daemon: running the operations of a Rush project folder nested under common/config no longer changes the workspace configuration fingerprint and forces a graph reload on the next request.", + "type": "none" + } + ] +} diff --git a/common/changes/@microsoft/rush/swarm-r05-warm-set-retention_2026-09-28-12-26.json b/common/changes/@microsoft/rush/swarm-r05-warm-set-retention_2026-09-28-12-26.json new file mode 100644 index 0000000000..c81cf84f7a --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r05-warm-set-retention_2026-09-28-12-26.json @@ -0,0 +1,9 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Document that daemon warm-set idle expiry and memory budget do not evict retained results of resource-free projects.", + "type": "none" + } + ] +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-transition-admission_2026-09-28-12-45.json b/common/changes/@rushstack/rush-daemon/swarm-r05-transition-admission_2026-09-28-12-45.json new file mode 100644 index 0000000000..73f03d8079 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-transition-admission_2026-09-28-12-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Admission: a client-default wait budget no longer runs while another request loads or reloads the workspace graph, or while its own admitted request is routed and executed, so early arrivals and graph-gate waiters no longer fail with wait-timeout behind a slow cold load.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-warm-set-retention_2026-09-28-12-26.json b/common/changes/@rushstack/rush-daemon/swarm-r05-warm-set-retention_2026-09-28-12-26.json new file mode 100644 index 0000000000..dc03220e72 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-warm-set-retention_2026-09-28-12-26.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Warm set: memory pressure and idle expiry release only runners and watchers, so retained results of resource-free projects keep warm no-op skips; the pressure warning is reported once per state.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/docs/rush/environment-variables.md b/docs/rush/environment-variables.md index 35f266c2f2..5ca22ad77b 100644 --- a/docs/rush/environment-variables.md +++ b/docs/rush/environment-variables.md @@ -27,8 +27,8 @@ variables and invalid values are errors, not ignored settings. | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | `30` | Maximum request admission wait. Nonnegative seconds, at most 2147483.647; converted to whole milliseconds by rounding down. Per-invocation `--wait-timeout` or `--no-wait` takes precedence. Overrides `queueTimeoutSeconds`. | | `RUSH_DAEMON_WATCH` | `0` | Observe requested warm projects between requests. Never schedules builds and does not enable `--watch` mode. Root/config guards and request-time input reconciliation remain active when disabled. Overrides `watch`. | | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | `0` | Enable explicitly configured `operationSettings[].daemonIpc` Node workers for supported incremental daemon builds. Does not convert arbitrary shell scripts into persistent workers. Overrides `usePersistentIpcRunners`. | -| `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | `300` | Idle expiration for retained runners, project watchers and results. Positive seconds, at most 2147483.647. Overrides `warmIdleTimeoutSeconds`. | -| `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | `512` | Best-effort sampled RSS budget in MiB. Positive number, at most 9007199254740991. Not a hard process-tree memory ceiling; active/protected work is exempt. Overrides `warmMemoryBudgetMB`. | +| `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | `300` | Idle expiration for retained runners and project watchers, together with those projects' results. Results of resource-free (shell/null) projects do not expire. Positive seconds, at most 2147483.647. Overrides `warmIdleTimeoutSeconds`. | +| `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | `512` | Best-effort sampled RSS budget in MiB. Positive number, at most 9007199254740991. Not a hard process-tree memory ceiling; active/protected work and results of resource-free projects are exempt. Overrides `warmMemoryBudgetMB`. | | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | `20` | Best-effort retained-project limit. Positive safe integer, at most 9007199254740991; never trims the requested execution set. Overrides `warmSetMaxProjects`. | | `RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY` | `0` | Rank retention using measured time saved, frequency and memory, with conservative LRU fallback. Never speculatively executes scripts. Overrides `autoWarmByTelemetry`. | | `RUSH_DAEMON_EXPERIMENTAL` | `0` | Enable the experimental `rush-client daemon graph` command surface. Does not start a daemon or enable builds; no corresponding `rush.json` property. | diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 31171ac55d..2e489bbb43 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -313,9 +313,9 @@ preserves its records and diagnostics without failing an otherwise successful bu | Policy | Runtime behavior | | --- | --- | | `watch` | Retains host observation of requested warm projects between requests when true. False (the default) keeps root/config guards only. Never schedules builds. | -| `warmIdleTimeoutSeconds` | Expires unused project runners, watchers and retained results after requests finish. Unchanged requests refresh recency too. | +| `warmIdleTimeoutSeconds` | Expires unused project runners and watchers, together with those projects' retained results, after requests finish. Unchanged requests refresh recency too. Projects whose only retained state is operation results from resource-free (shell/null) runners do not expire: those results are revalidated on every request and stay until the generation ends, so an agent that returns after a long pause still gets no-op skips. | | `warmSetMaxProjects` | Limits the projects that hold warm **resources** (an active runner such as a persistent IPC child, or a `watch: true` file watcher). The lowest-ranked holders are released (runners closed, watchers removed, records deleted); executing/prepared and explicitly protected work is exempt. Projects whose only retained state is operation results from resource-free (shell/null) runners neither count toward nor are evicted for this limit, so no-op re-requests of large workspaces stay skipped. | -| `warmMemoryBudgetMB` | Attempts idle eviction under sampled daemon-plus-measured-child RSS pressure. The comparison uses the **whole daemon process RSS** (graph, Node heap and retained records, typically 130-190 MiB) plus measured child RSS, so a budget below the daemon's baseline evicts every idle project on each pass and disables warm skipping. Never treats cache files as memory or claims a hard RSS ceiling. | +| `warmMemoryBudgetMB` | Attempts idle eviction of resource-holding projects under sampled daemon-plus-measured-child RSS pressure. The comparison uses the **whole daemon process RSS** (graph, Node heap and retained records, typically 130-190 MiB for a small workspace and more for a large one) plus measured child RSS, so a budget below the daemon's baseline releases every idle runner and watcher on each pass. Retained results of resource-free projects are not evicted for the budget, so warm skipping keeps working, and the pressure warning is reported once per distinct state. Never treats cache files as memory or claims a hard RSS ceiling. | | `autoWarmByTelemetry` | Promotes already-requested high-value work instead of pure LRU. Never schedules or executes speculative scripts. | One deterministic best-first comparator is shared by retention and reverse-order eviction. With complete @@ -459,10 +459,16 @@ the static built-in command policy (`SHARED-BUILD`, `SHARED-READ`, or `EXCLUSIVE built-in names fail closed to `EXCLUSIVE`, including plugin replacements of built-in names. Queued clients receive ordered, one-based position controls and can request fail-fast or bounded waiting. One progress channel covers both workspace admission and the temporary phased graph-execution gate. An explicit `noWait` or `waitTimeoutMs` is one -absolute deadline for both waits. When the client marks `waitTimeoutMs` as its default (`waitTimeoutIsDefault`), the -deadline bounds workspace admission only: a `SHARED-BUILD` request that arrives after the current batch has closed waits +absolute deadline for both waits. When the client marks `waitTimeoutMs` as its default (`waitTimeoutIsDefault`), it is +a budget that only contention spends: a `SHARED-BUILD` request that arrives after the current batch has closed waits on the graph-execution gate without a deadline, because it is queued only behind running compatible shared builds, and -then runs in the next batch. Cancellation, disconnect, or queue-output failure removes queued work before it can execute. +then runs in the next batch. A request queued behind another request that holds exclusive workspace admission to load +or reload the graph does not spend the budget during that load, so every build that arrives while the first build +after startup loads the graph is admitted when the load finishes. The budget does run while that other request still +waits for exclusive admission, so requests behind a reload that cannot start, for example behind a long build, still +time out. Routing and executing an admitted request do not spend the budget either: a request that re-enters +workspace admission to reload the graph after its inputs changed keeps the budget it had when it was admitted. +Cancellation, disconnect, or queue-output failure removes queued work before it can execute. A requesting client receives only its enabled dependency closure's WS1 raw chunks and structured events through backpressured, ordered callbacks, followed exactly once by a typed final command result after all preceding output drains. The result translates only that client's operation subset to Rush's success, warning, failure, or abort exit diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index a541976266..edf357db67 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -96,12 +96,78 @@ class QueuePositionWriter { } } +/** + * Reports whether a request that other requests wait behind is doing work on their behalf, such as loading the + * workspace graph that they need. + */ +export class AdmissionProgress { + readonly #listeners: Set<() => void> = new Set(); + #active: boolean = false; + + public get active(): boolean { + return this.#active; + } + + public setActive(active: boolean): void { + if (this.#active === active) return; + this.#active = active; + for (const listener of [...this.#listeners]) listener(); + } + + public subscribe(listener: () => void): () => void { + this.#listeners.add(listener); + return () => this.#listeners.delete(listener); + } +} + +/** A wait budget that is spent only while `progress` is inactive. */ +class ProgressPausedBudget { + readonly #onExhausted: () => void; + readonly #progress: AdmissionProgress; + readonly #unsubscribe: () => void; + #remainingMs: number; + #runningSinceMs: number | undefined; + #timer: ReturnType | undefined; + + public constructor(remainingMs: number, progress: AdmissionProgress, onExhausted: () => void) { + this.#remainingMs = remainingMs; + this.#progress = progress; + this.#onExhausted = onExhausted; + this.#unsubscribe = progress.subscribe(() => this.#update()); + this.#update(); + } + + /** Stops spending and returns the unspent budget. */ + public stop(): number { + this.#unsubscribe(); + this.#pause(); + return this.#remainingMs; + } + + #update(): void { + if (this.#progress.active) { + this.#pause(); + } else if (this.#runningSinceMs === undefined) { + this.#runningSinceMs = Date.now(); + this.#timer = setTimeout(this.#onExhausted, this.#remainingMs); + } + } + + #pause(): void { + if (this.#runningSinceMs === undefined) return; + this.#remainingMs = Math.max(0, this.#remainingMs - (Date.now() - this.#runningSinceMs)); + this.#runningSinceMs = undefined; + clearTimeout(this.#timer); + this.#timer = undefined; + } +} + export class RequestAdmissionController { readonly #abortController: AbortController = new AbortController(); readonly #abortFromClient: () => void; readonly #admission: IDaemonRequestAdmissionOptions | undefined; readonly #client: IRequestAdmissionClient; - readonly #deadlineMs: number | undefined; + #deadlineMs: number | undefined; readonly #writer: QueuePositionWriter | undefined; public constructor(options: IRequestAdmissionControllerOptions) { @@ -124,7 +190,7 @@ export class RequestAdmissionController { } } - /** Waits for workspace admission, bounded by the request's absolute admission deadline. */ + /** Waits for workspace admission within the request's remaining admission budget. */ public async acquireAsync( scheduler: RequestScheduler, exclusivityClass: RequestExclusivityClass @@ -161,17 +227,83 @@ export class RequestAdmissionController { ); } + /** + * Waits for shared-build workspace admission while another request loads or reloads the workspace graph. + * + * @remarks + * While `transition` reports progress, the other request holds the exclusive gate and is loading the graph that + * this request needs, so a client-default timeout is not spent: at a cold start every concurrent build waits for + * the first build's graph load. The default budget is still spent while the transition itself waits for another + * request, so a transition that cannot start does not hold its followers indefinitely. Unspent budget carries over + * to later waits of this request. An explicit `noWait` or `waitTimeoutMs` applies unchanged. + */ + public async acquireBehindTransitionAsync( + scheduler: RequestScheduler, + transition: AdmissionProgress + ): Promise { + const waitingFor: string = "another request's load or reload of the workspace graph"; + const remainingMs: number | undefined = this.#getRemainingWaitTimeoutMs(); + if (!this.#admission?.waitTimeoutIsDefault || remainingMs === undefined) { + return await this.#acquireAsync(scheduler, RequestExclusivityClass.SharedBuild, remainingMs, waitingFor); + } + const exhausted: AbortController = new AbortController(); + const budget: ProgressPausedBudget = new ProgressPausedBudget(remainingMs, transition, () => + exhausted.abort() + ); + try { + return await this.#acquireAsync( + scheduler, + RequestExclusivityClass.SharedBuild, + undefined, + waitingFor, + AbortSignal.any([this.#abortController.signal, exhausted.signal]) + ); + } catch (error) { + if (!exhausted.signal.aborted || this.#abortController.signal.aborted) throw error; + throw this.#getReportedError( + new RequestSchedulerError( + RequestSchedulerErrorCode.WaitTimeout, + `The request was not admitted within ${this.#admission.waitTimeoutMs}ms.` + ), + waitingFor + ); + } finally { + this.#deadlineMs = Date.now() + budget.stop(); + } + } + + /** + * Runs `action`, such as routing and executing an admitted request, without spending a client-default budget. + * + * @remarks + * Work after admission either runs or waits behind progress, such as the exempt graph-execution gate. A request that + * re-enters workspace admission afterwards, for example to reload the graph after its inputs changed, therefore + * keeps the budget it had before `action`. An explicit `noWait` or `waitTimeoutMs` keeps its absolute deadline. + */ + public async runOutsideDefaultBudgetAsync(action: () => Promise): Promise { + const remainingMs: number | undefined = this.#getRemainingWaitTimeoutMs(); + if (!this.#admission?.waitTimeoutIsDefault || remainingMs === undefined) { + return await action(); + } + try { + return await action(); + } finally { + this.#deadlineMs = Date.now() + remainingMs; + } + } + async #acquireAsync( scheduler: RequestScheduler, exclusivityClass: RequestExclusivityClass, waitTimeoutMs: number | undefined, - waitingFor: string + waitingFor: string, + abortSignal: AbortSignal = this.#abortController.signal ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; let lease: IRequestLease | undefined; try { lease = await scheduler.acquireAsync({ - abortSignal: this.#abortController.signal, + abortSignal, exclusivityClass, noWait: this.#admission?.noWait, onQueuePositionChanged: writer ? (position: number) => writer.enqueue(position) : undefined, diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 2ee815aad2..699c1eec21 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -35,6 +35,7 @@ import { type IRequestLease } from './RequestScheduler'; import { + AdmissionProgress, RequestAdmissionController, getRequestAdmissionErrorCode, getWorkspaceRequestScheduler @@ -126,6 +127,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { #restartPending: boolean = false; #lastReloadTier: WorkspaceInputChangeTier = WorkspaceInputChangeTier.Reuse; #transitioning: boolean = false; + /** Active while the transition owner holds the exclusive gate and loads or reloads the workspace graph. */ + readonly #transitionProgress: AdmissionProgress = new AdmissionProgress(); #cleanupFailure: unknown; #disposePromise: Promise | undefined; @@ -204,25 +207,28 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { try { for (let attempt: number = 0; ; attempt++) { try { - generation = await this.#prepareAsync(envelope, client, admission, ticket); + const prepared: IPreparedGeneration = await this.#prepareAsync(envelope, client, admission, ticket); + generation = prepared; const requestEnvelope: IDaemonRequestEnvelope = { ...envelope, admission: admission.remainingAdmission }; - if (isMutation(envelope)) { - await this.#executeMutationAsync(generation, requestEnvelope, client, state, dispatchAsync); - } else { - await dispatchAsync({ - envelope: requestEnvelope, - client, - workspaceSession: generation.session, - resolver: generation.resolver, - onExecutionStarting: () => { - this.#assertGeneration(generation!); - state.began = true; - } - }); - } + await admission.runOutsideDefaultBudgetAsync(async () => { + if (isMutation(envelope)) { + await this.#executeMutationAsync(prepared, requestEnvelope, client, state, dispatchAsync); + } else { + await dispatchAsync({ + envelope: requestEnvelope, + client, + workspaceSession: prepared.session, + resolver: prepared.resolver, + onExecutionStarting: () => { + this.#assertGeneration(prepared); + state.began = true; + } + }); + } + }); return; } catch (error) { if ( @@ -294,7 +300,10 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { admittedLease?: IRequestLease ): Promise { let lease: IRequestLease = - admittedLease ?? (await admission.acquireAsync(this.#gate, RequestExclusivityClass.SharedBuild)); + admittedLease ?? + (this.#transitioning + ? await admission.acquireBehindTransitionAsync(this.#gate, this.#transitionProgress) + : await admission.acquireAsync(this.#gate, RequestExclusivityClass.SharedBuild)); let ownsTransition: boolean = false; try { if (this.#restartPending) throw new RestartPendingBeforeExecution(); @@ -429,15 +438,16 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { if (this.#restartPending) throw new RestartPendingBeforeExecution(); } if (this.#transitioning) { - const shared: IRequestLease = await admission.acquireAsync( + const shared: IRequestLease = await admission.acquireBehindTransitionAsync( this.#gate, - RequestExclusivityClass.SharedBuild + this.#transitionProgress ); return await this.#prepareAsync(envelope, client, admission, ticket, shared); } this.#transitioning = ownsTransition = true; this.#cancelObservers(); lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); + this.#transitionProgress.setActive(true); if (this.#restartPending) throw new RestartPendingBeforeExecution(); if (this.#closing) throw new Error('The workspace is restarting. No operation was scheduled or executed.'); @@ -609,7 +619,10 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { if (!(error instanceof RestartBeforeExecution)) lease.release(); throw error; } finally { - if (ownsTransition) this.#transitioning = false; + if (ownsTransition) { + this.#transitioning = false; + this.#transitionProgress.setActive(false); + } } } diff --git a/libraries/rush-daemon/src/WorkspaceWarmSet.ts b/libraries/rush-daemon/src/WorkspaceWarmSet.ts index 3b7e5aa5f5..bf037cabc0 100644 --- a/libraries/rush-daemon/src/WorkspaceWarmSet.ts +++ b/libraries/rush-daemon/src/WorkspaceWarmSet.ts @@ -233,11 +233,13 @@ export class WorkspaceWarmSet implements AsyncDisposable { const now: number = performance.now(); const delay: number = Math.min( MAX_POLL_DELAY_MS, - ...this.#rankProjects().map((project) => { - const remaining: number = - project.lastUsed + this.#configuration.warmIdleTimeoutSeconds * 1000 - now; - return remaining > 0 ? Math.max(1, remaining) : RETRY_DELAY_MS; - }) + ...this.#rankProjects() + .filter((project) => this.#mayRelease(project)) + .map((project) => { + const remaining: number = + project.lastUsed + this.#configuration.warmIdleTimeoutSeconds * 1000 - now; + return remaining > 0 ? Math.max(1, remaining) : RETRY_DELAY_MS; + }) ); this.#schedule( this.#deferredReason || this.#watcherPolicyFailure ? Math.min(delay, RETRY_DELAY_MS) : delay @@ -365,8 +367,13 @@ export class WorkspaceWarmSet implements AsyncDisposable { project.operations.every( (operation) => !graph.resultByOperation.has(operation) && !operation.runner?.isActive ); - const overProjectLimit: boolean = status.overProjectLimit && project.holdsResources; - if (!unrequested && !expired && !status.overMemoryBudget && !overProjectLimit) continue; + // Idle expiry and every limit release runners and watchers, and finish an eviction that failed earlier. + // Otherwise the retained results of a resource-free project stay until the generation ends: they are + // revalidated on every request, they are what makes a warm no-op skip possible, and dropping them cannot + // bring daemon RSS below the budget. + const release: boolean = + this.#mayRelease(project) && (expired || status.overMemoryBudget || status.overProjectLimit); + if (!unrequested && !release) continue; try { await graph.closeRunnersAsync(project.operations); if (project.operations.some((operation) => operation.runner?.isActive)) { @@ -384,6 +391,10 @@ export class WorkspaceWarmSet implements AsyncDisposable { } } + #mayRelease(project: IWarmProject): boolean { + return project.holdsResources || this.#cleanupFailures.has(project.key); + } + #rankProjects(): IWarmProject[] { const { operationGraph: graph, watcher, getProtectedOperations } = this.#options; const protectedOperations: ReadonlySet | undefined = getProtectedOperations?.(); @@ -463,14 +474,14 @@ export class WorkspaceWarmSet implements AsyncDisposable { } #reportPressure(status: IWorkspaceWarmSetStatus): void { - // A queued project-cap cleanup is normal during a request; status still exposes the deferral. - if (status.deferredReason && !status.overMemoryBudget) return; + // Cleanup queued behind a request is normal and status still exposes the deferral. Only a completed pass + // shows what remains, and the key ignores the deferral so alternating busy/idle passes do not repeat it. + if (status.deferredReason) return; const key: string | undefined = status.overMemoryBudget || status.overProjectLimit ? JSON.stringify([ status.overMemoryBudget, status.overProjectLimit, - status.deferredReason, status.protectedProjectNames, status.cleanupFailures, status.unmeasuredRunnerCount @@ -483,7 +494,7 @@ export class WorkspaceWarmSet implements AsyncDisposable { `limit ${this.#configuration.warmSetMaxProjects} projects): daemon RSS ${status.daemonResidentMemoryBytes} bytes, ` + `measured child RSS ${status.measuredRunnerMemoryBytes} bytes, ${status.unmeasuredRunnerCount} unmeasured runners, ` + `${status.retainedProjectNames.length} retained projects, ${status.protectedProjectNames.length} protected. ` + - `Deferred: ${status.deferredReason ?? 'no'}. Active/protected resources and remaining daemon memory cannot be forced below the budget.` + `Active or protected resources, resource-free retained results and remaining daemon memory are not released to meet the budget.` ) ); } diff --git a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts index 21b01d472c..844b581ed1 100644 --- a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts +++ b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts @@ -34,6 +34,8 @@ export class DaemonGraphTestFixture implements AsyncDisposable { public session!: WorkspaceSession; public host!: RushDaemonHost; public getSuccessorLaunchAsync: GetWorkspaceSuccessorLaunchAsync | undefined; + /** Awaited before each workspace session is created, including a request's graph load or reload. */ + public beforeCreateSessionAsync: (() => Promise) | undefined; public readonly folder: string = fs.realpathSync.native( fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-graph-')) ); @@ -154,6 +156,7 @@ export class DaemonGraphTestFixture implements AsyncDisposable { } }, createWorkspaceSessionAsync: async (options) => { + await this.beforeCreateSessionAsync?.(); this.session = await WorkspaceSession.createAsync(options); return this.session; } diff --git a/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts new file mode 100644 index 0000000000..232687ffb5 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts @@ -0,0 +1,159 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; + +import { + type IRequestLease, + RequestExclusivityClass, + RequestScheduler, + RequestSchedulerErrorCode +} from '../RequestScheduler'; +import { AdmissionProgress, RequestAdmissionController } from '../WorkspaceRequestAdmission'; + +const DEFAULT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 100, waitTimeoutIsDefault: true }; +const EXPLICIT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 100 }; + +interface IAcquisition { + settled: boolean; + lease?: IRequestLease; + error?: unknown; +} + +function track(promise: Promise): IAcquisition { + const acquisition: IAcquisition = { settled: false }; + promise.then( + (lease: IRequestLease) => { + acquisition.settled = true; + acquisition.lease = lease; + }, + (error: unknown) => { + acquisition.settled = true; + acquisition.error = error; + } + ); + return acquisition; +} + +function createController( + admission: IDaemonRequestAdmissionOptions, + abortSignal: AbortSignal = new AbortController().signal +): RequestAdmissionController { + return new RequestAdmissionController({ admission, client: { abortSignal }, requestId: 'request' }); +} + +describe(RequestAdmissionController.name, () => { + let scheduler: RequestScheduler; + let owner: IRequestLease; + let transition: AdmissionProgress; + + beforeEach(async () => { + jest.useFakeTimers(); + scheduler = new RequestScheduler(); + owner = await scheduler.acquireAsync({ exclusivityClass: RequestExclusivityClass.Exclusive }); + transition = new AdmissionProgress(); + }); + + afterEach(() => { + owner.release(); + jest.useRealTimers(); + }); + + it('does not spend a default budget while the transition it waits behind makes progress', async () => { + const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); + transition.setActive(true); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(10_000); + expect(waiting.settled).toBe(false); + + scheduler.downgradeExclusiveLease(owner, RequestExclusivityClass.SharedBuild); + transition.setActive(false); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.settled).toBe(true); + expect(waiting.error).toBeUndefined(); + expect(waiting.lease?.exclusivityClass).toBe(RequestExclusivityClass.SharedBuild); + waiting.lease?.release(); + controller.dispose(); + }); + + it('spends a default budget while the transition itself waits, and carries the rest across progress', async () => { + const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(60); + transition.setActive(true); + await jest.advanceTimersByTimeAsync(10_000); + transition.setActive(false); + await jest.advanceTimersByTimeAsync(39); + expect(waiting.settled).toBe(false); + + await jest.advanceTimersByTimeAsync(1); + expect(waiting.error).toMatchObject({ + code: RequestSchedulerErrorCode.WaitTimeout, + message: expect.stringContaining( + "not admitted within 100ms while waiting for another request's load or reload of the workspace graph" + ) + }); + expect(scheduler.queuedRequestCount).toBe(0); + controller.dispose(); + }); + + it('keeps an explicit deadline behind a transition that makes progress', async () => { + const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); + transition.setActive(true); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(99); + expect(waiting.settled).toBe(false); + + await jest.advanceTimersByTimeAsync(1); + expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.WaitTimeout }); + controller.dispose(); + }); + + it('reports cancellation behind a transition as an abort', async () => { + const client: AbortController = new AbortController(); + const controller: RequestAdmissionController = createController(DEFAULT_BUDGET, client.signal); + transition.setActive(true); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + client.abort(); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.Aborted }); + expect(scheduler.queuedRequestCount).toBe(0); + controller.dispose(); + }); + + it('keeps an unspent default budget across admitted work for a later admission wait', async () => { + const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); + await jest.advanceTimersByTimeAsync(40); + const work: Promise = controller.runOutsideDefaultBudgetAsync( + () => new Promise((resolve) => setTimeout(resolve, 10_000)) + ); + await jest.advanceTimersByTimeAsync(10_000); + await work; + expect(controller.remainingAdmission).toEqual({ ...DEFAULT_BUDGET, waitTimeoutMs: 60 }); + + const waiting: IAcquisition = track(controller.acquireAsync(scheduler, RequestExclusivityClass.SharedBuild)); + await jest.advanceTimersByTimeAsync(59); + expect(waiting.settled).toBe(false); + await jest.advanceTimersByTimeAsync(1); + expect(waiting.error).toMatchObject({ + code: RequestSchedulerErrorCode.WaitTimeout, + message: expect.stringContaining('not admitted within 100ms while waiting for workspace admission') + }); + controller.dispose(); + }); + + it('keeps an explicit deadline running across admitted work', async () => { + const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); + const work: Promise = controller.runOutsideDefaultBudgetAsync( + () => new Promise((resolve) => setTimeout(resolve, 10_000)) + ); + await jest.advanceTimersByTimeAsync(10_000); + await work; + expect(controller.remainingAdmission).toEqual({ waitTimeoutMs: 0 }); + + const waiting: IAcquisition = track(controller.acquireAsync(scheduler, RequestExclusivityClass.SharedBuild)); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.WaitTimeout }); + controller.dispose(); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts new file mode 100644 index 0000000000..1337cb515f --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts @@ -0,0 +1,111 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; + +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { createDeferred, type IDeferred, type ITerminalExchange } from './DaemonRequestWireTestUtilities'; + +jest.setTimeout(60_000); + +const BUILD_A: string[] = ['build', '--to', 'a', '--parallelism', '3']; +const BUILD_B: string[] = ['build', '--to', 'b', '--parallelism', '3']; +const DEFAULT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 300, waitTimeoutIsDefault: true }; + +function delayAsync(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +async function waitForAsync(predicate: () => boolean, description: string): Promise { + const deadline: number = Date.now() + 30_000; + while (!predicate()) { + if (Date.now() > deadline) throw new Error(`Timed out waiting for ${description}.`); + await delayAsync(20); + } +} + +function expectSuccess(exchange: ITerminalExchange): void { + expect(exchange.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); +} + +function expectWaitTimeout(exchange: ITerminalExchange): void { + expect(exchange.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, admissionErrorCode: 'wait-timeout' } + }); +} + +describe('workspace admission behind a graph transition', () => { + it('admits builds that arrive while another build loads the graph without spending their default budget', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); + try { + const loadStarted: IDeferred = createDeferred(); + const releaseLoad: IDeferred = createDeferred(); + fixture.beforeCreateSessionAsync = async () => { + loadStarted.resolve(); + await releaseLoad.promise; + }; + const first: Promise = fixture.runAsync(BUILD_B); + await loadStarted.promise; + + const withDefaultBudget: Promise = fixture.runAsync(BUILD_B, { + admission: DEFAULT_BUDGET + }); + const withExplicitBudget: ITerminalExchange = await fixture.runAsync(BUILD_B, { + admission: { waitTimeoutMs: 300 } + }); + expectWaitTimeout(withExplicitBudget); + await delayAsync(600); + + fixture.beforeCreateSessionAsync = undefined; + releaseLoad.resolve(); + expectSuccess(await first); + expectSuccess(await withDefaultBudget); + expect(fixture.runs()).toEqual(['a', 'b']); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + + it('keeps a graph-gate waiter default budget for its reload after an input change', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => { + created.write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\nruns.txt\nrelease-a\n'); + created.write( + 'a/build.cjs', + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','a\\n');" + + "const wait=()=>fs.existsSync('../release-a')?console.log('finished-a'):setTimeout(wait,20);wait();" + ); + }); + const releaseFile: string = path.join(fixture.folder, 'release-a'); + try { + const long: Promise = fixture.runAsync(BUILD_A); + await waitForAsync(() => fixture.runs().includes('a'), 'the long build to start'); + + // Reuses the generation, then waits at the graph-execution gate, which a default budget does not limit. + const graphGateWaiter: Promise = fixture.runAsync(BUILD_A, { + admission: DEFAULT_BUDGET + }); + await delayAsync(500); + const packageJsonPath: string = path.join(fixture.folder, 'c/package.json'); + const packageJson: Record = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8')); + fs.writeFileSync(packageJsonPath, JSON.stringify({ ...packageJson, description: 'changed' })); + + // Needs a reload, so it waits for exclusive admission until the long build ends. + const reload: Promise = fixture.runAsync(BUILD_A); + await delayAsync(500); + // A reload that cannot start is not progress, so a default budget behind it is still spent. + expectWaitTimeout(await fixture.runAsync(BUILD_A, { admission: DEFAULT_BUDGET })); + + fs.writeFileSync(releaseFile, ''); + expectSuccess(await long); + expectSuccess(await reload); + // The waiter's batch finds the changed input and re-enters admission behind the reload. + expectSuccess(await graphGateWaiter); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts index f4193dbd27..8f3d81c702 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts @@ -260,6 +260,37 @@ describe('warm policies attached to native graphs and real filesystem watchers', expect(fixture.runs()).toEqual(['a', 'b']); }); + it('keeps resource-free retained results under memory pressure and idle expiry, and warns once', async () => { + test = await WarmSetTestFixture.createAsync(); + test.configuration = { ...GENEROUS_WARM_CONFIGURATION, watch: false }; + const { fixture } = test; + await fixture.buildSuccessfullyAsync(); + const { warm, graph, watcher } = test; + const records: Map = new Map( + ['a', 'b'].map((name) => [name, graph.resultByOperation.get(test!.operation(name))]) + ); + expect([...records.values()].every((record) => record !== undefined)).toBe(true); + const runs: string[] = fixture.runs(); + test.update({ watch: false, warmMemoryBudgetMB: 0.01, warmIdleTimeoutSeconds: 0.01 }); + await delayAsync(50); + for (let pass: number = 0; pass < 2; pass++) { + const status = await warm.maintainAsync(); + expect(status).toMatchObject({ overMemoryBudget: true, overProjectLimit: false }); + expect(status.deferredReason).toBeUndefined(); + expect([...status.retainedProjectNames].sort()).toEqual(['a', 'b']); + } + for (const [name, record] of records) { + expect(graph.resultByOperation.get(test.operation(name))).toBe(record); + } + expect(watcher.watchedProjectNames.size).toBe(0); + + // The retained results still make an unchanged build a warm no-op skip. + await fixture.buildSuccessfullyAsync(); + await warm.maintainAsync(); + expect(fixture.runs()).toEqual(runs); + expect(test.diagnostics.filter((error) => error.message.includes('Warm-set pressure'))).toHaveLength(1); + }); + it('reports missing child measurements and uses conservative LRU instead of inventing memory scores', async () => { const { fixture, warm } = await startAsync({ ipc: true }); const missing = ['a', 'b'].map((name) => test!.operation(name).runner!); diff --git a/libraries/rush-lib/src/api/DaemonConfiguration.ts b/libraries/rush-lib/src/api/DaemonConfiguration.ts index 695dcaa548..0869ebacb3 100644 --- a/libraries/rush-lib/src/api/DaemonConfiguration.ts +++ b/libraries/rush-lib/src/api/DaemonConfiguration.ts @@ -15,9 +15,15 @@ export interface IDaemonConfigurationJson { readonly usePersistentIpcRunners?: boolean; /** Maximum admission queue wait in seconds. Defaults to 30. */ readonly queueTimeoutSeconds?: number; - /** Idle resource expiration in an attached daemon warm set. Defaults to 300 seconds. */ + /** + * Idle resource expiration in an attached daemon warm set; retained results of resource-free projects do not + * expire. Defaults to 300 seconds. + */ readonly warmIdleTimeoutSeconds?: number; - /** Best-effort sampled RSS budget for an attached warm set, not a hard ceiling. Defaults to 512 MiB. */ + /** + * Best-effort sampled RSS budget for an attached warm set, not a hard ceiling; active/protected work and retained + * results of resource-free projects are exempt. Defaults to 512 MiB. + */ readonly warmMemoryBudgetMB?: number; /** * Best-effort limit on projects holding warm resources (active runners or file watchers); active/protected work diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index 9bf466e493..b4262da1e2 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -236,7 +236,20 @@ export async function captureWorkspaceInputFingerprintAsync( installation.add(path.join(subspace.getSubspaceTempFolderPath(), 'last-install.flag')); } installation.add(path.join(rushConfiguration.commonTempFolder, 'current-variants.json')); - const configurationFiles: string[] = await listFilesAsync(path.join(root, 'common', 'config'), false); + const projectFolders: string[] = []; + for (const project of rushJson.projects) { + const projectFolder: string = path.resolve(root, project.projectFolder); + if (!Path.isUnderOrEqual(projectFolder, root)) { + throw new Error('A fingerprint project folder must be inside the workspace.'); + } + projectFolders.push(projectFolder); + } + const commonConfigFolder: string = path.join(root, 'common', 'config'); + const configurationFiles: string[] = await listFilesAsync( + commonConfigFolder, + false, + getNestedProjectFolders(commonConfigFolder, projectFolders, rushConfiguration) + ); for (const filename of configurationFiles) { (isProcessBoundConfiguration(filename) ? installation : definitions).add(filename); } @@ -249,11 +262,7 @@ export async function captureWorkspaceInputFingerprintAsync( definitions.add(filename); } } - for (const project of rushJson.projects) { - const projectFolder: string = path.resolve(root, project.projectFolder); - if (!Path.isUnderOrEqual(projectFolder, root)) { - throw new Error('A fingerprint project folder must be inside the workspace.'); - } + for (const projectFolder of projectFolders) { for (const relativePath of [ 'package.json', '.gitignore', @@ -333,7 +342,38 @@ async function hashFilesAsync(filenames: Iterable): Promise { return hashText(JSON.stringify(entries)); } -async function listFilesAsync(folderOrFile: string, runtime: boolean): Promise { +/** + * Returns the Rush project folders nested inside `common/config`. Rush reads such a project only through the + * project definition files fingerprinted for every project, and running its operations rewrites logs, build + * outputs and `.rush/temp` state inside it, which must not look like a configuration change. A project folder + * that is inside or contains a Rush configuration folder is never excluded. + */ +function getNestedProjectFolders( + commonConfigFolder: string, + projectFolders: ReadonlyArray, + rushConfiguration: RushConfiguration +): ReadonlySet { + const rushConfigurationFolders: string[] = [ + rushConfiguration.commonRushConfigFolder, + path.join(commonConfigFolder, 'subspaces'), + ...rushConfiguration.subspaces.map((subspace) => subspace.getSubspaceConfigFolderPath()) + ]; + return new Set( + projectFolders.filter( + (projectFolder) => + Path.isUnder(projectFolder, commonConfigFolder) && + !rushConfigurationFolders.some( + (folder) => Path.isUnderOrEqual(folder, projectFolder) || Path.isUnderOrEqual(projectFolder, folder) + ) + ) + ); +} + +async function listFilesAsync( + folderOrFile: string, + runtime: boolean, + excludedFolders: ReadonlySet = new Set() +): Promise { try { const stat: Awaited> = await fs.stat(folderOrFile); if (!stat.isDirectory()) return [folderOrFile]; @@ -341,8 +381,14 @@ async function listFilesAsync(folderOrFile: string, runtime: boolean): Promise { fs.rmSync(folder, { recursive: true, force: true }); } }); + + it('ignores operation state inside a project nested in common/config but not its definitions', async () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-fingerprint-')); + try { + const write = (relativePath: string, content: string): void => { + const filename: string = path.join(folder, relativePath); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, content); + }; + write( + 'rush.json', + JSON.stringify({ + rushVersion: '5.179.0', + pnpmVersion: '10.27.0', + projectFolderMaxDepth: 4, + projects: [ + { packageName: 'pipelines', projectFolder: 'common/config/pipelines' }, + { packageName: 'inside-rush', projectFolder: 'common/config/rush/inside-rush' } + ] + }) + ); + write('common/config/pipelines/package.json', '{"name":"pipelines","version":"1.0.0"}'); + write('common/config/rush/inside-rush/package.json', '{"name":"inside-rush","version":"1.0.0"}'); + write('common/config/rush/command-line.json', '{"commands":[]}'); + write('common/config/other/settings.json', '{}'); + const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile( + path.join(folder, 'rush.json') + ); + const runtimeCache: WorkspaceRuntimeFingerprintCache = new WorkspaceRuntimeFingerprintCache(); + const captureAsync = (): Promise => + captureWorkspaceInputFingerprintAsync({ rushConfiguration, runtimeCache, environment: {} }); + + let previous: IWorkspaceInputFingerprint = await captureAsync(); + for (const relativePath of [ + 'common/config/pipelines/rush-logs/pipelines._phase_build.log', + 'common/config/pipelines/.rush/temp/operation/_phase_build/state.json', + 'common/config/pipelines/lib/index.js', + 'common/config/pipelines/config/heft.json' + ]) { + write(relativePath, String(Math.random())); + const next: IWorkspaceInputFingerprint = await captureAsync(); + expect(next.configurationHash).toBe(previous.configurationHash); + expect(classifyWorkspaceInputChange(previous, next)).toBe(WorkspaceInputChangeTier.Reuse); + } + for (const [relativePath, content] of [ + ['common/config/pipelines/package.json', '{"name":"pipelines","version":"1.0.1"}'], + ['common/config/pipelines/config/rush-project.json', '{"operationSettings":[]}'], + ['common/config/pipelines/config/rig.json', '{"rigPackageName":"rig"}'], + ['common/config/rush/command-line.json', '{"commands":[],"parameters":[]}'], + ['common/config/rush/inside-rush/rush-logs/inside-rush._phase_build.log', 'log'], + ['common/config/other/settings.json', '{"changed":true}'] + ]) { + write(relativePath, content); + const next: IWorkspaceInputFingerprint = await captureAsync(); + expect(classifyWorkspaceInputChange(previous, next)).toBe(WorkspaceInputChangeTier.Reload); + previous = next; + } + } finally { + fs.rmSync(folder, { recursive: true, force: true }); + } + }); }); diff --git a/libraries/rush-lib/src/schemas/rush.schema.json b/libraries/rush-lib/src/schemas/rush.schema.json index 801ccdafc3..980113494a 100644 --- a/libraries/rush-lib/src/schemas/rush.schema.json +++ b/libraries/rush-lib/src/schemas/rush.schema.json @@ -290,7 +290,7 @@ "exclusiveMinimum": true, "maximum": 2147483.647, "default": 300, - "description": "Idle resource expiration for an attached warm set in seconds, at most 2147483.647 (the signed 32-bit millisecond limit of setTimeout). RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS overrides." + "description": "Idle resource expiration for an attached warm set in seconds, at most 2147483.647 (the signed 32-bit millisecond limit of setTimeout). Retained results of resource-free projects do not expire. RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS overrides." }, "warmMemoryBudgetMB": { "type": "number", @@ -298,7 +298,7 @@ "exclusiveMinimum": true, "maximum": 9007199254740991, "default": 512, - "description": "Best-effort sampled RSS budget in MiB, not a hard process/tree memory ceiling. At most 9007199254740991 (Number.MAX_SAFE_INTEGER). Active/protected work is exempt. RUSH_DAEMON_WARM_MEMORY_BUDGET_MB overrides." + "description": "Best-effort sampled RSS budget in MiB, not a hard process/tree memory ceiling. At most 9007199254740991 (Number.MAX_SAFE_INTEGER). Active/protected work and retained results of resource-free projects are exempt. RUSH_DAEMON_WARM_MEMORY_BUDGET_MB overrides." }, "warmSetMaxProjects": { "type": "integer", From 593dbcfc723e9bb591549d896d3ae73d05beb876 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 13:31:54 +0000 Subject: [PATCH 017/265] [rush-daemon] Warm set: read status once per maintenance pass, not once per project (and 1 more) Swarm integration step 2; original commit 8553b2be55 (merge of swarm/r07 at d94054e04d). Commits folded into this step (2): - 57dd6a4b45 [rush-daemon] Warm set: read status once per maintenance pass, not once per project - d94054e04d [rush-lib] Share rig profile loads across projects in the uncached rush-project.json load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...rig-profile-realpath_2026-09-28-13-00.json | 10 + ...warm-set-status-once_2026-09-28-12-50.json | 10 + libraries/rush-daemon/src/WorkspaceWarmSet.ts | 6 +- .../WorkspaceWarmSetMaintenanceCost.test.ts | 176 ++++++++++++++++++ .../src/api/RushProjectConfiguration.ts | 60 +++++- .../api/test/RushProjectConfiguration.test.ts | 56 ++++++ 6 files changed, 313 insertions(+), 5 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r07-rig-profile-realpath_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r07-warm-set-status-once_2026-09-28-12-50.json create mode 100644 libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts diff --git a/common/changes/@microsoft/rush/swarm-r07-rig-profile-realpath_2026-09-28-13-00.json b/common/changes/@microsoft/rush/swarm-r07-rig-profile-realpath_2026-09-28-13-00.json new file mode 100644 index 0000000000..c0a5dec20c --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r07-rig-profile-realpath_2026-09-28-13-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "The daemon's uncached rush-project.json load resolves each rig profile folder to its real path, so projects that reach one rig through their own node_modules symlinks share one load of the rig's files instead of re-reading and re-resolving them per project.", + "type": "none" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r07-warm-set-status-once_2026-09-28-12-50.json b/common/changes/@rushstack/rush-daemon/swarm-r07-warm-set-status-once_2026-09-28-12-50.json new file mode 100644 index 0000000000..cf46ceb8d9 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r07-warm-set-status-once_2026-09-28-12-50.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Warm set: an idle maintenance pass reads its status once, and again only after an eviction attempt, instead of once per retained project, so a pass that evicts nothing no longer blocks the daemon for seconds in large repos.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/WorkspaceWarmSet.ts b/libraries/rush-daemon/src/WorkspaceWarmSet.ts index bf037cabc0..0be59f2249 100644 --- a/libraries/rush-daemon/src/WorkspaceWarmSet.ts +++ b/libraries/rush-daemon/src/WorkspaceWarmSet.ts @@ -355,11 +355,14 @@ export class WorkspaceWarmSet implements AsyncDisposable { async #evictIdleAsync(): Promise { const { operationGraph: graph, watcher } = this.#options; + // getStatus() re-ranks every project. Only an eviction attempt (which awaits) can change it during a pass, so + // it is reused until then; recomputing it per project made a pass without evictions quadratic. + let currentStatus: IWorkspaceWarmSetStatus | undefined; // Retention and eviction use exactly the same ordering, reversed only to release the lowest value first. for (const project of this.#rankProjects().reverse()) { if (this.#disposed) break; if (project.protected) continue; - const status: IWorkspaceWarmSetStatus = this.getStatus(); + const status: IWorkspaceWarmSetStatus = (currentStatus ??= this.getStatus()); const expired: boolean = performance.now() - project.lastUsed >= this.#configuration.warmIdleTimeoutSeconds * 1000; const unrequested: boolean = @@ -388,6 +391,7 @@ export class WorkspaceWarmSet implements AsyncDisposable { this.#cleanupFailures.set(project.key, message); this.#diagnose(new Error(message, { cause: error })); } + currentStatus = undefined; } } diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts new file mode 100644 index 0000000000..b9d42c040c --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts @@ -0,0 +1,176 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { + OperationStatus, + type IOperationGraph, + type IOperationRunner, + type Operation +} from '@microsoft/rush-lib'; + +import type { RequestScheduler } from '../RequestScheduler'; +import type { WorkspaceSessionFileWatcher } from '../WorkspaceSessionFileWatcher'; +import { + WorkspaceWarmSet, + type IWorkspaceWarmSetStatus, + type WorkspaceWarmSetConfiguration +} from '../WorkspaceWarmSet'; + +type Tap = (...args: unknown[]) => unknown; + +const CONFIGURATION: WorkspaceWarmSetConfiguration = { + watch: false, + warmIdleTimeoutSeconds: 3600, + warmMemoryBudgetMB: 1024 * 1024, + warmSetMaxProjects: 20, + autoWarmByTelemetry: false +}; + +interface ITestGraph { + readonly graph: IOperationGraph; + readonly operations: Operation[]; + readonly requestAll: () => void; +} + +function createHook(taps: Tap[]): { tap: (options: unknown, fn: Tap) => void } { + return { + tap: (options: unknown, fn: Tap) => { + taps.push(fn); + } + }; +} + +// A graph shaped like a warm odsp-web generation: every project has retained results and nothing is running. +function createGraph(projectCount: number): ITestGraph { + const operations: Operation[] = []; + for (let i: number = 0; i < projectCount; i++) { + operations.push({ + name: `p${i} (build)`, + associatedProject: { packageName: `p${i}` }, + enabled: true, + runner: undefined, + consumers: new Set() + } as unknown as Operation); + } + const configureIterationTaps: Tap[] = []; + const resultByOperation: Map = new Map( + operations.map((operation) => [operation, { status: OperationStatus.Success }]) + ); + const graph: IOperationGraph = { + operations: new Set(operations), + resultByOperation, + hooks: { + configureIteration: createHook(configureIterationTaps), + beforeExecuteOperationAsync: createHook([]), + afterExecuteIterationAsync: createHook([]), + onIdle: createHook([]) + }, + hasScheduledIteration: false, + status: OperationStatus.Ready, + abortController: new AbortController(), + deleteResults: (deleted: Iterable): void => { + for (const operation of deleted) resultByOperation.delete(operation); + }, + closeRunnersAsync: async (closed: Iterable): Promise => { + for (const operation of closed) await operation.runner?.closeAsync?.(); + } + } as unknown as IOperationGraph; + return { + graph, + operations, + requestAll: () => { + for (const tap of configureIterationTaps) tap(new Map(), new Map(), {}); + } + }; +} + +function attach( + graph: IOperationGraph, + configuration: Partial = {} +): WorkspaceWarmSet { + return WorkspaceWarmSet.attach({ + operationGraph: graph, + configuration: { ...CONFIGURATION, ...configuration }, + scheduler: { acquireAsync: async () => ({ release: () => undefined }) } as unknown as RequestScheduler, + acquireExecutionLeaseAsync: async () => ({ [Symbol.asyncDispose]: async () => undefined }), + watcher: { + watchedProjectNames: new Set(), + watchProjects: () => undefined, + unwatchProjectsAsync: async () => undefined + } as unknown as WorkspaceSessionFileWatcher + }); +} + +function createResidentRunner(): IOperationRunner { + let active: boolean = true; + return { + name: 'resident', + isNoOp: false, + cacheable: false, + reportTiming: false, + silent: false, + warningsAreAllowed: false, + get isActive(): boolean { + return active; + }, + getConfigHash: () => '', + executeAsync: async () => OperationStatus.Success, + closeAsync: async () => { + active = false; + } + }; +} + +async function countStatusReadsInPassAsync(warm: WorkspaceWarmSet): Promise { + // Let the pass scheduled by attachment and by the simulated request settle, so only the measured pass is counted. + await new Promise((resolve) => setTimeout(resolve, 10)); + await warm.maintainAsync(); + const getStatus: jest.SpyInstance = jest.spyOn(warm, 'getStatus'); + try { + await warm.maintainAsync(); + return getStatus.mock.calls.length; + } finally { + getStatus.mockRestore(); + } +} + +describe('warm-set maintenance cost', () => { + const disposables: WorkspaceWarmSet[] = []; + afterEach(async () => { + for (const warm of disposables.splice(0)) await warm[Symbol.asyncDispose](); + }); + + it('reads status a constant number of times in a pass that evicts nothing, regardless of project count', async () => { + const counts: number[] = []; + for (const projectCount of [10, 2000]) { + const { graph, requestAll } = createGraph(projectCount); + const warm: WorkspaceWarmSet = attach(graph); + disposables.push(warm); + requestAll(); + counts.push(await countStatusReadsInPassAsync(warm)); + expect(graph.resultByOperation.size).toBe(projectCount); + expect(warm.getStatus().retainedProjectNames).toHaveLength(projectCount); + } + expect(counts[1]).toBe(counts[0]); + expect(counts[0]).toBeLessThanOrEqual(2); + }); + + it('re-reads status after each eviction attempt, so it stops evicting once back under the project cap', async () => { + const { graph, operations, requestAll } = createGraph(40); + for (const operation of operations.slice(0, 5)) operation.runner = createResidentRunner(); + const warm: WorkspaceWarmSet = attach(graph, { warmSetMaxProjects: 3 }); + disposables.push(warm); + requestAll(); + await new Promise((resolve) => setTimeout(resolve, 10)); + const getStatus: jest.SpyInstance = jest.spyOn(warm, 'getStatus'); + const status: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + // Two evictions: one read before the first attempt, one after each attempt, one for the pass result. + expect(getStatus.mock.calls.length).toBeLessThanOrEqual(4); + getStatus.mockRestore(); + expect(status.overProjectLimit).toBe(false); + const evicted: Operation[] = operations.filter((operation) => !graph.resultByOperation.has(operation)); + expect(evicted).toHaveLength(2); + expect(evicted.every((operation) => operation.runner?.isActive === false)).toBe(true); + expect(graph.resultByOperation.size).toBe(38); + }); +}); diff --git a/libraries/rush-lib/src/api/RushProjectConfiguration.ts b/libraries/rush-lib/src/api/RushProjectConfiguration.ts index d3e356756b..f9c0668547 100644 --- a/libraries/rush-lib/src/api/RushProjectConfiguration.ts +++ b/libraries/rush-lib/src/api/RushProjectConfiguration.ts @@ -6,7 +6,12 @@ import * as path from 'node:path'; import { AlreadyReportedError, Async, FileSystem, JsonFile, Path } from '@rushstack/node-core-library'; import type { ITerminal } from '@rushstack/terminal'; import { ProjectConfigurationFile, InheritanceType } from '@rushstack/heft-config-file'; -import { RigConfig, type IRigConfigJson, type ILoadForProjectFolderOptions } from '@rushstack/rig-package'; +import { + RigConfig, + type IRigConfig, + type IRigConfigJson, + type ILoadForProjectFolderOptions +} from '@rushstack/rig-package'; import type { RushConfigurationProject } from './RushConfigurationProject'; import { RushConstants } from '../logic/RushConstants'; @@ -660,7 +665,7 @@ async function _tryLoadJsonForProjectAsync( loaders?.configurationFile ?? RUSH_PROJECT_CONFIGURATION_FILE; const oldConfigurationFile: ProjectConfigurationFile = loaders?.oldConfigurationFile ?? OLD_RUSH_PROJECT_CONFIGURATION_FILE; - const rigConfig: RigConfig | undefined = loaders + const rigConfig: IRigConfig | undefined = loaders ? await loadIsolatedRigConfigAsync(project.projectFolder) : await RigConfig.loadForProjectFolderAsync({ projectFolderPath: project.projectFolder }); @@ -699,7 +704,50 @@ async function _tryLoadJsonForProjectAsync( } } -async function loadIsolatedRigConfigAsync(projectFolder: string): Promise { +/** + * A found rig whose resolved profile folder is the real path of the rig package's profile folder. + */ +class RealProfileFolderRigConfig implements IRigConfig { + public readonly projectFolderOriginalPath: string; + public readonly projectFolderPath: string; + public readonly rigFound: boolean; + public readonly filePath: string; + public readonly rigPackageName: string; + public readonly rigProfile: string; + public readonly relativeProfileFolderPath: string; + readonly #rigConfig: IRigConfig; + readonly #profileFolder: string; + + public constructor(rigConfig: IRigConfig, profileFolder: string) { + this.projectFolderOriginalPath = rigConfig.projectFolderOriginalPath; + this.projectFolderPath = rigConfig.projectFolderPath; + this.rigFound = rigConfig.rigFound; + this.filePath = rigConfig.filePath; + this.rigPackageName = rigConfig.rigPackageName; + this.rigProfile = rigConfig.rigProfile; + this.relativeProfileFolderPath = rigConfig.relativeProfileFolderPath; + this.#rigConfig = rigConfig; + this.#profileFolder = profileFolder; + } + + public getResolvedProfileFolder(): string { + return this.#profileFolder; + } + + public async getResolvedProfileFolderAsync(): Promise { + return this.#profileFolder; + } + + public tryResolveConfigFilePath(configFileRelativePath: string): string | undefined { + return this.#rigConfig.tryResolveConfigFilePath(configFileRelativePath); + } + + public async tryResolveConfigFilePathAsync(configFileRelativePath: string): Promise { + return await this.#rigConfig.tryResolveConfigFilePathAsync(configFileRelativePath); + } +} + +async function loadIsolatedRigConfigAsync(projectFolder: string): Promise { let rigJson: IRigConfigJson; try { rigJson = await JsonFile.loadAsync(path.join(projectFolder, 'config', 'rig.json')); @@ -718,14 +766,18 @@ async function loadIsolatedRigConfigAsync(projectFolder: string): Promise { const configurations = await loadAsync(ownFile); expect(getOutputFolderNames(configurations.get(ownFile))).toEqual(['from-project']); }); + + it('loads a rig profile shared through per-project node_modules symlinks once, with native results', async () => { + write('store/example-rig/package.json', { name: 'example-rig', version: '1.0.0' }); + write('store/example-rig/profiles/default/config/rush-project.json', { + extends: '../../../shared/rush-project.json', + operationSettings: [{ operationName: '_phase:build', outputFolderNames: ['from-rig'] }] + }); + write('store/example-rig/shared/rush-project.json', { + operationSettings: [{ operationName: '_phase:test', outputFolderNames: ['from-shared'] }] + }); + const names: string[] = ['linked-1', 'linked-2', 'linked-3', 'linked-own']; + for (const name of names) { + write(`${name}/package.json`, { name, version: '1.0.0' }); + write(`${name}/config/rig.json`, { rigPackageName: 'example-rig' }); + fs.mkdirSync(path.join(folder, name, 'node_modules')); + fs.symlinkSync( + path.join(folder, 'store/example-rig'), + path.join(folder, name, 'node_modules/example-rig'), + 'junction' + ); + } + write('linked-own/config/rush-project.json', { + operationSettings: [{ operationName: '_phase:build', outputFolderNames: ['from-project'] }] + }); + const projects: RushConfigurationProject[] = names.map(project); + + const readFileAsync: jest.SpyInstance = jest.spyOn(FileSystem, 'readFileAsync'); + let configurations: ReadonlyMap; + let readPaths: string[]; + try { + configurations = await loadAsync(...projects); + readPaths = readFileAsync.mock.calls.map(([filePath]) => Path.convertToSlashes(filePath)); + } finally { + readFileAsync.mockRestore(); + } + expect(readPaths.filter((p) => p.endsWith('/profiles/default/config/rush-project.json'))).toHaveLength(1); + expect(readPaths.filter((p) => p.endsWith('/shared/rush-project.json'))).toHaveLength(1); + + for (const linked of projects.slice(0, 3)) { + expect(getOutputFolderNames(configurations.get(linked))).toEqual(['from-rig']); + expect([ + ...configurations.get(linked)!.operationSettingsByOperationName.get('_phase:test')!.outputFolderNames! + ]).toEqual(['from-shared']); + } + expect(getOutputFolderNames(configurations.get(projects[3]))).toEqual(['from-project']); + + const terminal: Terminal = new Terminal(new StringBufferTerminalProvider()); + for (const linked of projects) { + const native: RushProjectConfiguration | undefined = await RushProjectConfiguration.tryLoadForProjectAsync( + linked, + terminal + ); + expect(configurations.get(linked)!._getJsonForFingerprint()).toBe(native!._getJsonForFingerprint()); + } + }); }); describe('operationSettingsByOperationName', () => { From cd7a2a5b738c5efad7e06980b6bba8d3db5e8e6a Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 13:57:14 +0000 Subject: [PATCH 018/265] [rush-lib] Keep RUSH_DAEMON* variables out of lifecycle script environments (and 3 more) Swarm integration step 3; original commit 3b2d24277d (merge of swarm/r06 at 6c10e241a4). Commits folded into this step (4): - 803345396b [rush-lib] Keep RUSH_DAEMON* variables out of lifecycle script environments - b49d8b10cb [rush-daemon] Keep one daemon across per-session environment differences - 36892176cb [rush-daemon-transport] Reap detached operation groups of a daemon that died uncleanly - 6c10e241a4 [rush-daemon] Run each operation with its requester's per-session environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../src/test/RushXDaemonBoundaries.test.ts | 19 ++ .../rush/rushd-env-identity_2026-09-28.json | 11 ++ .../rush/rushd-operation-env_2026-09-28.json | 11 ++ ...ushd-request-operation-env_2026-09-28.json | 11 ++ ...ushd-request-operation-env_2026-09-28.json | 11 ++ .../rushd-orphan-groups_2026-09-28.json | 11 ++ .../rushd-env-identity_2026-09-28.json | 11 ++ ...ushd-request-operation-env_2026-09-28.json | 11 ++ common/reviews/api/rush-lib.api.md | 13 +- .../src/DaemonGroupTermination.ts | 52 ++++++ .../src/DaemonListener.ts | 7 +- .../src/DaemonListenerLifetime.ts | 14 +- .../src/DaemonOperationGroupReaper.ts | 75 ++++++++ .../src/DaemonOperationGroupRecorder.ts | 70 ++++++++ .../src/DaemonOperationGroups.ts | 65 +++++++ .../src/DaemonOrphanReaper.ts | 98 +++------- .../src/DaemonProcessGroup.ts | 10 +- .../src/DaemonProcessStat.ts | 72 ++++++++ .../src/DaemonReapOptions.ts | 63 +++++++ .../src/DaemonReclaim.ts | 11 +- .../test/DaemonOperationGroupReaper.test.ts | 77 ++++++++ .../test/DaemonOperationGroupRecorder.test.ts | 61 +++++++ .../src/test/DaemonOrphanReaper.test.ts | 2 +- .../src/test/DaemonProcessStat.test.ts | 50 ++++++ .../src/test/FakeOperationDaemonFixture.ts | 64 +++++++ .../src/test/OperationGroupFixture.ts | 49 +++++ .../test/OperationGroupReapOnReclaim.test.ts | 59 ++++++ .../test/OperationGroupTermination.test.ts | 83 +++++++++ .../src/test/OrphanReaperFixture.ts | 73 ++++++-- .../src/test/ProcessWaitFixture.ts | 53 ++++++ .../rush-daemon/src/PhasedRequestRouter.ts | 30 +++- .../src/ProductionDaemonRequestResolver.ts | 8 +- .../src/RushXDaemonRequestResolver.ts | 3 + .../src/VersionSelectedDaemonLauncher.ts | 11 +- .../src/test/PhasedRequestBatching.test.ts | 62 ++++++- .../ProductionDaemonRequestResolver.test.ts | 75 +++++++- .../VersionSelectedDaemonLauncher.test.ts | 20 +++ .../test/WorkspaceReloadTierStatus.test.ts | 21 ++- .../rush-lib/src/api/PhasedCommandEngine.ts | 10 +- .../src/api/WorkspaceInputFingerprint.ts | 160 ++++++++++++++--- ...asedCommandEngineParameterIdentity.test.ts | 39 ++++ .../test/WorkspaceInputFingerprint.test.ts | 97 ++++++++++ .../rush-lib/src/cli/RushCommandLineParser.ts | 11 ++ .../cli/scriptActions/PhasedScriptAction.ts | 32 +++- libraries/rush-lib/src/index.ts | 3 + .../src/logic/incremental/InputsSnapshot.ts | 93 ++++++---- .../incremental/test/InputsSnapshot.test.ts | 36 ++++ .../src/logic/operations/IOperationGraph.ts | 12 ++ .../operations/OperationExecutionRecord.ts | 11 +- .../src/logic/operations/OperationGraph.ts | 18 +- ...OperationGraphOperationEnvironment.test.ts | 170 ++++++++++++++++++ libraries/rush-lib/src/utilities/Utilities.ts | 7 + .../src/utilities/test/Utilities.test.ts | 32 ++++ 53 files changed, 1994 insertions(+), 184 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-env-identity_2026-09-28.json create mode 100644 common/changes/@microsoft/rush/rushd-operation-env_2026-09-28.json create mode 100644 common/changes/@microsoft/rush/rushd-request-operation-env_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-cli-client/rushd-request-operation-env_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon-transport/rushd-orphan-groups_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-env-identity_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-request-operation-env_2026-09-28.json create mode 100644 libraries/rush-daemon-transport/src/DaemonGroupTermination.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonOperationGroupRecorder.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonOperationGroups.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonProcessStat.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonReapOptions.ts create mode 100644 libraries/rush-daemon-transport/src/test/DaemonOperationGroupReaper.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/DaemonOperationGroupRecorder.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/DaemonProcessStat.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/FakeOperationDaemonFixture.ts create mode 100644 libraries/rush-daemon-transport/src/test/OperationGroupFixture.ts create mode 100644 libraries/rush-daemon-transport/src/test/OperationGroupReapOnReclaim.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/OperationGroupTermination.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/ProcessWaitFixture.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts diff --git a/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts b/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts index ecb58c4df2..37534a31ab 100644 --- a/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts +++ b/apps/rush-cli-client/src/test/RushXDaemonBoundaries.test.ts @@ -156,6 +156,25 @@ describe('native Rushx execution boundaries', () => { } ); + it("serves a request that sets a variable the daemon host starts without, with the request's value", async () => { + const parallelism: string | undefined = process.env.RUSH_PARALLELISM; + // A daemon host does not inherit the request-scoped variables of the client that started it. + delete process.env.RUSH_PARALLELISM; + let cwd: string; + try { + cwd = await startAsync(); + } finally { + if (parallelism !== undefined) process.env.RUSH_PARALLELISM = parallelism; + } + fixture.write('projects/a/script.cjs', 'console.log(process.env.RUSH_PARALLELISM);'); + const result: IRequestResult = await fixture.runAsync( + fixture.request(['-q', 'build'], cwd, fixture.environment({ RUSH_PARALLELISM: '2' })) + ); + expect(result.outcome.kind).toBe('result'); + expect(result.exitCode).toBe(0); + expect(result.stdout.toString()).toBe('2\n'); + }); + it('runs a script without queueing behind an exclusive workspace request', async () => { const cwd: string = await startAsync(); let release: () => void = () => {}; diff --git a/common/changes/@microsoft/rush/rushd-env-identity_2026-09-28.json b/common/changes/@microsoft/rush/rushd-env-identity_2026-09-28.json new file mode 100644 index 0000000000..dee50fac66 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-env-identity_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Stop per-session variables (`RUSHD_OUTPUT`, `RUSH_PARALLELISM`, `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, `RUSH_INVOKED_FOLDER`, systemd, editor, terminal and coding-agent session markers) and repeated `PATH` entries from changing a long-lived host's workspace identity, and read an engine request's `RUSH_PARALLELISM` from that request's environment.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@microsoft/rush/rushd-operation-env_2026-09-28.json b/common/changes/@microsoft/rush/rushd-operation-env_2026-09-28.json new file mode 100644 index 0000000000..b959f07eaa --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-operation-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Keep `RUSH_DAEMON*` variables out of lifecycle script environments, so that project tooling that starts an older Rush release, which rejects them as unknown `RUSH_*` variables, works when the command was routed through the Rush daemon.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@microsoft/rush/rushd-request-operation-env_2026-09-28.json b/common/changes/@microsoft/rush/rushd-request-operation-env_2026-09-28.json new file mode 100644 index 0000000000..1396212768 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-request-operation-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add `IOperationGraphIterationOptions.getOperationEnvironment` and `getWorkspaceRequestOperationEnvironment`, so that a long-lived host can start each operation, and hash its `dependsOnEnvVars`, from the environment of the request that selected it rather than from its own `process.env`.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/rushd-request-operation-env_2026-09-28.json b/common/changes/@rushstack/rush-cli-client/rushd-request-operation-env_2026-09-28.json new file mode 100644 index 0000000000..67822bf202 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/rushd-request-operation-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Cover a `rushx` request that sets `RUSH_PARALLELISM`, which the daemon now serves.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-transport/rushd-orphan-groups_2026-09-28.json b/common/changes/@rushstack/rush-daemon-transport/rushd-orphan-groups_2026-09-28.json new file mode 100644 index 0000000000..96e2b98b92 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-transport/rushd-orphan-groups_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-transport", + "comment": "On Linux, record the detached process groups that a daemon's operations lead, and when reclaiming the socket of a daemon that died uncleanly (SIGKILL/OOM), terminate the recorded groups that are proven to still belong to it (SIGTERM, then SIGKILL after a grace period), so that a successor daemon does not re-run those operations while the orphans are still writing.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-transport", + "email": "TheLarkInn@users.noreply.github.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-env-identity_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-env-identity_2026-09-28.json new file mode 100644 index 0000000000..5f0bfa1a78 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-env-identity_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Keep one daemon across clients whose environments differ only in per-session variables, apply each request's own `RUSH_PARALLELISM`, and start the daemon without the starting client's `RUSH_PARALLELISM` and `COPILOT_AGENT_SESSION_ID`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-request-operation-env_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-request-operation-env_2026-09-28.json new file mode 100644 index 0000000000..ca002cab9e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-request-operation-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Run each phased operation with the per-session variables (such as `COPILOT_AGENT_SESSION_ID`, `GIT_ASKPASS`, terminal markers and `RUSH_INVOKED_FOLDER`) of the request that selected it, and hash its `dependsOnEnvVars` from those same values, rather than from the environment of the client that started the daemon. Serve `rushx` requests that set a per-request variable such as `RUSH_PARALLELISM` instead of running them in-process.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 7b82e2af5d..0215d58fda 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -384,6 +384,12 @@ export type GetInputsSnapshotAsyncFn = () => Promise>): [string, string][]; +// @alpha +export function getWorkspaceHostEnvironment(environment: Readonly>): Record; + +// @alpha +export function getWorkspaceRequestOperationEnvironment(hostEnvironment: Readonly>, requestEnvironment: Readonly>): Record; + // @alpha (undocumented) export interface IBaseOperationExecutionResult { getStateHash(): string; @@ -616,7 +622,7 @@ export interface IIndividualVersionJson extends IVersionPolicyJson { // @beta export interface IInputsSnapshot { - getOperationOwnStateHash(project: IRushConfigurationProjectForSnapshot, operationName?: string): string; + getOperationOwnStateHash(project: IRushConfigurationProjectForSnapshot, operationName?: string, environment?: Readonly>): string; getTrackedFileHashesForOperation(project: IRushConfigurationProjectForSnapshot, operationName?: string): ReadonlyMap; readonly hashes: ReadonlyMap; readonly hasUncommittedChanges: boolean; @@ -766,6 +772,7 @@ export interface _IOperationGraphEventSink { // @alpha export interface IOperationGraphIterationOptions { + getOperationEnvironment?: (operation: Operation) => Readonly>; // (undocumented) inputsSnapshot?: IInputsSnapshot; startTime?: number; @@ -903,6 +910,7 @@ export interface IParsePhasedCommandOptions { readonly argv: ReadonlyArray; // (undocumented) readonly cwd: string; + readonly environment?: Readonly>; // (undocumented) readonly rushConfiguration: RushConfiguration; // (undocumented) @@ -2125,6 +2133,9 @@ export enum WorkspaceInputChangeTier { Reuse = 0 } +// @alpha +export const workspaceRequestScopedEnvironmentVariables: ReadonlySet; + // @alpha export class WorkspaceRuntimeFingerprintCache { get changedPaths(): ReadonlyArray; diff --git a/libraries/rush-daemon-transport/src/DaemonGroupTermination.ts b/libraries/rush-daemon-transport/src/DaemonGroupTermination.ts new file mode 100644 index 0000000000..df74a2495e --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonGroupTermination.ts @@ -0,0 +1,52 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IReapContext } from './DaemonReapOptions'; + +const POLL_INTERVAL_MS: number = 20; + +/** Outcome of reaping a dead daemon's orphaned process groups. */ +export type DaemonOrphanReapOutcome = 'none' | 'terminated' | 'killed'; + +function listExistingGroups(context: IReapContext, groupIds: readonly number[]): number[] { + return groupIds.filter((groupId: number) => context.ops.groupExists(groupId)); +} + +function haveAllExited(context: IReapContext, groupIds: readonly number[]): boolean { + return !groupIds.some((groupId: number) => context.ops.groupExists(groupId)); +} + +async function waitForGroupsExitAsync(context: IReapContext, groupIds: readonly number[]): Promise { + const deadline: number = context.ops.now() + context.graceMs; + while (context.ops.now() < deadline) { + if (haveAllExited(context, groupIds)) return true; + await context.ops.delayAsync(POLL_INTERVAL_MS); + } + return haveAllExited(context, groupIds); +} + +function signalExistingGroups( + context: IReapContext, + groupIds: readonly number[], + signal: NodeJS.Signals +): void { + for (const groupId of listExistingGroups(context, groupIds)) { + context.ops.signalGroup(groupId, signal); + } +} + +/** + * Sends SIGTERM to every group, then SIGKILL to those still present after `graceMs`, and throws if any is + * still present after a further `graceMs`. The groups share each grace period. Call only for groups proven + * to belong to the dead daemon, under the reclaim mutex. + */ +export async function terminateProcessGroupsAsync( + context: IReapContext, + groupIds: readonly number[] +): Promise { + signalExistingGroups(context, groupIds, 'SIGTERM'); + if (await waitForGroupsExitAsync(context, groupIds)) return 'terminated'; + signalExistingGroups(context, groupIds, 'SIGKILL'); + if (await waitForGroupsExitAsync(context, groupIds)) return 'killed'; + throw new Error(`Processes of dead daemon ${context.deadPid} survived SIGKILL; not reclaiming its socket.`); +} diff --git a/libraries/rush-daemon-transport/src/DaemonListener.ts b/libraries/rush-daemon-transport/src/DaemonListener.ts index df37a5749b..083ba3ae50 100644 --- a/libraries/rush-daemon-transport/src/DaemonListener.ts +++ b/libraries/rush-daemon-transport/src/DaemonListener.ts @@ -9,6 +9,8 @@ import { DaemonFrameConnection } from './DaemonFrameConnection'; import { listenWithReclaimAsync } from './DaemonListenerBinding'; import { DaemonListenerLifetime } from './DaemonListenerLifetime'; import { ensureDaemonRuntimeDir, writeDaemonLockfile } from './DaemonLockfile'; +import { startOperationGroupRecording } from './DaemonOperationGroupRecorder'; +import { getOperationGroupsFolder } from './DaemonOperationGroups'; import { assertDaemonOwnershipAvailable } from './DaemonOwnership'; import type { IDaemonPaths } from './DaemonPaths'; @@ -31,7 +33,10 @@ export interface IDaemonListenerOptions { export class DaemonFrameListener { readonly #lifetime: DaemonListenerLifetime; private constructor(server: net.Server, paths: IDaemonPaths) { - this.#lifetime = new DaemonListenerLifetime(server, paths); + // Record detached operation groups for as long as this process owns the lockfile, so a successor can + // reap them if this daemon dies uncleanly. + const folder: string = getOperationGroupsFolder(paths.lockfilePath, process.pid); + this.#lifetime = new DaemonListenerLifetime(server, paths, startOperationGroupRecording(folder)); } /** Binds the socket/pipe path and writes the PID lockfile. */ public static async listenAsync( diff --git a/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts b/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts index ee6f22141b..90f79a48b6 100644 --- a/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts +++ b/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts @@ -4,17 +4,28 @@ import type * as net from 'node:net'; import { removeDaemonArtifacts } from './DaemonLockfile'; +import type { StopOperationGroupRecording } from './DaemonOperationGroupRecorder'; import type { IDaemonPaths } from './DaemonPaths'; +function keepNoRecords(): void { + // Nothing was recorded before the lockfile was written. +} + export class DaemonListenerLifetime { readonly #paths: IDaemonPaths; readonly #server: net.Server; + readonly #stopRecording: StopOperationGroupRecording; #closePromise: Promise | undefined; #stopPromise: Promise | undefined; - public constructor(server: net.Server, paths: IDaemonPaths) { + public constructor( + server: net.Server, + paths: IDaemonPaths, + stopRecording: StopOperationGroupRecording = keepNoRecords + ) { this.#server = server; this.#paths = paths; + this.#stopRecording = stopRecording; } public stopAcceptingAsync(): Promise { @@ -29,6 +40,7 @@ export class DaemonListenerLifetime { async #closeOnceAsync(): Promise { await this.stopAcceptingAsync(); + this.#stopRecording(); removeDaemonArtifacts(this.#paths.lockfilePath, this.#paths.socketPath); } } diff --git a/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts b/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts new file mode 100644 index 0000000000..f1d4a45068 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts @@ -0,0 +1,75 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { terminateProcessGroupsAsync } from './DaemonGroupTermination'; +import type { DaemonOrphanReapOutcome } from './DaemonGroupTermination'; +import { + getOperationGroupsFolder, + readOperationGroupRecords, + removeOperationGroupRecords +} from './DaemonOperationGroups'; +import type { IOperationGroupRecord } from './DaemonOperationGroups'; +import type { IProcessStat } from './DaemonProcessStat'; +import { createReapContext, isSignalableGroup } from './DaemonReapOptions'; +import type { IDaemonOrphanReaperOptions, IReapContext } from './DaemonReapOptions'; + +const NO_MEMBERS: number = 0; +const LIST_SEPARATOR: string = ', '; + +function isSameLeader(leader: IProcessStat, record: IOperationGroupRecord): boolean { + const { groupId } = record; + return leader.startTime === record.startTime && leader.groupId === groupId && leader.sessionId === groupId; +} + +// Every group lives inside one session. A group whose session is its own id was created by setsid() of +// the recorded leader (or of a reused pid that also called setsid(), which needs the pid space to wrap); +// a shell job's group lives in its shell's session and is rejected. +function isLeaderlessOperationGroup(groupId: number, context: IReapContext): boolean { + const members: IProcessStat[] = context.ops.listLiveGroupMembers(groupId); + return members.length > NO_MEMBERS && members.every((member: IProcessStat) => member.sessionId === groupId); +} + +function isProvenOperationGroup(record: IOperationGroupRecord, context: IReapContext): boolean { + if (!isSignalableGroup(record.groupId, context)) return false; + const leader: IProcessStat | undefined = context.ops.readProcessStat(record.groupId); + return leader ? isSameLeader(leader, record) : isLeaderlessOperationGroup(record.groupId, context); +} + +async function terminateAndLogAsync( + context: IReapContext, + groupIds: number[] +): Promise { + const outcome: DaemonOrphanReapOutcome = await terminateProcessGroupsAsync(context, groupIds); + context.ops.log( + `Reclaimed dead daemon ${context.deadPid}: its orphaned operation process groups ` + + `${groupIds.join(LIST_SEPARATOR)} were ${outcome}.` + ); + return outcome; +} + +/** + * Terminates the detached operation process groups that dead daemon `deadPid` recorded while it ran + * (see `startOperationGroupRecording`), then deletes the records. + * + * @remarks + * A record is signaled only when it provably still names the daemon's operation: its leader is alive with + * the recorded start time and still leads group and session `groupId` (a reused pid has another start + * time), or its leader has exited and every live member of the group is in session `groupId`. Unproven + * records are dropped without a signal. Records survive a failed reap, so the next reclaim retries. + * Call only under the reclaim mutex, after the daemon has been proven dead. + */ +export async function reapDeadDaemonOperationGroupsAsync( + lockfilePath: string, + deadPid: number, + options: IDaemonOrphanReaperOptions = {} +): Promise { + const context: IReapContext = createReapContext(deadPid, options); + const folder: string = getOperationGroupsFolder(lockfilePath, deadPid); + const groupIds: number[] = readOperationGroupRecords(folder) + .filter((record: IOperationGroupRecord) => isProvenOperationGroup(record, context)) + .map((record: IOperationGroupRecord) => record.groupId); + const outcome: DaemonOrphanReapOutcome = + groupIds.length > NO_MEMBERS ? await terminateAndLogAsync(context, groupIds) : 'none'; + removeOperationGroupRecords(folder); + return outcome; +} diff --git a/libraries/rush-daemon-transport/src/DaemonOperationGroupRecorder.ts b/libraries/rush-daemon-transport/src/DaemonOperationGroupRecorder.ts new file mode 100644 index 0000000000..9ea4368ddb --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonOperationGroupRecorder.ts @@ -0,0 +1,70 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { ChildProcess } from 'node:child_process'; +import * as diagnosticsChannel from 'node:diagnostics_channel'; + +import { + removeOperationGroupRecord, + removeOperationGroupRecords, + writeOperationGroupRecord +} from './DaemonOperationGroups'; +import type { IOperationGroupRecord } from './DaemonOperationGroups'; +import { readProcessStat } from './DaemonProcessStat'; +import type { IProcessStat } from './DaemonProcessStat'; + +// Node publishes every new ChildProcess on this built-in channel (since v16.18; built-in channels are +// experimental). If it ever stops publishing, nothing is recorded and reclaim falls back to the daemon group. +const CHILD_PROCESS_CHANNEL: string = 'child_process'; +const SPAWN_EVENT: string = 'spawn'; +const EXIT_EVENT: string = 'exit'; + +interface IChildProcessMessage { + readonly process: ChildProcess; +} + +/** Stops recording and deletes the records; call it when the daemon releases its lockfile. */ +export type StopOperationGroupRecording = () => void; + +function bestEffort(action: () => void): void { + try { + action(); + } catch { + // A lost record only means this group is not reaped if the daemon later dies uncleanly. + } +} + +// A `detached` child (SubprocessTerminator.RECOMMENDED_OPTIONS on POSIX) leads its own group and session, +// so it is outside the daemon's process group and needs a record of its own. +function isGroupAndSessionLeader(stat: IProcessStat | undefined): stat is IProcessStat { + return stat !== undefined && stat.groupId === stat.pid && stat.sessionId === stat.pid; +} + +function recordWhileRunning(child: ChildProcess, folder: string): void { + // 'spawn' is emitted on the next tick after exec, before the child can be reaped, so its pid is still its own. + const stat: IProcessStat | undefined = child.pid === undefined ? undefined : readProcessStat(child.pid); + if (!isGroupAndSessionLeader(stat)) return; + const record: IOperationGroupRecord = { groupId: stat.pid, startTime: stat.startTime }; + bestEffort(() => writeOperationGroupRecord(folder, record)); + child.once(EXIT_EVENT, () => bestEffort(() => removeOperationGroupRecord(folder, record))); +} + +/** + * Records every child this process spawns into its own process group and session, until that child exits, + * so that a successor can reap them if this daemon dies without killing them (SIGKILL, OOM). + * + * @remarks + * Linux only: records need `/proc` start times to rule out pid reuse. Elsewhere this records nothing. + */ +export function startOperationGroupRecording(folder: string): StopOperationGroupRecording { + if (readProcessStat(process.pid) === undefined) return () => undefined; + const onChildProcess = (message: unknown): void => { + const child: ChildProcess = (message as IChildProcessMessage).process; + child.once(SPAWN_EVENT, () => recordWhileRunning(child, folder)); + }; + diagnosticsChannel.subscribe(CHILD_PROCESS_CHANNEL, onChildProcess); + return () => { + diagnosticsChannel.unsubscribe(CHILD_PROCESS_CHANNEL, onChildProcess); + bestEffort(() => removeOperationGroupRecords(folder)); + }; +} diff --git a/libraries/rush-daemon-transport/src/DaemonOperationGroups.ts b/libraries/rush-daemon-transport/src/DaemonOperationGroups.ts new file mode 100644 index 0000000000..57fc2f4fb4 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonOperationGroups.ts @@ -0,0 +1,65 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const FOLDER_INFIX: string = '.groups-'; +const NAME_SEPARATOR: string = '-'; +const RECORD_NAME_PATTERN: RegExp = /^(\d+)-(\d+)$/; +const GROUP_ID_MATCH: number = 1; +const START_TIME_MATCH: number = 2; +const DIR_MODE: number = 0o700; +const FILE_MODE: number = 0o600; +const EMPTY_FILE: string = ''; + +/** A running operation process group: its leader's pid (which is the group id) and start time. */ +export interface IOperationGroupRecord { + readonly groupId: number; + readonly startTime: string; +} + +/** The sidecar folder in which daemon `daemonPid` records the process groups of its running operations. */ +export function getOperationGroupsFolder(lockfilePath: string, daemonPid: number): string { + return `${lockfilePath}${FOLDER_INFIX}${daemonPid}`; +} + +function getRecordPath(folder: string, record: IOperationGroupRecord): string { + return path.join(folder, `${record.groupId}${NAME_SEPARATOR}${record.startTime}`); +} + +/** Records a running group as an empty file whose name is the record, so no record is ever half-written. */ +export function writeOperationGroupRecord(folder: string, record: IOperationGroupRecord): void { + fs.mkdirSync(folder, { recursive: true, mode: DIR_MODE }); + fs.writeFileSync(getRecordPath(folder, record), EMPTY_FILE, { mode: FILE_MODE }); +} + +/** Forgets a group whose leader has exited. */ +export function removeOperationGroupRecord(folder: string, record: IOperationGroupRecord): void { + fs.rmSync(getRecordPath(folder, record), { force: true }); +} + +function parseRecordName(name: string): IOperationGroupRecord | undefined { + const match: RegExpExecArray | null = RECORD_NAME_PATTERN.exec(name); + return match ? { groupId: Number(match[GROUP_ID_MATCH]), startTime: match[START_TIME_MATCH] } : undefined; +} + +function isRecord(record: IOperationGroupRecord | undefined): record is IOperationGroupRecord { + return record !== undefined; +} + +/** Reads the recorded groups; a missing or unreadable folder holds none. */ +export function readOperationGroupRecords(folder: string): IOperationGroupRecord[] { + let names: string[]; + try { + names = fs.readdirSync(folder); + } catch { + return []; + } + return names.map(parseRecordName).filter(isRecord); +} + +/** Deletes the sidecar folder and every record in it; idempotent. */ +export function removeOperationGroupRecords(folder: string): void { + fs.rmSync(folder, { recursive: true, force: true }); +} diff --git a/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts b/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts index bd4c61534b..9de74cb9d3 100644 --- a/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts +++ b/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts @@ -1,75 +1,25 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import { terminateProcessGroupsAsync } from './DaemonGroupTermination'; +import type { DaemonOrphanReapOutcome } from './DaemonGroupTermination'; import type { IDaemonLockfile } from './DaemonLockfile'; -import type { IDaemonProcessGroupOps } from './DaemonProcessGroup'; -import { POSIX_PROCESS_GROUP_OPS } from './DaemonProcessGroup'; +import { reapDeadDaemonOperationGroupsAsync } from './DaemonOperationGroupReaper'; +import { createReapContext, isSignalableGroup } from './DaemonReapOptions'; +import type { IDaemonOrphanReaperOptions, IReapContext } from './DaemonReapOptions'; -const WINDOWS_PLATFORM: NodeJS.Platform = 'win32'; -// 0 and 1 are never daemons, and kill(-0)/kill(-1) would signal our own group or every process. -const FIRST_USER_PID: number = 2; -const POLL_INTERVAL_MS: number = 20; -const DEFAULT_GRACE_MS: number = 2000; - -/** Outcome of reaping a dead daemon's process group. */ -export type DaemonOrphanReapOutcome = 'none' | 'terminated' | 'killed'; - -/** Options for {@link reapDeadDaemonProcessGroupAsync}; every field defaults to the real process. */ -export interface IDaemonOrphanReaperOptions { - readonly ops?: IDaemonProcessGroupOps; - readonly platform?: NodeJS.Platform; - readonly selfPid?: number; - /** How long SIGTERM'd (and then SIGKILL'd) processes get to exit. */ - readonly graceMs?: number; -} - -interface IReapContext { - readonly ops: IDaemonProcessGroupOps; - readonly deadPid: number; - readonly graceMs: number; -} -type OrphanCheck = (pid: number) => boolean; - -function isNeitherSelfNorOwnGroup(pid: number, selfPid: number, ops: IDaemonProcessGroupOps): boolean { - // Fail closed: an unknown own group might be `pid` (e.g. rush-client run by an operation). - const ownGroupId: number | undefined = ops.ownGroupId(); - return pid !== selfPid && ownGroupId !== undefined && pid !== ownGroupId; -} - -function orphanChecks(options: IDaemonOrphanReaperOptions, ops: IDaemonProcessGroupOps): OrphanCheck[] { - return [ - () => (options.platform ?? process.platform) !== WINDOWS_PLATFORM, - (pid: number) => Number.isSafeInteger(pid) && pid >= FIRST_USER_PID, - (pid: number) => isNeitherSelfNorOwnGroup(pid, options.selfPid ?? process.pid, ops), - (pid: number) => !ops.isProcessAlive(pid), - (pid: number) => ops.groupExists(pid) - ]; -} - -async function waitForGroupExitAsync(context: IReapContext): Promise { - const deadline: number = context.ops.now() + context.graceMs; - while (context.ops.now() < deadline) { - if (!context.ops.groupExists(context.deadPid)) return true; - await context.ops.delayAsync(POLL_INTERVAL_MS); - } - return !context.ops.groupExists(context.deadPid); -} - -async function terminateGroupAsync(context: IReapContext): Promise { - context.ops.signalGroup(context.deadPid, 'SIGTERM'); - if (await waitForGroupExitAsync(context)) return 'terminated'; - context.ops.signalGroup(context.deadPid, 'SIGKILL'); - if (await waitForGroupExitAsync(context)) return 'killed'; - throw new Error(`Processes of dead daemon ${context.deadPid} survived SIGKILL; not reclaiming its socket.`); +function isOrphanedDaemonGroup(context: IReapContext): boolean { + return isSignalableGroup(context.deadPid, context) && context.ops.groupExists(context.deadPid); } /** - * Terminates operation processes left behind by a daemon that died without joining them (SIGKILL, OOM). + * Terminates processes left behind in a dead daemon's own process group (SIGKILL, OOM). * * @remarks - * The daemon is spawned detached, so its pid is its process group id, and phased operation children inherit - * that group (children spawned with their own detached group are out of scope). Sends SIGTERM, then SIGKILL - * after `graceMs`, and throws if the group still has not exited after a further `graceMs`. + * The daemon is spawned detached, so its pid is its process group id, and children it spawns without + * `detached` inherit that group. Detached operation children lead groups of their own; see + * {@link reapDeadDaemonOperationGroupsAsync}. Sends SIGTERM, then SIGKILL after `graceMs`, and throws if the + * group still has not exited after a further `graceMs`. * PID-reuse guard: only group `deadPid` is signaled, and only once `deadPid` is proven dead while the group * still exists; POSIX never reuses a pid still in use as a process group id, so every remaining member * belongs to the dead daemon. Never signals the caller's pid or group, and does nothing when the caller's @@ -80,21 +30,19 @@ export async function reapDeadDaemonProcessGroupAsync( options: IDaemonOrphanReaperOptions = {} ): Promise { const context: IReapContext = createReapContext(deadPid, options); - if (!orphanChecks(options, context.ops).every((check: OrphanCheck) => check(deadPid))) return 'none'; - const outcome: DaemonOrphanReapOutcome = await terminateGroupAsync(context); + if (!isOrphanedDaemonGroup(context)) return 'none'; + const outcome: DaemonOrphanReapOutcome = await terminateProcessGroupsAsync(context, [deadPid]); context.ops.log(`Reclaimed dead daemon ${deadPid}: its orphaned operation process group was ${outcome}.`); return outcome; } -function createReapContext(deadPid: number, options: IDaemonOrphanReaperOptions): IReapContext { - return { - ops: options.ops ?? POSIX_PROCESS_GROUP_OPS, - deadPid, - graceMs: options.graceMs ?? DEFAULT_GRACE_MS - }; -} - -/** Reaps the orphaned operations of a reclaimed daemon's recorded owner, if there is one. */ -export async function reapOrphansOfDeadOwnerAsync(owner: IDaemonLockfile | undefined): Promise { - if (owner) await reapDeadDaemonProcessGroupAsync(owner.pid); +/** Reaps the orphaned processes of a reclaimed daemon's recorded owner, if there is one. */ +export async function reapOrphansOfDeadOwnerAsync( + lockfilePath: string, + owner: IDaemonLockfile | undefined, + options: IDaemonOrphanReaperOptions = {} +): Promise { + if (!owner) return; + await reapDeadDaemonProcessGroupAsync(owner.pid, options); + await reapDeadDaemonOperationGroupsAsync(lockfilePath, owner.pid, options); } diff --git a/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts b/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts index a8eb871383..09ba56928d 100644 --- a/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts +++ b/libraries/rush-daemon-transport/src/DaemonProcessGroup.ts @@ -5,6 +5,8 @@ import * as fs from 'node:fs'; import { setTimeout as delayAsync } from 'node:timers/promises'; import { isDaemonProcessAlive } from './DaemonLockfile'; +import { listLiveGroupMembers, readProcessStat } from './DaemonProcessStat'; +import type { IProcessStat } from './DaemonProcessStat'; const NO_SIGNAL: number = 0; const NO_SUCH_PROCESS: string = 'ESRCH'; @@ -15,13 +17,17 @@ const FIELD_SEPARATOR: string = ' '; // After the ")" that ends the command name come: " ...". const PGRP_FIELD_INDEX: number = 3; -/** Process probing/signaling used to reap a dead daemon's process group; injectable for tests. */ +/** Process probing/signaling used to reap a dead daemon's orphaned process groups; injectable for tests. */ export interface IDaemonProcessGroupOps { readonly isProcessAlive: (pid: number) => boolean; readonly groupExists: (groupId: number) => boolean; readonly signalGroup: (groupId: number, signal: NodeJS.Signals) => void; /** The caller's own process group id, or `undefined` when the platform cannot report it. */ readonly ownGroupId: () => number | undefined; + /** Reads a process's `/proc` identity, or `undefined` when it is gone (or there is no `/proc`). */ + readonly readProcessStat: (pid: number) => IProcessStat | undefined; + /** The processes in a group that have not exited. */ + readonly listLiveGroupMembers: (groupId: number) => IProcessStat[]; readonly delayAsync: (ms: number) => Promise; readonly now: () => number; readonly log: (message: string) => void; @@ -70,6 +76,8 @@ export const POSIX_PROCESS_GROUP_OPS: IDaemonProcessGroupOps = { groupExists, signalGroup, ownGroupId, + readProcessStat, + listLiveGroupMembers, delayAsync: async (ms: number) => { await delayAsync(ms); }, diff --git a/libraries/rush-daemon-transport/src/DaemonProcessStat.ts b/libraries/rush-daemon-transport/src/DaemonProcessStat.ts new file mode 100644 index 0000000000..6a1401d8ab --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonProcessStat.ts @@ -0,0 +1,72 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +const PROC_ROOT: string = '/proc'; +const STAT_FILE_NAME: string = 'stat'; +const UTF8: BufferEncoding = 'utf8'; +const COMM_END: string = ')'; +const FIELD_SEPARATOR: string = ' '; +const PID_PATTERN: RegExp = /^\d+$/; +// Indices into the fields after the ")" that ends the command name (proc_pid_stat(5)): +// ") ...", where starttime is field 22 of the whole record. +const STATE_INDEX: number = 1; +const GROUP_INDEX: number = 3; +const SESSION_INDEX: number = 4; +const START_TIME_INDEX: number = 20; +const EXITED_STATES: ReadonlySet = new Set(['Z', 'X']); + +/** The identity fields of one Linux `/proc//stat` record. */ +export interface IProcessStat { + readonly pid: number; + readonly groupId: number; + readonly sessionId: number; + /** Clock ticks after boot when the process started; a reused pid gets a different value. */ + readonly startTime: string; + /** `true` for a zombie: it has exited but its parent has not reaped it yet. */ + readonly exited: boolean; +} + +function parseProcessStat(pid: number, stat: string): IProcessStat { + // The command name may contain spaces and ")", so parse after its last ")". + const fields: string[] = stat.slice(stat.lastIndexOf(COMM_END)).split(FIELD_SEPARATOR); + return { + pid, + groupId: Number(fields[GROUP_INDEX]), + sessionId: Number(fields[SESSION_INDEX]), + startTime: fields[START_TIME_INDEX], + exited: EXITED_STATES.has(fields[STATE_INDEX]) + }; +} + +/** Reads `/proc//stat`; `undefined` when the process is gone or the platform has no `/proc`. */ +export function readProcessStat(pid: number): IProcessStat | undefined { + try { + return parseProcessStat(pid, fs.readFileSync(`${PROC_ROOT}/${pid}/${STAT_FILE_NAME}`, UTF8)); + } catch { + return undefined; + } +} + +function isLiveMemberOf(groupId: number, stat: IProcessStat | undefined): stat is IProcessStat { + return stat !== undefined && stat.groupId === groupId && !stat.exited; +} + +function listProcessIds(): number[] { + try { + return fs + .readdirSync(PROC_ROOT) + .filter((name: string) => PID_PATTERN.test(name)) + .map(Number); + } catch { + return []; + } +} + +/** Lists the processes of group `groupId` that have not exited, by scanning `/proc`. */ +export function listLiveGroupMembers(groupId: number): IProcessStat[] { + return listProcessIds() + .map(readProcessStat) + .filter((stat: IProcessStat | undefined): stat is IProcessStat => isLiveMemberOf(groupId, stat)); +} diff --git a/libraries/rush-daemon-transport/src/DaemonReapOptions.ts b/libraries/rush-daemon-transport/src/DaemonReapOptions.ts new file mode 100644 index 0000000000..550183106c --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonReapOptions.ts @@ -0,0 +1,63 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonProcessGroupOps } from './DaemonProcessGroup'; +import { POSIX_PROCESS_GROUP_OPS } from './DaemonProcessGroup'; + +const WINDOWS_PLATFORM: NodeJS.Platform = 'win32'; +// 0 and 1 are never daemons, and kill(-0)/kill(-1) would signal our own group or every process. +const FIRST_USER_PID: number = 2; +const DEFAULT_GRACE_MS: number = 2000; + +/** Options for reaping a dead daemon's orphans; every field defaults to the real process. */ +export interface IDaemonOrphanReaperOptions { + readonly ops?: IDaemonProcessGroupOps; + readonly platform?: NodeJS.Platform; + readonly selfPid?: number; + /** How long SIGTERM'd (and then SIGKILL'd) processes get to exit. */ + readonly graceMs?: number; +} + +/** {@link IDaemonOrphanReaperOptions} with every default applied, for the dead daemon `deadPid`. */ +export interface IReapContext { + readonly ops: IDaemonProcessGroupOps; + readonly platform: NodeJS.Platform; + readonly selfPid: number; + readonly deadPid: number; + readonly graceMs: number; +} + +function resolveCaller(options: IDaemonOrphanReaperOptions): Pick { + return { platform: options.platform ?? process.platform, selfPid: options.selfPid ?? process.pid }; +} + +/** Applies the defaults of {@link IDaemonOrphanReaperOptions}. */ +export function createReapContext(deadPid: number, options: IDaemonOrphanReaperOptions): IReapContext { + return { + ...resolveCaller(options), + ops: options.ops ?? POSIX_PROCESS_GROUP_OPS, + deadPid, + graceMs: options.graceMs ?? DEFAULT_GRACE_MS + }; +} + +function isNeitherSelfNorOwnGroup(groupId: number, context: IReapContext): boolean { + // Fail closed: an unknown own group might be `groupId` (e.g. rush-client run by an operation). + const ownGroupId: number | undefined = context.ops.ownGroupId(); + return groupId !== context.selfPid && ownGroupId !== undefined && groupId !== ownGroupId; +} + +type GroupCheck = (groupId: number, context: IReapContext) => boolean; + +const SIGNALABLE_GROUP_CHECKS: readonly GroupCheck[] = [ + (groupId: number, context: IReapContext) => context.platform !== WINDOWS_PLATFORM, + (groupId: number) => Number.isSafeInteger(groupId) && groupId >= FIRST_USER_PID, + isNeitherSelfNorOwnGroup, + // A live daemon still owns its operations. + (groupId: number, context: IReapContext) => !context.ops.isProcessAlive(context.deadPid) +]; + +/** `true` when group `groupId` may be signaled at all: POSIX, a user pid, not ours, and its daemon is dead. */ +export function isSignalableGroup(groupId: number, context: IReapContext): boolean { + return SIGNALABLE_GROUP_CHECKS.every((check: GroupCheck) => check(groupId, context)); +} diff --git a/libraries/rush-daemon-transport/src/DaemonReclaim.ts b/libraries/rush-daemon-transport/src/DaemonReclaim.ts index 92f0ed9323..5ce5a8d0e4 100644 --- a/libraries/rush-daemon-transport/src/DaemonReclaim.ts +++ b/libraries/rush-daemon-transport/src/DaemonReclaim.ts @@ -5,11 +5,7 @@ import * as fs from 'node:fs'; import { connectDaemonAsync } from './DaemonConnector'; import type { DaemonFrameConnection } from './DaemonFrameConnection'; -import { - isDaemonProcessAlive, - readDaemonLockfile, - removeDaemonArtifacts -} from './DaemonLockfile'; +import { isDaemonProcessAlive, readDaemonLockfile, removeDaemonArtifacts } from './DaemonLockfile'; import { reapOrphansOfDeadOwnerAsync } from './DaemonOrphanReaper'; import type { IDaemonPaths } from './DaemonPaths'; import { tryAcquireReclaimLock } from './DaemonReclaimLock'; @@ -26,7 +22,8 @@ import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTranspor * lockfile mutex ({@link tryAcquireReclaimLock}): only the mutex holder may * unlink the socket path, so a concurrent starter cannot delete a socket that * another process just bound. Operation processes still running in the dead - * daemon's process group are terminated first (see `DaemonOrphanReaper`). + * daemon's process group, or in the operation process groups it recorded, are + * terminated first (see `DaemonOrphanReaper`). * * @throws {@link DaemonTransportError} with code `daemonAlreadyRunning` when a * live (or plausibly live) daemon owns the path, or when another starter holds @@ -68,7 +65,7 @@ async function reclaimUnderLockAsync(paths: IDaemonPaths): Promise { throwAlreadyRunning(paths, 'it answers a connect probe'); } // A daemon that died uncleanly leaves its operations running; stop them before a successor re-runs them. - await reapOrphansOfDeadOwnerAsync(owner); + await reapOrphansOfDeadOwnerAsync(paths.lockfilePath, owner); removeDaemonArtifacts(paths.lockfilePath, paths.socketPath); } diff --git a/libraries/rush-daemon-transport/src/test/DaemonOperationGroupReaper.test.ts b/libraries/rush-daemon-transport/src/test/DaemonOperationGroupReaper.test.ts new file mode 100644 index 0000000000..a77feb6589 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/DaemonOperationGroupReaper.test.ts @@ -0,0 +1,77 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { reapDeadDaemonOperationGroupsAsync } from '../DaemonOperationGroupReaper'; +import type { IProcessStat } from '../DaemonProcessStat'; + +import { + OPERATION_CHILD, + OPERATION_GROUP, + operationTree, + recordGroups, + recordsRemain, + stat +} from './OperationGroupFixture'; +import { DEAD_PID, createFakeGroup } from './OrphanReaperFixture'; +import type { IFakeGroup } from './OrphanReaperFixture'; + +const REUSED_START: string = '999'; +const SHELL_SESSION: number = 3000; + +async function reapAsync(processes: readonly IProcessStat[]): Promise { + const fake: IFakeGroup = createFakeGroup({ exitsOn: 'SIGTERM', processes }); + const lockfilePath: string = recordGroups([OPERATION_GROUP]); + const outcome: string = await reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, fake.options); + expect(recordsRemain(lockfilePath)).toBe(false); + return { ...fake, outcome }; +} + +it('signals a recorded group whose leader is alive with the recorded start time', async () => { + const result: IFakeGroup & { outcome: string } = await reapAsync(operationTree(OPERATION_GROUP)); + expect(result.outcome).toBe('terminated'); + expect(result.targets).toEqual([OPERATION_GROUP]); + expect(result.logs).toEqual([expect.stringContaining(`groups ${OPERATION_GROUP} were terminated`)]); +}); + +it('signals a group whose leader has exited when every live member is in its session', async () => { + const result: IFakeGroup & { outcome: string } = await reapAsync(operationTree(OPERATION_GROUP, false)); + expect(result.outcome).toBe('terminated'); + expect(result.targets).toEqual([OPERATION_GROUP]); +}); + +it('never signals a reused pid: the leader has another start time', async () => { + const reused: IProcessStat = { ...stat(OPERATION_GROUP, OPERATION_GROUP), startTime: REUSED_START }; + const result: IFakeGroup & { outcome: string } = await reapAsync([reused]); + expect(result.outcome).toBe('none'); + expect(result.signals).toEqual([]); +}); + +it('never signals a leader that no longer leads its own session', async () => { + const result: IFakeGroup & { outcome: string } = await reapAsync([ + stat(OPERATION_GROUP, OPERATION_GROUP, SHELL_SESSION) + ]); + expect(result.signals).toEqual([]); +}); + +it('never signals a leaderless group of another session, such as a shell job with a reused pid', async () => { + const result: IFakeGroup & { outcome: string } = await reapAsync([ + stat(OPERATION_CHILD, OPERATION_GROUP, SHELL_SESSION) + ]); + expect(result.signals).toEqual([]); +}); + +it('never signals a group whose only members have exited', async () => { + const zombie: IProcessStat = { ...stat(OPERATION_CHILD, OPERATION_GROUP), exited: true }; + const result: IFakeGroup & { outcome: string } = await reapAsync([zombie]); + expect(result.outcome).toBe('none'); + expect(result.signals).toEqual([]); +}); + +it('does nothing when the dead daemon recorded no groups', async () => { + const fake: IFakeGroup = createFakeGroup({ processes: operationTree(OPERATION_GROUP) }); + const lockfilePath: string = recordGroups([]); + await expect( + reapDeadDaemonOperationGroupsAsync(`${lockfilePath}.missing`, DEAD_PID, fake.options) + ).resolves.toBe('none'); + expect(fake.signals).toEqual([]); +}); diff --git a/libraries/rush-daemon-transport/src/test/DaemonOperationGroupRecorder.test.ts b/libraries/rush-daemon-transport/src/test/DaemonOperationGroupRecorder.test.ts new file mode 100644 index 0000000000..5b760e4ca7 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/DaemonOperationGroupRecorder.test.ts @@ -0,0 +1,61 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn } from 'node:child_process'; +import type { ChildProcess, SpawnOptions } from 'node:child_process'; +import { once } from 'node:events'; +import * as fs from 'node:fs'; + +import { startOperationGroupRecording } from '../DaemonOperationGroupRecorder'; +import type { StopOperationGroupRecording } from '../DaemonOperationGroupRecorder'; +import { getOperationGroupsFolder, readOperationGroupRecords } from '../DaemonOperationGroups'; +import type { IOperationGroupRecord } from '../DaemonOperationGroups'; +import { readProcessStat } from '../DaemonProcessStat'; + +import { identifyStarted, killStillRunning, waitUntilAsync } from './ProcessWaitFixture'; +import type { IStartedProcess } from './ProcessWaitFixture'; +import { createTestDaemonPaths } from './TestDaemonFixture'; + +const linuxIt: jest.It = process.platform === 'linux' ? it : it.skip; +const SLEEP_ARGS: string[] = ['-e', 'setTimeout(() => {}, 30000)']; +// SubprocessTerminator.RECOMMENDED_OPTIONS on POSIX. +const DETACHED: SpawnOptions = { detached: true, stdio: 'ignore' }; +const ATTACHED: SpawnOptions = { stdio: 'ignore' }; +const NO_RECORDS: number = 0; + +let started: IStartedProcess[] = []; +afterEach(() => { + killStillRunning(started); + started = []; +}); + +async function spawnAsync(options: SpawnOptions): Promise { + const child: ChildProcess = spawn(process.execPath, SLEEP_ARGS, options); + await once(child, 'spawn'); + started = [...started, ...identifyStarted([Number(child.pid)])]; + return child; +} + +function recordOf(child: ChildProcess): IOperationGroupRecord { + const pid: number = Number(child.pid); + return { groupId: pid, startTime: String(readProcessStat(pid)?.startTime) }; +} + +linuxIt('records a detached child while it runs, and no child that shares the daemon group', async () => { + const folder: string = getOperationGroupsFolder(createTestDaemonPaths().lockfilePath, process.pid); + const stop: StopOperationGroupRecording = startOperationGroupRecording(folder); + const detached: ChildProcess = await spawnAsync(DETACHED); + await spawnAsync(ATTACHED); + expect(readOperationGroupRecords(folder)).toEqual([recordOf(detached)]); + detached.kill('SIGKILL'); + expect(await waitUntilAsync(() => readOperationGroupRecords(folder).length === NO_RECORDS)).toBe(true); + stop(); + expect(fs.existsSync(folder)).toBe(false); +}); + +linuxIt('records nothing after it stops', async () => { + const folder: string = getOperationGroupsFolder(createTestDaemonPaths().lockfilePath, process.pid); + startOperationGroupRecording(folder)(); + await spawnAsync(DETACHED); + expect(readOperationGroupRecords(folder)).toEqual([]); +}); diff --git a/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts b/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts index a775495413..1defa97d7c 100644 --- a/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts +++ b/libraries/rush-daemon-transport/src/test/DaemonOrphanReaper.test.ts @@ -2,7 +2,7 @@ // See LICENSE in the project root for license information. import { reapDeadDaemonProcessGroupAsync } from '../DaemonOrphanReaper'; -import type { IDaemonOrphanReaperOptions } from '../DaemonOrphanReaper'; +import type { IDaemonOrphanReaperOptions } from '../DaemonReapOptions'; import { DEAD_PID, SELF_PID, createFakeGroup } from './OrphanReaperFixture'; import type { IFakeGroup } from './OrphanReaperFixture'; diff --git a/libraries/rush-daemon-transport/src/test/DaemonProcessStat.test.ts b/libraries/rush-daemon-transport/src/test/DaemonProcessStat.test.ts new file mode 100644 index 0000000000..fdb0bd95b5 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/DaemonProcessStat.test.ts @@ -0,0 +1,50 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn } from 'node:child_process'; +import type { ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { POSIX_PROCESS_GROUP_OPS } from '../DaemonProcessGroup'; +import { listLiveGroupMembers, readProcessStat } from '../DaemonProcessStat'; +import type { IProcessStat } from '../DaemonProcessStat'; + +const linuxIt: jest.It = process.platform === 'linux' ? it : it.skip; +// Linux caps pid_max at 2^22, so this pid never exists. +const IMPOSSIBLE_PID: number = 4194305; +const SLEEP_SECONDS: string = '30'; +const DIGITS: RegExp = /^\d+$/; + +linuxIt('reads the group, session and start time of a live process', () => { + const self: IProcessStat | undefined = readProcessStat(process.pid); + expect(self).toMatchObject({ + pid: process.pid, + groupId: POSIX_PROCESS_GROUP_OPS.ownGroupId(), + exited: false + }); + expect(self?.startTime).toMatch(DIGITS); + expect(readProcessStat(IMPOSSIBLE_PID)).toBeUndefined(); +}); + +linuxIt('lists the live members of a process group', () => { + const members: IProcessStat[] = listLiveGroupMembers(Number(POSIX_PROCESS_GROUP_OPS.ownGroupId())); + expect(members.map((member: IProcessStat) => member.pid)).toContain(process.pid); +}); + +linuxIt('parses a command name that contains spaces and parentheses', async () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-stat-')); + const executable: string = path.join(folder, 'a) b (c'); + fs.symlinkSync('/bin/sleep', executable); + const child: ChildProcess = spawn(executable, [SLEEP_SECONDS], { detached: true, stdio: 'ignore' }); + await once(child, 'spawn'); + const pid: number = Number(child.pid); + const stat: IProcessStat | undefined = readProcessStat(pid); + child.kill('SIGKILL'); + await once(child, 'exit'); + fs.rmSync(folder, { recursive: true, force: true }); + expect(stat).toMatchObject({ pid, groupId: pid, sessionId: pid, exited: false }); + expect(stat?.startTime).toMatch(DIGITS); +}); diff --git a/libraries/rush-daemon-transport/src/test/FakeOperationDaemonFixture.ts b/libraries/rush-daemon-transport/src/test/FakeOperationDaemonFixture.ts new file mode 100644 index 0000000000..c545fb3a84 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/FakeOperationDaemonFixture.ts @@ -0,0 +1,64 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn } from 'node:child_process'; +import type { ChildProcess, SpawnOptions } from 'node:child_process'; +import { once } from 'node:events'; +import * as path from 'node:path'; +import type { Readable } from 'node:stream'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import { writeDaemonLockfile } from '../DaemonLockfile'; +import type { IDaemonPaths } from '../DaemonPaths'; + +const FAKE_DAEMON_OPTIONS: SpawnOptions = { detached: true, stdio: ['ignore', 'pipe', 'ignore'] }; +const SPACE: string = ' '; +// The fake daemon runs the real recorder, then spawns two operations the way Rush does +// (`detached: true` = SubprocessTerminator.RECOMMENDED_OPTIONS on POSIX), each with a grandchild. The +// first operation's shell waits for its grandchild; the second exits when the daemon's stdin pipe closes, +// which leaves a leaderless group behind. +const FAKE_DAEMON_SCRIPT: string = ` +const [recorder, groups, lockfile] = process.argv.slice(1); +require(recorder).startOperationGroupRecording(require(groups).getOperationGroupsFolder(lockfile, process.pid)); +const spawnOperation = (tail) => require('node:child_process').spawn('/bin/sh', + ['-c', 'sleep 30 /dev/null 2>&1 & echo $!; ' + tail], { detached: true, stdio: ['pipe', 'pipe', 'ignore'] }); +const pidsOf = (op) => new Promise((resolve) => op.stdout.once('data', (d) => resolve(op.pid + ' ' + String(d).trim()))); +Promise.all([spawnOperation('wait'), spawnOperation('read line')].map(pidsOf)) + .then((pids) => process.stdout.write(pids.join(' ') + '\\n')); +setInterval(() => {}, 1000);`; + +/** A fake daemon that recorded two detached operations, each with a grandchild. */ +export interface IFakeDaemon { + readonly daemon: ChildProcess; + /** Leader and grandchild of the waiting operation, then of the exiting one. */ + readonly pids: number[]; +} + +/** Starts the fake daemon and waits until both operations and their grandchildren run. */ +export async function startFakeDaemonAsync(paths: IDaemonPaths): Promise { + const modules: string[] = ['DaemonOperationGroupRecorder', 'DaemonOperationGroups'].map((name: string) => + path.join(__dirname, '..', `${name}.js`) + ); + const args: string[] = ['-e', FAKE_DAEMON_SCRIPT, ...modules, paths.lockfilePath]; + const daemon: ChildProcess = spawn(process.execPath, args, FAKE_DAEMON_OPTIONS); + const [chunk] = (await once(daemon.stdout as Readable, 'data')) as [Buffer]; + return { daemon, pids: chunk.toString().trim().split(SPACE).map(Number) }; +} + +/** SIGKILLs the fake daemon, as the OOM killer would, leaving its operations behind. */ +export async function killFakeDaemonAsync(fake: IFakeDaemon): Promise { + fake.daemon.kill('SIGKILL'); + await once(fake.daemon, 'exit'); + (fake.daemon.stdout as Readable).destroy(); +} + +/** Writes the lockfile the dead fake daemon would have left. */ +export function writeDeadOwnerLockfile(paths: IDaemonPaths, fake: IFakeDaemon): void { + writeDaemonLockfile(paths.lockfilePath, { + pid: Number(fake.daemon.pid), + protocolVersion: DAEMON_PROTOCOL_VERSION, + startedAt: new Date().toISOString(), + socketPath: paths.socketPath + }); +} diff --git a/libraries/rush-daemon-transport/src/test/OperationGroupFixture.ts b/libraries/rush-daemon-transport/src/test/OperationGroupFixture.ts new file mode 100644 index 0000000000..6b045d7e93 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/OperationGroupFixture.ts @@ -0,0 +1,49 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +import { getOperationGroupsFolder, writeOperationGroupRecord } from '../DaemonOperationGroups'; +import type { IOperationGroupRecord } from '../DaemonOperationGroups'; +import type { IProcessStat } from '../DaemonProcessStat'; + +import { DEAD_PID } from './OrphanReaperFixture'; +import { createTestDaemonPaths } from './TestDaemonFixture'; + +/** The leader pid (and group and session id) of the first fake detached operation. */ +export const OPERATION_GROUP: number = 5000; +/** A second fake detached operation. */ +export const OTHER_OPERATION_GROUP: number = 6000; +/** The start time recorded for every fake operation leader. */ +export const RECORDED_START: string = '777'; +const CHILD_OFFSET: number = 1; +/** The pid of the child in {@link OPERATION_GROUP}'s tree. */ +export const OPERATION_CHILD: number = OPERATION_GROUP + CHILD_OFFSET; + +/** A fake `/proc` stat record for a live process. */ +export function stat(pid: number, groupId: number, sessionId: number = groupId): IProcessStat { + return { pid, groupId, sessionId, startTime: RECORDED_START, exited: false }; +} + +/** A fake operation tree: the leader (unless it exited) and one child, both in group and session `groupId`. */ +export function operationTree(groupId: number, leaderAlive: boolean = true): IProcessStat[] { + const child: IProcessStat = stat(groupId + CHILD_OFFSET, groupId); + return leaderAlive ? [stat(groupId, groupId), child] : [child]; +} + +/** A lockfile path whose dead daemon ({@link DEAD_PID}) recorded `groupIds`. */ +export function recordGroups(groupIds: readonly number[]): string { + const { lockfilePath } = createTestDaemonPaths(); + const folder: string = getOperationGroupsFolder(lockfilePath, DEAD_PID); + fs.mkdirSync(folder, { recursive: true }); + for (const groupId of groupIds) { + const record: IOperationGroupRecord = { groupId, startTime: RECORDED_START }; + writeOperationGroupRecord(folder, record); + } + return lockfilePath; +} + +/** `true` when the dead daemon's records for `lockfilePath` still exist. */ +export function recordsRemain(lockfilePath: string): boolean { + return fs.existsSync(getOperationGroupsFolder(lockfilePath, DEAD_PID)); +} diff --git a/libraries/rush-daemon-transport/src/test/OperationGroupReapOnReclaim.test.ts b/libraries/rush-daemon-transport/src/test/OperationGroupReapOnReclaim.test.ts new file mode 100644 index 0000000000..8051359a9e --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/OperationGroupReapOnReclaim.test.ts @@ -0,0 +1,59 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { getOperationGroupsFolder, readOperationGroupRecords } from '../DaemonOperationGroups'; +import type { IOperationGroupRecord } from '../DaemonOperationGroups'; +import type { IDaemonPaths } from '../DaemonPaths'; +import { reclaimStaleDaemonAsync } from '../DaemonReclaim'; + +import { + killFakeDaemonAsync, + startFakeDaemonAsync, + writeDeadOwnerLockfile +} from './FakeOperationDaemonFixture'; +import type { IFakeDaemon } from './FakeOperationDaemonFixture'; +import { identifyStarted, isAlive, isGone, killStillRunning, waitUntilAsync } from './ProcessWaitFixture'; +import type { IStartedProcess } from './ProcessWaitFixture'; +import { createTestDaemonPaths } from './TestDaemonFixture'; + +const posixIt: jest.It = process.platform === 'linux' ? it : it.skip; +// Above the worst case of two 5 s polls plus the reaper's SIGTERM and SIGKILL grace periods, so that a +// regression fails an assertion instead of timing out. +const REAP_TEST_TIMEOUT_MS: number = 30000; + +function recordedGroupIds(folder: string): Set { + return new Set(readOperationGroupRecords(folder).map((record: IOperationGroupRecord) => record.groupId)); +} + +let started: IStartedProcess[] = []; +afterEach(() => { + // A failed assertion must not leave this test's processes running. + killStillRunning(started); + started = []; + jest.restoreAllMocks(); +}); + +async function reapsOrphanedOperationGroupsAsync(): Promise { + const warning: jest.SpyInstance = jest.spyOn(process, 'emitWarning').mockImplementation(() => undefined); + const paths: IDaemonPaths = createTestDaemonPaths(); + const fake: IFakeDaemon = await startFakeDaemonAsync(paths); + started = identifyStarted([Number(fake.daemon.pid), ...fake.pids]); + const [waitingLeader, , exitingLeader] = fake.pids; + const folder: string = getOperationGroupsFolder(paths.lockfilePath, Number(fake.daemon.pid)); + expect(recordedGroupIds(folder)).toEqual(new Set([waitingLeader, exitingLeader])); + await killFakeDaemonAsync(fake); + // Reaped, not merely a zombie, so reclaim finds a group without its leader. + expect(await waitUntilAsync(() => isGone(exitingLeader))).toBe(true); + writeDeadOwnerLockfile(paths, fake); + await reclaimStaleDaemonAsync(paths); + expect(await waitUntilAsync(() => !fake.pids.some(isAlive))).toBe(true); + expect(readOperationGroupRecords(folder)).toEqual([]); + expect(warning).toHaveBeenCalledWith(expect.stringContaining(`${waitingLeader}`), expect.anything()); + expect(warning).toHaveBeenCalledWith(expect.stringContaining(`${exitingLeader}`), expect.anything()); +} + +posixIt( + 'reaps detached operation groups orphaned by a SIGKILLed daemon, with or without their leader', + reapsOrphanedOperationGroupsAsync, + REAP_TEST_TIMEOUT_MS +); diff --git a/libraries/rush-daemon-transport/src/test/OperationGroupTermination.test.ts b/libraries/rush-daemon-transport/src/test/OperationGroupTermination.test.ts new file mode 100644 index 0000000000..dfb3c7dc42 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/OperationGroupTermination.test.ts @@ -0,0 +1,83 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { reapDeadDaemonOperationGroupsAsync } from '../DaemonOperationGroupReaper'; +import type { IProcessStat } from '../DaemonProcessStat'; +import type { IDaemonOrphanReaperOptions } from '../DaemonReapOptions'; + +import { + OPERATION_GROUP, + OTHER_OPERATION_GROUP, + operationTree, + recordGroups, + recordsRemain, + stat +} from './OperationGroupFixture'; +import { DEAD_PID, SELF_PID, createFakeGroup } from './OrphanReaperFixture'; +import type { IFakeGroup, IFakeGroupSpec } from './OrphanReaperFixture'; + +const BOTH_GROUPS: readonly number[] = [OPERATION_GROUP, OTHER_OPERATION_GROUP]; +const SWAPPER_PID: number = 0; +const INIT_PID: number = 1; +const UNSAFE_GROUPS: readonly number[] = [SWAPPER_PID, INIT_PID, SELF_PID]; + +function createOperations(spec: IFakeGroupSpec): IFakeGroup { + return createFakeGroup({ + ...spec, + processes: [...operationTree(OPERATION_GROUP), ...operationTree(OTHER_OPERATION_GROUP)] + }); +} + +it('signals every proven group together and escalates the survivors to SIGKILL', async () => { + const fake: IFakeGroup = createOperations({ exitsOn: 'SIGKILL' }); + const lockfilePath: string = recordGroups(BOTH_GROUPS); + await expect(reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, fake.options)).resolves.toBe( + 'killed' + ); + expect(fake.signals).toEqual(['SIGTERM', 'SIGTERM', 'SIGKILL', 'SIGKILL']); + expect(fake.targets).toEqual([...BOTH_GROUPS, ...BOTH_GROUPS]); + expect(fake.logs).toEqual([ + expect.stringContaining(`${OPERATION_GROUP}, ${OTHER_OPERATION_GROUP} were killed`) + ]); + expect(recordsRemain(lockfilePath)).toBe(false); +}); + +it('fails the reclaim and keeps the records when a group survives SIGKILL', async () => { + const fake: IFakeGroup = createOperations({}); + const lockfilePath: string = recordGroups(BOTH_GROUPS); + await expect(reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, fake.options)).rejects.toThrow( + /survived SIGKILL/ + ); + expect(recordsRemain(lockfilePath)).toBe(true); +}); + +it.each<[string, IFakeGroupSpec, IDaemonOrphanReaperOptions]>([ + ['the daemon is alive', { daemonAlive: true }, {}], + ['the caller leads the group', { ownGroupId: OPERATION_GROUP }, {}], + ['the caller group is unknown', { unknownOwnGroup: true }, {}], + ['the caller is the leader', {}, { selfPid: OPERATION_GROUP }], + ['on Windows', {}, { platform: 'win32' }] +])( + 'never signals a recorded group when %s', + async (name: string, spec: IFakeGroupSpec, overrides: IDaemonOrphanReaperOptions) => { + const fake: IFakeGroup = createOperations({ ...spec, exitsOn: 'SIGTERM' }); + const lockfilePath: string = recordGroups([OPERATION_GROUP]); + const options: IDaemonOrphanReaperOptions = { ...fake.options, ...overrides }; + await expect(reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, options)).resolves.toBe('none'); + expect(fake.signals).toEqual([]); + } +); + +it('never signals the kernel, init or the caller, even as proven leaders', async () => { + const leaders: IProcessStat[] = UNSAFE_GROUPS.map((groupId: number) => stat(groupId, groupId)); + const fake: IFakeGroup = createFakeGroup({ + ownGroupId: OPERATION_GROUP, + exitsOn: 'SIGTERM', + processes: leaders + }); + const lockfilePath: string = recordGroups(UNSAFE_GROUPS); + await expect(reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, fake.options)).resolves.toBe( + 'none' + ); + expect(fake.signals).toEqual([]); +}); diff --git a/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts b/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts index c0e1f8bede..59fdcb4014 100644 --- a/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts +++ b/libraries/rush-daemon-transport/src/test/OrphanReaperFixture.ts @@ -1,8 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import type { IDaemonOrphanReaperOptions } from '../DaemonOrphanReaper'; import type { IDaemonProcessGroupOps } from '../DaemonProcessGroup'; +import type { IProcessStat } from '../DaemonProcessStat'; +import type { IDaemonOrphanReaperOptions } from '../DaemonReapOptions'; /** The pid of the fake dead daemon, which is also its process group id. */ export const DEAD_PID: number = 4242; @@ -14,6 +15,8 @@ const CLOCK_START: number = 0; /** A fake process table recording every signal sent and every message logged. */ export interface IFakeGroup { readonly signals: NodeJS.Signals[]; + /** The group id of every signal in {@link IFakeGroup.signals}, in the same order. */ + readonly targets: number[]; readonly logs: string[]; readonly options: IDaemonOrphanReaperOptions; } @@ -21,30 +24,74 @@ export interface IFakeGroup { /** Describes the fake process table. */ export interface IFakeGroupSpec { readonly daemonAlive?: boolean; - /** The signal after which the group is gone; omitted means it never exits. */ + /** The signal after which every group is gone; omitted means they never exit. */ readonly exitsOn?: NodeJS.Signals; readonly ownGroupId?: number; readonly unknownOwnGroup?: boolean; readonly anyGroupExists?: boolean; + /** Processes outside the dead daemon's own group, such as detached operation trees. */ + readonly processes?: readonly IProcessStat[]; } -/** Creates a fake process table with a virtual clock. */ -export function createFakeGroup(spec: IFakeGroupSpec): IFakeGroup { - const signals: NodeJS.Signals[] = []; - const logs: string[] = []; +interface IFakeTable extends Omit { + readonly spec: IFakeGroupSpec; +} + +function hasExited(table: IFakeTable): boolean { + return table.signals.some((signal: NodeJS.Signals) => signal === table.spec.exitsOn); +} + +function liveProcesses(table: IFakeTable): readonly IProcessStat[] { + return hasExited(table) ? [] : (table.spec.processes ?? []); +} + +function isKnownGroup(table: IFakeTable, groupId: number): boolean { + return table.spec.anyGroupExists === true || groupId === DEAD_PID; +} + +function groupExists(table: IFakeTable, groupId: number): boolean { + const inTable: boolean = liveProcesses(table).some((stat: IProcessStat) => stat.groupId === groupId); + return !hasExited(table) && (isKnownGroup(table, groupId) || inTable); +} + +function createProcessOps( + table: IFakeTable +): Pick { + return { + readProcessStat: (pid: number) => liveProcesses(table).find((stat: IProcessStat) => stat.pid === pid), + listLiveGroupMembers: (groupId: number) => + liveProcesses(table).filter((stat: IProcessStat) => stat.groupId === groupId && !stat.exited) + }; +} + +function createGroupOps(table: IFakeTable): IDaemonProcessGroupOps { + const { spec } = table; let clock: number = CLOCK_START; - const ops: IDaemonProcessGroupOps = { + return { isProcessAlive: () => spec.daemonAlive === true, - groupExists: (groupId: number) => - (spec.anyGroupExists === true || groupId === DEAD_PID) && - !signals.some((signal: NodeJS.Signals) => signal === spec.exitsOn), - signalGroup: (groupId: number, signal: NodeJS.Signals) => signals.push(signal), + groupExists: (groupId: number) => groupExists(table, groupId), + signalGroup: (groupId: number, signal: NodeJS.Signals) => { + table.targets.push(groupId); + table.signals.push(signal); + }, ownGroupId: () => (spec.unknownOwnGroup === true ? undefined : (spec.ownGroupId ?? SELF_PID)), + ...createProcessOps(table), delayAsync: async (ms: number) => { clock += ms; }, now: () => clock, - log: (message: string) => logs.push(message) + log: (message: string) => table.logs.push(message) + }; +} + +/** Creates a fake process table with a virtual clock. */ +export function createFakeGroup(spec: IFakeGroupSpec): IFakeGroup { + const table: IFakeTable = { spec, signals: [], targets: [], logs: [] }; + const options: IDaemonOrphanReaperOptions = { + ops: createGroupOps(table), + platform: 'linux', + selfPid: SELF_PID, + graceMs: GRACE_MS }; - return { signals, logs, options: { ops, platform: 'linux', selfPid: SELF_PID, graceMs: GRACE_MS } }; + return { signals: table.signals, targets: table.targets, logs: table.logs, options }; } diff --git a/libraries/rush-daemon-transport/src/test/ProcessWaitFixture.ts b/libraries/rush-daemon-transport/src/test/ProcessWaitFixture.ts new file mode 100644 index 0000000000..546c6007c6 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/ProcessWaitFixture.ts @@ -0,0 +1,53 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { readProcessStat } from '../DaemonProcessStat'; +import type { IProcessStat } from '../DaemonProcessStat'; + +const FIRST_ATTEMPT: number = 0; +const POLL_ATTEMPTS: number = 250; +const POLL_INTERVAL_MS: number = 20; + +/** `true` while `pid` runs; a zombie that has exited does not count. */ +export function isAlive(pid: number): boolean { + const stat: IProcessStat | undefined = readProcessStat(pid); + return stat !== undefined && !stat.exited; +} + +/** `true` once `pid` has exited and been reaped, so `/proc` has no record of it. */ +export function isGone(pid: number): boolean { + return readProcessStat(pid) === undefined; +} + +/** Polls `condition` for up to 5 s. */ +export async function waitUntilAsync(condition: () => boolean): Promise { + for (let attempt: number = FIRST_ATTEMPT; attempt < POLL_ATTEMPTS; attempt++) { + if (condition()) return true; + await delayAsync(POLL_INTERVAL_MS); + } + return condition(); +} + +/** A process a test started, identified by pid and start time so that a reused pid is never signaled. */ +export interface IStartedProcess { + readonly pid: number; + readonly startTime: string | undefined; +} + +/** Identifies processes a test has just started. */ +export function identifyStarted(pids: readonly number[]): IStartedProcess[] { + return pids.map((pid: number) => ({ pid, startTime: readProcessStat(pid)?.startTime })); +} + +function isStillRunning({ pid, startTime }: IStartedProcess): boolean { + return startTime !== undefined && readProcessStat(pid)?.startTime === startTime; +} + +/** SIGKILLs the started processes that are still the same processes; for test cleanup only. */ +export function killStillRunning(started: readonly IStartedProcess[]): void { + for (const { pid } of started.filter(isStillRunning)) { + process.kill(pid, 'SIGKILL'); + } +} diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index dda743a498..b132616378 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -8,7 +8,7 @@ import type { Operation, _IOperationGraphEventSink } from '@microsoft/rush-lib'; -import { OperationStatus } from '@microsoft/rush-lib'; +import { getWorkspaceRequestOperationEnvironment, OperationStatus } from '@microsoft/rush-lib'; import { Sort } from '@rushstack/node-core-library'; import type { IDaemonPhasedEngineShape, @@ -458,7 +458,8 @@ class PhasedRequestBatchCoordinator { try { for (const entry of participants) entry.onExecutionStarting?.(); scheduled = await this.#graph.scheduleIterationAsync({ - inputsSnapshot: this.#workspaceSession.inputsSnapshot + inputsSnapshot: this.#workspaceSession.inputsSnapshot, + getOperationEnvironment: createOperationEnvironmentLookup(participants) }); if (scheduled) { await Promise.all( @@ -875,6 +876,31 @@ function validateRequestIdentity(request: IDaemonPhasedRequest): void { } } +/** + * Gives each operation of an iteration the environment of the first participant that selected it, as a native + * command would run it in its invoker's environment. An operation that several participants share runs once, in + * the first participant's environment. + */ +function createOperationEnvironmentLookup( + participants: ReadonlyArray +): (operation: Operation) => Readonly> { + const environmentByOperation: Map>> = new Map(); + let firstEnvironment: Readonly> | undefined; + for (const entry of participants) { + const environment: Readonly> = getWorkspaceRequestOperationEnvironment( + process.env, + entry.request.environment + ); + firstEnvironment ??= environment; + for (const operation of entry.selection.activeOperations) { + if (!environmentByOperation.has(operation)) { + environmentByOperation.set(operation, environment); + } + } + } + return (operation: Operation) => environmentByOperation.get(operation) ?? firstEnvironment ?? process.env; +} + function validateNonemptyName(value: string, kind: string): void { if (value.length === 0 || value.trim() !== value) { throw new Error(`Invalid phased request ${kind}: "${value}".`); diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 4cdf68c46e..73249df527 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -5,6 +5,7 @@ import * as path from 'node:path'; import type { LockFile } from '@rushstack/node-core-library'; import { + EnvironmentVariableNames, getWorkspaceFingerprintEnvironmentEntries, PhasedCommandEngine, PhasedCommandEngineBusyError, @@ -129,7 +130,11 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { commandName: envelope.commandName, commandOrigin: envelope.commandOrigin, engineShape: shape, - environment: envelope.environment, + // Native Rush assigns the invocation's folder at CLI startup (Rush._assignRushInvokedFolder). + environment: { + ...envelope.environment, + [EnvironmentVariableNames.RUSH_INVOKED_FOLDER]: envelope.cwd + }, operationSelection, requestId: envelope.requestId, terminalRequirement: envelope.terminal.terminalRequirement @@ -162,6 +167,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { command = await PhasedCommandEngine.parseAsync({ argv: envelope.argv, cwd: envelope.cwd, + environment: envelope.environment, rushConfiguration: workspaceSession.rushConfiguration, terminalProvider: terminal }); diff --git a/libraries/rush-daemon/src/RushXDaemonRequestResolver.ts b/libraries/rush-daemon/src/RushXDaemonRequestResolver.ts index 0d08c2f830..0ccd334714 100644 --- a/libraries/rush-daemon/src/RushXDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/RushXDaemonRequestResolver.ts @@ -12,6 +12,7 @@ import { daemonEnvironmentVariables, RushConfiguration, RushXCommand, + workspaceRequestScopedEnvironmentVariables, type IRushXCommandLineArguments } from '@microsoft/rush-lib'; import { EnvironmentMap, FileSystem, JsonFile } from '@rushstack/node-core-library'; @@ -40,6 +41,8 @@ export class RushXDaemonRequestResolver implements IDaemonRequestResolver { readonly #startupEnvironment: NodeJS.ProcessEnv = { ...process.env }; readonly #requestLocalRushVariables: ReadonlySet = new Set([ ...Object.values(daemonEnvironmentVariables), + // A daemon host starts without these, so any request that sets one differs from the startup environment. + ...workspaceRequestScopedEnvironmentVariables, 'RUSH_DAEMON_EXPERIMENTAL', 'RUSH_INVOKED_FOLDER', 'RUSH_QUIET_MODE' diff --git a/libraries/rush-daemon/src/VersionSelectedDaemonLauncher.ts b/libraries/rush-daemon/src/VersionSelectedDaemonLauncher.ts index f931b84277..f5c82ab50b 100644 --- a/libraries/rush-daemon/src/VersionSelectedDaemonLauncher.ts +++ b/libraries/rush-daemon/src/VersionSelectedDaemonLauncher.ts @@ -6,7 +6,7 @@ import { once } from 'node:events'; import * as fs from 'node:fs/promises'; import * as path from 'node:path'; -import { _FlagFile } from '@microsoft/rush-lib'; +import { _FlagFile, getWorkspaceHostEnvironment } from '@microsoft/rush-lib'; import { EnvironmentConfiguration } from '@microsoft/rush-lib/lib/api/EnvironmentConfiguration'; import { DependencySpecifier, @@ -77,13 +77,8 @@ export function getSelectedDaemonStartCommand( context.repoRoot ], cwd: context.repoRoot, - environment: Object.freeze( - Object.fromEntries( - Object.entries(context.environment).filter( - (entry): entry is [string, string] => entry[1] !== undefined - ) - ) - ) + // The daemon outlives the client that starts it, so it keeps none of that client's request-scoped values. + environment: Object.freeze(getWorkspaceHostEnvironment(context.environment)) }; } diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index 1cde7c35a6..aab404064f 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -10,7 +10,7 @@ import type { } from '@rushstack/rush-daemon-protocol'; import { RUSHD_OPERATION_HEADER, RUSHD_OPERATION_STREAM_CLOSED } from '@rushstack/rush-daemon-protocol'; import { OperationStatus } from '@microsoft/rush-lib'; -import type { IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; +import type { IOperationRunnerContext, IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; import { @@ -24,6 +24,9 @@ import type { ITestClientWrite, ITestRoutingFixture } from './PhasedRequestRoute const OPERATION_A: string = 'project-a (_phase:test)'; const OPERATION_B: string = 'project-b (_phase:test)'; const OPERATION_C: string = 'project-c (_phase:test)'; +const SESSION_VARIABLE: string = 'COPILOT_AGENT_SESSION_ID'; + +type TestOperationAction = (terminal: ITerminal, context: IOperationRunnerContext) => Promise; interface IDeferred { readonly promise: Promise; @@ -57,9 +60,9 @@ function createRequest( } function createFixture(options?: { - readonly actionAAsync?: (terminal: ITerminal) => Promise; - readonly actionBAsync?: (terminal: ITerminal) => Promise; - readonly actionCAsync?: (terminal: ITerminal) => Promise; + readonly actionAAsync?: TestOperationAction; + readonly actionBAsync?: TestOperationAction; + readonly actionCAsync?: TestOperationAction; readonly statusA?: OperationStatus; }): ITestRoutingFixture { return createRoutingFixture( @@ -197,6 +200,57 @@ describe('shared phased request batching', () => { expect(getResultOperationIds(consumer)).toEqual([OPERATION_A, OPERATION_B]); }); + it('gives each operation of a shared iteration the environment of the first request that selected it', async () => { + const sessions: Map = new Map(); + const record = + (operationId: string): TestOperationAction => + async (terminal: ITerminal, context: IOperationRunnerContext): Promise => { + sessions.set(operationId, context.environment?.[SESSION_VARIABLE]); + }; + const fixture: ITestRoutingFixture = createFixture({ + actionAAsync: record(OPERATION_A), + actionBAsync: record(OPERATION_B), + actionCAsync: record(OPERATION_C) + }); + const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const withSession = (request: IDaemonPhasedRequest, session?: string): IDaemonPhasedRequest => ({ + ...request, + environment: session === undefined ? {} : { [SESSION_VARIABLE]: session } + }); + const daemonSession: string | undefined = process.env[SESSION_VARIABLE]; + process.env[SESSION_VARIABLE] = 'daemon'; + try { + await Promise.all([ + router.executeAsync( + withSession(createRequest('a', OPERATION_A), 'session-A'), + new TestPhasedRequestClient('one') + ), + // Project B depends on project A, which the first request already selected. + router.executeAsync( + withSession(createRequest('b', OPERATION_B), 'session-B'), + new TestPhasedRequestClient('two') + ), + router.executeAsync( + withSession(createRequest('c', OPERATION_C)), + new TestPhasedRequestClient('three') + ) + ]); + } finally { + if (daemonSession === undefined) delete process.env[SESSION_VARIABLE]; + else process.env[SESSION_VARIABLE] = daemonSession; + } + + expect(scheduleSpy).toHaveBeenCalledTimes(1); + expect(sessions).toEqual( + new Map([ + [OPERATION_A, 'session-A'], + [OPERATION_B, 'session-B'], + [OPERATION_C, undefined] + ]) + ); + }); + it('shares one iteration for disjoint selections while isolating streams, events, and results', async () => { const fixture: ITestRoutingFixture = createFixture({ actionAAsync: async (terminal: ITerminal): Promise => terminal.writeLine('only-a'), diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index bdf1634744..dd82aa7c09 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -218,6 +218,11 @@ if (fs.existsSync(gateFile)) { }); } fs.appendFileSync('../../runs.txt', name + ':' + input + ':' + process.argv.slice(2).join(' ') + '\\n'); +const environmentFile = path.resolve('../../common/temp/operation-environment.txt'); +if (fs.existsSync(environmentFile)) { + const { COPILOT_AGENT_SESSION_ID = null, RUSH_INVOKED_FOLDER = null } = process.env; + fs.appendFileSync(environmentFile, JSON.stringify([name, COPILOT_AGENT_SESSION_ID, RUSH_INVOKED_FOLDER]) + '\\n'); +} fs.mkdirSync('lib', { recursive: true }); fs.writeFileSync('lib/output.txt', input); console.log('built-' + name + '-' + input); @@ -349,6 +354,49 @@ function requestEnvironment(): Record { } describe('native production daemon engine', () => { + it("runs each request's operations with its requester's session and invocation folder", async () => { + const fixture: IFixture = await createFixtureAsync(); + const recordFile: string = path.join(fixture.repoRoot, 'common/temp/operation-environment.txt'); + const projectFolder: string = path.join(fixture.repoRoot, 'projects/a'); + const daemonSession: string | undefined = process.env.COPILOT_AGENT_SESSION_ID; + // This in-process host's own value stands in for the session that started the daemon. + process.env.COPILOT_AGENT_SESSION_ID = 'daemon-starter'; + try { + fs.writeFileSync(recordFile, ''); + const requests: ReadonlyArray = [ + ['first-a', 'session-A', fixture.repoRoot], + ['then-b', 'session-B', projectFolder], + ['again-a', 'session-A', fixture.repoRoot], + ['unset', undefined, fixture.repoRoot] + ]; + for (const [requestId, session, cwd] of requests) { + const environment: Record = requestEnvironment(); + delete environment.COPILOT_AGENT_SESSION_ID; + if (session !== undefined) environment.COPILOT_AGENT_SESSION_ID = session; + const exchange: ITerminalExchange = await runAsync(fixture, requestId, ['rebuild', '--only', 'a'], { + cwd, + environment + }); + expect(exchange.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + } + const records: unknown[] = fs + .readFileSync(recordFile, 'utf8') + .trim() + .split('\n') + .map((line: string) => JSON.parse(line)); + expect(records).toEqual([ + ['a', 'session-A', fixture.repoRoot], + ['a', 'session-B', projectFolder], + ['a', 'session-A', fixture.repoRoot], + ['a', null, fixture.repoRoot] + ]); + } finally { + if (daemonSession === undefined) delete process.env.COPILOT_AGENT_SESSION_ID; + else process.env.COPILOT_AGENT_SESSION_ID = daemonSession; + await fixture[Symbol.asyncDispose](); + } + }); + it('preserves resolver decoration and isolated invocation routing across native generation reloads', async () => { const events: string[] = []; const fixture: IFixture = await createFixtureAsync(false, 'direct', { @@ -418,6 +466,31 @@ describe('native production daemon engine', () => { } }); + it('serves client output and request-scoped settings in the same generation instead of restarting', async () => { + const fixture: IFixture = await createFixtureAsync(); + try { + await runAsync(fixture, 'initial-settings', ['build', '--only', 'a']); + const session: WorkspaceSession = fixture.session; + const graph: IOperationGraph | undefined = session.operationGraph; + const environment: Record = { + ...requestEnvironment(), + RUSH_PARALLELISM: process.env.RUSH_PARALLELISM === '1' ? '2' : '1', + RUSHD_OUTPUT: process.env.RUSHD_OUTPUT === 'legacy' ? 'agent' : 'legacy', + COPILOT_AGENT_SESSION_ID: 'another-agent-session' + }; + expect( + (await runAsync(fixture, 'request-scoped-settings', ['build', '--only', 'a'], { environment })) + .terminal + ).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0, scheduled: false } }); + expect(fixture.session).toBe(session); + expect(fixture.session.operationGraph).toBe(graph); + expect(readDaemonLockfile(fixture.host.paths.lockfilePath)?.pid).toBe(process.pid); + expect(runs(fixture)).toEqual(['a:one:']); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('keeps the selected native SDK handoff instead of restarting for a foreign client engine path', async () => { const fixture: IFixture = await createFixtureAsync(); try { @@ -598,7 +671,7 @@ describe('native production daemon engine', () => { const oldGraph: IOperationGraph = fixture.session.operationGraph!; const environment: Record = { ...requestEnvironment(), - RUSH_PARALLELISM: process.env.RUSH_PARALLELISM === '1' ? '2' : '1' + RUSHSTACK_DAEMON_TEST_HARD_INPUT: 'changed' }; expect( (await runAsync(fixture, 'hard', ['build', '--only', 'a'], { environment })).terminal diff --git a/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts b/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts index 34ea43c4ee..1187628df5 100644 --- a/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts +++ b/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts @@ -90,6 +90,26 @@ describe('version-selected daemon launcher', () => { expect(process.env._RUSH_LIB_PATH).toBe(originalRushLibPath); }); + it("starts the daemon without the starting client's request-scoped variables", () => { + const command = getSelectedDaemonStartCommand(path.join(repoRoot, 'package.json'), { + ...context, + environment: { + HOME: '/home/user', + RUSH_PARALLELISM: '48', + COPILOT_AGENT_SESSION_ID: 'session-1', + RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', + RUSHD_OUTPUT: 'agent', + UNSET: undefined + } + }); + expect(command.environment).toEqual({ + HOME: '/home/user', + RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', + RUSHD_OUTPUT: 'agent' + }); + expect(Object.isFrozen(command.environment)).toBe(true); + }); + it('never relabels the bundled engine as a different requested version', async () => { fs.writeFileSync(path.join(repoRoot, 'rush.json'), JSON.stringify({ rushVersion: '5.178.1' })); await expect( diff --git a/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts b/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts index afc7bc6a13..b8d770ac40 100644 --- a/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as path from 'node:path'; + import { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; @@ -81,7 +83,24 @@ it('reuses the warm generation when only volatile per-shell environment variable _: '/usr/bin/env' }, terminal: { ...fixture.environment, TERM: 'dumb', COLUMNS: '91', WSL_INTEROP: '/run/WSL/1_interop' }, - routing: { ...fixture.environment, RUSH_DAEMON: '1', RUSH_DAEMON_EXPERIMENTAL: '1' } + routing: { ...fixture.environment, RUSH_DAEMON: '1', RUSH_DAEMON_EXPERIMENTAL: '1' }, + client: { + ...fixture.environment, + RUSHD_OUTPUT: 'legacy', + RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: '600', + RUSH_PARALLELISM: '2' + }, + session: { + ...fixture.environment, + INVOCATION_ID: 'b0f1', + COPILOT_CLI: '1', + COPILOT_AGENT_SESSION_ID: 'another-session', + VSCODE_IPC_HOOK_CLI: '/run/vscode-ipc.sock' + }, + repeatedPath: { + ...fixture.environment, + PATH: [fixture.environment.PATH, fixture.environment.PATH].join(path.delimiter) + } })) { const result = await fixture.runAsync(['build', '--to', 'b', '--parallelism', '3'], { environment }); expect(result.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); diff --git a/libraries/rush-lib/src/api/PhasedCommandEngine.ts b/libraries/rush-lib/src/api/PhasedCommandEngine.ts index 5c085cf201..707faae6ec 100644 --- a/libraries/rush-lib/src/api/PhasedCommandEngine.ts +++ b/libraries/rush-lib/src/api/PhasedCommandEngine.ts @@ -41,6 +41,12 @@ export interface IPhasedCommandEngine extends AsyncDisposable { export interface IParsePhasedCommandOptions { readonly argv: ReadonlyArray; readonly cwd: string; + /** + * The environment of the client that sent the command. It supplies the defaults of environment-backed + * parameters (`RUSH_PARALLELISM`), because a long-lived host's own environment belongs to no request. + * Defaults to `process.env`. + */ + readonly environment?: Readonly>; readonly rushConfiguration: RushConfiguration; readonly terminalProvider: ITerminalProvider; } @@ -83,7 +89,7 @@ export class PhasedCommandEngine { } public static async parseAsync(options: IParsePhasedCommandOptions): Promise { - const { rushConfiguration, terminalProvider, cwd, argv } = options; + const { rushConfiguration, terminalProvider, cwd, argv, environment = process.env } = options; const resolvedCwd: string = await resolvePhasedCommandCwdAsync(cwd, rushConfiguration.rushJsonFolder); if (argv.length === 0 || argv.includes('--help') || argv.includes('-h')) { throw new Error('Command help must be handled by the native CLI, not by an engine request.'); @@ -95,7 +101,7 @@ export class PhasedCommandEngine { } const parser: RushCommandLineParser = new RushCommandLineParser({ cwd: resolvedCwd, - engine: { rushConfiguration, terminalProvider } + engine: { rushConfiguration, terminalProvider, environment } }); await parser.executeWithoutErrorHandlingAsync([...argv]); const action: CommandLineAction | undefined = parser.selectedAction; diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index b4262da1e2..a6ea56664e 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -38,27 +38,42 @@ export interface IWorkspaceInputFingerprintOptions { * Environment variable names that are excluded from {@link IWorkspaceInputFingerprint.environmentHash}. * * @remarks - * These variables are maintained per shell, terminal, remote session or client invocation. Rush never reads them - * to configure the engine, construct the operation graph or compute operation hashes, so a difference must not - * discard a warm workspace: + * These variables are maintained per shell, terminal, remote session, service unit, agent session or client + * invocation. Rush never reads them to configure the engine, construct the operation graph or compute operation + * hashes, so a difference must not discard a warm workspace: * * - shell bookkeeping: `_`, `PWD`, `OLDPWD`, `SHLVL`, `PS1`, `HISTFILE`, `HISTSIZE` * (a child shell recomputes `PWD`/`SHLVL`/`_` for its own working directory) * - terminal presentation: `TERM`, `TERM_PROGRAM`, `TERM_PROGRAM_VERSION`, `TERM_SESSION_ID`, `COLORTERM`, - * `COLUMNS`, `LINES`, `LS_COLORS`, `WINDOWID` + * `COLUMNS`, `LINES`, `LS_COLORS`, `WINDOWID`, and the per-window handles of terminal emulators: + * `WT_SESSION`, `WT_PROFILE_ID`, `ITERM_SESSION_ID` * - session and multiplexer handles: `WSL_INTEROP`, `WSLENV`, `SSH_CLIENT`, `SSH_CONNECTION`, `SSH_TTY`, * `SSH_AUTH_SOCK`, `TMUX`, `TMUX_PANE`, `STY`, `XDG_SESSION_ID`, `XDG_SESSION_TYPE`, `DBUS_SESSION_BUS_ADDRESS` - * - `INIT_CWD`, which Rush removes from every lifecycle script environment and sets explicitly where needed - * - client routing: `RUSH_DAEMON` and `RUSH_DAEMON_AUTO_START` only select and start a daemon, and - * `RUSH_DAEMON_EXPERIMENTAL` is read from each request rather than from the process + * - service manager metadata that systemd assigns to every unit and scope: `INVOCATION_ID`, `JOURNAL_STREAM`, + * `MANAGERPID`, `SYSTEMD_EXEC_PID`, `MEMORY_PRESSURE_WATCH`, `MEMORY_PRESSURE_WRITE` + * - editor and credential-prompt handles of an integrated terminal: `VSCODE_IPC_HOOK_CLI`, + * `VSCODE_GIT_IPC_HANDLE`, `VSCODE_GIT_ASKPASS_MAIN`, `VSCODE_GIT_ASKPASS_NODE`, + * `VSCODE_GIT_ASKPASS_EXTRA_ARGS`, `VSCODE_INJECTION`, `VSCODE_NONCE`, `GIT_ASKPASS`, `SSH_ASKPASS` + * - coding agent session markers: `COPILOT_CLI`, `COPILOT_AGENT_SESSION_ID`, `COPILOT_LOADER_PID`, + * `COPILOT_CLI_BINARY_VERSION`, `COPILOT_CLI_RESOLVED_DIST_DIR`, `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT` + * - `INIT_CWD`, which Rush removes from every lifecycle script environment and sets explicitly where needed, + * and `RUSH_INVOKED_FOLDER`, which Rush assigns for each invocation + * - client routing and presentation: `RUSH_DAEMON` and `RUSH_DAEMON_AUTO_START` only select and start a daemon, + * `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` is sent as each request's admission deadline, `RUSHD_OUTPUT` selects + * the client's output mode, and `RUSH_DAEMON_EXPERIMENTAL` is read from each request rather than from the process + * - `RUSH_PARALLELISM`, which a long-lived host applies to each request as its `--parallelism` default * * Every other variable remains a process-bound input, including the remaining `RUSH_*` settings (such as * `RUSH_BUILD_CACHE_*` and the daemon's own `RUSH_DAEMON_*` resource settings), `NODE_*`, npm/pnpm - * configuration, `PATH` and `HOME`. On Windows, names are matched case-insensitively. + * configuration, credentials, `PATH` and `HOME`. On Windows, names are matched case-insensitively. + * `PATH` is compared without repeated entries, because a later duplicate can never change which executable + * a lookup finds. * - * A long-lived host that ignores these variables keeps the values from its own startup environment for the - * processes it launches. Projects that need one of these values as an operation input should not rely on it - * being request-specific in such a host. + * A long-lived host that ignores these variables must not give the processes it launches the values of the + * client that started it. Each operation instead takes every one of these variables from the request that it + * serves ({@link getWorkspaceRequestOperationEnvironment}), and does not receive the variable when that + * request does not define it. The host's own process also drops {@link workspaceRequestScopedEnvironmentVariables}, + * because code running inside it reads them from `process.env`. * * @alpha */ @@ -79,6 +94,9 @@ export const workspaceFingerprintIgnoredEnvironmentVariables: ReadonlySet = new Set([ + 'RUSH_PARALLELISM', + 'COPILOT_AGENT_SESSION_ID' ]); /** * Returns the defined environment entries that participate in workspace fingerprints, sorted by name. * * @remarks - * Omits undefined values and {@link workspaceFingerprintIgnoredEnvironmentVariables}. Hosts that compare - * environments outside {@link captureWorkspaceInputFingerprintAsync} must use this function so that every - * comparison applies the same normalization. + * Omits undefined values and {@link workspaceFingerprintIgnoredEnvironmentVariables}, and removes repeated + * `PATH` entries. Hosts that compare environments outside {@link captureWorkspaceInputFingerprintAsync} must use + * this function so that every comparison applies the same normalization. * * @alpha */ @@ -111,13 +172,68 @@ export function getWorkspaceFingerprintEnvironmentEntries( environment: Readonly> ): [string, string][] { const isWindows: boolean = process.platform === 'win32'; - return Object.entries(environment) - .filter( - (entry): entry is [string, string] => - entry[1] !== undefined && - !workspaceFingerprintIgnoredEnvironmentVariables.has(isWindows ? entry[0].toUpperCase() : entry[0]) - ) - .sort(([left], [right]) => Sort.compareByValue(left, right)); + const entries: [string, string][] = []; + for (const [name, value] of Object.entries(environment)) { + if (value === undefined) continue; + const normalizedName: string = isWindows ? name.toUpperCase() : name; + if (workspaceFingerprintIgnoredEnvironmentVariables.has(normalizedName)) continue; + entries.push([name, normalizedName === 'PATH' ? removeRepeatedPathEntries(value) : value]); + } + return entries.sort(([left], [right]) => Sort.compareByValue(left, right)); +} + +/** + * Returns a copy of a host startup environment without {@link workspaceRequestScopedEnvironmentVariables}. + * + * @alpha + */ +export function getWorkspaceHostEnvironment( + environment: Readonly> +): Record { + const isWindows: boolean = process.platform === 'win32'; + const hostEnvironment: Record = {}; + for (const [name, value] of Object.entries(environment)) { + if ( + value !== undefined && + !workspaceRequestScopedEnvironmentVariables.has(isWindows ? name.toUpperCase() : name) + ) { + hostEnvironment[name] = value; + } + } + return hostEnvironment; +} + +/** + * Returns the environment that an operation starts from when a long-lived host runs it for a request. + * + * @remarks + * Every variable in {@link workspaceFingerprintIgnoredEnvironmentVariables} takes the request's value, and is + * omitted when the request does not define it, so that the operation sees its own requester's session, terminal + * and credential-helper variables. Every other variable comes from the host, whose environment matches the + * request's for identity. A host returns the result from `IOperationGraphIterationOptions.getOperationEnvironment`. + * On Windows, names are matched case-insensitively. + * + * @alpha + */ +export function getWorkspaceRequestOperationEnvironment( + hostEnvironment: Readonly>, + requestEnvironment: Readonly> +): Record { + const isWindows: boolean = process.platform === 'win32'; + const isRequestValue = (name: string): boolean => + workspaceFingerprintIgnoredEnvironmentVariables.has(isWindows ? name.toUpperCase() : name); + const environment: Record = {}; + for (const [name, value] of Object.entries(hostEnvironment)) { + if (value !== undefined && !isRequestValue(name)) environment[name] = value; + } + for (const [name, value] of Object.entries(requestEnvironment)) { + if (value !== undefined && isRequestValue(name)) environment[name] = value; + } + return environment; +} + +function removeRepeatedPathEntries(value: string): string { + return Array.from(new Set(value.split(path.delimiter))).join(path.delimiter); } /** diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts index 6aebbe7d35..49b1c296e1 100644 --- a/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts @@ -107,4 +107,43 @@ describe(`${PhasedCommandEngine.name} parameter identity`, () => { parallelism: { scalar: 0.5 } }); }); + + it("takes the RUSH_PARALLELISM default from the request's environment, not the host's", async () => { + const parseInEnvironmentAsync = async ( + environment: Record, + ...argv: string[] + ): Promise => + await PhasedCommandEngine.parseAsync({ + argv, + cwd: folder, + environment, + rushConfiguration, + terminalProvider: new NoOpTerminalProvider() + }); + const hostValue: string | undefined = process.env.RUSH_PARALLELISM; + process.env.RUSH_PARALLELISM = '7'; + try { + expect((await parseInEnvironmentAsync({ RUSH_PARALLELISM: '3' }, 'build')).requestSettings).toEqual({ + quietMode: true, + parallelism: 3 + }); + expect( + (await parseInEnvironmentAsync({ RUSH_PARALLELISM: '3' }, 'build', '-p', '5')).requestSettings + .parallelism + ).toBe(5); + expect((await parseInEnvironmentAsync({}, 'build')).requestSettings.parallelism).toEqual( + parseParallelism(undefined) + ); + const baseline: string = (await parseInEnvironmentAsync({}, 'build')).parameterIdentity; + expect((await parseInEnvironmentAsync({ RUSH_PARALLELISM: '3' }, 'build')).parameterIdentity).toBe( + baseline + ); + } finally { + if (hostValue === undefined) { + delete process.env.RUSH_PARALLELISM; + } else { + process.env.RUSH_PARALLELISM = hostValue; + } + } + }); }); diff --git a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts index 8982ac0494..e6496a515e 100644 --- a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts +++ b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts @@ -9,6 +9,10 @@ import { captureWorkspaceInputFingerprintAsync, classifyWorkspaceInputChange, getWorkspaceFingerprintEnvironmentEntries, + getWorkspaceHostEnvironment, + getWorkspaceRequestOperationEnvironment, + workspaceFingerprintIgnoredEnvironmentVariables, + workspaceRequestScopedEnvironmentVariables, WorkspaceInputChangeTier, WorkspaceRuntimeFingerprintCache, type IWorkspaceInputFingerprint @@ -117,6 +121,21 @@ describe('workspace input fingerprints', () => { { SSH_CONNECTION: '10.0.0.1 1 10.0.0.2 22', SSH_AUTH_SOCK: '/tmp/agent', TMUX: '/tmp/tmux' }, { INIT_CWD: '/repo/packages/p03' }, { RUSH_DAEMON: '1', RUSH_DAEMON_AUTO_START: '0', RUSH_DAEMON_EXPERIMENTAL: '1' }, + { RUSHD_OUTPUT: 'legacy', RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: '600' }, + { RUSH_PARALLELISM: '48', RUSH_INVOKED_FOLDER: '/repo/packages/p03' }, + { INVOCATION_ID: 'a1b2', JOURNAL_STREAM: '8:123', MANAGERPID: '1', SYSTEMD_EXEC_PID: '42' }, + { + VSCODE_IPC_HOOK_CLI: '/run/vscode.sock', + VSCODE_GIT_IPC_HANDLE: '/run/git.sock', + GIT_ASKPASS: '/a' + }, + { + COPILOT_CLI: '1', + COPILOT_AGENT_SESSION_ID: 'session-2', + COPILOT_LOADER_PID: '77', + WT_SESSION: 'w' + }, + { PATH: `${base.PATH}${path.delimiter}${base.PATH}` }, { TERM: undefined, PWD: undefined } ]) { expect(await getHashAsync({ ...base, ...volatile })).toBe(baseHash); @@ -126,9 +145,12 @@ describe('workspace input fingerprints', () => { { RUSH_BUILD_CACHE_ENABLED: '1' }, { RUSH_BUILD_CACHE_WRITE_ALLOWED: '0' }, { RUSH_DAEMON_WATCH: '1' }, + { RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400' }, { NODE_OPTIONS: '--max-old-space-size=8192' }, { NPM_CONFIG_REGISTRY: 'https://example.invalid/' }, + { GIT_CONFIG_COUNT: '1', GIT_CONFIG_KEY_0: 'core.fsmonitor', GIT_CONFIG_VALUE_0: 'false' }, { PATH: '/usr/bin:/usr/local/bin' }, + { PATH: ['/opt/node/bin', '/usr/local/bin', '/usr/bin'].join(path.delimiter) }, { HOME: '/home/other' } ]) { expect(await getHashAsync({ ...base, ...relevant })).not.toBe(baseHash); @@ -137,11 +159,86 @@ describe('workspace input fingerprints', () => { ['HOME', '/home/user'], ['PATH', '/usr/local/bin:/usr/bin'] ]); + const repeatedPath: string = ['/a', '/b', '/a', '', '/c', '', '/b'].join(path.delimiter); + expect(getWorkspaceFingerprintEnvironmentEntries({ PATH: repeatedPath })).toEqual([ + ['PATH', ['/a', '/b', '', '/c'].join(path.delimiter)] + ]); } finally { fs.rmSync(folder, { recursive: true, force: true }); } }); + it('keeps request-scoped variables out of a host environment', () => { + const environment: Record = { + HOME: '/home/user', + RUSH_PARALLELISM: '48', + COPILOT_AGENT_SESSION_ID: 'session-1', + RUSHD_OUTPUT: 'agent', + RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', + UNSET: undefined + }; + expect(getWorkspaceHostEnvironment(environment)).toEqual({ + HOME: '/home/user', + RUSHD_OUTPUT: 'agent', + RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400' + }); + for (const name of workspaceRequestScopedEnvironmentVariables) { + expect(workspaceFingerprintIgnoredEnvironmentVariables.has(name)).toBe(true); + } + }); + + it("gives operations the request's values of variables that are not host identity", () => { + const hostEnvironment: Record = { + HOME: '/home/user', + PATH: '/usr/bin', + NODE_OPTIONS: '--max-old-space-size=8192', + COPILOT_AGENT_SESSION_ID: 'session-A', + COPILOT_CLI: '1', + GIT_ASKPASS: '/window-A/askpass.sh', + WT_SESSION: 'wt-A', + UNSET: undefined + }; + const requestEnvironment: Record = { + HOME: '/home/other', + PATH: '/other/bin', + COPILOT_AGENT_SESSION_ID: 'session-B', + RUSH_PARALLELISM: '2', + WT_SESSION: 'wt-B', + RUSH_INVOKED_FOLDER: '/repo/apps/b', + TERM: undefined + }; + expect(getWorkspaceRequestOperationEnvironment(hostEnvironment, requestEnvironment)).toEqual({ + HOME: '/home/user', + PATH: '/usr/bin', + NODE_OPTIONS: '--max-old-space-size=8192', + COPILOT_AGENT_SESSION_ID: 'session-B', + RUSH_PARALLELISM: '2', + WT_SESSION: 'wt-B', + RUSH_INVOKED_FOLDER: '/repo/apps/b' + }); + expect(getWorkspaceRequestOperationEnvironment(hostEnvironment, {})).toEqual({ + HOME: '/home/user', + PATH: '/usr/bin', + NODE_OPTIONS: '--max-old-space-size=8192' + }); + }); + + it('matches request variable names case-insensitively on Windows', () => { + const platform: PropertyDescriptor = Object.getOwnPropertyDescriptor(process, 'platform')!; + const mixedCaseSessionVariable: string = 'Copilot_Agent_Session_Id'; + Object.defineProperty(process, 'platform', { ...platform, value: 'win32' }); + try { + expect( + getWorkspaceRequestOperationEnvironment( + { Path: 'C:\\bin', [mixedCaseSessionVariable]: 'session-A' }, + { Path: 'D:\\bin', COPILOT_AGENT_SESSION_ID: 'session-B' } + ) + ).toEqual({ Path: 'C:\\bin', COPILOT_AGENT_SESSION_ID: 'session-B' }); + } finally { + Object.defineProperty(process, 'platform', platform); + } + }); + it('classifies content, configuration and process-bound identities', () => { const current: IWorkspaceInputFingerprint = { configurationHash: 'configuration', diff --git a/libraries/rush-lib/src/cli/RushCommandLineParser.ts b/libraries/rush-lib/src/cli/RushCommandLineParser.ts index 3127c27e7d..5c6cd6521e 100644 --- a/libraries/rush-lib/src/cli/RushCommandLineParser.ts +++ b/libraries/rush-lib/src/cli/RushCommandLineParser.ts @@ -98,6 +98,8 @@ export interface IRushCommandLineParserOptions { engine?: { rushConfiguration: RushConfiguration; terminalProvider: ITerminalProvider; + /** The environment of the request; see {@link RushCommandLineParser.engineEnvironment}. */ + environment?: Readonly>; }; } @@ -203,6 +205,15 @@ export class RushCommandLineParser extends CommandLineParser { return this.#rushOptions.cwd; } + /** + * For a parser that serves a long-lived engine host, the environment of the request being parsed. Actions read + * environment-backed parameter defaults from it instead of from `process.env`. + */ + public get engineEnvironment(): Readonly> | undefined { + const engine: IRushCommandLineParserOptions['engine'] = this.#rushOptions.engine; + return engine ? (engine.environment ?? process.env) : undefined; + } + public constructor(options?: Partial) { super({ toolFilename: 'rush', diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index 2ff886b205..fe92a4c5ac 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -176,6 +176,7 @@ export class PhasedScriptAction extends BaseScriptAction i readonly #alwaysInstall: boolean | undefined; readonly #includeAllProjectsInWatchGraph: boolean; readonly #terminal: ITerminal; + readonly #engineEnvironment: Readonly> | undefined; readonly #changedProjectsOnlyParameter: CommandLineFlagParameter | undefined; readonly #selectionParameters: SelectionParameterSet; @@ -225,13 +226,15 @@ export class PhasedScriptAction extends BaseScriptAction i this.hooks = new PhasedCommandHooks(); this.#terminal = new Terminal(this.rushSession.terminalProvider); + this.#engineEnvironment = options.parser.engineEnvironment; this.#parallelismParameter = this.#enableParallelism ? this.defineStringParameter({ parameterLongName: '--parallelism', parameterShortName: '-p', argumentName: 'COUNT', - environmentVariable: EnvironmentVariableNames.RUSH_PARALLELISM, + // An engine host reads this default from the request's environment instead; see #getParallelism(). + environmentVariable: this.#engineEnvironment ? undefined : EnvironmentVariableNames.RUSH_PARALLELISM, description: 'Specifies the maximum number of concurrent processes to launch during a build.' + ' The COUNT should be a positive integer, a percentage value (eg. "50%") or the word "max"' + @@ -408,10 +411,31 @@ export class PhasedScriptAction extends BaseScriptAction i public getEngineRequestSettings(): IPhasedCommandEngineRequestSettings { return { quietMode: !this.#verboseParameter.value, - parallelism: this.#enableParallelism ? parseParallelism(this.#parallelismParameter?.value) : 1 + parallelism: this.#getParallelism() }; } + /** + * The `--parallelism` value, else the `RUSH_PARALLELISM` default, or 1 if the command does not run in parallel. + * An engine parser takes the default from the request's environment, because the host process belongs to no + * request; on Windows its name is matched case-insensitively, like `process.env`. + */ + #getParallelism(): Parallelism { + if (!this.#enableParallelism) return 1; + const engineEnvironment: Readonly> | undefined = + this.#engineEnvironment; + let value: string | undefined = this.#parallelismParameter?.value; + if (value === undefined && engineEnvironment) { + const name: string = EnvironmentVariableNames.RUSH_PARALLELISM; + const key: string | undefined = + process.platform === 'win32' + ? Object.keys(engineEnvironment).find((candidate: string) => candidate.toUpperCase() === name) + : name; + value = key === undefined ? undefined : engineEnvironment[key]; + } + return parseParallelism(value); + } + public async selectEngineOperationsAsync( graph: IOperationGraph ): Promise> { @@ -512,9 +536,7 @@ export class PhasedScriptAction extends BaseScriptAction i // if this is parallelizable, then use the value from the flag (undefined or a number), // if parallelism is not enabled, then restrict to 1 core const maxParallelism: number = getNumberOfCores(); - const parallelism: Parallelism = this.#enableParallelism - ? parseParallelism(this.#parallelismParameter?.value) - : 1; + const parallelism: Parallelism = this.#getParallelism(); await measureAsyncFn(`${PERF_PREFIX}:applyStandardPlugins`, async () => { // Generates the default operation graph diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index a2346d9e99..33caab8118 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -188,7 +188,10 @@ export { captureProjectConfigurationFingerprintAsync, classifyWorkspaceInputChange, getWorkspaceFingerprintEnvironmentEntries, + getWorkspaceHostEnvironment, + getWorkspaceRequestOperationEnvironment, workspaceFingerprintIgnoredEnvironmentVariables, + workspaceRequestScopedEnvironmentVariables, WorkspaceInputChangeTier, WorkspaceRuntimeFingerprintCache, type IWorkspaceInputFingerprint, diff --git a/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts b/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts index 4fa4d1b0ac..f48fd9b356 100644 --- a/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts +++ b/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts @@ -167,9 +167,15 @@ export interface IInputsSnapshot { * the command being executed and the final hashes of the operation's dependencies to compute the final hash for the operation. * @param project - The Rush project to compute the state hash for * @param operationName - The name of the operation (phase) to get hashes for. If omitted, returns a generic hash for the whole project, as used for bulk commands. + * @param environment - The environment that the operation runs with, if it differs from the environment of the + * snapshot. The operation's `dependsOnEnvVars` are hashed from it. * @returns The local state hash for the project. This is a hash of the environment, the project's tracked files, and any additional files. */ - getOperationOwnStateHash(project: IRushConfigurationProjectForSnapshot, operationName?: string): string; + getOperationOwnStateHash( + project: IRushConfigurationProjectForSnapshot, + operationName?: string, + environment?: Readonly> + ): string; } /** @@ -377,59 +383,80 @@ export class InputsSnapshot implements IInputsSnapshot { */ public getOperationOwnStateHash( project: IRushConfigurationProjectForSnapshot, - operationName?: string + operationName?: string, + environment?: Readonly> ): string { const record: IInternalInputsSnapshotProjectMetadata | undefined = this.#projectMetadataMap.get(project); if (!record) { throw new Error(`No information available for project at ${project.projectFolder}`); } + const operationSettings: Readonly | undefined = operationName + ? record.projectConfig?.operationSettingsByOperationName.get(operationName) + : undefined; + const snapshotEnvironment: Readonly> = this.#environment; + if ( + environment && + operationSettings?.dependsOnEnvVars?.some( + (envVar: string) => (environment[envVar] || '') !== (snapshotEnvironment[envVar] || '') + ) + ) { + // The operation's environment differs from the snapshot's in a variable that it hashes, so don't memoize. + return this.#computeOperationOwnStateHash(project, operationName, operationSettings, environment); + } + const { hashByOperationName } = record; let hash: string | undefined = hashByOperationName.get(operationName); if (!hash) { - const hashes: ReadonlyMap = this.getTrackedFileHashesForOperation( + hash = this.#computeOperationOwnStateHash( project, - operationName + operationName, + operationSettings, + snapshotEnvironment ); + hashByOperationName.set(operationName, hash); + } - const hasher: Hash = createHash('sha1'); - // If this is for a specific operation, apply operation-specific options - if (operationName) { - const operationSettings: Readonly | undefined = - record.projectConfig?.operationSettingsByOperationName.get(operationName); - if (operationSettings) { - const { dependsOnEnvVars, dependsOnNodeVersion, outputFolderNames } = operationSettings; - if (dependsOnEnvVars) { - // As long as we enumerate environment variables in a consistent order, we will get a stable hash. - // Changing the order in rush-project.json will change the hash anyway since the file contents are part of the hash. - for (const envVar of dependsOnEnvVars) { - hasher.update(`${hashDelimiter}$${envVar}=${this.#environment[envVar] || ''}`); - } - } - - if (dependsOnNodeVersion) { - const granularity: NodeVersionGranularity = - dependsOnNodeVersion === true ? 'patch' : dependsOnNodeVersion; - hasher.update(`${hashDelimiter}nodeVersion=${this.#nodeVersionByGranularity[granularity]}`); - } + return hash; + } - if (outputFolderNames) { - hasher.update(`${hashDelimiter}${JSON.stringify(outputFolderNames)}`); - } + #computeOperationOwnStateHash( + project: IRushConfigurationProjectForSnapshot, + operationName: string | undefined, + operationSettings: Readonly | undefined, + environment: Readonly> + ): string { + const hashes: ReadonlyMap = this.getTrackedFileHashesForOperation(project, operationName); + + const hasher: Hash = createHash('sha1'); + // If this is for a specific operation, apply operation-specific options + if (operationSettings) { + const { dependsOnEnvVars, dependsOnNodeVersion, outputFolderNames } = operationSettings; + if (dependsOnEnvVars) { + // As long as we enumerate environment variables in a consistent order, we will get a stable hash. + // Changing the order in rush-project.json will change the hash anyway since the file contents are part of the hash. + for (const envVar of dependsOnEnvVars) { + hasher.update(`${hashDelimiter}$${envVar}=${environment[envVar] || ''}`); } } - // Hash the base project files - for (const [filePath, fileHash] of hashes) { - hasher.update(`${hashDelimiter}${filePath}${hashDelimiter}${fileHash}`); + if (dependsOnNodeVersion) { + const granularity: NodeVersionGranularity = + dependsOnNodeVersion === true ? 'patch' : dependsOnNodeVersion; + hasher.update(`${hashDelimiter}nodeVersion=${this.#nodeVersionByGranularity[granularity]}`); } - hash = hasher.digest('hex'); + if (outputFolderNames) { + hasher.update(`${hashDelimiter}${JSON.stringify(outputFolderNames)}`); + } + } - hashByOperationName.set(operationName, hash); + // Hash the base project files + for (const [filePath, fileHash] of hashes) { + hasher.update(`${hashDelimiter}${filePath}${hashDelimiter}${fileHash}`); } - return hash; + return hasher.digest('hex'); } *#resolveHashes(filePaths: Iterable): Generator<[string, string]> { diff --git a/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts b/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts index 0ace6c1472..901fad97cc 100644 --- a/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts +++ b/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts @@ -713,5 +713,41 @@ describe(InputsSnapshot.name, () => { expect(result2).not.toEqual(baseline); expect(result2).not.toEqual(result1); }); + + it("Hashes dependsOnEnvVars from an operation's own environment when one is supplied", () => { + const { project, options } = getTestConfig(); + const projectConfig: Pick = { + operationSettingsByOperationName: new Map([ + ['_phase:build', { operationName: '_phase:build', dependsOnEnvVars: ['ENV_VAR'] }] + ]) + }; + const createSnapshot = (environment: Record): InputsSnapshot => + new InputsSnapshot({ + ...options, + projectMap: new Map([[project, { projectConfig: projectConfig as RushProjectConfiguration }]]), + environment + }); + const snapshotA: InputsSnapshot = createSnapshot({ ENV_VAR: 'a', OTHER: 'a' }); + const snapshotB: InputsSnapshot = createSnapshot({ ENV_VAR: 'b' }); + const hashA: string = createSnapshot({ ENV_VAR: 'a' }).getOperationOwnStateHash( + project, + '_phase:build' + ); + const hashB: string = snapshotB.getOperationOwnStateHash(project, '_phase:build'); + expect(hashB).not.toEqual(hashA); + + // An operation that runs with another environment gets the hash of a snapshot of that environment, + // whether or not the snapshot's own hash was already computed. + expect(snapshotA.getOperationOwnStateHash(project, '_phase:build', { ENV_VAR: 'b' })).toEqual(hashB); + expect(snapshotA.getOperationOwnStateHash(project, '_phase:build')).toEqual(hashA); + expect(snapshotA.getOperationOwnStateHash(project, '_phase:build', { ENV_VAR: 'b' })).toEqual(hashB); + // Variables that the operation does not depend on never change its hash. + expect( + snapshotA.getOperationOwnStateHash(project, '_phase:build', { ENV_VAR: 'a', OTHER: 'b' }) + ).toEqual(hashA); + expect(snapshotA.getOperationOwnStateHash(project, undefined, { ENV_VAR: 'b' })).toEqual( + snapshotB.getOperationOwnStateHash(project, undefined) + ); + }); }); }); diff --git a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts index e6b4316fbe..9b72d55383 100644 --- a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts @@ -21,6 +21,18 @@ export interface IOperationGraphIterationOptions { * The time when the iteration was scheduled, if available, as returned by `performance.now()`. */ startTime?: number; + + /** + * Returns the environment that an operation of this iteration starts from, before any + * `createEnvironmentForOperation` tap. The operation's `dependsOnEnvVars` are hashed from the same environment. + * When omitted, every operation starts from `process.env` and hashes the environment of the inputs snapshot. + * + * @remarks + * A long-lived host serves requests from clients whose environments differ from its own, and one iteration can + * serve several requests. The host gives each operation the environment of a request that selected it + * (see `getWorkspaceRequestOperationEnvironment`). + */ + getOperationEnvironment?: (operation: Operation) => Readonly>; } /** diff --git a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts index 129d5c04c5..f786b97113 100644 --- a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts +++ b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts @@ -48,6 +48,11 @@ export interface IOperationExecutionRecordContext { streamCollator: StreamCollator | undefined; onOperationStateChanged?: (record: OperationExecutionRecord) => void; createEnvironment?: (record: OperationExecutionRecord) => IEnvironment; + /** + * The environment that an operation starts from, before any `createEnvironmentForOperation` tap, when it is not + * the environment of the inputs snapshot. The operation's `dependsOnEnvVars` are hashed from it. + */ + getOperationEnvironment?: (operation: Operation) => Readonly>; invalidate?: (operations: Iterable, reason: string) => void; inputsSnapshot: IInputsSnapshot | undefined; maxParallelism: number; @@ -429,7 +434,11 @@ export class OperationExecutionRecord implements IOperationRunnerContext, IOpera // - Git hashes of tracked files in the associated project // - Git hash of the shrinkwrap file for the project // - Git hashes of any files specified in `dependsOnAdditionalFiles` (must not be associated with a project) - const local: string = inputsSnapshot.getOperationOwnStateHash(associatedProject, associatedPhase.name); + const local: string = inputsSnapshot.getOperationOwnStateHash( + associatedProject, + associatedPhase.name, + this.#context.getOperationEnvironment?.(this.operation) + ); // Examples of data in the config hash: // - CLI parameters (ShellOperationRunner) diff --git a/libraries/rush-lib/src/logic/operations/OperationGraph.ts b/libraries/rush-lib/src/logic/operations/OperationGraph.ts index a3ed25bb1c..09da8be3a0 100644 --- a/libraries/rush-lib/src/logic/operations/OperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/OperationGraph.ts @@ -691,9 +691,16 @@ export class OperationGraph implements IOperationGraph { const getInputsSnapshotAsync: (() => Promise) | undefined = this.#getInputsSnapshotAsync; - const { startTime = performance.now(), inputsSnapshot = await getInputsSnapshotAsync?.() } = - iterationOptions; - const iterationOptionsForCallbacks: IOperationGraphIterationOptions = { startTime, inputsSnapshot }; + const { + startTime = performance.now(), + inputsSnapshot = await getInputsSnapshotAsync?.(), + getOperationEnvironment + } = iterationOptions; + const iterationOptionsForCallbacks: IOperationGraphIterationOptions = { + startTime, + inputsSnapshot, + getOperationEnvironment + }; const { hooks } = this; @@ -719,7 +726,9 @@ export class OperationGraph implements IOperationGraph { const graph: OperationGraph = this; function createEnvironmentForOperation(record: OperationExecutionRecord): IEnvironment { - return hooks.createEnvironmentForOperation.call({ ...process.env }, record); + const baseEnvironment: Readonly> = + getOperationEnvironment?.(record.operation) ?? process.env; + return hooks.createEnvironmentForOperation.call({ ...baseEnvironment }, record); } const terminateController: AbortController | undefined = this.#supportsTerminateRunning @@ -739,6 +748,7 @@ export class OperationGraph implements IOperationGraph { maxParallelism: this.#maxParallelism, onOperationStateChanged: undefined, createEnvironment: createEnvironmentForOperation, + getOperationEnvironment, invalidate: graph.invalidateOperations.bind(graph), get debugMode(): boolean { return graph.debugMode; diff --git a/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts new file mode 100644 index 0000000000..653c027eb2 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts @@ -0,0 +1,170 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('../OperationStateFile'); +// Mock project log file creation to avoid filesystem writes. +jest.mock('../ProjectLogWritable', () => { + const actual = jest.requireActual('../ProjectLogWritable'); + const { TerminalWritable } = jest.requireActual('@rushstack/terminal'); + class MockTerminalWritable extends TerminalWritable { + protected onWriteChunk(): void { + /* noop */ + } + protected onClose(): void { + /* noop */ + } + } + return { + ...actual, + initializeProjectLogFilesAsync: jest.fn(async () => new MockTerminalWritable()) + }; +}); + +import { MockWritable } from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import type { IEnvironment } from '../../../utilities/Utilities'; +import type { IInputsSnapshot, IRushConfigurationProjectForSnapshot } from '../../incremental/InputsSnapshot'; +import type { IOperationGraphIterationOptions } from '../IOperationGraph'; +import type { IOperationRunnerContext } from '../IOperationRunner'; +import { Operation } from '../Operation'; +import { OperationGraph } from '../OperationGraph'; +import { OperationStatus } from '../OperationStatus'; +import { MockOperationRunner } from './MockOperationRunner'; + +const SESSION_VARIABLE: string = 'RUSH_TEST_OPERATION_ENVIRONMENT_SESSION'; + +const mockPhase: IPhase = { + name: 'phase', + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + dependencies: { + self: new Set(), + upstream: new Set() + }, + isSynthetic: false, + logFilenameIdentifier: 'phase', + missingScriptBehavior: 'silent' +}; + +class EnvironmentRecordingRunner extends MockOperationRunner { + public readonly sessions: (string | undefined)[] = []; + + public override async executeAsync(context: IOperationRunnerContext): Promise { + this.sessions.push(context.environment?.[SESSION_VARIABLE]); + return await super.executeAsync(context); + } +} + +function createGraph(runners: ReadonlyArray): OperationGraph { + const operations: Operation[] = runners.map( + (runner: EnvironmentRecordingRunner) => + new Operation({ + runner, + logFilenameIdentifier: runner.name, + phase: mockPhase, + project: { packageName: runner.name } as unknown as RushConfigurationProject + }) + ); + return new OperationGraph(new Set(operations), { + quietMode: false, + debugMode: false, + parallelism: 1, + allowOversubscription: true, + destinations: [new MockWritable()], + abortController: new AbortController() + }); +} + +describe('OperationGraph operation environment', () => { + afterEach(() => { + delete process.env[SESSION_VARIABLE]; + }); + + it('starts operations from process.env by default', async () => { + process.env[SESSION_VARIABLE] = 'host'; + const runner: EnvironmentRecordingRunner = new EnvironmentRecordingRunner('operation'); + const graph: OperationGraph = createGraph([runner]); + + expect((await graph.executeAsync({})).status).toBe(OperationStatus.Success); + expect(runner.sessions).toEqual(['host']); + }); + + it('starts each operation from the environment that the iteration chooses for it, before plugin taps', async () => { + process.env[SESSION_VARIABLE] = 'host'; + const first: EnvironmentRecordingRunner = new EnvironmentRecordingRunner('first'); + const second: EnvironmentRecordingRunner = new EnvironmentRecordingRunner('second'); + const graph: OperationGraph = createGraph([first, second]); + const tapSessions: (string | undefined)[] = []; + graph.hooks.createEnvironmentForOperation.tap('test', (environment: IEnvironment) => { + tapSessions.push(environment[SESSION_VARIABLE]); + // A tap edits its own copy, never the environment that the iteration supplied. + environment[SESSION_VARIABLE] = `${environment[SESSION_VARIABLE]}+tap`; + return environment; + }); + const sessionA: Readonly> = { [SESSION_VARIABLE]: 'A' }; + const sessionB: Readonly> = { [SESSION_VARIABLE]: 'B' }; + + const iterations: (IOperationGraphIterationOptions['getOperationEnvironment'] | undefined)[] = [ + (operation: Operation) => (operation.runner === first ? sessionA : sessionB), + () => sessionB, + undefined + ]; + for (const getOperationEnvironment of iterations) { + expect((await graph.executeAsync({ getOperationEnvironment })).status).toBe(OperationStatus.Success); + } + + expect(first.sessions).toEqual(['A+tap', 'B+tap', 'host+tap']); + expect(second.sessions).toEqual(['B+tap', 'B+tap', 'host+tap']); + expect(tapSessions.sort()).toEqual(['A', 'B', 'B', 'B', 'host', 'host']); + expect([sessionA, sessionB, process.env[SESSION_VARIABLE]]).toEqual([ + { [SESSION_VARIABLE]: 'A' }, + { [SESSION_VARIABLE]: 'B' }, + 'host' + ]); + }); + + it("hashes each operation's inputs with the environment that it starts from", async () => { + const first: EnvironmentRecordingRunner = new EnvironmentRecordingRunner('first'); + const second: EnvironmentRecordingRunner = new EnvironmentRecordingRunner('second'); + const graph: OperationGraph = createGraph([first, second]); + const hashedEnvironments: Map> | undefined> = + new Map(); + const inputsSnapshot: IInputsSnapshot = { + hashes: new Map(), + rootDirectory: '/repo', + hasUncommittedChanges: false, + getTrackedFileHashesForOperation: () => new Map(), + getOperationOwnStateHash: ( + project: IRushConfigurationProjectForSnapshot, + operationName?: string, + environment?: Readonly> + ) => { + hashedEnvironments.set((project as RushConfigurationProject).packageName, environment); + return 'local'; + } + }; + const sessionA: Readonly> = { [SESSION_VARIABLE]: 'A' }; + + await graph.executeAsync({ + inputsSnapshot, + getOperationEnvironment: (operation: Operation) => (operation.runner === first ? sessionA : {}) + }); + expect(hashedEnvironments).toEqual( + new Map([ + ['first', sessionA], + ['second', {}] + ]) + ); + + hashedEnvironments.clear(); + await graph.executeAsync({ inputsSnapshot }); + expect(hashedEnvironments).toEqual( + new Map([ + ['first', undefined], + ['second', undefined] + ]) + ); + }); +}); diff --git a/libraries/rush-lib/src/utilities/Utilities.ts b/libraries/rush-lib/src/utilities/Utilities.ts index 8c7a4b6078..df68af0e61 100644 --- a/libraries/rush-lib/src/utilities/Utilities.ts +++ b/libraries/rush-lib/src/utilities/Utilities.ts @@ -820,6 +820,13 @@ function _createEnvironmentForRushCommand(options: ICreateEnvironmentForRushComm continue; } + // RUSH_DAEMON* variables only route a client to the daemon and configure the daemon host. Project + // tooling never reads them, and a Rush release that predates the daemon rejects them as unknown + // RUSH_* variables, which breaks tools that start their own copy of Rush. + if (normalizedKey.startsWith('RUSH_DAEMON')) { + continue; + } + // Use the uppercased environment variable name on Windows because environment variable names // are case-insensitive on Windows environment[normalizedKey] = options.initialEnvironment[key]; diff --git a/libraries/rush-lib/src/utilities/test/Utilities.test.ts b/libraries/rush-lib/src/utilities/test/Utilities.test.ts index 5ab24d3ab6..b69fad4d4d 100644 --- a/libraries/rush-lib/src/utilities/test/Utilities.test.ts +++ b/libraries/rush-lib/src/utilities/test/Utilities.test.ts @@ -349,6 +349,38 @@ describe(Utilities.name, () => { expect(options.shell).toBeUndefined(); } }); + + it('does not pass the daemon routing and host settings to lifecycle commands', () => { + const spawn = jest + .fn, SpawnOptions]>() + .mockReturnValue(new ChildProcess()); + Utilities.executeLifecycleCommandAsync('echo request', { + rushConfiguration: undefined, + workingDirectory: process.cwd(), + initCwd: process.cwd(), + handleOutput: false, + environmentPathOptions: {}, + initialEnvironment: { + RUSH_DAEMON: '1', + RUSH_DAEMON_AUTO_START: '0', + RUSH_DAEMON_EXPERIMENTAL: '1', + RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', + RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: '65536', + RUSH_BUILD_CACHE_ENABLED: '1', + RUSH_PREVIEW_VERSION: '5.0.0', + REQUEST_VALUE: 'request' + }, + stdio: 'pipe', + spawn + }); + const environment: NodeJS.ProcessEnv = spawn.mock.calls[0][2].env!; + expect(Object.keys(environment).filter((name: string) => /^RUSH_DAEMON/i.test(name))).toEqual([]); + expect(environment).toMatchObject({ + RUSH_BUILD_CACHE_ENABLED: '1', + RUSH_PREVIEW_VERSION: '5.0.0', + REQUEST_VALUE: 'request' + }); + }); }); }); }); From ba3ff35857c4bde32fa4ca5c5a7080a588ac09e9 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 14:12:02 +0000 Subject: [PATCH 019/265] [rush-lib] Let daemon engines serve builds with declared daemon-compatible plugins (and 1 more) Swarm integration step 4; original commit 1e786a4252 (merge of swarm/r01 at 162ff05eaf). Commits folded into this step (2): - 5fe505bf1a [rush-lib] Let daemon engines serve builds with declared daemon-compatible plugins - 162ff05eaf [rush-lib] Walk each plugin package once in the daemon runtime fingerprint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...n-compatible-plugins_2026-09-28-13-00.json | 10 + ...n-compatible-plugins_2026-09-28-13-00.json | 10 + common/reviews/api/rush-lib.api.md | 4 +- docs/rush/dogfooding-rush-daemon.md | 67 ++++- docs/rush/environment-variables.md | 1 + .../src/ProductionDaemonRequestResolver.ts | 13 +- libraries/rush-lib/assets/rush-init/rush.json | 5 +- .../rush-lib/src/api/DaemonConfiguration.ts | 43 ++- .../src/api/EnvironmentConfiguration.ts | 3 + .../rush-lib/src/api/PhasedCommandEngine.ts | 57 +++- .../src/api/WorkspaceInputFingerprint.ts | 18 +- .../src/api/test/DaemonConfiguration.test.ts | 65 ++++- .../src/api/test/PhasedCommandEngine.test.ts | 262 +++++++++++++++--- .../test/WorkspaceInputFingerprint.test.ts | 87 ++++++ .../src/api/test/WorkspaceRuntimeWalk.test.ts | 53 ++++ .../PluginLoader/AutoinstallerPluginLoader.ts | 12 + .../PluginLoader/PluginLoaderBase.ts | 2 + .../src/pluginFramework/PluginManager.ts | 45 +-- .../schemas/rush-plugin-manifest.schema.json | 4 + .../rush-lib/src/schemas/rush.schema.json | 9 + 20 files changed, 688 insertions(+), 82 deletions(-) create mode 100644 common/changes/@microsoft/rush/daemon-compatible-plugins_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-daemon/daemon-compatible-plugins_2026-09-28-13-00.json create mode 100644 libraries/rush-lib/src/api/test/WorkspaceRuntimeWalk.test.ts diff --git a/common/changes/@microsoft/rush/daemon-compatible-plugins_2026-09-28-13-00.json b/common/changes/@microsoft/rush/daemon-compatible-plugins_2026-09-28-13-00.json new file mode 100644 index 0000000000..bdc4a73d31 --- /dev/null +++ b/common/changes/@microsoft/rush/daemon-compatible-plugins_2026-09-28-13-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Let daemon engines serve commands that a configured Rush plugin participates in when the plugin is declared daemon-compatible, through the `daemonCompatible` manifest field, the rush.json `daemon.compatiblePlugins` setting, or the `RUSH_DAEMON_COMPATIBLE_PLUGINS` environment variable. Treat the installed code of every configured plugin as a daemon runtime input.", + "type": "minor" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-daemon/daemon-compatible-plugins_2026-09-28-13-00.json b/common/changes/@rushstack/rush-daemon/daemon-compatible-plugins_2026-09-28-13-00.json new file mode 100644 index 0000000000..4c2528cae8 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/daemon-compatible-plugins_2026-09-28-13-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Serve build and rebuild requests that a declared daemon-compatible plugin participates in, and log compatible plugin names that match no configured plugin.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 0215d58fda..4e3184e502 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -326,6 +326,7 @@ export const EnvironmentVariableNames: { readonly RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: "RUSH_DAEMON_WARM_MEMORY_BUDGET_MB"; readonly RUSH_DAEMON_WARM_SET_MAX_PROJECTS: "RUSH_DAEMON_WARM_SET_MAX_PROJECTS"; readonly RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY: "RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY"; + readonly RUSH_DAEMON_COMPATIBLE_PLUGINS: "RUSH_DAEMON_COMPATIBLE_PLUGINS"; readonly RUSH_DAEMON_EXPERIMENTAL: "RUSH_DAEMON_EXPERIMENTAL"; }; @@ -524,6 +525,7 @@ export interface ICustomTipsJson { export interface IDaemonConfigurationJson { readonly autoStart?: boolean; readonly autoWarmByTelemetry?: boolean; + readonly compatiblePlugins?: ReadonlyArray; readonly enabled?: boolean; readonly idleTimeoutSeconds?: number; readonly queueTimeoutSeconds?: number; @@ -1268,7 +1270,6 @@ export interface IWorkspaceInputFingerprint { readonly environmentHash: string; // (undocumented) readonly installationHash: string; - // (undocumented) readonly runtimeHash: string; // (undocumented) readonly selectedRushVersion: string; @@ -1533,6 +1534,7 @@ export class PhasedCommandEngine { static parseAsync(options: IParsePhasedCommandOptions): Promise; get requestSettings(): IPhasedCommandEngineRequestSettings; selectOperationsAsync(graph: IOperationGraph): Promise>; + readonly unmatchedCompatiblePluginNames: ReadonlyArray; } // @alpha diff --git a/docs/rush/dogfooding-rush-daemon.md b/docs/rush/dogfooding-rush-daemon.md index 5ab364cda1..7cd51776c7 100644 --- a/docs/rush/dogfooding-rush-daemon.md +++ b/docs/rush/dogfooding-rush-daemon.md @@ -189,7 +189,9 @@ Remove the snapshot with `rm -rf common/temp/rush-daemon-dogfood` (or `rush purg - **Plugins.** This repository's only configured plugin, `@rushstack/rush-published-versions-json-plugin`, is associated only with `record-published-versions` and is inert for builds. A plugin without `associatedCommands`, a plugin associated with `build` or `rebuild`, or a plugin command-line that defines - the command, one of its phases, or a parameter for either would make those builds fall back to native Rush. + the command, one of its phases, or a parameter for either would make those builds fall back to native Rush, + unless the plugin is declared daemon-compatible. See + [Rush plugins in daemon engines](#rush-plugins-in-daemon-engines). - **Windows.** Native Windows validation of the daemon code saw unresolved, intermittent failures in which Git `hash-object --stdin-paths` exited with `0xC0000142` (DLL initialization failed) while the daemon captured workspace snapshots under Jest. Direct fixtures did not reproduce it, and it did not occur while @@ -200,6 +202,69 @@ Remove the snapshot with `rm -rf common/temp/rush-daemon-dogfood` (or `rush purg tool; if you see that, [refresh the snapshot](#refresh-the-snapshot). - **CI** stays in-process unless `RUSH_DAEMON=1` is set; do not set it in CI workflows. +## Rush plugins in daemon engines + +A daemon engine applies each Rush plugin once and then serves many requests from one long-lived process. +A plugin written for a single native command can therefore misbehave, so the daemon serves a `build` or +`rebuild` that a configured plugin participates in only if the plugin is declared daemon-compatible. A plugin +participates if it has no `associatedCommands` (Rush initializes it for every command), is associated with +the command, or its command-line.json defines the command, one of its phases, or a parameter for either. +Any other plugin is inert for the build and needs no declaration. + +Declare a plugin in one of these ways: + +- **The plugin author** sets `"daemonCompatible": true` on the plugin's entry in `rush-plugin-manifest.json`. + Releases whose schemas predate this setting reject such a manifest, and then every command fails, not only + builds. Set it only in plugin versions that are installed with a Rush release that accepts it, as with the + `@rushstack` plugins, which are versioned with Rush; otherwise leave the declaration to repositories. +- **The repository** lists the plugin's `pluginName` from `rush-plugins.json` in the `rush.json` setting + `"daemon": { "compatiblePlugins": [...] }`, after verifying the plugin against the contract below. +- **One shell** sets `RUSH_DAEMON_COMPATIBLE_PLUGINS` to a comma-separated list of plugin names, which + replaces the `rush.json` list; an empty value lists no plugins. Releases whose schemas predate these + settings reject the manifest and `rush.json` keys, so a repository that still selects such a release can + only use this variable, and only for `rush-client`. The value is part of the daemon's environment + identity, so changing it starts a new daemon. + +A listed name that matches no configured plugin has no effect; the request that creates the daemon's engine +prints a warning, which the daemon's launcher log also keeps. An undeclared participating plugin makes the +request fall back to native Rush, with a message that names the plugin and each reason. + +The engine lifecycle that a declared plugin must support: + +- **Once per engine:** the plugin's `apply()`, `runAnyPhasedCommand`, `runPhasedCommand.for()`, + `createOperationsAsync` and `onGraphCreatedAsync`. The engine builds the graph for every project, with + `isWatch` false; each request then selects operations from it. A request whose parameters differ from the + engine's (other than project selection, `--verbose`, `--parallelism` and `--timeline`) or a changed + configuration file replaces the engine, and the new engine applies the plugin again in the same process. + Module-level state therefore outlives an engine. +- **Once per iteration:** the operation graph hooks, such as `configureIteration`, + `beforeExecuteIterationAsync`, `before`/`afterExecuteOperationAsync`, `createEnvironmentForOperation` and + `afterExecuteIterationAsync`. One iteration can serve several concurrent requests. If every operation that + a request selects is up to date, the daemon aborts the iteration before it runs anything, so + `beforeExecuteIterationAsync` and the operation hooks don't fire. Don't rely on iteration hooks for + per-request work. +- **Disposal:** the engine aborts `IOperationGraph.abortController`, cancels the current iteration, and then + closes every operation runner. Release resources from the abort signal or the runner's `closeAsync()`. + +Rules for a daemon-compatible plugin: + +- Don't change process globals: no writes to `process.env`, `process.exitCode`, the current directory or + `process.argv`, and no patching or clearing of global functions such as `setTimeout`. They belong to the + daemon and to every other request. +- Keep request-specific state per iteration, not per process or per engine (session IDs, start times, + performance marks). +- Never prompt. The daemon has no terminal; fail fast with a message that the requesting client sees, and + continue without the feature where possible. +- Read the operation's environment (from `createEnvironmentForOperation`), not the daemon's `process.env`. +- Write output through the session's logger. Output written while an iteration runs reaches that iteration's + clients, and output written while the engine is created reaches the request that created it; the daemon + drops output written between iterations. + +Node.js loads a plugin's code only once, so the daemon treats the installed package folder of every configured +plugin as part of its implementation: a change to a plugin's `.js` or `.json` files, including through a +`link:` dependency, starts a new daemon for the next request. Declaration (`lib-dts`) and ES module (`lib-esm`) +output folders are left out, because a plugin's CommonJS entry point doesn't load them. + ## Version skew and `RUSH_PREVIEW_VERSION` `rush-client` runs the daemon with its bundled engine only if that engine's version equals the selected Rush diff --git a/docs/rush/environment-variables.md b/docs/rush/environment-variables.md index 5ca22ad77b..721e370fc3 100644 --- a/docs/rush/environment-variables.md +++ b/docs/rush/environment-variables.md @@ -31,6 +31,7 @@ variables and invalid values are errors, not ignored settings. | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | `512` | Best-effort sampled RSS budget in MiB. Positive number, at most 9007199254740991. Not a hard process-tree memory ceiling; active/protected work and results of resource-free projects are exempt. Overrides `warmMemoryBudgetMB`. | | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | `20` | Best-effort retained-project limit. Positive safe integer, at most 9007199254740991; never trims the requested execution set. Overrides `warmSetMaxProjects`. | | `RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY` | `0` | Rank retention using measured time saved, frequency and memory, with conservative LRU fallback. Never speculatively executes scripts. Overrides `autoWarmByTelemetry`. | +| `RUSH_DAEMON_COMPATIBLE_PLUGINS` | empty | Comma-separated `pluginName` values of configured Rush plugins that the repository has verified for long-lived daemon engines, in addition to plugins whose manifest sets `daemonCompatible`. Entries are trimmed; an empty entry is an error, and an empty value lists no plugins. Part of the daemon's environment identity. Overrides `compatiblePlugins`. See [Rush plugins in daemon engines](./dogfooding-rush-daemon.md#rush-plugins-in-daemon-engines). | | `RUSH_DAEMON_EXPERIMENTAL` | `0` | Enable the experimental `rush-client daemon graph` command surface. Does not start a daemon or enable builds; no corresponding `rush.json` property. | The timeout maximum is the signed 32-bit millisecond limit used by Node.js `setTimeout`, diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 73249df527..141749594b 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -45,9 +45,9 @@ import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; * scheduling parameters (`--verbose`, `--parallelism`, `--timeline`) are applied per request instead. * Incompatible parameters, * environments, or graph inputs are rejected before scheduling; no request is retried automatically. - * The initial supported surface excludes external plugins that participate in the requested command, - * .env initialization, install/watch, event-hook scripts, and rushx/global commands. Use the unchanged - * native CLI for those surfaces. + * The initial supported surface excludes external plugins that participate in the requested command + * (unless their manifest or the repository declares them daemon-compatible), .env initialization, + * install/watch, event-hook scripts, and rushx/global commands. Use the unchanged native CLI for those surfaces. * @beta */ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { @@ -195,6 +195,13 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { session: IWorkspaceSession ): Promise { if (!session.initializeEngineAsync) throw new Error('This session cannot bind a native engine.'); + if (command.unmatchedCompatiblePluginNames.length > 0) { + // The binding request's output also carries this warning; the launcher log keeps it for daemon diagnostics. + process.stderr.write( + `Warning: the daemon's compatible plugin list names plugins that are not configured in ` + + `rush-plugins.json: ${command.unmatchedCompatiblePluginNames.join(', ')}\n` + ); + } await session.initializeEngineAsync(async (options) => { let engine: IPhasedCommandEngine; try { diff --git a/libraries/rush-lib/assets/rush-init/rush.json b/libraries/rush-lib/assets/rush-init/rush.json index 33fec40008..2e66256750 100644 --- a/libraries/rush-lib/assets/rush-init/rush.json +++ b/libraries/rush-lib/assets/rush-init/rush.json @@ -337,6 +337,8 @@ * Warm-set policies require the generation-owned WorkspaceWarmSet host attachment. * They retain requested work without starting scripts and impose no hard process-RSS ceiling. * The watch option retains host observation of warm project files between requests, without running builds. + * The daemon serves a command that a configured Rush plugin participates in only if the plugin's manifest + * declares "daemonCompatible", or if "compatiblePlugins" lists the plugin's name. Otherwise it runs in-process. */ /*[LINE "HYPOTHETICAL"]*/ "daemon": { /*[LINE "HYPOTHETICAL"]*/ "enabled": false, @@ -348,7 +350,8 @@ /*[LINE "HYPOTHETICAL"]*/ "warmIdleTimeoutSeconds": 300, /*[LINE "HYPOTHETICAL"]*/ "warmMemoryBudgetMB": 512, /*[LINE "HYPOTHETICAL"]*/ "warmSetMaxProjects": 20, - /*[LINE "HYPOTHETICAL"]*/ "autoWarmByTelemetry": false + /*[LINE "HYPOTHETICAL"]*/ "autoWarmByTelemetry": false, + /*[LINE "HYPOTHETICAL"]*/ "compatiblePlugins": [] /*[LINE "HYPOTHETICAL"]*/ }, /** diff --git a/libraries/rush-lib/src/api/DaemonConfiguration.ts b/libraries/rush-lib/src/api/DaemonConfiguration.ts index 0869ebacb3..09c17f999d 100644 --- a/libraries/rush-lib/src/api/DaemonConfiguration.ts +++ b/libraries/rush-lib/src/api/DaemonConfiguration.ts @@ -32,6 +32,11 @@ export interface IDaemonConfigurationJson { readonly warmSetMaxProjects?: number; /** Prefer measured time-saved * frequency / resident-memory retention over LRU. Never starts scripts. Defaults to false. */ readonly autoWarmByTelemetry?: boolean; + /** + * Names of configured Rush plugins (their `pluginName` in rush-plugins.json) that the repository has verified + * for long-lived daemon engines, in addition to plugins whose manifest sets `daemonCompatible`. Defaults to none. + */ + readonly compatiblePlugins?: ReadonlyArray; } const defaults: Required = { @@ -44,7 +49,8 @@ const defaults: Required = { warmIdleTimeoutSeconds: 300, warmMemoryBudgetMB: 512, warmSetMaxProjects: 20, - autoWarmByTelemetry: false + autoWarmByTelemetry: false, + compatiblePlugins: Object.freeze([]) }; /** The exact recognized environment names. Unknown RUSH_DAEMON* names are rejected. @beta */ @@ -59,7 +65,8 @@ export const daemonEnvironmentVariables: Readonly> +): ReadonlyArray { + const configured: unknown = json.compatiblePlugins; + if ( + configured !== undefined && + (!Array.isArray(configured) || + configured.some((name: unknown) => typeof name !== 'string' || !isPluginName(name))) + ) { + throw new Error('daemon.compatiblePlugins must be an array of plugin names.'); + } + const name: string = daemonEnvironmentVariables.compatiblePlugins; + const value: string | undefined = environment[name]; + if (value === undefined) { + return configured ? Object.freeze([...(configured as string[])]) : defaults.compatiblePlugins; + } + // An empty value overrides rush.json with no plugins. + const names: string[] = value.trim() === '' ? [] : value.split(',').map((entry: string) => entry.trim()); + if (!names.every(isPluginName)) { + throw new Error(`${name} must be a comma-separated list of plugin names.`); + } + return Object.freeze(names); +} + +function isPluginName(name: string): boolean { + return name !== '' && name === name.trim() && !name.includes(','); +} + function booleanOption( key: 'enabled' | 'autoStart' | 'watch' | 'autoWarmByTelemetry' | 'usePersistentIpcRunners', json: IDaemonConfigurationJson, diff --git a/libraries/rush-lib/src/api/EnvironmentConfiguration.ts b/libraries/rush-lib/src/api/EnvironmentConfiguration.ts index b8f3357b17..00357ce7d2 100644 --- a/libraries/rush-lib/src/api/EnvironmentConfiguration.ts +++ b/libraries/rush-lib/src/api/EnvironmentConfiguration.ts @@ -286,6 +286,8 @@ export const EnvironmentVariableNames = { RUSH_DAEMON_WARM_SET_MAX_PROJECTS: 'RUSH_DAEMON_WARM_SET_MAX_PROJECTS', /** Enables telemetry-weighted retention of requested work in an attached warm set. */ RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY: 'RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY', + /** Overrides the plugins that the repository has verified for long-lived daemon engines. */ + RUSH_DAEMON_COMPATIBLE_PLUGINS: 'RUSH_DAEMON_COMPATIBLE_PLUGINS', /** Gates the experimental graph client; requires host graph integration. */ RUSH_DAEMON_EXPERIMENTAL: 'RUSH_DAEMON_EXPERIMENTAL' } as const; @@ -697,6 +699,7 @@ export class EnvironmentConfiguration { case EnvironmentVariableNames.RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: case EnvironmentVariableNames.RUSH_DAEMON_WARM_SET_MAX_PROJECTS: case EnvironmentVariableNames.RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY: + case EnvironmentVariableNames.RUSH_DAEMON_COMPATIBLE_PLUGINS: case EnvironmentVariableNames.RUSH_DAEMON_EXPERIMENTAL: // Validated together by resolveDaemonConfiguration(). break; diff --git a/libraries/rush-lib/src/api/PhasedCommandEngine.ts b/libraries/rush-lib/src/api/PhasedCommandEngine.ts index 707faae6ec..56940a44ca 100644 --- a/libraries/rush-lib/src/api/PhasedCommandEngine.ts +++ b/libraries/rush-lib/src/api/PhasedCommandEngine.ts @@ -4,7 +4,7 @@ import * as path from 'node:path'; import { FileSystem, LockFile } from '@rushstack/node-core-library'; -import type { ITerminalProvider } from '@rushstack/terminal'; +import { Terminal, type ITerminalProvider } from '@rushstack/terminal'; import type { CommandLineAction } from '@rushstack/ts-command-line'; import { RushCommandLineParser } from '../cli/RushCommandLineParser'; @@ -67,10 +67,13 @@ export interface IPhasedCommandEngineRequestSettings { * * @remarks * The initial engine surface deliberately rejects watch/install, event-hook scripts, .env files, and - * external plugins that Rush would initialize for the command or whose command-line.json shapes it. - * Those require request-scoped initialization and asynchronous disposal contracts before they can - * safely run in a shared process. Plugins associated only with other commands are inert and permitted. - * Native graph/cache plugins are not replaced. + * external plugins that Rush would initialize for the command or whose command-line.json shapes it, unless + * the plugin's manifest (`daemonCompatible`) or the repository (`daemon.compatiblePlugins`) declares that the + * plugin supports the engine lifecycle. An engine applies each plugin once and serves many requests: session + * hooks, `createOperationsAsync` (with every project and `isWatch` false) and `onGraphCreatedAsync` run once + * per engine, operation graph hooks run for each iteration (which can serve several coalesced requests), and + * disposal aborts `IOperationGraph.abortController` and then closes every operation runner. Plugins associated + * only with other commands are inert and permitted. Native graph/cache plugins are not replaced. * @alpha */ export class PhasedCommandEngine { @@ -80,12 +83,22 @@ export class PhasedCommandEngine { public readonly parameterIdentity: string; public readonly commandName: string; + /** + * Names listed by `daemon.compatiblePlugins` (or `RUSH_DAEMON_COMPATIBLE_PLUGINS`) that match no plugin + * configured in rush-plugins.json. They have no effect and are usually misspellings, so hosts should report them. + */ + public readonly unmatchedCompatiblePluginNames: ReadonlyArray; - private constructor(parser: RushCommandLineParser, action: PhasedScriptAction) { + private constructor( + parser: RushCommandLineParser, + action: PhasedScriptAction, + unmatchedCompatiblePluginNames: ReadonlyArray + ) { this._parser = parser; this._action = action; this.commandName = action.actionName; this.parameterIdentity = action.getEngineParameterIdentity(); + this.unmatchedCompatiblePluginNames = unmatchedCompatiblePluginNames; } public static async parseAsync(options: IParsePhasedCommandOptions): Promise { @@ -108,20 +121,38 @@ export class PhasedCommandEngine { if (!(action instanceof PhasedScriptAction) || !['build', 'rebuild'].includes(action.actionName)) { throw new Error('The production daemon engine currently supports native build and rebuild only.'); } + const compatiblePluginNames: ReadonlySet = new Set(rushConfiguration.daemon.compatiblePlugins); + const configuredPluginNames: ReadonlySet = parser.pluginManager.configuredPluginNames; + const unmatchedCompatiblePluginNames: ReadonlyArray = Array.from(compatiblePluginNames).filter( + (pluginName) => !configuredPluginNames.has(pluginName) + ); + if (unmatchedCompatiblePluginNames.length > 0) { + new Terminal(terminalProvider).writeWarningLine( + `The daemon's compatible plugin list (rush.json "daemon.compatiblePlugins" or ` + + `RUSH_DAEMON_COMPATIBLE_PLUGINS) names plugins that are not configured in rush-plugins.json: ` + + `${unmatchedCompatiblePluginNames.map((pluginName) => `"${pluginName}"`).join(', ')}. ` + + `Check that each entry is the plugin's "pluginName".` + ); + } // Plugins which the command would never initialize, and whose command-line.json does not shape - // this command, cannot affect a shared engine. Every other external plugin still requires native Rush. - const participatingPlugins: ReadonlyArray = parser.pluginManager.getPluginsParticipatingInCommand( + // this command, cannot affect a shared engine. Every other external plugin must be declared compatible + // with the engine lifecycle; otherwise the command still requires native Rush. + const incompatiblePlugins: ReadonlyArray = parser.pluginManager.getPluginsIncompatibleWithEngine( action.actionName, - action.schedulablePhaseNames + action.schedulablePhaseNames, + compatiblePluginNames ); - if (participatingPlugins.length > 0) { + if (incompatiblePlugins.length > 0) { throw new Error( - `Daemon engine execution does not yet support Rush plugins that participate in "${action.actionName}": ` + - `${participatingPlugins.join('; ')}. Use --no-daemon.` + `Daemon engine execution does not support Rush plugins that participate in "${action.actionName}" ` + + `unless they are declared daemon-compatible: ${incompatiblePlugins.join('; ')}. A plugin declares this ` + + `with "daemonCompatible" in its rush-plugin-manifest.json; a repository can list plugins it has ` + + `verified in the rush.json "daemon.compatiblePlugins" setting or RUSH_DAEMON_COMPATIBLE_PLUGINS. ` + + `Use --no-daemon.` ); } action.validateEngineCommand(); - return new PhasedCommandEngine(parser, action); + return new PhasedCommandEngine(parser, action, unmatchedCompatiblePluginNames); } /** diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index a6ea56664e..5fa2acbe4c 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -20,6 +20,10 @@ export interface IWorkspaceInputFingerprint { readonly configurationHash: string; readonly environmentHash: string; readonly installationHash: string; + /** + * The Node.js executable and version, the running Rush package, the host's `runtimePaths`, and the installed + * package folder of every configured Rush plugin. A change requires a new process. + */ readonly runtimeHash: string; readonly selectedRushVersion: string; } @@ -264,7 +268,8 @@ export class WorkspaceRuntimeFingerprintCache { /** @internal */ public _hashPaths(paths: ReadonlyArray): string { const filenames: Set = new Set(); - for (const filename of paths) { + // Several configured plugins often come from one package, whose folder is walked only once. + for (const filename of new Set(paths)) { for (const file of listRuntimeFilesSync(filename)) filenames.add(file); } const entries: ReadonlyArray[] = []; @@ -399,6 +404,11 @@ export async function captureWorkspaceInputFingerprintAsync( path.join(packageFolder, 'dist'), ...(options.runtimePaths ?? []) ]; + // A host loads plugins with require(), and Node.js never reloads a module, so a plugin's implementation is + // bound to the process that loaded it: an engine recreated in the same process would reuse the old code. + for (const pluginConfiguration of rushConfiguration._rushPluginsConfiguration.configuration.plugins) { + runtimePaths.push(AutoinstallerPluginLoader.getPluginPackageFolder(rushConfiguration, pluginConfiguration)); + } const runtimeHash: string = (options.runtimeCache ?? new WorkspaceRuntimeFingerprintCache())._hashPaths( runtimePaths ); @@ -521,12 +531,16 @@ function isProcessBoundConfiguration(filename: string): boolean { ); } +// Declaration and ES module output, which a plugin's CommonJS entry point doesn't load. Listing them would only +// slow down the synchronous walk that every request runs. +const NON_RUNTIME_FOLDER_NAMES: ReadonlySet = new Set(['node_modules', 'test', 'lib-dts', 'lib-esm']); + function listRuntimeFilesSync(folderOrFile: string): string[] { try { if (!fsSync.statSync(folderOrFile).isDirectory()) return [folderOrFile]; const files: string[] = []; for (const entry of fsSync.readdirSync(folderOrFile, { withFileTypes: true })) { - if (entry.name === 'node_modules' || entry.name === 'test') continue; + if (NON_RUNTIME_FOLDER_NAMES.has(entry.name)) continue; const filename: string = path.join(folderOrFile, entry.name); if (entry.isDirectory()) files.push(...listRuntimeFilesSync(filename)); else if (/\.(?:js|cjs|mjs|json)$/.test(entry.name) && !entry.name.endsWith('.test.js')) diff --git a/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts b/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts index c2e46f39ad..fb16ad2fd5 100644 --- a/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts @@ -52,7 +52,12 @@ describe('daemon configuration', () => { { warmMemoryBudgetMB: 0 }, { warmSetMaxProjects: 0.5 }, { autoWarmByTelemetry: 1 }, - { usePersistentIpcRunners: 'true' } + { usePersistentIpcRunners: 'true' }, + { compatiblePlugins: 'rush-example-plugin' }, + { compatiblePlugins: [''] }, + { compatiblePlugins: [' rush-example-plugin'] }, + { compatiblePlugins: ['rush-a-plugin,rush-b-plugin'] }, + { compatiblePlugins: [1] } ])('publishes schema rejection for %j', (daemon) => { const schema = JsonSchema.fromLoadedObject(schemaJson); expect(() => @@ -63,6 +68,55 @@ describe('daemon configuration', () => { ).toThrow(); }); + it('resolves compatible plugin names from the environment, then configuration, then no plugins', () => { + expect(resolveDaemonConfiguration({}, {}).compatiblePlugins).toEqual([]); + const configured: string[] = ['rush-a-plugin', 'rush-b-plugin']; + expect(resolveDaemonConfiguration({ compatiblePlugins: configured }, {}).compatiblePlugins).toEqual( + configured + ); + expect( + resolveDaemonConfiguration( + { compatiblePlugins: configured }, + { RUSH_DAEMON_COMPATIBLE_PLUGINS: ' rush-c-plugin , rush-d-plugin' } + ).compatiblePlugins + ).toEqual(['rush-c-plugin', 'rush-d-plugin']); + // An empty value is an explicit override that declares no plugins. + for (const value of ['', ' ']) { + expect( + resolveDaemonConfiguration({ compatiblePlugins: configured }, { RUSH_DAEMON_COMPATIBLE_PLUGINS: value }) + .compatiblePlugins + ).toEqual([]); + } + const resolved: readonly string[] = resolveDaemonConfiguration( + { compatiblePlugins: configured }, + {} + ).compatiblePlugins; + expect(Object.isFrozen(resolved)).toBe(true); + expect(resolved).not.toBe(configured); + }); + + it.each([',', 'rush-a-plugin,', 'rush-a-plugin,,rush-b-plugin', ' , rush-a-plugin'])( + 'rejects compatible plugin override %j with an empty entry', + (value) => { + expect(() => resolveDaemonConfiguration({}, { RUSH_DAEMON_COMPATIBLE_PLUGINS: value })).toThrow( + 'RUSH_DAEMON_COMPATIBLE_PLUGINS must be a comma-separated list of plugin names.' + ); + } + ); + + it.each([ + 'rush-example-plugin', + [''], + [' rush-example-plugin'], + ['rush-a-plugin,rush-b-plugin'], + [1], + [null] + ])('rejects configured compatible plugins %j', (compatiblePlugins) => { + expect(() => + resolveDaemonConfiguration({ compatiblePlugins: compatiblePlugins as unknown as string[] }, {}) + ).toThrow('daemon.compatiblePlugins must be an array of plugin names.'); + }); + it('accepts all valid knobs in the published schema', () => { JsonSchema.fromLoadedObject(schemaJson).validateObject( { @@ -73,5 +127,14 @@ describe('daemon configuration', () => { }, 'rush.json' ); + JsonSchema.fromLoadedObject(schemaJson).validateObject( + { + rushVersion: '5.179.0', + pnpmVersion: '10.27.0', + projects: [], + daemon: resolveDaemonConfiguration({ compatiblePlugins: ['rush-example-plugin'] }, {}) + }, + 'rush.json' + ); }); }); diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts index 1faae1ece9..8b3f371b9b 100644 --- a/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts @@ -5,7 +5,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; -import { NoOpTerminalProvider } from '@rushstack/terminal'; +import { NoOpTerminalProvider, StringBufferTerminalProvider } from '@rushstack/terminal'; import { PhasedCommandEngine } from '../PhasedCommandEngine'; import { RushConfiguration } from '../RushConfiguration'; @@ -13,11 +13,23 @@ import { RushConfiguration } from '../RushConfiguration'; const PACKAGE_NAME: string = '@example/rush-example-plugin'; const PLUGIN_NAME: string = 'rush-example-plugin'; const PLUGIN_COMMAND: string = 'record-example'; +const OTHER_PLUGIN_NAME: string = 'rush-other-plugin'; +const COMPATIBLE_PLUGINS_VARIABLE: string = 'RUSH_DAEMON_COMPATIBLE_PLUGINS'; interface IPluginFixture { + /** Defaults to PLUGIN_NAME. The plugin's package is `@example/`. */ + readonly pluginName?: string; readonly associatedCommands?: string[]; readonly commandLineJson?: object; readonly writeManifest?: boolean; + readonly daemonCompatible?: boolean; +} + +interface IRepoFixture { + readonly plugins: ReadonlyArray; + /** The rush.json `daemon.compatiblePlugins` setting. */ + readonly compatiblePlugins?: string[]; + readonly commandLineJson?: object; } // Mirrors the command-scoped plugin that rushstack configures in common/config/rush/rush-plugins.json. @@ -34,6 +46,21 @@ const COMMAND_SCOPED_COMMAND_LINE_JSON: object = { ] }; +// A plugin that shapes the build phase, like odsp-web's fstrace plugin. +const PHASE_SHAPING_COMMAND_LINE_JSON: object = { + ...COMMAND_SCOPED_COMMAND_LINE_JSON, + phases: [{ name: '_phase:build' }], + parameters: [ + { + parameterKind: 'flag', + longName: '--example-flag', + description: 'An example flag.', + associatedCommands: [PLUGIN_COMMAND], + associatedPhases: ['_phase:build'] + } + ] +}; + const REPO_COMMAND_LINE_JSON: object = { commands: [ { @@ -48,45 +75,66 @@ const REPO_COMMAND_LINE_JSON: object = { phases: [{ name: '_phase:build', dependencies: { upstream: ['_phase:build'] } }] }; -function createRepo(plugin: IPluginFixture): string { +function getPackageName(plugin: IPluginFixture): string { + return `@example/${plugin.pluginName ?? PLUGIN_NAME}`; +} + +function createRepo(repo: IRepoFixture): string { const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-engine-plugins-')); const write = (relativePath: string, json: object): void => { const filename: string = path.join(folder, relativePath); fs.mkdirSync(path.dirname(filename), { recursive: true }); fs.writeFileSync(filename, JSON.stringify(json)); }; - write('rush.json', { rushVersion: '5.179.0', pnpmVersion: '10.27.0', projects: [] }); - write('common/config/rush/command-line.json', REPO_COMMAND_LINE_JSON); + write('rush.json', { + rushVersion: '5.179.0', + pnpmVersion: '10.27.0', + projects: [], + ...(repo.compatiblePlugins ? { daemon: { compatiblePlugins: repo.compatiblePlugins } } : {}) + }); + write('common/config/rush/command-line.json', repo.commandLineJson ?? REPO_COMMAND_LINE_JSON); write('common/config/rush/rush-plugins.json', { - plugins: [{ packageName: PACKAGE_NAME, pluginName: PLUGIN_NAME, autoinstallerName: 'plugins' }] + plugins: repo.plugins.map((plugin) => ({ + packageName: getPackageName(plugin), + pluginName: plugin.pluginName ?? PLUGIN_NAME, + autoinstallerName: 'plugins' + })) }); write('common/autoinstallers/plugins/package.json', { name: 'plugins', version: '1.0.0', private: true, - dependencies: { [PACKAGE_NAME]: '1.0.0' } + dependencies: Object.fromEntries(repo.plugins.map((plugin) => [getPackageName(plugin), '1.0.0'])) }); - const storeFolder: string = `common/autoinstallers/plugins/rush-plugins/${PACKAGE_NAME}`; - if (plugin.writeManifest !== false) { - write(`${storeFolder}/rush-plugin-manifest.json`, { - plugins: [ - { - pluginName: PLUGIN_NAME, - description: 'An example plugin.', - entryPoint: './lib/index.js', - associatedCommands: plugin.associatedCommands, - commandLineJsonFilePath: './command-line.json' - } - ] - }); - } - if (plugin.commandLineJson) { - write(`${storeFolder}/${PLUGIN_NAME}/command-line.json`, plugin.commandLineJson); + for (const plugin of repo.plugins) { + const pluginName: string = plugin.pluginName ?? PLUGIN_NAME; + const storeFolder: string = `common/autoinstallers/plugins/rush-plugins/${getPackageName(plugin)}`; + if (plugin.writeManifest !== false) { + write(`${storeFolder}/rush-plugin-manifest.json`, { + plugins: [ + { + pluginName, + description: 'An example plugin.', + entryPoint: './lib/index.js', + associatedCommands: plugin.associatedCommands, + commandLineJsonFilePath: './command-line.json', + daemonCompatible: plugin.daemonCompatible + } + ] + }); + } + if (plugin.commandLineJson) { + write(`${storeFolder}/${pluginName}/command-line.json`, plugin.commandLineJson); + } } return folder; } -async function parseBuildAsync(folder: string, argv: string[] = ['build']): Promise { +async function parseBuildAsync( + folder: string, + argv: string[] = ['build'], + terminalProvider: NoOpTerminalProvider | StringBufferTerminalProvider = new NoOpTerminalProvider() +): Promise { const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile( path.join(folder, 'rush.json') ); @@ -94,19 +142,28 @@ async function parseBuildAsync(folder: string, argv: string[] = ['build']): Prom argv, cwd: folder, rushConfiguration, - terminalProvider: new NoOpTerminalProvider() + terminalProvider }); } describe(PhasedCommandEngine.name, () => { const folders: string[] = []; - function createTestRepo(plugin: IPluginFixture): string { - const folder: string = createRepo(plugin); + function createTestRepo(plugin: IPluginFixture, repo?: Omit): string { + return createMultiPluginTestRepo({ ...repo, plugins: [plugin] }); + } + function createMultiPluginTestRepo(repo: IRepoFixture): string { + const folder: string = createRepo(repo); folders.push(folder); return folder; } + const originalCompatiblePlugins: string | undefined = process.env[COMPATIBLE_PLUGINS_VARIABLE]; afterEach(() => { + if (originalCompatiblePlugins === undefined) { + delete process.env[COMPATIBLE_PLUGINS_VARIABLE]; + } else { + process.env[COMPATIBLE_PLUGINS_VARIABLE] = originalCompatiblePlugins; + } for (const folder of folders.splice(0)) { fs.rmSync(folder, { recursive: true, force: true }); } @@ -132,8 +189,11 @@ describe(PhasedCommandEngine.name, () => { it('rejects an unassociated plugin, which Rush initializes for every command', async () => { const folder: string = createTestRepo({ commandLineJson: COMMAND_SCOPED_COMMAND_LINE_JSON }); await expect(parseBuildAsync(folder)).rejects.toThrow( - `Daemon engine execution does not yet support Rush plugins that participate in "build": ` + - `"${PLUGIN_NAME}" (${PACKAGE_NAME}) is initialized for every command. Use --no-daemon.` + `Daemon engine execution does not support Rush plugins that participate in "build" unless they are ` + + `declared daemon-compatible: "${PLUGIN_NAME}" (${PACKAGE_NAME}) is initialized for every command. ` + + `A plugin declares this with "daemonCompatible" in its rush-plugin-manifest.json; a repository can ` + + `list plugins it has verified in the rush.json "daemon.compatiblePlugins" setting or ` + + `RUSH_DAEMON_COMPATIBLE_PLUGINS. Use --no-daemon.` ); }); @@ -177,19 +237,7 @@ describe(PhasedCommandEngine.name, () => { it('rejects a plugin parameter associated with a phase of the requested command', async () => { const folder: string = createTestRepo({ associatedCommands: [PLUGIN_COMMAND], - commandLineJson: { - ...COMMAND_SCOPED_COMMAND_LINE_JSON, - phases: [{ name: '_phase:build' }], - parameters: [ - { - parameterKind: 'flag', - longName: '--example-flag', - description: 'An example flag.', - associatedCommands: [PLUGIN_COMMAND], - associatedPhases: ['_phase:build'] - } - ] - } + commandLineJson: PHASE_SHAPING_COMMAND_LINE_JSON }); await expect(parseBuildAsync(folder)).rejects.toThrow( `"${PLUGIN_NAME}" (${PACKAGE_NAME}) associates "--example-flag" with the "_phase:build" phase` @@ -202,4 +250,136 @@ describe(PhasedCommandEngine.name, () => { `"${PLUGIN_NAME}" (${PACKAGE_NAME}): its manifest could not be read` ); }); + + describe('daemon-compatible declarations', () => { + // Associated with build and rebuild, and shapes a build phase and parameter, like odsp-web's fstrace plugin. + const BUILD_PLUGIN: IPluginFixture = { + associatedCommands: [PLUGIN_COMMAND, 'build', 'rebuild'], + commandLineJson: PHASE_SHAPING_COMMAND_LINE_JSON + }; + + async function expectAcceptedAsync(folder: string): Promise { + for (const commandName of ['build', 'rebuild']) { + const command: PhasedCommandEngine = await parseBuildAsync(folder, [commandName]); + expect(command.commandName).toBe(commandName); + expect(command.unmatchedCompatiblePluginNames).toEqual([]); + } + } + + it('reports every reason for an undeclared plugin', async () => { + const folder: string = createTestRepo(BUILD_PLUGIN); + const label: string = `"${PLUGIN_NAME}" (${PACKAGE_NAME})`; + await expect(parseBuildAsync(folder)).rejects.toThrow( + `${label} is associated with "build"; ${label} defines the "_phase:build" phase; ` + + `${label} associates "--example-flag" with the "_phase:build" phase. A plugin declares this` + ); + }); + + it('accepts a plugin whose manifest declares it daemon-compatible', async () => { + await expectAcceptedAsync(createTestRepo({ ...BUILD_PLUGIN, daemonCompatible: true })); + }); + + it('accepts a plugin that rush.json declares daemon-compatible', async () => { + await expectAcceptedAsync(createTestRepo(BUILD_PLUGIN, { compatiblePlugins: [PLUGIN_NAME] })); + }); + + it('accepts a plugin that RUSH_DAEMON_COMPATIBLE_PLUGINS declares, overriding rush.json', async () => { + const folder: string = createTestRepo(BUILD_PLUGIN, { compatiblePlugins: [] }); + process.env[COMPATIBLE_PLUGINS_VARIABLE] = ` ${PLUGIN_NAME} `; + await expectAcceptedAsync(folder); + }); + + it('lets an empty RUSH_DAEMON_COMPATIBLE_PLUGINS withdraw the rush.json declarations', async () => { + const folder: string = createTestRepo(BUILD_PLUGIN, { compatiblePlugins: [PLUGIN_NAME] }); + process.env[COMPATIBLE_PLUGINS_VARIABLE] = ''; + await expect(parseBuildAsync(folder)).rejects.toThrow( + `"${PLUGIN_NAME}" (${PACKAGE_NAME}) is associated with "build"` + ); + }); + + it('accepts a declared plugin that Rush initializes for every command', async () => { + await expectAcceptedAsync( + createTestRepo({ commandLineJson: COMMAND_SCOPED_COMMAND_LINE_JSON, daemonCompatible: true }) + ); + await expectAcceptedAsync( + createTestRepo({ commandLineJson: COMMAND_SCOPED_COMMAND_LINE_JSON }, { compatiblePlugins: [PLUGIN_NAME] }) + ); + }); + + it('still rejects an undeclared plugin next to a declared one, and reports only the undeclared one', async () => { + const folder: string = createMultiPluginTestRepo({ + plugins: [BUILD_PLUGIN, { pluginName: OTHER_PLUGIN_NAME, associatedCommands: ['build'] }], + compatiblePlugins: [PLUGIN_NAME] + }); + const error: Error | undefined = await parseBuildAsync(folder).then( + () => undefined, + (reason: Error) => reason + ); + expect(error?.message).toContain( + `unless they are declared daemon-compatible: "${OTHER_PLUGIN_NAME}" (@example/${OTHER_PLUGIN_NAME}) ` + + `is associated with "build". A plugin declares this` + ); + expect(error?.message).not.toContain(PLUGIN_NAME); + }); + + it('fails closed for a declared plugin whose manifest or command-line.json cannot be read', async () => { + await expect( + parseBuildAsync(createTestRepo({ writeManifest: false }, { compatiblePlugins: [PLUGIN_NAME] })) + ).rejects.toThrow(`"${PLUGIN_NAME}" (${PACKAGE_NAME}): its manifest could not be read`); + // The native parser also reads the file, and may fail first; either way, parsing fails. + await expect( + parseBuildAsync( + createTestRepo({ ...BUILD_PLUGIN, daemonCompatible: true, commandLineJson: { commands: 'invalid' } }) + ) + ).rejects.toThrow(/command-line\.json/); + }); + + it('keeps the parameters that a declared plugin reads in the engine parameter identity', async () => { + // odsp-web's fstrace plugin reads --fstrace-mode, which the repository's command-line.json defines. + const folder: string = createTestRepo( + { ...BUILD_PLUGIN, daemonCompatible: true }, + { + commandLineJson: { + ...REPO_COMMAND_LINE_JSON, + parameters: [ + { + parameterKind: 'choice', + longName: '--example-mode', + description: 'An example mode.', + associatedCommands: ['build'], + alternatives: [ + { name: 'on', description: 'On.' }, + { name: 'off', description: 'Off.' } + ], + defaultValue: 'on' + } + ] + } + } + ); + const defaultIdentity: string = (await parseBuildAsync(folder)).parameterIdentity; + expect((await parseBuildAsync(folder)).parameterIdentity).toBe(defaultIdentity); + expect((await parseBuildAsync(folder, ['build', '--example-mode', 'off'])).parameterIdentity).not.toBe( + defaultIdentity + ); + }); + + it('warns about declared names that match no configured plugin', async () => { + const folder: string = createTestRepo( + { ...BUILD_PLUGIN, daemonCompatible: true }, + { compatiblePlugins: ['rush-exmaple-plugin', PLUGIN_NAME] } + ); + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const command: PhasedCommandEngine = await parseBuildAsync(folder, ['build'], terminalProvider); + expect(command.unmatchedCompatiblePluginNames).toEqual(['rush-exmaple-plugin']); + expect(terminalProvider.getWarningOutput()).toContain( + 'names plugins that are not configured in rush-plugins.json: "rush-exmaple-plugin".' + ); + // A misspelled name does not declare the plugin that it was meant to name. + process.env[COMPATIBLE_PLUGINS_VARIABLE] = 'rush-exmaple-plugin'; + await expect(parseBuildAsync(createTestRepo(BUILD_PLUGIN))).rejects.toThrow( + `"${PLUGIN_NAME}" (${PACKAGE_NAME}) is associated with "build"` + ); + }); + }); }); diff --git a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts index e6496a515e..90f646331c 100644 --- a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts +++ b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts @@ -92,6 +92,91 @@ describe('workspace input fingerprints', () => { } }); + it('restarts when the implementation of a configured plugin changes, including through a link', async () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-fingerprint-')); + try { + const write = (relativePath: string, content: string): void => { + const filename: string = path.join(folder, relativePath); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, content); + }; + write('rush.json', JSON.stringify({ rushVersion: '5.179.0', pnpmVersion: '10.27.0', projects: [] })); + write( + 'common/config/rush/rush-plugins.json', + JSON.stringify({ + plugins: [ + { packageName: '@example/installed', pluginName: 'installed', autoinstallerName: 'plugins' }, + { packageName: '@example/linked', pluginName: 'linked', autoinstallerName: 'plugins' } + ] + }) + ); + write('common/autoinstallers/plugins/package.json', '{"name":"plugins","version":"1.0.0"}'); + write('common/autoinstallers/plugins/node_modules/@example/installed/lib/index.js', 'exports.v = 1;'); + // Like a `link:` dependency, whose implementation is checked in outside node_modules. + write('common/autoinstallers/plugins/linked-plugin/release/index.js', 'exports.v = 1;'); + fs.symlinkSync( + path.join(folder, 'common/autoinstallers/plugins/linked-plugin'), + path.join(folder, 'common/autoinstallers/plugins/node_modules/@example/linked'), + 'junction' + ); + const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile( + path.join(folder, 'rush.json') + ); + const runtimeCache: WorkspaceRuntimeFingerprintCache = new WorkspaceRuntimeFingerprintCache(); + const captureAsync = (): Promise => + captureWorkspaceInputFingerprintAsync({ rushConfiguration, runtimeCache, environment: {} }); + + let previous: IWorkspaceInputFingerprint = await captureAsync(); + expect(classifyWorkspaceInputChange(previous, await captureAsync())).toBe(WorkspaceInputChangeTier.Reuse); + for (const [relativePath, content] of [ + ['common/autoinstallers/plugins/node_modules/@example/installed/lib/index.js', 'exports.v = 2;'], + ['common/autoinstallers/plugins/linked-plugin/release/index.js', 'exports.v = 2;'], + ['common/autoinstallers/plugins/linked-plugin/release/worker.js', 'exports.w = 1;'] + ]) { + write(relativePath, content); + const next: IWorkspaceInputFingerprint = await captureAsync(); + expect(classifyWorkspaceInputChange(previous, next)).toBe(WorkspaceInputChangeTier.Restart); + expect(runtimeCache.changedPaths).toContain( + path.join( + folder, + relativePath.replace('plugins/linked-plugin/', 'plugins/node_modules/@example/linked/') + ) + ); + previous = next; + } + // Files that Node.js never loads as plugin code do not restart the host. + write('common/autoinstallers/plugins/linked-plugin/README.md', 'Documentation'); + write('common/autoinstallers/plugins/linked-plugin/lib-esm/index.js', 'export const v = 2;'); + write('common/autoinstallers/plugins/linked-plugin/lib-dts/tsdoc-metadata.json', '{}'); + expect(classifyWorkspaceInputChange(previous, await captureAsync())).toBe(WorkspaceInputChangeTier.Reuse); + } finally { + fs.rmSync(folder, { recursive: true, force: true }); + } + }); + + it('reloads when rush.json declares daemon-compatible plugins', async () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-fingerprint-')); + try { + const rushJsonPath: string = path.join(folder, 'rush.json'); + const rushJson: object = { rushVersion: '5.179.0', pnpmVersion: '10.27.0', projects: [] }; + fs.writeFileSync(rushJsonPath, JSON.stringify(rushJson)); + const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile(rushJsonPath); + const runtimeCache: WorkspaceRuntimeFingerprintCache = new WorkspaceRuntimeFingerprintCache(); + const captureAsync = (): Promise => + captureWorkspaceInputFingerprintAsync({ rushConfiguration, runtimeCache, environment: {} }); + const previous: IWorkspaceInputFingerprint = await captureAsync(); + fs.writeFileSync( + rushJsonPath, + JSON.stringify({ ...rushJson, daemon: { compatiblePlugins: ['rush-example-plugin'] } }) + ); + expect(classifyWorkspaceInputChange(previous, await captureAsync())).toBe( + WorkspaceInputChangeTier.Reload + ); + } finally { + fs.rmSync(folder, { recursive: true, force: true }); + } + }); + it('ignores volatile per-shell variables but not engine, Node.js or tool resolution inputs', async () => { const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-fingerprint-')); try { @@ -142,6 +227,8 @@ describe('workspace input fingerprints', () => { } for (const relevant of [ { FOO: '1' }, + // Declaring a plugin daemon-compatible changes which plugins the engine applies. + { RUSH_DAEMON_COMPATIBLE_PLUGINS: 'rush-example-plugin' }, { RUSH_BUILD_CACHE_ENABLED: '1' }, { RUSH_BUILD_CACHE_WRITE_ALLOWED: '0' }, { RUSH_DAEMON_WATCH: '1' }, diff --git a/libraries/rush-lib/src/api/test/WorkspaceRuntimeWalk.test.ts b/libraries/rush-lib/src/api/test/WorkspaceRuntimeWalk.test.ts new file mode 100644 index 0000000000..192c572255 --- /dev/null +++ b/libraries/rush-lib/src/api/test/WorkspaceRuntimeWalk.test.ts @@ -0,0 +1,53 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('node:fs', () => { + const actual: typeof import('node:fs') = jest.requireActual('node:fs'); + return { ...actual, readdirSync: jest.fn(actual.readdirSync) }; +}); + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { captureWorkspaceInputFingerprintAsync } from '../WorkspaceInputFingerprint'; +import { RushConfiguration } from '../RushConfiguration'; + +describe('workspace runtime walk', () => { + it('walks a package folder that several configured plugins share only once per capture', async () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-fingerprint-')); + try { + const write = (relativePath: string, content: string): void => { + const filename: string = path.join(folder, relativePath); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, content); + }; + write('rush.json', JSON.stringify({ rushVersion: '5.179.0', pnpmVersion: '10.27.0', projects: [] })); + write( + 'common/config/rush/rush-plugins.json', + JSON.stringify({ + plugins: ['first', 'second', 'third'].map((pluginName: string) => ({ + packageName: '@example/plugins', + pluginName, + autoinstallerName: 'plugins' + })) + }) + ); + const packageFolder: string = 'common/autoinstallers/plugins/node_modules/@example/plugins'; + write(`${packageFolder}/release/first.js`, 'exports.v = 1;'); + const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile( + path.join(folder, 'rush.json') + ); + + const readdirSync: jest.Mock = fs.readdirSync as unknown as jest.Mock; + readdirSync.mockClear(); + await captureWorkspaceInputFingerprintAsync({ rushConfiguration, environment: {} }); + const walks: unknown[] = readdirSync.mock.calls.filter( + ([walked]: unknown[]) => walked === path.join(folder, packageFolder) + ); + expect(walks).toHaveLength(1); + } finally { + fs.rmSync(folder, { recursive: true, force: true }); + } + }); +}); diff --git a/libraries/rush-lib/src/pluginFramework/PluginLoader/AutoinstallerPluginLoader.ts b/libraries/rush-lib/src/pluginFramework/PluginLoader/AutoinstallerPluginLoader.ts index 8c464bcc6d..fb4ed3a438 100644 --- a/libraries/rush-lib/src/pluginFramework/PluginLoader/AutoinstallerPluginLoader.ts +++ b/libraries/rush-lib/src/pluginFramework/PluginLoader/AutoinstallerPluginLoader.ts @@ -80,6 +80,18 @@ export class AutoinstallerPluginLoader extends PluginLoaderBase/node_modules/`). It may be a link to a folder elsewhere. + */ + public static getPluginPackageFolder( + rushConfiguration: RushConfiguration, + pluginConfiguration: IRushPluginConfiguration + ): string { + const { autoinstallerName, packageName } = pluginConfiguration; + return path.join(rushConfiguration.commonAutoinstallersFolder, autoinstallerName, 'node_modules', packageName); + } + public update(): void { const packageName: string = this.packageName; const pluginName: string = this.pluginName; diff --git a/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts b/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts index d9fd335af0..4c26791a62 100644 --- a/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts +++ b/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts @@ -30,6 +30,8 @@ export interface IRushPluginManifest { associatedCommands?: string[]; commandLineJsonFilePath?: string; rushVersionRange?: string; + /** Declares that the plugin honors the long-lived daemon engine lifecycle. */ + daemonCompatible?: boolean; } export interface IRushPluginManifestJson { diff --git a/libraries/rush-lib/src/pluginFramework/PluginManager.ts b/libraries/rush-lib/src/pluginFramework/PluginManager.ts index 8321713a1a..2322d6f071 100644 --- a/libraries/rush-lib/src/pluginFramework/PluginManager.ts +++ b/libraries/rush-lib/src/pluginFramework/PluginManager.ts @@ -10,7 +10,7 @@ import { BuiltInPluginLoader, type IBuiltInPluginConfiguration } from './PluginL import type { IRushPlugin } from './IRushPlugin'; import { AutoinstallerPluginLoader } from './PluginLoader/AutoinstallerPluginLoader'; import { _createRushSessionForPlugin, type RushSession } from './RushSession'; -import type { PluginLoaderBase } from './PluginLoader/PluginLoaderBase'; +import type { PluginLoaderBase, IRushPluginManifest } from './PluginLoader/PluginLoaderBase'; import { Rush } from '../api/Rush'; import type { RushGlobalFolder } from '../api/RushGlobalFolder'; @@ -123,6 +123,11 @@ export class PluginManager { return this.#loadedPluginNames; } + /** The `pluginName` of every plugin configured in rush-plugins.json, whether or not it has been loaded. */ + public get configuredPluginNames(): ReadonlySet { + return new Set(this.#autoinstallerPluginLoaders.map((pluginLoader) => pluginLoader.pluginName)); + } + public async updateAsync(): Promise { await this._preparePluginAutoinstallersAsync(this.#autoinstallerPluginLoaders); const preparedAutoinstallerNames: Set = new Set(); @@ -201,35 +206,33 @@ export class PluginManager { } /** - * Explains why configured autoinstaller plugins could participate in the specified phased command. + * Explains why configured autoinstaller plugins prevent a long-lived engine from serving the specified + * phased command. * * @remarks - * A plugin is inert for the command only if Rush will neither initialize it (it is associated with - * specific commands, none of which is this command) nor use its command-line.json to define the - * command, a phase of the command, or a parameter associated with either. A manifest or command-line - * file that cannot be read is reported rather than assumed to be inert. + * A plugin is compatible with a long-lived engine if its manifest sets `daemonCompatible` or its name is in + * `compatiblePluginNames`. Any other plugin is inert for the command only if Rush will neither initialize it + * (it is associated with specific commands, none of which is this command) nor use its command-line.json to + * define the command, a phase of the command, or a parameter associated with either. A manifest or + * command-line file that cannot be read is reported rather than assumed to be compatible or inert. * - * @returns An empty array if every configured autoinstaller plugin is inert for the command. + * @returns An empty array if every configured autoinstaller plugin is compatible with, or inert for, the command. */ - public getPluginsParticipatingInCommand( + public getPluginsIncompatibleWithEngine( commandName: string, - phaseNames: ReadonlySet + phaseNames: ReadonlySet, + compatiblePluginNames: ReadonlySet ): ReadonlyArray { const reasons: string[] = []; for (const pluginLoader of this.#autoinstallerPluginLoaders) { const pluginLabel: string = `"${pluginLoader.pluginName}" (${pluginLoader.packageName})`; - let associatedCommands: ReadonlyArray | undefined; + let manifest: IRushPluginManifest; try { - associatedCommands = pluginLoader.pluginManifest.associatedCommands; + manifest = pluginLoader.pluginManifest; } catch (error) { reasons.push(`${pluginLabel}: its manifest could not be read: ${(error as Error).message}`); continue; } - if (!associatedCommands) { - reasons.push(`${pluginLabel} is initialized for every command`); - } else if (associatedCommands.includes(commandName)) { - reasons.push(`${pluginLabel} is associated with "${commandName}"`); - } let commandLineConfiguration: CommandLineConfiguration | undefined; try { @@ -238,6 +241,16 @@ export class PluginManager { reasons.push(`${pluginLabel}: its command-line.json could not be read: ${(error as Error).message}`); continue; } + if (manifest.daemonCompatible || compatiblePluginNames.has(pluginLoader.pluginName)) { + continue; + } + + const { associatedCommands } = manifest; + if (!associatedCommands) { + reasons.push(`${pluginLabel} is initialized for every command`); + } else if (associatedCommands.includes(commandName)) { + reasons.push(`${pluginLabel} is associated with "${commandName}"`); + } if (!commandLineConfiguration) { continue; } diff --git a/libraries/rush-lib/src/schemas/rush-plugin-manifest.schema.json b/libraries/rush-lib/src/schemas/rush-plugin-manifest.schema.json index 43cfaf13b7..2a37a30e76 100644 --- a/libraries/rush-lib/src/schemas/rush-plugin-manifest.schema.json +++ b/libraries/rush-lib/src/schemas/rush-plugin-manifest.schema.json @@ -46,6 +46,10 @@ "description": "Specifies the semver range of Rush versions supported by this plugin.", "type": "string", "minLength": 1 + }, + "daemonCompatible": { + "description": "If true, the plugin declares that it works in a long-lived Rush daemon engine, where one applied instance serves many requests: its session hooks and createOperationsAsync run once per engine (with isWatch false), and its operation graph hooks run for each iteration. Such a plugin does not change process globals (such as process.env, process.exitCode, the working directory or global timers), keeps no per-request state in the process, never prompts, reads the operation environment rather than the daemon's process.env, and releases its resources when the engine is disposed. Without it, the daemon runs commands that the plugin participates in with in-process Rush, unless the repository lists the plugin in the rush.json \"daemon.compatiblePlugins\" setting.", + "type": "boolean" } } } diff --git a/libraries/rush-lib/src/schemas/rush.schema.json b/libraries/rush-lib/src/schemas/rush.schema.json index 980113494a..b702606300 100644 --- a/libraries/rush-lib/src/schemas/rush.schema.json +++ b/libraries/rush-lib/src/schemas/rush.schema.json @@ -311,6 +311,15 @@ "type": "boolean", "default": false, "description": "Prefer measured time-saved * frequency / resident-memory retention over LRU. Never starts unrequested scripts. RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY overrides." + }, + "compatiblePlugins": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[^\\s,](?:[^,]*[^\\s,])?$" + }, + "default": [], + "description": "Names of configured Rush plugins (their pluginName in rush-plugins.json) that this repository has verified for long-lived daemon engines, in addition to plugins whose manifest sets daemonCompatible. A daemon request for a command that any other configured plugin participates in runs with in-process Rush. RUSH_DAEMON_COMPATIBLE_PLUGINS (a comma-separated list) overrides." } } }, From 78b29c21557553fbcf1a109b0ea59b518badff89 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 14:23:19 +0000 Subject: [PATCH 020/265] [rush-daemon] Abort a shared iteration's leftover work once no remaining client needs it Swarm integration step 5; original commit 02f83a99ae (merge of swarm/r05 at bd7ac88caa). Refs: board 207. Commits folded into this step (1): - bd7ac88caa [rush-daemon] Abort a shared iteration's leftover work once no remaining client needs it (board 207) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...m-r05-cancel-orphans_2026-09-28-13-14.json | 10 ++ libraries/rush-daemon/README.md | 5 +- .../rush-daemon/src/PhasedIterationDemand.ts | 93 +++++++++++ .../src/PhasedRequestEventMultiplexer.ts | 2 +- .../rush-daemon/src/PhasedRequestEventSink.ts | 2 +- .../rush-daemon/src/PhasedRequestRouter.ts | 40 ++++- .../src/test/PhasedIterationDemand.test.ts | 158 ++++++++++++++++++ .../src/test/PhasedRequestBatching.test.ts | 4 +- .../test/PhasedRequestCancellation.test.ts | 119 +++++++++++++ 9 files changed, 427 insertions(+), 6 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-cancel-orphans_2026-09-28-13-14.json create mode 100644 libraries/rush-daemon/src/PhasedIterationDemand.ts create mode 100644 libraries/rush-daemon/src/test/PhasedIterationDemand.test.ts diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-cancel-orphans_2026-09-28-13-14.json b/common/changes/@rushstack/rush-daemon/swarm-r05-cancel-orphans_2026-09-28-13-14.json new file mode 100644 index 0000000000..7cc8049401 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-cancel-orphans_2026-09-28-13-14.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Cancellation: when a client leaves a shared iteration, the daemon aborts the iteration as soon as every operation a remaining client needs has finished, so work that only the departed client needed is never started (or is terminated) instead of delaying the remaining client's result and every later request.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 2e489bbb43..4cadfb0bda 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -484,7 +484,10 @@ remaining clients. The last client that still needs the iteration receives its r release, as for a single client. An early result is not published when any of the client's operations was aborted; iteration-wide failures that occur after an early result are reported only to the remaining clients. Requests admitted after scheduling starts form a later batch. Cancelling or disconnecting one client removes its subscription without aborting work needed by other -clients; the graph iteration is aborted only after every client in that batch has stopped needing it. +clients; the graph iteration is aborted once every client in that batch has stopped needing it, or once every +operation that a remaining client needs has finished while work that only departed clients needed has not. In the +latter case unstarted operations never start and running ones are terminated, so neither the remaining client's +result nor later requests wait for work that nobody needs. The typed phased router remains separate from native initialization. `ProductionDaemonRequestResolver` supplies validated exact selections from `PhasedCommandEngine`; other integrations retain the existing dependency-closure diff --git a/libraries/rush-daemon/src/PhasedIterationDemand.ts b/libraries/rush-daemon/src/PhasedIterationDemand.ts new file mode 100644 index 0000000000..2349b884e8 --- /dev/null +++ b/libraries/rush-daemon/src/PhasedIterationDemand.ts @@ -0,0 +1,93 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IOperationExecutionResult, Operation } from '@microsoft/rush-lib'; + +import type { IRequestEventSink } from './PhasedRequestEventMultiplexer'; +import { TERMINAL_OPERATION_STATUSES } from './PhasedRequestEventSink'; + +/** + * Tracks whether the unfinished work of a shared iteration is still needed by one of the batch's remaining clients. + * + * @remarks + * A batch arms the tracker when a participating client leaves (cancellation or output failure) while other + * participants remain. From then on, the tracker reports the iteration as abandoned, exactly once, as soon as every + * operation that a remaining client needs has reached a terminal status while enabled operations that only departed + * clients needed are still unfinished. Disabled operations finish without running anything, so they alone never + * make an iteration abandoned. + * + * The tracker observes status changes rather than completion events: the graph dispatches an operation's dependents + * before that operation's completion event, so aborting on completion could start a process only to terminate it. + */ +export class PhasedIterationDemand implements IRequestEventSink { + readonly #onAbandoned: () => void; + readonly #unfinished: Set = new Set(); + #abandoned: boolean = false; + #needed: ReadonlySet | undefined; + #neededUnfinished: number = 0; + + public constructor(onAbandoned: () => void) { + this.#onAbandoned = onAbandoned; + } + + /** Whether the iteration was reported as abandoned. */ + public get abandoned(): boolean { + return this.#abandoned; + } + + /** + * Arms the tracker with, or narrows it to, the operations that the remaining clients still need. + */ + public restrictTo(neededOperations: Iterable): void { + this.#needed = new Set(neededOperations); + this.#countNeededUnfinished(); + this.#evaluate(); + } + + public onIterationScheduled(records: Iterable): void { + this.#unfinished.clear(); + for (const record of records) { + if (!TERMINAL_OPERATION_STATUSES.has(record.status)) { + this.#unfinished.add(record); + } + } + this.#countNeededUnfinished(); + this.#evaluate(); + } + + public onOperationStatusChanged(record: IOperationExecutionResult): void { + if (!TERMINAL_OPERATION_STATUSES.has(record.status) || !this.#unfinished.delete(record)) { + return; + } + if (this.#needed?.has(record.operation)) { + this.#neededUnfinished--; + this.#evaluate(); + } + } + + #countNeededUnfinished(): void { + let neededUnfinished: number = 0; + if (this.#needed) { + for (const record of this.#unfinished) { + if (this.#needed.has(record.operation)) { + neededUnfinished++; + } + } + } + this.#neededUnfinished = neededUnfinished; + } + + #evaluate(): void { + if (this.#abandoned || !this.#needed || this.#neededUnfinished > 0) { + return; + } + // Every unfinished operation is now one that no remaining client needs. + for (const record of this.#unfinished) { + if (record.enabled) { + this.#abandoned = true; + this.#onAbandoned(); + return; + } + } + } +} diff --git a/libraries/rush-daemon/src/PhasedRequestEventMultiplexer.ts b/libraries/rush-daemon/src/PhasedRequestEventMultiplexer.ts index 25f7850339..39991db255 100644 --- a/libraries/rush-daemon/src/PhasedRequestEventMultiplexer.ts +++ b/libraries/rush-daemon/src/PhasedRequestEventMultiplexer.ts @@ -9,7 +9,7 @@ import type { } from '@microsoft/rush-lib'; import type { ITerminalChunk } from '@rushstack/terminal'; -interface IRequestEventSink extends _IOperationGraphEventSink { +export interface IRequestEventSink extends _IOperationGraphEventSink { onIterationScheduled(records: Iterable): void; } diff --git a/libraries/rush-daemon/src/PhasedRequestEventSink.ts b/libraries/rush-daemon/src/PhasedRequestEventSink.ts index d4a315526f..0fc35cbf01 100644 --- a/libraries/rush-daemon/src/PhasedRequestEventSink.ts +++ b/libraries/rush-daemon/src/PhasedRequestEventSink.ts @@ -29,7 +29,7 @@ const EVENT_SOURCE_PACKAGE: string = '@microsoft/rush-lib'; const EVENT_SOURCE_COMPONENT: string = 'OperationGraph'; const TEXT_ENCODER: InstanceType = new TextEncoder(); // Mirrors rush-lib's TERMINAL_STATUSES, which is not part of its public API. -const TERMINAL_OPERATION_STATUSES: ReadonlySet = new Set([ +export const TERMINAL_OPERATION_STATUSES: ReadonlySet = new Set([ OperationStatus.Success, OperationStatus.SuccessWithWarning, OperationStatus.Skipped, diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index b132616378..76ccbba728 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -20,6 +20,7 @@ import type { import { PhasedRequestEventSink } from './PhasedRequestEventSink'; import { PhasedRequestEventMultiplexer } from './PhasedRequestEventMultiplexer'; +import { PhasedIterationDemand } from './PhasedIterationDemand'; import { writePhasedRequestSummary } from './PhasedRequestSummary'; import type { IPhasedRequestClient } from './PhasedRequestClient'; import { DaemonRequiresInProcessError, evaluateDaemonTerminalPolicy } from './DaemonTerminalPolicy'; @@ -244,6 +245,8 @@ class PhasedRequestBatchCoordinator { readonly #abortErrors: unknown[] = []; #abortTail: Promise = Promise.resolve(); #acceptingCurrentBatch: boolean = false; + /** Set while the current batch's iteration may execute; see `#restrictBatchDemand`. */ + #batchDemand: PhasedIterationDemand | undefined; #currentBatch: ReadonlyArray | undefined; #drainScheduled: boolean = false; #nextGraphLeasePromise: Promise | undefined; @@ -434,6 +437,9 @@ class PhasedRequestBatchCoordinator { this.#finishDetachedEntry(entry); } } + const demand: PhasedIterationDemand = new PhasedIterationDemand(() => this.#onBatchAbandoned(demand)); + this.#batchDemand = demand; + const unsubscribeDemand: () => void = this.#multiplexer.subscribe(demand); for (const entry of participants) { entry.participated = true; const activeOperationIds: ReadonlySet = new Set( @@ -472,7 +478,7 @@ class PhasedRequestBatchCoordinator { }) ); const executionPromise: Promise = this.#graph.executeScheduledIterationAsync(); - if (!participants.some((entry: IBatchEntry) => this.#needsIteration(entry))) { + if (!participants.some((entry: IBatchEntry) => this.#needsIteration(entry)) || demand.abandoned) { // Let executeScheduledIterationAsync promote the scheduled iteration before aborting it. await Promise.resolve(); this.#requestIterationAbort(); @@ -494,6 +500,8 @@ class PhasedRequestBatchCoordinator { } } } finally { + this.#batchDemand = undefined; + unsubscribeDemand(); for (const entry of participants) { entry.unsubscribe?.(); entry.unsubscribe = undefined; @@ -551,6 +559,9 @@ class PhasedRequestBatchCoordinator { ) { // Other live participants still need the shared work: detach this client and answer it now. this.#finishDetachedEntry(entry); + if (entry.participated) { + this.#restrictBatchDemand(); + } return; } @@ -580,6 +591,30 @@ class PhasedRequestBatchCoordinator { }); } + /** + * Narrows the running iteration to the remaining participants' selections after a participant left it. + * + * @remarks + * The departed client's selection stays merged into the iteration, so without this the last remaining + * participant would wait for, and queued requests would queue behind, work that only the departed client needed. + * Once every operation a remaining participant needs has finished, the iteration is aborted instead: operations + * that have not started are never started, and running ones are terminated, as when every client cancels. + */ + #restrictBatchDemand(): void { + this.#batchDemand?.restrictTo( + (this.#currentBatch ?? []).flatMap((candidate: IBatchEntry) => + this.#needsIteration(candidate) ? candidate.selection.activeOperations : [] + ) + ); + } + + #onBatchAbandoned(demand: PhasedIterationDemand): void { + // Before the iteration starts executing, `#executeBatchAsync` aborts it once it has been promoted. + if (this.#batchDemand === demand && this.#graph.status === OperationStatus.Executing) { + this.#requestIterationAbort(); + } + } + #isEntryLive(entry: IBatchEntry): boolean { return !entry.abortRequested && !entry.client.abortSignal.aborted && entry.outputError === undefined; } @@ -614,7 +649,8 @@ class PhasedRequestBatchCoordinator { * The sink invokes this after the operations' final events and log chunks were enqueued, and `#finishEntryAsync` * drains them before writing the result. The iteration, graph lease and execution lease stay owned by the batch. * The last participant that needs the iteration keeps the ordinary contract: its result follows iteration end - * and execution lease release, so single-client requests and warm-state retention are unchanged. + * and execution lease release, so single-client requests and warm-state retention are unchanged. When other + * participants left the batch, `#restrictBatchDemand` makes that end prompt by aborting work only they needed. */ #finishSettledEntry(entry: IBatchEntry): void { if ( diff --git a/libraries/rush-daemon/src/test/PhasedIterationDemand.test.ts b/libraries/rush-daemon/src/test/PhasedIterationDemand.test.ts new file mode 100644 index 0000000000..476d3d4c62 --- /dev/null +++ b/libraries/rush-daemon/src/test/PhasedIterationDemand.test.ts @@ -0,0 +1,158 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IOperationExecutionResult, Operation } from '@microsoft/rush-lib'; +import { OperationStatus } from '@microsoft/rush-lib'; + +import { PhasedIterationDemand } from '../PhasedIterationDemand'; + +interface ITestRecord { + readonly enabled: boolean; + readonly operation: Operation; + status: OperationStatus; +} + +function createRecord(name: string, enabled: boolean = true): ITestRecord { + return { enabled, operation: { name } as unknown as Operation, status: OperationStatus.Waiting }; +} + +function asResult(record: ITestRecord): IOperationExecutionResult { + return record as unknown as IOperationExecutionResult; +} + +interface ITestDemand { + readonly abandonedCount: () => number; + readonly demand: PhasedIterationDemand; + readonly schedule: (...records: ReadonlyArray) => void; + readonly setStatus: (record: ITestRecord, status: OperationStatus) => void; +} + +function createDemand(): ITestDemand { + let abandonedCount: number = 0; + const demand: PhasedIterationDemand = new PhasedIterationDemand(() => abandonedCount++); + return { + abandonedCount: () => abandonedCount, + demand, + schedule: (...records: ReadonlyArray) => demand.onIterationScheduled(records.map(asResult)), + setStatus: (record: ITestRecord, status: OperationStatus) => { + record.status = status; + demand.onOperationStatusChanged(asResult(record)); + } + }; +} + +describe(PhasedIterationDemand.name, () => { + it('never reports an iteration abandoned until it is armed', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const needed: ITestRecord = createRecord('needed'); + const other: ITestRecord = createRecord('other'); + schedule(needed, other); + + setStatus(needed, OperationStatus.Success); + + expect(abandonedCount()).toBe(0); + expect(demand.abandoned).toBe(false); + }); + + it('reports abandonment once, when the needed work finishes before enabled unneeded work', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const needed: ITestRecord = createRecord('needed'); + const running: ITestRecord = createRecord('running'); + const waiting: ITestRecord = createRecord('waiting'); + schedule(needed, running, waiting); + demand.restrictTo([needed.operation]); + setStatus(running, OperationStatus.Executing); + setStatus(needed, OperationStatus.Executing); + expect(abandonedCount()).toBe(0); + + setStatus(needed, OperationStatus.Success); + expect(abandonedCount()).toBe(1); + expect(demand.abandoned).toBe(true); + + setStatus(running, OperationStatus.Aborted); + setStatus(waiting, OperationStatus.Aborted); + expect(abandonedCount()).toBe(1); + }); + + it('does not report abandonment when unneeded work finishes first', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const needed: ITestRecord = createRecord('needed'); + const other: ITestRecord = createRecord('other'); + schedule(needed, other); + demand.restrictTo([needed.operation]); + + setStatus(other, OperationStatus.Success); + setStatus(needed, OperationStatus.Success); + + expect(abandonedCount()).toBe(0); + }); + + it('ignores unfinished records that are disabled, since they run nothing', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const needed: ITestRecord = createRecord('needed'); + schedule(needed, createRecord('disabled', false)); + demand.restrictTo([needed.operation]); + + setStatus(needed, OperationStatus.Success); + + expect(abandonedCount()).toBe(0); + }); + + it('counts only the first terminal status of an iteration record', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const first: ITestRecord = createRecord('first'); + const second: ITestRecord = createRecord('second'); + schedule(first, second, createRecord('unneeded')); + demand.restrictTo([first.operation, second.operation]); + + setStatus(first, OperationStatus.Success); + // For example, a failing cache write after the operation succeeded. + setStatus(first, OperationStatus.Failure); + // A record outside the iteration, such as a retained result that is invalidated, is ignored. + demand.onOperationStatusChanged(asResult({ ...second, status: OperationStatus.Success })); + expect(abandonedCount()).toBe(0); + + setStatus(second, OperationStatus.Blocked); + expect(abandonedCount()).toBe(1); + }); + + it('reports abandonment at once when it is armed after the needed work finished', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const needed: ITestRecord = createRecord('needed'); + schedule(needed, createRecord('unneeded')); + setStatus(needed, OperationStatus.FromCache); + expect(abandonedCount()).toBe(0); + + demand.restrictTo([needed.operation]); + + expect(abandonedCount()).toBe(1); + }); + + it('narrows to the remaining clients when another client leaves', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const small: ITestRecord = createRecord('small'); + const large: ITestRecord = createRecord('large'); + schedule(small, large, createRecord('departed')); + demand.restrictTo([small.operation, large.operation]); + setStatus(small, OperationStatus.Success); + expect(abandonedCount()).toBe(0); + + demand.restrictTo([small.operation]); + + expect(abandonedCount()).toBe(1); + }); + + it('applies an arming that precedes scheduling, and treats records that are already terminal as finished', () => { + const { abandonedCount, demand, schedule, setStatus } = createDemand(); + const needed: ITestRecord = createRecord('needed'); + const alreadyFinished: ITestRecord = createRecord('already-finished'); + alreadyFinished.status = OperationStatus.Success; + demand.restrictTo([needed.operation, alreadyFinished.operation]); + schedule(needed, alreadyFinished, createRecord('unneeded')); + expect(abandonedCount()).toBe(0); + + setStatus(needed, OperationStatus.Success); + + expect(abandonedCount()).toBe(1); + }); +}); diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index aab404064f..bca1bde260 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -476,12 +476,14 @@ describe('shared phased request batching', () => { const abortCallCountBeforeCancellation: number = abortSpy.mock.calls.length; cancelledClient.abortController.abort(); + // The continuing client still needs its running operation, so the cancellation itself aborts nothing. Once that + // operation finishes, work that only the cancelled client needed may be aborted (PhasedRequestCancellation.test). + expect(abortSpy).toHaveBeenCalledTimes(abortCallCountBeforeCancellation); releaseOperation.resolve(); const [cancelledResult, continuingResult] = await Promise.all([cancelled, continuing]); expect(cancelledResult).toMatchObject({ aborted: true, outcome: 'aborted' }); expect(continuingResult).toMatchObject({ exitCode: 0, outcome: 'success' }); - expect(abortSpy).toHaveBeenCalledTimes(abortCallCountBeforeCancellation); expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); }); diff --git a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts index 7642e99cb4..44c3358cd4 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts @@ -16,7 +16,23 @@ import type { ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; const OPERATION_A: string = 'project-a (_phase:test)'; const OPERATION_B: string = 'project-b (_phase:test)'; +const OPERATION_C: string = 'project-c (_phase:test)'; const PROMPT_CANCELLATION_MS: number = 1000; +const TIMED_OUT: 'timed out' = 'timed out'; + +async function raceWithTimeoutAsync(promise: Promise, timeoutMs: number): Promise { + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + promise, + new Promise((resolve) => { + timer = setTimeout(() => resolve(TIMED_OUT), timeoutMs); + }) + ]); + } finally { + clearTimeout(timer); + } +} function createRequest(requestId: string, operationId: string): IDaemonPhasedRequest { return { @@ -186,4 +202,107 @@ describe('phased request client cancellation', () => { hanging.release(); expect(await continuing).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); }); + + it('never starts work that only a cancelled client needed once the remaining client has its operations', async () => { + const { hanging: shared, actionAsync: sharedActionAsync } = createHangingOperation(); + const { hanging: orphan, actionAsync: orphanActionAsync } = createHangingOperation(); + // The cancelled client needs A, then B, then C. The remaining client needs only A. + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A, OperationStatus.Success, sharedActionAsync)], + [OPERATION_B, new TestOperationRunner(OPERATION_B, OperationStatus.Success, orphanActionAsync)], + [OPERATION_C, new TestOperationRunner(OPERATION_C)] + ]), + [ + [OPERATION_B, OPERATION_A], + [OPERATION_C, OPERATION_B] + ], + { supportsTerminateRunning: true } + ); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const cancelledClient: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const cancelled: Promise = router.executeAsync( + createRequest('cancelled', OPERATION_C), + cancelledClient + ); + const continuing: Promise = router.executeAsync( + createRequest('continuing', OPERATION_A), + new TestPhasedRequestClient('two') + ); + await shared.started; + + cancelledClient.abortController.abort(); + expect(await cancelled).toMatchObject({ aborted: true, outcome: 'aborted' }); + // The remaining client still needs the running shared operation. + expect(abortSpy).not.toHaveBeenCalledWith({ terminateRunning: true }); + + const releasedAt: number = Date.now(); + shared.release(); + const result: IDaemonPhasedRequestResult | typeof TIMED_OUT = await raceWithTimeoutAsync( + continuing, + PROMPT_CANCELLATION_MS + ); + // Lets an iteration that started the abandoned work finish, so a failure below is reported cleanly. + orphan.release(); + + expect(result).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); + expect(Date.now() - releasedAt).toBeLessThan(PROMPT_CANCELLATION_MS); + expect((result as IDaemonPhasedRequestResult).operationResults).toEqual([ + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Success }) + ]); + expect(abortSpy).toHaveBeenCalledWith({ terminateRunning: true }); + expect(fixture.runners.get(OPERATION_B)?.runCount).toBe(0); + expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(0); + // The remaining client's result stays warm; the abandoned operations are not retained. + expect(new Set([...fixture.graph.resultByOperation.keys()].map(({ name }) => name))).toEqual( + new Set([OPERATION_A]) + ); + }); + + it('terminates running work that only a cancelled client needed once the remaining client has its operations', async () => { + const { hanging: remaining, actionAsync: remainingActionAsync } = createHangingOperation(); + const { hanging: orphan, actionAsync: orphanActionAsync } = createHangingOperation(); + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A, OperationStatus.Success, remainingActionAsync)], + [OPERATION_B, new TestOperationRunner(OPERATION_B, OperationStatus.Success, orphanActionAsync)] + ]), + [], + { supportsTerminateRunning: true } + ); + fixture.graph.parallelism = 2; + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const cancelledClient: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const cancelled: Promise = router.executeAsync( + createRequest('cancelled', OPERATION_B), + cancelledClient + ); + const continuing: Promise = router.executeAsync( + createRequest('continuing', OPERATION_A), + new TestPhasedRequestClient('two') + ); + await Promise.all([remaining.started, orphan.started]); + + cancelledClient.abortController.abort(); + expect(await cancelled).toMatchObject({ aborted: true, outcome: 'aborted' }); + // While the remaining client's operation runs, the iteration is left alone. + expect(orphan.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([false]); + + const releasedAt: number = Date.now(); + remaining.release(); + const result: IDaemonPhasedRequestResult | typeof TIMED_OUT = await raceWithTimeoutAsync( + continuing, + PROMPT_CANCELLATION_MS + ); + orphan.release(); + + expect(result).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); + expect(Date.now() - releasedAt).toBeLessThan(PROMPT_CANCELLATION_MS); + expect((result as IDaemonPhasedRequestResult).operationResults).toEqual([ + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Success }) + ]); + expect(orphan.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([true]); + await orphan.terminated; + }); }); From cf3ff0106a2d5c74ace62b7dbcfe1829432bd11f Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 13:47:41 +0000 Subject: [PATCH 021/265] [rush-lib] Run the initial script for every command outside watch mode Swarm integration step 6; original commit 519ae33c8d. Refs: task 50, board 115. ShellOperationRunner runs a phase's `:incremental` script whenever the operation has a last state. Native `rush build` executes one iteration, so only watch mode ever had one, and watch mode turns cache writes off. rushd keeps one operation graph across requests, so from its second request on a daemon `build` ran `_phase:build:incremental` (in odsp-web, `heft run --only build --` without --clean) with cache writes on. Outputs of a deleted or renamed source file stayed in lib/ and were cached under the key native Rush computes for the clean build, so later native builds restored them (t04 board 115, board 450). ShellOperationRunnerPlugin now sets the incremental command only in watch mode. Every other command runs the initial script, exactly as a single `rush build` does, which is option (a) of task 50. Native `rush build` is unchanged (it never has a last state) and watch mode keeps `:incremental` with writes off. Tests: a rush-lib unit test runs a repeated operation with and without watch mode, and a daemon integration test runs three warm builds of a project with an `:incremental` script. Before the fix the second daemon build logged "Invoking (incremental): node build.cjs --incremental" followed by "Successfully set cache entry". (cherry picked from commit e382d2ac80f881451b7d527ea9a97e3037880923) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...script-outside-watch_2026-09-28-14-05.json | 11 +++ ...script-outside-watch_2026-09-28-14-05.json | 11 +++ .../ProductionDaemonRequestResolver.test.ts | 33 ++++++- .../operations/ShellOperationRunnerPlugin.ts | 14 ++- .../test/ShellOperationRunnerPlugin.test.ts | 94 ++++++++++++++++++- 5 files changed, 155 insertions(+), 8 deletions(-) create mode 100644 common/changes/@microsoft/rush/initial-script-outside-watch_2026-09-28-14-05.json create mode 100644 common/changes/@rushstack/rush-daemon/initial-script-outside-watch_2026-09-28-14-05.json diff --git a/common/changes/@microsoft/rush/initial-script-outside-watch_2026-09-28-14-05.json b/common/changes/@microsoft/rush/initial-script-outside-watch_2026-09-28-14-05.json new file mode 100644 index 0000000000..f5dedf2dae --- /dev/null +++ b/common/changes/@microsoft/rush/initial-script-outside-watch_2026-09-28-14-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Run a phase's `:incremental` script only for repeated watch-mode iterations. A long-lived host that runs separate commands on one operation graph now runs the initial script, as a single `rush build` does, so outputs of deleted inputs are not kept or written to the build cache.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/initial-script-outside-watch_2026-09-28-14-05.json b/common/changes/@rushstack/rush-daemon/initial-script-outside-watch_2026-09-28-14-05.json new file mode 100644 index 0000000000..3086d1cb95 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/initial-script-outside-watch_2026-09-28-14-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Test that every warm daemon build request runs the initial script of an operation, as native `rush build` does.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index dd82aa7c09..704df32f27 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -77,6 +77,8 @@ interface IFixtureOptions { readonly getSuccessorLaunchAsync?: GetWorkspaceSuccessorLaunchAsync; readonly onSessionCreated?: (session: WorkspaceSession) => void; readonly resolver?: IDaemonRequestResolver; + /** Adds the watch-only `_phase:compile:incremental` script, which passes `--incremental` to build.cjs. */ + readonly incrementalScript?: boolean; } class DecoratedTestResolver implements IDaemonRequestResolver { @@ -189,7 +191,12 @@ async function createFixtureAsync( JSON.stringify({ name, version: '1.0.0', - scripts: { '_phase:compile': 'node build.cjs' }, + scripts: { + '_phase:compile': 'node build.cjs', + ...(options.incrementalScript + ? { '_phase:compile:incremental': 'node build.cjs --incremental' } + : {}) + }, dependencies: name === 'b' ? { a: '1.0.0' } : {} }) ); @@ -1403,6 +1410,30 @@ process.exit(23); } }); + it('runs the initial script for every warm build request, as native rush build does, not the watch-only one', async () => { + const fixture: IFixture = await createFixtureAsync(true, 'direct', { incrementalScript: true }); + try { + for (const [requestId, input] of [ + ['initial-script-1', 'one'], + ['initial-script-2', 'two'], + ['initial-script-3', 'three'] + ]) { + fs.writeFileSync(path.join(fixture.repoRoot, 'projects/a/input.txt'), input); + const exchange: ITerminalExchange = await runAsync(fixture, requestId, ['build', '--only', 'a']); + expect(exchange.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, operationResults: [{ operationId: 'a (compile)', status: 'SUCCESS' }] } + }); + expect(logText(exchange)).toContain('Invoking (initial): node build.cjs'); + } + // A watch-only incremental script can keep outputs of deleted inputs, and its output would be cached + // under the key of the initial script that native Rush runs. + expect(runs(fixture)).toEqual(['a:one:', 'a:two:', 'a:three:']); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('re-runs only the operation whose declared outputs were deleted after a warm build', async () => { const fixture: IFixture = await createFixtureAsync(); try { diff --git a/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts b/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts index 990aedb617..3330ac8e32 100644 --- a/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts @@ -30,7 +30,7 @@ export class ShellOperationRunnerPlugin implements IPhasedCommandPlugin { operations: Set, context: ICreateOperationsContext ): Set { - const { rushConfiguration, isIncrementalBuildAllowed } = context; + const { rushConfiguration, isIncrementalBuildAllowed, isWatch } = context; const getCustomParameterValues: (operation: Operation) => ICustomParameterValuesForOperation = getCustomParameterValuesByOperation(); @@ -52,12 +52,16 @@ export class ShellOperationRunnerPlugin implements IPhasedCommandPlugin { // This is the command that will be used to identify the cache entry for this operation const commandForHash: string | undefined = shellCommand ?? scripts?.[phaseName]; - // For execution of non-initial iterations, prefer the `:incremental` script if it exists. + // For execution of non-initial watch iterations, prefer the `:incremental` script if it exists. // However, the `shellCommand` value still takes precedence per the spec for that feature. + // Outside watch mode, every command runs the initial script, as a single `rush build` does, even + // when a long-lived host (rushd) executes it on a graph that already ran the operation. The + // incremental script may keep outputs of deleted inputs, and only watch mode disables cache writes. const initialCommand: string | undefined = shellCommand ?? scripts?.[phaseName]; - const incrementalCommand: string | undefined = isIncrementalBuildAllowed - ? (shellCommand ?? scripts?.[`${phaseName}:incremental`]) - : undefined; + const incrementalCommand: string | undefined = + isIncrementalBuildAllowed && isWatch + ? (shellCommand ?? scripts?.[`${phaseName}:incremental`]) + : undefined; operation.runner = initializeShellOperationRunner({ phase, diff --git a/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts index 3e558acc90..b93ecb3d91 100644 --- a/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts @@ -1,9 +1,19 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type * as childProcess from 'node:child_process'; +import { EventEmitter } from 'node:events'; import path from 'node:path'; +import { PassThrough } from 'node:stream'; + import { JsonFile } from '@rushstack/node-core-library'; -import { ConsoleTerminalProvider, Terminal } from '@rushstack/terminal'; +import { + ConsoleTerminalProvider, + StringBufferTerminalProvider, + Terminal, + type ITerminal, + type ITerminalProvider +} from '@rushstack/terminal'; import { CommandLineAction, CommandLineParser, type CommandLineParameter } from '@rushstack/ts-command-line'; import { RushConfiguration } from '../../../api/RushConfiguration'; @@ -13,7 +23,11 @@ import { type IParameterJson, type IPhase } from '../../../api/CommandLineConfiguration'; -import type { Operation } from '../Operation'; +import { Operation } from '../Operation'; +import type { IOperationRunnerContext } from '../IOperationRunner'; +import { OperationStatus } from '../OperationStatus'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import { Utilities } from '../../../utilities/Utilities'; import type { ICommandLineJson } from '../../../api/CommandLineJson'; import { PhasedOperationPlugin } from '../PhasedOperationPlugin'; import { ShellOperationRunnerPlugin } from '../ShellOperationRunnerPlugin'; @@ -259,4 +273,80 @@ describe(ShellOperationRunnerPlugin.name, () => { // All projects snapshot expect(Array.from(operations, serializeOperation)).toMatchSnapshot(); }); + + it.each([ + [false, ['node build.js', 'node build.js']], + [true, ['node build.js', 'node build.js --incremental']] + ])( + 'runs the :incremental script for a repeated operation only in watch mode (isWatch: %s)', + async (isWatch: boolean, expectedCommands: string[]) => { + const phase: IPhase = { + name: '_phase:build', + isSynthetic: false, + missingScriptBehavior: 'error', + allowWarningsOnSuccess: false, + associatedParameters: new Set() + } as unknown as IPhase; + const project: RushConfigurationProject = { + packageName: 'a', + projectFolder: process.cwd(), + packageJson: { + scripts: { + '_phase:build': 'node build.js', + '_phase:build:incremental': 'node build.js --incremental' + } + }, + rushConfiguration: { commonTempFolder: process.cwd() } + } as unknown as RushConfigurationProject; + const operation: Operation = new Operation({ phase, project, logFilenameIdentifier: 'a' }); + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new ShellOperationRunnerPlugin().apply(hooks); + await hooks.createOperationsAsync.promise(new Set([operation]), { + isIncrementalBuildAllowed: true, + isWatch + } as unknown as ICreateOperationsContext); + + const commands: string[] = []; + const executeSpy = jest + .spyOn(Utilities, 'executeLifecycleCommandAsync') + .mockImplementation((command) => { + commands.push(command.trim()); + const stdout: PassThrough = new PassThrough(); + const stderr: PassThrough = new PassThrough(); + const child: childProcess.ChildProcess = Object.assign(new EventEmitter(), { + stdout, + stderr, + stdio: [] + }) as unknown as childProcess.ChildProcess; + queueMicrotask(() => { + stdout.end(); + stderr.end(); + child.emit('close', 0, null); + }); + return child; + }); + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const context: IOperationRunnerContext = { + environment: undefined, + async runWithTerminalAsync( + callback: ( + terminal: ITerminal, + operationTerminalProvider: ITerminalProvider, + structuredChildOutputTerminalProvider: ITerminalProvider + ) => Promise + ): Promise { + return await callback(new Terminal(terminalProvider), terminalProvider, terminalProvider); + } + } as unknown as IOperationRunnerContext; + try { + await expect(operation.runner!.executeAsync(context)).resolves.toBe(OperationStatus.Success); + await expect( + operation.runner!.executeAsync(context, { status: OperationStatus.Success }) + ).resolves.toBe(OperationStatus.Success); + } finally { + executeSpy.mockRestore(); + } + expect(commands).toEqual(expectedCommands); + } + ); }); From 889b2b4007d3dc4039bc7cbf02bfb090d8a11d22 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 15:23:57 +0000 Subject: [PATCH 022/265] [rush-daemon] Share input captures between concurrent requests Swarm integration step 7; original commit f7ac88dc9d (merge of swarm/r07 at 55a16f14ef). Refs: F1 coalescer, board 555. GATE OK ch01 board 930 (tree 41b2227d59). CONFIRMED by ch03 board 681, t06 board 701/board 749, t01 board 861. Only 55a16f14ef; b6692d09a5 (task 74) is not gated. Commits folded into this step (1): - 55a16f14ef [rush-daemon] Share input captures between concurrent requests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...hared-input-captures_2026-09-28-13-45.json | 10 ++ .../rush-daemon/src/FreshCaptureCoalescer.ts | 49 ++++++ .../src/WorkspaceRequestLifecycle.ts | 58 ++++--- .../src/test/FreshCaptureCoalescer.test.ts | 150 ++++++++++++++++++ .../test/WorkspaceInputCaptureSharing.test.ts | 83 ++++++++++ 5 files changed, 327 insertions(+), 23 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r07-shared-input-captures_2026-09-28-13-45.json create mode 100644 libraries/rush-daemon/src/FreshCaptureCoalescer.ts create mode 100644 libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts diff --git a/common/changes/@rushstack/rush-daemon/swarm-r07-shared-input-captures_2026-09-28-13-45.json b/common/changes/@rushstack/rush-daemon/swarm-r07-shared-input-captures_2026-09-28-13-45.json new file mode 100644 index 0000000000..00bed838d1 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r07-shared-input-captures_2026-09-28-13-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Concurrent requests share workspace input and project configuration captures instead of each running its own. A request never receives a capture that started before it arrived, so a burst of N warm builds costs at most two captures instead of N.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/FreshCaptureCoalescer.ts b/libraries/rush-daemon/src/FreshCaptureCoalescer.ts new file mode 100644 index 0000000000..1b16286973 --- /dev/null +++ b/libraries/rush-daemon/src/FreshCaptureCoalescer.ts @@ -0,0 +1,49 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +interface ICapture { + readonly running: Promise; + next: Promise | undefined; +} + +function ignore(): void {} + +/** + * Shares workspace input captures between concurrent requests, without ever giving a caller a capture that + * began before the caller asked for one. + * + * @remarks + * A capture reads the workspace while it runs, so a capture that is already running can miss a change made just + * before a later caller arrived. A caller that finds a capture running therefore waits for the next capture, + * which starts when the running one settles and is shared by every caller that arrived in the meantime. + * Concurrent callers cost at most two captures instead of one each, and every caller receives a result that is + * at least as fresh as a capture it started itself. Nothing is retained once a capture settles. + * + * Callers that pass the same scope and key must request the same capture. + */ +export class FreshCaptureCoalescer { + readonly #captures: WeakMap>> = new WeakMap(); + + public captureAsync(scope: TScope, key: string, captureAsync: () => Promise): Promise { + const capture: ICapture | undefined = this.#captures.get(scope)?.get(key); + if (!capture) return this.#start(scope, key, captureAsync); + capture.next ??= capture.running.then(ignore, ignore).then(() => this.#start(scope, key, captureAsync)); + return capture.next; + } + + #start(scope: TScope, key: string, captureAsync: () => Promise): Promise { + let captures: Map> | undefined = this.#captures.get(scope); + if (!captures) { + captures = new Map(); + this.#captures.set(scope, captures); + } + const running: Promise = new Promise((resolve) => resolve(captureAsync())); + const capture: ICapture = { running, next: undefined }; + captures.set(key, capture); + const forget: () => void = () => { + if (captures.get(key) === capture) captures.delete(key); + }; + running.then(forget, forget); + return running; + } +} diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 699c1eec21..0cf37b2483 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -8,11 +8,13 @@ import { captureWorkspaceInputFingerprintAsync, classifyWorkspaceInputChange, EnvironmentVariableNames, + getWorkspaceFingerprintEnvironmentEntries, PhasedCommandEngineBusyError, Rush, WorkspaceInputChangeTier, WorkspaceRuntimeFingerprintCache, - type IWorkspaceInputFingerprint + type IWorkspaceInputFingerprint, + type RushConfiguration } from '@microsoft/rush-lib'; import { LockFile } from '@rushstack/node-core-library'; import { NoOpTerminalProvider, Terminal } from '@rushstack/terminal'; @@ -52,6 +54,7 @@ import type { IWorkspaceSuccessorLaunch } from './WorkspaceProcessRestart'; import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from './WorkspaceRestartArbiter'; +import { FreshCaptureCoalescer } from './FreshCaptureCoalescer'; interface IExecutionState { began: boolean; @@ -116,6 +119,11 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { readonly #runtimePaths: ReadonlyArray = [__dirname, path.resolve(__dirname, '../package.json')]; readonly #startupFingerprint: IWorkspaceInputFingerprint; readonly #runtimeCache: WorkspaceRuntimeFingerprintCache; + // Concurrent requests share captures; each capture still starts after the requests it serves arrived. + readonly #fingerprintCaptures: FreshCaptureCoalescer = + new FreshCaptureCoalescer(); + readonly #projectFingerprintCaptures: FreshCaptureCoalescer = + new FreshCaptureCoalescer(); #fingerprint: IWorkspaceInputFingerprint; #projectFingerprint: string | undefined; #commandIdentity: string | undefined; @@ -364,10 +372,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { currentTier !== WorkspaceInputChangeTier.Reuse || (this.#boundSession && this.#projectFingerprint !== - (await captureProjectConfigurationFingerprintAsync( - session.rushConfiguration, - this.#terminal - ))) + (await this.#captureProjectFingerprintAsync(session))) ) { throw new Error( 'Graph inputs changed. Load the new generation with a supported build request; no operation was scheduled or executed.' @@ -405,10 +410,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { abortSignal: client.abortSignal }); if (tier === WorkspaceInputChangeTier.Reuse) { - projectFingerprint = await captureProjectConfigurationFingerprintAsync( - session.rushConfiguration, - this.#terminal - ); + projectFingerprint = await this.#captureProjectFingerprintAsync(session); if ( this.#boundSession !== session || this.#commandIdentity !== commandIdentity || @@ -502,10 +504,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { session.invalidations.getSnapshot().isWatcherHealthy && !session.invalidations.hasUnattributedUnknownChanges ) { - projectFingerprint = await captureProjectConfigurationFingerprintAsync( - session.rushConfiguration, - this.#terminal - ); + projectFingerprint = await this.#captureProjectFingerprintAsync(session); if (projectFingerprint === this.#projectFingerprint) { this.#lastReloadTier = WorkspaceInputChangeTier.Reuse; this.#gate.downgradeExclusiveLease(lease, RequestExclusivityClass.SharedBuild); @@ -587,10 +586,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { expectedFingerprint = after; this.#boundSession = session; this.#fingerprint = after; - this.#projectFingerprint = await captureProjectConfigurationFingerprintAsync( - session.rushConfiguration, - this.#terminal - ); + this.#projectFingerprint = await this.#captureProjectFingerprintAsync(session); this.#commandIdentity = await getResolverLifecycle(resolver).getCommandParameterIdentityAsync({ envelope, workspaceSession: session, @@ -640,12 +636,28 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { session: IWorkspaceSession, envelope: IDaemonRequestEnvelope ): Promise { - return captureWorkspaceInputFingerprintAsync({ - rushConfiguration: session.rushConfiguration, - environment: envelope.environment, - runtimePaths: this.#runtimePaths, - runtimeCache: this.#runtimeCache - }); + const { rushConfiguration } = session; + const { environment } = envelope; + // The capture reads only these parts of an environment, normalized as every fingerprint comparison is. + const key: string = JSON.stringify([ + environment.RUSH_PREVIEW_VERSION ?? null, + getWorkspaceFingerprintEnvironmentEntries(environment) + ]); + return this.#fingerprintCaptures.captureAsync(rushConfiguration, key, () => + captureWorkspaceInputFingerprintAsync({ + rushConfiguration, + environment, + runtimePaths: this.#runtimePaths, + runtimeCache: this.#runtimeCache + }) + ); + } + + #captureProjectFingerprintAsync(session: IWorkspaceSession): Promise { + const { rushConfiguration } = session; + return this.#projectFingerprintCaptures.captureAsync(rushConfiguration, '', () => + captureProjectConfigurationFingerprintAsync(rushConfiguration, this.#terminal) + ); } #classify(fingerprint: IWorkspaceInputFingerprint, mutation: boolean): WorkspaceInputChangeTier { diff --git a/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts b/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts new file mode 100644 index 0000000000..207f2f79e1 --- /dev/null +++ b/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts @@ -0,0 +1,150 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { FreshCaptureCoalescer } from '../FreshCaptureCoalescer'; + +interface IStartedCapture { + readonly index: number; + readonly resolve: (value: string) => void; + readonly reject: (error: Error) => void; +} + +/** Captures that start in order and settle only when the test says so. */ +class ControlledCaptures { + public readonly started: IStartedCapture[] = []; + + public readonly captureAsync = (): Promise => { + return new Promise((resolve, reject) => { + this.started.push({ index: this.started.length, resolve, reject }); + }); + }; +} + +async function flushAsync(): Promise { + await new Promise((resolve) => setImmediate(resolve)); +} + +describe(FreshCaptureCoalescer.name, () => { + it('starts a capture for a caller when none is running and retains nothing once it settles', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + expect(captures.started).toHaveLength(1); + captures.started[0].resolve('first'); + await expect(first).resolves.toBe('first'); + + const second: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + expect(captures.started).toHaveLength(2); + captures.started[1].resolve('second'); + await expect(second).resolves.toBe('second'); + }); + + it('gives every caller that arrives during a capture one shared capture that starts after it settles', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + const joiners: Promise[] = [1, 2, 3].map(() => + coalescer.captureAsync(scope, 'key', captures.captureAsync) + ); + await flushAsync(); + expect(captures.started).toHaveLength(1); + + captures.started[0].resolve('before the joiners asked'); + await expect(first).resolves.toBe('before the joiners asked'); + await flushAsync(); + expect(captures.started).toHaveLength(2); + + captures.started[1].resolve('after the joiners asked'); + await expect(Promise.all(joiners)).resolves.toEqual([ + 'after the joiners asked', + 'after the joiners asked', + 'after the joiners asked' + ]); + expect(captures.started).toHaveLength(2); + }); + + it('never gives a caller a capture that started before the caller asked', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + const results: Promise<[number, string]>[] = []; + const ask = (): void => { + const startedBeforeAsking: number = captures.started.length; + results.push( + coalescer + .captureAsync(scope, 'key', captures.captureAsync) + .then((value: string): [number, string] => [startedBeforeAsking, value]) + ); + }; + + ask(); // starts capture 0 + ask(); // waits for capture 1 + captures.started[0].resolve('0'); + await flushAsync(); + ask(); // capture 1 is running, so this waits for capture 2 + ask(); + captures.started[1].resolve('1'); + await flushAsync(); + captures.started[2].resolve('2'); + await flushAsync(); + ask(); // nothing is running, so this starts capture 3 + captures.started[3].resolve('3'); + + const settled: [number, string][] = await Promise.all(results); + expect(settled).toEqual([ + [0, '0'], + [1, '1'], + [2, '2'], + [2, '2'], + [3, '3'] + ]); + for (const [startedBeforeAsking, value] of settled) { + expect(Number(value)).toBeGreaterThanOrEqual(startedBeforeAsking); + } + }); + + it('rejects the callers of a failed capture and still runs the next capture for callers that arrived during it', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + const failed: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + const joiner: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + captures.started[0].reject(new Error('a configuration file is being rewritten')); + await expect(failed).rejects.toThrow('a configuration file is being rewritten'); + await flushAsync(); + expect(captures.started).toHaveLength(2); + captures.started[1].resolve('rewritten'); + await expect(joiner).resolves.toBe('rewritten'); + }); + + it('returns a rejected promise when the capture function throws synchronously', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const scope: object = {}; + const result: Promise = coalescer.captureAsync(scope, 'key', () => { + throw new Error('cannot start'); + }); + await expect(result).rejects.toThrow('cannot start'); + await expect(coalescer.captureAsync(scope, 'key', async () => 'started')).resolves.toBe('started'); + }); + + it('never shares a capture between different scopes or keys', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const firstScope: object = {}; + const secondScope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + const results: Promise[] = [ + coalescer.captureAsync(firstScope, 'a', captures.captureAsync), + coalescer.captureAsync(firstScope, 'b', captures.captureAsync), + coalescer.captureAsync(secondScope, 'a', captures.captureAsync) + ]; + expect(captures.started).toHaveLength(3); + for (const capture of captures.started) capture.resolve(String(capture.index)); + await expect(Promise.all(results)).resolves.toEqual(['0', '1', '2']); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts b/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts new file mode 100644 index 0000000000..2f8b3b6a27 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts @@ -0,0 +1,83 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('@microsoft/rush-lib', () => { + const actual: typeof import('@microsoft/rush-lib') = jest.requireActual('@microsoft/rush-lib'); + return { + ...actual, + captureProjectConfigurationFingerprintAsync: jest.fn(actual.captureProjectConfigurationFingerprintAsync) + }; +}); + +import * as rushLib from '@microsoft/rush-lib'; + +import { FreshCaptureCoalescer } from '../FreshCaptureCoalescer'; +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { setDaemonPolicy } from './WarmGenerationTestUtilities'; + +jest.setTimeout(60_000); + +const WAIT_TIMEOUT_MS: number = 20_000; +const captureMock: jest.MockedFunction = + jest.mocked(rushLib.captureProjectConfigurationFingerprintAsync); + +async function waitForAsync(description: string, condition: () => boolean): Promise { + const deadline: number = Date.now() + WAIT_TIMEOUT_MS; + while (!condition()) { + if (Date.now() > deadline) throw new Error(`Timed out waiting until ${description}`); + await new Promise((resolve) => setTimeout(resolve, 10)); + } +} + +it('serves concurrent warm builds from one project configuration capture that started after they arrived', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => + setDaemonPolicy(created, {}) + ); + let release: () => void = () => {}; + try { + await fixture.buildSuccessfullyAsync(); + await fixture.buildSuccessfullyAsync(); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + const generation: number = fixture.host.workspaceGeneration; + + const requests: jest.SpyInstance = jest.spyOn(FreshCaptureCoalescer.prototype, 'captureAsync'); + // The project configuration coalescer uses one key per workspace configuration. + const projectCaptureRequests = (): number => + requests.mock.calls.filter(([, key]: unknown[]) => key === '').length; + captureMock.mockClear(); + const gate: Promise = new Promise((resolve) => (release = resolve)); + const actual: typeof rushLib.captureProjectConfigurationFingerprintAsync = jest.requireActual< + typeof rushLib + >('@microsoft/rush-lib').captureProjectConfigurationFingerprintAsync; + captureMock.mockImplementationOnce(async (...args) => { + await gate; + return await actual(...args); + }); + + const first: ReturnType = fixture.buildAsync(); + await waitForAsync('the first build is capturing', () => captureMock.mock.calls.length === 1); + const joiners: ReturnType[] = [ + fixture.buildAsync(), + fixture.buildAsync(), + fixture.buildAsync() + ]; + await waitForAsync( + 'every build asked for a capture', + () => Math.max(projectCaptureRequests(), captureMock.mock.calls.length) === 4 + ); + expect(captureMock).toHaveBeenCalledTimes(1); + release(); + + for (const exchange of await Promise.all([first, ...joiners])) { + expect(exchange.terminal).toMatchObject({ payload: { exitCode: 0 } }); + } + // The joiners arrived while the first capture was running, so they shared one that started after them. + expect(captureMock).toHaveBeenCalledTimes(2); + expect(fixture.host.workspaceGeneration).toBe(generation); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + requests.mockRestore(); + } finally { + release(); + await fixture[Symbol.asyncDispose](); + } +}); From d5ccd5b7e024e2cf207629e328c566ac32300c60 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 15:50:25 +0000 Subject: [PATCH 023/265] [rush-lib] Skip cache writes when input files change while the inputs snapshot is taken Swarm integration step 8; original commit bc03023091 (merge of swarm/r06 at 29f86b181c). Refs: task 77. ch01 GATE OK board 1019 (tree 80c9659bac); claim board 698, CONFIRMED by o04 board 738, o03 board 742, o05 board 748. Commits folded into this step (2): - a644819daf [rush-lib] Don't write a build cache entry for inputs saved while the snapshot was taken - 29f86b181c [rush-lib] Make the parameter identity test independent of the host's RUSH_PARALLELISM Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...napshot-window-input-guard_2026-09-28.json | 11 ++ common/reviews/api/rush-lib.api.md | 1 + ...asedCommandEngineParameterIdentity.test.ts | 2 + .../src/logic/ProjectChangeAnalyzer.ts | 5 +- .../src/logic/incremental/InputsSnapshot.ts | 22 +++- .../incremental/test/InputsSnapshot.test.ts | 8 ++ .../operations/CacheableOperationPlugin.ts | 103 +++++++++++---- .../operations/InputFilesStatSignature.ts | 66 +++++++++- .../test/CacheableOperationPlugin.test.ts | 119 ++++++++++++++++-- .../test/InputFilesStatSignature.test.ts | 80 +++++++++++- .../logic/test/ProjectChangeAnalyzer.test.ts | 14 +++ 11 files changed, 387 insertions(+), 44 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-snapshot-window-input-guard_2026-09-28.json diff --git a/common/changes/@microsoft/rush/rushd-snapshot-window-input-guard_2026-09-28.json b/common/changes/@microsoft/rush/rushd-snapshot-window-input-guard_2026-09-28.json new file mode 100644 index 0000000000..d6ba28db8e --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-snapshot-window-input-guard_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Fix an issue where a tracked input file that was saved while Rush was reading the working tree could produce a build cache entry whose outputs don't match its cache key. Such files are now hashed again before the entry is written, and the write is skipped if the hash no longer matches. Adds `IInputsSnapshot.workingTreeReadStartTimeMs`.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 4e3184e502..94c081b8f9 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -629,6 +629,7 @@ export interface IInputsSnapshot { readonly hashes: ReadonlyMap; readonly hasUncommittedChanges: boolean; readonly rootDirectory: string; + readonly workingTreeReadStartTimeMs?: number; } // @public diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts index 49b1c296e1..e0799d6a68 100644 --- a/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts @@ -26,6 +26,8 @@ describe(`${PhasedCommandEngine.name} parameter identity`, () => { return await PhasedCommandEngine.parseAsync({ argv, cwd: folder, + // Otherwise environment-backed defaults such as RUSH_PARALLELISM come from the test host. + environment: {}, rushConfiguration, terminalProvider: new NoOpTerminalProvider() }); diff --git a/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts b/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts index 960c313db3..14dcadd0ba 100644 --- a/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts +++ b/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts @@ -424,6 +424,8 @@ export class ProjectChangeAnalyzer { } return async function tryGetSnapshotAsync(): Promise { + // Recorded before Git reads the working tree: a file saved after this may be newer than its hash. + const workingTreeReadStartTimeMs: number = Date.now(); try { const [{ files: hashes, symlinks, hasUncommittedChanges }, additionalFiles] = await Promise.all([ getDetailedRepoStateAsync(rootDirectory, additionalRelativePathsToHash, gitPath, filterPath), @@ -459,7 +461,8 @@ export class ProjectChangeAnalyzer { hasUncommittedChanges, lookupByPath, projectMap, - rootDir: rootDirectory + rootDir: rootDirectory, + workingTreeReadStartTimeMs }); } catch (e) { // If getRepoState fails, don't fail the whole build. Treat this case as if we don't know anything about diff --git a/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts b/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts index f48fd9b356..14949e73d4 100644 --- a/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts +++ b/libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts @@ -123,6 +123,10 @@ export interface IInputsSnapshotParameters { * The directory that all relative paths are relative to. */ rootDir: string; + /** + * {@inheritdoc IInputsSnapshot.workingTreeReadStartTimeMs} + */ + workingTreeReadStartTimeMs?: number; } const { hashDelimiter } = RushConstants; @@ -149,6 +153,16 @@ export interface IInputsSnapshot { */ readonly hasUncommittedChanges: boolean; + /** + * The time, in milliseconds since the epoch, at which this snapshot began reading the state of the working tree, + * if known. + * + * @remarks + * A tracked file that was modified at or after this time may be newer than its hash in `hashes`, because Git + * may have read the file before it was saved. + */ + readonly workingTreeReadStartTimeMs?: number; + /** * Gets the map of file paths to Git hashes that will be used to compute the local state hash of the operation. * Exposed separately from the final state hash to facilitate detailed change detection. @@ -200,6 +214,10 @@ export class InputsSnapshot implements IInputsSnapshot { * {@inheritdoc IInputsSnapshot.rootDirectory} */ public readonly rootDirectory: string; + /** + * {@inheritdoc IInputsSnapshot.workingTreeReadStartTimeMs} + */ + public readonly workingTreeReadStartTimeMs: number | undefined; /** * The metadata for each project. This is a superset of the information in `projectMap` and includes caching of queries. @@ -239,7 +257,8 @@ export class InputsSnapshot implements IInputsSnapshot { hasUncommittedChanges, lookupByPath, nodeVersion = process.version, - rootDir + rootDir, + workingTreeReadStartTimeMs } = params; const projectMetadataMap: Map< IRushConfigurationProjectForSnapshot, @@ -299,6 +318,7 @@ export class InputsSnapshot implements IInputsSnapshot { this.hashes = hashes; this.hasUncommittedChanges = hasUncommittedChanges; this.rootDirectory = rootDir; + this.workingTreeReadStartTimeMs = workingTreeReadStartTimeMs; } /** diff --git a/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts b/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts index 901fad97cc..c0b85794b3 100644 --- a/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts +++ b/libraries/rush-lib/src/logic/incremental/test/InputsSnapshot.test.ts @@ -49,6 +49,14 @@ describe(InputsSnapshot.name, () => { return { project, input }; } + it('Exposes the time at which it began reading the working tree', () => { + const { options } = getTestConfig(); + expect(new InputsSnapshot(options).workingTreeReadStartTimeMs).toBeUndefined(); + expect( + new InputsSnapshot({ ...options, workingTreeReadStartTimeMs: 1234 }).workingTreeReadStartTimeMs + ).toBe(1234); + }); + describe(InputsSnapshot.prototype.getTrackedFileHashesForOperation.name, () => { it('Handles trivial input', () => { const { project, input } = getTrivialSnapshot(); diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index 6213db3ede..55037496af 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -5,6 +5,7 @@ import * as crypto from 'node:crypto'; import * as path from 'node:path'; import { InternalError, NewlineKind, Sort, Executable } from '@rushstack/node-core-library'; +import { hashFilesAsync } from '@rushstack/package-deps-hash'; import { CollatedTerminal, type CollatedWriter } from '@rushstack/stream-collator'; import { DiscardStdoutTransform, @@ -80,9 +81,11 @@ export interface IOperationBuildCacheContext { isCacheReadAttempted: boolean; // The on-disk state of the tracked input files whose hashes produced the cache key, captured right after - // the iteration's inputs snapshot. Used to refuse cache writes if the inputs changed while the operation - // was executing. + // the iteration's inputs snapshot. Used to refuse cache writes if the inputs changed while the snapshot was + // being taken or while the operation was executing. inputFilesState?: IInputFilesState; + // The hashes of the tracked input files in the iteration's inputs snapshot + inputFileHashes?: ReadonlyMap; } export interface ICacheableOperationPluginOptions { @@ -122,24 +125,54 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { this.#options = options; } + #getGitPath(): string | undefined { + if (!this.#gitPathResolved) { + this.#gitPath = EnvironmentConfiguration.gitBinaryPath || Executable.tryResolve('git'); + this.#gitPathResolved = true; + } + return this.#gitPath; + } + #isNewInput( newEntryPaths: ReadonlyArray, rootDirectory: string, projectFolder: string, outputFolderNames: ReadonlyArray ): boolean { - if (!this.#gitPathResolved) { - this.#gitPath = EnvironmentConfiguration.gitBinaryPath || Executable.tryResolve('git'); - this.#gitPathResolved = true; - } - if (!this.#gitPath) { + const gitPath: string | undefined = this.#getGitPath(); + if (!gitPath) { // Without Git we cannot tell whether the new entries are ignored, so assume they are inputs. return true; } const outputFolderPaths: string[] = outputFolderNames.map((folderName: string) => path.resolve(projectFolder, folderName) ); - return hasUntrackedGitFiles(this.#gitPath, rootDirectory, newEntryPaths, outputFolderPaths); + return hasUntrackedGitFiles(gitPath, rootDirectory, newEntryPaths, outputFolderPaths); + } + + /** + * Returns true if the current Git hash of any of the specified files differs from its hash in the inputs + * snapshot. If the files cannot be hashed, conservatively returns true. + */ + async #haveSnapshotHashesChangedAsync( + rootDirectory: string, + filePaths: ReadonlyArray, + snapshotHashes: ReadonlyMap | undefined + ): Promise { + const gitPath: string | undefined = this.#getGitPath(); + if (!gitPath || !snapshotHashes) { + return true; + } + try { + for (const [filePath, hash] of await hashFilesAsync(rootDirectory, filePaths, gitPath)) { + if (snapshotHashes.get(filePath) !== hash) { + return true; + } + } + return false; + } catch { + return true; + } } public apply(hooks: PhasedCommandHooks): void { @@ -222,7 +255,11 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { const inputFilesState: IInputFilesState | undefined = cacheWriteEnabled && !cacheDisabledReason && record.enabled - ? captureInputFilesState(inputsSnapshot.rootDirectory, fileHashes.keys()) + ? captureInputFilesState( + inputsSnapshot.rootDirectory, + fileHashes.keys(), + inputsSnapshot.workingTreeReadStartTimeMs + ) : undefined; const buildCacheContext: IOperationBuildCacheContext = { @@ -242,7 +279,8 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { }), cacheRestored: false, isCacheReadAttempted: false, - inputFilesState + inputFilesState, + inputFileHashes: inputFilesState ? fileHashes : undefined }; // Upstream runners may mutate the property of build cache context for downstream runners this.#buildCacheContextByOperation.set(operation, buildCacheContext); @@ -587,26 +625,41 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { if (!setCacheEntryPromise && taskIsSuccessful && isCacheWriteAllowed && operationBuildCache) { setCacheEntryPromise = () => operationBuildCache.trySetCacheEntryAsync(buildCacheTerminal); } - const { inputFilesState } = buildCacheContext; - if ( - !cacheRestored && - isCacheWriteAllowed && - inputFilesState && - haveInputFilesChanged(inputFilesState, (newEntryPaths: ReadonlyArray) => - this.#isNewInput( - newEntryPaths, + const { inputFilesState, inputFileHashes } = buildCacheContext; + let inputFilesChangedMessage: string | undefined; + if (!cacheRestored && isCacheWriteAllowed && inputFilesState) { + // If Git hashed a file that was saved during the snapshot before it was saved, the outputs were + // built from newer content than the cache key describes. + const haveSnapshotHashesChanged: boolean = + inputFilesState.filesChangedDuringSnapshot.length > 0 && + (await this.#haveSnapshotHashesChangedAsync( inputFilesState.rootDirectory, - project.projectFolder, - buildCacheContext.outputFolderNames + inputFilesState.filesChangedDuringSnapshot, + inputFileHashes + )); + const { outputFolderNames } = buildCacheContext; + if ( + haveInputFilesChanged(inputFilesState, (newEntryPaths: ReadonlyArray) => + this.#isNewInput( + newEntryPaths, + inputFilesState.rootDirectory, + project.projectFolder, + outputFolderNames + ) ) - ) - ) { + ) { + inputFilesChangedMessage = + 'Input files changed while this operation was executing; not writing a build cache entry.'; + } else if (haveSnapshotHashesChanged) { + inputFilesChangedMessage = + 'Input files changed while the inputs snapshot was being taken; not writing a build cache entry.'; + } + } + if (inputFilesChangedMessage) { // The cache key was derived from the iteration's inputs snapshot. Storing outputs produced from // edited inputs under that key would poison the cache for every consumer of the entry. // Consumers' cache keys also embed this operation's pre-edit state, so block their writes too. - buildCacheTerminal.writeLine( - 'Input files changed while this operation was executing; not writing a build cache entry.' - ); + buildCacheTerminal.writeLine(inputFilesChangedMessage); buildCacheContext.isCacheWriteAllowed = false; setCacheEntryPromise = undefined; } diff --git a/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts b/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts index aef3eaff54..1e32a0da61 100644 --- a/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts +++ b/libraries/rush-lib/src/logic/operations/InputFilesStatSignature.ts @@ -7,6 +7,18 @@ import * as path from 'node:path'; import { Executable } from '@rushstack/node-core-library'; +/** + * How far outside of the snapshot window a file time may be and still count as a save during the window. + * File times can trail `Date.now()` by a clock tick (about 16 ms on Windows), or by up to 2 seconds on file systems + * with coarse time stamps (FAT). A file that is flagged by mistake only costs a `git hash-object` call. + */ +export const FILE_TIME_TOLERANCE_MS: number = 2000; +const NANOSECONDS_PER_MILLISECOND: bigint = BigInt(1000000); + +function millisecondsToNanoseconds(timeMs: number): bigint { + return BigInt(Math.floor(timeMs)) * NANOSECONDS_PER_MILLISECOND; +} + /** * The on-disk state of an operation's tracked input files, captured right after the inputs snapshot * (from which the operation's build cache key is derived) was taken. @@ -29,6 +41,13 @@ export interface IInputFilesState { * Used to detect files (or folders) that were created after the snapshot was taken. */ readonly folderEntries: ReadonlyMap>; + /** + * The tracked input file paths, as passed to `captureInputFilesState`, whose modification or status change time + * falls between the time that the inputs snapshot began reading the working tree and the time that this state + * was captured. Git may have hashed such a file before it was saved, so its hash in the snapshot (from which the + * build cache key was derived) may be stale even though its stats do not change again. + */ + readonly filesChangedDuringSnapshot: ReadonlyArray; } /** @@ -42,14 +61,24 @@ export type IsNewInputCallback = (newEntryPaths: ReadonlyArray) => boole * Missing files are included in the signature, so deleting or creating a listed file also changes it. */ export function getInputFilesStatSignature(filePaths: Iterable): string { + return hashInputFilesStats(filePaths); +} + +function hashInputFilesStats( + filePaths: Iterable, + onStats?: (index: number, stats: fs.BigIntStats) => void +): string { const hasher: crypto.Hash = crypto.createHash('sha1'); + let index: number = 0; for (const filePath of filePaths) { const stats: fs.BigIntStats | undefined = fs.statSync(filePath, { bigint: true, throwIfNoEntry: false }); if (stats) { hasher.update(`${filePath}\0${stats.size}\0${stats.mtimeNs}\0${stats.ino}\n`); + onStats?.(index, stats); } else { hasher.update(`${filePath}\0missing\n`); } + index++; } return hasher.digest('hex'); } @@ -69,15 +98,20 @@ function tryReadFolderEntries(folderPath: string): Set | undefined { * @param inputFilePaths - The tracked input file paths. Relative paths are resolved against `rootDirectory`; * absolute paths (e.g. `dependsOnAdditionalFiles` outside of the repository) are stat'ed but their folders * are not watched for new entries. + * @param snapshotStartTimeMs - When the inputs snapshot began reading the working tree + * (`IInputsSnapshot.workingTreeReadStartTimeMs`), if known. Used to compute `filesChangedDuringSnapshot`. */ export function captureInputFilesState( rootDirectory: string, - inputFilePaths: Iterable + inputFilePaths: Iterable, + snapshotStartTimeMs?: number ): IInputFilesState { + const originalFilePaths: string[] = []; const filePaths: string[] = []; const folderEntries: Map> = new Map(); for (const inputFilePath of inputFilePaths) { const absolutePath: string = path.resolve(rootDirectory, inputFilePath); + originalFilePaths.push(inputFilePath); filePaths.push(absolutePath); if (!path.isAbsolute(inputFilePath)) { const folderPath: string = path.dirname(absolutePath); @@ -86,7 +120,33 @@ export function captureInputFilesState( } } } - return { rootDirectory, filePaths, statSignature: getInputFilesStatSignature(filePaths), folderEntries }; + const windowStartNs: bigint | undefined = + snapshotStartTimeMs === undefined + ? undefined + : millisecondsToNanoseconds(snapshotStartTimeMs - FILE_TIME_TOLERANCE_MS); + // For each file with a time at or after the start of the window, the earliest such time + const earliestTimeInWindowNsByIndex: Map = new Map(); + const statSignature: string = hashInputFilesStats(filePaths, (index: number, stats: fs.BigIntStats) => { + if (windowStartNs === undefined) { + return; + } + for (const timeNs of [stats.mtimeNs, stats.ctimeNs]) { + const earliestTimeNs: bigint | undefined = earliestTimeInWindowNsByIndex.get(index); + if (timeNs >= windowStartNs && (earliestTimeNs === undefined || timeNs < earliestTimeNs)) { + earliestTimeInWindowNsByIndex.set(index, timeNs); + } + } + }); + // The window ends after the files were stat'ed, so that a save during the loop is inside it. A later file time + // (e.g. an mtime set in the future) is not a save during the window. + const windowEndNs: bigint = millisecondsToNanoseconds(Date.now() + FILE_TIME_TOLERANCE_MS); + const filesChangedDuringSnapshot: string[] = []; + for (const [index, timeNs] of earliestTimeInWindowNsByIndex) { + if (timeNs <= windowEndNs) { + filesChangedDuringSnapshot.push(originalFilePaths[index]); + } + } + return { rootDirectory, filePaths, statSignature, folderEntries, filesChangedDuringSnapshot }; } /** @@ -148,4 +208,4 @@ export function hasUntrackedGitFiles( return true; } return result.stdout.length > 0; -} \ No newline at end of file +} diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts index a9b3ad395a..ffdee0680e 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts @@ -42,7 +42,19 @@ jest.mock('../OperationMetadataManager', () => { jest.mock('../../buildCache/OperationBuildCache', () => ({ OperationBuildCache: { forOperation: jest.fn() } })); +jest.mock('@rushstack/package-deps-hash', () => { + const actual: typeof import('@rushstack/package-deps-hash') = jest.requireActual( + '@rushstack/package-deps-hash' + ); + return { ...actual, hashFilesAsync: jest.fn(actual.hashFilesAsync) }; +}); +import * as crypto from 'node:crypto'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { hashFilesAsync } from '@rushstack/package-deps-hash'; import { MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import type { IPhase } from '../../../api/CommandLineConfiguration'; @@ -60,6 +72,7 @@ import { OperationStatus } from '../OperationStatus'; import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; import type { IExecutionResult } from '../IOperationExecutionResult'; import type { OperationExecutionRecord } from '../OperationExecutionRecord'; +import { FILE_TIME_TOLERANCE_MS } from '../InputFilesStatSignature'; const mockPhase: IPhase = { name: 'phase', @@ -99,18 +112,21 @@ interface ITestGraph { graph: OperationGraph; operations: Map; localHashes: Map; + // The tracked input file hashes of each operation in the inputs snapshot, by name + trackedFileHashes: Map>; executions: string[]; cacheWrites: string[]; - executeAsync(): Promise; + executeAsync(workingTreeReadStartTimeMs?: number): Promise; } /** * Creates a linear chain of cacheable operations: names[0] <- names[1] <- ... (each depends on the previous). */ -async function createTestGraphAsync(names: string[]): Promise { +async function createTestGraphAsync(names: string[], rootDirectory: string = '/repo'): Promise { const executions: string[] = []; const cacheWrites: string[] = []; const localHashes: Map = new Map(); + const trackedFileHashes: Map> = new Map(); const operations: Map = new Map(); const projectConfigurations: Map = new Map(); @@ -118,7 +134,7 @@ async function createTestGraphAsync(names: string[]): Promise { for (const name of names) { const project: RushConfigurationProject = { packageName: name, - projectFolder: `/repo/${name}` + projectFolder: `${rootDirectory}/${name}` } as unknown as RushConfigurationProject; projectConfigurations.set(project, { getCacheDisabledReason: () => undefined @@ -176,33 +192,51 @@ async function createTestGraphAsync(names: string[]): Promise { projectConfigurations } as unknown as IOperationGraphContext); - const inputsSnapshot: IInputsSnapshot = { - hashes: new Map(), - rootDirectory: '/repo', - hasUncommittedChanges: false, - getTrackedFileHashesForOperation: () => new Map(), - getOperationOwnStateHash: (project: RushConfigurationProject) => localHashes.get(project.packageName)! - }; - return { graph, operations, localHashes, + trackedFileHashes, executions, cacheWrites, - executeAsync: async () => { + executeAsync: async (workingTreeReadStartTimeMs?: number) => { executions.length = 0; cacheWrites.length = 0; + const inputsSnapshot: IInputsSnapshot = { + hashes: new Map(), + rootDirectory, + hasUncommittedChanges: false, + workingTreeReadStartTimeMs, + getTrackedFileHashesForOperation: (project: RushConfigurationProject) => + trackedFileHashes.get(project.packageName) ?? new Map(), + getOperationOwnStateHash: (project: RushConfigurationProject) => localHashes.get(project.packageName)! + }; return await graph.executeAsync({ inputsSnapshot }); } }; } +function getGitBlobHash(content: string): string { + return crypto + .createHash('sha1') + .update(`blob ${Buffer.byteLength(content)}\0${content}`) + .digest('hex'); +} + +function getLatestFileTimeMs(filePath: string): number { + const { mtimeNs, ctimeNs } = fs.statSync(filePath, { bigint: true }); + return Number((mtimeNs > ctimeNs ? mtimeNs : ctimeNs) / BigInt(1000000)); +} + function getStatus(testGraph: ITestGraph, result: IExecutionResult, name: string): OperationStatus { return (result.operationResults.get(testGraph.operations.get(name)!) as OperationExecutionRecord).status; } describe(CacheableOperationPlugin.name, () => { + beforeEach(() => { + jest.mocked(hashFilesAsync).mockClear(); + }); + it('writes cache entries for all operations in a cold iteration', async () => { const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); @@ -300,4 +334,65 @@ describe(CacheableOperationPlugin.name, () => { expect(testGraph.executions).toEqual(['a', 'c']); expect(testGraph.cacheWrites).toEqual(['a']); }); + + describe('input files saved while the inputs snapshot was being taken', () => { + const inputFile: string = 'a/src/index.ts'; + let rootDirectory: string; + + beforeEach(() => { + rootDirectory = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-cacheable-'))); + fs.mkdirSync(path.join(rootDirectory, 'a', 'src'), { recursive: true }); + fs.writeFileSync(path.join(rootDirectory, inputFile), 'export const a = 2;'); + }); + + afterEach(() => { + fs.rmSync(rootDirectory, { recursive: true, force: true }); + }); + + it('does not write cache entries if Git hashed the file before it was saved', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b'], rootDirectory); + testGraph.trackedFileHashes.set('a', new Map([[inputFile, getGitBlobHash('export const a = 1;')]])); + + await testGraph.executeAsync(Date.now()); + + expect(testGraph.executions).toEqual(['a', 'b']); + // "b" consumed outputs of "a" that do not match the state hash of "a", so it must not write either. + expect(testGraph.cacheWrites).toEqual([]); + expect(jest.mocked(hashFilesAsync)).toHaveBeenCalledTimes(1); + expect(jest.mocked(hashFilesAsync).mock.calls[0].slice(0, 2)).toEqual([rootDirectory, [inputFile]]); + + // The next snapshot hashes the saved content, so the outputs are rebuilt and cached. + testGraph.trackedFileHashes.set('a', new Map([[inputFile, getGitBlobHash('export const a = 2;')]])); + testGraph.localHashes.set('a', 'a-v2'); + await testGraph.executeAsync( + getLatestFileTimeMs(path.join(rootDirectory, inputFile)) + FILE_TIME_TOLERANCE_MS + 1 + ); + + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual(['a', 'b']); + }); + + it('writes cache entries if Git hashed the file after it was saved', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b'], rootDirectory); + testGraph.trackedFileHashes.set('a', new Map([[inputFile, getGitBlobHash('export const a = 2;')]])); + + await testGraph.executeAsync(Date.now()); + + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual(['a', 'b']); + expect(jest.mocked(hashFilesAsync)).toHaveBeenCalledTimes(1); + }); + + it('does not hash files that were saved before the snapshot started', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b'], rootDirectory); + testGraph.trackedFileHashes.set('a', new Map([[inputFile, getGitBlobHash('export const a = 2;')]])); + + await testGraph.executeAsync( + getLatestFileTimeMs(path.join(rootDirectory, inputFile)) + FILE_TIME_TOLERANCE_MS + 1 + ); + + expect(testGraph.cacheWrites).toEqual(['a', 'b']); + expect(jest.mocked(hashFilesAsync)).not.toHaveBeenCalled(); + }); + }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts b/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts index 0e0839c269..ca626bcc18 100644 --- a/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/InputFilesStatSignature.test.ts @@ -8,6 +8,7 @@ import * as path from 'node:path'; import { captureInputFilesState, + FILE_TIME_TOLERANCE_MS, getNewFolderEntries, hasUntrackedGitFiles, haveInputFilesChanged, @@ -22,9 +23,14 @@ describe('InputFilesStatSignature', () => { let noNewInputs: jest.Mock]>; function capture(...absolutePaths: string[]): IInputFilesState { + return captureAt(undefined, ...absolutePaths); + } + + function captureAt(snapshotStartTimeMs: number | undefined, ...absolutePaths: string[]): IInputFilesState { return captureInputFilesState( tempFolder, - absolutePaths.map((filePath: string) => path.relative(tempFolder, filePath)) + absolutePaths.map((filePath: string) => path.relative(tempFolder, filePath)), + snapshotStartTimeMs ); } @@ -105,6 +111,76 @@ describe('InputFilesStatSignature', () => { }); }); + describe('filesChangedDuringSnapshot', () => { + const nanosecondsPerMillisecond: bigint = BigInt(1000000); + + function getLatestFileTimeMs(filePath: string): number { + const { mtimeNs, ctimeNs } = fs.statSync(filePath, { bigint: true }); + return Number((mtimeNs > ctimeNs ? mtimeNs : ctimeNs) / nanosecondsPerMillisecond); + } + + function getStatusChangeTimeMs(filePath: string): number { + return Number(fs.statSync(filePath, { bigint: true }).ctimeNs / nanosecondsPerMillisecond); + } + + it('is empty if the snapshot start time is unknown', () => { + expect(capture(fileA, fileB).filesChangedDuringSnapshot).toEqual([]); + }); + + it('lists a file modified at or after the snapshot start, within the tolerance', () => { + const fileTimeMs: number = getLatestFileTimeMs(fileA); + expect(captureAt(fileTimeMs + FILE_TIME_TOLERANCE_MS, fileA).filesChangedDuringSnapshot).toEqual([ + path.relative(tempFolder, fileA) + ]); + expect(captureAt(fileTimeMs + FILE_TIME_TOLERANCE_MS + 1, fileA).filesChangedDuringSnapshot).toEqual( + [] + ); + }); + + it('lists only the files that changed after the snapshot start', () => { + const snapshotStartTimeMs: number = getLatestFileTimeMs(fileB) + FILE_TIME_TOLERANCE_MS + 1; + // Wait for the file system clock to pass the start of the window + const deadlineMs: number = Date.now() + 10000; + do { + fs.writeFileSync(fileB, `export const b = ${Date.now()}; // saved during the snapshot`); + } while ( + getLatestFileTimeMs(fileB) < snapshotStartTimeMs - FILE_TIME_TOLERANCE_MS && + Date.now() < deadlineMs + ); + + expect(captureAt(snapshotStartTimeMs, fileA, fileB).filesChangedDuringSnapshot).toEqual([ + path.relative(tempFolder, fileB) + ]); + }); + + it('uses the status change time if the modification time was set back', () => { + const past: Date = new Date(Date.now() - 3600 * 1000); + fs.utimesSync(fileA, past, past); + const statusChangeTimeMs: number = getStatusChangeTimeMs(fileA); + expect(fs.statSync(fileA).mtimeMs).toBeLessThan(statusChangeTimeMs - FILE_TIME_TOLERANCE_MS); + + expect(captureAt(statusChangeTimeMs, fileA).filesChangedDuringSnapshot).toEqual([ + path.relative(tempFolder, fileA) + ]); + }); + + it('ignores file times after the end of the window', () => { + const future: Date = new Date(Date.now() + 3600 * 1000); + fs.utimesSync(fileA, future, future); + // Only the modification time, which is in the future, is at or after the start of the window + const snapshotStartTimeMs: number = getStatusChangeTimeMs(fileA) + FILE_TIME_TOLERANCE_MS + 1; + + expect(captureAt(snapshotStartTimeMs, fileA).filesChangedDuringSnapshot).toEqual([]); + }); + + it('skips missing files', () => { + const missingFile: string = path.join(srcFolder, 'missing.ts'); + expect(captureAt(0, missingFile, fileA).filesChangedDuringSnapshot).toEqual([ + path.relative(tempFolder, fileA) + ]); + }); + }); + describe(hasUntrackedGitFiles.name, () => { const gitPath: string = 'git'; @@ -144,4 +220,4 @@ describe('InputFilesStatSignature', () => { expect(hasUntrackedGitFiles(gitPath, tempFolder, [path.join(srcFolder, 'nested')], [])).toBe(true); }); }); -}); \ No newline at end of file +}); diff --git a/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts b/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts index c9138b3c85..8bb271d8bd 100644 --- a/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts +++ b/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts @@ -25,6 +25,7 @@ const mockHashes: Map = new Map([ // Mock function for customizing repo changes in each test const mockGetRepoChanges: jest.MockedFunction = jest.fn(); +const mockOnGetDetailedRepoState: jest.Mock = jest.fn(); jest.mock(`@rushstack/package-deps-hash`, () => { return { @@ -32,6 +33,7 @@ jest.mock(`@rushstack/package-deps-hash`, () => { return dir; }, getDetailedRepoStateAsync(): IDetailedRepoState { + mockOnGetDetailedRepoState(); return { hasSubmodules: false, hasUncommittedChanges: false, @@ -152,8 +154,17 @@ describe(ProjectChangeAnalyzer.name, () => { const terminal: Terminal = new Terminal(terminalProvider); const mockSnapshotValue: {} = {}; mockSnapshot.mockImplementation(() => mockSnapshotValue); + let repoStateReadTimeMs: number | undefined; + mockOnGetDetailedRepoState.mockImplementationOnce(() => { + repoStateReadTimeMs = Date.now(); + // Make sure that a time recorded after the working tree was read is later than this one + while (Date.now() <= repoStateReadTimeMs + 1) { + // Busy wait + } + }); const snapshotProvider: GetInputsSnapshotAsyncFn | undefined = await projectChangeAnalyzer._tryGetSnapshotProviderAsync(new Map(), terminal); + const beforeSnapshotTimeMs: number = Date.now(); const snapshot: IInputsSnapshot | undefined = await snapshotProvider?.(); expect(snapshot).toBe(mockSnapshotValue); @@ -167,6 +178,9 @@ describe(ProjectChangeAnalyzer.name, () => { expect(mockInput.hashes).toEqual(mockHashes); expect(mockInput.rootDir).toEqual(rootDir); expect(mockInput.additionalHashes).toEqual(new Map()); + // The start time is recorded before Git reads the working tree + expect(mockInput.workingTreeReadStartTimeMs).toBeGreaterThanOrEqual(beforeSnapshotTimeMs); + expect(mockInput.workingTreeReadStartTimeMs).toBeLessThanOrEqual(repoStateReadTimeMs!); }); }); From ef51bf74247a8a8e4954bd0116d8c36b3a15a8d5 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 16:37:37 +0000 Subject: [PATCH 024/265] [rush-daemon] don't let a request received during a batch's reconcile join that batch Swarm integration step 9; original commit d0ed461613 (merge of swarm/r05 at 199c69347c). Refs: board 371, task 72. ch01 GATE OK board 1152 (gated tree 5c2b6b6cbf). Commits folded into this step (1): - 199c69347c [rush-daemon] Don't let a request received during a batch's reconcile join that batch (board 371) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...swarm-r05-stale-join_2026-09-28-14-10.json | 10 ++ common/reviews/api/rush-daemon.api.md | 3 +- .../src/DaemonRequestDispatcher.ts | 10 +- .../rush-daemon/src/PhasedRequestRouter.ts | 24 ++- .../src/WorkspaceRequestLifecycle.ts | 2 + .../src/test/PhasedRequestBatching.test.ts | 167 ++++++++++++++++++ .../test/PhasedRequestCancellation.test.ts | 58 +++++- .../ProductionDaemonRequestResolver.test.ts | 57 ++++-- 8 files changed, 303 insertions(+), 28 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-stale-join_2026-09-28-14-10.json diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-stale-join_2026-09-28-14-10.json b/common/changes/@rushstack/rush-daemon/swarm-r05-stale-join_2026-09-28-14-10.json new file mode 100644 index 0000000000..0f55dd4c37 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-stale-join_2026-09-28-14-10.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A shared build batch no longer accepts a compatible request that the daemon received after the batch started to reconcile its inputs. That request runs in a later batch that reconciles again, instead of on a snapshot read before its changes. Requests received before the reconcile started still join the batch.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index fd40da63ca..0d30e1b2dc 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -206,6 +206,7 @@ export interface IDispatchWorkspaceRequestOptions { readonly envelope: IDaemonRequestEnvelope; // (undocumented) readonly onExecutionStarting?: () => void; + readonly receivedTimeMs?: number; // (undocumented) readonly resolver: IDaemonRequestResolver | undefined; // (undocumented) @@ -691,7 +692,7 @@ export type MapWorkspaceInvalidationsToOperationsAsync = (options: IMapWorkspace // @beta export class PhasedRequestRouter { constructor(workspaceSession: IWorkspaceSession); - executeAsync(request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection?: boolean, onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings): Promise; + executeAsync(request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection?: boolean, onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings, receivedTimeMs?: number): Promise; } // @beta diff --git a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts index d984b44f91..8a72ae6f9c 100644 --- a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts +++ b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts @@ -94,6 +94,11 @@ export interface IDispatchWorkspaceRequestOptions { readonly workspaceSession: IWorkspaceSession; readonly resolver: IDaemonRequestResolver | undefined; readonly onExecutionStarting?: () => void; + /** + * The `performance.now()` timestamp at which the daemon received the request. A phased request can join a batch + * whose input reconcile started after this time; see {@link PhasedRequestRouter.executeAsync}. + */ + readonly receivedTimeMs?: number; } /** Executes an already admitted workspace generation without resolving against another session. @beta */ @@ -158,7 +163,7 @@ export class DaemonRequestDispatcher implements AsyncDisposable { async function dispatchWorkspaceRequestAsync( options: IDispatchWorkspaceRequestOptions ): Promise { - const { envelope, client, workspaceSession, resolver, onExecutionStarting } = options; + const { envelope, client, workspaceSession, resolver, onExecutionStarting, receivedTimeMs } = options; workspaceSession.assertActive?.(); if ( !isRushxInvocation(envelope) && @@ -191,7 +196,8 @@ async function dispatchWorkspaceRequestAsync( createPhasedClient(client), resolved.exactSelection, onExecutionStarting, - resolved.requestSettings + resolved.requestSettings, + receivedTimeMs ); } const globalRouter: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(workspaceSession); diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index 76ccbba728..1d2dfd9fd7 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -73,6 +73,8 @@ interface IPreparedPhasedRequest { readonly requestSettings: IPhasedCommandEngineRequestSettings | undefined; readonly requestSettingsKey: string; readonly selection: IResolvedSelection; + /** The `performance.now()` timestamp at which the daemon received the request; see `#canJoinCurrentBatch`. */ + readonly receivedTimeMs: number; /** The `performance.now()` timestamp at which the router received the request. */ readonly startTimeMs: number; readonly warningsAllowedByEnvironment: boolean; @@ -126,13 +128,20 @@ export class PhasedRequestRouter { this.#workspaceSession = workspaceSession; } - /** Validates and executes one resolved phased request against the warm graph. */ + /** + * Validates and executes one resolved phased request against the warm graph. + * + * @remarks + * `receivedTimeMs` is the `performance.now()` timestamp at which the daemon received the request. The request can + * join a batch whose input reconcile started after this time. It defaults to the time of this call. + */ public async executeAsync( request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection: boolean = false, onExecutionStarting?: () => void, - requestSettings?: IPhasedCommandEngineRequestSettings + requestSettings?: IPhasedCommandEngineRequestSettings, + receivedTimeMs?: number ): Promise { const startTimeMs: number = performance.now(); validateRequestIdentity(request); @@ -210,6 +219,7 @@ export class PhasedRequestRouter { client, exclusivityClass, interactiveSession, + receivedTimeMs: receivedTimeMs ?? startTimeMs, request, requestSettings, requestSettingsKey: JSON.stringify(requestSettings ?? null), @@ -250,6 +260,8 @@ class PhasedRequestBatchCoordinator { #currentBatch: ReadonlyArray | undefined; #drainScheduled: boolean = false; #nextGraphLeasePromise: Promise | undefined; + /** When the current batch's input reconcile started, or undefined before it starts. */ + #reconcileStartTimeMs: number | undefined; #running: boolean = false; public constructor( @@ -335,6 +347,7 @@ class PhasedRequestBatchCoordinator { } this.#currentBatch = batch; this.#acceptingCurrentBatch = first.exclusivityClass === RequestExclusivityClass.SharedBuild; + this.#reconcileStartTimeMs = undefined; for (const entry of batch) { entry.executionStarted = true; } @@ -344,6 +357,7 @@ class PhasedRequestBatchCoordinator { await Promise.all(batch.map((entry: IBatchEntry) => this.#rejectEntryAsync(entry, error))); } finally { this.#acceptingCurrentBatch = false; + this.#reconcileStartTimeMs = undefined; this.#currentBatch = undefined; } } @@ -364,6 +378,8 @@ class PhasedRequestBatchCoordinator { } return ( this.#acceptingCurrentBatch && + // A request received after the reconcile started may have changed an input that the reconcile already read. + (this.#reconcileStartTimeMs === undefined || request.receivedTimeMs < this.#reconcileStartTimeMs) && this.#currentBatch?.[0]?.exclusivityClass === RequestExclusivityClass.SharedBuild && this.#currentBatch[0].requestSettingsKey === request.requestSettingsKey ); @@ -407,12 +423,16 @@ class PhasedRequestBatchCoordinator { throw new Error('The warm workspace operation graph is not idle.'); } executionLease = await this.#workspaceSession.acquireExecutionLeaseAsync?.(); + // Requests received before this point made their changes before the reconcile reads the inputs, so they can + // still join while it runs. Requests received later wait for the next batch, which reconciles again. + this.#reconcileStartTimeMs = performance.now(); await this.#workspaceSession.reconcileInvalidationsAsync(); if (batch[0].exclusivityClass === RequestExclusivityClass.SharedBuild) { this.#takeCompatiblePending(batch); } this.#acceptingCurrentBatch = false; + const participants: IBatchEntry[] = batch.filter((entry: IBatchEntry) => this.#isEntryLive(entry)); if (participants.length === 0) { const beforeResultAsync: (() => Promise) | undefined = executionLease diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 0cf37b2483..b13176adca 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -176,6 +176,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { destination: IDaemonRequestDispatchClient, dispatchAsync: DispatchWorkspaceRequestAsync ): Promise { + const receivedTimeMs: number = performance.now(); // Native Rush owns its SDK handoff; a foreign client's bundled engine must not override this one. const envelope: IDaemonRequestEnvelope = { ...request, @@ -230,6 +231,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { client, workspaceSession: prepared.session, resolver: prepared.resolver, + receivedTimeMs, onExecutionStarting: () => { this.#assertGeneration(prepared); state.began = true; diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index bca1bde260..6333cea8d6 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -86,6 +86,30 @@ function getResultOperationIds(result: IDaemonPhasedRequestResult): ReadonlyArra return result.operationResults.map(({ operationId }) => operationId); } +/** One input file of project-c: the version on disk, and the version that the latest reconcile read. */ +class TestWorkspaceInput { + #diskVersion: number = 0; + #snapshotVersion: number | undefined; + + public edit(): void { + this.#diskVersion++; + } + + public read(): void { + this.#snapshotVersion = this.#diskVersion; + } + + public describe(): string { + return `snapshot=${this.#snapshotVersion} disk=${this.#diskVersion}`; + } +} + +async function settleAsync(): Promise { + for (let turn: number = 0; turn < 20; turn++) { + await new Promise((resolve) => setImmediate(resolve)); + } +} + interface IExecutionLeaseTracker { readonly events: string[]; } @@ -638,6 +662,149 @@ describe('shared phased request batching', () => { expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); }); + it('puts a compatible request received during the reconcile into a later batch that reconciles again', async () => { + const input: TestWorkspaceInput = new TestWorkspaceInput(); + const events: string[] = []; + const fixture: ITestRoutingFixture = createFixture({ + actionCAsync: async (): Promise => { + events.push(`run:C ${input.describe()}`); + } + }); + const inputRead: IDeferred = createDeferred(); + const finishFirstReconcile: IDeferred = createDeferred(); + let reconcileCount: number = 0; + fixture.session.onReconcileAsync = async (): Promise => { + reconcileCount++; + events.push('reconcile:start'); + input.read(); + if (reconcileCount === 1) { + inputRead.resolve(); + // The reconcile has read project-c's input and is still reading the rest of the workspace. + await finishFirstReconcile.promise; + } + events.push('reconcile:end'); + }; + const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const first = router.executeAsync( + createRequest('first', OPERATION_A), + new TestPhasedRequestClient('one') + ); + await inputRead.promise; + + // Another client changes project-c's input, then submits a compatible build that needs it. + input.edit(); + events.push('edit'); + const second = router.executeAsync( + createRequest('second', OPERATION_C), + new TestPhasedRequestClient('two') + ); + await settleAsync(); + finishFirstReconcile.resolve(); + + expect(await first).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(await second).toMatchObject({ exitCode: 0, outcome: 'success' }); + // Joining the first batch would run project-c on the inputs read before its change ("snapshot=0 disk=1"). + expect(events).toEqual([ + 'reconcile:start', + 'edit', + 'reconcile:end', + 'reconcile:start', + 'reconcile:end', + 'run:C snapshot=1 disk=1' + ]); + expect(scheduleSpy).toHaveBeenCalledTimes(2); + }); + + it('lets a compatible request that is pending before the reconcile join the batch and see its change', async () => { + const input: TestWorkspaceInput = new TestWorkspaceInput(); + const events: string[] = []; + const fixture: ITestRoutingFixture = createFixture({ + actionCAsync: async (): Promise => { + events.push(`run:C ${input.describe()}`); + } + }); + const leaseRequested: IDeferred = createDeferred(); + const grantLease: IDeferred = createDeferred(); + fixture.session.acquireExecutionLeaseAsync = async (): Promise => { + leaseRequested.resolve(); + await grantLease.promise; + return { [Symbol.asyncDispose]: async (): Promise => undefined }; + }; + fixture.session.onReconcileAsync = async (): Promise => { + events.push('reconcile'); + input.read(); + }; + const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const first = router.executeAsync( + createRequest('first', OPERATION_A), + new TestPhasedRequestClient('one') + ); + await leaseRequested.promise; + + input.edit(); + events.push('edit'); + const second = router.executeAsync( + createRequest('second', OPERATION_C), + new TestPhasedRequestClient('two') + ); + await settleAsync(); + grantLease.resolve(); + + expect(await first).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(await second).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(events).toEqual(['edit', 'reconcile', 'run:C snapshot=1 disk=1']); + expect(scheduleSpy).toHaveBeenCalledTimes(1); + }); + + it('lets a request received before the reconcile began join the batch when it reaches the router during it', async () => { + const input: TestWorkspaceInput = new TestWorkspaceInput(); + const events: string[] = []; + const fixture: ITestRoutingFixture = createFixture({ + actionCAsync: async (): Promise => { + events.push(`run:C ${input.describe()}`); + } + }); + const reconcileStarted: IDeferred = createDeferred(); + const finishReconcile: IDeferred = createDeferred(); + fixture.session.onReconcileAsync = async (): Promise => { + events.push('reconcile:start'); + input.read(); + reconcileStarted.resolve(); + await finishReconcile.promise; + events.push('reconcile:end'); + }; + const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + + // The second client changes project-c's input and the daemon receives its request, which then waits (for + // example behind a graph load) and reaches the router only while the first batch reconciles. + input.edit(); + events.push('edit'); + const secondReceivedTimeMs: number = performance.now(); + const first = router.executeAsync( + createRequest('first', OPERATION_A), + new TestPhasedRequestClient('one') + ); + await reconcileStarted.promise; + const second = router.executeAsync( + createRequest('second', OPERATION_C), + new TestPhasedRequestClient('two'), + false, + undefined, + undefined, + secondReceivedTimeMs + ); + await settleAsync(); + finishReconcile.resolve(); + + expect(await first).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(await second).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(events).toEqual(['edit', 'reconcile:start', 'reconcile:end', 'run:C snapshot=1 disk=1']); + expect(scheduleSpy).toHaveBeenCalledTimes(1); + }); + it('lets a late shared build wait past a default timeout while a compatible batch executes', async () => { const operationStarted: IDeferred = createDeferred(); const releaseOperation: IDeferred = createDeferred(); diff --git a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts index 44c3358cd4..7941f76a7f 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts @@ -20,7 +20,10 @@ const OPERATION_C: string = 'project-c (_phase:test)'; const PROMPT_CANCELLATION_MS: number = 1000; const TIMED_OUT: 'timed out' = 'timed out'; -async function raceWithTimeoutAsync(promise: Promise, timeoutMs: number): Promise { +async function raceWithTimeoutAsync( + promise: Promise, + timeoutMs: number +): Promise { let timer: NodeJS.Timeout | undefined; try { return await Promise.race([ @@ -167,11 +170,54 @@ describe('phased request client cancellation', () => { const { hanging, actionAsync } = createHangingOperation(); const fixture: ITestRoutingFixture = createFixture(actionAsync); const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + let onLeaseRequested: () => void = () => undefined; + const leaseRequested: Promise = new Promise((resolve) => (onLeaseRequested = resolve)); + let grantLease: () => void = () => undefined; + const leaseGranted: Promise = new Promise((resolve) => (grantLease = resolve)); + fixture.session.acquireExecutionLeaseAsync = async (): Promise => { + onLeaseRequested(); + await leaseGranted; + return { [Symbol.asyncDispose]: async (): Promise => undefined }; + }; + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const cancelledClient: TestPhasedRequestClient = new TestPhasedRequestClient('one'); + const cancelled: Promise = router.executeAsync( + createRequest('cancelled', OPERATION_A), + cancelledClient + ); + await leaseRequested; + // Accepted while the batch waits for its execution lease, so it joins before the batch reconciles. + const continuing: Promise = router.executeAsync( + createRequest('continuing', OPERATION_A), + new TestPhasedRequestClient('two') + ); + // Let the continuing request finish preparation and enter the pending queue. + for (let tick: number = 0; tick < 20; tick++) { + await new Promise((resolve) => setImmediate(resolve)); + } + + cancelledClient.abortController.abort(); + expect(await cancelled).toMatchObject({ aborted: true, outcome: 'aborted' }); + + grantLease(); + await hanging.started; + expect(hanging.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([false]); + expect(abortSpy).not.toHaveBeenCalledWith({ terminateRunning: true }); + hanging.release(); + expect(await continuing).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); + }); + + it('keeps a request received during the reconcile out of the batch when its only participant cancels', async () => { + const { hanging, actionAsync } = createHangingOperation(); + const fixture: ITestRoutingFixture = createFixture(actionAsync); + const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); let onReconciling: () => void = () => undefined; const reconciling: Promise = new Promise((resolve) => (onReconciling = resolve)); let releaseReconcile: () => void = () => undefined; const reconcileReleased: Promise = new Promise((resolve) => (releaseReconcile = resolve)); + let reconciles: number = 0; fixture.session.onReconcileAsync = async () => { + reconciles++; onReconciling(); await reconcileReleased; }; @@ -182,23 +228,25 @@ describe('phased request client cancellation', () => { cancelledClient ); await reconciling; - // Accepted while the batch is still being prepared, so it joins once preparation finishes. + // Received after the reconcile started, so it waits for the next batch, which reconciles again. const continuing: Promise = router.executeAsync( createRequest('continuing', OPERATION_A), new TestPhasedRequestClient('two') ); - // Let the continuing request finish preparation and enter the pending queue. for (let tick: number = 0; tick < 20; tick++) { await new Promise((resolve) => setImmediate(resolve)); } cancelledClient.abortController.abort(); + // The cancelled client is the batch's last participant, so its result still follows the batch's reconcile. + expect(await raceWithTimeoutAsync(cancelled, 100)).toBe(TIMED_OUT); + releaseReconcile(); expect(await cancelled).toMatchObject({ aborted: true, outcome: 'aborted' }); - releaseReconcile(); await hanging.started; + expect(reconciles).toBe(2); + expect(scheduleSpy).toHaveBeenCalledTimes(1); expect(hanging.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([false]); - expect(abortSpy).not.toHaveBeenCalledWith({ terminateRunning: true }); hanging.release(); expect(await continuing).toMatchObject({ aborted: false, exitCode: 0, outcome: 'success' }); }); diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 704df32f27..408f74dbd5 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -7,13 +7,11 @@ import * as path from 'node:path'; import { execFileSync } from 'node:child_process'; import { - PhasedCommandEngine, Rush, RushProjectConfiguration, RushUserConfiguration, type IOperationGraph, type Operation, - type OperationEnabledState, type RushConfigurationProject } from '@microsoft/rush-lib'; import { @@ -48,6 +46,8 @@ import { type IWorkspaceResolverLifecycle } from '../WorkspaceResolverLifecycle'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import type { IRequestLease } from '../RequestScheduler'; +import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; import { TestPhasedRequestClient } from './PhasedRequestRouterTestUtilities'; import { createNativeScriptGateAsync, @@ -909,9 +909,11 @@ process.exit(23); it('uses one native lease for a merged batch and excludes native actions throughout reconciliation and execution', async () => { const fixture: IFixture = await createFixtureAsync(); + const leaseRequested: IDeferred = createDeferred(); + const grantLease: IDeferred = createDeferred(); const reconcileEntered: IDeferred = createDeferred(); const releaseReconciliation: IDeferred = createDeferred(); - const secondSelection: IDeferred = createDeferred(); + const secondAdmitted: IDeferred = createDeferred(); let secondClient: DaemonRequestWireClient | undefined; let gate: INativeScriptGate | undefined; let first: Promise | undefined; @@ -922,7 +924,15 @@ process.exit(23); await fixture.session.quiesceWarmSetAsync(); const graph: IOperationGraph = fixture.session.operationGraph!; const scheduleSpy: jest.SpyInstance = jest.spyOn(graph, 'scheduleIterationAsync'); - const leaseSpy: jest.SpyInstance = jest.spyOn(fixture.session, 'acquireExecutionLeaseAsync'); + const acquireExecutionLeaseAsync: WorkspaceSession['acquireExecutionLeaseAsync'] = + fixture.session.acquireExecutionLeaseAsync.bind(fixture.session); + const leaseSpy: jest.SpyInstance = jest + .spyOn(fixture.session, 'acquireExecutionLeaseAsync') + .mockImplementationOnce(async () => { + leaseRequested.resolve(); + await grantLease.promise; + return await acquireExecutionLeaseAsync(); + }); const reconcileAsync: WorkspaceSession['reconcileInvalidationsAsync'] = fixture.session.reconcileInvalidationsAsync.bind(fixture.session); jest.spyOn(fixture.session, 'reconcileInvalidationsAsync').mockImplementationOnce(async () => { @@ -930,28 +940,38 @@ process.exit(23); await releaseReconciliation.promise; return await reconcileAsync(); }); - const selectAsync: PhasedCommandEngine['selectOperationsAsync'] = - PhasedCommandEngine.prototype.selectOperationsAsync; - let selections: number = 0; - jest.spyOn(PhasedCommandEngine.prototype, 'selectOperationsAsync').mockImplementation(async function ( - this: PhasedCommandEngine, - selectedGraph: IOperationGraph + const routeAsync: PhasedRequestRouter['executeAsync'] = PhasedRequestRouter.prototype.executeAsync; + let routedRequests: number = 0; + jest.spyOn(PhasedRequestRouter.prototype, 'executeAsync').mockImplementation(async function ( + this: PhasedRequestRouter, + ...args: Parameters ) { - const selection: ReadonlyMap = await selectAsync.call( - this, - selectedGraph - ); - if (++selections === 2) secondSelection.resolve(); - return selection; + routedRequests++; + return await routeAsync.apply(this, args); + }); + const admitAsync: RequestAdmissionController['acquireAsync'] = + RequestAdmissionController.prototype.acquireAsync; + jest.spyOn(RequestAdmissionController.prototype, 'acquireAsync').mockImplementation(async function ( + this: RequestAdmissionController, + ...args: Parameters + ) { + const lease: IRequestLease = await admitAsync.apply(this, args); + // The router admits each request once, after the workspace lifecycle has, and then enqueues it. + if (routedRequests === 2) secondAdmitted.resolve(); + return lease; }); gate = await createNativeScriptGateAsync(fixture.repoRoot, 'a'); secondClient = await DaemonRequestWireClient.connectAsync(fixture.host.paths.socketPath); await secondClient.handshakeAsync(); first = runAsync(fixture, 'dependency', ['build', '--to', 'a']); - await reconcileEntered.promise; + await leaseRequested.promise; + // Accepted while the batch waits for its native lease, so it joins before the batch reconciles. second = runAsync({ ...fixture, client: secondClient }, 'consumer', ['build', '--to', 'b']); - await secondSelection.promise; + await secondAdmitted.promise; + // After admission, the router enqueues the request without awaiting. await new Promise((resolve) => setImmediate(resolve)); + grantLease.resolve(); + await reconcileEntered.promise; expect(await runNativeCommandAsync(fixture.repoRoot, ['rebuild', '--only', 'c'])).toMatchObject({ exitCode: 1, stdout: expect.stringContaining('Another Rush command') @@ -980,6 +1000,7 @@ process.exit(23); expect(scheduleSpy).toHaveBeenCalledTimes(1); expect(runs(fixture)).toEqual(['c:one:', 'a:one:', 'b:one:']); } finally { + grantLease.resolve(); releaseReconciliation.resolve(); await gate?.releaseAsync(); await first; From 103bd5e050c5a004b682ba2e0a4abe763db04048 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 17:04:32 +0000 Subject: [PATCH 025/265] [rush-daemon] Another request's graph load doesn't spend any finite wait timeout Swarm integration step 10; original commit 908dcc1f27 (merge of swarm/r05 at 868e4b9299). Refs: task 7 part 3, board 891. ch01 GATE OK board 1196 (gated tree 55698be825). Commits folded into this step (1): - 868e4b9299 [rush-daemon] Admission: another request's graph load doesn't spend any finite wait timeout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 27 ++- .../src/ClientAdmissionControls.ts | 11 +- .../src/test/ClientAdmissionControls.test.ts | 13 +- ...mission-wait-timeout_2026-09-28-13-45.json | 10 + ...mission-wait-timeout_2026-09-28-13-45.json | 10 + docs/rush/environment-variables.md | 2 +- libraries/rush-daemon/README.md | 26 +-- .../src/WorkspaceRequestAdmission.ts | 158 +++++++++++----- .../src/WorkspaceRequestLifecycle.ts | 2 +- .../src/WorkspaceRestartArbiter.ts | 2 +- .../test/WorkspaceRequestAdmission.test.ts | 175 +++++++++++------- .../test/WorkspaceTransitionAdmission.test.ts | 46 ++++- 12 files changed, 335 insertions(+), 147 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index ba0dcca5da..f651b98bab 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -49,15 +49,22 @@ rounded down to milliseconds. These controls are mutually exclusive and are consumed before forwarding, never appended to a project script. Arguments after `--` remain literal script arguments. -The queue timeout is measured from when the daemon receives the request. An -explicit `--no-wait`, `--wait-timeout`, `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, or -`daemon.queueTimeoutSeconds` in `rush.json` bounds the entire wait: waiting for -workspace admission and waiting for a running build that the request could not -join. The built-in 30-second default bounds only workspace admission (for example, -waiting for a command that needs exclusive access). With the default, a build that -arrives while a compatible build is already running waits for it to finish and then -runs, instead of failing after 30 seconds. On a timeout, the client exits with -code 1 and says how to wait longer. +The queue timeout is measured from when the daemon receives the request, and only +time spent waiting for other requests counts against it, whether it comes from +`--wait-timeout`, `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, `daemon.queueTimeoutSeconds` +in `rush.json`, or the built-in 30-second default. Waiting while another request +loads or reloads the workspace graph does not count, so every build that arrives +while the first build after startup loads the graph runs once the load finishes. +That wait fails after 10 times the timeout (5 minutes with the default), so a load +that never finishes does not hold other requests forever. The request's own routing +and execution do not count either. A configured or per-invocation timeout also +limits waiting for a running build that the request could not join; the built-in +default does not. With the default, a build that arrives while a compatible build is +already running waits for it to finish and then runs, instead of failing after 30 +seconds. `--no-wait` fails wherever the request would wait. On a timeout, the client +exits with code 1 and suggests `--wait-timeout`. It does not suggest exporting +`RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, because Rush versions that do not recognize a +`RUSH_` environment variable fail every command while it is set. Admission controls also apply to experimental graph requests, but not `start|stop|restart|status|logs`. They affect daemon admission only; native fallback @@ -212,7 +219,7 @@ keys and unknown `RUSH_DAEMON*` variables fail validation. | `enabled` | `RUSH_DAEMON` | false | Client routing | | `autoStart` | `RUSH_DAEMON_AUTO_START` | true | Only after opt-in | | `idleTimeoutSeconds` | `RUSH_DAEMON_IDLE_TIMEOUT_SECONDS` | 900 | Host idle shutdown after request/output/cleanup drain | -| `queueTimeoutSeconds` | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | 30 | Admission wait limit. The default does not bound waiting behind a running compatible build; an explicit value does | +| `queueTimeoutSeconds` | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | 30 | Admission wait limit. Time behind another request's graph load (up to 10 times the limit) and the request's own work do not count. The default does not limit waiting behind a running compatible build; an explicit value does | | `watch` | `RUSH_DAEMON_WATCH` | false | Persistent host observation of requested warm projects; false keeps root/config guards only. Never schedules builds | | `usePersistentIpcRunners` | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | false | Enables explicit per-operation `daemonIpc` Node launchers for unsharded incremental daemon builds | | `warmIdleTimeoutSeconds` | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | 300 | Idle runner, project-watcher and retained-result eviction | diff --git a/apps/rush-cli-client/src/ClientAdmissionControls.ts b/apps/rush-cli-client/src/ClientAdmissionControls.ts index a0dc6cb017..bd3d04edeb 100644 --- a/apps/rush-cli-client/src/ClientAdmissionControls.ts +++ b/apps/rush-cli-client/src/ClientAdmissionControls.ts @@ -82,12 +82,15 @@ export function formatAdmissionFailure( return `${prefix}: another daemon request is using this workspace and --no-wait was specified.\n`; } if (code === 'wait-timeout') { - const seconds: string = - admission?.waitTimeoutMs === undefined ? '' : ` after ${admission.waitTimeoutMs / 1000}s`; + const timeout: string = + admission?.waitTimeoutMs === undefined + ? '' + : ` after its ${admission.waitTimeoutMs / 1000}s wait timeout`; return ( - `${prefix}: timed out${seconds} waiting for another daemon request in this workspace to finish ` + + `${prefix}: timed out${timeout} waiting for another daemon request in this workspace to finish ` + '(a command that needs exclusive access, or a running build). ' + - 'To wait longer, use --wait-timeout or set RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS.\n' + // Only the per-invocation flag is offered: Rush versions that do not recognize the variable reject it. + 'To wait longer, pass --wait-timeout .\n' ); } return `${prefix}.\n`; diff --git a/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts b/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts index 5aea17bc37..c835699a2f 100644 --- a/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts +++ b/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts @@ -68,17 +68,20 @@ describe(getConfiguredAdmission.name, () => { }); }); - it('keeps an explicitly configured timeout as one absolute deadline', () => { + it('does not mark an explicitly configured timeout as the default', () => { expect(getConfiguredAdmission({ queueTimeoutSeconds: 1.5, explicit: true })).toEqual({ waitTimeoutMs: 1500 }); }); }); describe(formatAdmissionFailure.name, () => { - it('explains a wait timeout and how to wait longer', () => { + it('explains a wait timeout and offers only the per-invocation way to wait longer', () => { const message: string = formatAdmissionFailure('wait-timeout', { waitTimeoutMs: 5000 }); - expect(message).toContain('daemon admission failed (wait-timeout): timed out after 5s waiting for'); - expect(message).toContain('--wait-timeout '); - expect(message).toContain('RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS'); + expect(message).toContain( + 'daemon admission failed (wait-timeout): timed out after its 5s wait timeout waiting for' + ); + expect(message).toContain('pass --wait-timeout '); + // Exporting it would break later commands of Rush versions that reject unknown RUSH_ variables. + expect(message).not.toContain('RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS'); }); it('explains a no-wait failure', () => { diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json b/common/changes/@rushstack/rush-cli-client/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json new file mode 100644 index 0000000000..f0547ce6ea --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "A daemon admission timeout names the timeout that expired and suggests only --wait-timeout, because exporting RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS breaks Rush versions that do not recognize it.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json b/common/changes/@rushstack/rush-daemon/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json new file mode 100644 index 0000000000..454b715b4f --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-admission-wait-timeout_2026-09-28-13-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Admission: time spent waiting while another request loads or reloads the workspace graph no longer counts against any finite wait timeout, explicit or default, and is limited to 10 times that timeout instead; an admitted request's own routing and execution no longer spend an explicit timeout either. Timeout messages suggest --wait-timeout rather than exporting RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/docs/rush/environment-variables.md b/docs/rush/environment-variables.md index 721e370fc3..590a0932a3 100644 --- a/docs/rush/environment-variables.md +++ b/docs/rush/environment-variables.md @@ -24,7 +24,7 @@ variables and invalid values are errors, not ignored settings. | `RUSH_DAEMON` | `0` | Opt into daemon execution through the separate clients. `0` disables it. Overrides `daemon.enabled`. | | `RUSH_DAEMON_AUTO_START` | `1` | Start an absent compatible daemon after daemon execution is selected. Does not itself opt in. Overrides `autoStart`. | | `RUSH_DAEMON_IDLE_TIMEOUT_SECONDS` | `900` | Shut down after the last pending request, output drain and cleanup are complete. Positive seconds, at most 2147483.647. Overrides `idleTimeoutSeconds`. | -| `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | `30` | Maximum request admission wait. Nonnegative seconds, at most 2147483.647; converted to whole milliseconds by rounding down. Per-invocation `--wait-timeout` or `--no-wait` takes precedence. Overrides `queueTimeoutSeconds`. | +| `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | `30` | Maximum request admission wait. Nonnegative seconds, at most 2147483.647; converted to whole milliseconds by rounding down. Time behind another request's load of the workspace graph (up to 10 times this value) does not count. Per-invocation `--wait-timeout` or `--no-wait` takes precedence. Overrides `queueTimeoutSeconds`. | | `RUSH_DAEMON_WATCH` | `0` | Observe requested warm projects between requests. Never schedules builds and does not enable `--watch` mode. Root/config guards and request-time input reconciliation remain active when disabled. Overrides `watch`. | | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | `0` | Enable explicitly configured `operationSettings[].daemonIpc` Node workers for supported incremental daemon builds. Does not convert arbitrary shell scripts into persistent workers. Overrides `usePersistentIpcRunners`. | | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | `300` | Idle expiration for retained runners and project watchers, together with those projects' results. Results of resource-free (shell/null) projects do not expire. Positive seconds, at most 2147483.647. Overrides `warmIdleTimeoutSeconds`. | diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 4cadfb0bda..250201ec20 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -457,17 +457,21 @@ the router validates both, reconciles retained invalidations, applies the select and runs at most one scheduled iteration. A workspace-wide `RequestScheduler` admits phased and global routes using the static built-in command policy (`SHARED-BUILD`, `SHARED-READ`, or `EXCLUSIVE`); custom-origin commands and unknown built-in names fail closed to `EXCLUSIVE`, including plugin replacements of built-in names. Queued clients receive -ordered, one-based position controls and can request fail-fast or bounded waiting. One progress channel covers both -workspace admission and the temporary phased graph-execution gate. An explicit `noWait` or `waitTimeoutMs` is one -absolute deadline for both waits. When the client marks `waitTimeoutMs` as its default (`waitTimeoutIsDefault`), it is -a budget that only contention spends: a `SHARED-BUILD` request that arrives after the current batch has closed waits -on the graph-execution gate without a deadline, because it is queued only behind running compatible shared builds, and -then runs in the next batch. A request queued behind another request that holds exclusive workspace admission to load -or reload the graph does not spend the budget during that load, so every build that arrives while the first build -after startup loads the graph is admitted when the load finishes. The budget does run while that other request still -waits for exclusive admission, so requests behind a reload that cannot start, for example behind a long build, still -time out. Routing and executing an admitted request do not spend the budget either: a request that re-enters -workspace admission to reload the graph after its inputs changed keeps the budget it had when it was admitted. +ordered, one-based position controls and can request fail-fast or time-limited waiting. One progress channel covers +both workspace admission and the temporary phased graph-execution gate. `noWait` fails at once wherever the request +would wait. A finite `waitTimeoutMs` is a budget that only contention spends, whether it is the client's default or an +explicit value. A request queued behind another request that holds exclusive workspace admission to load or reload +the graph does not spend its budget during that load, so every build that arrives while the first build after startup +loads the graph is admitted when the load finishes. That wait is limited separately, to 10 times `waitTimeoutMs`, so +a load that never finishes does not hold the requests behind it indefinitely. The budget does run while that other +request still waits for exclusive admission, so requests behind a reload that cannot start, for example behind a long +build, still time out. Routing and executing an admitted request do not spend the budget either. Routing boundaries +such as the graph-execution gate apply the remaining budget they receive, and a request that re-enters workspace +admission to reload the graph after its inputs changed starts again from the budget it had when it was admitted; time +it spent at those boundaries is not charged again. The default and an explicit value differ only at the +graph-execution gate. When the client marks `waitTimeoutMs` as its default (`waitTimeoutIsDefault`), a `SHARED-BUILD` +request that arrives after the current batch has closed waits there without a deadline, because it is queued only +behind running compatible shared builds, and then runs in the next batch. An explicit value still limits that wait. Cancellation, disconnect, or queue-output failure removes queued work before it can execute. A requesting client receives only its enabled dependency closure's WS1 raw chunks and structured events through backpressured, ordered callbacks, followed exactly once by a typed final command result after all preceding output diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index edf357db67..40e49c264e 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -1,7 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { validateDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; +import { + MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS, + validateDaemonRequestAdmissionOptions +} from '@rushstack/rush-daemon-protocol'; import type { DaemonRequestAdmissionErrorCode, IDaemonRequestAdmissionOptions, @@ -33,6 +36,14 @@ export interface IRequestAdmissionControllerOptions { } const REQUEST_SCHEDULER_BY_SESSION: WeakMap = new WeakMap(); +/** A request waits for another request's graph load or reload for up to this many times its wait timeout. */ +const GRAPH_LOAD_WAIT_FACTOR: number = 10; +// Only the per-invocation flag is offered: Rush versions that do not recognize the environment variable reject it. +const WAIT_LONGER_HINT: string = 'Use --wait-timeout to wait longer.'; + +function formatSeconds(ms: number): string { + return `${Math.round(ms / 100) / 10}s`; +} class WorkspaceRequestScheduler extends RequestScheduler { readonly #session: IWorkspaceSession; @@ -120,45 +131,75 @@ export class AdmissionProgress { } } -/** A wait budget that is spent only while `progress` is inactive. */ +/** + * A wait budget that is spent only while `progress` is inactive. + * + * @remarks + * Waiting while `progress` is active is limited separately, to `maxPausedMs` in total, so that a wedged transition + * does not hold the requests behind it indefinitely. `onExhausted` receives whether that limit, rather than the + * budget, ran out. + */ class ProgressPausedBudget { - readonly #onExhausted: () => void; + readonly #maxPausedMs: number; + readonly #onExhausted: (pausedLimitReached: boolean) => void; readonly #progress: AdmissionProgress; readonly #unsubscribe: () => void; + #intervalStartMs: number = 0; + #paused: boolean = false; + #pausedMs: number = 0; #remainingMs: number; - #runningSinceMs: number | undefined; #timer: ReturnType | undefined; - public constructor(remainingMs: number, progress: AdmissionProgress, onExhausted: () => void) { + public constructor( + remainingMs: number, + maxPausedMs: number, + progress: AdmissionProgress, + onExhausted: (pausedLimitReached: boolean) => void + ) { this.#remainingMs = remainingMs; + this.#maxPausedMs = maxPausedMs; this.#progress = progress; this.#onExhausted = onExhausted; - this.#unsubscribe = progress.subscribe(() => this.#update()); - this.#update(); + this.#unsubscribe = progress.subscribe(() => { + this.#endInterval(); + this.#startInterval(); + }); + this.#startInterval(); } - /** Stops spending and returns the unspent budget. */ - public stop(): number { - this.#unsubscribe(); - this.#pause(); + /** The time that was not spent from the budget because `progress` was active. */ + public get pausedMs(): number { + return this.#pausedMs; + } + + /** The unspent budget. */ + public get remainingMs(): number { return this.#remainingMs; } - #update(): void { - if (this.#progress.active) { - this.#pause(); - } else if (this.#runningSinceMs === undefined) { - this.#runningSinceMs = Date.now(); - this.#timer = setTimeout(this.#onExhausted, this.#remainingMs); - } + /** Stops spending; `pausedMs` and `remainingMs` are final afterwards. Calling it again has no effect. */ + public stop(): void { + this.#unsubscribe(); + this.#endInterval(); } - #pause(): void { - if (this.#runningSinceMs === undefined) return; - this.#remainingMs = Math.max(0, this.#remainingMs - (Date.now() - this.#runningSinceMs)); - this.#runningSinceMs = undefined; + #startInterval(): void { + this.#paused = this.#progress.active; + this.#intervalStartMs = Date.now(); + const delayMs: number = this.#paused ? this.#maxPausedMs - this.#pausedMs : this.#remainingMs; + this.#timer = setTimeout(() => this.#onExhausted(this.#paused), Math.max(0, delayMs)); + } + + #endInterval(): void { + if (this.#timer === undefined) return; clearTimeout(this.#timer); this.#timer = undefined; + const elapsedMs: number = Date.now() - this.#intervalStartMs; + if (this.#paused) { + this.#pausedMs += elapsedMs; + } else { + this.#remainingMs = Math.max(0, this.#remainingMs - elapsedMs); + } } } @@ -209,7 +250,7 @@ export class RequestAdmissionController { * @remarks * A shared-build request that reaches this gate is only waiting behind running compatible shared builds, which is * progress rather than contention. A client-default timeout therefore does not apply to that wait; an explicit - * `noWait` or `waitTimeoutMs` still applies, using the same absolute deadline as workspace admission. + * `noWait` or `waitTimeoutMs` still applies, using the request's remaining admission budget. */ public async acquireGraphExecutionAsync( scheduler: RequestScheduler, @@ -232,10 +273,12 @@ export class RequestAdmissionController { * * @remarks * While `transition` reports progress, the other request holds the exclusive gate and is loading the graph that - * this request needs, so a client-default timeout is not spent: at a cold start every concurrent build waits for - * the first build's graph load. The default budget is still spent while the transition itself waits for another - * request, so a transition that cannot start does not hold its followers indefinitely. Unspent budget carries over - * to later waits of this request. An explicit `noWait` or `waitTimeoutMs` applies unchanged. + * this request needs, so the request's wait timeout is not spent: at a cold start every concurrent build waits for + * the first build's graph load, whether its timeout is the client default or explicit. That wait is limited + * separately, to `GRAPH_LOAD_WAIT_FACTOR` times the wait timeout, so a wedged load does not hold its followers + * indefinitely. The timeout is still spent while the transition itself waits for another request, so a transition + * that cannot start does not hold its followers either. Unspent time carries over to later waits of this request. + * `noWait` still fails at once. */ public async acquireBehindTransitionAsync( scheduler: RequestScheduler, @@ -243,12 +286,27 @@ export class RequestAdmissionController { ): Promise { const waitingFor: string = "another request's load or reload of the workspace graph"; const remainingMs: number | undefined = this.#getRemainingWaitTimeoutMs(); - if (!this.#admission?.waitTimeoutIsDefault || remainingMs === undefined) { - return await this.#acquireAsync(scheduler, RequestExclusivityClass.SharedBuild, remainingMs, waitingFor); + const waitTimeoutMs: number | undefined = this.#admission?.waitTimeoutMs; + if (remainingMs === undefined || waitTimeoutMs === undefined) { + return await this.#acquireAsync( + scheduler, + RequestExclusivityClass.SharedBuild, + remainingMs, + waitingFor + ); } const exhausted: AbortController = new AbortController(); - const budget: ProgressPausedBudget = new ProgressPausedBudget(remainingMs, transition, () => - exhausted.abort() + let pausedLimitReached: boolean = false; + const budget: ProgressPausedBudget = new ProgressPausedBudget( + remainingMs, + Math.min(GRAPH_LOAD_WAIT_FACTOR * waitTimeoutMs, MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS), + transition, + (reachedPausedLimit: boolean) => { + if (!exhausted.signal.aborted) { + pausedLimitReached = reachedPausedLimit; + exhausted.abort(); + } + } ); try { return await this.#acquireAsync( @@ -259,30 +317,38 @@ export class RequestAdmissionController { AbortSignal.any([this.#abortController.signal, exhausted.signal]) ); } catch (error) { + budget.stop(); if (!exhausted.signal.aborted || this.#abortController.signal.aborted) throw error; - throw this.#getReportedError( - new RequestSchedulerError( - RequestSchedulerErrorCode.WaitTimeout, - `The request was not admitted within ${this.#admission.waitTimeoutMs}ms.` - ), - waitingFor + const message: string = pausedLimitReached + ? `The request was not admitted within ${GRAPH_LOAD_WAIT_FACTOR} times its ${waitTimeoutMs}ms wait ` + + `timeout because ${waitingFor} was still running after ${formatSeconds(budget.pausedMs)}.` + : `The request was not admitted within its ${waitTimeoutMs}ms wait timeout while waiting for ` + + `${waitingFor}` + + (budget.pausedMs > 0 + ? `; ${formatSeconds(budget.pausedMs)} spent while that request loaded the graph did not count.` + : '.'); + throw new RequestSchedulerError( + RequestSchedulerErrorCode.WaitTimeout, + `${message} ${WAIT_LONGER_HINT}` ); } finally { - this.#deadlineMs = Date.now() + budget.stop(); + budget.stop(); + this.#deadlineMs = Date.now() + budget.remainingMs; } } /** - * Runs `action`, such as routing and executing an admitted request, without spending a client-default budget. + * Runs `action`, such as routing and executing an admitted request, without spending the request's wait timeout. * * @remarks - * Work after admission either runs or waits behind progress, such as the exempt graph-execution gate. A request that - * re-enters workspace admission afterwards, for example to reload the graph after its inputs changed, therefore - * keeps the budget it had before `action`. An explicit `noWait` or `waitTimeoutMs` keeps its absolute deadline. + * Work after admission either runs or waits at a routing boundary that applies the timeout it received, such as the + * graph-execution gate. A request that re-enters workspace admission afterwards, for example to reload the graph + * after its inputs changed, therefore keeps the unspent timeout it had before `action`, whether that timeout is the + * client default or explicit. */ - public async runOutsideDefaultBudgetAsync(action: () => Promise): Promise { + public async runOutsideWaitBudgetAsync(action: () => Promise): Promise { const remainingMs: number | undefined = this.#getRemainingWaitTimeoutMs(); - if (!this.#admission?.waitTimeoutIsDefault || remainingMs === undefined) { + if (remainingMs === undefined) { return await action(); } try { @@ -361,8 +427,8 @@ export class RequestAdmissionController { ) { return new RequestSchedulerError( RequestSchedulerErrorCode.WaitTimeout, - `The request was not admitted within ${waitTimeoutMs}ms while waiting for ${waitingFor}. ` + - 'Use --wait-timeout or RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS to wait longer.' + `The request was not admitted within its ${waitTimeoutMs}ms wait timeout while waiting for ${waitingFor}. ` + + WAIT_LONGER_HINT ); } return error; diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index b13176adca..9b51147ecb 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -222,7 +222,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { ...envelope, admission: admission.remainingAdmission }; - await admission.runOutsideDefaultBudgetAsync(async () => { + await admission.runOutsideWaitBudgetAsync(async () => { if (isMutation(envelope)) { await this.#executeMutationAsync(prepared, requestEnvelope, client, state, dispatchAsync); } else { diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts index 9fb26f6455..b1228eaca7 100644 --- a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -114,7 +114,7 @@ export class WorkspaceRestartArbiter { new RequestSchedulerError( RequestSchedulerErrorCode.WaitTimeout, 'The request was not admitted before the daemon could restart for its environment. ' + - 'Use --wait-timeout or RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS to wait longer.' + 'Use --wait-timeout to wait longer.' ) ), Math.min(MAX_TIMER_DELAY_MS, Math.max(0, deadline - Date.now())) diff --git a/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts index 232687ffb5..23b261466b 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts @@ -1,7 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; +import { + type IDaemonRequestAdmissionOptions, + MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS +} from '@rushstack/rush-daemon-protocol'; import { type IRequestLease, @@ -13,6 +16,10 @@ import { AdmissionProgress, RequestAdmissionController } from '../WorkspaceReque const DEFAULT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 100, waitTimeoutIsDefault: true }; const EXPLICIT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 100 }; +const BUDGETS: ReadonlyArray<{ kind: string; budget: IDaemonRequestAdmissionOptions }> = [ + { kind: 'a default', budget: DEFAULT_BUDGET }, + { kind: 'an explicit', budget: EXPLICIT_BUDGET } +]; interface IAcquisition { settled: boolean; @@ -59,53 +66,99 @@ describe(RequestAdmissionController.name, () => { jest.useRealTimers(); }); - it('does not spend a default budget while the transition it waits behind makes progress', async () => { - const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); - transition.setActive(true); - const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); - await jest.advanceTimersByTimeAsync(10_000); - expect(waiting.settled).toBe(false); + it.each(BUDGETS)( + 'does not spend $kind timeout while the transition it waits behind makes progress', + async ({ budget }) => { + const controller: RequestAdmissionController = createController(budget); + transition.setActive(true); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(999); + expect(waiting.settled).toBe(false); + + scheduler.downgradeExclusiveLease(owner, RequestExclusivityClass.SharedBuild); + transition.setActive(false); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.settled).toBe(true); + expect(waiting.error).toBeUndefined(); + expect(waiting.lease?.exclusivityClass).toBe(RequestExclusivityClass.SharedBuild); + waiting.lease?.release(); + controller.dispose(); + } + ); - scheduler.downgradeExclusiveLease(owner, RequestExclusivityClass.SharedBuild); - transition.setActive(false); - await jest.advanceTimersByTimeAsync(0); - expect(waiting.settled).toBe(true); - expect(waiting.error).toBeUndefined(); - expect(waiting.lease?.exclusivityClass).toBe(RequestExclusivityClass.SharedBuild); - waiting.lease?.release(); - controller.dispose(); - }); + it.each(BUDGETS)( + 'spends $kind timeout while the transition itself waits, and carries the rest across progress', + async ({ budget }) => { + const controller: RequestAdmissionController = createController(budget); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(60); + transition.setActive(true); + await jest.advanceTimersByTimeAsync(900); + transition.setActive(false); + await jest.advanceTimersByTimeAsync(39); + expect(waiting.settled).toBe(false); + + await jest.advanceTimersByTimeAsync(1); + expect(waiting.error).toMatchObject({ + code: RequestSchedulerErrorCode.WaitTimeout, + message: + "The request was not admitted within its 100ms wait timeout while waiting for another request's load " + + 'or reload of the workspace graph; 0.9s spent while that request loaded the graph did not count. ' + + 'Use --wait-timeout to wait longer.' + }); + expect(scheduler.queuedRequestCount).toBe(0); + controller.dispose(); + } + ); - it('spends a default budget while the transition itself waits, and carries the rest across progress', async () => { - const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); - const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); - await jest.advanceTimersByTimeAsync(60); + it('fails once the transition it waits behind has made progress for ten times its timeout', async () => { + const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); transition.setActive(true); - await jest.advanceTimersByTimeAsync(10_000); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(600); + // Time spent while the transition waits is budget, not paused time, so it does not advance the paused limit. transition.setActive(false); - await jest.advanceTimersByTimeAsync(39); + await jest.advanceTimersByTimeAsync(50); + transition.setActive(true); + await jest.advanceTimersByTimeAsync(399); expect(waiting.settled).toBe(false); await jest.advanceTimersByTimeAsync(1); expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.WaitTimeout, - message: expect.stringContaining( - "not admitted within 100ms while waiting for another request's load or reload of the workspace graph" - ) + message: + "The request was not admitted within 10 times its 100ms wait timeout because another request's load " + + 'or reload of the workspace graph was still running after 1s. Use --wait-timeout to wait longer.' }); expect(scheduler.queuedRequestCount).toBe(0); controller.dispose(); }); - it('keeps an explicit deadline behind a transition that makes progress', async () => { - const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); + it('limits the paused wait for the largest timeout to the timer range', async () => { + const client: AbortController = new AbortController(); + const controller: RequestAdmissionController = createController( + { waitTimeoutMs: MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS }, + client.signal + ); transition.setActive(true); const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); - await jest.advanceTimersByTimeAsync(99); + // A delay beyond the timer range would fire after 1ms. + await jest.advanceTimersByTimeAsync(10_000); expect(waiting.settled).toBe(false); - await jest.advanceTimersByTimeAsync(1); - expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.WaitTimeout }); + client.abort(); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.Aborted }); + controller.dispose(); + }); + + it('fails at once behind a transition when the request does not wait', async () => { + const controller: RequestAdmissionController = createController({ noWait: true }); + transition.setActive(true); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.NoWait }); + expect(scheduler.queuedRequestCount).toBe(0); controller.dispose(); }); @@ -121,39 +174,31 @@ describe(RequestAdmissionController.name, () => { controller.dispose(); }); - it('keeps an unspent default budget across admitted work for a later admission wait', async () => { - const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); - await jest.advanceTimersByTimeAsync(40); - const work: Promise = controller.runOutsideDefaultBudgetAsync( - () => new Promise((resolve) => setTimeout(resolve, 10_000)) - ); - await jest.advanceTimersByTimeAsync(10_000); - await work; - expect(controller.remainingAdmission).toEqual({ ...DEFAULT_BUDGET, waitTimeoutMs: 60 }); - - const waiting: IAcquisition = track(controller.acquireAsync(scheduler, RequestExclusivityClass.SharedBuild)); - await jest.advanceTimersByTimeAsync(59); - expect(waiting.settled).toBe(false); - await jest.advanceTimersByTimeAsync(1); - expect(waiting.error).toMatchObject({ - code: RequestSchedulerErrorCode.WaitTimeout, - message: expect.stringContaining('not admitted within 100ms while waiting for workspace admission') - }); - controller.dispose(); - }); - - it('keeps an explicit deadline running across admitted work', async () => { - const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); - const work: Promise = controller.runOutsideDefaultBudgetAsync( - () => new Promise((resolve) => setTimeout(resolve, 10_000)) - ); - await jest.advanceTimersByTimeAsync(10_000); - await work; - expect(controller.remainingAdmission).toEqual({ waitTimeoutMs: 0 }); - - const waiting: IAcquisition = track(controller.acquireAsync(scheduler, RequestExclusivityClass.SharedBuild)); - await jest.advanceTimersByTimeAsync(0); - expect(waiting.error).toMatchObject({ code: RequestSchedulerErrorCode.WaitTimeout }); - controller.dispose(); - }); + it.each(BUDGETS)( + 'keeps the unspent part of $kind timeout across admitted work for a later admission wait', + async ({ budget }) => { + const controller: RequestAdmissionController = createController(budget); + await jest.advanceTimersByTimeAsync(40); + const work: Promise = controller.runOutsideWaitBudgetAsync( + () => new Promise((resolve) => setTimeout(resolve, 10_000)) + ); + await jest.advanceTimersByTimeAsync(10_000); + await work; + expect(controller.remainingAdmission).toEqual({ ...budget, waitTimeoutMs: 60 }); + + const waiting: IAcquisition = track( + controller.acquireAsync(scheduler, RequestExclusivityClass.SharedBuild) + ); + await jest.advanceTimersByTimeAsync(59); + expect(waiting.settled).toBe(false); + await jest.advanceTimersByTimeAsync(1); + expect(waiting.error).toMatchObject({ + code: RequestSchedulerErrorCode.WaitTimeout, + message: + 'The request was not admitted within its 100ms wait timeout while waiting for workspace admission. ' + + 'Use --wait-timeout to wait longer.' + }); + controller.dispose(); + } + ); }); diff --git a/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts index 1337cb515f..2d8586d358 100644 --- a/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceTransitionAdmission.test.ts @@ -39,7 +39,7 @@ function expectWaitTimeout(exchange: ITerminalExchange): void { } describe('workspace admission behind a graph transition', () => { - it('admits builds that arrive while another build loads the graph without spending their default budget', async () => { + it('admits builds that arrive while another build loads the graph without spending their wait timeout', async () => { const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); try { const loadStarted: IDeferred = createDeferred(); @@ -54,22 +54,60 @@ describe('workspace admission behind a graph transition', () => { const withDefaultBudget: Promise = fixture.runAsync(BUILD_B, { admission: DEFAULT_BUDGET }); - const withExplicitBudget: ITerminalExchange = await fixture.runAsync(BUILD_B, { + // Raising the timeout must not make a build fail behind a load that the default timeout waits for. + const withExplicitBudget: Promise = fixture.runAsync(BUILD_B, { admission: { waitTimeoutMs: 300 } }); - expectWaitTimeout(withExplicitBudget); await delayAsync(600); fixture.beforeCreateSessionAsync = undefined; releaseLoad.resolve(); expectSuccess(await first); expectSuccess(await withDefaultBudget); + expectSuccess(await withExplicitBudget); expect(fixture.runs()).toEqual(['a', 'b']); } finally { await fixture[Symbol.asyncDispose](); } }); + it('fails a build that has waited behind a graph load for ten times its wait timeout', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); + const loadStarted: IDeferred = createDeferred(); + const releaseLoad: IDeferred = createDeferred(); + try { + fixture.beforeCreateSessionAsync = async () => { + loadStarted.resolve(); + await releaseLoad.promise; + }; + const first: Promise = fixture.runAsync(BUILD_B); + await loadStarted.promise; + + const startedAt: number = Date.now(); + const behindWedgedLoad: ITerminalExchange = await fixture.runAsync(BUILD_B, { + admission: { waitTimeoutMs: 100 } + }); + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(900); + expectWaitTimeout(behindWedgedLoad); + expect(behindWedgedLoad.terminal).toMatchObject({ + payload: { + errorMessage: expect.stringContaining( + "within 10 times its 100ms wait timeout because another request's load or reload of the workspace " + + 'graph was still running' + ) + } + }); + + fixture.beforeCreateSessionAsync = undefined; + releaseLoad.resolve(); + expectSuccess(await first); + } finally { + // A failed expectation must not leave the load held, which would keep the fixture from shutting down. + releaseLoad.resolve(); + await fixture[Symbol.asyncDispose](); + } + }); + it('keeps a graph-gate waiter default budget for its reload after an input change', async () => { const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => { created.write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\nruns.txt\nrelease-a\n'); @@ -105,6 +143,8 @@ describe('workspace admission behind a graph transition', () => { // The waiter's batch finds the changed input and re-enters admission behind the reload. expectSuccess(await graphGateWaiter); } finally { + // A failed expectation must not leave the long build running, which would keep the fixture from shutting down. + fs.writeFileSync(releaseFile, ''); await fixture[Symbol.asyncDispose](); } }); From 86780e6b4ffd0561e2ce9d81812a636d432b2b89 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 17:04:33 +0000 Subject: [PATCH 026/265] [rush-lib] Keep cache trust for unselected retained results and re-verify untrusted ones Swarm integration step 11; original commit 4456f17460 (merge of swarm/r07 at 53654af37c). Refs: board 401, task 74, board 982, board 1024. ch01 GATE OK board 1197 (gated tree 3014a03bb8). Commits folded into this step (3): - b6692d09a5 [rush-lib] Keep cache trust for unselected retained results and re-verify untrusted ones - 492126a051 [rush-lib] Re-verify retained results built against unverified dependency outputs without the build cache - 53654af37c [rush-daemon] Test that input captures are shared only between equal fingerprint environments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...etained-result-trust_2026-09-28-14-10.json | 10 + ...ify-retained-results_2026-09-28-15-25.json | 10 + ...ure-environment-test_2026-09-28-15-47.json | 10 + .../test/WorkspaceInputCaptureSharing.test.ts | 67 ++- .../operations/CacheableOperationPlugin.ts | 31 +- .../logic/operations/PhasedOperationPlugin.ts | 98 +++- .../operations/RetainedResultVerification.ts | 76 ++++ .../test/CacheableOperationPlugin.test.ts | 26 +- ...ableOperationPluginRetainedResults.test.ts | 414 +++++++++++++++++ ...asedOperationPluginRetainedResults.test.ts | 418 ++++++++++++++++++ 10 files changed, 1144 insertions(+), 16 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r07-retained-result-trust_2026-09-28-14-10.json create mode 100644 common/changes/@microsoft/rush/swarm-r07-verify-retained-results_2026-09-28-15-25.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r07-input-capture-environment-test_2026-09-28-15-47.json create mode 100644 libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts diff --git a/common/changes/@microsoft/rush/swarm-r07-retained-result-trust_2026-09-28-14-10.json b/common/changes/@microsoft/rush/swarm-r07-retained-result-trust_2026-09-28-14-10.json new file mode 100644 index 0000000000..f77be5bea1 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r07-retained-result-trust_2026-09-28-14-10.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "In long-lived operation graphs such as the Rush daemon, an operation that a request did not select no longer blocks build cache writes for its consumers if its retained result is trusted at its current state hash. A selected operation whose retained result is not trusted is now restored from the build cache or run again instead of being skipped, so it no longer stops cache writes for its consumers or keeps outputs that were built against dependencies that have since been rebuilt.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@microsoft/rush/swarm-r07-verify-retained-results_2026-09-28-15-25.json b/common/changes/@microsoft/rush/swarm-r07-verify-retained-results_2026-09-28-15-25.json new file mode 100644 index 0000000000..82bc56e33d --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r07-verify-retained-results_2026-09-28-15-25.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "In long-lived operation graphs such as the Rush daemon, a selected operation whose retained result was built against outputs of a dependency that have since been rebuilt is now run again instead of being skipped, even if build cache writes are disabled or the build cache is off.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r07-input-capture-environment-test_2026-09-28-15-47.json b/common/changes/@rushstack/rush-daemon/swarm-r07-input-capture-environment-test_2026-09-28-15-47.json new file mode 100644 index 0000000000..b1cf0aeb7c --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r07-input-capture-environment-test_2026-09-28-15-47.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Test that concurrent requests share workspace input captures only when their fingerprint environments are equal.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts b/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts index 2f8b3b6a27..23bce5cc3d 100644 --- a/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceInputCaptureSharing.test.ts @@ -5,7 +5,8 @@ jest.mock('@microsoft/rush-lib', () => { const actual: typeof import('@microsoft/rush-lib') = jest.requireActual('@microsoft/rush-lib'); return { ...actual, - captureProjectConfigurationFingerprintAsync: jest.fn(actual.captureProjectConfigurationFingerprintAsync) + captureProjectConfigurationFingerprintAsync: jest.fn(actual.captureProjectConfigurationFingerprintAsync), + captureWorkspaceInputFingerprintAsync: jest.fn(actual.captureWorkspaceInputFingerprintAsync) }; }); @@ -20,6 +21,8 @@ jest.setTimeout(60_000); const WAIT_TIMEOUT_MS: number = 20_000; const captureMock: jest.MockedFunction = jest.mocked(rushLib.captureProjectConfigurationFingerprintAsync); +const inputCaptureMock: jest.MockedFunction = + jest.mocked(rushLib.captureWorkspaceInputFingerprintAsync); async function waitForAsync(description: string, condition: () => boolean): Promise { const deadline: number = Date.now() + WAIT_TIMEOUT_MS; @@ -81,3 +84,65 @@ it('serves concurrent warm builds from one project configuration capture that st await fixture[Symbol.asyncDispose](); } }); + +it('shares input fingerprint captures only between requests whose fingerprint environments are equal', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => + setDaemonPolicy(created, {}) + ); + let release: () => void = () => {}; + const requests: jest.SpyInstance = jest.spyOn(FreshCaptureCoalescer.prototype, 'captureAsync'); + try { + await fixture.buildSuccessfullyAsync(); + await fixture.buildSuccessfullyAsync(); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + const generation: number = fixture.host.workspaceGeneration; + + // A variable that is part of the workspace fingerprint environment, unlike TERM and COLUMNS. + const variable: string = 'RUSHD_CAPTURE_SHARING_TEST'; + const capturedValues = (): (string | undefined)[] => + inputCaptureMock.mock.calls.map(([options]) => options.environment[variable]); + // The project configuration coalescer uses the empty key; input captures use environment keys. + const inputCaptureRequests = (): number => + requests.mock.calls.filter(([, key]: unknown[]) => key !== '').length; + requests.mockClear(); + inputCaptureMock.mockClear(); + const gate: Promise = new Promise((resolve) => (release = resolve)); + const actual: typeof rushLib.captureWorkspaceInputFingerprintAsync = jest.requireActual< + typeof rushLib + >('@microsoft/rush-lib').captureWorkspaceInputFingerprintAsync; + inputCaptureMock.mockImplementationOnce(async (...args) => { + await gate; + return await actual(...args); + }); + + const first: ReturnType = fixture.buildAsync(); + await waitForAsync('the first build is capturing', () => inputCaptureMock.mock.calls.length === 1); + const argv: string[] = ['build', '--to', 'b', '--parallelism', '3']; + const volatile: ReturnType = fixture.runAsync(argv, { + environment: { ...fixture.environment, TERM: 'dumb', COLUMNS: '91' } + }); + const other: ReturnType = fixture.runAsync(argv, { + environment: { ...fixture.environment, [variable]: 'other' } + }); + await waitForAsync('every build asked for an input capture', () => inputCaptureRequests() === 3); + // The request with another environment captured it at once. The request that differs only in volatile + // variables waits for the next capture that it can share. + expect(capturedValues()).toEqual([undefined, 'other']); + release(); + + const [firstResult, volatileResult, otherResult] = await Promise.all([first, volatile, other]); + expect(firstResult.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(volatileResult.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + // Only the request with another environment needs a new daemon process, which this host cannot launch. + expect(otherResult.terminal).toMatchObject({ + kind: 'requestRejected', + payload: { message: expect.stringContaining('A new daemon process is required (environment)') } + }); + expect(fixture.host.workspaceGeneration).toBe(generation); + expect(fixture.runs()).toEqual(['a', 'b']); + } finally { + requests.mockRestore(); + release(); + await fixture[Symbol.asyncDispose](); + } +}); diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index 55037496af..b3b3e10ea2 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -48,11 +48,14 @@ import type { } from '../../pluginFramework/PhasedCommandHooks'; import type { IOperationGraph, IOperationGraphIterationOptions } from './IOperationGraph'; import type { BuildCacheConfiguration } from '../../api/BuildCacheConfiguration'; -import type { IOperationExecutionResult } from './IOperationExecutionResult'; +import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; import type { OperationExecutionRecord } from './OperationExecutionRecord'; +import { enableUnverifiedRetainedOperations } from './RetainedResultVerification'; const PLUGIN_NAME: 'CacheablePhasedOperationPlugin' = 'CacheablePhasedOperationPlugin'; const PERIODIC_CALLBACK_INTERVAL_IN_SECONDS: number = 10; +// Runs after the default-stage taps (e.g. PhasedOperationPlugin) have decided which operations to enable. +const RE_ENABLE_UNTRUSTED_RESULTS_STAGE: number = 1; export interface IProjectDeps { files: { [filePath: string]: string }; @@ -201,6 +204,21 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { } this.#buildCacheContextByOperation.clear(); }); + graph.hooks.configureIteration.tap( + { name: PLUGIN_NAME, stage: RE_ENABLE_UNTRUSTED_RESULTS_STAGE }, + ( + currentStates: ReadonlyMap, + lastStates: ReadonlyMap, + iterationOptions: IOperationGraphIterationOptions + ) => { + // PhasedOperationPlugin re-verifies results that were built against unverified dependency outputs. + // This also re-verifies results that are not trusted for other reasons, e.g. because their input files + // changed while they were executing. Without cache writes, nothing is ever trusted. + if (buildCacheConfiguration.cacheWriteEnabled && iterationOptions.inputsSnapshot) { + enableUnverifiedRetainedOperations(currentStates, lastStates, trustedStateHashByOperation); + } + } + ); graph.hooks.beforeExecuteIterationAsync.tap( PLUGIN_NAME, ( @@ -692,13 +710,10 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { // Skipping generally means we cannot guarantee integrity, so prevent cache writes in dependents. // The exception is an operation that was not re-run because a previous iteration of this graph // produced a trusted result at exactly the same state hash (e.g. a result retained by a - // long-lived graph such as the Rush daemon). Since the state hash of an operation covers the - // state hashes of all of its dependencies, a consumer's cache key fully describes this input. - if ( - blockCacheWrite || - record.operation.enabled === false || - trustedStateHashByOperation.get(operation) !== record.getStateHash() - ) { + // long-lived graph such as the Rush daemon), whether or not this iteration selected it. Since the + // state hash of an operation covers the state hashes of all of its dependencies, a consumer's + // cache key fully describes this input. + if (blockCacheWrite || trustedStateHashByOperation.get(operation) !== record.getStateHash()) { blockCacheWrite = true; trustedStateHashByOperation.delete(operation); } diff --git a/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts index ac98661fc6..1584cba611 100644 --- a/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts @@ -17,8 +17,9 @@ import type { IOperationExecutionResult, IOperationStateHashComponents } from './IOperationExecutionResult'; -import { SUCCESS_STATUSES } from './OperationStatus'; +import { OperationStatus, SUCCESS_STATUSES } from './OperationStatus'; import type { IInputsSnapshot } from '../incremental/InputsSnapshot'; +import { enableUnverifiedRetainedOperations } from './RetainedResultVerification'; const PLUGIN_NAME: 'PhasedOperationPlugin' = 'PhasedOperationPlugin'; @@ -124,6 +125,18 @@ function createOperations( } function configureExecutionManager(graph: IOperationGraph, context: IOperationGraphContext): void { + // The state hash at which each operation last produced its outputs, or restored them from the build cache, + // in an iteration of this graph in which the outputs of all of its dependencies were verified. + const verifiedStateHashByOperation: Map = new Map(); + // The records of the executing iteration, if its state hashes are available. + let iterationRecords: ReadonlyMap | undefined; + + graph.hooks.beforeDeleteResults.tap(PLUGIN_NAME, (operations: ReadonlySet) => { + for (const operation of operations) { + verifiedStateHashByOperation.delete(operation); + } + }); + graph.hooks.configureIteration.tap( PLUGIN_NAME, ( @@ -132,8 +145,91 @@ function configureExecutionManager(graph: IOperationGraph, context: IOperationGr iterationOptions: IOperationGraphIterationOptions ) => { configureOperations(currentStates, lastStates, iterationOptions); + if (iterationOptions.inputsSnapshot) { + // A retained result that is current by state hash can still have been built against outputs of a + // dependency that were not current, e.g. by an `--only` request. + enableUnverifiedRetainedOperations(currentStates, lastStates, verifiedStateHashByOperation); + } + } + ); + + graph.hooks.beforeExecuteIterationAsync.tap( + PLUGIN_NAME, + ( + records: ReadonlyMap, + iterationOptions: IOperationGraphIterationOptions + ): void => { + if (iterationOptions.inputsSnapshot) { + iterationRecords = records; + } else { + // Without state hashes, nothing can be verified. + iterationRecords = undefined; + verifiedStateHashByOperation.clear(); + } } ); + + graph.hooks.afterExecuteOperationAsync.tap(PLUGIN_NAME, (record: IOperationExecutionResult) => { + if (iterationRecords) { + updateVerifiedStateHash(record, iterationRecords, verifiedStateHashByOperation); + } + }); + + graph.hooks.afterExecuteIterationAsync.tap(PLUGIN_NAME, (status: OperationStatus) => { + iterationRecords = undefined; + return status; + }); +} + +function updateVerifiedStateHash( + record: IOperationExecutionResult, + records: ReadonlyMap, + verifiedStateHashByOperation: Map +): void { + const { operation } = record; + switch (record.status) { + case OperationStatus.Skipped: { + // The outputs were left as they were. + return; + } + + case OperationStatus.FromCache: { + // The outputs were restored from the build cache entry for this state hash. + verifiedStateHashByOperation.set(operation, record.getStateHash()); + return; + } + + case OperationStatus.Success: + case OperationStatus.SuccessWithWarning: + case OperationStatus.NoOp: { + if (areDependenciesVerified(operation, records, verifiedStateHashByOperation)) { + verifiedStateHashByOperation.set(operation, record.getStateHash()); + return; + } + break; + } + + default: { + // The outputs may be incomplete. + break; + } + } + + verifiedStateHashByOperation.delete(operation); +} + +function areDependenciesVerified( + operation: Operation, + records: ReadonlyMap, + verifiedStateHashByOperation: ReadonlyMap +): boolean { + for (const dependency of operation.dependencies) { + const dependencyRecord: IOperationExecutionResult | undefined = records.get(dependency); + if (!dependencyRecord || verifiedStateHashByOperation.get(dependency) !== dependencyRecord.getStateHash()) { + return false; + } + } + return true; } function shouldEnableOperation( diff --git a/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts b/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts new file mode 100644 index 0000000000..e354e44463 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts @@ -0,0 +1,76 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { Operation } from './Operation'; +import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; +import { SUCCESS_STATUSES } from './OperationStatus'; + +/** + * Re-enables selected operations whose successful result retained by a previous iteration of a long-lived graph + * (e.g. the Rush daemon) is current by state hash, but not verified at that state hash, so that they are restored + * from the build cache or executed instead of being skipped. Such a result was produced while the outputs of one + * of its dependencies were not verified (e.g. that dependency was not selected and had changed), so its outputs + * may not match its state hash. Skipping it would keep outputs that were built against dependency outputs that + * have since been rebuilt. + * + * An operation is only re-enabled if none of its dependencies will remain unverified, since otherwise running it + * again would not produce a verified result either. An operation that is disabled for another reason than a + * current retained result, e.g. by a plugin that performs the work itself, is left disabled. + * + * @param records - The records of the iteration that is being configured + * @param lastStates - The results retained by previous iterations of the graph + * @param verifiedStateHashByOperation - The state hash at which the retained result of each operation is verified + */ +export function enableUnverifiedRetainedOperations( + records: ReadonlyMap, + lastStates: ReadonlyMap, + verifiedStateHashByOperation: ReadonlyMap +): void { + // Whether the result of each operation will still be unverified at the end of this iteration. + const remainsUnverifiedByOperation: Map = new Map(); + + function remainsUnverified(operation: Operation): boolean { + const known: boolean | undefined = remainsUnverifiedByOperation.get(operation); + if (known !== undefined) { + return known; + } + // Treat a dependency cycle as unverified; the real value is assigned below. + remainsUnverifiedByOperation.set(operation, true); + + let unverified: boolean = false; + for (const dependency of operation.dependencies) { + if (remainsUnverified(dependency)) { + unverified = true; + break; + } + } + + const record: IConfigurableOperation | undefined = records.get(operation); + if (!record) { + unverified = true; + } else if (!unverified && !record.enabled && !operation.isNoOp) { + // The operation will be skipped, which only keeps a verified result if it is verified at this state hash. + const stateHash: string = record.getStateHash(); + if (verifiedStateHashByOperation.get(operation) !== stateHash) { + const lastState: IOperationExecutionResult | undefined = lastStates.get(operation); + if ( + operation.enabled === true && + lastState && + SUCCESS_STATUSES.has(lastState.status) && + lastState.getStateHash() === stateHash + ) { + record.enabled = true; + } else { + unverified = true; + } + } + } + + remainsUnverifiedByOperation.set(operation, unverified); + return unverified; + } + + for (const operation of records.keys()) { + remainsUnverified(operation); + } +} diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts index ffdee0680e..6c7f1cb80f 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts @@ -276,7 +276,7 @@ describe(CacheableOperationPlugin.name, () => { expect(testGraph.cacheWrites).toEqual(['c']); }); - it('does not write a cache entry when a dependency was skipped by the user (e.g. --only)', async () => { + it('writes a cache entry when a dependency that was not selected (e.g. --only) is trusted at its state hash', async () => { const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); await testGraph.executeAsync(); @@ -284,6 +284,20 @@ describe(CacheableOperationPlugin.name, () => { testGraph.localHashes.set('b', 'b-v2'); const result: IExecutionResult = await testGraph.executeAsync(); + expect(getStatus(testGraph, result, 'a')).toBe(OperationStatus.Skipped); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual(['b']); + }); + + it('does not write a cache entry when a dependency that was not selected (e.g. --only) has changed', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); + await testGraph.executeAsync(); + + testGraph.operations.get('a')!.enabled = false; + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + const result: IExecutionResult = await testGraph.executeAsync(); + expect(getStatus(testGraph, result, 'a')).toBe(OperationStatus.Skipped); expect(testGraph.executions).toEqual(['b']); expect(testGraph.cacheWrites).toEqual([]); @@ -299,7 +313,7 @@ describe(CacheableOperationPlugin.name, () => { expect(testGraph.cacheWrites).toEqual([]); }); - it('does not trust a retained result that was produced while one of its dependencies was skipped', async () => { + it('re-executes a retained result that was produced while one of its dependencies was skipped', async () => { const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); const a: Operation = testGraph.operations.get('a')!; @@ -314,10 +328,10 @@ describe(CacheableOperationPlugin.name, () => { testGraph.localHashes.set('c', 'c-v2'); const result: IExecutionResult = await testGraph.executeAsync(); - expect(getStatus(testGraph, result, 'b')).toBe(OperationStatus.Skipped); - expect(testGraph.executions).toEqual(['a', 'c']); - // "b" was built against an unknown "a", so "c" must not write to the cache. - expect(testGraph.cacheWrites).toEqual(['a']); + // "b" was built against an unknown "a", so it must not be skipped now that "a" has executed. + expect(getStatus(testGraph, result, 'b')).toBe(OperationStatus.Success); + expect(testGraph.executions).toEqual(['a', 'b', 'c']); + expect(testGraph.cacheWrites).toEqual(['a', 'b', 'c']); }); it('does not trust results whose state hash changed without re-execution', async () => { diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts new file mode 100644 index 0000000000..4581d7c702 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts @@ -0,0 +1,414 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('../../../utilities/Utilities'); +jest.mock('../OperationStateFile'); +jest.mock('../ProjectLogWritable', () => { + const actual = jest.requireActual('../ProjectLogWritable'); + const { TerminalWritable } = jest.requireActual('@rushstack/terminal'); + class MockTerminalWritable extends TerminalWritable { + protected onWriteChunk(): void { + /* noop */ + } + protected onClose(): void { + /* noop */ + } + } + return { + ...actual, + initializeProjectLogFilesAsync: jest.fn(async () => new MockTerminalWritable()) + }; +}); +jest.mock('../OperationMetadataManager', () => { + class MockOperationMetadataManager { + public readonly logFilenameIdentifier: string; + public readonly metadataFolderPath: string = '.rush/temp/operation/mock'; + public readonly stateFile: { state: undefined } = { state: undefined }; + public constructor({ operation }: { operation: { logFilenameIdentifier: string } }) { + this.logFilenameIdentifier = operation.logFilenameIdentifier; + } + public async saveAsync(): Promise { + /* noop */ + } + public async tryRestoreAsync(): Promise { + /* noop */ + } + public tryRestoreStopwatch(originalStopwatch: T): T { + return originalStopwatch; + } + } + return { OperationMetadataManager: MockOperationMetadataManager }; +}); +jest.mock('../../buildCache/OperationBuildCache', () => ({ + OperationBuildCache: { forOperation: jest.fn() } +})); + +import { MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import type { BuildCacheConfiguration } from '../../../api/BuildCacheConfiguration'; +import type { RushProjectConfiguration } from '../../../api/RushProjectConfiguration'; +import { PhasedCommandHooks, type IOperationGraphContext } from '../../../pluginFramework/PhasedCommandHooks'; +import type { IInputsSnapshot } from '../../incremental/InputsSnapshot'; +import { OperationBuildCache } from '../../buildCache/OperationBuildCache'; +import { CacheableOperationPlugin } from '../CacheableOperationPlugin'; +import { PhasedOperationPlugin } from '../PhasedOperationPlugin'; +import { OperationGraph } from '../OperationGraph'; +import { Operation } from '../Operation'; +import { OperationStatus } from '../OperationStatus'; +import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; +import type { IExecutionResult } from '../IOperationExecutionResult'; +import type { OperationExecutionRecord } from '../OperationExecutionRecord'; + +const mockPhase: IPhase = { + name: 'phase', + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + dependencies: { self: new Set(), upstream: new Set() }, + isSynthetic: false, + logFilenameIdentifier: 'phase', + missingScriptBehavior: 'silent' +}; + +class CacheableMockRunner implements IOperationRunner { + public readonly reportTiming: boolean = true; + public readonly silent: boolean = false; + public readonly cacheable: boolean = true; + public readonly warningsAreAllowed: boolean = false; + public readonly isNoOp: boolean = false; + public readonly name: string; + readonly #executions: string[]; + + public constructor(name: string, executions: string[]) { + this.name = name; + this.#executions = executions; + } + + public async executeAsync(context: IOperationRunnerContext): Promise { + this.#executions.push(this.name); + return OperationStatus.Success; + } + + public getConfigHash(): string { + return 'config'; + } +} + +interface ITestGraph { + graph: OperationGraph; + operations: Map; + localHashes: Map; + executions: string[]; + cacheWrites: string[]; + cacheRestores: string[]; + executeAsync(): Promise; +} + +interface ITestGraphOptions { + /** + * The names of the dependencies of each operation. By default, each operation depends on the previous one. + */ + dependencies?: Record; + cacheWriteEnabled?: boolean; +} + +/** + * Creates a graph of cacheable operations. By default it is a linear chain: names[0] <- names[1] <- ... + * The mock build cache stores an entry per operation and state hash, and restores it if it exists. + */ +async function createTestGraphAsync(names: string[], options: ITestGraphOptions = {}): Promise { + const { dependencies, cacheWriteEnabled = true } = options; + const executions: string[] = []; + const cacheWrites: string[] = []; + const cacheRestores: string[] = []; + const cacheEntries: Set = new Set(); + const localHashes: Map = new Map(); + const operations: Map = new Map(); + const projectConfigurations: Map = new Map(); + + let previous: Operation | undefined; + for (const name of names) { + const project: RushConfigurationProject = { + packageName: name, + projectFolder: `/repo/${name}` + } as unknown as RushConfigurationProject; + projectConfigurations.set(project, { + getCacheDisabledReason: () => undefined + } as unknown as RushProjectConfiguration); + const operation: Operation = new Operation({ + runner: new CacheableMockRunner(name, executions), + logFilenameIdentifier: name, + phase: mockPhase, + project + }); + if (previous && !dependencies) { + operation.addDependency(previous); + } + previous = operation; + operations.set(name, operation); + localHashes.set(name, `${name}-v1`); + } + for (const [name, dependencyNames] of Object.entries(dependencies ?? {})) { + for (const dependencyName of dependencyNames) { + operations.get(name)!.addDependency(operations.get(dependencyName)!); + } + } + + jest.mocked(OperationBuildCache.forOperation).mockImplementation((record) => { + const name: string = record.operation.associatedProject.packageName; + const getCacheKey = (): string => `${name}@${record.getStateHash()}`; + return { + tryRestoreFromCacheAsync: async () => { + const restored: boolean = cacheEntries.has(getCacheKey()); + if (restored) { + cacheRestores.push(name); + } + return restored; + }, + trySetCacheEntryAsync: async () => { + cacheWrites.push(name); + cacheEntries.add(getCacheKey()); + return true; + } + } as unknown as OperationBuildCache; + }); + + const terminal: Terminal = new Terminal(new StringBufferTerminalProvider()); + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new PhasedOperationPlugin().apply(hooks); + new CacheableOperationPlugin({ + allowWarningsInSuccessfulBuild: false, + buildCacheConfiguration: { + buildCacheEnabled: true, + cacheWriteEnabled + } as unknown as BuildCacheConfiguration, + cobuildConfiguration: undefined, + terminal, + excludeAppleDoubleFiles: false, + useDirectFileTransfersForBuildCache: false + }).apply(hooks); + + const graph: OperationGraph = new OperationGraph(new Set(operations.values()), { + quietMode: true, + debugMode: false, + parallelism: 1, + allowOversubscription: true, + destinations: [new MockWritable()], + abortController: new AbortController() + }); + await hooks.onGraphCreatedAsync.promise(graph, { + isIncrementalBuildAllowed: true, + projectConfigurations + } as unknown as IOperationGraphContext); + + const inputsSnapshot: IInputsSnapshot = { + hashes: new Map(), + rootDirectory: '/repo', + hasUncommittedChanges: false, + getTrackedFileHashesForOperation: () => new Map(), + getOperationOwnStateHash: (project: RushConfigurationProject) => localHashes.get(project.packageName)! + }; + + return { + graph, + operations, + localHashes, + executions, + cacheWrites, + cacheRestores, + executeAsync: async () => { + executions.length = 0; + cacheWrites.length = 0; + cacheRestores.length = 0; + return await graph.executeAsync({ inputsSnapshot }); + } + }; +} + +function getStatus(testGraph: ITestGraph, result: IExecutionResult, name: string): OperationStatus { + return (result.operationResults.get(testGraph.operations.get(name)!) as OperationExecutionRecord).status; +} + +// How results retained by earlier iterations of a long-lived graph (e.g. the Rush daemon) are trusted. +describe(`${CacheableOperationPlugin.name} retained results`, () => { + it('resumes cache writes for consumers after an --only request', async () => { + // "lib" <- "tool" <- "app", like @rushstack/node-core-library <- @rushstack/ts-command-line <- @rushstack/heft + const testGraph: ITestGraph = await createTestGraphAsync(['lib', 'tool', 'app']); + const lib: Operation = testGraph.operations.get('lib')!; + const app: Operation = testGraph.operations.get('app')!; + await testGraph.executeAsync(); + + // S1: edit "app", then --to app + testGraph.localHashes.set('app', 'app-v2'); + await testGraph.executeAsync(); + expect(testGraph.cacheWrites).toEqual(['app']); + + // S2: edit "tool", then --only tool + testGraph.localHashes.set('tool', 'tool-v2'); + lib.enabled = false; + app.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['tool']); + expect(testGraph.cacheWrites).toEqual(['tool']); + + // S3: --to app + lib.enabled = true; + app.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['app']); + expect(testGraph.cacheWrites).toEqual(['app']); + + // S4: edit "app", then --to app + testGraph.localHashes.set('app', 'app-v3'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['app']); + expect(testGraph.cacheWrites).toEqual(['app']); + }); + + it('keeps trusting operations that a request did not select', async () => { + // "a" <- "b" <- "c", and "a" <- "tool" + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c', 'tool'], { + dependencies: { b: ['a'], c: ['b'], tool: ['a'] } + }); + const { operations } = testGraph; + await testGraph.executeAsync(); + + for (let round: number = 2; round <= 3; round++) { + // --to tool + testGraph.localHashes.set('tool', `tool-v${round}`); + operations.get('b')!.enabled = false; + operations.get('c')!.enabled = false; + operations.get('tool')!.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['tool']); + expect(testGraph.cacheWrites).toEqual(['tool']); + + // --to c + testGraph.localHashes.set('c', `c-v${round}`); + operations.get('b')!.enabled = true; + operations.get('c')!.enabled = true; + operations.get('tool')!.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + expect(testGraph.cacheWrites).toEqual(['c']); + } + }); + + it('re-executes a retained result that was built against a dependency that has since been rebuilt', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + + // --only b, after editing both: "b" is built against the outputs of the previous "a". + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual([]); + + // --to b: "b" has the same state hash as its retained result, but "a" is rebuilt. + a.enabled = true; + const result: IExecutionResult = await testGraph.executeAsync(); + expect(getStatus(testGraph, result, 'b')).toBe(OperationStatus.Success); + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual(['a', 'b']); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('does not re-execute an untrusted retained result while a dependency that was not selected is untrusted', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); + testGraph.operations.get('a')!.enabled = false; + + // Cold --only b: "a" has never executed in this graph. + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual([]); + + // Repeating the request cannot produce a trusted result for "b", so it is skipped. + const result: IExecutionResult = await testGraph.executeAsync(); + expect(result.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + expect(testGraph.cacheWrites).toEqual([]); + }); + + it('restores an untrusted retained result from the build cache when an entry exists', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + + // --only b, after editing "a": "b" is built against the outputs of the previous "a". + testGraph.localHashes.set('a', 'a-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual([]); + + // Revert "a", then --to b: both have entries from the first iteration. + testGraph.localHashes.set('a', 'a-v1'); + a.enabled = true; + const result: IExecutionResult = await testGraph.executeAsync(); + expect(getStatus(testGraph, result, 'a')).toBe(OperationStatus.FromCache); + expect(getStatus(testGraph, result, 'b')).toBe(OperationStatus.FromCache); + expect(testGraph.executions).toEqual([]); + expect(testGraph.cacheRestores).toEqual(['a', 'b']); + + // Both are trusted again. + testGraph.localHashes.set('b', 'b-v2'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual(['b']); + }); + + it('does not re-execute untrusted retained results when cache writes are disabled', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b'], { cacheWriteEnabled: false }); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + testGraph.localHashes.set('b', 'b-v2'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual([]); + }); + + it('re-executes a retained result that was built against a dependency that has since been rebuilt when cache writes are disabled', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b'], { cacheWriteEnabled: false }); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + + // --only b, after editing both + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b + a.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual([]); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('does not re-enable operations that another plugin disabled', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); + // Like a plugin that performs the work itself. This tap runs after PhasedOperationPlugin's. + testGraph.graph.hooks.configureIteration.tap('TestPlugin', (records) => { + for (const record of records.values()) { + record.enabled = false; + } + }); + + const result: IExecutionResult = await testGraph.executeAsync(); + expect(result.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); +}); diff --git a/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts new file mode 100644 index 0000000000..325a1ea315 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts @@ -0,0 +1,418 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('../../../utilities/Utilities'); +jest.mock('../OperationStateFile'); +jest.mock('../ProjectLogWritable', () => { + const actual = jest.requireActual('../ProjectLogWritable'); + const { TerminalWritable } = jest.requireActual('@rushstack/terminal'); + class MockTerminalWritable extends TerminalWritable { + protected onWriteChunk(): void { + /* noop */ + } + protected onClose(): void { + /* noop */ + } + } + return { + ...actual, + initializeProjectLogFilesAsync: jest.fn(async () => new MockTerminalWritable()) + }; +}); + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import { PhasedCommandHooks, type IOperationGraphContext } from '../../../pluginFramework/PhasedCommandHooks'; +import type { IInputsSnapshot } from '../../incremental/InputsSnapshot'; +import { LegacySkipPlugin } from '../LegacySkipPlugin'; +import { PhasedOperationPlugin } from '../PhasedOperationPlugin'; +import { OperationGraph } from '../OperationGraph'; +import { Operation } from '../Operation'; +import { OperationStatus } from '../OperationStatus'; +import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; +import type { IExecutionResult } from '../IOperationExecutionResult'; + +const mockPhase: IPhase = { + name: 'phase', + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + dependencies: { self: new Set(), upstream: new Set() }, + isSynthetic: false, + logFilenameIdentifier: 'phase', + missingScriptBehavior: 'silent' +}; + +class MockRunner implements IOperationRunner { + public readonly reportTiming: boolean = true; + public readonly silent: boolean = false; + public readonly cacheable: boolean = true; + public readonly warningsAreAllowed: boolean = false; + public readonly name: string; + public readonly isNoOp: boolean; + readonly #executions: string[]; + + public constructor(name: string, isNoOp: boolean, executions: string[]) { + this.name = name; + this.isNoOp = isNoOp; + this.#executions = executions; + } + + public async executeAsync(context: IOperationRunnerContext): Promise { + if (this.isNoOp) { + return OperationStatus.NoOp; + } + this.#executions.push(this.name); + return OperationStatus.Success; + } + + public getConfigHash(): string { + return 'config'; + } +} + +interface ITestGraph { + graph: OperationGraph; + operations: Map; + localHashes: Map; + executions: string[]; + executeAsync(): Promise; +} + +interface ITestGraphOptions { + /** + * Operations that have no work. + */ + noOps?: ReadonlySet; + /** + * If set, the incremental state files of the legacy skip detection are stored in this folder. + */ + legacySkipFolder?: string; +} + +/** + * Creates a graph from the names of the dependencies of each operation, without the build cache. + */ +async function createTestGraphAsync( + dependencies: Record, + options: ITestGraphOptions = {} +): Promise { + const { noOps, legacySkipFolder } = options; + const executions: string[] = []; + const localHashes: Map = new Map(); + const operations: Map = new Map(); + + for (const name of Object.keys(dependencies)) { + const projectFolder: string = path.join(legacySkipFolder ?? '/repo', name); + const project: RushConfigurationProject = { + packageName: name, + projectFolder, + projectRushTempFolder: projectFolder + } as unknown as RushConfigurationProject; + const operation: Operation = new Operation({ + runner: new MockRunner(name, !!noOps?.has(name), executions), + logFilenameIdentifier: name, + phase: mockPhase, + project + }); + operations.set(name, operation); + localHashes.set(name, `${name}-v1`); + } + for (const [name, dependencyNames] of Object.entries(dependencies)) { + for (const dependencyName of dependencyNames) { + operations.get(name)!.addDependency(operations.get(dependencyName)!); + } + } + + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new PhasedOperationPlugin().apply(hooks); + if (legacySkipFolder) { + new LegacySkipPlugin({ + allowWarningsInSuccessfulBuild: false, + terminal: new Terminal(new StringBufferTerminalProvider()), + changedProjectsOnly: false, + isIncrementalBuildAllowed: true + }).apply(hooks); + } + + const graph: OperationGraph = new OperationGraph(new Set(operations.values()), { + quietMode: true, + debugMode: false, + parallelism: 1, + allowOversubscription: true, + destinations: [new MockWritable()], + abortController: new AbortController() + }); + await hooks.onGraphCreatedAsync.promise(graph, { + isIncrementalBuildAllowed: true, + projectConfigurations: new Map() + } as unknown as IOperationGraphContext); + + const inputsSnapshot: IInputsSnapshot = { + hashes: new Map(), + rootDirectory: '/repo', + hasUncommittedChanges: false, + getTrackedFileHashesForOperation: (project: RushConfigurationProject) => + new Map([[`${project.packageName}/src/index.ts`, localHashes.get(project.packageName)!]]), + getOperationOwnStateHash: (project: RushConfigurationProject) => localHashes.get(project.packageName)! + }; + + return { + graph, + operations, + localHashes, + executions, + executeAsync: async () => { + executions.length = 0; + return await graph.executeAsync({ inputsSnapshot }); + } + }; +} + +// How results retained by earlier iterations of a long-lived graph (e.g. the Rush daemon) are verified, +// whether or not the build cache is in use. +describe(`${PhasedOperationPlugin.name} retained results`, () => { + it('re-executes a retained result that was built against a dependency that has since been rebuilt', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + + // --only b, after editing both: "b" is built against the outputs of the previous "a". + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b: "b" has the same state hash as its retained result, but "a" is rebuilt. + a.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('re-executes every retained result in a chain that was built against unverified outputs', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'], c: ['b'] }); + const a: Operation = testGraph.operations.get('a')!; + const b: Operation = testGraph.operations.get('b')!; + const c: Operation = testGraph.operations.get('c')!; + await testGraph.executeAsync(); + + // --only b, then --only c, after editing "a" and "b" + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + c.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + b.enabled = false; + c.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + + // --to c + a.enabled = true; + b.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b', 'c']); + }); + + it('verifies results through operations without work', async () => { + const testGraph: ITestGraph = await createTestGraphAsync( + { a: [], n: ['a'], b: ['n'] }, + { noOps: new Set(['n']) } + ); + const a: Operation = testGraph.operations.get('a')!; + const n: Operation = testGraph.operations.get('n')!; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + // --only b, after editing "a" and "b" + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + n.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b + a.enabled = true; + n.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('does not re-execute a retained result that was built against unchanged dependencies that were not selected', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + + // --only b, after editing "b" + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b + a.enabled = true; + const result: IExecutionResult = await testGraph.executeAsync(); + expect(result.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('does not re-execute a retained result while a dependency that was not selected is unverified', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); + testGraph.operations.get('a')!.enabled = false; + + // Cold --only b: "a" has never executed in this graph. + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // Repeating the request cannot produce a verified result for "b", so it is skipped. + const result: IExecutionResult = await testGraph.executeAsync(); + expect(result.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('re-executes a consumer whose dependency was rebuilt by a request that did not select it', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ b: [], c: ['b'] }); + const c: Operation = testGraph.operations.get('c')!; + await testGraph.executeAsync(); + + // --to b, after editing "b" + testGraph.localHashes.set('b', 'b-v2'); + c.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to c + c.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + }); + + it('does not re-execute retained results of operations that ignore dependency changes', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); + const a: Operation = testGraph.operations.get('a')!; + const b: Operation = testGraph.operations.get('b')!; + await testGraph.executeAsync(); + + // --only b, after editing both + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b --changed-projects-only + a.enabled = 'ignore-dependency-changes'; + b.enabled = 'ignore-dependency-changes'; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a']); + }); + + it('does not re-enable operations that a later configureIteration tap disabled', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + + // --only b, after editing both + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // Like a plugin that performs the work itself. + testGraph.graph.hooks.configureIteration.tap({ name: 'TestPlugin', stage: 1 }, (records) => { + for (const record of records.values()) { + record.enabled = false; + } + }); + a.enabled = true; + const result: IExecutionResult = await testGraph.executeAsync(); + expect(result.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + describe('with legacy skip detection', () => { + let legacySkipFolder: string; + + beforeEach(() => { + legacySkipFolder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-lib-legacy-skip-')); + }); + + afterEach(() => { + fs.rmSync(legacySkipFolder, { recursive: true, force: true }); + }); + + it('re-executes a retained result that was built against a dependency that has since been rebuilt', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder }); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + // --only b, after editing both + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b + a.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('re-executes a consumer whose dependency was rebuilt by a request that did not select it', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ b: [], c: ['b'] }, { legacySkipFolder }); + const c: Operation = testGraph.operations.get('c')!; + await testGraph.executeAsync(); + + // --to b, after editing "b" + testGraph.localHashes.set('b', 'b-v2'); + c.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to c: the files of "c" did not change, and "b" does not execute. + c.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + }); + + it('still skips an unverified retained result if no dependency executes', async () => { + // Build once in another process, then start a long-lived graph. + await (await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder })).executeAsync(); + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder }); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual([]); + + // --to b, after editing "b": "a" is skipped, so it is not verified in this graph. + testGraph.localHashes.set('b', 'b-v2'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + + // --to b: the legacy skip detection still skips "b". + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual([]); + }); + }); +}); From f3cc6afdf5f62f988cf49fd9d487eb9e75526024 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 17:15:47 +0000 Subject: [PATCH 027/265] [rush-cli-client] Show engine and plugin warnings in agent output; keep the daemon alive when git hash-object fails Swarm integration step 12; original commit cc5c6aab7e (merge of swarm/r01 at 575cbf5698). Refs: board 601, task 80, board 899. ch01 GATE OK board 1272 and board 1273 (gated tree b691997c35). Commits folded into this step (3): - c11ee10320 [rush-cli-client] Show engine and plugin warnings in agent output - c77aa1dbef [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 575cbf5698 [package-deps-hash] Name the git command that failed in the error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 4 +- apps/rush-cli-client/src/AgentNotices.ts | 51 ++++++++++ .../src/AgentProgressRenderer.ts | 12 ++- .../src/test/AgentNotices.test.ts | 97 +++++++++++++++++++ ...01-git-error-command_2026-09-28-15-02.json | 11 +++ ...ash-object-rejection_2026-09-28-14-47.json | 11 +++ ...tput-engine-warnings_2026-09-28-13-52.json | 10 ++ ...tput-engine-warnings_2026-09-28-13-52.json | 10 ++ ...tput-engine-warnings_2026-09-28-13-52.json | 10 ++ .../reviews/api/rush-daemon-protocol.api.md | 1 + .../package-deps-hash/src/getRepoState.ts | 13 ++- .../src/test/getRepoDeps.test.ts | 76 ++++++++++++++- .../src/DaemonOperationPayloads.ts | 6 ++ .../rush-daemon/src/EngineActivityOptions.ts | 32 ++++++ .../rush-daemon/src/EngineTerminalProvider.ts | 5 +- .../rush-daemon/src/PhasedRequestEventSink.ts | 25 ++--- .../src/test/EngineActivity.test.ts | 90 +++++++++++++++++ 17 files changed, 441 insertions(+), 23 deletions(-) create mode 100644 apps/rush-cli-client/src/AgentNotices.ts create mode 100644 apps/rush-cli-client/src/test/AgentNotices.test.ts create mode 100644 common/changes/@rushstack/package-deps-hash/swarm-r01-git-error-command_2026-09-28-15-02.json create mode 100644 common/changes/@rushstack/package-deps-hash/swarm-r01-hash-object-rejection_2026-09-28-14-47.json create mode 100644 common/changes/@rushstack/rush-cli-client/agent-output-engine-warnings_2026-09-28-13-52.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/agent-output-engine-warnings_2026-09-28-13-52.json create mode 100644 common/changes/@rushstack/rush-daemon/agent-output-engine-warnings_2026-09-28-13-52.json create mode 100644 libraries/rush-daemon/src/EngineActivityOptions.ts create mode 100644 libraries/rush-daemon/src/test/EngineActivity.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index f651b98bab..1fb55ae3fb 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -98,7 +98,9 @@ use `--reporter=ai` for machine-parsed records. It writes a first status line be `@microsoft/rush-lib` is loaded, then at most three live rows on a TTY (append-only lines throttled to one per 2 seconds on a pipe), the queue position when waiting for admission, and always one final summary line (`rush build: SUCCESS 12/12 operations (...) in 3.1s`, or -`up to date (no operations needed)`). On failure, it lists failed operations and a +`up to date (no operations needed)`). Warnings and errors that Rush or a Rush plugin writes +outside any operation (for example a plugin that continues without the cloud build cache) +precede the summary line, at most three lines of them. On failure, it lists failed operations and a bounded tail (10 lines) of their stderr, or of their stdout when they wrote no stderr. Operation logs are otherwise not printed; use `RUSHD_OUTPUT=legacy` for full logs. When a request falls back to in-process Rush, agent mode stops and native output follows. diff --git a/apps/rush-cli-client/src/AgentNotices.ts b/apps/rush-cli-client/src/AgentNotices.ts new file mode 100644 index 0000000000..154f44ff47 --- /dev/null +++ b/apps/rush-cli-client/src/AgentNotices.ts @@ -0,0 +1,51 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// Keep this module free of heavy imports: start.ts loads the agent renderer before @microsoft/rush-lib. + +/** The most warning and error lines printed before the summary line. */ +const MAX_NOTICE_LINES: number = 3; +/** A longer line is cut to this many characters. */ +const MAX_NOTICE_LENGTH: number = 300; + +function clip(line: string): string { + return line.length > MAX_NOTICE_LENGTH ? `${line.slice(0, MAX_NOTICE_LENGTH - 1)}…` : line; +} + +/** + * Collects the warnings and errors that Rush or a Rush plugin wrote while the daemon engine loaded or ran, for + * example a plugin that continues without the cloud build cache, or a daemon-compatible plugin name that + * matches no configured plugin. They arrive as `activityChanged` events with a `severity` and no operation + * scope. Agent output shows other activity only in its live row, which a pipe never prints. + */ +export class AgentNotices { + readonly #lines: Set = new Set(); + + /** Records the lines of an activity payload that is a warning or an error outside any operation. */ + public add(payload: Readonly>, operationId: string | undefined): void { + const { severity, text } = payload; + if (operationId !== undefined || typeof text !== 'string') { + return; + } + if (severity !== 'warning' && severity !== 'error') { + return; + } + for (const line of text.split('\n')) { + const trimmed: string = line.trim(); + if (trimmed) { + this.#lines.add(trimmed); + } + } + } + + /** The first distinct lines, each cut to a maximum length, followed by the number of lines left out. */ + public getLines(): string[] { + const lines: string[] = [...this.#lines]; + const shown: string[] = lines.slice(0, MAX_NOTICE_LINES).map(clip); + const omitted: number = lines.length - shown.length; + if (omitted > 0) { + shown.push(`+${omitted} more warning and error lines; RUSHD_OUTPUT=legacy prints them all`); + } + return shown; + } +} diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index 08fa108796..87e8a41d97 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -6,6 +6,8 @@ import type { IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; +import { AgentNotices } from './AgentNotices'; + const SPINNER_FRAMES: readonly string[] = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']; const TERMINAL_STATUSES: ReadonlySet = new Set([ 'SUCCESS', @@ -52,6 +54,7 @@ export class AgentProgressRenderer { readonly #failed: string[] = []; readonly #stderrTails: Map = new Map(); readonly #stdoutTails: Map = new Map(); + readonly #notices: AgentNotices = new AgentNotices(); #total: number = 0; #done: number = 0; #lastActivity: string = ''; @@ -135,6 +138,7 @@ export class AgentProgressRenderer { break; } case 'activityChanged': { + this.#notices.add(payload, event.scope?.operationId); if (typeof payload.text === 'string' && payload.text.trim()) { this.#lastActivity = payload.text.trim().split('\n')[0]; if (this.#phase !== 'running') { @@ -179,11 +183,17 @@ export class AgentProgressRenderer { this.#stop(); } - /** Stops the live region and writes the final summary line, at most once. */ + /** + * Stops the live region and writes the final summary line, at most once. Warnings and errors that Rush or a + * plugin wrote outside any operation precede it. + */ public finish(result: IAgentFinalResult | undefined): void { if (!this.#stop()) { return; } + for (const notice of this.#notices.getLines()) { + this.#options.write(`${notice}\n`); + } const succeeded: boolean = result !== undefined && result.exitCode === 0; const total: number = this.#getTotal(); const parts: string[] = [...this.#counts].map(([status, count]) => `${count} ${status.toLowerCase()}`); diff --git a/apps/rush-cli-client/src/test/AgentNotices.test.ts b/apps/rush-cli-client/src/test/AgentNotices.test.ts new file mode 100644 index 0000000000..5a3f96995e --- /dev/null +++ b/apps/rush-cli-client/src/test/AgentNotices.test.ts @@ -0,0 +1,97 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { + DAEMON_PROTOCOL_VERSION, + type IDaemonActivityPayload, + type IDaemonEventEnvelope +} from '@rushstack/rush-daemon-protocol'; + +import { AgentNotices } from '../AgentNotices'; +import { AgentProgressRenderer } from '../AgentProgressRenderer'; + +const UNMATCHED_PLUGIN_WARNING: string = + "Warning: the daemon's compatible plugin list names plugins that are not configured in rush-plugins.json: typo-plugin\n"; +const CACHE_WARNING: string = + 'The Rush daemon never signs in interactively, so this build continues without signing in to the cloud build cache.\n'; + +function activity(payload: IDaemonActivityPayload, operationId?: string): IDaemonEventEnvelope { + return { + eventId: 'event', + sessionId: 'session', + sequence: 1, + timestamp: new Date().toISOString(), + protocolVersion: DAEMON_PROTOCOL_VERSION, + source: { packageName: 'test', packageVersion: '1.0.0' }, + privacy: 'public', + required: true, + type: 'activityChanged', + scope: operationId === undefined ? undefined : { operationId }, + payload + }; +} + +function createRenderer(isTTY: boolean): { renderer: AgentProgressRenderer; output: string[] } { + const output: string[] = []; + const renderer: AgentProgressRenderer = new AgentProgressRenderer({ + commandName: 'build', + isTTY, + columns: 60, + write: (text: string) => output.push(text), + now: () => 0, + startTimeMs: 0 + }); + return { renderer, output }; +} + +describe(AgentNotices.name, () => { + it('keeps the distinct lines of warnings and errors outside any operation', () => { + const notices: AgentNotices = new AgentNotices(); + notices.add({ severity: 'warning', stream: 'stderr', text: UNMATCHED_PLUGIN_WARNING }, undefined); + notices.add({ severity: 'warning', stream: 'stderr', text: UNMATCHED_PLUGIN_WARNING }, undefined); + notices.add({ severity: 'error', stream: 'stderr', text: '\nFirst line\n second line \n' }, undefined); + notices.add({ stream: 'stderr', text: 'Operations failed.\n' }, undefined); + notices.add({ stream: 'stdout', text: 'FSTrace: Enabled\n' }, undefined); + notices.add({ severity: 'warning', stream: 'stderr', text: 'operation warning\n' }, 'a (build)'); + expect(notices.getLines()).toEqual([UNMATCHED_PLUGIN_WARNING.trim(), 'First line', 'second line']); + }); + + it('prints at most three lines, each cut to 300 characters, and counts the rest', () => { + const notices: AgentNotices = new AgentNotices(); + notices.add({ severity: 'warning', text: `${'x'.repeat(400)}\nb\nc\nd\ne\n` }, undefined); + const lines: string[] = notices.getLines(); + expect(lines).toEqual([ + `${'x'.repeat(299)}…`, + 'b', + 'c', + '+2 more warning and error lines; RUSHD_OUTPUT=legacy prints them all' + ]); + }); +}); + +describe(`${AgentProgressRenderer.name} notices`, () => { + it.each([false, true])('prints plugin warnings before the summary line (isTTY: %s)', (isTTY: boolean) => { + const { renderer, output } = createRenderer(isTTY); + renderer.start(); + renderer.onEvent(activity({ severity: 'warning', stream: 'stderr', text: UNMATCHED_PLUGIN_WARNING })); + renderer.onEvent(activity({ severity: 'warning', stream: 'stderr', text: CACHE_WARNING })); + renderer.onEvent(activity({ stream: 'stdout', text: 'rush build (0.50 seconds)\n' })); + const beforeFinish: number = output.length; + renderer.finish({ exitCode: 0 }); + expect(output.slice(beforeFinish).filter((text: string) => !text.startsWith('\x1b'))).toEqual([ + UNMATCHED_PLUGIN_WARNING, + CACHE_WARNING, + 'rush build: SUCCESS up to date (no operations needed) in 0.0s\n' + ]); + }); + + it('prints nothing more when no warning or error was written', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(activity({ stream: 'stderr', text: 'Operations succeeded with warnings.\n' })); + const beforeFinish: number = output.length; + renderer.finish({ exitCode: 0 }); + expect(output.slice(beforeFinish)).toEqual([ + 'rush build: SUCCESS up to date (no operations needed) in 0.0s\n' + ]); + }); +}); diff --git a/common/changes/@rushstack/package-deps-hash/swarm-r01-git-error-command_2026-09-28-15-02.json b/common/changes/@rushstack/package-deps-hash/swarm-r01-git-error-command_2026-09-28-15-02.json new file mode 100644 index 0000000000..e460c63b55 --- /dev/null +++ b/common/changes/@rushstack/package-deps-hash/swarm-r01-git-error-command_2026-09-28-15-02.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/package-deps-hash", + "comment": "Name the Git command that failed (for example `git hash-object`) in the error from `getRepoStateAsync` and `getDetailedRepoStateAsync`, instead of `git --no-optional-locks`.", + "type": "patch" + } + ], + "packageName": "@rushstack/package-deps-hash", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/package-deps-hash/swarm-r01-hash-object-rejection_2026-09-28-14-47.json b/common/changes/@rushstack/package-deps-hash/swarm-r01-hash-object-rejection_2026-09-28-14-47.json new file mode 100644 index 0000000000..31552615ca --- /dev/null +++ b/common/changes/@rushstack/package-deps-hash/swarm-r01-hash-object-rejection_2026-09-28-14-47.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/package-deps-hash", + "comment": "Fix an unhandled promise rejection in `getDetailedRepoStateAsync` when `git hash-object` fails before `git ls-files` and `git status` finish, for example on an additional path that does not exist. The error now reaches the caller instead of ending the process.", + "type": "patch" + } + ], + "packageName": "@rushstack/package-deps-hash", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/agent-output-engine-warnings_2026-09-28-13-52.json b/common/changes/@rushstack/rush-cli-client/agent-output-engine-warnings_2026-09-28-13-52.json new file mode 100644 index 0000000000..995e8af912 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-output-engine-warnings_2026-09-28-13-52.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, print the warnings and errors that Rush or a Rush plugin wrote outside any operation before the summary line (at most three lines), instead of showing them only in the live row.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/agent-output-engine-warnings_2026-09-28-13-52.json b/common/changes/@rushstack/rush-daemon-protocol/agent-output-engine-warnings_2026-09-28-13-52.json new file mode 100644 index 0000000000..410d064d19 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/agent-output-engine-warnings_2026-09-28-13-52.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add an optional `severity` to `activityChanged` payloads, set on the warnings and errors that Rush or a Rush plugin wrote while the engine loaded or ran.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol" +} diff --git a/common/changes/@rushstack/rush-daemon/agent-output-engine-warnings_2026-09-28-13-52.json b/common/changes/@rushstack/rush-daemon/agent-output-engine-warnings_2026-09-28-13-52.json new file mode 100644 index 0000000000..c4722622fe --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/agent-output-engine-warnings_2026-09-28-13-52.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Mark the warnings and errors that Rush or a Rush plugin writes through the engine terminal with a `severity` in their activity events.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index 0584bcb366..a00763848c 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -216,6 +216,7 @@ export const FRAME_HEADER_BYTES: number; // @beta export interface IDaemonActivityPayload { + readonly severity?: 'warning' | 'error'; readonly stream?: 'stdout' | 'stderr'; readonly text: string; } diff --git a/libraries/package-deps-hash/src/getRepoState.ts b/libraries/package-deps-hash/src/getRepoState.ts index 45cb07a8dd..e88cf1900c 100644 --- a/libraries/package-deps-hash/src/getRepoState.ts +++ b/libraries/package-deps-hash/src/getRepoState.ts @@ -363,7 +363,9 @@ async function spawnGitAsync( if (status !== 0) { ensureGitMinimumVersion(gitPath); - throw new Error(`git ${args[0]} exited with code ${status}:\n${stderr}`); + // Name the git command itself, not the first of the STANDARD_GIT_OPTIONS in front of it + const command: string | undefined = args.find((arg: string) => !STANDARD_GIT_OPTIONS.includes(arg)); + throw new Error(`git ${command} exited with code ${status}:\n${stderr}`); } return stdout; @@ -554,13 +556,16 @@ export async function getDetailedRepoStateAsync( gitPath ); - const [{ files, symlinks, submodules }, locallyModifiedFiles] = await Promise.all([ + // Await all three at once. `git hash-object` can fail before the other two finish, for example on an additional + // path that does not exist. Its rejection needs a handler right away, or Node reports it as unhandled and exits. + const [{ files, symlinks, submodules }, locallyModifiedFiles, hashObjectResult] = await Promise.all([ statePromise, - locallyModifiedPromise + locallyModifiedPromise, + hashObjectPromise ]); // The result of "git hash-object" will be a list of file hashes delimited by newlines - for (const [filePath, hash] of await hashObjectPromise) { + for (const [filePath, hash] of hashObjectResult) { files.set(filePath, hash); } diff --git a/libraries/package-deps-hash/src/test/getRepoDeps.test.ts b/libraries/package-deps-hash/src/test/getRepoDeps.test.ts index c78e034386..2694e2dbe6 100644 --- a/libraries/package-deps-hash/src/test/getRepoDeps.test.ts +++ b/libraries/package-deps-hash/src/test/getRepoDeps.test.ts @@ -2,7 +2,7 @@ // See LICENSE in the project root for license information. import * as path from 'node:path'; -import { execSync, type SpawnSyncReturns } from 'node:child_process'; +import { execSync, type ChildProcess, type SpawnSyncReturns } from 'node:child_process'; import { getDetailedRepoStateAsync, @@ -12,7 +12,7 @@ import { parseGitHashObject } from '../getRepoState'; -import { Executable, FileSystem } from '@rushstack/node-core-library'; +import { Executable, FileSystem, type IExecutableSpawnOptions } from '@rushstack/node-core-library'; const SOURCE_PATH: string = path .join(__dirname) @@ -326,4 +326,76 @@ describe(getDetailedRepoStateAsync.name, () => { FileSystem.deleteFile(tempFilePath1); } }); + + it('rejects when git hash-object fails, without waiting for git ls-files and git status', async () => { + // `git hash-object` fails fast on an additional path that does not exist. Its promise used to be awaited only + // after `git ls-files` and `git status` finished, so until then its rejection had no handler. Under Node's + // default `--unhandled-rejections=throw`, that ended the process (a Rush daemon, in the reported crash). + // + // To make the ordering deterministic, the 'close' events of ls-files and status are held back until the end. + let releaseHeldProcesses: () => void = () => {}; + const heldProcessesReleased: Promise = new Promise((resolve) => { + releaseHeldProcesses = resolve; + }); + let onHashObjectClosed: () => void = () => {}; + const hashObjectClosed: Promise = new Promise((resolve) => { + onHashObjectClosed = resolve; + }); + + const spawn: typeof Executable.spawn = Executable.spawn.bind(Executable); + const spawnSpy: jest.SpyInstance = jest + .spyOn(Executable, 'spawn') + .mockImplementation( + (filename: string, args: string[], options?: IExecutableSpawnOptions): ChildProcess => { + const childProcess: ChildProcess = spawn(filename, args, options); + const emit: ChildProcess['emit'] = childProcess.emit.bind(childProcess); + const isHashObject: boolean = args.includes('hash-object'); + childProcess.emit = ((eventName: string | symbol, ...eventArgs: unknown[]): boolean => { + if (eventName === 'close' && !isHashObject) { + void heldProcessesReleased.then(() => emit(eventName, ...eventArgs)); + return true; + } + const hadListeners: boolean = emit(eventName, ...eventArgs); + if (eventName === 'close') { + onHashObjectClosed(); + } + return hadListeners; + }) as ChildProcess['emit']; + return childProcess; + } + ); + + let outcome: { error: unknown } | undefined; + const settled: Promise = getDetailedRepoStateAsync( + SOURCE_PATH, + [`${TEST_PREFIX}testProject/does-not-exist.txt`], + undefined, + FILTERS + ).then( + () => { + outcome = { error: undefined }; + }, + (error: unknown) => { + outcome = { error }; + } + ); + + try { + await hashObjectClosed; + // Every step from the hash-object exit to the rejection runs as a microtask, so one macrotask is enough. + await new Promise((resolve) => setImmediate(resolve)); + + expect(spawnSpy).toHaveBeenCalledTimes(3); + // Before the fix, the promise was still pending at this point. + expect(outcome).toEqual({ + error: expect.objectContaining({ + message: expect.stringMatching(/^git hash-object exited with code 128:[\s\S]*does-not-exist\.txt/) + }) + }); + } finally { + releaseHeldProcesses(); + await settled; + spawnSpy.mockRestore(); + } + }); }); diff --git a/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts b/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts index 926e20e449..9f700d26c7 100644 --- a/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts +++ b/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts @@ -44,4 +44,10 @@ export interface IDaemonActivityPayload { readonly text: string; /** The stream the line was written to. Defaults to `stdout`. */ readonly stream?: 'stdout' | 'stderr'; + /** + * Set when Rush or a Rush plugin wrote the text as a warning or an error while the engine loaded or ran, + * which native Rush prints in yellow or red. Absent on other activity, including the end-of-run summary. + * Clients that print only a summary can still show these lines. + */ + readonly severity?: 'warning' | 'error'; } diff --git a/libraries/rush-daemon/src/EngineActivityOptions.ts b/libraries/rush-daemon/src/EngineActivityOptions.ts new file mode 100644 index 0000000000..bd87825125 --- /dev/null +++ b/libraries/rush-daemon/src/EngineActivityOptions.ts @@ -0,0 +1,32 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { _IOperationActivityOptions } from '@microsoft/rush-lib'; +import { TerminalProviderSeverity } from '@rushstack/terminal'; + +/** + * Activity options for text that Rush or a Rush plugin wrote through a daemon engine's terminal. The graph's event + * sink (PhasedRequestEventMultiplexer) passes the options object unchanged to each request's event sink. + */ +export interface IEngineActivityOptions extends _IOperationActivityOptions { + /** + * Set when the text is a warning or an error. Request event sinks copy it into the activity payload, so + * clients that print only a summary can still show these lines. + */ + readonly severity?: 'warning' | 'error'; +} + +const WARNING_ACTIVITY: IEngineActivityOptions = { stderr: true, severity: 'warning' }; +const ERROR_ACTIVITY: IEngineActivityOptions = { stderr: true, severity: 'error' }; +const OUTPUT_ACTIVITY: IEngineActivityOptions = { stderr: false }; + +export function getEngineActivityOptions(severity: TerminalProviderSeverity): IEngineActivityOptions { + switch (severity) { + case TerminalProviderSeverity.warning: + return WARNING_ACTIVITY; + case TerminalProviderSeverity.error: + return ERROR_ACTIVITY; + default: + return OUTPUT_ACTIVITY; + } +} diff --git a/libraries/rush-daemon/src/EngineTerminalProvider.ts b/libraries/rush-daemon/src/EngineTerminalProvider.ts index 88345d0372..cb13c5d782 100644 --- a/libraries/rush-daemon/src/EngineTerminalProvider.ts +++ b/libraries/rush-daemon/src/EngineTerminalProvider.ts @@ -4,6 +4,7 @@ import type { IOperationGraph, _IOperationGraphEventSink } from '@microsoft/rush-lib'; import { TerminalProviderSeverity, type ITerminalProvider } from '@rushstack/terminal'; +import { getEngineActivityOptions } from './EngineActivityOptions'; import { WorkspaceEngineRecreationRequiredError } from './WorkspaceEngineComponentFactory'; export class EngineTerminalProvider implements ITerminalProvider { @@ -95,8 +96,6 @@ export class EngineTerminalProvider implements ITerminalProvider { ) { return; } - this.#graph?.eventSink?.onActivity?.(text, { - stderr: severity === TerminalProviderSeverity.error || severity === TerminalProviderSeverity.warning - }); + this.#graph?.eventSink?.onActivity?.(text, getEngineActivityOptions(severity)); } } diff --git a/libraries/rush-daemon/src/PhasedRequestEventSink.ts b/libraries/rush-daemon/src/PhasedRequestEventSink.ts index 0fc35cbf01..4a849617c8 100644 --- a/libraries/rush-daemon/src/PhasedRequestEventSink.ts +++ b/libraries/rush-daemon/src/PhasedRequestEventSink.ts @@ -3,12 +3,7 @@ import { randomUUID } from 'node:crypto'; -import type { - IOperationExecutionResult, - Operation, - _IOperationActivityOptions, - _IOperationGraphEventSink -} from '@microsoft/rush-lib'; +import type { IOperationExecutionResult, Operation, _IOperationGraphEventSink } from '@microsoft/rush-lib'; import { OperationStatus } from '@microsoft/rush-lib'; import { DAEMON_PROTOCOL_VERSION, @@ -17,12 +12,14 @@ import { } from '@rushstack/rush-daemon-protocol'; import type { DaemonEventType, + IDaemonActivityPayload, IDaemonEventEnvelope, IDaemonEventScope } from '@rushstack/rush-daemon-protocol'; import { TerminalChunkKind } from '@rushstack/terminal'; import type { ITerminalChunk } from '@rushstack/terminal'; +import type { IEngineActivityOptions } from './EngineActivityOptions'; import type { IPhasedRequestClient } from './PhasedRequestClient'; const EVENT_SOURCE_PACKAGE: string = '@microsoft/rush-lib'; @@ -236,16 +233,20 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { } } - public onActivity(text: string, options?: _IOperationActivityOptions): void { + public onActivity(text: string, options?: IEngineActivityOptions): void { const operationId: string | undefined = options?.operationId; if (operationId !== undefined && !this.#activeOperationIds.has(operationId)) { return; } - this.#emitEvent( - 'activityChanged', - { stream: options?.stderr === true ? 'stderr' : 'stdout', text }, - { required: true, scope: operationId === undefined ? undefined : { operationId } } - ); + const payload: IDaemonActivityPayload = { + stream: options?.stderr === true ? 'stderr' : 'stdout', + text, + ...(options?.severity === undefined ? undefined : { severity: options.severity }) + }; + this.#emitEvent('activityChanged', payload, { + required: true, + scope: operationId === undefined ? undefined : { operationId } + }); } #emitEvent(type: DaemonEventType, payload: unknown, options?: IEventOptions): void { diff --git a/libraries/rush-daemon/src/test/EngineActivity.test.ts b/libraries/rush-daemon/src/test/EngineActivity.test.ts new file mode 100644 index 0000000000..87859ca4f1 --- /dev/null +++ b/libraries/rush-daemon/src/test/EngineActivity.test.ts @@ -0,0 +1,90 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IOperationGraph } from '@microsoft/rush-lib'; +import type { IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; +import { TerminalProviderSeverity } from '@rushstack/terminal'; + +import { EngineTerminalProvider } from '../EngineTerminalProvider'; +import { PhasedRequestEventMultiplexer } from '../PhasedRequestEventMultiplexer'; +import { PhasedRequestEventSink } from '../PhasedRequestEventSink'; +import { TestPhasedRequestClient } from './PhasedRequestRouterTestUtilities'; + +type HookName = 'configureIteration' | 'beforeExecuteIterationAsync' | 'afterExecuteIterationAsync'; + +interface IActivityHarness { + readonly client: TestPhasedRequestClient; + readonly sink: PhasedRequestEventSink; + readonly terminal: EngineTerminalProvider; + callHook(name: HookName): void; +} + +function createHarness(): IActivityHarness { + const taps: Map void> = new Map(); + const hook = (name: HookName): { tap(options: unknown, fn: () => void): void } => ({ + tap: (options: unknown, fn: () => void) => taps.set(name, fn) + }); + const multiplexer: PhasedRequestEventMultiplexer = new PhasedRequestEventMultiplexer(undefined); + const graph: IOperationGraph = { + debugMode: false, + eventSink: multiplexer, + hooks: { + configureIteration: hook('configureIteration'), + beforeExecuteIterationAsync: hook('beforeExecuteIterationAsync'), + afterExecuteIterationAsync: hook('afterExecuteIterationAsync') + } + } as unknown as IOperationGraph; + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + const sink: PhasedRequestEventSink = new PhasedRequestEventSink({ + activeOperationIds: new Set(), + client, + getNextSequence: () => client.getNextEventSequence(), + onWriteFailure: () => undefined, + rushVersion: '5.178.1' + }); + multiplexer.subscribe(sink); + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + terminal.attach(graph); + return { client, sink, terminal, callHook: (name: HookName) => taps.get(name)!() }; +} + +async function getActivityPayloadsAsync(harness: IActivityHarness): Promise { + await harness.sink.flushAsync(); + return harness.client.writes + .map(({ event }) => event) + .filter((event: IDaemonEventEnvelope | undefined) => event?.type === 'activityChanged') + .map((event: IDaemonEventEnvelope | undefined) => event!.payload); +} + +describe('engine terminal activity', () => { + it('marks warnings and errors that Rush or a plugin writes, whether buffered or written during an iteration', async () => { + const harness: IActivityHarness = createHarness(); + harness.terminal.write('buffered warning\n', TerminalProviderSeverity.warning); + harness.terminal.write('buffered output\n', TerminalProviderSeverity.log); + harness.callHook('configureIteration'); + harness.callHook('beforeExecuteIterationAsync'); + harness.terminal.write('plugin warning\n', TerminalProviderSeverity.warning); + harness.terminal.write('plugin error\n', TerminalProviderSeverity.error); + harness.terminal.write('plugin output\n', TerminalProviderSeverity.log); + harness.terminal.write('plugin detail\n', TerminalProviderSeverity.verbose); + + expect(await getActivityPayloadsAsync(harness)).toStrictEqual([ + { severity: 'warning', stream: 'stderr', text: 'buffered warning\n' }, + { stream: 'stdout', text: 'buffered output\n' }, + { severity: 'warning', stream: 'stderr', text: 'plugin warning\n' }, + { severity: 'error', stream: 'stderr', text: 'plugin error\n' }, + { stream: 'stdout', text: 'plugin output\n' } + ]); + }); + + it('leaves other activity, such as the end-of-run summary, without a severity', async () => { + const harness: IActivityHarness = createHarness(); + harness.sink.onActivity('Operations failed.\n', { stderr: true }); + harness.sink.onActivity('rush build (1.00 seconds)\n'); + + expect(await getActivityPayloadsAsync(harness)).toStrictEqual([ + { stream: 'stderr', text: 'Operations failed.\n' }, + { stream: 'stdout', text: 'rush build (1.00 seconds)\n' } + ]); + }); +}); From 6a05b01dee46e87bed03fbaa9baaf07624b3ad8c Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 17:52:56 +0000 Subject: [PATCH 028/265] [node-core-library] LockFile ties, TZ/locale rule, /proc start times and own-start memo Swarm integration step 13; original commit 0347cd0dbb. Refs: tasks 67, D and 62. ch01 GATE OK board 1389; o05 board 1360 (task 62 on ch01-t62: 32/32 served in 3 of 3 bursts); t01 board 1106/board 1186. Commits folded into this step (6): - 3dfb75ec86 [node-core-library] Don't grant a LockFile twice when lockfile birthtimes tie - 074f207217 [node-core-library] Keep the LockFile of a running process with another time zone or locale - 4e43a1ef8f [node-core-library] Don't set dirtyWhenAcquired when the holder releases the lock during a check - ad9c1211b4 [node-core-library] Read the start times of other LockFile processes from /proc on Linux - 698af4ef92 [node-core-library] Run ps for the start time of the current LockFile process only once on Linux - 9249e85d63 [rush-client-core] Test clients that start the daemon at the same time Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../lockfile-birthtime-tie_2026-09-28.json | 10 + .../lockfile-own-start-time_2026-09-28.json | 10 + .../lockfile-proc-start-time_2026-09-28.json | 10 + ...lockfile-start-time-locale_2026-09-28.json | 10 + .../startup-burst-test_2026-09-28.json | 10 + libraries/node-core-library/src/LockFile.ts | 356 +++++++++- .../src/test/LockFile.test.ts | 608 +++++++++++++++++- .../src/test/connectOrStartDaemon.test.ts | 52 +- .../src/test/fixtures/daemon.ts | 8 +- 9 files changed, 1040 insertions(+), 34 deletions(-) create mode 100644 common/changes/@rushstack/node-core-library/lockfile-birthtime-tie_2026-09-28.json create mode 100644 common/changes/@rushstack/node-core-library/lockfile-own-start-time_2026-09-28.json create mode 100644 common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json create mode 100644 common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-client-core/startup-burst-test_2026-09-28.json diff --git a/common/changes/@rushstack/node-core-library/lockfile-birthtime-tie_2026-09-28.json b/common/changes/@rushstack/node-core-library/lockfile-birthtime-tie_2026-09-28.json new file mode 100644 index 0000000000..c2d5042e06 --- /dev/null +++ b/common/changes/@rushstack/node-core-library/lockfile-birthtime-tie_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/node-core-library", + "comment": "Fix `LockFile` on Linux and macOS granting a lock to two processes at once when their lockfiles have the same birthtime. A tie no longer lets either process acquire the lock in that attempt; instead, `LockFile.tryAcquire()` waits a random time and tries again with a new lockfile, up to 4 attempts.", + "type": "patch" + } + ], + "packageName": "@rushstack/node-core-library" +} diff --git a/common/changes/@rushstack/node-core-library/lockfile-own-start-time_2026-09-28.json b/common/changes/@rushstack/node-core-library/lockfile-own-start-time_2026-09-28.json new file mode 100644 index 0000000000..04de10dff9 --- /dev/null +++ b/common/changes/@rushstack/node-core-library/lockfile-own-start-time_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/node-core-library", + "comment": "On Linux, `LockFile` now runs `ps` for the start time of the current process only once, instead of each time it tries to acquire a lock. It runs `ps` again if the system clock is set or `process.env.TZ` changes.", + "type": "patch" + } + ], + "packageName": "@rushstack/node-core-library" +} diff --git a/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json b/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json new file mode 100644 index 0000000000..e06b6e046b --- /dev/null +++ b/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/node-core-library", + "comment": "Make `LockFile` on Linux much faster when other processes hold or wait for the same lock. It now confirms that another process's lockfile is valid by reading /proc, instead of running a `ps` command for each lockfile on each attempt.", + "type": "patch" + } + ], + "packageName": "@rushstack/node-core-library" +} diff --git a/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json b/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json new file mode 100644 index 0000000000..0c02c7c84f --- /dev/null +++ b/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/node-core-library", + "comment": "Fix `LockFile` on Linux and macOS deleting the lockfile of a running process, and granting the lock again, when that process has a different time zone or locale. A lockfile whose start time doesn't match is now kept if its process started before the lockfile was created.", + "type": "patch" + } + ], + "packageName": "@rushstack/node-core-library" +} diff --git a/common/changes/@rushstack/rush-client-core/startup-burst-test_2026-09-28.json b/common/changes/@rushstack/rush-client-core/startup-burst-test_2026-09-28.json new file mode 100644 index 0000000000..bbb09270b5 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/startup-burst-test_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Add a test in which 32 clients start at once against a daemon that takes 8 seconds to start. All of them must connect, and `ps` must run about once per client.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-client-core" +} diff --git a/libraries/node-core-library/src/LockFile.ts b/libraries/node-core-library/src/LockFile.ts index c8c913056e..ea39a31b97 100644 --- a/libraries/node-core-library/src/LockFile.ts +++ b/libraries/node-core-library/src/LockFile.ts @@ -138,6 +138,147 @@ export function getProcessStartTime(pid: number): string | undefined { throw new Error(`Unexpected output from the "ps" command`); } +const LSTART_MONTHS: string[] = [ + 'Jan', + 'Feb', + 'Mar', + 'Apr', + 'May', + 'Jun', + 'Jul', + 'Aug', + 'Sep', + 'Oct', + 'Nov', + 'Dec' +]; + +/** + * Helper function that is exported for unit tests only. + * Returns the time when the process started, in milliseconds since the epoch, rounded down to a whole second. + * Unlike getProcessStartTime(), the result doesn't depend on the time zone or locale of the current process. + * Returns undefined if the process doesn't exist with that pid, or if its start time can't be determined. + */ +export function getProcessStartTimeMs(pid: number): number | undefined { + const pidString: string = pid.toString(); + if (pid < 0 || pidString.indexOf('e') >= 0 || pidString.indexOf('E') >= 0) { + return undefined; + } + let args: string[]; + if (process.platform === 'darwin') { + args = [`-p ${pidString}`, '-o lstart']; + } else if (process.platform === 'linux') { + args = ['-p', pidString, '-o', 'lstart']; + } else { + return undefined; + } + + // "ps -o lstart" formats the time using the time zone and locale of the "ps" process + const psResult: child_process.SpawnSyncReturns = child_process.spawnSync('ps', args, { + encoding: 'utf8', + env: { ...process.env, LC_ALL: 'C', TZ: 'UTC0' } + }); + + // For example: "Sun Sep 27 17:15:08 2026" + const match: RegExpExecArray | null = + /^\s*[A-Za-z]{3} ([A-Za-z]{3}) +(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) (\d{4})\s*$/.exec( + (psResult.stdout || '').split('\n')[1] || '' + ); + if (!match) { + return undefined; + } + const month: number = LSTART_MONTHS.indexOf(match[1]); + if (month < 0) { + return undefined; + } + return Date.UTC( + Number(match[6]), + month, + Number(match[2]), + Number(match[3]), + Number(match[4]), + Number(match[5]) + ); +} + +const LSTART_DAYS: string[] = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']; + +// The number of clock ticks per second in /proc/[pid]/stat. This is sysconf(_SC_CLK_TCK), which Linux fixes +// at 100 (USER_HZ) on every architecture that Node.js supports. +const LINUX_CLOCK_TICKS_PER_SECOND: number = 100; + +/** + * Helper function that is exported for unit tests only. + * Linux only: returns the time when the system booted, in seconds since the epoch, from the "btime" line of + * /proc/stat. "ps -o lstart" adds the start time of a process to this time. + */ +export function getLinuxBootTimeSeconds(): number { + const match: RegExpExecArray | null = /^btime (\d+)$/m.exec(FileSystem.readFile('/proc/stat')); + if (!match) { + throw new Error('The contents of /proc/stat have an unexpected format'); + } + return Number(match[1]); +} + +/** + * The start time of a Linux process, in the formats that getProcessStartTime() returns. + */ +export interface ILinuxProcessStartTime { + /** + * What "ps -o lstart" prints with the C locale and the current time zone, for example "Mon Sep 28 13:52:39 2026" + */ + lstart: string; + /** + * The start time in clock ticks after boot, from /proc/[pid]/stat. getProcessStartTime() returns this if + * "ps" can't be run. + */ + ticks: string; +} + +/** + * Helper function that is exported for unit tests only. + * Linux only: returns the start time of a process from /proc, without running "ps" like getProcessStartTime() + * does. Returns undefined if the process doesn't exist with that pid. Throws if /proc can't be read for another + * reason, or if it has an unexpected format. + * @param pid - The process ID + * @param getBootTimeSeconds - Returns what getLinuxBootTimeSeconds() returns + */ +export function getLinuxProcessStartTime( + pid: number, + getBootTimeSeconds: () => number +): ILinuxProcessStartTime | undefined { + const pidString: string = pid.toString(); + if (pid < 0 || pidString.indexOf('e') >= 0 || pidString.indexOf('E') >= 0) { + throw new Error(`"pid" is negative or too large`); + } + let stat: string; + try { + stat = FileSystem.readFile(`/proc/${pidString}/stat`); + } catch (error) { + // ESRCH means that the process exited while we were reading the file. + if (FileSystem.isNotExistError(error as Error) || (error as NodeJS.ErrnoException).code === 'ESRCH') { + return undefined; + } + throw error; + } + const ticks: string | undefined = getProcessStartTimeFromProcStat(stat); + if (ticks === undefined || !/^[0-9]+$/.test(ticks)) { + throw new Error(`The contents of /proc/${pidString}/stat have an unexpected format`); + } + + // Like "ps", round down to a whole second and use "%a %b %e %H:%M:%S %Y" in the local time zone. + const date: Date = new Date( + (getBootTimeSeconds() + Math.floor(Number(ticks) / LINUX_CLOCK_TICKS_PER_SECOND)) * 1000 + ); + const twoDigits: (value: number) => string = (value: number) => (value < 10 ? `0${value}` : `${value}`); + const lstart: string = + `${LSTART_DAYS[date.getDay()]} ${LSTART_MONTHS[date.getMonth()]} ` + + `${date.getDate() < 10 ? ' ' : ''}${date.getDate()} ` + + `${twoDigits(date.getHours())}:${twoDigits(date.getMinutes())}:${twoDigits(date.getSeconds())} ` + + `${date.getFullYear()}`; + return { lstart, ticks }; +} + // A set of locks that currently exist in the current process, to be used when // multiple locks are acquired in the same process. const IN_PROC_LOCKS: Set = new Set(); @@ -145,12 +286,41 @@ const IN_PROC_LOCKS: Set = new Set(); // The function used to determine a process's start time. Overridable for unit testing. let _getStartTime: (pid: number) => string | undefined = getProcessStartTime; +// On Linux, what _getStartTime() returned for the current process, and what getLinuxProcessStartTime() +// returned for it at the same time. See _getCurrentProcessStartTime(). +let _currentProcessStartTime: { startTime: string; linuxLstart: string } | undefined; + /** * For unit testing only: overrides the function used to determine a process's start time. * @internal */ export function _setLockFileGetProcessStartTime(fn: (pid: number) => string | undefined): void { _getStartTime = fn; + _currentProcessStartTime = undefined; +} + +/** + * Returns the start time of the current process, which its lockfiles contain. On Linux, this runs "ps" again only + * if the start time that /proc gives changes, which happens when the system clock is set or process.env.TZ + * changes. A process that acquires locks often would otherwise run "ps" each time, which is slow when there are + * many processes. + */ +function _getCurrentProcessStartTime(getLinuxBootTime: () => number): string | undefined { + let linuxLstart: string | undefined; + if (process.platform === 'linux') { + try { + linuxLstart = getLinuxProcessStartTime(process.pid, getLinuxBootTime)?.lstart; + } catch (error) { + // /proc can't be read, so run "ps" every time. + } + } + if (linuxLstart !== undefined && _currentProcessStartTime?.linuxLstart === linuxLstart) { + return _currentProcessStartTime.startTime; + } + const startTime: string | undefined = _getStartTime(process.pid); + _currentProcessStartTime = + startTime !== undefined && linuxLstart !== undefined ? { startTime, linuxLstart } : undefined; + return startTime; } /** @@ -415,6 +585,67 @@ function _tryAcquireInner( } } +// How much later than a lockfile's birthtime its process may appear to have started. "ps" reports whole +// seconds, and the system clock can be adjusted while a process runs. +const START_TIME_TOLERANCE_MS: number = 5000; + +/** + * Called when the start time in the lockfile of another running process differs from the start time that + * we got for its PID. Returns true if the lockfile still belongs to that process. + */ +function _isLockFileOfRunningProcess( + pid: string, + lockFileStartTime: string | undefined, + lockFileBirthtimeMs: number | undefined +): boolean { + // "ps -o lstart" formats the start time using the time zone and locale of the process that runs it, + // so a process whose TZ, LANG, LC_TIME or LC_ALL differs from ours wrote its start time differently. + // If the start time differs because the lockfile's process exited and the OS gave its PID to a new + // process, then the new process started after the lockfile was created. An empty lockfile is still + // treated as stale here, as before. + if (!lockFileStartTime || lockFileBirthtimeMs === undefined) { + return false; + } + const startTimeMs: number | undefined = getProcessStartTimeMs(parseInt(pid, 10)); + return startTimeMs !== undefined && startTimeMs <= lockFileBirthtimeMs + START_TIME_TOLERANCE_MS; +} + +/** + * Returns true if /proc shows that the lockfile of another process belongs to the running process with its PID. + * This is much faster than running "ps", which reads the files of every process in /proc. Returns false if the + * start time in the lockfile is different, if /proc can't tell, or if this isn't Linux. Then the caller must + * run "ps", so this never makes a lockfile stale. + */ +function _isLockFileOfLinuxProcess( + pid: string, + lockFileStartTime: string | undefined, + getBootTimeSeconds: () => number +): boolean { + if (process.platform !== 'linux' || !lockFileStartTime) { + return false; + } + let startTime: ILinuxProcessStartTime | undefined; + try { + startTime = getLinuxProcessStartTime(parseInt(pid, 10), getBootTimeSeconds); + } catch (error) { + // For example, /proc isn't mounted, or it doesn't let us read the files of this process. + return false; + } + // These are the formats that getProcessStartTime() returns. If the other process wrote its start time with a + // locale other than C, the caller runs "ps". + return ( + startTime !== undefined && + (lockFileStartTime === startTime.lstart || lockFileStartTime === startTime.ticks) + ); +} + +// Returned by _tryAcquireMacOrLinuxOnce() when the lockfile of another process has the same birthtime as ours. +const TIED: unique symbol = Symbol('tied'); +// The maximum number of attempts that _tryAcquireMacOrLinux() makes while lockfiles keep tying. +const MAX_TIED_ATTEMPTS: number = 4; +// After a tie, the next attempt waits a random time of up to this many milliseconds times the attempt number. +const TIE_RETRY_DELAY_MS: number = 20; + /** * Attempts to acquire the lock on a Linux or OSX machine */ @@ -423,19 +654,59 @@ function _tryAcquireMacOrLinux( resourceName: string, pidLockFilePath: string ): ITryAcquireResult | undefined { - // get the current process identifier (PID) - const pid: number = process.pid; + // On Linux, the time when the system booted, which we read from /proc/stat at most once per call + let linuxBootTimeSeconds: number | undefined; + const getLinuxBootTime: () => number = () => { + if (linuxBootTimeSeconds === undefined) { + linuxBootTimeSeconds = getLinuxBootTimeSeconds(); + } + return linuxBootTimeSeconds; + }; // Suppose that a process terminates unexpectedly without deleting its PID-based lockfile, // then we check to see if the process is still alive. The OS may have given the same PID // to a new process, how to detect that? We will rely on getProcessStartTime() which // is stored in the file itself for comparison. - const startTime: string | undefined = _getStartTime(pid); + const startTime: string | undefined = _getCurrentProcessStartTime(getLinuxBootTime); if (!startTime) { throw new Error(`Unable to calculate start time for current process.`); } + for (let attempt: number = 1; ; attempt++) { + const result: ITryAcquireResult | typeof TIED | undefined = _tryAcquireMacOrLinuxOnce( + resourceFolder, + resourceName, + pidLockFilePath, + startTime, + getLinuxBootTime + ); + if (result !== TIED) { + return result; + } + if (attempt >= MAX_TIED_ATTEMPTS) { + return undefined; + } + // If the other process also saw the tie, neither of us has the lock. Wait a random time so that + // our next lockfile is unlikely to tie again, and try again. + const delayMs: number = 1 + Math.floor(Math.random() * TIE_RETRY_DELAY_MS * attempt); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, delayMs); + } +} + +/** + * Makes one attempt to acquire the lock on a Linux or OSX machine + */ +function _tryAcquireMacOrLinuxOnce( + resourceFolder: string, + resourceName: string, + pidLockFilePath: string, + startTime: string, + getLinuxBootTime: () => number +): ITryAcquireResult | typeof TIED | undefined { + // get the current process identifier (PID) + const pid: number = process.pid; + let lockFileHandle: FileWriter | undefined; let result: ITryAcquireResult | undefined; @@ -448,8 +719,8 @@ function _tryAcquireMacOrLinux( lockFileHandle.write(startTime); const currentBirthTimeMs: number = lockFileHandle.getStatistics().birthtime.getTime(); - let smallestBirthTimeMs: number = currentBirthTimeMs; - let smallestBirthTimePid: string = pid.toString(); + // Set if another process has a lockfile with the same birthtime as ours + let tied: boolean = false; // now, scan the directory for all lockfiles const files: string[] = FileSystem.readFolderItemNames(resourceFolder); @@ -475,9 +746,6 @@ function _tryAcquireMacOrLinux( // console.log(`FOUND OTHER LOCKFILE: ${otherPid}`); - // Actual start time of the other PID - const otherPidCurrentStartTime: string | undefined = _getStartTime(parseInt(otherPid, 10)); - // The start time from the file, which we will compare with otherPidCurrentStartTime // to determine whether the PID got reused by a new process. let otherPidOldStartTime: string | undefined; @@ -506,6 +774,13 @@ function _tryAcquireMacOrLinux( // console.log(`Ignoring lock for pid ${otherPid} because its lockfile is newer than ours.`); continue; + } else if (otherBirthtimeMs === currentBirthTimeMs) { + // ==> Tie + // The other process's file has the same birthtime as ours, and they may acquire the lock + // after they finish writing the contents, so we must not treat this file as stale below. + // See the comment about ties below. + tied = true; + continue; } else if ( otherBirthtimeMs - currentBirthTimeMs < 0 && otherBirthtimeMs - currentBirthTimeMs > -1000 @@ -522,16 +797,39 @@ function _tryAcquireMacOrLinux( } // console.log(`Other pid ${otherPid} lockfile has start time: "${otherPidOldStartTime}"`); + + // Actual start time of the other PID. On Linux, /proc usually shows that the file belongs to the + // process with that PID, and then we don't need to run "ps", which is slow when there are many + // processes. When many processes wait for the same lock, each of their attempts checks the file + // of every other process. + const otherPidCurrentStartTime: string | undefined = _isLockFileOfLinuxProcess( + otherPid, + otherPidOldStartTime, + getLinuxBootTime + ) + ? otherPidOldStartTime + : _getStartTime(parseInt(otherPid, 10)); + // console.log(`Other pid ${otherPid} actually has start time: "${otherPidCurrentStartTime}"`); // Time to compare - if (!otherPidCurrentStartTime || otherPidOldStartTime !== otherPidCurrentStartTime) { + if ( + !otherPidCurrentStartTime || + (otherPidOldStartTime !== otherPidCurrentStartTime && + !_isLockFileOfRunningProcess(otherPid, otherPidOldStartTime, otherBirthtimeMs)) + ) { // ==> Stale lockfile // This file doesn't prevent us from acquiring the lock, but it does indicate that // the resource was left in a dirty state. (If we delete the file right now, that // information would be lost, so we clean up later when we acquire successfully.) // console.log(`Other pid ${otherPid} is no longer executing!`); + + // We checked the other process after we read its file. If the file is gone now, the other process + // released the lock in between, so the resource isn't dirty. + if (!FileSystem.exists(fileInFolderPath)) { + continue; + } staleFilesToDelete.push(fileInFolderPath); continue; } @@ -541,33 +839,29 @@ function _tryAcquireMacOrLinux( if (otherBirthtimeMs !== undefined) { // ==> We found a valid file belonging to another process. - // With multiple parties trying to acquire, the winner is the smallestBirthTime, - // so we need to sort. - - // the other lock file was created before the current earliest lock file - // or the other lock file was created at the same exact time, but has earlier pid - - // note that it is acceptable to do a direct comparison of the PIDs in this case - // since we are establishing a consistent order to apply to the lock files in all - // execution instances. - - // it doesn't matter that the PIDs roll over, we've already - // established that these processes all started at the same time, so we just - // need to get all instances of the lock test to agree which one won. - if ( - otherBirthtimeMs < smallestBirthTimeMs || - (otherBirthtimeMs === smallestBirthTimeMs && otherPid < smallestBirthTimePid) - ) { - smallestBirthTimeMs = otherBirthtimeMs; - smallestBirthTimePid = otherPid; + // With multiple parties trying to acquire, the winner is the one with the earliest file. + if (otherBirthtimeMs < currentBirthTimeMs) { + // we do not have the lock + return undefined; + } + + if (otherBirthtimeMs === currentBirthTimeMs) { + // ==> Tie + // Birthtimes have millisecond precision (often coarser), so files created at about the + // same time can have equal birthtimes. We read the folder only once, so the other process + // may have read it before our file existed and concluded that it holds the lock. Breaking + // the tie by PID could then let both processes acquire the lock. Instead, a tie means that + // neither process acquires the lock in this attempt. At least one of the two processes + // sees the other's file, so at most one of them acquires the lock. + tied = true; } } } } - if (smallestBirthTimePid !== pid.toString()) { - // we do not have the lock - return undefined; + if (tied) { + // we do not have the lock, but we may acquire it if we try again with a new file + return TIED; } let dirtyWhenAcquired: boolean = false; diff --git a/libraries/node-core-library/src/test/LockFile.test.ts b/libraries/node-core-library/src/test/LockFile.test.ts index 44957de4b9..f9ea13aee2 100644 --- a/libraries/node-core-library/src/test/LockFile.test.ts +++ b/libraries/node-core-library/src/test/LockFile.test.ts @@ -2,14 +2,18 @@ // See LICENSE in the project root for license information. import * as path from 'node:path'; +import * as child_process from 'node:child_process'; import { LockFile, type ILockFileHandle, getProcessStartTime, getProcessStartTimeFromProcStat, + getProcessStartTimeMs, + getLinuxBootTimeSeconds, + getLinuxProcessStartTime, _setLockFileGetProcessStartTime } from '../LockFile'; -import { FileSystem } from '../FileSystem'; +import { FileSystem, type FileSystemStats, type IFileSystemReadFileOptions } from '../FileSystem'; import { FileWriter } from '../FileWriter'; import * as WindowsLockFile from '../WindowsLockFile'; @@ -17,6 +21,41 @@ function setLockFileGetProcessStartTime(fn: (process: number) => string | undefi _setLockFileGetProcessStartTime(fn); } +/** + * Replaces the contents of a file that FileSystem.readFile() reads, or makes reading it fail. + */ +function mockReadFile(filePath: string, contents: string | NodeJS.ErrnoException): void { + const originalReadFile: typeof FileSystem.readFile = FileSystem.readFile; + jest + .spyOn(FileSystem, 'readFile') + .mockImplementation((readFilePath: string, options?: IFileSystemReadFileOptions) => { + if (readFilePath !== filePath) { + return originalReadFile(readFilePath, options); + } + if (typeof contents !== 'string') { + throw contents; + } + return contents; + }); +} + +/** + * Runs "ps -o lstart" for a process with the C locale, and returns what it prints. + */ +function getLstartWithCLocale(pid: number, timeZone?: string): string { + return child_process + .spawnSync('ps', ['-p', `${pid}`, '-o', 'lstart'], { + encoding: 'utf8', + env: { ...process.env, LC_ALL: 'C', ...(timeZone === undefined ? {} : { TZ: timeZone }) } + }) + .stdout.split('\n')[1] + .trim(); +} + +function createEaccesError(): NodeJS.ErrnoException { + return Object.assign(new Error('EACCES: permission denied'), { code: 'EACCES' }); +} + // lib/test const libTestFolder: string = path.resolve(__dirname, '../../lib-commonjs/test'); @@ -450,6 +489,573 @@ describe(LockFile.name, () => { lock!.release(); }); + + test("doesn't mark dirtyWhenAcquired if other process releases the lock and exits after its lockfile is read", () => { + // ensure test folder is clean + const testFolder: string = path.join(libTestFolder, '17'); + FileSystem.ensureEmptyFolder(testFolder); + + const resourceName: string = 'test'; + + const otherPid: number = 999999999; + const otherPidStartTime: string = '2012-01-02 12:53:12'; + + const otherPidLockFileName: string = LockFile.getLockFilePath(testFolder, resourceName, otherPid); + + // create an open lockfile for other process + const lockFileHandle: FileWriter = FileWriter.open(otherPidLockFileName); + lockFileHandle.write(otherPidStartTime); + lockFileHandle.close(); + + // simulate other process lock release and exit while the current process checks whether it is running + setLockFileGetProcessStartTime((pid: number) => { + if (pid === otherPid) { + FileSystem.deleteFile(otherPidLockFileName); + return undefined; + } + return getProcessStartTime(pid); + }); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(false); + expect(lock!.isReleased).toEqual(false); + + lock!.release(); + }); + + describe('when lockfiles have the same birthtime', () => { + const resourceName: string = 'test'; + // Compared as strings, this is larger than any real PID, so a PID tie-break would favor this process. + const otherPid: number = 999999999; + const otherPidStartTime: string = '2012-01-02 12:53:12'; + const birthtime: Date = new Date(1500000000000); + + interface IOtherLockFile { + otherPidLockFileName: string; + ourGetStatisticsSpy: jest.SpyInstance; + } + + function prepareOtherLockFile( + testFolder: string, + contents: string, + otherBirthtime: Date, + deleteAfterFirstRead: boolean = false + ): IOtherLockFile { + FileSystem.ensureEmptyFolder(testFolder); + const otherPidLockFileName: string = LockFile.getLockFilePath(testFolder, resourceName, otherPid); + const lockFileHandle: FileWriter = FileWriter.open(otherPidLockFileName); + lockFileHandle.write(contents); + lockFileHandle.close(); + + // the other process is still running + setLockFileGetProcessStartTime((pid: number) => { + return pid === otherPid ? otherPidStartTime : getProcessStartTime(pid); + }); + + const ourGetStatisticsSpy: jest.SpyInstance = jest + .spyOn(FileWriter.prototype, 'getStatistics') + .mockReturnValue({ birthtime } as FileSystemStats); + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + if (path.resolve(filePath) !== otherPidLockFileName) { + return originalGetStatistics(filePath); + } + if (deleteAfterFirstRead) { + // Like us, the other process saw the tie, so it deletes its lockfile before trying again. + FileSystem.deleteFile(otherPidLockFileName); + } + return { birthtime: otherBirthtime } as FileSystemStats; + }); + return { otherPidLockFileName, ourGetStatisticsSpy }; + } + + test('cannot acquire a lock if another valid lock has the same birthtime and a larger pid', () => { + const testFolder: string = path.join(libTestFolder, '6'); + const { otherPidLockFileName, ourGetStatisticsSpy } = prepareOtherLockFile( + testFolder, + otherPidStartTime, + birthtime + ); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + // The other process may have read the folder before our lockfile existed, in which case it + // already holds the lock, so a tie must not let us acquire it. We try again with a new + // lockfile a few times before giving up. + expect(lock).toBeUndefined(); + expect(ourGetStatisticsSpy).toHaveBeenCalledTimes(4); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + expect(FileSystem.exists(LockFile.getLockFilePath(testFolder, resourceName))).toEqual(false); + }); + + test('cannot acquire a lock or delete the other lockfile if it is still empty', () => { + const testFolder: string = path.join(libTestFolder, '7'); + const { otherPidLockFileName } = prepareOtherLockFile(testFolder, '', birthtime); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + // The other process hasn't written its start time yet, so its lockfile is not stale. + expect(lock).toBeUndefined(); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + }); + + test('can acquire a lock after a tie if the other process gives up', () => { + const testFolder: string = path.join(libTestFolder, '8'); + const { otherPidLockFileName, ourGetStatisticsSpy } = prepareOtherLockFile( + testFolder, + otherPidStartTime, + birthtime, + true + ); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(false); + expect(ourGetStatisticsSpy).toHaveBeenCalledTimes(2); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(false); + lock!.release(); + }); + + test('does not try again if another valid lock is older', () => { + const testFolder: string = path.join(libTestFolder, '9'); + const { otherPidLockFileName, ourGetStatisticsSpy } = prepareOtherLockFile( + testFolder, + otherPidStartTime, + new Date(birthtime.getTime() - 1) + ); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeUndefined(); + expect(ourGetStatisticsSpy).toHaveBeenCalledTimes(1); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + }); + + test('can acquire a lock if the other valid lock is newer', () => { + const testFolder: string = path.join(libTestFolder, '10'); + const { otherPidLockFileName } = prepareOtherLockFile( + testFolder, + otherPidStartTime, + new Date(birthtime.getTime() + 1) + ); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(false); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + lock!.release(); + }); + }); + + describe('when the start time in a lockfile differs from the start time of its running process', () => { + const resourceName: string = 'test'; + // The parent process is still running, and it started before any lockfile that this test creates. + const otherPid: number = process.ppid; + + function createOtherLockFile(testFolder: string, contents: string): string { + FileSystem.ensureEmptyFolder(testFolder); + const otherPidLockFileName: string = LockFile.getLockFilePath(testFolder, resourceName, otherPid); + const lockFileHandle: FileWriter = FileWriter.open(otherPidLockFileName); + lockFileHandle.write(contents); + lockFileHandle.close(); + return otherPidLockFileName; + } + + test('cannot acquire a lock if the other process wrote its start time in another time zone', () => { + const testFolder: string = path.join(libTestFolder, '11'); + // This is UTC+14 (or UTC-12 if that is the local time zone). POSIX time zones need no time zone database. + const otherTimeZone: string = new Date().getTimezoneOffset() === -840 ? 'XYZ+12' : 'XYZ-14'; + // What the other process writes if it runs with TZ=otherTimeZone + const otherPidStartTime: string = child_process + .spawnSync('ps', ['-p', `${otherPid}`, '-o', 'lstart'], { + encoding: 'utf8', + env: { ...process.env, TZ: otherTimeZone } + }) + .stdout.split('\n')[1] + .trim(); + expect(otherPidStartTime).not.toEqual(getProcessStartTime(otherPid)); + const otherPidLockFileName: string = createOtherLockFile(testFolder, otherPidStartTime); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + // The other process is running and started before its lockfile was created, so it holds the lock. + expect(lock).toBeUndefined(); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + }); + + test('deletes the lockfile if its PID now belongs to a process that started after the lockfile was created', () => { + const testFolder: string = path.join(libTestFolder, '12'); + const otherPidLockFileName: string = createOtherLockFile(testFolder, 'Thu Jan 1 00:00:10 1970'); + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + return path.resolve(filePath) === otherPidLockFileName + ? ({ birthtime: new Date(10000) } as FileSystemStats) + : originalGetStatistics(filePath); + }); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(true); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(false); + lock!.release(); + }); + + test('deletes an empty lockfile that is more than 1 second old', () => { + const testFolder: string = path.join(libTestFolder, '13'); + const otherPidLockFileName: string = createOtherLockFile(testFolder, ''); + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + return path.resolve(filePath) === otherPidLockFileName + ? ({ birthtime: new Date(Date.now() - 2000) } as FileSystemStats) + : originalGetStatistics(filePath); + }); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(true); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(false); + lock!.release(); + }); + + test("doesn't mark dirtyWhenAcquired if the process releases the lock and exits before its start time is checked again", () => { + const testFolder: string = path.join(libTestFolder, '19'); + // No process has this PID, so the second check of its start time finds that the process exited. + const exitedPid: number = 999999999; + FileSystem.ensureEmptyFolder(testFolder); + const exitedPidLockFileName: string = LockFile.getLockFilePath(testFolder, resourceName, exitedPid); + const lockFileHandle: FileWriter = FileWriter.open(exitedPidLockFileName); + lockFileHandle.write('Mon Jan 2 12:53:12 2012'); + lockFileHandle.close(); + + // Before its lockfile is read, the process is running, and its start time is formatted differently, + // as if it had another time zone. + setLockFileGetProcessStartTime((pid: number) => { + return pid === exitedPid ? 'Mon Jan 2 04:53:12 2012' : getProcessStartTime(pid); + }); + // Right after its lockfile is read, the process releases the lock and exits. + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + const statistics: FileSystemStats = originalGetStatistics(filePath); + if (path.resolve(filePath) === exitedPidLockFileName) { + FileSystem.deleteFile(exitedPidLockFileName); + } + return statistics; + }); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(false); + lock!.release(); + }); + }); + + describe(getProcessStartTimeMs.name, () => { + test('returns the start time of a process', () => { + const startTimeMs: number | undefined = getProcessStartTimeMs(process.pid); + expect(startTimeMs).toBeDefined(); + expect(startTimeMs! % 1000).toEqual(0); + expect(Math.abs(startTimeMs! - (Date.now() - process.uptime() * 1000))).toBeLessThan(5000); + }); + + test('does not depend on the time zone', () => { + const startTimeMs: number | undefined = getProcessStartTimeMs(process.pid); + const originalTimeZone: string | undefined = process.env.TZ; + process.env.TZ = 'XYZ-14'; + try { + expect(getProcessStartTimeMs(process.pid)).toEqual(startTimeMs); + } finally { + if (originalTimeZone === undefined) { + delete process.env.TZ; + } else { + process.env.TZ = originalTimeZone; + } + } + }); + + test('returns undefined if the process does not exist', () => { + expect(getProcessStartTimeMs(999999999)).toBeUndefined(); + }); + }); + + if (process.platform === 'linux') { + describe(getLinuxProcessStartTime.name, () => { + test('returns what "ps -o lstart" prints with the C locale', () => { + for (const pid of [process.pid, process.ppid, 1]) { + expect(getLinuxProcessStartTime(pid, getLinuxBootTimeSeconds)!.lstart).toEqual( + getLstartWithCLocale(pid) + ); + } + }); + + test('uses the time zone of the process, like "ps"', () => { + // A test can't change the time zone of its own process in Jest, so this runs another process. + const timeZone: string = new Date().getTimezoneOffset() === -840 ? 'XYZ+12' : 'XYZ-14'; + const lockFileModulePath: string = require.resolve('../LockFile'); + const script: string = + `const { getLinuxProcessStartTime, getLinuxBootTimeSeconds } = require(${JSON.stringify(lockFileModulePath)});` + + `console.log(getLinuxProcessStartTime(${process.pid}, getLinuxBootTimeSeconds).lstart);`; + const lstart: string = child_process + .spawnSync(process.execPath, ['-e', script], { + encoding: 'utf8', + env: { ...process.env, TZ: timeZone } + }) + .stdout.trim(); + + expect(lstart).toEqual(getLstartWithCLocale(process.pid, timeZone)); + expect(lstart).not.toEqual( + getLinuxProcessStartTime(process.pid, getLinuxBootTimeSeconds)!.lstart + ); + }); + + test('returns the start time in clock ticks from /proc/[pid]/stat', () => { + expect(getLinuxProcessStartTime(process.pid, getLinuxBootTimeSeconds)!.ticks).toEqual( + getProcessStartTimeFromProcStat(FileSystem.readFile(`/proc/${process.pid}/stat`)) + ); + }); + + test('reads /proc/[pid]/stat if the command name contains spaces and parentheses', () => { + mockReadFile( + '/proc/12345/stat', + '12345 (a) (b) S 1 1 1 0 -1 4194304 0 0 0 0 0 0 0 0 20 0 1 0 250 0 0\n' + ); + // 250 ticks are 2.5 seconds, which "ps" rounds down. This is September 8 or 9, 2001 in any time zone. + const expectedLstart: string = child_process + .spawnSync('date', ['-d', '@1000000002', '+%a %b %e %H:%M:%S %Y'], { + encoding: 'utf8', + env: { ...process.env, LC_ALL: 'C' } + }) + .stdout.trim(); + + expect(getLinuxProcessStartTime(12345, () => 1000000000)).toEqual({ + lstart: expectedLstart, + ticks: '250' + }); + expect(expectedLstart).toMatch(/^[A-Z][a-z]{2} Sep {2}[89] \d\d:\d\d:\d\d 2001$/); + }); + + test('returns undefined if the process does not exist', () => { + expect(getLinuxProcessStartTime(999999999, getLinuxBootTimeSeconds)).toBeUndefined(); + }); + + test('throws if /proc/[pid]/stat cannot be read', () => { + mockReadFile('/proc/12345/stat', createEaccesError()); + expect(() => getLinuxProcessStartTime(12345, getLinuxBootTimeSeconds)).toThrow('EACCES'); + }); + + test('throws if /proc/[pid]/stat has an unexpected format', () => { + mockReadFile('/proc/12345/stat', '12345 (node) S 1\n'); + expect(() => getLinuxProcessStartTime(12345, getLinuxBootTimeSeconds)).toThrow( + 'unexpected format' + ); + }); + }); + + describe('when other lockfiles belong to running processes', () => { + const resourceName: string = 'test'; + let childProcesses: child_process.ChildProcess[] = []; + + afterEach(() => { + for (const childProcess of childProcesses) { + childProcess.kill(); + } + childProcesses = []; + }); + + function startProcesses(count: number): number[] { + for (let i: number = 0; i < count; i++) { + childProcesses.push(child_process.spawn('sleep', ['60'], { stdio: 'ignore' })); + } + return childProcesses.map((childProcess: child_process.ChildProcess) => childProcess.pid!); + } + + // Creates a lockfile for each process that is newer than the lockfile of this process, + // so that tryAcquire() checks all of them and then acquires the lock. + function createNewerLockFiles( + testFolder: string, + otherPids: number[], + getContents: (pid: number) => string + ): string[] { + FileSystem.ensureEmptyFolder(testFolder); + const otherPidLockFileNames: string[] = otherPids.map((otherPid: number) => { + const otherPidLockFileName: string = LockFile.getLockFilePath( + testFolder, + resourceName, + otherPid + ); + FileSystem.writeFile(otherPidLockFileName, getContents(otherPid)); + return otherPidLockFileName; + }); + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + return otherPidLockFileNames.includes(path.resolve(filePath)) + ? ({ birthtime: new Date(Date.now() + 60000) } as FileSystemStats) + : originalGetStatistics(filePath); + }); + return otherPidLockFileNames; + } + + function expectToAcquireAndKeep(testFolder: string, otherPidLockFileNames: string[]): void { + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(false); + for (const otherPidLockFileName of otherPidLockFileNames) { + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + } + lock!.release(); + } + + test('does not run "ps" for them', () => { + const testFolder: string = path.join(libTestFolder, '14'); + const otherPidLockFileNames: string[] = createNewerLockFiles( + testFolder, + [process.ppid, 1, ...startProcesses(6)], + getLstartWithCLocale + ); + // LockFile calls the functions of this module object, so they can be spied on here. + const nativeChildProcess: typeof child_process = jest.requireActual('node:child_process'); + const spawnSyncSpy: jest.SpyInstance = jest.spyOn(nativeChildProcess, 'spawnSync'); + const readFileSpy: jest.SpyInstance = jest.spyOn(FileSystem, 'readFile'); + + expectToAcquireAndKeep(testFolder, otherPidLockFileNames); + + // "ps" ran only for the start time of this process. + expect(spawnSyncSpy).toHaveBeenCalledTimes(1); + expect(spawnSyncSpy.mock.calls[0][1]).toEqual(['-p', `${process.pid}`, '-o', 'lstart']); + // The boot time was read once. + expect(readFileSpy.mock.calls.filter((args: unknown[]) => args[0] === '/proc/stat')).toHaveLength( + 1 + ); + }); + + test('does not run "ps" if a lockfile has the start time in clock ticks', () => { + const testFolder: string = path.join(libTestFolder, '15'); + const otherPidLockFileNames: string[] = createNewerLockFiles( + testFolder, + startProcesses(1), + (otherPid: number) => getLinuxProcessStartTime(otherPid, getLinuxBootTimeSeconds)!.ticks + ); + const getStartTimeSpy: jest.Mock = jest.fn(getProcessStartTime); + setLockFileGetProcessStartTime(getStartTimeSpy); + + expectToAcquireAndKeep(testFolder, otherPidLockFileNames); + + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid]]); + }); + + test('runs "ps" if /proc/[pid]/stat cannot be read', () => { + const testFolder: string = path.join(libTestFolder, '16'); + const otherPids: number[] = startProcesses(1); + const otherPidLockFileNames: string[] = createNewerLockFiles( + testFolder, + otherPids, + getLstartWithCLocale + ); + mockReadFile(`/proc/${otherPids[0]}/stat`, createEaccesError()); + const getStartTimeSpy: jest.Mock = jest.fn(getProcessStartTime); + setLockFileGetProcessStartTime(getStartTimeSpy); + + expectToAcquireAndKeep(testFolder, otherPidLockFileNames); + + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [otherPids[0]]]); + }); + }); + + describe('when this process acquires locks again', () => { + const resourceName: string = 'test'; + const testFolder: string = path.join(libTestFolder, '18'); + + beforeEach(() => { + FileSystem.ensureEmptyFolder(testFolder); + }); + + // Returns what the lockfile of this process contained + function acquireAndRelease(): string { + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + expect(lock).toBeDefined(); + const contents: string = FileSystem.readFile(LockFile.getLockFilePath(testFolder, resourceName)); + lock!.release(); + return contents; + } + + test('runs "ps" for the start time of this process only once', () => { + const getStartTimeSpy: jest.Mock = jest.fn(getProcessStartTime); + setLockFileGetProcessStartTime(getStartTimeSpy); + const expectedStartTime: string = getProcessStartTime(process.pid)!; + + for (let i: number = 0; i < 3; i++) { + expect(acquireAndRelease()).toEqual(expectedStartTime); + } + + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid]]); + }); + + test('runs "ps" again if the boot time changes, as it does when the system clock is set', () => { + const getStartTimeSpy: jest.Mock = jest.fn(getProcessStartTime); + setLockFileGetProcessStartTime(getStartTimeSpy); + + acquireAndRelease(); + mockReadFile('/proc/stat', `btime ${getLinuxBootTimeSeconds() - 3600}\n`); + acquireAndRelease(); + acquireAndRelease(); + + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [process.pid]]); + }); + + test('runs "ps" again if it did not return the start time', () => { + const getStartTimeSpy: jest.Mock = jest + .fn(getProcessStartTime) + .mockImplementationOnce(() => undefined); + setLockFileGetProcessStartTime(getStartTimeSpy); + + expect(() => LockFile.tryAcquire(testFolder, resourceName)).toThrow( + 'Unable to calculate start time for current process.' + ); + acquireAndRelease(); + acquireAndRelease(); + + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [process.pid]]); + }); + + test('runs "ps" for other processes each time', () => { + const otherPid: number = 999999999; + const otherPidStartTime: string = '2012-01-02 12:53:12'; + const otherPidLockFileName: string = LockFile.getLockFilePath(testFolder, resourceName, otherPid); + FileSystem.writeFile(otherPidLockFileName, otherPidStartTime); + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + return path.resolve(filePath) === otherPidLockFileName + ? ({ birthtime: new Date(Date.now() - 60000) } as FileSystemStats) + : originalGetStatistics(filePath); + }); + const getStartTimeSpy: jest.Mock = jest.fn((pid: number) => + pid === otherPid ? otherPidStartTime : getProcessStartTime(pid) + ); + setLockFileGetProcessStartTime(getStartTimeSpy); + + for (let i: number = 0; i < 3; i++) { + expect(LockFile.tryAcquire(testFolder, resourceName)).toBeUndefined(); + } + + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [otherPid], [otherPid], [otherPid]]); + }); + + test('uses the new function after the function is replaced', () => { + setLockFileGetProcessStartTime(() => 'Mon Jan 1 00:00:00 2024'); + expect(acquireAndRelease()).toEqual('Mon Jan 1 00:00:00 2024'); + + setLockFileGetProcessStartTime(() => 'Tue Jan 2 00:00:00 2024'); + expect(acquireAndRelease()).toEqual('Tue Jan 2 00:00:00 2024'); + }); + }); + } }); } diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index bc11e815bd..027d2e22bf 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -343,6 +343,53 @@ describe('detached daemon startup', () => { await client.closeAsync(); }, 15000); + // On Linux, LockFile checks the owners of other lockfiles. It used to run "ps" for each of them on every attempt, + // and "ps" reads every process on the system, so clients that started at once timed out on a busy machine. + (process.platform === 'linux' ? it : it.skip)( + 'connects 32 concurrent clients to a daemon that takes 8 seconds to start, running "ps" once per client', + async () => { + const clientCount: number = 32; + fs.writeFileSync(path.join(folder, 'startup-delay-ms'), '8000'); + // A "ps" script ahead of the real one on the PATH records the client that runs each "ps" command. + const binFolder: string = path.join(folder, 'bin'); + const psCallsPath: string = path.join(folder, 'ps-calls'); + fs.mkdirSync(binFolder); + fs.writeFileSync( + path.join(binFolder, 'ps'), + `#!/bin/sh\necho "$PPID" >> '${psCallsPath}'\nPATH='${process.env.PATH}' exec ps "$@"\n`, + { mode: 0o755 } + ); + // The default deadline, instead of the 7 seconds that this suite uses. + const startOptions: IConnectOrStartDaemonOptions = { ...options, startupTimeoutMs: 15000 }; + const starters: ChildProcess[] = Array.from({ length: clientCount }, () => + spawn(process.execPath, [path.join(__dirname, 'fixtures/starter.js'), JSON.stringify(startOptions)], { + // With another locale, "ps" can print localized names, and LockFile then runs "ps" to check them. + env: { ...process.env, PATH: `${binFolder}${path.delimiter}${process.env.PATH}`, LC_ALL: 'C' }, + stdio: ['ignore', 'ignore', 'pipe'] + }) + ); + starterProcesses.push(...starters); + const results = await Promise.all( + starters.map(async (starter) => { + let stderr: string = ''; + starter.stderr!.on('data', (chunk: Buffer) => { + stderr += chunk.toString(); + }); + const [code] = await once(starter, 'close'); + return { code, stderr }; + }) + ); + + expect(results).toEqual(Array.from({ length: clientCount }, () => ({ code: 0, stderr: '' }))); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + // Each client runs "ps" once, for its own start time. Lockfiles of other clients are checked with /proc. + const psCalls: number = fs.readFileSync(psCallsPath, 'utf8').trim().split('\n').length; + expect(psCalls).toBeGreaterThan(0); + expect(psCalls).toBeLessThan(2 * clientCount); + }, + 30000 + ); + it('replaces a mismatched daemon exactly once for concurrent clients before requests start', async () => { const old = await connectOrStartDaemonAsync(options); const previous = await old.status; @@ -468,7 +515,10 @@ describe('detached daemon startup', () => { expect(starts).toBeGreaterThanOrEqual(2); expect(starts).toBeLessThanOrEqual(7); // The deadline may expire after the last successor started but before the request was resubmitted. - const requests: number = fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n').length; + const requests: number = fs + .readFileSync(path.join(folder, 'requests'), 'utf8') + .trim() + .split('\n').length; expect(requests).toBeGreaterThanOrEqual(starts - 1); expect(requests).toBeLessThanOrEqual(starts); } else { diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index e03e1cc12e..5d04da0f96 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -39,7 +39,7 @@ async function mainAsync(): Promise { await new Promise((resolve) => setTimeout(resolve, 20)); } } - await new Promise((resolve) => setTimeout(resolve, 250)); + await new Promise((resolve) => setTimeout(resolve, readStartupDelayMs(folder))); const listener = await DaemonFrameListener.listenAsync(paths, { protocolVersion: DAEMON_PROTOCOL_VERSION, onConnection: (connection) => { @@ -127,6 +127,12 @@ async function mainAsync(): Promise { } } +/** How long the daemon waits before it listens: 250 milliseconds, or the number in the "startup-delay-ms" file. */ +function readStartupDelayMs(folder: string): number { + const delayPath: string = path.join(folder, 'startup-delay-ms'); + return fs.existsSync(delayPath) ? Number(fs.readFileSync(delayPath, 'utf8')) : 250; +} + mainAsync().catch((error: Error) => { process.stderr.write(`${error.stack}\n`); process.exitCode = 1; From bc5e6c9e40d5c0fa10f051f00ca71f363c86d7e2 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 18:05:55 +0000 Subject: [PATCH 029/265] [rush-daemon] A request can share an input capture that started after it was received Swarm integration step 14; original commit f2ec9ffc26. ch01 GATE OK board 1430 (tree ae49a66e47 with o07's stack); t06 CONFIRMED board 1365; o05 board 1412 (cold split gone in auto-start mode). Commits folded into this step (1): - a4de3fe5fe [rush-daemon] Share running input captures that started after a request was received Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...time-capture-sharing_2026-09-28-16-31.json | 10 + .../rush-daemon/src/FreshCaptureCoalescer.ts | 42 ++- .../src/WorkspaceRequestLifecycle.ts | 48 ++- .../src/test/FreshCaptureCoalescer.test.ts | 144 +++++++++ .../test/WorkspaceInputCaptureReceipt.test.ts | 294 ++++++++++++++++++ 5 files changed, 520 insertions(+), 18 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r07-receipt-time-capture-sharing_2026-09-28-16-31.json create mode 100644 libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts diff --git a/common/changes/@rushstack/rush-daemon/swarm-r07-receipt-time-capture-sharing_2026-09-28-16-31.json b/common/changes/@rushstack/rush-daemon/swarm-r07-receipt-time-capture-sharing_2026-09-28-16-31.json new file mode 100644 index 0000000000..ca7fb34a1e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r07-receipt-time-capture-sharing_2026-09-28-16-31.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A build request can share a workspace input or project configuration capture that is already running if the capture started after the daemon received the request, instead of waiting for the next capture. Builds that arrive together, or that wait behind a graph load, now share one capture. Captures that detect changes during a graph reload, and graph control captures, still start after they are requested.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/FreshCaptureCoalescer.ts b/libraries/rush-daemon/src/FreshCaptureCoalescer.ts index 1b16286973..a9f4865264 100644 --- a/libraries/rush-daemon/src/FreshCaptureCoalescer.ts +++ b/libraries/rush-daemon/src/FreshCaptureCoalescer.ts @@ -2,15 +2,26 @@ // See LICENSE in the project root for license information. interface ICapture { + readonly startTimeMs: number; readonly running: Promise; next: Promise | undefined; } +/** + * Options for {@link FreshCaptureCoalescer}. + */ +export interface IFreshCaptureCoalescerOptions { + /** + * Returns the current time on the clock that callers use for `notBeforeMs`. Defaults to `performance.now()`. + */ + readonly now?: () => number; +} + function ignore(): void {} /** * Shares workspace input captures between concurrent requests, without ever giving a caller a capture that - * began before the caller asked for one. + * began before the caller asked for one, or before the time that the caller passes as `notBeforeMs`. * * @remarks * A capture reads the workspace while it runs, so a capture that is already running can miss a change made just @@ -19,14 +30,38 @@ function ignore(): void {} * Concurrent callers cost at most two captures instead of one each, and every caller receives a result that is * at least as fresh as a capture it started itself. Nothing is retained once a capture settles. * + * A caller that knows an earlier time after which every change it depends on was made, such as the time at which + * the daemon received its request, can pass that time as `notBeforeMs`. It then shares a running capture that + * started at or after that time instead of waiting for the next one. + * * Callers that pass the same scope and key must request the same capture. */ export class FreshCaptureCoalescer { readonly #captures: WeakMap>> = new WeakMap(); + readonly #now: () => number; + + public constructor(options: IFreshCaptureCoalescerOptions = {}) { + this.#now = options.now ?? (() => performance.now()); + } - public captureAsync(scope: TScope, key: string, captureAsync: () => Promise): Promise { + /** + * Returns a capture that started after this call, or at or after `notBeforeMs` if it is specified. + * + * @param scope - Captures are shared only within a scope. + * @param key - Identifies the capture within the scope. + * @param captureAsync - Starts a capture when the caller cannot share one. + * @param notBeforeMs - A time on this coalescer's clock. A running capture that started at or after this time is + * shared with the caller. If it is not specified, the caller shares only a capture that starts after this call. + */ + public captureAsync( + scope: TScope, + key: string, + captureAsync: () => Promise, + notBeforeMs?: number + ): Promise { const capture: ICapture | undefined = this.#captures.get(scope)?.get(key); if (!capture) return this.#start(scope, key, captureAsync); + if (notBeforeMs !== undefined && capture.startTimeMs >= notBeforeMs) return capture.running; capture.next ??= capture.running.then(ignore, ignore).then(() => this.#start(scope, key, captureAsync)); return capture.next; } @@ -37,8 +72,9 @@ export class FreshCaptureCoalescer { captures = new Map(); this.#captures.set(scope, captures); } + const startTimeMs: number = this.#now(); const running: Promise = new Promise((resolve) => resolve(captureAsync())); - const capture: ICapture = { running, next: undefined }; + const capture: ICapture = { startTimeMs, running, next: undefined }; captures.set(key, capture); const forget: () => void = () => { if (captures.get(key) === capture) captures.delete(key); diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 9b51147ecb..d6d3e60a3d 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -120,6 +120,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { readonly #startupFingerprint: IWorkspaceInputFingerprint; readonly #runtimeCache: WorkspaceRuntimeFingerprintCache; // Concurrent requests share captures; each capture still starts after the requests it serves arrived. + // The coalescers' default clock, performance.now(), is also the clock of receivedTimeMs. readonly #fingerprintCaptures: FreshCaptureCoalescer = new FreshCaptureCoalescer(); readonly #projectFingerprintCaptures: FreshCaptureCoalescer = @@ -216,7 +217,13 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { try { for (let attempt: number = 0; ; attempt++) { try { - const prepared: IPreparedGeneration = await this.#prepareAsync(envelope, client, admission, ticket); + const prepared: IPreparedGeneration = await this.#prepareAsync( + envelope, + client, + admission, + ticket, + receivedTimeMs + ); generation = prepared; const requestEnvelope: IDaemonRequestEnvelope = { ...envelope, @@ -307,6 +314,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { client: IDaemonRequestDispatchClient, admission: RequestAdmissionController, ticket: IWorkspaceRestartTicket | undefined, + receivedTimeMs: number, admittedLease?: IRequestLease ): Promise { let lease: IRequestLease = @@ -401,7 +409,9 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { fingerprint: this.#fingerprint }; } - let fingerprint: IWorkspaceInputFingerprint = await this.#captureAsync(session, envelope); + // The client changes the workspace before it sends a request, so any capture that started after the request + // was received sees those changes. Captures that must detect changes made during a transition stay strict. + let fingerprint: IWorkspaceInputFingerprint = await this.#captureAsync(session, envelope, receivedTimeMs); let tier: WorkspaceInputChangeTier = this.#classify(fingerprint, isMutation(envelope)); let commandIdentity: string | undefined; let projectFingerprint: string | undefined; @@ -412,7 +422,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { abortSignal: client.abortSignal }); if (tier === WorkspaceInputChangeTier.Reuse) { - projectFingerprint = await this.#captureProjectFingerprintAsync(session); + projectFingerprint = await this.#captureProjectFingerprintAsync(session, receivedTimeMs); if ( this.#boundSession !== session || this.#commandIdentity !== commandIdentity || @@ -446,7 +456,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { this.#gate, this.#transitionProgress ); - return await this.#prepareAsync(envelope, client, admission, ticket, shared); + return await this.#prepareAsync(envelope, client, admission, ticket, receivedTimeMs, shared); } this.#transitioning = ownsTransition = true; this.#cancelObservers(); @@ -636,7 +646,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { #captureAsync( session: IWorkspaceSession, - envelope: IDaemonRequestEnvelope + envelope: IDaemonRequestEnvelope, + notBeforeMs?: number ): Promise { const { rushConfiguration } = session; const { environment } = envelope; @@ -645,20 +656,27 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { environment.RUSH_PREVIEW_VERSION ?? null, getWorkspaceFingerprintEnvironmentEntries(environment) ]); - return this.#fingerprintCaptures.captureAsync(rushConfiguration, key, () => - captureWorkspaceInputFingerprintAsync({ - rushConfiguration, - environment, - runtimePaths: this.#runtimePaths, - runtimeCache: this.#runtimeCache - }) + return this.#fingerprintCaptures.captureAsync( + rushConfiguration, + key, + () => + captureWorkspaceInputFingerprintAsync({ + rushConfiguration, + environment, + runtimePaths: this.#runtimePaths, + runtimeCache: this.#runtimeCache + }), + notBeforeMs ); } - #captureProjectFingerprintAsync(session: IWorkspaceSession): Promise { + #captureProjectFingerprintAsync(session: IWorkspaceSession, notBeforeMs?: number): Promise { const { rushConfiguration } = session; - return this.#projectFingerprintCaptures.captureAsync(rushConfiguration, '', () => - captureProjectConfigurationFingerprintAsync(rushConfiguration, this.#terminal) + return this.#projectFingerprintCaptures.captureAsync( + rushConfiguration, + '', + () => captureProjectConfigurationFingerprintAsync(rushConfiguration, this.#terminal), + notBeforeMs ); } diff --git a/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts b/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts index 207f2f79e1..fe2dbb66f6 100644 --- a/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts +++ b/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts @@ -20,6 +20,12 @@ class ControlledCaptures { }; } +/** A clock that moves only when the test says so. */ +class ManualClock { + public nowMs: number = 0; + public readonly now = (): number => this.nowMs; +} + async function flushAsync(): Promise { await new Promise((resolve) => setImmediate(resolve)); } @@ -132,6 +138,144 @@ describe(FreshCaptureCoalescer.name, () => { await expect(coalescer.captureAsync(scope, 'key', async () => 'started')).resolves.toBe('started'); }); + it('shares a running capture with callers whose notBeforeMs is at or before the time it started', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 10); + clock.nowMs = 20; + const receivedWhenItStarted: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + 10 + ); + const receivedBeforeItStarted: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + 5 + ); + await flushAsync(); + expect(captures.started).toHaveLength(1); + + captures.started[0].resolve('started at 10'); + await expect(Promise.all([first, receivedWhenItStarted, receivedBeforeItStarted])).resolves.toEqual([ + 'started at 10', + 'started at 10', + 'started at 10' + ]); + await flushAsync(); + expect(captures.started).toHaveLength(1); + }); + + it('makes callers whose notBeforeMs is after the running capture started wait for the next capture', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + clock.nowMs = 11; + const receivedLater: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 11); + const strict: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + await flushAsync(); + expect(captures.started).toHaveLength(1); + + captures.started[0].resolve('started at 10'); + await expect(first).resolves.toBe('started at 10'); + await flushAsync(); + expect(captures.started).toHaveLength(2); + captures.started[1].resolve('started at 11'); + await expect(Promise.all([receivedLater, strict])).resolves.toEqual(['started at 11', 'started at 11']); + }); + + it('records when a next capture starts, not when it was queued', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + clock.nowMs = 12; + const queued: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + // The running capture started too early for this caller, so it shares the next capture that is already queued. + const queuedWithReceipt: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 12); + clock.nowMs = 30; + captures.started[0].resolve('started at 10'); + await expect(first).resolves.toBe('started at 10'); + await flushAsync(); + expect(captures.started).toHaveLength(2); + + const receivedBeforeNextStarted: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + 25 + ); + const receivedAfterNextStarted: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + 31 + ); + captures.started[1].resolve('started at 30'); + await expect(Promise.all([queued, queuedWithReceipt, receivedBeforeNextStarted])).resolves.toEqual([ + 'started at 30', + 'started at 30', + 'started at 30' + ]); + await flushAsync(); + expect(captures.started).toHaveLength(3); + captures.started[2].resolve('started after 31'); + await expect(receivedAfterNextStarted).resolves.toBe('started after 31'); + }); + + it('rejects a caller that shared a running capture that failed, as if it had asked before it started', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const failed: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + const shared: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 10); + captures.started[0].reject(new Error('a configuration file is being rewritten')); + await expect(failed).rejects.toThrow('a configuration file is being rewritten'); + await expect(shared).rejects.toThrow('a configuration file is being rewritten'); + await flushAsync(); + expect(captures.started).toHaveLength(1); + }); + + it('compares notBeforeMs with performance.now() by default', async () => { + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + const receivedTimeMs: number = performance.now(); + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + const shared: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, receivedTimeMs); + const receivedLater: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + performance.now() + 60_000 + ); + await flushAsync(); + expect(captures.started).toHaveLength(1); + captures.started[0].resolve('first'); + await expect(Promise.all([first, shared])).resolves.toEqual(['first', 'first']); + await flushAsync(); + expect(captures.started).toHaveLength(2); + captures.started[1].resolve('second'); + await expect(receivedLater).resolves.toBe('second'); + }); + it('never shares a capture between different scopes or keys', async () => { const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); const firstScope: object = {}; diff --git a/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts b/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts new file mode 100644 index 0000000000..93132421a0 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts @@ -0,0 +1,294 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('@microsoft/rush-lib', () => { + const actual: typeof import('@microsoft/rush-lib') = jest.requireActual('@microsoft/rush-lib'); + return { + ...actual, + captureWorkspaceInputFingerprintAsync: jest.fn(actual.captureWorkspaceInputFingerprintAsync), + captureProjectConfigurationFingerprintAsync: jest.fn(actual.captureProjectConfigurationFingerprintAsync) + }; +}); + +import * as rushLib from '@microsoft/rush-lib'; + +import { FreshCaptureCoalescer } from '../FreshCaptureCoalescer'; +import { DaemonGraphTestFixture, responseSnapshot } from './DaemonGraphTestFixture'; +import { setDaemonPolicy } from './WarmGenerationTestUtilities'; + +jest.setTimeout(60_000); + +type CaptureRequest = [scope: object, key: string, captureAsync: () => Promise, notBeforeMs?: number]; +type BuildExchange = ReturnType; + +interface IGate { + readonly promise: Promise; + readonly release: () => void; +} + +const WAIT_TIMEOUT_MS: number = 20_000; +const actualRushLib: typeof rushLib = jest.requireActual('@microsoft/rush-lib'); +const workspaceCaptureMock: jest.MockedFunction = + jest.mocked(rushLib.captureWorkspaceInputFingerprintAsync); +const projectCaptureMock: jest.MockedFunction = + jest.mocked(rushLib.captureProjectConfigurationFingerprintAsync); +const originalCaptureAsync: FreshCaptureCoalescer['captureAsync'] = + FreshCaptureCoalescer.prototype.captureAsync; + +function createGate(): IGate { + let release: () => void = () => {}; + const promise: Promise = new Promise((resolve) => (release = resolve)); + return { promise, release: () => release() }; +} + +async function waitForAsync(description: string, condition: () => boolean): Promise { + const deadline: number = Date.now() + WAIT_TIMEOUT_MS; + while (!condition()) { + if (Date.now() > deadline) throw new Error(`Timed out waiting until ${description}`); + await new Promise((resolve) => setTimeout(resolve, 10)); + } +} + +// The project configuration coalescer uses one key per workspace configuration. +function isProjectRequest(request: unknown[]): boolean { + return request[1] === ''; +} + +function describeRequest(request: unknown[]): [string, string] { + return [isProjectRequest(request) ? 'project' : 'workspace', typeof request[3]]; +} + +function countRequests(requests: jest.SpyInstance, kind: 'workspace' | 'project'): number { + return requests.mock.calls.filter((request: unknown[]) => describeRequest(request)[0] === kind).length; +} + +/** + * Makes the first `count` workspace input requests wait until all of them have arrived, and then makes them in + * one loop, as requests that were received together would if none of them was delayed on the way. + */ +function holdWorkspaceRequests(requests: jest.SpyInstance, count: number): void { + const waiting: (() => void)[] = []; + requests.mockImplementation(function ( + this: FreshCaptureCoalescer, + ...request: CaptureRequest + ): Promise { + if (waiting.length === count || isProjectRequest(request)) return originalCaptureAsync.apply(this, request); + return new Promise((resolve) => { + waiting.push(() => resolve(originalCaptureAsync.apply(this, request))); + if (waiting.length === count) for (const start of waiting) start(); + }); + }); +} + +function gateNextCapture( + mock: jest.MockedFunction<(...args: TArgs) => Promise>, + actual: (...args: TArgs) => Promise +): IGate { + const gate: IGate = createGate(); + mock.mockImplementationOnce(async (...args: TArgs) => { + await gate.promise; + return await actual(...args); + }); + return gate; +} + +async function expectSuccessfulBuildsAsync(builds: BuildExchange[]): Promise { + for (const exchange of await Promise.all(builds)) { + expect(exchange.terminal).toMatchObject({ payload: { exitCode: 0 } }); + } +} + +async function createWarmFixtureAsync(): Promise { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => + setDaemonPolicy(created, {}) + ); + try { + await fixture.buildSuccessfullyAsync(); + await fixture.buildSuccessfullyAsync(); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + return fixture; + } catch (error) { + await fixture[Symbol.asyncDispose](); + throw error; + } +} + +describe('workspace input captures shared from the time a request was received', () => { + const gates: IGate[] = []; + let requests: jest.SpyInstance | undefined; + let fixture: DaemonGraphTestFixture | undefined; + + function spyOnRequests(): jest.SpyInstance { + requests = jest.spyOn(FreshCaptureCoalescer.prototype, 'captureAsync'); + return requests; + } + + function gate(next: IGate): IGate { + gates.push(next); + return next; + } + + afterEach(async () => { + for (const next of gates.splice(0)) next.release(); + requests?.mockRestore(); + requests = undefined; + const disposing: DaemonGraphTestFixture | undefined = fixture; + fixture = undefined; + try { + await disposing?.[Symbol.asyncDispose](); + } finally { + workspaceCaptureMock.mockReset(); + workspaceCaptureMock.mockImplementation(actualRushLib.captureWorkspaceInputFingerprintAsync); + projectCaptureMock.mockReset(); + projectCaptureMock.mockImplementation(actualRushLib.captureProjectConfigurationFingerprintAsync); + } + }); + + it('shares the first captures of warm builds that were all received before those captures started', async () => { + fixture = await createWarmFixtureAsync(); + const warm: DaemonGraphTestFixture = fixture; + const generation: number = warm.host.workspaceGeneration; + const spy: jest.SpyInstance = spyOnRequests(); + holdWorkspaceRequests(spy, 3); + workspaceCaptureMock.mockClear(); + projectCaptureMock.mockClear(); + const projectGate: IGate = gate( + gateNextCapture(projectCaptureMock, actualRushLib.captureProjectConfigurationFingerprintAsync) + ); + + const builds: BuildExchange[] = [warm.buildAsync(), warm.buildAsync(), warm.buildAsync()]; + await waitForAsync( + 'every build asked for a project configuration capture', + () => countRequests(spy, 'project') === 3 + ); + // A strict coalescer would start a second capture for the builds that reached it after the first one started. + expect(workspaceCaptureMock).toHaveBeenCalledTimes(1); + expect(projectCaptureMock).toHaveBeenCalledTimes(1); + projectGate.release(); + + await expectSuccessfulBuildsAsync(builds); + expect(projectCaptureMock).toHaveBeenCalledTimes(1); + expect(warm.host.workspaceGeneration).toBe(generation); + expect(warm.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + }); + + it('never shares a capture that started before a build was received', async () => { + fixture = await createWarmFixtureAsync(); + const warm: DaemonGraphTestFixture = fixture; + const spy: jest.SpyInstance = spyOnRequests(); + workspaceCaptureMock.mockClear(); + projectCaptureMock.mockClear(); + const workspaceGate: IGate = gate( + gateNextCapture(workspaceCaptureMock, actualRushLib.captureWorkspaceInputFingerprintAsync) + ); + const projectGate: IGate = gate( + gateNextCapture(projectCaptureMock, actualRushLib.captureProjectConfigurationFingerprintAsync) + ); + + const first: BuildExchange = warm.buildAsync(); + await waitForAsync('the first build is capturing', () => workspaceCaptureMock.mock.calls.length === 1); + const late: BuildExchange = warm.buildAsync(); + await waitForAsync('the late build asked for a capture', () => countRequests(spy, 'workspace') === 2); + workspaceGate.release(); + await waitForAsync( + 'both builds asked for a project configuration capture', + () => countRequests(spy, 'project') === 2 + ); + // The late build was received after the first workspace capture started, so it waited for its own. + expect(workspaceCaptureMock).toHaveBeenCalledTimes(2); + projectGate.release(); + + await expectSuccessfulBuildsAsync([first, late]); + // It was received before the first project configuration capture started, so it shared that one. + expect(projectCaptureMock).toHaveBeenCalledTimes(1); + }); + + it('keeps the receipt time for builds that wait behind a transition', async () => { + fixture = await DaemonGraphTestFixture.createAsync((created) => setDaemonPolicy(created, {})); + const cold: DaemonGraphTestFixture = fixture; + const spy: jest.SpyInstance = spyOnRequests(); + workspaceCaptureMock.mockClear(); + const workspaceGate: IGate = gate( + gateNextCapture(workspaceCaptureMock, actualRushLib.captureWorkspaceInputFingerprintAsync) + ); + + const first: BuildExchange = cold.buildAsync(); + await waitForAsync('the first build is capturing', () => workspaceCaptureMock.mock.calls.length === 1); + const waiters: BuildExchange[] = [cold.buildAsync(), cold.buildAsync()]; + await waitForAsync('every build asked for a capture', () => countRequests(spy, 'workspace') === 3); + workspaceGate.release(); + await expectSuccessfulBuildsAsync([first, ...waiters]); + expect(cold.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + + const byReceipt: Map = new Map(); + for (const request of spy.mock.calls as unknown[][]) { + const notBeforeMs: unknown = request[3]; + if (typeof notBeforeMs !== 'number') continue; + byReceipt.set(notBeforeMs, [...(byReceipt.get(notBeforeMs) ?? []), describeRequest(request)[0]]); + } + // One build loads the graph. The others were admitted before it began, wait behind its transition, and then + // prepare again with the time at which they were received. + expect([...byReceipt.values()].sort((a: string[], b: string[]) => a.length - b.length)).toEqual([ + ['workspace', 'project'], + ['workspace', 'project', 'workspace', 'project'], + ['workspace', 'project', 'workspace', 'project'] + ]); + }); + + it('passes the receipt time only to the captures that decide whether a request can reuse the workspace', async () => { + fixture = await createWarmFixtureAsync(); + const warm: DaemonGraphTestFixture = fixture; + const spy: jest.SpyInstance = spyOnRequests(); + + await warm.buildSuccessfullyAsync(); + expect(warm.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + // The last capture validates the graph inputs before execution and must start after it was requested. + expect(spy.mock.calls.map(describeRequest)).toEqual([ + ['workspace', 'number'], + ['project', 'number'], + ['workspace', 'undefined'] + ]); + + spy.mockClear(); + setDaemonPolicy(warm, { warmSetMaxProjects: 19 }); + await warm.buildSuccessfullyAsync(); + expect(warm.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reload); + // A transition detects changes made while it runs, so every capture after the first one is strict. + const reload: [string, string][] = spy.mock.calls.map(describeRequest); + expect(reload[0]).toEqual(['workspace', 'number']); + expect(reload.slice(1).map(([, notBeforeMs]) => notBeforeMs)).toEqual( + reload.slice(1).map(() => 'undefined') + ); + expect(reload).toContainEqual(['project', 'undefined']); + + spy.mockClear(); + responseSnapshot(await warm.graphAsync('invalidate', '--project', 'a')); + // Graph control checks the inputs of the generation it changes, so its captures are strict as well. + expect(spy.mock.calls.map(describeRequest)).toEqual([ + ['workspace', 'undefined'], + ['project', 'undefined'] + ]); + }); + + it('checks the project configurations again with a strict capture when a transition can reuse the workspace', async () => { + fixture = await createWarmFixtureAsync(); + const warm: DaemonGraphTestFixture = fixture; + const generation: number = warm.host.workspaceGeneration; + const spy: jest.SpyInstance = spyOnRequests(); + projectCaptureMock.mockClear(); + // The project configurations changed before the build was received and changed back before its transition. + projectCaptureMock.mockImplementationOnce(async () => 'a project configuration that was changed back'); + + await warm.buildSuccessfullyAsync(); + expect(warm.host.workspaceGeneration).toBe(generation); + expect(warm.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + expect(projectCaptureMock).toHaveBeenCalledTimes(2); + expect(spy.mock.calls.map(describeRequest)).toEqual([ + ['workspace', 'number'], + ['project', 'number'], + ['workspace', 'undefined'], + ['project', 'undefined'], + ['workspace', 'undefined'] + ]); + }); +}); From 14fbc543a80b547b1ab0c1776179c8ce15ed99cc Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 18:33:59 +0000 Subject: [PATCH 030/265] [rush-lib] Hand requests to in-process Rush when a project's configuration cannot load Swarm integration step 15; original commit 69f30e84ab (merge of swarm/r01-t70 at c1b5c95adb). Scope: task 70. ch01 GATE OK board 1522 (tree 3a564b4533; WorkspaceRequestLifecycle.ts resolved with r07's board 1231 recipe: the Reuse capture stays tolerant and keeps task 94's receipt); o01 board 1291; r01 board 1250. Commits folded into this step (5): - 08aed7efd2 [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 3d786c1f78 [package-deps-hash] Name the git command that failed in the error message - a11aa6b774 [rush-lib] Name the project whose configuration an engine cannot load - 24cf595d6a [rush-lib] Report a missing project dependency file to an engine's host - c1b5c95adb [rush-daemon] Hand requests to in-process Rush when a project's configuration cannot load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...sing-shrinkwrap-deps_2026-09-28-16-20.json | 11 ++ ...-configuration-error_2026-09-28-16-20.json | 11 ++ ...nfiguration-fallback_2026-09-28-16-20.json | 11 ++ common/reviews/api/rush-lib.api.md | 10 +- .../src/ProductionDaemonRequestResolver.ts | 31 +++++ .../src/WorkspaceRequestLifecycle.ts | 27 ++++- .../ProductionDaemonRequestResolver.test.ts | 112 +++++++++++++++++- ...dCommandEngineProjectConfigurationError.ts | 27 +++++ .../src/api/RushProjectConfiguration.ts | 33 ++++-- .../src/api/WorkspaceInputFingerprint.ts | 6 +- .../api/test/RushProjectConfiguration.test.ts | 36 +++++- .../cli/scriptActions/PhasedScriptAction.ts | 30 ++++- libraries/rush-lib/src/index.ts | 1 + .../src/logic/ProjectChangeAnalyzer.ts | 79 +++++++++++- .../logic/test/ProjectChangeAnalyzer.test.ts | 106 ++++++++++++++++- 15 files changed, 498 insertions(+), 33 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r01-engine-missing-shrinkwrap-deps_2026-09-28-16-20.json create mode 100644 common/changes/@microsoft/rush/swarm-r01-engine-project-configuration-error_2026-09-28-16-20.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r01-project-configuration-fallback_2026-09-28-16-20.json create mode 100644 libraries/rush-lib/src/api/PhasedCommandEngineProjectConfigurationError.ts diff --git a/common/changes/@microsoft/rush/swarm-r01-engine-missing-shrinkwrap-deps_2026-09-28-16-20.json b/common/changes/@microsoft/rush/swarm-r01-engine-missing-shrinkwrap-deps_2026-09-28-16-20.json new file mode 100644 index 0000000000..c0323e2384 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r01-engine-missing-shrinkwrap-deps_2026-09-28-16-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "An engine that cannot take an inputs snapshot because a project's shrinkwrap-deps.json is missing, including one deleted after the engine started, now reports Rush's \"A project dependency file (...) is missing\" error instead of continuing without a snapshot. That error also takes precedence over a project configuration error, because \"rush install\" fixes both. Rush's warning for a file deleted during watch mode names the file instead of the failed Git command.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@microsoft/rush/swarm-r01-engine-project-configuration-error_2026-09-28-16-20.json b/common/changes/@microsoft/rush/swarm-r01-engine-project-configuration-error_2026-09-28-16-20.json new file mode 100644 index 0000000000..b9c6b1b1c6 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r01-engine-project-configuration-error_2026-09-28-16-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "An engine reports a project whose rush-project.json or rig cannot be loaded with the new PhasedCommandEngineProjectConfigurationError, which names the project. A host such as the Rush daemon, which loads every project, can then hand the request to in-process Rush, which loads only the projects that it selects.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r01-project-configuration-fallback_2026-09-28-16-20.json b/common/changes/@rushstack/rush-daemon/swarm-r01-project-configuration-fallback_2026-09-28-16-20.json new file mode 100644 index 0000000000..ad5c2594d6 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r01-project-configuration-fallback_2026-09-28-16-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "When the daemon cannot load a project's configuration, for example after a filtered install left the rig package of an unselected project uninstalled, it hands each request to in-process Rush with a one-line reason instead of rejecting every build, and writes the full error to the daemon log. A missing shrinkwrap-deps.json is still rejected, because in-process Rush cannot build without it either, with Rush's \"rush install\" instruction as the last line.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 94c081b8f9..35b81910c8 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -1548,6 +1548,12 @@ export class PhasedCommandEngineConfigurationChangedError extends Error { constructor(); } +// @alpha +export class PhasedCommandEngineProjectConfigurationError extends Error { + constructor(projectName: string, cause: unknown); + readonly projectName: string; +} + // @alpha export class PhasedCommandHooks { readonly createOperationsAsync: AsyncSeriesWaterfallHook<[ @@ -1625,7 +1631,9 @@ export class ProjectChangeAnalyzer { // (undocumented) protected getChangesByProject(lookup: LookupByPath, changedFiles: Map): Map>; // @internal - _tryGetSnapshotProviderAsync(projectConfigurations: ReadonlyMap, terminal: ITerminal, projectSelection?: ReadonlySet): Promise; + _tryGetSnapshotProviderAsync(projectConfigurations: ReadonlyMap, terminal: ITerminal, projectSelection?: ReadonlySet, options?: { + readonly throwOnMissingProjectShrinkwrapFile?: boolean; + }): Promise; } export { ReporterExtensionEventName } diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 141749594b..2edb24d3b7 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -3,6 +3,7 @@ import * as path from 'node:path'; +import { AlreadyReportedError } from '@rushstack/node-core-library'; import type { LockFile } from '@rushstack/node-core-library'; import { EnvironmentVariableNames, @@ -10,6 +11,7 @@ import { PhasedCommandEngine, PhasedCommandEngineBusyError, PhasedCommandEngineConfigurationChangedError, + PhasedCommandEngineProjectConfigurationError, type IPhasedCommandEngine, type IInputsSnapshot, type IOperationGraph, @@ -208,6 +210,9 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { engine = await command.createEngineAsync(this.#preparationLock); } catch (error) { if (error instanceof PhasedCommandEngineBusyError) throw error; + if (error instanceof PhasedCommandEngineProjectConfigurationError) { + throw createProjectConfigurationFallback(error); + } throw new Error(terminal.describeError(error), { cause: error }); } try { @@ -269,6 +274,32 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { } } +/** + * An engine loads the configuration of every project, whereas native Rush loads only the projects that a request + * selects. In-process Rush can therefore serve a request that the daemon cannot, for example after a filtered + * install, and it reports the error itself if the request selects the project. + */ +function createProjectConfigurationFallback( + error: PhasedCommandEngineProjectConfigurationError +): DaemonRequestDispatchError { + // The launcher log keeps the whole error for daemon diagnostics; the client prints one line. + process.stderr.write(`Warning: ${error.message}\n`); + const cause: unknown = error.cause; + const detail: string = + cause instanceof Error && !(cause instanceof AlreadyReportedError) + ? `: ${cause.message.split('\n', 1)[0]}` + : ''; + const hint: string = + (cause as { code?: unknown } | undefined)?.code === 'MODULE_NOT_FOUND' + ? ' (the daemon loads every project, so it needs a full "rush install")' + : ''; + return new DaemonRequestDispatchError( + 'unsupported', + `The daemon could not load the configuration of project "${error.projectName}"${detail}${hint}`, + { cause: error } + ); +} + function environmentIdentity(environment: Readonly>): string { return JSON.stringify(getWorkspaceFingerprintEnvironmentEntries(environment)); } diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index d6d3e60a3d..96349a42b2 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -10,6 +10,7 @@ import { EnvironmentVariableNames, getWorkspaceFingerprintEnvironmentEntries, PhasedCommandEngineBusyError, + PhasedCommandEngineProjectConfigurationError, Rush, WorkspaceInputChangeTier, WorkspaceRuntimeFingerprintCache, @@ -422,10 +423,11 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { abortSignal: client.abortSignal }); if (tier === WorkspaceInputChangeTier.Reuse) { - projectFingerprint = await this.#captureProjectFingerprintAsync(session, receivedTimeMs); + projectFingerprint = await this.#tryCaptureProjectFingerprintAsync(session, receivedTimeMs); if ( this.#boundSession !== session || this.#commandIdentity !== commandIdentity || + projectFingerprint === undefined || this.#projectFingerprint !== projectFingerprint || this.#forceReload || !session.invalidations.getSnapshot().isWatcherHealthy || @@ -516,8 +518,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { session.invalidations.getSnapshot().isWatcherHealthy && !session.invalidations.hasUnattributedUnknownChanges ) { - projectFingerprint = await this.#captureProjectFingerprintAsync(session); - if (projectFingerprint === this.#projectFingerprint) { + projectFingerprint = await this.#tryCaptureProjectFingerprintAsync(session); + if (projectFingerprint !== undefined && projectFingerprint === this.#projectFingerprint) { this.#lastReloadTier = WorkspaceInputChangeTier.Reuse; this.#gate.downgradeExclusiveLease(lease, RequestExclusivityClass.SharedBuild); return { @@ -598,7 +600,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { expectedFingerprint = after; this.#boundSession = session; this.#fingerprint = after; - this.#projectFingerprint = await this.#captureProjectFingerprintAsync(session); + this.#projectFingerprint = await this.#tryCaptureProjectFingerprintAsync(session); this.#commandIdentity = await getResolverLifecycle(resolver).getCommandParameterIdentityAsync({ envelope, workspaceSession: session, @@ -680,6 +682,23 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { ); } + /** + * Returns undefined, which must not match any fingerprint, if a project's configuration cannot be loaded. + * Binding a new generation then reports the error, or hands the request to in-process Rush, which loads only + * the projects that a request selects. + */ + async #tryCaptureProjectFingerprintAsync( + session: IWorkspaceSession, + notBeforeMs?: number + ): Promise { + try { + return await this.#captureProjectFingerprintAsync(session, notBeforeMs); + } catch (error) { + if (error instanceof PhasedCommandEngineProjectConfigurationError) return undefined; + throw error; + } + } + #classify(fingerprint: IWorkspaceInputFingerprint, mutation: boolean): WorkspaceInputChangeTier { if ( fingerprint.selectedRushVersion !== Rush.version || diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 408f74dbd5..c9c31ac580 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -79,6 +79,8 @@ interface IFixtureOptions { readonly resolver?: IDaemonRequestResolver; /** Adds the watch-only `_phase:compile:incremental` script, which passes `--incremental` to build.cjs. */ readonly incrementalScript?: boolean; + /** Uses PNPM, which installs a dependency file (shrinkwrap-deps.json) that change detection hashes per project. */ + readonly pnpm?: boolean; } class DecoratedTestResolver implements IDaemonRequestResolver { @@ -137,7 +139,7 @@ async function createFixtureAsync( 'rush.json', JSON.stringify({ rushVersion: RUSH_VERSION, - npmVersion: '10.0.0', + ...(options.pnpm ? { pnpmVersion: '9.15.9' } : { npmVersion: '10.0.0' }), // Retention assertions must not depend on the surrounding Jest worker's accumulated RSS. daemon: { warmMemoryBudgetMB: 100_000 }, projectFolderMinDepth: 2, @@ -150,7 +152,11 @@ async function createFixtureAsync( ); write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\n**/lib/\n**/node_modules/\nruns.txt\n'); write('common/temp/last-link.flag', '{}'); - write('common/config/rush/npm-shrinkwrap.json', '{"lockfileVersion":3,"packages":{}}'); + if (options.pnpm) { + write('common/config/rush/pnpm-lock.yaml', "lockfileVersion: '9.0'\n"); + } else { + write('common/config/rush/npm-shrinkwrap.json', '{"lockfileVersion":3,"packages":{}}'); + } write( 'common/config/rush/command-line.json', JSON.stringify({ @@ -207,6 +213,7 @@ async function createFixtureAsync( }) ); write(`projects/${name}/input.txt`, 'one'); + if (options.pnpm) write(`projects/${name}/.rush/temp/shrinkwrap-deps.json`, '{}'); write( `projects/${name}/build.cjs`, ` @@ -1585,16 +1592,115 @@ process.exit(23); it('rejects invalid inherited project configuration before creating a graph or running scripts', async () => { const fixture: IFixture = await createFixtureAsync(); + const stderrWrite: jest.SpyInstance = jest.spyOn(process.stderr, 'write').mockImplementation(() => true); try { fs.writeFileSync( path.join(fixture.repoRoot, 'projects/a/config/rush-project.json'), '{"extends":"./unowned.json"}' ); expect((await runAsync(fixture, 'unsupported', ['build'])).terminal).toMatchObject({ - kind: 'requestRejected' + kind: 'requestRejected', + // In-process Rush reports the error natively if the request selects the project. + payload: { + code: 'unsupported', + message: expect.stringMatching(/^The daemon could not load the configuration of project "a": /) + } }); + expect(stderrWrite).toHaveBeenCalledWith( + expect.stringMatching(/^Warning: Rush could not load the configuration of project "a": /) + ); expect(fixture.session.operationGraph).toBeUndefined(); expect(runs(fixture)).toEqual([]); + } finally { + stderrWrite.mockRestore(); + await fixture[Symbol.asyncDispose](); + } + }); + + it('hands requests to in-process Rush while a project that a filtered install skipped has no rig package', async () => { + const fixture: IFixture = await createFixtureAsync(false, 'rig'); + const rigFolder: string = path.join(fixture.repoRoot, 'projects/a/node_modules/fixture-rig'); + const installedRigFolder: string = `${rigFolder}-installed`; + const stderr: string[] = []; + const stderrWrite: jest.SpyInstance = jest + .spyOn(process.stderr, 'write') + .mockImplementation((chunk: string | Uint8Array) => stderr.push(chunk.toString()) > 0); + const expectFallbackAsync = async (requestId: string): Promise => { + stderr.length = 0; + const rejected: ITerminalExchange = await runAsync(fixture, requestId, ['build', '--only', 'c']); + expect(rejected.terminal).toMatchObject({ + kind: 'requestRejected', + payload: { + code: 'unsupported', + message: + `The daemon could not load the configuration of project "a": Cannot find module ` + + `'fixture-rig/package.json' from '${path.join(fixture.repoRoot, 'projects/a')}' ` + + '(the daemon loads every project, so it needs a full "rush install")' + } + }); + expect(stderr.join('')).toContain( + `Warning: Rush could not load the configuration of project "a": Cannot find module 'fixture-rig/package.json'` + ); + }; + try { + fs.renameSync(rigFolder, installedRigFolder); + await expectFallbackAsync('unbound'); + expect(fixture.session.operationGraph).toBeUndefined(); + + fs.renameSync(installedRigFolder, rigFolder); + const installed: ITerminalExchange = await runAsync(fixture, 'installed', ['build', '--only', 'c']); + expect(installed.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + + // Removing an installed package changes no watched file, but the next request must not reuse the graph. + fs.renameSync(rigFolder, installedRigFolder); + await expectFallbackAsync('uninstalled'); + fs.renameSync(installedRigFolder, rigFolder); + const reinstalled: ITerminalExchange = await runAsync(fixture, 'reinstalled', ['build', '--only', 'c']); + expect(reinstalled.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(runs(fixture)).toEqual(['c:one:']); + } finally { + stderrWrite.mockRestore(); + await fixture[Symbol.asyncDispose](); + } + }); + + it('rejects a missing project dependency file with the native instruction as the last line', async () => { + const fixture: IFixture = await createFixtureAsync(false, 'direct', { pnpm: true }); + const dependencyFile: string = path.join(fixture.repoRoot, 'projects/c/.rush/temp/shrinkwrap-deps.json'); + const instruction: string = + `A project dependency file (${dependencyFile}) is missing. ` + + 'You may need to run "rush install" or "rush update".'; + const expectRejectedAsync = async (requestId: string): Promise => { + const { terminal } = await runAsync(fixture, requestId, ['build', '--only', 'a']); + expect(terminal).toMatchObject({ kind: 'requestRejected', payload: { code: 'routingFailed' } }); + const { message } = (terminal as { payload: { message: string } }).payload; + expect(message.split('\n').pop()).toBe(instruction); + }; + try { + fs.rmSync(dependencyFile); + await expectRejectedAsync('cold'); + expect(fixture.session.operationGraph).toBeUndefined(); + + fs.writeFileSync(dependencyFile, '{}'); + const initial: ITerminalExchange = await runAsync(fixture, 'initial', ['build', '--only', 'a']); + expect(initial.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + fs.rmSync(dependencyFile); + await expectRejectedAsync('warm'); + + fs.writeFileSync(dependencyFile, '{}'); + const restored: ITerminalExchange = await runAsync(fixture, 'restored', ['build', '--only', 'a']); + expect(restored.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + + // A project added before "rush install" has no dependency file, so in-process Rush cannot hash the repo + // state either. Its missing rig package must not hide that instruction. + fs.rmSync(dependencyFile); + fs.rmSync(path.join(fixture.repoRoot, 'projects/c/config/rush-project.json')); + fs.writeFileSync( + path.join(fixture.repoRoot, 'projects/c/config/rig.json'), + '{"rigPackageName":"uninstalled-rig"}' + ); + await expectRejectedAsync('added'); + expect(runs(fixture)).toEqual(['a:one:']); } finally { await fixture[Symbol.asyncDispose](); } diff --git a/libraries/rush-lib/src/api/PhasedCommandEngineProjectConfigurationError.ts b/libraries/rush-lib/src/api/PhasedCommandEngineProjectConfigurationError.ts new file mode 100644 index 0000000000..063271df68 --- /dev/null +++ b/libraries/rush-lib/src/api/PhasedCommandEngineProjectConfigurationError.ts @@ -0,0 +1,27 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { AlreadyReportedError } from '@rushstack/node-core-library'; + +/** + * The engine could not load the native configuration of a project. + * + * @remarks + * An engine loads every project in the workspace, whereas a native command loads only the projects that it + * selects. A native command may therefore succeed where the engine cannot, for example after a filtered + * install that left the rig package of an unselected project uninstalled. No operation has begun. + * @alpha + */ +export class PhasedCommandEngineProjectConfigurationError extends Error { + /** The package name of the project whose configuration could not be loaded. */ + public readonly projectName: string; + + public constructor(projectName: string, cause: unknown) { + // An AlreadyReportedError has written its details to the terminal; its own message adds nothing. + const detail: string = + cause instanceof Error && !(cause instanceof AlreadyReportedError) ? `: ${cause.message}` : '.'; + super(`Rush could not load the configuration of project "${projectName}"${detail}`, { cause }); + this.name = 'PhasedCommandEngineProjectConfigurationError'; + this.projectName = projectName; + } +} diff --git a/libraries/rush-lib/src/api/RushProjectConfiguration.ts b/libraries/rush-lib/src/api/RushProjectConfiguration.ts index f9c0668547..9ce3da893d 100644 --- a/libraries/rush-lib/src/api/RushProjectConfiguration.ts +++ b/libraries/rush-lib/src/api/RushProjectConfiguration.ts @@ -21,6 +21,7 @@ import schemaJson from '../schemas/rush-project.schema.json'; import anythingSchemaJson from '../schemas/anything.schema.json'; import { HotlinkManager } from '../utilities/HotlinkManager'; import type { RushConfiguration } from './RushConfiguration'; +import { PhasedCommandEngineProjectConfigurationError } from './PhasedCommandEngineProjectConfigurationError'; /** * Describes the file structure for the `/config/rush-project.json` config file. @@ -572,6 +573,10 @@ export class RushProjectConfiguration { /** * Loads a fresh native configuration snapshot without reading or modifying process-wide * project, inherited-file, or rig caches. The loaders are owned only by this invocation. + * + * @remarks + * Throws a {@link PhasedCommandEngineProjectConfigurationError} that names a project whose + * configuration could not be loaded. * @internal */ public static async _tryLoadForProjectsUncachedAsync( @@ -589,20 +594,24 @@ export class RushProjectConfiguration { await Async.forEachAsync( projects, async (project) => { - const rushProjectJson: IRushProjectJson | undefined = await _tryLoadJsonForProjectAsync( - project, - terminal, - loaders - ); - if (rushProjectJson) { - result.set( + try { + const rushProjectJson: IRushProjectJson | undefined = await _tryLoadJsonForProjectAsync( project, - new RushProjectConfiguration( - project, - rushProjectJson, - _getRushProjectConfiguration(project, rushProjectJson, terminal) - ) + terminal, + loaders ); + if (rushProjectJson) { + result.set( + project, + new RushProjectConfiguration( + project, + rushProjectJson, + _getRushProjectConfiguration(project, rushProjectJson, terminal) + ) + ); + } + } catch (error) { + throw new PhasedCommandEngineProjectConfigurationError(project.packageName, error); } }, { concurrency: 50 } diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index 5fa2acbe4c..b8914cfab9 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -421,7 +421,11 @@ export async function captureWorkspaceInputFingerprintAsync( }; } -/** Fingerprints native merged project/rig/inherited configuration using invocation-owned loader caches. @alpha */ +/** + * Fingerprints native merged project/rig/inherited configuration using invocation-owned loader caches. + * Throws a {@link PhasedCommandEngineProjectConfigurationError} if a project's configuration cannot be loaded. + * @alpha + */ export async function captureProjectConfigurationFingerprintAsync( rushConfiguration: RushConfiguration, terminal: ITerminal diff --git a/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts b/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts index 61e4dbee64..8db1a1fb44 100644 --- a/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts @@ -5,13 +5,14 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; -import { FileSystem, Path } from '@rushstack/node-core-library'; +import { AlreadyReportedError, FileSystem, Path } from '@rushstack/node-core-library'; import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import type { CommandLineParameter } from '@rushstack/ts-command-line'; import type { IPhase } from '../CommandLineConfiguration'; import type { RushConfigurationProject } from '../RushConfigurationProject'; import { RushProjectConfiguration } from '../RushProjectConfiguration'; +import { PhasedCommandEngineProjectConfigurationError } from '../PhasedCommandEngineProjectConfigurationError'; // eslint-disable-next-line @typescript-eslint/no-explicit-any function stripSymbolsFromObject(obj: any | undefined): void { @@ -167,6 +168,39 @@ describe(RushProjectConfiguration.name, () => { expect(getOutputFolderNames(configurations.get(ownFile))).toEqual(['from-project']); }); + it('names the project whose rig package is not installed, with the native error as the cause', async () => { + write('uninstalled/package.json', { name: 'uninstalled', version: '1.0.0' }); + write('uninstalled/config/rig.json', { rigPackageName: 'uninstalled-rig' }); + const uninstalled: RushConfigurationProject = project('uninstalled'); + const nativeError: Error = await RushProjectConfiguration.tryLoadForProjectAsync( + uninstalled, + new Terminal(new StringBufferTerminalProvider()) + ).then( + () => { + throw new Error('The native load succeeded.'); + }, + (error: Error) => error + ); + expect(nativeError).toMatchObject({ code: 'MODULE_NOT_FOUND' }); + + const error: unknown = await loadAsync(project('rigged'), uninstalled).catch((e: unknown) => e); + expect(error).toBeInstanceOf(PhasedCommandEngineProjectConfigurationError); + expect(error).toMatchObject({ + projectName: 'uninstalled', + message: `Rush could not load the configuration of project "uninstalled": ${nativeError.message}`, + cause: { code: 'MODULE_NOT_FOUND', message: nativeError.message } + }); + }); + + it('does not repeat the message of an error that has already been reported', () => { + expect( + new PhasedCommandEngineProjectConfigurationError('reported', new AlreadyReportedError()) + ).toMatchObject({ + projectName: 'reported', + message: 'Rush could not load the configuration of project "reported".' + }); + }); + it('loads a rig profile shared through per-project node_modules symlinks once, with native results', async () => { write('store/example-rig/package.json', { name: 'example-rig', version: '1.0.0' }); write('store/example-rig/profiles/default/config/rush-project.json', { diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index fe92a4c5ac..0e6b9ba4a1 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -55,7 +55,10 @@ import { associateParametersByPhase } from '../parsing/associateParametersByPhas import { PhasedOperationPlugin } from '../../logic/operations/PhasedOperationPlugin'; import { ShellOperationRunnerPlugin } from '../../logic/operations/ShellOperationRunnerPlugin'; import { Event } from '../../api/EventHooks'; -import { ProjectChangeAnalyzer } from '../../logic/ProjectChangeAnalyzer'; +import { + ProjectChangeAnalyzer, + tryGetMissingProjectShrinkwrapFileErrorAsync +} from '../../logic/ProjectChangeAnalyzer'; import { OperationStatus } from '../../logic/operations/OperationStatus'; import type { IExecutionResult } from '../../logic/operations/IOperationExecutionResult'; import { OperationResultSummarizerPlugin } from '../../logic/operations/OperationResultSummarizerPlugin'; @@ -732,7 +735,7 @@ export class PhasedScriptAction extends BaseScriptAction i ? new Map() : await measureAsyncFn(`${PERF_PREFIX}:loadProjectConfigurations`, () => onEngine - ? RushProjectConfiguration._tryLoadForProjectsUncachedAsync(relevantProjects, terminal) + ? this.#loadEngineProjectConfigurationsAsync(relevantProjects, terminal) : RushProjectConfiguration.tryLoadForProjectsAsync(relevantProjects, terminal) ); const projectConfigurationIdentity: string | undefined = onEngine @@ -781,7 +784,9 @@ export class PhasedScriptAction extends BaseScriptAction i projectConfigurations, terminal, // We need to include all dependencies, otherwise build cache id calculation will be incorrect - relevantProjects + relevantProjects, + // An engine cannot continue without a snapshot, so it reports why none could be taken. + { throwOnMissingProjectShrinkwrapFile: !!onEngine } ); const innerInitialSnapshot: IInputsSnapshot | undefined = innerGetInputsSnapshotAsync ? await innerGetInputsSnapshotAsync() @@ -820,7 +825,7 @@ export class PhasedScriptAction extends BaseScriptAction i ? async () => { await this.#validateInstallStateAsync(); const currentConfigurations: ReadonlyMap = - await RushProjectConfiguration._tryLoadForProjectsUncachedAsync(relevantProjects, terminal); + await this.#loadEngineProjectConfigurationsAsync(relevantProjects, terminal); if ( (await getProjectConfigurationIdentityAsync( currentConfigurations, @@ -1025,6 +1030,23 @@ export class PhasedScriptAction extends BaseScriptAction i } } + /** + * Loads the configuration of every specified project for an engine, which loads projects that a native + * command might not select. + */ + async #loadEngineProjectConfigurationsAsync( + projects: ReadonlySet, + terminal: ITerminal + ): Promise> { + try { + return await RushProjectConfiguration._tryLoadForProjectsUncachedAsync(projects, terminal); + } catch (error) { + // An incomplete install can leave both a project dependency file and a rig package missing. Without the + // file, native Rush cannot analyze the repo state either, so report it: "rush install" fixes both. + throw (await tryGetMissingProjectShrinkwrapFileErrorAsync(this.rushConfiguration)) ?? error; + } + } + async #validateInstallStateAsync(): Promise { if (!this._runsBeforeInstall) { await measureAsyncFn(`${PERF_PREFIX}:checkInstallFlag`, async () => { diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index 33caab8118..b45248e94f 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -182,6 +182,7 @@ export { type IParsePhasedCommandOptions } from './api/PhasedCommandEngine'; export { PhasedCommandEngineConfigurationChangedError } from './api/PhasedCommandEngineConfigurationChangedError'; +export { PhasedCommandEngineProjectConfigurationError } from './api/PhasedCommandEngineProjectConfigurationError'; export { PhasedCommandEngineBusyError } from './api/PhasedCommandEngineBusyError'; export { captureWorkspaceInputFingerprintAsync, diff --git a/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts b/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts index 14dcadd0ba..ed5353620a 100644 --- a/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts +++ b/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts @@ -86,6 +86,18 @@ export interface IRawRepoState { rawHashes: Map; } +/** + * A project dependency file (shrinkwrap-deps.json) that change detection hashes is missing. + */ +class MissingProjectShrinkwrapFileError extends Error { + public constructor(projectShrinkwrapFilePath: string) { + super( + `A project dependency file (${projectShrinkwrapFilePath}) is missing. You may need to run ` + + '"rush install" or "rush update".' + ); + } +} + /** * @beta */ @@ -316,13 +328,21 @@ export class ProjectChangeAnalyzer { /** * Gets a snapshot of the input state of the Rush workspace that can be queried for incremental * build operations and use by the build cache. + * + * @remarks + * If the state cannot be calculated, this writes a warning and continues without a snapshot. With + * `throwOnMissingProjectShrinkwrapFile`, a missing project dependency file (shrinkwrap-deps.json) instead + * throws its error, whether it is detected when the provider is created or when a snapshot fails. A host that + * cannot continue without a snapshot can then report that actionable cause. * @internal */ public async _tryGetSnapshotProviderAsync( projectConfigurations: ReadonlyMap, terminal: ITerminal, - projectSelection?: ReadonlySet + projectSelection?: ReadonlySet, + options?: { readonly throwOnMissingProjectShrinkwrapFile?: boolean } ): Promise { + const throwOnMissingProjectShrinkwrapFile: boolean = !!options?.throwOnMissingProjectShrinkwrapFile; try { const gitPath: string = this.#git.getGitPathOrThrow(); @@ -375,6 +395,7 @@ export class ProjectChangeAnalyzer { // Include project shrinkwrap files as part of the computation const additionalRelativePathsToHash: string[] = []; + const projectShrinkwrapFilePaths: string[] = []; const globalAdditionalFiles: string[] = []; if (rushConfiguration.isPnpm) { await Async.forEachAsync(rushConfiguration.projects, async (project: RushConfigurationProject) => { @@ -384,16 +405,16 @@ export class ProjectChangeAnalyzer { return; } - throw new Error( - `A project dependency file (${projectShrinkwrapFilePath}) is missing. You may need to run ` + - '"rush install" or "rush update".' - ); + throw new MissingProjectShrinkwrapFileError(projectShrinkwrapFilePath); } const relativeProjectShrinkwrapFilePath: string = Path.convertToSlashes( path.relative(rootDirectory, projectShrinkwrapFilePath) ); additionalRelativePathsToHash.push(relativeProjectShrinkwrapFilePath); + if (!rushConfiguration.subspacesFeatureEnabled) { + projectShrinkwrapFilePaths.push(projectShrinkwrapFilePath); + } }); } else { // Add the shrinkwrap file to every project's dependencies @@ -465,11 +486,18 @@ export class ProjectChangeAnalyzer { workingTreeReadStartTimeMs }); } catch (e) { + // The files were checked once, when this provider was created. A file removed since then fails + // "git hash-object" with an obscure message, so check them again, only after a failure. + const error: Error = (await tryGetMissingFileErrorAsync(projectShrinkwrapFilePaths)) ?? e; + if (throwOnMissingProjectShrinkwrapFile && error instanceof MissingProjectShrinkwrapFileError) { + throw error; + } + // If getRepoState fails, don't fail the whole build. Treat this case as if we don't know anything about // the state of the files in the repo. This can happen if the environment doesn't have Git. terminal.writeWarningLine( `Error calculating the state of the repo. (inner error: ${ - e.stack ?? e.message ?? e + error.stack ?? error.message ?? error }). Continuing without diffing files.` ); @@ -477,6 +505,10 @@ export class ProjectChangeAnalyzer { } }; } catch (e) { + if (throwOnMissingProjectShrinkwrapFile && e instanceof MissingProjectShrinkwrapFileError) { + throw e; + } + // If getRepoState fails, don't fail the whole build. Treat this case as if we don't know anything about // the state of the files in the repo. This can happen if the environment doesn't have Git. terminal.writeWarningLine( @@ -687,6 +719,41 @@ async function isVersionBumpChangeAsync( } } +/** + * Returns an error naming the first of the specified project dependency files that is missing, if any. + */ +async function tryGetMissingFileErrorAsync( + projectShrinkwrapFilePaths: ReadonlyArray +): Promise { + const exists: boolean[] = await Async.mapAsync( + projectShrinkwrapFilePaths, + (filePath: string) => FileSystem.existsAsync(filePath), + { concurrency: 50 } + ); + const missingIndex: number = exists.indexOf(false); + return missingIndex < 0 + ? undefined + : new MissingProjectShrinkwrapFileError(projectShrinkwrapFilePaths[missingIndex]); +} + +/** + * Returns the error that {@link ProjectChangeAnalyzer._tryGetSnapshotProviderAsync} reports for a missing + * project dependency file, if any is missing. This happens when a project is added before "rush install". + */ +export async function tryGetMissingProjectShrinkwrapFileErrorAsync( + rushConfiguration: RushConfiguration +): Promise { + if (!rushConfiguration.isPnpm || rushConfiguration.subspacesFeatureEnabled) { + return undefined; + } + + return await tryGetMissingFileErrorAsync( + rushConfiguration.projects.map((project: RushConfigurationProject) => + BaseProjectShrinkwrapFile.getFilePathForProject(project) + ) + ); +} + interface IAdditionalGlob { project: RushConfigurationProject; operationName: string; diff --git a/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts b/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts index 8bb271d8bd..5d85cffa1d 100644 --- a/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts +++ b/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts @@ -114,6 +114,7 @@ jest.mock('../incremental/InputsSnapshot', () => { }; }); +import * as fs from 'node:fs'; import { resolve } from 'node:path'; import type { IDetailedRepoState, IFileDiffStatus } from '@rushstack/package-deps-hash'; @@ -122,7 +123,8 @@ import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import { ProjectChangeAnalyzer, isPackageJsonVersionBumpChange, - isPackageJsonVersionOnlyChange + isPackageJsonVersionOnlyChange, + tryGetMissingProjectShrinkwrapFileErrorAsync } from '../ProjectChangeAnalyzer'; import { RushConfiguration } from '../../api/RushConfiguration'; import type { @@ -182,6 +184,108 @@ describe(ProjectChangeAnalyzer.name, () => { expect(mockInput.workingTreeReadStartTimeMs).toBeGreaterThanOrEqual(beforeSnapshotTimeMs); expect(mockInput.workingTreeReadStartTimeMs).toBeLessThanOrEqual(repoStateReadTimeMs!); }); + + describe('with PNPM project dependency files', () => { + let folder: string; + let rushConfiguration: RushConfiguration; + let terminalProvider: StringBufferTerminalProvider; + let terminal: Terminal; + const dependencyFile = (name: string): string => + resolve(folder, name, '.rush/temp/shrinkwrap-deps.json'); + const instruction = (name: string): string => + `A project dependency file (${dependencyFile(name)}) is missing. ` + + 'You may need to run "rush install" or "rush update".'; + const gitFailure: () => never = () => { + throw new Error('git hash-object failed'); + }; + const hostOptions: { throwOnMissingProjectShrinkwrapFile: boolean } = { + throwOnMissingProjectShrinkwrapFile: true + }; + + beforeEach(() => { + // Change detection requires a Git working tree, which contains this folder. + folder = fs.mkdtempSync(resolve(__dirname, 'shrinkwrap-deps-')); + fs.writeFileSync( + resolve(folder, 'rush.json'), + JSON.stringify({ + rushVersion: '5.162.0', + pnpmVersion: '9.15.9', + projects: ['a', 'b'].map((name) => ({ packageName: name, projectFolder: name })) + }) + ); + for (const name of ['a', 'b']) { + fs.mkdirSync(resolve(folder, name, '.rush/temp'), { recursive: true }); + fs.writeFileSync(resolve(folder, name, 'package.json'), JSON.stringify({ name, version: '1.0.0' })); + fs.writeFileSync(dependencyFile(name), '{}'); + } + rushConfiguration = RushConfiguration.loadFromConfigurationFile(resolve(folder, 'rush.json')); + terminalProvider = new StringBufferTerminalProvider(); + terminal = new Terminal(terminalProvider); + }); + + afterEach(() => { + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it('warns and continues without a snapshot provider if a file is missing', async () => { + fs.rmSync(dependencyFile('b')); + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + await expect(analyzer._tryGetSnapshotProviderAsync(new Map(), terminal)).resolves.toBeUndefined(); + expect(terminalProvider.getWarningOutput()).toContain(instruction('b')); + }); + + it('throws the error of a missing file for a host that requires a snapshot', async () => { + fs.rmSync(dependencyFile('b')); + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + await expect( + analyzer._tryGetSnapshotProviderAsync(new Map(), terminal, undefined, hostOptions) + ).rejects.toThrow(new Error(instruction('b'))); + expect(terminalProvider.getWarningOutput()).toBe(''); + }); + + it('reports a file that was removed after the provider was created instead of the Git failure', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const nativeProvider: GetInputsSnapshotAsyncFn | undefined = + await analyzer._tryGetSnapshotProviderAsync(new Map(), terminal); + const hostProvider: GetInputsSnapshotAsyncFn | undefined = + await analyzer._tryGetSnapshotProviderAsync(new Map(), terminal, undefined, hostOptions); + fs.rmSync(dependencyFile('a')); + mockOnGetDetailedRepoState.mockImplementationOnce(gitFailure).mockImplementationOnce(gitFailure); + + await expect(nativeProvider!()).resolves.toBeUndefined(); + expect(terminalProvider.getWarningOutput()).toContain(instruction('a')); + expect(terminalProvider.getWarningOutput()).not.toContain('git hash-object failed'); + await expect(hostProvider!()).rejects.toThrow(new Error(instruction('a'))); + }); + + it('continues without a snapshot after other failures, even for a host that requires one', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const hostProvider: GetInputsSnapshotAsyncFn | undefined = + await analyzer._tryGetSnapshotProviderAsync(new Map(), terminal, undefined, hostOptions); + mockOnGetDetailedRepoState.mockImplementationOnce(gitFailure); + + await expect(hostProvider!()).resolves.toBeUndefined(); + expect(terminalProvider.getWarningOutput()).toContain('git hash-object failed'); + }); + + it('finds the first missing file in project order', async () => { + await expect( + tryGetMissingProjectShrinkwrapFileErrorAsync(rushConfiguration) + ).resolves.toBeUndefined(); + fs.rmSync(dependencyFile('b')); + fs.rmSync(dependencyFile('a')); + await expect(tryGetMissingProjectShrinkwrapFileErrorAsync(rushConfiguration)).resolves.toEqual( + new Error(instruction('a')) + ); + }); + }); + + it('never reports a missing project dependency file for a package manager that has none', async () => { + const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile( + resolve(__dirname, 'repo', 'rush.json') + ); + await expect(tryGetMissingProjectShrinkwrapFileErrorAsync(rushConfiguration)).resolves.toBeUndefined(); + }); }); describe(ProjectChangeAnalyzer.prototype.getChangedProjectsAsync.name, () => { From bd7bc498d42e6b1514f388475937c53f1eb63248 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 18:39:57 +0000 Subject: [PATCH 031/265] [rush-lib] With cache writes off, a result whose inputs changed during the run is not trusted as up to date Swarm integration step 16; original commit d6ac7c43f3 (merge of swarm/r06-t92 at 72c2f0e4f8). Scope: task 92. ch01 GATE OK board 1563 (tree 5c0b0ac25d); r06 board 1270; CONFIRMED by t03, t04, ch05 and o03. Commits folded into this step (1): - 72c2f0e4f8 [rush-lib] Re-run operations whose inputs changed during the iteration, also with cache writes off Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...nverifiable-changed-inputs_2026-09-28.json | 11 ++ ...nverifiable-changed-inputs_2026-09-28.json | 11 ++ libraries/rush-daemon/README.md | 4 + .../operations/CacheableOperationPlugin.ts | 20 +++- .../logic/operations/PhasedOperationPlugin.ts | 21 +++- .../operations/RetainedResultVerification.ts | 23 ++++ .../test/CacheableOperationPlugin.test.ts | 106 +++++++++++++++++- ...asedOperationPluginRetainedResults.test.ts | 29 ++++- 8 files changed, 208 insertions(+), 17 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-unverifiable-changed-inputs_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-unverifiable-changed-inputs_2026-09-28.json diff --git a/common/changes/@microsoft/rush/rushd-unverifiable-changed-inputs_2026-09-28.json b/common/changes/@microsoft/rush/rushd-unverifiable-changed-inputs_2026-09-28.json new file mode 100644 index 0000000000..dd6bcf7778 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-unverifiable-changed-inputs_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Fix an issue where a long-lived operation graph, such as the Rush daemon, with build cache writes disabled kept an operation as up to date after its tracked input files changed while the inputs snapshot was being taken or while the operation was executing, so that a later build after the files were changed back skipped the operation and its consumers over outputs built from the changed files.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-unverifiable-changed-inputs_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-unverifiable-changed-inputs_2026-09-28.json new file mode 100644 index 0000000000..76c1432beb --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-unverifiable-changed-inputs_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Document that a cacheable operation whose tracked input files changed while the inputs snapshot was taken or while it executed is run again by the next request.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} \ No newline at end of file diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 250201ec20..a28234b3df 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -98,6 +98,10 @@ unchanged and are checked normally. Phased operation processes inherit the daemo see the daemon's startup values for the ignored variables rather than the submitting shell's values. Compatible selections reuse the same graph and records. An unchanged successful build schedules no work; rebuild still invalidates the graph on each request. Every execution refreshes operation inputs under its native lease. +With the build cache enabled, a cacheable operation whose tracked input files change while the inputs snapshot is +taken or while it executes is not kept as up to date, whether or not cache writes are allowed: the next request runs +it and its consumers again, even if the files were changed back in between. Operations whose build cache is +disabled, and workspaces without a build cache, don't get this check. A generation lease spans resolution through final output. Reload also takes exclusive workspace admission and the native preparation lock, discards paused prepared work, and awaits old runner/plugin/watcher cleanup before diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index b3b3e10ea2..5f96073993 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -50,7 +50,7 @@ import type { IOperationGraph, IOperationGraphIterationOptions } from './IOperat import type { BuildCacheConfiguration } from '../../api/BuildCacheConfiguration'; import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; import type { OperationExecutionRecord } from './OperationExecutionRecord'; -import { enableUnverifiedRetainedOperations } from './RetainedResultVerification'; +import { enableUnverifiedRetainedOperations, markResultUnverifiable } from './RetainedResultVerification'; const PLUGIN_NAME: 'CacheablePhasedOperationPlugin' = 'CacheablePhasedOperationPlugin'; const PERIODIC_CALLBACK_INTERVAL_IN_SECONDS: number = 10; @@ -84,8 +84,9 @@ export interface IOperationBuildCacheContext { isCacheReadAttempted: boolean; // The on-disk state of the tracked input files whose hashes produced the cache key, captured right after - // the iteration's inputs snapshot. Used to refuse cache writes if the inputs changed while the snapshot was - // being taken or while the operation was executing. + // the iteration's inputs snapshot. Used to refuse cache writes, and to keep a long-lived graph from skipping + // the operation later, if the inputs changed while the snapshot was being taken or while the operation was + // executing. inputFilesState?: IInputFilesState; // The hashes of the tracked input files in the iteration's inputs snapshot inputFileHashes?: ReadonlyMap; @@ -271,8 +272,10 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { disjointSet?.add(operation); + // Captured even if cache writes are disabled, since a long-lived graph (e.g. the Rush daemon) must not + // retain outputs that were built from input files that changed during the iteration. const inputFilesState: IInputFilesState | undefined = - cacheWriteEnabled && !cacheDisabledReason && record.enabled + !cacheDisabledReason && record.enabled ? captureInputFilesState( inputsSnapshot.rootDirectory, fileHashes.keys(), @@ -645,7 +648,7 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { } const { inputFilesState, inputFileHashes } = buildCacheContext; let inputFilesChangedMessage: string | undefined; - if (!cacheRestored && isCacheWriteAllowed && inputFilesState) { + if (!cacheRestored && inputFilesState) { // If Git hashed a file that was saved during the snapshot before it was saved, the outputs were // built from newer content than the cache key describes. const haveSnapshotHashesChanged: boolean = @@ -677,9 +680,14 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { // The cache key was derived from the iteration's inputs snapshot. Storing outputs produced from // edited inputs under that key would poison the cache for every consumer of the entry. // Consumers' cache keys also embed this operation's pre-edit state, so block their writes too. - buildCacheTerminal.writeLine(inputFilesChangedMessage); + if (isCacheWriteAllowed) { + buildCacheTerminal.writeLine(inputFilesChangedMessage); + } buildCacheContext.isCacheWriteAllowed = false; setCacheEntryPromise = undefined; + // For the same reason, a long-lived graph must not skip this operation, or the consumers built against + // its outputs, while the state hash is unchanged, whether or not cache writes are enabled. + markResultUnverifiable(record); } if (!cacheRestored) { const cacheWriteSuccess: boolean | undefined = await setCacheEntryPromise?.(); diff --git a/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts index 1584cba611..093ce3fc7f 100644 --- a/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts @@ -19,9 +19,12 @@ import type { } from './IOperationExecutionResult'; import { OperationStatus, SUCCESS_STATUSES } from './OperationStatus'; import type { IInputsSnapshot } from '../incremental/InputsSnapshot'; -import { enableUnverifiedRetainedOperations } from './RetainedResultVerification'; +import { enableUnverifiedRetainedOperations, isResultUnverifiable } from './RetainedResultVerification'; const PLUGIN_NAME: 'PhasedOperationPlugin' = 'PhasedOperationPlugin'; +// Runs after the default-stage taps (e.g. CacheableOperationPlugin's input file checks), which can mark a result as +// unverifiable. +const VERIFY_RESULT_STAGE: number = 1; /** * Core phased command plugin that provides the functionality for generating a base operation graph @@ -169,11 +172,14 @@ function configureExecutionManager(graph: IOperationGraph, context: IOperationGr } ); - graph.hooks.afterExecuteOperationAsync.tap(PLUGIN_NAME, (record: IOperationExecutionResult) => { - if (iterationRecords) { - updateVerifiedStateHash(record, iterationRecords, verifiedStateHashByOperation); + graph.hooks.afterExecuteOperationAsync.tap( + { name: PLUGIN_NAME, stage: VERIFY_RESULT_STAGE }, + (record: IOperationExecutionResult) => { + if (iterationRecords) { + updateVerifiedStateHash(record, iterationRecords, verifiedStateHashByOperation); + } } - }); + ); graph.hooks.afterExecuteIterationAsync.tap(PLUGIN_NAME, (status: OperationStatus) => { iterationRecords = undefined; @@ -202,7 +208,10 @@ function updateVerifiedStateHash( case OperationStatus.Success: case OperationStatus.SuccessWithWarning: case OperationStatus.NoOp: { - if (areDependenciesVerified(operation, records, verifiedStateHashByOperation)) { + if ( + !isResultUnverifiable(record) && + areDependenciesVerified(operation, records, verifiedStateHashByOperation) + ) { verifiedStateHashByOperation.set(operation, record.getStateHash()); return; } diff --git a/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts b/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts index e354e44463..f15be001ff 100644 --- a/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts +++ b/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts @@ -5,6 +5,29 @@ import type { Operation } from './Operation'; import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; import { SUCCESS_STATUSES } from './OperationStatus'; +const unverifiableResults: WeakSet = new WeakSet(); + +/** + * Records that the outputs of a result of the executing iteration may not match its state hash, e.g. because input + * files of the operation changed while the inputs snapshot was being taken or while the operation was executing. + * Such a result is never verified at its state hash, so a later iteration of a long-lived graph runs the operation + * again, and the consumers that were built against its outputs, instead of skipping them. + * + * @remarks + * Call this from an `afterExecuteOperationAsync` tap with the default stage. The taps that verify results use a + * later stage. + */ +export function markResultUnverifiable(result: IOperationExecutionResult): void { + unverifiableResults.add(result); +} + +/** + * Returns true if `markResultUnverifiable` was called for the result. + */ +export function isResultUnverifiable(result: IOperationExecutionResult): boolean { + return unverifiableResults.has(result); +} + /** * Re-enables selected operations whose successful result retained by a previous iteration of a long-lived graph * (e.g. the Rush daemon) is current by state hash, but not verified at that state hash, so that they are restored diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts index 6c7f1cb80f..b0d532c5c8 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts @@ -92,14 +92,17 @@ class CacheableMockRunner implements IOperationRunner { public readonly isNoOp: boolean = false; public readonly name: string; readonly #executions: string[]; + readonly #onExecute: ((name: string) => void) | undefined; - public constructor(name: string, executions: string[]) { + public constructor(name: string, executions: string[], onExecute?: (name: string) => void) { this.name = name; this.#executions = executions; + this.#onExecute = onExecute; } public async executeAsync(context: IOperationRunnerContext): Promise { this.#executions.push(this.name); + this.#onExecute?.(this.name); return OperationStatus.Success; } @@ -116,19 +119,26 @@ interface ITestGraph { trackedFileHashes: Map>; executions: string[]; cacheWrites: string[]; + // Called when an operation executes, e.g. to save one of its input files while it executes + onExecute: ((name: string) => void) | undefined; executeAsync(workingTreeReadStartTimeMs?: number): Promise; } /** * Creates a linear chain of cacheable operations: names[0] <- names[1] <- ... (each depends on the previous). */ -async function createTestGraphAsync(names: string[], rootDirectory: string = '/repo'): Promise { +async function createTestGraphAsync( + names: string[], + rootDirectory: string = '/repo', + cacheWriteEnabled: boolean = true +): Promise { const executions: string[] = []; const cacheWrites: string[] = []; const localHashes: Map = new Map(); const trackedFileHashes: Map> = new Map(); const operations: Map = new Map(); const projectConfigurations: Map = new Map(); + let onExecute: ((name: string) => void) | undefined; let previous: Operation | undefined; for (const name of names) { @@ -140,7 +150,7 @@ async function createTestGraphAsync(names: string[], rootDirectory: string = '/r getCacheDisabledReason: () => undefined } as unknown as RushProjectConfiguration); const operation: Operation = new Operation({ - runner: new CacheableMockRunner(name, executions), + runner: new CacheableMockRunner(name, executions, (executedName: string) => onExecute?.(executedName)), logFilenameIdentifier: name, phase: mockPhase, project @@ -171,7 +181,7 @@ async function createTestGraphAsync(names: string[], rootDirectory: string = '/r allowWarningsInSuccessfulBuild: false, buildCacheConfiguration: { buildCacheEnabled: true, - cacheWriteEnabled: true + cacheWriteEnabled } as unknown as BuildCacheConfiguration, cobuildConfiguration: undefined, terminal, @@ -199,6 +209,12 @@ async function createTestGraphAsync(names: string[], rootDirectory: string = '/r trackedFileHashes, executions, cacheWrites, + get onExecute(): ((name: string) => void) | undefined { + return onExecute; + }, + set onExecute(value: ((name: string) => void) | undefined) { + onExecute = value; + }, executeAsync: async (workingTreeReadStartTimeMs?: number) => { executions.length = 0; cacheWrites.length = 0; @@ -408,5 +424,87 @@ describe(CacheableOperationPlugin.name, () => { expect(testGraph.cacheWrites).toEqual(['a', 'b']); expect(jest.mocked(hashFilesAsync)).not.toHaveBeenCalled(); }); + + it.each([true, false])( + 'runs the operations again if a later snapshot has the same state hashes (cache writes enabled: %s)', + async (cacheWriteEnabled: boolean) => { + const testGraph: ITestGraph = await createTestGraphAsync( + ['a', 'b'], + rootDirectory, + cacheWriteEnabled + ); + const inputFilePath: string = path.join(rootDirectory, inputFile); + testGraph.trackedFileHashes.set('a', new Map([[inputFile, getGitBlobHash('export const a = 1;')]])); + + await testGraph.executeAsync(Date.now()); + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual([]); + + // Reverting the save gives the next snapshot the same state hashes, but the outputs of "a" and "b" were + // built from the saved content. + fs.writeFileSync(inputFilePath, 'export const a = 1;'); + const afterRevertMs: number = getLatestFileTimeMs(inputFilePath) + FILE_TIME_TOLERANCE_MS + 1; + await testGraph.executeAsync(afterRevertMs); + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual(cacheWriteEnabled ? ['a', 'b'] : []); + + const hotResult: IExecutionResult = await testGraph.executeAsync(afterRevertMs); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + } + ); + }); + + describe('input files saved while an operation executes', () => { + const inputFile: string = 'a/src/index.ts'; + let rootDirectory: string; + let inputFilePath: string; + + beforeEach(() => { + rootDirectory = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-cacheable-'))); + fs.mkdirSync(path.join(rootDirectory, 'a', 'src'), { recursive: true }); + inputFilePath = path.join(rootDirectory, inputFile); + fs.writeFileSync(inputFilePath, 'export const a = 1;'); + }); + + afterEach(() => { + fs.rmSync(rootDirectory, { recursive: true, force: true }); + }); + + it.each([true, false])( + 'runs the operations again if a later snapshot has the same state hashes (cache writes enabled: %s)', + async (cacheWriteEnabled: boolean) => { + const testGraph: ITestGraph = await createTestGraphAsync( + ['a', 'b'], + rootDirectory, + cacheWriteEnabled + ); + testGraph.trackedFileHashes.set('a', new Map([[inputFile, getGitBlobHash('export const a = 1;')]])); + const afterWriteMs: number = getLatestFileTimeMs(inputFilePath) + FILE_TIME_TOLERANCE_MS + 1; + testGraph.onExecute = (name: string) => { + if (name === 'a') { + // A different size, so that the save is detected even if the file time does not change + fs.writeFileSync(inputFilePath, 'export const a = 22;'); + } + }; + + await testGraph.executeAsync(afterWriteMs); + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual([]); + + // Reverting the save gives the next snapshot the same state hashes, but the outputs of "a" and "b" may + // have been built from the saved content. + testGraph.onExecute = undefined; + fs.writeFileSync(inputFilePath, 'export const a = 1;'); + const afterRevertMs: number = getLatestFileTimeMs(inputFilePath) + FILE_TIME_TOLERANCE_MS + 1; + await testGraph.executeAsync(afterRevertMs); + expect(testGraph.executions).toEqual(['a', 'b']); + expect(testGraph.cacheWrites).toEqual(cacheWriteEnabled ? ['a', 'b'] : []); + + const hotResult: IExecutionResult = await testGraph.executeAsync(afterRevertMs); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + } + ); }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts index 325a1ea315..d748569f95 100644 --- a/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts @@ -36,7 +36,8 @@ import { OperationGraph } from '../OperationGraph'; import { Operation } from '../Operation'; import { OperationStatus } from '../OperationStatus'; import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; -import type { IExecutionResult } from '../IOperationExecutionResult'; +import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; +import { markResultUnverifiable } from '../RetainedResultVerification'; const mockPhase: IPhase = { name: 'phase', @@ -303,6 +304,32 @@ describe(`${PhasedOperationPlugin.name} retained results`, () => { expect(testGraph.executions).toEqual(['c']); }); + it('re-executes a result that a plugin marked unverifiable, and the results built against it', async () => { + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); + const a: Operation = testGraph.operations.get('a')!; + let isMarkingA: boolean = true; + // Like CacheableOperationPlugin when input files of "a" changed while "a" was executing + testGraph.graph.hooks.afterExecuteOperationAsync.tap( + 'TestPlugin', + (record: IOperationExecutionResult) => { + if (isMarkingA && record.operation === a) { + markResultUnverifiable(record); + } + } + ); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + // The same state hashes + isMarkingA = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b']); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + it('does not re-execute retained results of operations that ignore dependency changes', async () => { const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }); const a: Operation = testGraph.operations.get('a')!; From afcb54555b27065a101d3a040f8f00d1711cdac7 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 18:40:05 +0000 Subject: [PATCH 032/265] [rush-lib] A failed pnpm install releases the package-manager lock Swarm integration step 17; original commit 9a58fc2f85 (merge of swarm/r01-t103 at 93eacefb71). Scope: task 103. ch01 GATE OK board 1565 (tree e234c8d86d); o04 CONFIRMED board 1477; r01 board 1440. This is upstream row 16's commit. Commits folded into this step (1): - 93eacefb71 [rush-lib] Release the package manager lock when installing the package manager fails Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...package-manager-lock_2026-09-28-17-58.json | 11 ++ .../logic/installManager/InstallHelpers.ts | 110 +++++++++--------- .../src/logic/test/InstallHelpers.test.ts | 68 ++++++++++- 3 files changed, 136 insertions(+), 53 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r01-release-package-manager-lock_2026-09-28-17-58.json diff --git a/common/changes/@microsoft/rush/swarm-r01-release-package-manager-lock_2026-09-28-17-58.json b/common/changes/@microsoft/rush/swarm-r01-release-package-manager-lock_2026-09-28-17-58.json new file mode 100644 index 0000000000..decf751408 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r01-release-package-manager-lock_2026-09-28-17-58.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Release the package manager lock in `~/.rush` when installing the package manager fails. Before, a long-lived process such as the Rush daemon kept the lock after a failed install, and every later request waited for it forever.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-lib/src/logic/installManager/InstallHelpers.ts b/libraries/rush-lib/src/logic/installManager/InstallHelpers.ts index 8ae6806b00..0ff8126e63 100644 --- a/libraries/rush-lib/src/logic/installManager/InstallHelpers.ts +++ b/libraries/rush-lib/src/logic/installManager/InstallHelpers.ts @@ -494,66 +494,72 @@ export class InstallHelpers { logIfConsoleOutputIsNotRestricted(`Acquired lock for ${packageManagerAndVersion}`); - if (!(await packageManagerMarker.isValidAsync()) || lock.dirtyWhenAcquired) { - logIfConsoleOutputIsNotRestricted( - Colorize.bold(`Installing ${packageManager} version ${packageManagerVersion}\n`) - ); - - // note that this will remove the last-install flag from the directory - await Utilities.installPackageInDirectoryAsync({ - directory: packageManagerToolFolder, - packageName: packageManager, - version: rushConfiguration.packageManagerToolVersion, - tempPackageTitle: `${packageManager}-local-install`, - maxInstallAttempts: maxInstallAttempts, - // This is using a local configuration to install a package in a shared global location. - // Generally that's a bad practice, but in this case if we can successfully install - // the package at all, we can reasonably assume it's good for all the repositories. - // In particular, we'll assume that two different NPM registries cannot have two - // different implementations of the same version of the same package. - // This was needed for: https://github.com/microsoft/rushstack/issues/691 - commonRushConfigFolder: rushConfiguration.commonRushConfigFolder, - // Only filter npm-incompatible properties when the repo uses pnpm or yarn. - // If the repo uses npm, the .npmrc is already configured for npm, so don't filter. - filterNpmIncompatibleProperties: rushConfiguration.packageManager !== 'npm' - }); + try { + if (!(await packageManagerMarker.isValidAsync()) || lock.dirtyWhenAcquired) { + logIfConsoleOutputIsNotRestricted( + Colorize.bold(`Installing ${packageManager} version ${packageManagerVersion}\n`) + ); - logIfConsoleOutputIsNotRestricted( - `Successfully installed ${packageManager} version ${packageManagerVersion}` - ); - } else { - logIfConsoleOutputIsNotRestricted( - `Found ${packageManager} version ${packageManagerVersion} in ${packageManagerToolFolder}` - ); - } + // note that this will remove the last-install flag from the directory + await Utilities.installPackageInDirectoryAsync({ + directory: packageManagerToolFolder, + packageName: packageManager, + version: rushConfiguration.packageManagerToolVersion, + tempPackageTitle: `${packageManager}-local-install`, + maxInstallAttempts: maxInstallAttempts, + // This is using a local configuration to install a package in a shared global location. + // Generally that's a bad practice, but in this case if we can successfully install + // the package at all, we can reasonably assume it's good for all the repositories. + // In particular, we'll assume that two different NPM registries cannot have two + // different implementations of the same version of the same package. + // This was needed for: https://github.com/microsoft/rushstack/issues/691 + commonRushConfigFolder: rushConfiguration.commonRushConfigFolder, + // Only filter npm-incompatible properties when the repo uses pnpm or yarn. + // If the repo uses npm, the .npmrc is already configured for npm, so don't filter. + filterNpmIncompatibleProperties: rushConfiguration.packageManager !== 'npm' + }); + + logIfConsoleOutputIsNotRestricted( + `Successfully installed ${packageManager} version ${packageManagerVersion}` + ); + } else { + logIfConsoleOutputIsNotRestricted( + `Found ${packageManager} version ${packageManagerVersion} in ${packageManagerToolFolder}` + ); + } - await packageManagerMarker.createAsync(); + await packageManagerMarker.createAsync(); - // Example: "C:\MyRepo\common\temp" - FileSystem.ensureFolder(rushConfiguration.commonTempFolder); + // Example: "C:\MyRepo\common\temp" + FileSystem.ensureFolder(rushConfiguration.commonTempFolder); - // Example: "C:\MyRepo\common\temp\pnpm-local" - const localPackageManagerToolFolder: string = `${rushConfiguration.commonTempFolder}/${packageManager}-local`; + // Example: "C:\MyRepo\common\temp\pnpm-local" + const localPackageManagerToolFolder: string = `${rushConfiguration.commonTempFolder}/${packageManager}-local`; - logIfConsoleOutputIsNotRestricted(`\nSymlinking "${localPackageManagerToolFolder}"`); - logIfConsoleOutputIsNotRestricted(` --> "${packageManagerToolFolder}"`); + logIfConsoleOutputIsNotRestricted(`\nSymlinking "${localPackageManagerToolFolder}"`); + logIfConsoleOutputIsNotRestricted(` --> "${packageManagerToolFolder}"`); - // We cannot use FileSystem.exists() to test the existence of a symlink, because it will - // return false for broken symlinks. There is no way to test without catching an exception. - try { - await FileSystem.deleteFolderAsync(localPackageManagerToolFolder); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { - throw error; + // We cannot use FileSystem.exists() to test the existence of a symlink, because it will + // return false for broken symlinks. There is no way to test without catching an exception. + try { + await FileSystem.deleteFolderAsync(localPackageManagerToolFolder); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { + throw error; + } } - } - - await FileSystem.createSymbolicLinkJunctionAsync({ - linkTargetPath: packageManagerToolFolder, - newLinkPath: localPackageManagerToolFolder - }); - lock.release(); + await FileSystem.createSymbolicLinkJunctionAsync({ + linkTargetPath: packageManagerToolFolder, + newLinkPath: localPackageManagerToolFolder + }); + } finally { + // A long-lived process such as the Rush daemon calls this again after a failed install. + // LockFile keeps an in-process record of every held lock, so a lock that is never released + // makes that later call wait forever. A failed install empties the tool folder first, which + // removes the last-install flag, so the next caller installs again. + lock.release(); + } } } diff --git a/libraries/rush-lib/src/logic/test/InstallHelpers.test.ts b/libraries/rush-lib/src/logic/test/InstallHelpers.test.ts index e974122307..f7cc1ba6d4 100644 --- a/libraries/rush-lib/src/logic/test/InstallHelpers.test.ts +++ b/libraries/rush-lib/src/logic/test/InstallHelpers.test.ts @@ -1,12 +1,18 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { type IPackageJson, JsonFile } from '@rushstack/node-core-library'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { FileSystem, type IPackageJson, JsonFile, LockFile } from '@rushstack/node-core-library'; import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import { TestUtilities } from '@rushstack/heft-config-file'; import { InstallHelpers } from '../installManager/InstallHelpers'; import { RushConfiguration } from '../../api/RushConfiguration'; +import type { RushGlobalFolder } from '../../api/RushGlobalFolder'; +import { Utilities } from '../../utilities/Utilities'; import type { PnpmWorkspaceFile } from '../pnpm/PnpmWorkspaceFile'; describe(InstallHelpers.name, () => { @@ -70,6 +76,66 @@ describe(InstallHelpers.name, () => { }); }); + describe(InstallHelpers.ensureLocalPackageManagerAsync.name, () => { + const packageManagerVersion: string = '8.14.0'; + const lockResourceName: string = `pnpm-${packageManagerVersion}`; + let tempFolder: string; + let rushConfiguration: RushConfiguration; + let rushGlobalFolder: RushGlobalFolder; + let installPackageMock: jest.SpyInstance; + + beforeEach(() => { + tempFolder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-ensure-local-package-manager-')); + rushConfiguration = { + packageManager: 'pnpm', + packageManagerToolVersion: packageManagerVersion, + commonRushConfigFolder: `${tempFolder}/repo/common/config/rush`, + commonTempFolder: `${tempFolder}/repo/common/temp` + } as unknown as RushConfiguration; + rushGlobalFolder = { nodeSpecificPath: `${tempFolder}/rush-home` } as unknown as RushGlobalFolder; + installPackageMock = jest + .spyOn(Utilities, 'installPackageInDirectoryAsync') + .mockRejectedValue(new Error('Unexpected package manager install')); + }); + + afterEach(() => { + installPackageMock.mockRestore(); + fs.rmSync(tempFolder, { recursive: true, force: true }); + }); + + it('releases the package manager lock when the install fails, so the same process can retry', async () => { + installPackageMock + .mockRejectedValueOnce(new Error('npm error code E401')) + .mockResolvedValueOnce(undefined); + + await expect( + InstallHelpers.ensureLocalPackageManagerAsync(rushConfiguration, rushGlobalFolder, 1, true) + ).rejects.toThrow('npm error code E401'); + + // A Rush daemon calls this again in the same process. If the failed call still held the lock, + // tryAcquire would return undefined here and the retry below would wait forever. + const lockAfterFailure: LockFile | undefined = LockFile.tryAcquire( + rushGlobalFolder.nodeSpecificPath, + lockResourceName + ); + expect(lockAfterFailure).toBeDefined(); + lockAfterFailure?.release(); + + await InstallHelpers.ensureLocalPackageManagerAsync(rushConfiguration, rushGlobalFolder, 1, true); + + expect(installPackageMock).toHaveBeenCalledTimes(2); + await expect( + FileSystem.getLinkStatisticsAsync(`${rushConfiguration.commonTempFolder}/pnpm-local`) + ).resolves.toBeDefined(); + const lockAfterSuccess: LockFile | undefined = LockFile.tryAcquire( + rushGlobalFolder.nodeSpecificPath, + lockResourceName + ); + expect(lockAfterSuccess).toBeDefined(); + lockAfterSuccess?.release(); + }); + }); + describe(InstallHelpers.generateCommonPackageJsonAsync.name, () => { let mockJsonFileSaveAsync: jest.SpyInstance; let terminal: Terminal; From ab2fb7786b786369e718b9cd1fb86d2a1eca336c Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 19:16:03 +0000 Subject: [PATCH 033/265] [rush-daemon-transport] rush-lib path handoff, startup reservation, lost-connection message and a private runtime folder Swarm integration step 18; original commit 261e4320dd (merge of swarm/r03-t87 at d9366621c8). Scope: tasks 4+5, 68, 86 and 87. ch01 GATE OK board 1675 (tree f85ce62f88). Commits: bb1d272738 (tasks 4+5), 959cbb84ee (task 68), a68e4d96a5, 696dba1d3e (task 86) and d9366621c8 (task 87). Commits folded into this step (5): - bb1d272738 [rush-lib] Hand off a by-name resolvable rush-lib and load built-in cache plugins from deploy outputs - 959cbb84ee [rush-client-core] Resolve a retained startup reservation next to a ready daemon - a68e4d96a5 [rush-client-core] Resolve a remaining startup reservation before a plain daemon stop - 696dba1d3e [rush-client-core] Explain a daemon that exited before delivering a result - d9366621c8 [rush-daemon] Meet one daemon per checkout whatever TMPDIR or XDG_RUNTIME_DIR says (task 87) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 60 ++- apps/rush-cli-client/package.json | 3 + apps/rush-cli-client/src/daemonCommands.ts | 84 +++- .../src/daemonConnectionOptions.ts | 18 +- .../src/test/NativeBuildTestFixture.ts | 2 +- .../test/daemonConnectionSelection.test.ts | 24 ++ .../src/test/launchClient.test.ts | 134 +++++- .../r03-runtime-folder_2026-09-28-17-15.json | 11 + ...ush-lib-path-handoff_2026-09-28-13-10.json | 11 + ...explain-daemon-crash_2026-09-28-15-52.json | 11 + .../r03-runtime-folder_2026-09-28-17-15.json | 11 + ...ush-lib-path-handoff_2026-09-28-13-10.json | 11 + ...-startup-reservation_2026-09-28-14-25.json | 11 + ...resolves-reservation_2026-09-28-15-31.json | 11 + ...explain-daemon-crash_2026-09-28-15-52.json | 11 + .../r03-runtime-folder_2026-09-28-17-15.json | 11 + ...-startup-reservation_2026-09-28-14-25.json | 11 + ...resolves-reservation_2026-09-28-15-31.json | 11 + .../r03-runtime-folder_2026-09-28-17-15.json | 11 + .../r03-runtime-folder_2026-09-28-17-15.json | 11 + .../r03-runtime-folder_2026-09-28-17-15.json | 11 + ...ush-lib-path-handoff_2026-09-28-13-10.json | 11 + .../rush/deploy-rush-daemon-dogfood.json | 35 +- .../config/subspaces/default/pnpm-lock.yaml | 26 +- .../config/subspaces/default/repo-state.json | 2 +- common/reviews/api/rush-client-core.api.md | 19 + .../reviews/api/rush-daemon-protocol.api.md | 3 + .../reviews/api/rush-daemon-transport.api.md | 9 +- docs/rush/dogfooding-rush-daemon.md | 7 +- libraries/rush-client-core/README.md | 38 +- .../rush-client-core/src/DaemonClient.ts | 4 +- .../rush-client-core/src/DaemonClientError.ts | 4 + .../rush-client-core/src/DaemonDisconnect.ts | 185 ++++++++ .../rush-client-core/src/DaemonOwnership.ts | 2 +- .../src/DaemonRuntimeFolder.ts | 60 +++ .../rush-client-core/src/DaemonStartup.ts | 117 ++++- .../src/DaemonStartupReservation.ts | 114 +++++ .../rush-client-core/src/ProcessStartTime.ts | 23 +- .../src/connectOrStartDaemon.ts | 228 ++++++++-- .../src/executeWithDaemonRestart.ts | 30 +- libraries/rush-client-core/src/index.ts | 7 + .../src/test/DaemonDisconnect.test.ts | 121 ++++++ .../src/test/ProcessStartTime.test.ts | 14 +- .../src/test/UnreapedChildProcess.ts | 26 ++ .../src/test/connectOrStartDaemon.test.ts | 400 +++++++++++++++++- .../src/test/fixtures/daemon.ts | 43 +- .../src/DaemonProtocolVersion.ts | 9 +- libraries/rush-daemon-protocol/src/index.ts | 4 +- libraries/rush-daemon-transport/README.md | 13 +- .../src/DaemonFileIdentity.ts | 60 +++ .../src/DaemonListener.ts | 24 +- .../src/DaemonListenerBinding.ts | 18 +- .../src/DaemonListenerLifetime.ts | 23 +- .../src/DaemonLockfile.ts | 14 +- .../src/DaemonOperationGroupReaper.ts | 26 +- .../src/DaemonOrphanReaper.ts | 18 +- .../src/DaemonOwnedEntry.ts | 28 ++ .../rush-daemon-transport/src/DaemonPaths.ts | 56 ++- .../src/DaemonReapOptions.ts | 11 +- .../src/DaemonReclaim.ts | 8 +- .../src/DaemonRuntimeDir.ts | 52 +++ .../src/DaemonRuntimeFolderCheck.ts | 64 +++ .../src/DaemonSocketPublication.ts | 88 ++++ .../src/DaemonTransportError.ts | 4 +- libraries/rush-daemon-transport/src/index.ts | 9 +- .../src/test/DaemonPaths.test.ts | 36 +- .../src/test/FileIdentity.test.ts | 65 +++ .../src/test/ListenerSuccession.test.ts | 71 ++++ .../src/test/RecordOwnership.test.ts | 97 +++++ .../src/test/RuntimeFolder.test.ts | 98 +++++ .../src/test/TestDaemonFixture.ts | 33 +- libraries/rush-daemon/README.md | 4 +- libraries/rush-daemon/package.json | 3 + .../rush-daemon/src/RushLibPathHandoff.ts | 30 ++ .../src/SelectedDaemonBootstrap.ts | 4 +- .../src/WorkspaceRequestLifecycle.ts | 6 +- .../ProductionDaemonRequestResolver.test.ts | 42 ++ .../src/test/RushLibPathHandoff.test.ts | 47 ++ .../src/test/SuccessfulMutationFixture.ts | 2 +- .../VersionSelectedDaemonLauncher.test.ts | 10 +- .../src/api/WorkspaceInputFingerprint.ts | 17 +- .../test/WorkspaceInputFingerprint.test.ts | 12 +- .../src/pluginFramework/PluginManager.ts | 33 +- .../test/PluginManager.test.ts | 129 ++++++ .../src/utilities/RushLibPathHandoff.ts | 106 +++++ .../rush-lib/src/utilities/SetRushLibPath.ts | 22 +- .../utilities/test/RushLibPathHandoff.test.ts | 200 +++++++++ 87 files changed, 3428 insertions(+), 175 deletions(-) create mode 100644 common/changes/@microsoft/rush/r03-runtime-folder_2026-09-28-17-15.json create mode 100644 common/changes/@microsoft/rush/r03-rush-lib-path-handoff_2026-09-28-13-10.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-explain-daemon-crash_2026-09-28-15-52.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-runtime-folder_2026-09-28-17-15.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-rush-lib-path-handoff_2026-09-28-13-10.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-startup-reservation_2026-09-28-14-25.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-stop-resolves-reservation_2026-09-28-15-31.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-explain-daemon-crash_2026-09-28-15-52.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-runtime-folder_2026-09-28-17-15.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-startup-reservation_2026-09-28-14-25.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-stop-resolves-reservation_2026-09-28-15-31.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/r03-runtime-folder_2026-09-28-17-15.json create mode 100644 common/changes/@rushstack/rush-daemon-transport/r03-runtime-folder_2026-09-28-17-15.json create mode 100644 common/changes/@rushstack/rush-daemon/r03-runtime-folder_2026-09-28-17-15.json create mode 100644 common/changes/@rushstack/rush-daemon/r03-rush-lib-path-handoff_2026-09-28-13-10.json create mode 100644 libraries/rush-client-core/src/DaemonDisconnect.ts create mode 100644 libraries/rush-client-core/src/DaemonRuntimeFolder.ts create mode 100644 libraries/rush-client-core/src/DaemonStartupReservation.ts create mode 100644 libraries/rush-client-core/src/test/DaemonDisconnect.test.ts create mode 100644 libraries/rush-client-core/src/test/UnreapedChildProcess.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonFileIdentity.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonOwnedEntry.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonRuntimeFolderCheck.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonSocketPublication.ts create mode 100644 libraries/rush-daemon-transport/src/test/FileIdentity.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/RecordOwnership.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/RuntimeFolder.test.ts create mode 100644 libraries/rush-daemon/src/RushLibPathHandoff.ts create mode 100644 libraries/rush-daemon/src/test/RushLibPathHandoff.test.ts create mode 100644 libraries/rush-lib/src/pluginFramework/test/PluginManager.test.ts create mode 100644 libraries/rush-lib/src/utilities/RushLibPathHandoff.ts create mode 100644 libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 1fb55ae3fb..96c9d681da 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -176,9 +176,10 @@ coalesced iteration, not while idle; native commands and `--no-daemon` can run after a completed request without stopping the daemon. Native workspace dispatch copies the request envelope and normalizes only the -engine-owned `_RUSH_LIB_PATH` to this daemon's real engine. Foreign client SDK -paths therefore neither select the wrong SDK nor cause a false restart. All other -environment inputs remain unchanged and participate in normal lifecycle checks. +engine-owned `_RUSH_LIB_PATH` to this daemon's own engine, keeping the spelling +that the engine chose when it loaded. Foreign client SDK paths therefore neither +select the wrong SDK nor cause a false restart. All other environment inputs +remain unchanged and participate in normal lifecycle checks. Protocol 0.10 permits a bounded retry only when a pre-execution command result explicitly carries `retryAfterRestart: true`. `executeWithDaemonRestartAsync` @@ -190,6 +191,15 @@ only an unstarted request can receive the typed retry authorization. Accepted queued requests drain their typed restart results before the old connection closes. +When the connection is lost before a command's result, the command fails with exit code 1 +and is not retried. The diagnostic keeps "Daemon disconnected before delivering a result; the +command was not retried." and says what happened to rushd. If its process exited (a crash, an +out-of-memory kill or a signal), it names the PID, points to `rush-client daemon logs` and, if +the daemon exits again, to `--no-daemon` (`rushx-client --no-daemon` for Rushx), and quotes on a +second line the fatal error that the launcher log recorded after the command was sent. If +rushd still runs, it says that only the connection closed. Ctrl+C and an orderly `daemon stop` +or `daemon restart` still end a command as cancelled (exit code 130). + Piped input uses protocol 0.7's negotiated stdin admission and EOF. The client does not read input until the command attaches an input destination, and sends bounded chunks only as the daemon grants write credits. EOF follows all preceding writes; @@ -268,13 +278,44 @@ It does not replace a peer lacking safe shutdown support. Foreign package instal is a client preparation step; host self-restart selects only bundled or already cached compatible installations, never installing while the old workspace is being cleaned up. +Every client of a checkout finds its daemon in one per-user runtime folder: on Linux and +macOS, `/tmp/rushd-/`, whatever `TMPDIR` or `XDG_RUNTIME_DIR` a shell, job, service or +sandbox sets. It holds the socket (`.sock`), the ownership record +(`.pid.json`) and the launcher log. To move it, set `RUSHD_RUNTIME_DIR` to an absolute +path for every client of that checkout; the folder becomes `$RUSHD_RUNTIME_DIR/rushd-/`, +and its file system must support hard links. A client passes the folder to the daemon it starts. +Windows uses the named pipe `\\.\pipe\rushd-` and is unchanged. +The client refuses a runtime folder that is a symbolic link, is not a directory or belongs to +another user: commands run in-process with that reason, and `daemon` commands exit 1. Remove +the folder or set `RUSHD_RUNTIME_DIR`. A folder that others can open is made owner-only (`0700`). +`TMPDIR`, `TMP`, `TEMP`, `XDG_RUNTIME_DIR` and `RUSHD_RUNTIME_DIR` never select a different +daemon; each operation receives the requesting client's values. +Clients and daemons before protocol 0.12 used `$XDG_RUNTIME_DIR/rushd-/` or the +temporary folder instead. A daemon started there stays there, where current clients do not +look, until it idles out or is stopped with that older client (`rush-client daemon stop`). +An older client that starts a current engine while `XDG_RUNTIME_DIR` or `TMPDIR` is set does +not find it and runs in-process, so upgrade `rush-cli-client` with the engine. + `rush-client daemon status` only connects and checks hello/pong. It never starts a process, reclaims files, or treats a PID file as evidence of readiness. Both commands print one JSON object with `state: "ready"`, `socketPath`, and the actual pong fields (`uptimeMs`, available versions, optional `pid` and `residentMemoryBytes`, and an optional `workspace` snapshot). Exit code 0 means protocol readiness, not build support. An unreachable/incompatible endpoint, invalid -arguments, or startup failure returns exit code 1 with a diagnostic. +arguments, or startup failure returns exit code 1 with a diagnostic. When the endpoint +refuses connections and its ownership record (`.pid.json`) names a PID that no longer +exists, the diagnostic adds that rushd exited without shutting down (an orderly shutdown +removes the record) and that `daemon logs` may show why. + +A startup reservation (`.pid.json.starting`) refuses another daemon launch until +the daemon it reserved becomes ready. Status reports one that remains as +`startupReservation` with its `path`, the startup helper's `helperPid` when recorded, and +`helperState`: `running` (the helper still waits for readiness), `exited` (nothing else will +release it), or `unknown` (written by an older client). Status never removes it. Next to a +ready daemon, the next command that uses, stops or restarts that daemon removes it; when status +cannot connect, its diagnostic explains the reservation. After an `exited` helper, every automatic start is +refused at once unless that daemon still becomes ready: check `daemon logs`, and if the daemon +failed to start, run `daemon stop --force`. The optional workspace snapshot reports the provider generation/token, graph existence, and available warm accounting without initializing a graph. Missing fields are unknown, @@ -302,7 +343,11 @@ followed by EOF. It reports `state: "shutdownAccepted"` with exit code 0; this does not assert successful workspace disposal. Stop is idempotent: when nothing listens at the endpoint it reports `state: "notRunning"` with exit code 0. An unsupported protocol, missing acknowledgement, handshake failure, or timeout -returns exit code 1. It does not auto-start anything. +returns exit code 1. It does not auto-start anything. Before shutdown, it removes a startup +reservation that remains next to that daemon, as restart does, so that the reservation cannot +refuse the next start once the daemon is gone. It does so only for the live owner in the +ownership record, under the start mutex (waiting up to 15 seconds for it); a reservation that +it cannot resolve stays in place and is reported as `startupReservation`. `rush-client daemon stop --force` stops a running daemon the same way, then waits (up to 15 seconds) for it to release its listener and ownership record and removes @@ -319,7 +364,10 @@ that every fail-closed startup message points to. `rush-client daemon restart` first verifies that the selected Rush version has a launcher and captures the original lock's PID/start timestamp, checking that it matches pong's positive PID and the selected endpoint, then performs acknowledged -shutdown. It waits for original ownership release or a demonstrably dead owner +shutdown. Before shutdown, it removes a startup reservation that remains next to that +daemon (under the start mutex), so that the reservation cannot refuse the successor; if +another client holds the mutex for 15 seconds, restart fails without stopping the daemon. +It waits for original ownership release or a demonstrably dead owner before calling the existing locked starter. A live owner fails closed at the startup deadline; no PID is killed and no live ownership record is deleted. A newly diff --git a/apps/rush-cli-client/package.json b/apps/rush-cli-client/package.json index eb3f9a1609..3633851af4 100644 --- a/apps/rush-cli-client/package.json +++ b/apps/rush-cli-client/package.json @@ -36,6 +36,9 @@ }, "devDependencies": { "@rushstack/heft": "workspace:*", + "@rushstack/rush-amazon-s3-build-cache-plugin": "workspace:*", + "@rushstack/rush-azure-storage-build-cache-plugin": "workspace:*", + "@rushstack/rush-http-build-cache-plugin": "workspace:*", "eslint": "~9.37.0", "local-node-rig": "workspace:*" } diff --git a/apps/rush-cli-client/src/daemonCommands.ts b/apps/rush-cli-client/src/daemonCommands.ts index fd19088343..6568b5e298 100644 --- a/apps/rush-cli-client/src/daemonCommands.ts +++ b/apps/rush-cli-client/src/daemonCommands.ts @@ -6,14 +6,20 @@ import * as path from 'node:path'; import { DaemonClient, connectOrStartDaemonAsync, + inspectDaemonStartupReservation, requestDaemonShutdownAsync, resetDaemonArtifactsAsync, - type IConnectOrStartDaemonOptions + resolveDaemonStartupReservationAsync, + type IConnectOrStartDaemonOptions, + type IDaemonStartupReservationInfo } from '@rushstack/rush-client-core'; import { DaemonTransportError, DaemonTransportErrorCode, - type IDaemonLockfile + isDaemonProcessAlive, + readDaemonLockfile, + type IDaemonLockfile, + type IDaemonPaths } from '@rushstack/rush-daemon-transport'; import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; @@ -99,7 +105,8 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): await writeStatusAsync({ state: 'ready', socketPath: connectionOptions.paths.socketPath, - ...(await started.status) + ...(await started.status), + ...getStartupReservationStatus(connectionOptions.paths) }); } finally { await started.closeAsync(); @@ -113,12 +120,18 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): await writeStatusAsync({ state: removedPaths.length > 0 ? 'reset' : 'notRunning', socketPath: connectionOptions.paths.socketPath, - ...(options.argv[1] === '--force' ? { removedPaths } : {}) + ...(options.argv[1] === '--force' ? { removedPaths } : {}), + ...getStartupReservationStatus(connectionOptions.paths) }); return; } try { if (command === 'stop') { + if (options.argv[1] !== '--force') { + // Once the daemon is gone, nothing proves a remaining reservation stale, so it would refuse every later + // automatic start. A reservation that cannot be resolved is still reported; --force removes it below. + await resolveDaemonStartupReservationAsync(client, connectionOptions.paths).catch(() => false); + } const { activeRequests } = await client.shutdownAsync(); if (activeRequests) { await writeStreamAsync( @@ -147,7 +160,8 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): await writeStatusAsync({ state: 'shutdownAccepted', socketPath: connectionOptions.paths.socketPath, - ...cancelled + ...cancelled, + ...getStartupReservationStatus(connectionOptions.paths) }); return; } @@ -157,7 +171,8 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): await writeStatusAsync({ state: 'ready', socketPath: connectionOptions.paths.socketPath, - ...(await readyClient.status) + ...(await readyClient.status), + ...getStartupReservationStatus(connectionOptions.paths) }); } finally { if (readyClient !== client) await readyClient.closeAsync(); @@ -182,8 +197,63 @@ async function connectExistingAsync( ) { return undefined; } - throw error; + throw explainStartupReservation(explainExitedDaemon(error, options.paths), options.paths); + } +} + +/** + * Explains a refused connection whose ownership record names a daemon that no longer runs. An orderly + * shutdown removes the record, so that daemon exited without shutting down, for example after a crash. + */ +function explainExitedDaemon(error: unknown, paths: IDaemonPaths): unknown { + if (!(error instanceof DaemonTransportError) || error.code !== DaemonTransportErrorCode.connectionRefused) { + return error; + } + const owner: IDaemonLockfile | undefined = readDaemonLockfile(paths.lockfilePath); + if ( + owner?.socketPath !== paths.socketPath || + !Number.isSafeInteger(owner.pid) || + owner.pid <= 0 || + isDaemonProcessAlive(owner.pid) + ) { + return error; + } + return new DaemonTransportError( + error.code, + `${error.message} rushd (PID ${owner.pid}) exited without shutting down; "rush-client daemon logs" may show why.` + ); +} + +/** + * Reports a startup reservation, which refuses another daemon launch until it is resolved. Clients resolve it + * once the daemon it reserved is ready, so the next command that uses, stops or restarts a ready daemon resolves + * a remaining one; status only reports it. + */ +function getStartupReservationStatus(paths: IDaemonPaths): { + startupReservation?: IDaemonStartupReservationInfo; +} { + const startupReservation: IDaemonStartupReservationInfo | undefined = + inspectDaemonStartupReservation(paths); + return startupReservation ? { startupReservation } : {}; +} + +/** Explains that a remaining startup reservation refuses another daemon launch, and what can resolve it. */ +function explainStartupReservation(error: unknown, paths: IDaemonPaths): unknown { + const reservation: IDaemonStartupReservationInfo | undefined = inspectDaemonStartupReservation(paths); + if (!reservation || !(error instanceof Error)) return error; + const helper: string = `its startup helper (PID ${reservation.helperPid})`; + let explanation: string; + switch (reservation.helperState) { + case 'running': + explanation = `A daemon is starting: ${helper} is still waiting for it to become ready; retry shortly.`; + break; + case 'exited': + explanation = `The startup reservation at ${reservation.path} remains, but ${helper} exited before the daemon became ready, so the reservation refuses every automatic start unless that daemon still becomes ready. Check "rush-client daemon logs"; if the daemon failed to start, run "rush-client daemon stop --force" to remove it.`; + break; + default: + explanation = `The startup reservation at ${reservation.path} refuses another daemon launch. Check "rush-client daemon logs"; if no daemon is starting, run "rush-client daemon stop --force" to remove it.`; } + return new Error(`${error.message} ${explanation}`, { cause: error }); } async function restartDaemonAsync( diff --git a/apps/rush-cli-client/src/daemonConnectionOptions.ts b/apps/rush-cli-client/src/daemonConnectionOptions.ts index bb87818b85..26f7158a9d 100644 --- a/apps/rush-cli-client/src/daemonConnectionOptions.ts +++ b/apps/rush-cli-client/src/daemonConnectionOptions.ts @@ -4,8 +4,15 @@ import * as fs from 'node:fs'; import { JsonFile } from '@rushstack/node-core-library'; -import type { IConnectOrStartDaemonOptions } from '@rushstack/rush-client-core'; -import { computeDaemonWorkspaceKey, resolveDaemonPathsFromProcess } from '@rushstack/rush-daemon-transport'; +import { + assertDaemonRuntimeFolderIsPrivate, + type IConnectOrStartDaemonOptions +} from '@rushstack/rush-client-core'; +import { + computeDaemonWorkspaceKey, + resolveDaemonPathsFromProcess, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; import { readDaemonInstallationMetadata } from '@rushstack/rush-daemon/lib/DaemonInstallation'; import type * as VersionSelectedDaemonLauncherModule from '@rushstack/rush-daemon/lib/VersionSelectedDaemonLauncher'; @@ -26,8 +33,13 @@ export function getDaemonConnectionOptions( 'The synchronous launcher only supports its installed engine; use asynchronous version selection.' ); } + const paths: IDaemonPaths = resolveDaemonPathsFromProcess( + computeDaemonWorkspaceKey({ canonicalRepoRoot, rushVersion }) + ); + // Every daemon command trusts files in this folder: the socket, the lockfile, the log and the reservation. + assertDaemonRuntimeFolderIsPrivate(paths); return { - paths: resolveDaemonPathsFromProcess(computeDaemonWorkspaceKey({ canonicalRepoRoot, rushVersion })), + paths, expectedDaemonVersion: daemonPackage.version, startCommand: autoStart ? loadVersionSelectedDaemonLauncher().getSelectedDaemonStartCommand(daemonPackagePath, { diff --git a/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts b/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts index 07eb1bd763..ace2ae4152 100644 --- a/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts +++ b/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts @@ -47,7 +47,7 @@ export function createNativeBuildTestFixture(): INativeBuildTestFixture { CI: 'false', TF_BUILD: 'false', GITHUB_ACTIONS: 'false', - XDG_RUNTIME_DIR: folder + RUSHD_RUNTIME_DIR: folder }; const invocationClosures: Promise[] = []; const callbacks: Promise[] = []; diff --git a/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts b/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts index 98daf2fe4d..52e503a603 100644 --- a/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts +++ b/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts @@ -6,6 +6,7 @@ import * as os from 'node:os'; import * as path from 'node:path'; import { Rush } from '@microsoft/rush-lib'; +import { DaemonClientError } from '@rushstack/rush-client-core'; import { computeDaemonWorkspaceKey, resolveDaemonPathsFromProcess } from '@rushstack/rush-daemon-transport'; import { getDaemonConnectionOptions, getDaemonConnectionOptionsAsync } from '../daemonConnectionOptions'; @@ -48,6 +49,29 @@ describe('version-selected daemon connection options', () => { ); }); + (process.platform === 'win32' ? it.skip : it)( + 'refuses an unsafe runtime folder before any daemon command uses it', + () => { + const base: string = path.join(repoRoot, 'runtime'); + fs.mkdirSync(base); + fs.symlinkSync(repoRoot, path.join(base, `rushd-${process.getuid?.()}`)); + const previous: string | undefined = process.env.RUSHD_RUNTIME_DIR; + process.env.RUSHD_RUNTIME_DIR = base; + let error: unknown; + try { + getDaemonConnectionOptions(repoRoot, Rush.version, process.env, false); + } catch (thrown) { + error = thrown; + } finally { + if (previous === undefined) delete process.env.RUSHD_RUNTIME_DIR; + else process.env.RUSHD_RUNTIME_DIR = previous; + } + expect(error).toBeInstanceOf(DaemonClientError); + expect(error).toMatchObject({ code: 'startupFailed' }); + expect((error as Error).message).toContain('is unsafe: it is a symbolic link'); + } + ); + it('does not select or install a launcher for a connect-only invocation', async () => { const options = await getDaemonConnectionOptionsAsync(repoRoot, '5.178.1', {}, false); expect(options.startCommand).toBeUndefined(); diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index 39b091f31f..e8f70172b5 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -18,7 +18,7 @@ import { waitForTestProcessExitAsync } from '@rushstack/rush-daemon/lib/test/TestProcessExit'; import { captureTestDaemonListenerAsync } from '@rushstack/rush-daemon/lib/test/TestDaemonListener'; -import { readDaemonLockfile } from '@rushstack/rush-daemon-transport'; +import { readDaemonLockfile, type IDaemonLockfile } from '@rushstack/rush-daemon-transport'; import { getDaemonConnectionOptions } from '../daemonConnectionOptions'; import { @@ -306,6 +306,138 @@ describe('standalone rushx fallback', () => { } }, 30000); + it('stop resolves a startup reservation next to a ready daemon so that a later start launches', async () => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + const reservation: string = `${paths.lockfilePath}.starting`; + try { + const started: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'start']); + expect(started.code).toBe(0); + const { pid }: { pid: number } = JSON.parse(started.stdout); + fs.writeFileSync(reservation, 'abandoned'); + const stopped: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop']); + expect(stopped).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(stopped.stdout)).toEqual({ + state: 'shutdownAccepted', + socketPath: paths.socketPath, + cancelledRequests: 0 + }); + expect(fs.existsSync(reservation)).toBe(false); + const deadline: number = Date.now() + 7000; + while (fs.existsSync(paths.lockfilePath) && Date.now() < deadline) await delayAsync(50); + const restarted: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'start']); + expect(restarted).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(restarted.stdout).pid).not.toBe(pid); + expect((await invokeAsync(true, false, false, ['daemon', 'stop'])).code).toBe(0); + } finally { + const deadline: number = Date.now() + 7000; + while (fs.existsSync(paths.lockfilePath) && Date.now() < deadline) await delayAsync(50); + } + }, 30000); + + it('reports a startup reservation next to a ready daemon, and restart resolves it', async () => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + const reservation: string = `${paths.lockfilePath}.starting`; + try { + const started: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'start']); + expect(started.code).toBe(0); + const { pid }: { pid: number } = JSON.parse(started.stdout); + fs.writeFileSync(reservation, 'abandoned'); + const status: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'status']); + expect(status).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(status.stdout)).toMatchObject({ + state: 'ready', + pid, + startupReservation: { path: reservation, helperState: 'unknown' } + }); + // Status only reports the reservation. + expect(fs.readFileSync(reservation, 'utf8')).toBe('abandoned'); + const restarted: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'restart']); + expect(restarted).toMatchObject({ code: 0, stderr: '' }); + const successor: Record = JSON.parse(restarted.stdout); + expect(successor).toMatchObject({ state: 'ready', pid: expect.any(Number) }); + expect(successor.pid).not.toBe(pid); + expect(successor).not.toHaveProperty('startupReservation'); + expect(fs.existsSync(reservation)).toBe(false); + expect((await invokeAsync(true, false, false, ['daemon', 'stop'])).code).toBe(0); + } finally { + const deadline: number = Date.now() + 7000; + while (fs.existsSync(paths.lockfilePath) && Date.now() < deadline) await delayAsync(50); + } + }, 30000); + + it.each(['exited', 'running'])( + 'status explains a startup reservation (helper %s) while no daemon is ready', + async (helperState) => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + const reservation: string = `${paths.lockfilePath}.starting`; + fs.mkdirSync(path.dirname(paths.lockfilePath), { recursive: true, mode: 0o700 }); + let helperPid: number = process.pid; + if (helperState === 'exited') { + const exited: ChildProcess = spawn(process.execPath, ['-e', ''], { stdio: 'ignore' }); + await once(exited, 'close'); + helperPid = exited.pid!; + } + fs.writeFileSync( + reservation, + JSON.stringify({ token: 'fixture', helperPid, helperStartedAt: new Date().toISOString() }) + ); + const status: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'status']); + expect(status).toMatchObject({ code: 1, stdout: '' }); + expect(status.stderr).toContain('Could not connect to daemon'); + expect(status.stderr).toContain( + helperState === 'exited' + ? `its startup helper (PID ${helperPid}) exited before the daemon became ready, so the reservation refuses every automatic start unless that daemon still becomes ready. Check "rush-client daemon logs"; if the daemon failed to start, run "rush-client daemon stop --force" to remove it.` + : `A daemon is starting: its startup helper (PID ${helperPid}) is still waiting for it to become ready; retry shortly.` + ); + const stopped: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop']); + expect(stopped).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(stopped.stdout)).toEqual({ + state: 'notRunning', + socketPath: paths.socketPath, + startupReservation: { path: reservation, helperPid, helperState } + }); + const reset: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop', '--force']); + expect(reset).toMatchObject({ code: 0, stderr: '' }); + expect(JSON.parse(reset.stdout)).toEqual({ + state: 'reset', + socketPath: paths.socketPath, + removedPaths: [reservation] + }); + } + ); + + it('status names a recorded daemon that exited without shutting down', async () => { + const { paths } = getDaemonConnectionOptions(folder, Rush.version, {}, false); + fs.mkdirSync(path.dirname(paths.lockfilePath), { recursive: true, mode: 0o700 }); + const exited: ChildProcess = spawn(process.execPath, ['-e', ''], { stdio: 'ignore' }); + await once(exited, 'close'); + const record: IDaemonLockfile = { + pid: exited.pid!, + protocolVersion: { major: 0, minor: 11 }, + startedAt: new Date().toISOString(), + socketPath: paths.socketPath + }; + try { + fs.writeFileSync(paths.lockfilePath, JSON.stringify(record)); + const status: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'status']); + expect(status).toMatchObject({ code: 1, stdout: '' }); + expect(status.stderr).toContain( + `Could not connect to daemon at ${paths.socketPath}. rushd (PID ${record.pid}) exited without shutting down; "rush-client daemon logs" may show why.` + ); + // A record for another endpoint says nothing about the daemon at this one. + fs.writeFileSync( + paths.lockfilePath, + JSON.stringify({ ...record, socketPath: `${paths.socketPath}.x` }) + ); + const other: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'status']); + expect(other).toMatchObject({ code: 1, stdout: '' }); + expect(other.stderr).toContain('Could not connect to daemon'); + expect(other.stderr).not.toContain('exited without shutting down'); + } finally { + fs.rmSync(paths.lockfilePath, { force: true }); + } + }); + (process.platform === 'win32' ? it.skip : it)( 'stop --force removes stale artifacts left by a killed daemon', async () => { diff --git a/common/changes/@microsoft/rush/r03-runtime-folder_2026-09-28-17-15.json b/common/changes/@microsoft/rush/r03-runtime-folder_2026-09-28-17-15.json new file mode 100644 index 0000000000..8112f0cd39 --- /dev/null +++ b/common/changes/@microsoft/rush/r03-runtime-folder_2026-09-28-17-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Temporary and runtime folder variables (`TMPDIR`, `TMP`, `TEMP`, `XDG_RUNTIME_DIR` and `RUSHD_RUNTIME_DIR`) no longer select a different Rush daemon, and a daemon no longer keeps the `TMPDIR` or `XDG_RUNTIME_DIR` of the client that started it.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@microsoft/rush/r03-rush-lib-path-handoff_2026-09-28-13-10.json b/common/changes/@microsoft/rush/r03-rush-lib-path-handoff_2026-09-28-13-10.json new file mode 100644 index 0000000000..7fb30b2a6a --- /dev/null +++ b/common/changes/@microsoft/rush/r03-rush-lib-path-handoff_2026-09-28-13-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "When a source-built rush-lib cannot be found by name from its real path (for example in a `rush deploy` output), spell `_RUSH_LIB_PATH` through the host's `node_modules/@microsoft/rush-lib` link, and load the built-in build cache plugins that the host installs next to that link.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-explain-daemon-crash_2026-09-28-15-52.json b/common/changes/@rushstack/rush-cli-client/r03-explain-daemon-crash_2026-09-28-15-52.json new file mode 100644 index 0000000000..99273dcc38 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-explain-daemon-crash_2026-09-28-15-52.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "After a daemon crash, commands and `rush-client daemon status` say that rushd exited and point to `rush-client daemon logs` and `--no-daemon`, instead of only reporting a lost connection.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-runtime-folder_2026-09-28-17-15.json b/common/changes/@rushstack/rush-cli-client/r03-runtime-folder_2026-09-28-17-15.json new file mode 100644 index 0000000000..04b1a1ca29 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-runtime-folder_2026-09-28-17-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Use one daemon per checkout whatever `TMPDIR` or `XDG_RUNTIME_DIR` a shell, job or service sets; `RUSHD_RUNTIME_DIR` moves its folder from `/tmp/rushd-`. A runtime folder that is a symbolic link or belongs to another user is refused.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-rush-lib-path-handoff_2026-09-28-13-10.json b/common/changes/@rushstack/rush-cli-client/r03-rush-lib-path-handoff_2026-09-28-13-10.json new file mode 100644 index 0000000000..690b855dd3 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-rush-lib-path-handoff_2026-09-28-13-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-startup-reservation_2026-09-28-14-25.json b/common/changes/@rushstack/rush-cli-client/r03-startup-reservation_2026-09-28-14-25.json new file mode 100644 index 0000000000..b79ef158b2 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-startup-reservation_2026-09-28-14-25.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Report a remaining startup reservation and its helper's state as `startupReservation` in `rush-client daemon status` and `daemon stop`, explain it when status cannot connect, and let `daemon restart` resolve it before it stops the old daemon.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-stop-resolves-reservation_2026-09-28-15-31.json b/common/changes/@rushstack/rush-cli-client/r03-stop-resolves-reservation_2026-09-28-15-31.json new file mode 100644 index 0000000000..41881f9d8d --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-stop-resolves-reservation_2026-09-28-15-31.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Let `rush-client daemon stop` resolve a startup reservation that remains next to the ready daemon before it stops it, as `daemon restart` does, so that the next start is not refused.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-explain-daemon-crash_2026-09-28-15-52.json b/common/changes/@rushstack/rush-client-core/r03-explain-daemon-crash_2026-09-28-15-52.json new file mode 100644 index 0000000000..89e95ab426 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-explain-daemon-crash_2026-09-28-15-52.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "When the connection is lost before a result, `executeWithDaemonRestartAsync()` now says whether the daemon process exited, quotes the fatal error that its launcher log recorded, and names `rush-client daemon logs` and `--no-daemon`; the request is still never retried.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-runtime-folder_2026-09-28-17-15.json b/common/changes/@rushstack/rush-client-core/r03-runtime-folder_2026-09-28-17-15.json new file mode 100644 index 0000000000..c4ecee9621 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-runtime-folder_2026-09-28-17-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Pass the runtime folder to a daemon being started as `RUSHD_RUNTIME_DIR`, and report an unsafe runtime folder as `startupFailed` (`assertDaemonRuntimeFolderIsPrivate`).", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-startup-reservation_2026-09-28-14-25.json b/common/changes/@rushstack/rush-client-core/r03-startup-reservation_2026-09-28-14-25.json new file mode 100644 index 0000000000..6521eb05aa --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-startup-reservation_2026-09-28-14-25.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Record the startup helper's PID in the daemon startup reservation, and resolve a reservation that remains next to a ready daemon whose ownership is attested, so that a first start slower than the helper's wait no longer blocks every later start. Refuse another launch at once when the recorded helper has exited, resolve the reservation before an explicit restart's shutdown, and add `inspectDaemonStartupReservation()`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-stop-resolves-reservation_2026-09-28-15-31.json b/common/changes/@rushstack/rush-client-core/r03-stop-resolves-reservation_2026-09-28-15-31.json new file mode 100644 index 0000000000..7bda75c247 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-stop-resolves-reservation_2026-09-28-15-31.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Add `resolveDaemonStartupReservationAsync()`, which resolves a startup reservation that remains next to a connected, attested daemon before a caller stops that daemon.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/r03-runtime-folder_2026-09-28-17-15.json b/common/changes/@rushstack/rush-daemon-protocol/r03-runtime-folder_2026-09-28-17-15.json new file mode 100644 index 0000000000..c3d7c9edfb --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/r03-runtime-folder_2026-09-28-17-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Advertise protocol minor 12 (`DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR`): clients and daemons meet in `/tmp/rushd-` (or `$RUSHD_RUNTIME_DIR/rushd-`) instead of a folder chosen by `XDG_RUNTIME_DIR` or `TMPDIR`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-transport/r03-runtime-folder_2026-09-28-17-15.json b/common/changes/@rushstack/rush-daemon-transport/r03-runtime-folder_2026-09-28-17-15.json new file mode 100644 index 0000000000..0419d3749d --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-transport/r03-runtime-folder_2026-09-28-17-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-transport", + "comment": "Clients and daemons find each other in `/tmp/rushd-` (or `$RUSHD_RUNTIME_DIR/rushd-`) whatever `TMPDIR` or `XDG_RUNTIME_DIR` says. A listener publishes its socket with a hard link and removes its socket and ownership record on close only while they are still its own. The runtime folder must be an owned directory, not a symbolic link, and reapers skip records they do not own.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-transport", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r03-runtime-folder_2026-09-28-17-15.json b/common/changes/@rushstack/rush-daemon/r03-runtime-folder_2026-09-28-17-15.json new file mode 100644 index 0000000000..f061cf5f73 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r03-runtime-folder_2026-09-28-17-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Listen in `/tmp/rushd-` (or `$RUSHD_RUNTIME_DIR/rushd-`), where current clients look, instead of in `$XDG_RUNTIME_DIR` or the temporary folder.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r03-rush-lib-path-handoff_2026-09-28-13-10.json b/common/changes/@rushstack/rush-daemon/r03-rush-lib-path-handoff_2026-09-28-13-10.json new file mode 100644 index 0000000000..66ec1f2f82 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r03-rush-lib-path-handoff_2026-09-28-13-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Keep the selected engine's own spelling of `_RUSH_LIB_PATH` when it names the same rush-lib, so plugins in a deployed daemon can resolve rush-lib by name.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/config/rush/deploy-rush-daemon-dogfood.json b/common/config/rush/deploy-rush-daemon-dogfood.json index c83062bf53..e30a102fe0 100644 --- a/common/config/rush/deploy-rush-daemon-dogfood.json +++ b/common/config/rush/deploy-rush-daemon-dogfood.json @@ -11,5 +11,38 @@ { "$schema": "https://developer.microsoft.com/json-schemas/rush/v5/deploy-scenario.schema.json", - "deploymentProjectNames": ["@rushstack/rush-cli-client"] + "deploymentProjectNames": ["@rushstack/rush-cli-client"], + + /** + * A published @microsoft/rush-lib depends on its built-in build cache plugins. The source-built copy lists them + * in "publishOnlyDependencies" instead, and loads them from next to the rush-lib link of the host script that + * loaded it: @microsoft/rush (the in-process fallback), rush-client (when it loads rush-lib before handing off + * to @microsoft/rush) and rush-daemon. Each of these hosts declares the plugins as devDependencies. + */ + "projectSettings": [ + { + "projectName": "@microsoft/rush", + "additionalDependenciesToInclude": [ + "@rushstack/rush-amazon-s3-build-cache-plugin", + "@rushstack/rush-azure-storage-build-cache-plugin", + "@rushstack/rush-http-build-cache-plugin" + ] + }, + { + "projectName": "@rushstack/rush-cli-client", + "additionalDependenciesToInclude": [ + "@rushstack/rush-amazon-s3-build-cache-plugin", + "@rushstack/rush-azure-storage-build-cache-plugin", + "@rushstack/rush-http-build-cache-plugin" + ] + }, + { + "projectName": "@rushstack/rush-daemon", + "additionalDependenciesToInclude": [ + "@rushstack/rush-amazon-s3-build-cache-plugin", + "@rushstack/rush-azure-storage-build-cache-plugin", + "@rushstack/rush-http-build-cache-plugin" + ] + } + ] } diff --git a/common/config/subspaces/default/pnpm-lock.yaml b/common/config/subspaces/default/pnpm-lock.yaml index 2d4e19dd39..9254358cf5 100644 --- a/common/config/subspaces/default/pnpm-lock.yaml +++ b/common/config/subspaces/default/pnpm-lock.yaml @@ -451,6 +451,15 @@ importers: '@rushstack/heft': specifier: workspace:* version: link:../heft + '@rushstack/rush-amazon-s3-build-cache-plugin': + specifier: workspace:* + version: link:../../rush-plugins/rush-amazon-s3-build-cache-plugin + '@rushstack/rush-azure-storage-build-cache-plugin': + specifier: workspace:* + version: link:../../rush-plugins/rush-azure-storage-build-cache-plugin + '@rushstack/rush-http-build-cache-plugin': + specifier: workspace:* + version: link:../../rush-plugins/rush-http-build-cache-plugin eslint: specifier: ~9.37.0 version: 9.37.0 @@ -4197,6 +4206,15 @@ importers: '@rushstack/heft': specifier: workspace:* version: link:../../apps/heft + '@rushstack/rush-amazon-s3-build-cache-plugin': + specifier: workspace:* + version: link:../../rush-plugins/rush-amazon-s3-build-cache-plugin + '@rushstack/rush-azure-storage-build-cache-plugin': + specifier: workspace:* + version: link:../../rush-plugins/rush-azure-storage-build-cache-plugin + '@rushstack/rush-http-build-cache-plugin': + specifier: workspace:* + version: link:../../rush-plugins/rush-http-build-cache-plugin eslint: specifier: ~9.37.0 version: 9.37.0 @@ -4817,7 +4835,7 @@ importers: version: 1.3.1(@types/node@20.17.19) '@rushstack/heft-node-rig': specifier: 2.11.51 - version: 2.11.51(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19)(babel-plugin-macros@3.1.0)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0) + version: 2.11.51(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0) '@types/jest': specifier: 30.0.0 version: 30.0.0 @@ -24938,7 +24956,7 @@ snapshots: transitivePeerDependencies: - '@types/node' - '@rushstack/heft-jest-plugin@2.0.18(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/jest@30.0.0)(@types/node@20.17.19)(babel-plugin-macros@3.1.0)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0)(jest-environment-node@30.3.0)': + '@rushstack/heft-jest-plugin@2.0.18(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/jest@30.0.0)(@types/node@20.17.19)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0)(jest-environment-node@30.3.0)': dependencies: '@jest/core': 30.3.0(babel-plugin-macros@3.1.0)(esbuild-register@3.6.0(esbuild@0.28.0)) '@jest/reporters': 30.3.0 @@ -24971,13 +24989,13 @@ snapshots: transitivePeerDependencies: - '@types/node' - '@rushstack/heft-node-rig@2.11.51(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19)(babel-plugin-macros@3.1.0)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0)': + '@rushstack/heft-node-rig@2.11.51(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0)': dependencies: '@microsoft/api-extractor': 7.59.2(@types/node@20.17.19) '@rushstack/eslint-config': 4.8.0(eslint@9.37.0)(typescript@5.8.2) '@rushstack/heft': 1.3.1(@types/node@20.17.19) '@rushstack/heft-api-extractor-plugin': 1.3.28(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19) - '@rushstack/heft-jest-plugin': 2.0.18(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/jest@30.0.0)(@types/node@20.17.19)(babel-plugin-macros@3.1.0)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0)(jest-environment-node@30.3.0) + '@rushstack/heft-jest-plugin': 2.0.18(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/jest@30.0.0)(@types/node@20.17.19)(esbuild-register@3.6.0(esbuild@0.28.0))(jest-environment-jsdom@30.3.0)(jest-environment-node@30.3.0) '@rushstack/heft-lint-plugin': 1.3.0(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19) '@rushstack/heft-typescript-plugin': 1.3.23(@rushstack/heft@1.3.1(@types/node@20.17.19))(@types/node@20.17.19) '@types/jest': 30.0.0 diff --git a/common/config/subspaces/default/repo-state.json b/common/config/subspaces/default/repo-state.json index 7b9bfa0d93..c4bdac4d9b 100644 --- a/common/config/subspaces/default/repo-state.json +++ b/common/config/subspaces/default/repo-state.json @@ -1,5 +1,5 @@ // DO NOT MODIFY THIS FILE MANUALLY BUT DO COMMIT IT. It is generated and used by Rush. { - "pnpmShrinkwrapHash": "85270521602b3b33e51d632eb11e4defdabae46d", + "pnpmShrinkwrapHash": "0bba018dc100225f133b4d52644f2ce57c457132", "preferredVersionsHash": "029c99bd6e65c5e1f25e2848340509811ff9753c" } diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 465a2ce471..4a07758b95 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -16,6 +16,9 @@ import { IDaemonRequestRejectedMessage } from '@rushstack/rush-daemon-protocol'; import { IDaemonShutdownAckMessage } from '@rushstack/rush-daemon-protocol'; import type { Readable } from 'node:stream'; +// @beta +export function assertDaemonRuntimeFolderIsPrivate(paths: IDaemonPaths): void; + // @beta export function captureDaemonRequest(options: ICaptureDaemonRequestOptions): IDaemonRequestEnvelope; @@ -57,6 +60,9 @@ export type DaemonClientOutcome = { readonly rejection: IDaemonRequestRejectedMessage['payload']; }; +// @beta +export type DaemonStartupHelperState = 'running' | 'exited' | 'unknown'; + // @beta export function executeWithDaemonRestartAsync(client: DaemonClient, connection: IConnectOrStartDaemonOptions, execution: IDaemonClientExecuteOptions): Promise; @@ -139,10 +145,23 @@ export interface IDaemonStartCommand { readonly environment: Readonly>; } +// @beta +export interface IDaemonStartupReservationInfo { + readonly helperPid?: number; + readonly helperState: DaemonStartupHelperState; + readonly path: string; +} + +// @beta +export function inspectDaemonStartupReservation(paths: IDaemonPaths): IDaemonStartupReservationInfo | undefined; + // @beta export function requestDaemonShutdownAsync(client: DaemonClient, paths: IDaemonPaths, timeoutMs?: number): Promise>; // @beta export function resetDaemonArtifactsAsync(paths: IDaemonPaths, options?: IDaemonArtifactResetOptions): Promise; +// @beta +export function resolveDaemonStartupReservationAsync(client: DaemonClient, paths: IDaemonPaths, timeoutMs?: number): Promise; + ``` diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index a00763848c..1747837712 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -79,6 +79,9 @@ export const DAEMON_REQUEST_ADMISSION_PROTOCOL_MINOR: number; // @beta export const DAEMON_REQUEST_LIFECYCLE_PROTOCOL_MINOR: number; +// @beta +export const DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR: number; + // @beta export const DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR: number; diff --git a/common/reviews/api/rush-daemon-transport.api.md b/common/reviews/api/rush-daemon-transport.api.md index adb1942b0d..df4aaef215 100644 --- a/common/reviews/api/rush-daemon-transport.api.md +++ b/common/reviews/api/rush-daemon-transport.api.md @@ -8,12 +8,18 @@ import type { IDaemonFrame } from '@rushstack/rush-daemon-protocol'; import type { IDaemonProtocolVersion } from '@rushstack/rush-daemon-protocol'; import type * as net from 'node:net'; +// @beta +export function assertDaemonRuntimeDirIsPrivate(paths: IDaemonPaths): void; + // @beta export function computeDaemonWorkspaceKey(input: IWorkspaceKeyInput): string; // @beta export function connectDaemonAsync(socketPath: string, options?: IDaemonConnectorOptions): Promise; +// @beta +export const DAEMON_RUNTIME_DIR_ENV_VAR: 'RUSHD_RUNTIME_DIR'; + // @beta export class DaemonFrameConnection { constructor(socket: net.Socket); @@ -53,7 +59,8 @@ export enum DaemonTransportErrorCode { connectionRefused = "connectionRefused", connectionTimeout = "connectionTimeout", daemonAlreadyRunning = "daemonAlreadyRunning", - transportClosed = "transportClosed" + transportClosed = "transportClosed", + unsafeRuntimeDirectory = "unsafeRuntimeDirectory" } // @beta diff --git a/docs/rush/dogfooding-rush-daemon.md b/docs/rush/dogfooding-rush-daemon.md index 7cd51776c7..9a5c3d282d 100644 --- a/docs/rush/dogfooding-rush-daemon.md +++ b/docs/rush/dogfooding-rush-daemon.md @@ -55,8 +55,11 @@ workspace, run the following from the repository root. It must print a path unde node -p "require('fs').realpathSync(require.resolve('@microsoft/rush-lib', { paths: [require('path').resolve('common/temp/rush-daemon-dogfood/apps/rush-cli-client')] }))" ``` -The built-in cloud build-cache plugins are not part of the snapshot. That is fine for this repository, whose -build cache is `local-only`. +The snapshot also contains Rush's built-in cloud build-cache plugins (`amazon-s3`, `azure-blob-storage` and +`http`), next to the `@microsoft/rush-lib` links of `@microsoft/rush`, `rush-client` and the daemon. A source-built rush-lib loads them +from there, and it points `_RUSH_LIB_PATH` at that same link. So plugins that resolve `@microsoft/rush-lib` by name +from `_RUSH_LIB_PATH` work in both the daemon and the in-process fallback. This repository's own build cache is +`local-only`, so it doesn't need them. ## 3. Opt in and build diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index f2a1389416..3d3c57504a 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -41,13 +41,30 @@ returns a `restartRetriesExhausted` fallback outcome so the caller can run in-pr Cancellation stops waiting without killing a daemon. Disabling auto-start still permits waiting for a host-started successor, but never lets the client spawn one. +A connection lost before the result stays a `disconnected` `DaemonClientError`. Its message +starts with "Daemon disconnected before delivering a result; the command was not retried." +and `executeWithDaemonRestartAsync()` appends what happened to the daemon that served the +attempt, identified by its pong PID. If that process exited within about a second (on Linux, +an exited process that is not reaped yet counts), the message names the PID, `rush-client +daemon logs` and `--no-daemon` (`rushx-client` for Rushx requests), and adds a second line +with the first fatal error that `.log` gained after the request was sent: a +Node.js uncaught-exception report or a V8 `FATAL ERROR:` line, clipped to one printable line. +If the process still runs, the message says that only the connection closed. After the abort +signal fires, the error is unchanged, so the caller reports the cancellation. + `connectOrStartDaemonAsync()` accepts an **explicit, version-selected** executable, arguments, environment and cwd. It does not discover or install a Rush version. +It adds `RUSHD_RUNTIME_DIR`, set to the base of `paths.runtimeDir`, to that environment, so the +daemon resolves the same paths as its clients whatever environment it inherits. +A runtime folder that is a symbolic link, is not a directory or belongs to another user is +refused as `startupFailed` before anything in it is trusted +(`assertDaemonRuntimeFolderIsPrivate()`); one that others can open is made owner-only. It reuses transport paths/reclaim checks and node-core-library's process-identity aware `LockFile` for the first-start mutex, including kernel-enforced exclusive file sharing on Windows. The winning client rechecks readiness, -reclaims only an absent/dead owner, and reserves `.starting` before -handing the explicit command to a detached startup helper. The helper spawns without +reclaims only an absent/dead owner, spawns a detached startup helper, and reserves +`.starting` for it (recording the helper's PID and start time) before +handing it the explicit command. The helper spawns the launcher without a shell and retains that reservation until the daemon completes hello/ping readiness, independently of whether the requesting client survives. It waits for a live launcher for at least 120 seconds, even when the requesting client's own deadline is shorter, so a slow @@ -73,6 +90,23 @@ Recovery of an abandoned reservation requires operator confirmation that the ori startup cannot still publish an endpoint; normal successful startup releases it automatically. Cancellation stops the client waiting, not the detached handoff. +A client resolves a reservation on the same evidence the helper waits for, so a daemon +that became ready after its helper stopped waiting (for example a first start slower than +120 seconds) is still used: holding the start mutex, the client needs a daemon that +completes hello/ping at the endpoint and whose pong PID is the live owner in the +ownership record for that socket, and it removes the reservation only if it is unchanged. +Reservations written by older clients, without a helper, are resolved the same way. +The recorded helper decides how long a refused launch waits: while it is alive, a starting +client waits for it until the client's own deadline; once it is provably gone (its PID +no longer exists or was reused), nothing else can release the reservation, so clients +refuse another launch at once. `inspectDaemonStartupReservation(paths)` reports the +reservation and its helper's state without changing it (`rush-client daemon status`), and +`requestDaemonShutdownAsync()` resolves a reservation for the attested daemon before it +sends shutdown, so that its successor can start. `resolveDaemonStartupReservationAsync(client, +paths)` does the same for a caller that stops the daemon without replacing it (`rush-client +daemon stop`): it returns false and keeps the reservation when the daemon is not the attested +owner, the reservation changed, or another client holds the start mutex past the timeout. + If a wire-compatible daemon reports the wrong implementation version and an explicit replacement launcher is available, startup serializes replacement under that same mutex. It verifies the old endpoint's attested ownership, requests shutdown, waits for ownership diff --git a/libraries/rush-client-core/src/DaemonClient.ts b/libraries/rush-client-core/src/DaemonClient.ts index 9cdc0e5b21..2e7b2831c4 100644 --- a/libraries/rush-client-core/src/DaemonClient.ts +++ b/libraries/rush-client-core/src/DaemonClient.ts @@ -30,7 +30,7 @@ import { } from '@rushstack/rush-daemon-protocol'; import { connectDaemonAsync, type DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; -import { DaemonClientError } from './DaemonClientError'; +import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from './DaemonClientError'; const MAX_STDIN_CHUNK_BYTES: number = 64 * 1024; @@ -138,7 +138,7 @@ export class DaemonClient { 'disconnected', this.#shutdown ? 'Daemon disconnected before acknowledging shutdown.' - : 'Daemon disconnected before delivering a result; the command was not retried.' + : DAEMON_DISCONNECTED_MESSAGE ) ); }); diff --git a/libraries/rush-client-core/src/DaemonClientError.ts b/libraries/rush-client-core/src/DaemonClientError.ts index 09b21b881f..3b32286fa8 100644 --- a/libraries/rush-client-core/src/DaemonClientError.ts +++ b/libraries/rush-client-core/src/DaemonClientError.ts @@ -4,6 +4,10 @@ /** A failure before readiness, or a connection lost without an authoritative result. @beta */ export type DaemonClientErrorCode = 'timeout' | 'versionMismatch' | 'disconnected' | 'startupFailed'; +/** Guidance and tools match this sentence, so explanations of a lost connection are appended after it. */ +export const DAEMON_DISCONNECTED_MESSAGE: string = + 'Daemon disconnected before delivering a result; the command was not retried.'; + /** An actionable client failure. Never replay a request following a disconnect. @beta */ export class DaemonClientError extends Error { public readonly code: DaemonClientErrorCode; diff --git a/libraries/rush-client-core/src/DaemonDisconnect.ts b/libraries/rush-client-core/src/DaemonDisconnect.ts new file mode 100644 index 0000000000..14ad575411 --- /dev/null +++ b/libraries/rush-client-core/src/DaemonDisconnect.ts @@ -0,0 +1,185 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; +import { + DaemonTransportError, + DaemonTransportErrorCode, + readDaemonLockfile, + type IDaemonLockfile, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; + +import type { DaemonClient } from './DaemonClient'; +import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from './DaemonClientError'; +import { getDaemonLogFilePath } from './DaemonLogFile'; +import { isOwnerProcessAlive } from './DaemonOwnership'; +import { isProcessDefunct } from './ProcessStartTime'; + +/** A process closes its connections while it exits, so it can briefly outlive them. */ +const EXIT_WAIT_MS: number = 1000; +const EXIT_POLL_INTERVAL_MS: number = 20; +/** A crash report is the last thing a daemon writes; a successor started since then writes little. */ +const MAX_LOG_READ_BYTES: number = 64 * 1024; +const MAX_LOGGED_ERROR_LENGTH: number = 240; + +const V8_FATAL_ERROR: RegExp = /^FATAL ERROR: \S/; +const NODE_REPORT_TRAILER: RegExp = /^Node\.js v\d+\.\d+\.\d+/; +const SOURCE_LOCATION: RegExp = /:\d+$/; +const SOURCE_ARROW: RegExp = /^\s*\^[\^~]*\s*$/; +const STACK_FRAME: RegExp = /^\s+at /; +const TRACE_UNCAUGHT_HINT: string = '(Use `node --trace-uncaught'; +const CONTROL_CHARACTERS: RegExp = /\p{Cc}/gu; + +/** The daemon process that serves a request, and the size of its launcher log when the request was sent. */ +export interface IServingDaemon { + readonly pid: number; + /** The ownership record's start time when the record names this process; it detects a reused PID. */ + readonly startedAt: string | undefined; + readonly logFilePath: string; + readonly logOffset: number | undefined; +} + +/** Identifies the daemon behind a ready client, or returns undefined for a peer that does not report its PID. */ +export async function observeServingDaemonAsync( + client: DaemonClient, + paths: IDaemonPaths +): Promise { + const { pid } = await client.status; + if (pid === undefined) return undefined; + const owner: IDaemonLockfile | undefined = readDaemonLockfile(paths.lockfilePath); + const logFilePath: string = getDaemonLogFilePath(paths); + return { + pid, + startedAt: owner?.pid === pid ? owner.startedAt : undefined, + logFilePath, + logOffset: tryGetFileSize(logFilePath) + }; +} + +/** + * Explains a connection lost before a request's result by what happened to the daemon process: whether it + * exited, the fatal error its launcher log recorded, and how to recover. The request is never replayed. + * Other errors are returned unchanged. + */ +export async function explainLostConnectionAsync( + error: unknown, + daemon: IServingDaemon | undefined, + request: IDaemonRequestEnvelope +): Promise { + if (!daemon || !isConnectionLoss(error)) return error; + const exited: boolean | undefined = await waitForExitAsync(daemon); + if (exited === undefined) return error; + if (!exited) { + return new DaemonClientError( + 'disconnected', + `${DAEMON_DISCONNECTED_MESSAGE} The connection to rushd (PID ${daemon.pid}) closed, but the daemon is still running; run the command again.`, + { cause: error } + ); + } + const loggedError: string | undefined = readLoggedFatalError(daemon); + const client: string = request.invocationKind === 'rushx' ? 'rushx-client' : 'rush-client'; + const message: string = + `${DAEMON_DISCONNECTED_MESSAGE} rushd (PID ${daemon.pid}) exited while it ran the command; ` + + `"rush-client daemon logs" ${loggedError ? 'shows' : 'may show'} why. Run the command again; ` + + `if the daemon exits again, run the command with "${client} --no-daemon".`; + return new DaemonClientError( + 'disconnected', + loggedError ? `${message}\nThe daemon log reports: ${loggedError}` : message, + { cause: error } + ); +} + +/** + * Returns the first fatal error in launcher log lines: V8's `FATAL ERROR:` line, or the message of Node's report + * of an uncaught exception (its location, source line and caret, the error with its stack, then the Node.js + * version). Returns undefined when the first report does not have that shape. + */ +export function findLoggedFatalError(lines: ReadonlyArray): string | undefined { + for (let index: number = 0; index < lines.length; index++) { + if (V8_FATAL_ERROR.test(lines[index])) return lines[index].trim(); + if (NODE_REPORT_TRAILER.test(lines[index])) return findUncaughtErrorMessage(lines, index); + } + return undefined; +} + +function findUncaughtErrorMessage(lines: ReadonlyArray, trailer: number): string | undefined { + let arrow: number = trailer - 1; + while (arrow >= 2 && !SOURCE_ARROW.test(lines[arrow])) arrow--; + if (arrow < 2 || !SOURCE_LOCATION.test(lines[arrow - 2])) return undefined; + const message: string[] = []; + for (let index: number = arrow + 1; index < trailer && !STACK_FRAME.test(lines[index]); index++) { + const line: string = lines[index].trim(); + if (line && !line.startsWith(TRACE_UNCAUGHT_HINT)) message.push(line); + } + return message.length ? message.join(' ') : undefined; +} + +function isConnectionLoss(error: unknown): boolean { + if (error instanceof DaemonClientError) return error.code === 'disconnected'; + if (error instanceof DaemonTransportError) return error.code === DaemonTransportErrorCode.transportClosed; + const code: unknown = + typeof error === 'object' && error !== null && 'code' in error ? error.code : undefined; + return code === 'ECONNRESET' || code === 'EPIPE'; +} + +/** Resolves true once the process is gone, false if it still runs after the wait, or undefined if unknown. */ +async function waitForExitAsync(daemon: IServingDaemon): Promise { + const deadline: number = Date.now() + EXIT_WAIT_MS; + for (;;) { + try { + if (!isOwnerProcessAlive(daemon) || isProcessDefunct(daemon.pid)) return true; + } catch { + // For example EPERM: the PID exists but cannot be inspected. + return undefined; + } + if (Date.now() >= deadline) return false; + await delayAsync(EXIT_POLL_INTERVAL_MS); + } +} + +/** The fatal error that the launcher log gained since the request was sent, clipped to one printable line. */ +function readLoggedFatalError(daemon: IServingDaemon): string | undefined { + if (daemon.logOffset === undefined) return undefined; + const text: string | undefined = readLogTail(daemon.logFilePath, daemon.logOffset); + const loggedError: string | undefined = text && findLoggedFatalError(text.split(/\r?\n/)); + if (!loggedError) return undefined; + const printable: string = loggedError.replace(CONTROL_CHARACTERS, ''); + return printable.length > MAX_LOGGED_ERROR_LENGTH + ? `${printable.slice(0, MAX_LOGGED_ERROR_LENGTH - 1)}…` + : printable; +} + +/** Reads at most the last {@link MAX_LOG_READ_BYTES} after `offset`; undefined when the log cannot be read. */ +function readLogTail(logFilePath: string, offset: number): string | undefined { + let fd: number | undefined; + try { + fd = fs.openSync( + logFilePath, + // These distinct native flags have non-overlapping values. + fs.constants.O_RDONLY + + (process.platform === 'win32' ? 0 : fs.constants.O_NOFOLLOW + fs.constants.O_NONBLOCK) + ); + const stats: fs.Stats = fs.fstatSync(fd); + if (!stats.isFile() || stats.size < offset) return undefined; + const start: number = Math.max(offset, stats.size - MAX_LOG_READ_BYTES); + const buffer: Buffer = Buffer.alloc(stats.size - start); + return buffer.toString('utf8', 0, fs.readSync(fd, buffer, 0, buffer.length, start)); + } catch { + return undefined; + } finally { + if (fd !== undefined) fs.closeSync(fd); + } +} + +function tryGetFileSize(filePath: string): number | undefined { + try { + const stats: fs.Stats = fs.statSync(filePath); + return stats.isFile() ? stats.size : undefined; + } catch { + return undefined; + } +} diff --git a/libraries/rush-client-core/src/DaemonOwnership.ts b/libraries/rush-client-core/src/DaemonOwnership.ts index a7226d0ee9..c48949b75f 100644 --- a/libraries/rush-client-core/src/DaemonOwnership.ts +++ b/libraries/rush-client-core/src/DaemonOwnership.ts @@ -18,7 +18,7 @@ const RESET_RETRY_MS: number = 100; /** Printed wherever automatic recovery fails closed. */ export const DAEMON_RESET_HINT: string = - 'If no daemon is running for this workspace, run "rush-client daemon stop --force" to remove its stale files.'; + 'If "rush-client daemon status" cannot connect, run "rush-client daemon stop --force" to remove this workspace\'s stale daemon files.'; export type DaemonOwnership = Pick; diff --git a/libraries/rush-client-core/src/DaemonRuntimeFolder.ts b/libraries/rush-client-core/src/DaemonRuntimeFolder.ts new file mode 100644 index 0000000000..3d5694fc55 --- /dev/null +++ b/libraries/rush-client-core/src/DaemonRuntimeFolder.ts @@ -0,0 +1,60 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as path from 'node:path'; + +import { + DAEMON_RUNTIME_DIR_ENV_VAR, + DaemonTransportError, + DaemonTransportErrorCode, + assertDaemonRuntimeDirIsPrivate, + ensureDaemonRuntimeDir, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; + +import { DaemonClientError } from './DaemonClientError'; + +function toClientError(error: unknown): unknown { + return error instanceof DaemonTransportError && + error.code === DaemonTransportErrorCode.unsafeRuntimeDirectory + ? new DaemonClientError('startupFailed', error.message, { cause: error }) + : error; +} + +/** + * Checks the daemon runtime folder, when it exists, before a client trusts the socket, lockfile or log inside + * it (see `assertDaemonRuntimeDirIsPrivate` in `@rushstack/rush-daemon-transport`). + * + * @throws {@link DaemonClientError} with code `startupFailed` when the folder is unsafe, so that a caller that + * falls back to in-process Rush for startup failures also does so here. + * + * @beta + */ +export function assertDaemonRuntimeFolderIsPrivate(paths: IDaemonPaths): void { + try { + assertDaemonRuntimeDirIsPrivate(paths); + } catch (error) { + throw toClientError(error); + } +} + +/** Creates and checks the runtime folder, reporting an unsafe one as {@link assertDaemonRuntimeFolderIsPrivate} does. */ +export function ensureDaemonRuntimeFolder(paths: IDaemonPaths): void { + try { + ensureDaemonRuntimeDir(paths); + } catch (error) { + throw toClientError(error); + } +} + +/** + * Adds the base of the runtime folder that this client looks in to a daemon's start environment, so the daemon + * listens there whatever `XDG_RUNTIME_DIR`, `TMPDIR` or `RUSHD_RUNTIME_DIR` it would otherwise inherit. + */ +export function withDaemonRuntimeFolder( + environment: Readonly>, + paths: IDaemonPaths +): Readonly> { + if (paths.runtimeDir === undefined) return environment; + return { ...environment, [DAEMON_RUNTIME_DIR_ENV_VAR]: path.dirname(paths.runtimeDir) }; +} diff --git a/libraries/rush-client-core/src/DaemonStartup.ts b/libraries/rush-client-core/src/DaemonStartup.ts index 8fcd8e8477..56b01abdd4 100644 --- a/libraries/rush-client-core/src/DaemonStartup.ts +++ b/libraries/rush-client-core/src/DaemonStartup.ts @@ -24,31 +24,138 @@ export interface IDaemonStartupOptions { readonly timeoutMs: number; } +/** + * The detached startup helper recorded in a reservation. Only this process releases the reservation after + * readiness, so once it is provably gone, waiting for it cannot help. + */ +export interface IDaemonStartupHelper { + readonly pid: number; + /** Recorded after the helper was spawned, so a later process that reuses the PID is detectable. */ + readonly startedAt: string; +} + +/** A startup reservation as found on disk. */ +export interface IDaemonStartupReservation { + /** The exact contents, compared before removal; undefined when the entry cannot be read as a file. */ + readonly contents: string | undefined; + /** Undefined for a reservation that does not record a helper, for example one written by an older client. */ + readonly helper: IDaemonStartupHelper | undefined; +} + +interface IDaemonStartupRecord { + readonly token: string; + readonly helperPid: number; + readonly helperStartedAt: string; +} + export function getDaemonStartupFilePath(paths: IDaemonPaths): string { return `${paths.lockfilePath}.starting`; } -export function reserveDaemonStartup(paths: IDaemonPaths): string { +/** + * Reserves startup for a spawned helper that has not yet received its options, so the reservation names + * the helper before any launcher can start. + */ +export function reserveDaemonStartup(paths: IDaemonPaths, helper: IDaemonStartupHelper): string { const token: string = randomUUID(); - fs.writeFileSync(getDaemonStartupFilePath(paths), token, { flag: 'wx', mode: 0o600 }); + const record: IDaemonStartupRecord = { token, helperPid: helper.pid, helperStartedAt: helper.startedAt }; + fs.writeFileSync(getDaemonStartupFilePath(paths), JSON.stringify(record), { flag: 'wx', mode: 0o600 }); return token; } +export function readDaemonStartupReservation(paths: IDaemonPaths): IDaemonStartupReservation | undefined { + const filePath: string = getDaemonStartupFilePath(paths); + let contents: string; + try { + contents = fs.readFileSync(filePath, 'utf8'); + } catch (error) { + // Any other entry (for example a directory or a dangling link) still refuses another launch. + return isNotFound(error) && !fs.lstatSync(filePath, { throwIfNoEntry: false }) + ? undefined + : { contents: undefined, helper: undefined }; + } + const record: IDaemonStartupRecord | undefined = parseStartupRecord(contents); + return { + contents, + helper: record && { pid: record.helperPid, startedAt: record.helperStartedAt } + }; +} + +/** + * Removes `reservation` unless it changed since it was read, and reports whether it is gone. + * The caller must hold the start mutex, so the only concurrent change is the helper's own release. + */ +export function removeDaemonStartupIfUnchanged( + paths: IDaemonPaths, + reservation: IDaemonStartupReservation +): boolean { + const current: IDaemonStartupReservation | undefined = readDaemonStartupReservation(paths); + if (!current) return true; + if (reservation.contents === undefined || current.contents !== reservation.contents) return false; + unlinkIfPresent(getDaemonStartupFilePath(paths)); + return true; +} + function assertReservation(paths: IDaemonPaths, token: string): void { - if (fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8') !== token) { + const contents: string = fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8'); + if (parseStartupRecord(contents)?.token !== token) { throw new DaemonClientError('startupFailed', 'The daemon startup reservation changed ownership.'); } } +/** + * Releases the helper's own reservation. A missing reservation is already resolved: clients remove one only + * under the start mutex, after the same readiness evidence the helper waits for. + */ export function releaseDaemonStartup(paths: IDaemonPaths, token: string): void { - assertReservation(paths, token); - fs.unlinkSync(getDaemonStartupFilePath(paths)); + try { + assertReservation(paths, token); + } catch (error) { + if (isNotFound(error)) return; + throw error; + } + unlinkIfPresent(getDaemonStartupFilePath(paths)); +} + +function parseStartupRecord(contents: string): IDaemonStartupRecord | undefined { + let record: unknown; + try { + record = JSON.parse(contents); + } catch { + return undefined; + } + if (typeof record !== 'object' || record === null) return undefined; + const { token, helperPid, helperStartedAt } = record as Partial< + Record + >; + return typeof token === 'string' && + typeof helperPid === 'number' && + Number.isSafeInteger(helperPid) && + helperPid > 0 && + typeof helperStartedAt === 'string' && + Number.isFinite(Date.parse(helperStartedAt)) + ? { token, helperPid, helperStartedAt } + : undefined; +} + +function unlinkIfPresent(filePath: string): void { + try { + fs.unlinkSync(filePath); + } catch (error) { + if (!isNotFound(error)) throw error; + } +} + +function isNotFound(error: unknown): boolean { + return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT'; } /** * Runs independently of the requesting client. Once spawn succeeds, only protocol readiness releases * the reservation: an arbitrary launcher may outlive its parent or spawn descendants. * Failure before readiness deliberately leaves a durable reservation instead of guessing that a PID is safe. + * The reservation records this helper, so once it exits, later clients report the retained reservation at once + * instead of waiting for a release that cannot happen. */ export async function runDaemonStartupAsync(options: IDaemonStartupOptions): Promise { const { paths, startCommand: start, token, timeoutMs } = options; diff --git a/libraries/rush-client-core/src/DaemonStartupReservation.ts b/libraries/rush-client-core/src/DaemonStartupReservation.ts new file mode 100644 index 0000000000..301a21cff6 --- /dev/null +++ b/libraries/rush-client-core/src/DaemonStartupReservation.ts @@ -0,0 +1,114 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { + readDaemonLockfile, + type IDaemonLockfile, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; + +import type { DaemonClient } from './DaemonClient'; +import { isDaemonOwnership, isOwnerProcessAlive } from './DaemonOwnership'; +import { + getDaemonStartupFilePath, + readDaemonStartupReservation, + removeDaemonStartupIfUnchanged, + type IDaemonStartupHelper, + type IDaemonStartupReservation +} from './DaemonStartup'; +import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; + +/** + * What a startup reservation's recorded helper can still do. `running`: it may still release the reservation. + * `exited`: it is provably gone, so only a client that finds the daemon ready, or + * `resetDaemonArtifactsAsync()`, removes the reservation. `unknown`: the reservation records no helper, + * for example because an older client wrote it. + * @beta + */ +export type DaemonStartupHelperState = 'running' | 'exited' | 'unknown'; + +/** A daemon startup reservation (`.starting`), as reported by diagnostics. @beta */ +export interface IDaemonStartupReservationInfo { + /** The reservation file. */ + readonly path: string; + /** The detached startup helper that releases the reservation once the daemon is ready, when recorded. */ + readonly helperPid?: number; + /** Whether the helper can still release the reservation. */ + readonly helperState: DaemonStartupHelperState; +} + +/** + * Reads this workspace's startup reservation without changing it. + * @returns `undefined` when no reservation exists. + * @beta + */ +export function inspectDaemonStartupReservation( + paths: IDaemonPaths +): IDaemonStartupReservationInfo | undefined { + const reservation: IDaemonStartupReservation | undefined = readDaemonStartupReservation(paths); + if (!reservation) return undefined; + const { helper } = reservation; + return { + path: getDaemonStartupFilePath(paths), + ...(helper ? { helperPid: helper.pid } : {}), + helperState: getStartupHelperState(reservation) + }; +} + +export function getStartupHelperState(reservation: IDaemonStartupReservation): DaemonStartupHelperState { + if (!reservation.helper) return 'unknown'; + return isStartupHelperAlive(reservation.helper) ? 'running' : 'exited'; +} + +function isStartupHelperAlive(helper: IDaemonStartupHelper): boolean { + try { + return isOwnerProcessAlive(helper); + } catch { + // For example EPERM: the PID exists but belongs to another user, so the helper cannot be shown to be gone. + return true; + } +} + +/** + * Removes a retained startup reservation on the evidence that its helper waits for: a daemon that completed + * hello/ping at this endpoint. This also requires `pid` to be the live owner in the ownership record, so the + * endpoint cannot be handed to a second launch. The caller must hold the start mutex; a reservation made + * after this check is never removed. + * @returns true when no reservation remains. + */ +export function resolveStartupReservationForReadyDaemon( + paths: IDaemonPaths, + pid: number | undefined +): boolean { + const reservation: IDaemonStartupReservation | undefined = readDaemonStartupReservation(paths); + if (!reservation) return true; + if (pid === undefined || !isAttestedDaemonOwner(paths, pid)) return false; + return removeDaemonStartupIfUnchanged(paths, reservation); +} + +/** + * {@link resolveStartupReservationForReadyDaemon} for a connected client, taking the start mutex without waiting. + * Returns false, keeping the reservation, while another client or this process holds the mutex. + */ +export async function tryResolveStartupReservationAsync( + client: DaemonClient, + paths: IDaemonPaths +): Promise { + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + if (!lock) return false; + try { + return resolveStartupReservationForReadyDaemon(paths, (await client.status).pid); + } finally { + await lock.releaseAsync(); + } +} + +function isAttestedDaemonOwner(paths: IDaemonPaths, pid: number): boolean { + const owner: IDaemonLockfile | undefined = readDaemonLockfile(paths.lockfilePath); + if (!isDaemonOwnership(owner) || owner.pid !== pid || owner.socketPath !== paths.socketPath) return false; + try { + return isOwnerProcessAlive(owner); + } catch { + return false; + } +} diff --git a/libraries/rush-client-core/src/ProcessStartTime.ts b/libraries/rush-client-core/src/ProcessStartTime.ts index a258752e5d..b2d318507b 100644 --- a/libraries/rush-client-core/src/ProcessStartTime.ts +++ b/libraries/rush-client-core/src/ProcessStartTime.ts @@ -6,6 +6,7 @@ import { performance } from 'node:perf_hooks'; // USER_HZ is fixed at 100 on mainstream Linux ABIs; it is verified against this process before use. const USER_HZ: number = 100; +const PROC_STAT_STATE_FIELD: number = 3; const PROC_STAT_START_TIME_FIELD: number = 22; const MAX_CALIBRATION_ERROR_MS: number = 1000; /** A process that began this long after a record was written cannot be the record's writer. */ @@ -48,7 +49,25 @@ function readUptimeSeconds(): number | undefined { } } +/** + * True only on Linux when `pid` has exited but its parent has not reaped it yet. Such a zombie still accepts + * signal 0, but it can no longer run or hold connections. + */ +export function isProcessDefunct(pid: number): boolean { + if (process.platform !== 'linux') return false; + const state: string | undefined = readStatFields(pid)?.[PROC_STAT_STATE_FIELD - 3]; + return state === 'Z' || state === 'X'; +} + function readStartSeconds(pid: number | 'self'): number | undefined { + const fields: string[] | undefined = readStatFields(pid); + if (!fields) return undefined; + const jiffies: number = Number(fields[PROC_STAT_START_TIME_FIELD - 3]); + return Number.isSafeInteger(jiffies) && jiffies >= 0 ? jiffies / USER_HZ : undefined; +} + +/** The fields of `/proc//stat` from field 3 (the process state) on. */ +function readStatFields(pid: number | 'self'): string[] | undefined { let stat: string; try { stat = fs.readFileSync(`/proc/${pid}/stat`, 'utf8'); @@ -58,10 +77,8 @@ function readStartSeconds(pid: number | 'self'): number | undefined { // The command name (field 2) may contain spaces and parentheses; fields after it never do. const commandEnd: number = stat.lastIndexOf(')'); if (commandEnd < 0) return undefined; - const fields: string[] = stat + return stat .slice(commandEnd + 1) .trim() .split(' '); - const jiffies: number = Number(fields[PROC_STAT_START_TIME_FIELD - 3]); - return Number.isSafeInteger(jiffies) && jiffies >= 0 ? jiffies / USER_HZ : undefined; } diff --git a/libraries/rush-client-core/src/connectOrStartDaemon.ts b/libraries/rush-client-core/src/connectOrStartDaemon.ts index 93aa22ba0e..b6f0b46e4b 100644 --- a/libraries/rush-client-core/src/connectOrStartDaemon.ts +++ b/libraries/rush-client-core/src/connectOrStartDaemon.ts @@ -11,7 +11,6 @@ import { DAEMON_LIFECYCLE_PROTOCOL_MINOR } from '@rushstack/rush-daemon-protocol import { DaemonTransportError, DaemonTransportErrorCode, - ensureDaemonRuntimeDir, reclaimStaleDaemonAsync, readDaemonLockfile, type IDaemonLockfile, @@ -30,12 +29,24 @@ import { reclaimAbandonedOwnershipAsync, type DaemonOwnership } from './DaemonOwnership'; +import { + assertDaemonRuntimeFolderIsPrivate, + ensureDaemonRuntimeFolder, + withDaemonRuntimeFolder +} from './DaemonRuntimeFolder'; import { getDaemonStartupFilePath, + readDaemonStartupReservation, reserveDaemonStartup, - releaseDaemonStartup, - type IDaemonStartupOptions + type IDaemonStartupOptions, + type IDaemonStartupReservation } from './DaemonStartup'; +import { + getStartupHelperState, + resolveStartupReservationForReadyDaemon, + tryResolveStartupReservationAsync, + type DaemonStartupHelperState +} from './DaemonStartupReservation'; import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; interface IStartupHelper { @@ -50,6 +61,9 @@ interface IStartupHelper { */ const STARTUP_HELPER_READINESS_TIMEOUT_MS: number = 120_000; +/** Matches the default of {@link DaemonClient.shutdownAsync}. */ +const DEFAULT_SHUTDOWN_TIMEOUT_MS: number = 15000; + /** A version-selected launch command supplied by the embedding application, never guessed by the core. @beta */ export interface IDaemonStartCommand { readonly command: string; @@ -94,6 +108,7 @@ export async function connectOrStartDaemonAsync( } const deadline: number = Date.now() + timeoutMs; options.abortSignal?.throwIfAborted(); + assertDaemonRuntimeFolderIsPrivate(options.paths); await waitForPreviousDaemonAsync(options.paths, options.previousDaemon, deadline, options.abortSignal); const initial: DaemonClient | undefined = await tryConnectAsync(options, deadline); if (initial) return initial; @@ -121,7 +136,7 @@ async function startDaemonAsync( options: IConnectOrStartDaemonOptions & { readonly startCommand: IDaemonStartCommand }, deadline: number ): Promise { - ensureDaemonRuntimeDir(options.paths); + ensureDaemonRuntimeFolder(options.paths); let lock: IStartupLock | undefined; let backoffMs: number = 50; while (Date.now() < deadline) { @@ -137,19 +152,7 @@ async function startDaemonAsync( } if (!lock) throw startupError(options, 'timed out waiting for another starting client'); try { - while (fs.lstatSync(getDaemonStartupFilePath(options.paths), { throwIfNoEntry: false })) { - const ready: DaemonClient | undefined = await tryConnectAsync(options, deadline); - if (ready) return ready; - if (Date.now() >= deadline) { - throw startupError( - options, - `has an unresolved startup handoff at ${getDaemonStartupFilePath(options.paths)}; refusing another launch. ${DAEMON_RESET_HINT}` - ); - } - await delayAsync(Math.min(100, Math.max(1, deadline - Date.now())), undefined, { - signal: options.abortSignal - }); - } + await waitForStartupReservationAsync(options, deadline); const ready: DaemonClient | undefined = await tryConnectAsync(options, deadline); if (ready) return ready; const replacement: DaemonClient | undefined = await replaceMismatchedDaemonAsync(options, deadline); @@ -195,6 +198,76 @@ async function startDaemonAsync( } } +/** + * Holding the start mutex, waits until no startup reservation remains. The helper releases its reservation once + * the daemon completes hello/ping, and this client resolves it on the same evidence, so a daemon that became + * ready after its helper stopped waiting is still used. Another launch is refused while the reservation remains: + * at once when its helper exited, since nothing else will release it, and otherwise at the deadline. + */ +async function waitForStartupReservationAsync( + options: IConnectOrStartDaemonOptions, + deadline: number +): Promise { + while (true) { + options.abortSignal?.throwIfAborted(); + const reservation: IDaemonStartupReservation | undefined = readDaemonStartupReservation(options.paths); + if (!reservation) return; + if (await tryResolveForReadyDaemonAsync(options, deadline)) continue; + const helperState: DaemonStartupHelperState = getStartupHelperState(reservation); + if (helperState === 'exited' || Date.now() >= deadline) { + // The helper may have released its reservation just before it exited. + const current: IDaemonStartupReservation | undefined = readDaemonStartupReservation(options.paths); + if (!current || current.contents !== reservation.contents) continue; + throw startupError(options, describeUnresolvedReservation(options.paths, reservation, helperState)); + } + await delayAsync(Math.min(100, Math.max(1, deadline - Date.now())), undefined, { + signal: options.abortSignal + }); + } +} + +/** + * Resolves the startup reservation if a daemon of any version completes hello/ping at the endpoint, which is + * the readiness the helper waits for; the normal flow then replaces a mismatched daemon. The caller holds the + * start mutex. + */ +async function tryResolveForReadyDaemonAsync( + options: IConnectOrStartDaemonOptions, + deadline: number +): Promise { + let client: DaemonClient | undefined; + try { + client = await tryConnectEndpointAsync({ ...options, expectedDaemonVersion: undefined }, deadline); + } catch (error) { + // Nor would the helper treat a daemon with an incompatible protocol as ready. + if (error instanceof DaemonClientError && error.code === 'versionMismatch') return false; + throw error; + } + if (!client) return false; + try { + return resolveStartupReservationForReadyDaemon(options.paths, (await client.status).pid); + } finally { + await client.closeAsync(); + } +} + +function describeUnresolvedReservation( + paths: IDaemonPaths, + reservation: IDaemonStartupReservation, + helperState: DaemonStartupHelperState +): string { + const prefix: string = `has an unresolved startup handoff at ${getDaemonStartupFilePath(paths)}`; + const helperPid: number | undefined = reservation.helper?.pid; + switch (helperState) { + case 'exited': + return `${prefix}: its startup helper (PID ${helperPid}) exited before the daemon became ready; refusing another launch. ${DAEMON_RESET_HINT}`; + case 'running': + return `${prefix}: its startup helper (PID ${helperPid}) is still waiting for the daemon to become ready; refusing another launch`; + default: + return `${prefix}; refusing another launch. ${DAEMON_RESET_HINT}`; + } +} + /** * Captures attested ownership and requests shutdown without claiming that cleanup has finished. * Pass the returned identity as previousDaemon to connectOrStartDaemonAsync before replacement. @@ -220,10 +293,79 @@ export async function requestDaemonShutdownAsync( pid: owner.pid, startedAt: owner.startedAt }; + await resolveReservationBeforeShutdownAsync(paths, pid, timeoutMs ?? DEFAULT_SHUTDOWN_TIMEOUT_MS); await client.shutdownAsync(timeoutMs); return previousDaemon; } +/** + * A retained startup reservation would refuse the successor's launch, so it is resolved first; `pid` answered + * hello/ping and is attested as the owner. Waits for the start mutex, since another client may be resolving it. + */ +async function resolveReservationBeforeShutdownAsync( + paths: IDaemonPaths, + pid: number, + timeoutMs: number +): Promise { + if (!readDaemonStartupReservation(paths)) return; + const lock: IStartupLock | undefined = await waitForStartupLockAsync(paths, timeoutMs); + if (!lock) { + throw new DaemonClientError( + 'startupFailed', + `Another client is starting the daemon for ${paths.lockfilePath}; shutdown was not sent.` + ); + } + try { + if (!resolveStartupReservationForReadyDaemon(paths, pid)) { + throw new DaemonClientError( + 'startupFailed', + `The daemon startup reservation at ${getDaemonStartupFilePath(paths)} could not be resolved for PID ${pid}; shutdown was not sent.` + ); + } + } finally { + await lock.releaseAsync(); + } +} + +/** + * Resolves a startup reservation that remains next to a connected, ready daemon before that daemon is stopped. + * Afterwards no ready daemon would prove the reservation stale, so it would refuse every automatic start. The + * daemon must be the live owner in the ownership record for the endpoint, and the reservation is removed only + * if unchanged. Waits for the start mutex for up to `timeoutMs` (15000 milliseconds by default), since another + * client may be resolving it. + * @returns true when no reservation remains; false when one is kept. + * @beta + */ +export async function resolveDaemonStartupReservationAsync( + client: DaemonClient, + paths: IDaemonPaths, + timeoutMs: number = DEFAULT_SHUTDOWN_TIMEOUT_MS +): Promise { + if (!readDaemonStartupReservation(paths)) return true; + const { pid } = await client.status; + const lock: IStartupLock | undefined = await waitForStartupLockAsync(paths, timeoutMs); + if (!lock) return false; + try { + return resolveStartupReservationForReadyDaemon(paths, pid); + } finally { + await lock.releaseAsync(); + } +} + +/** Returns undefined when another client still holds the start mutex after `timeoutMs`. */ +async function waitForStartupLockAsync( + paths: IDaemonPaths, + timeoutMs: number +): Promise { + const deadline: number = Date.now() + timeoutMs; + let lock: IStartupLock | undefined; + while (!(lock = await tryAcquireStartupLockAsync(paths))) { + if (Date.now() >= deadline) return undefined; + await delayAsync(Math.min(100, Math.max(1, deadline - Date.now()))); + } + return lock; +} + async function replaceMismatchedDaemonAsync( options: IConnectOrStartDaemonOptions, deadline: number @@ -255,23 +397,36 @@ async function replaceMismatchedDaemonAsync( async function tryConnectAsync( options: IConnectOrStartDaemonOptions, deadline: number +): Promise { + const client: DaemonClient | undefined = await tryConnectEndpointAsync(options, deadline); + if (!client) return undefined; + // Do not expose a just-started daemon to shutdown/restart until its startup reservation is resolved: + // by the helper, or here once the daemon is ready. This never waits for the start mutex, and a client + // holding it (including this process) resolves the reservation itself. + let usable: boolean = false; + try { + options.abortSignal?.throwIfAborted(); + usable = + !readDaemonStartupReservation(options.paths) || + (await tryResolveStartupReservationAsync(client, options.paths)); + } finally { + if (!usable) await client.closeAsync(); + } + return usable ? client : undefined; +} + +/** Connects and completes hello/ping, whether or not a startup reservation remains. */ +async function tryConnectEndpointAsync( + options: IConnectOrStartDaemonOptions, + deadline: number ): Promise { options.abortSignal?.throwIfAborted(); try { - const client: DaemonClient = await DaemonClient.connectAsync({ + return await DaemonClient.connectAsync({ ...options, socketPath: options.paths.socketPath, timeoutMs: Math.min(options.timeoutMs ?? 1000, Math.max(1, deadline - Date.now())) }); - // Do not expose a just-started daemon to shutdown/restart until the helper finishes the handoff. - let pendingStartup: boolean = true; - try { - options.abortSignal?.throwIfAborted(); - pendingStartup = !!fs.lstatSync(getDaemonStartupFilePath(options.paths), { throwIfNoEntry: false }); - } finally { - if (pendingStartup) await client.closeAsync(); - } - return pendingStartup ? undefined : client; } catch (error) { options.abortSignal?.throwIfAborted(); if ( @@ -403,7 +558,8 @@ async function spawnDetachedAsync( ): Promise { const start: IDaemonStartCommand = { ...options.startCommand!, - cwd: path.resolve(options.startCommand!.cwd) + cwd: path.resolve(options.startCommand!.cwd), + environment: withDaemonRuntimeFolder(options.startCommand!.environment, options.paths) }; const logFilePath: string = getDaemonLogFilePath(options.paths); // These distinct native flags have non-overlapping values. @@ -427,7 +583,6 @@ async function spawnDetachedAsync( } fs.fchmodSync(logFd, 0o600); } - const token: string = reserveDaemonStartup(options.paths); let helper: IStartupHelper | undefined; try { const child: ChildProcess = spawn(process.execPath, [path.join(__dirname, 'runDaemonStartup.js')], { @@ -443,7 +598,6 @@ async function spawnDetachedAsync( await once(child, 'spawn'); } catch (error) { if (helper) await helper.closed; - releaseDaemonStartup(options.paths, token); throw new DaemonClientError( 'startupFailed', `Unable to start ${start.command}; inspect ${logFilePath}.`, @@ -451,6 +605,17 @@ async function spawnDetachedAsync( ); } const { child } = helper; + let token: string; + try { + // The helper launches nothing until it receives its options, so the reservation can name it first. + // Its start time is taken after the spawn, so the helper can never look like a later reuse of its PID. + token = reserveDaemonStartup(options.paths, { pid: child.pid!, startedAt: new Date().toISOString() }); + } catch (error) { + // Disconnected without options, the helper exits without launching. + child.disconnect(); + await helper.closed; + throw error; + } child.unref(); const startup: IDaemonStartupOptions = { paths: options.paths, @@ -500,8 +665,9 @@ async function waitForHelperExitAsync( } function startupError(options: IConnectOrStartDaemonOptions, reason: string): DaemonClientError { + const sentence: string = /[.!?]$/.test(reason) ? reason : `${reason}.`; return new DaemonClientError( 'startupFailed', - `Daemon startup ${reason}. Inspect ${getDaemonLogFilePath(options.paths)} and retry, or use --no-daemon.` + `Daemon startup ${sentence} Inspect ${getDaemonLogFilePath(options.paths)} and retry, or use --no-daemon.` ); } diff --git a/libraries/rush-client-core/src/executeWithDaemonRestart.ts b/libraries/rush-client-core/src/executeWithDaemonRestart.ts index e0d8563648..d63c6d3ce7 100644 --- a/libraries/rush-client-core/src/executeWithDaemonRestart.ts +++ b/libraries/rush-client-core/src/executeWithDaemonRestart.ts @@ -10,6 +10,11 @@ import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from './ import { captureDaemonRequest } from './captureDaemonRequest'; import type { DaemonClient, DaemonClientOutcome, IDaemonClientExecuteOptions } from './DaemonClient'; import { DaemonClientError } from './DaemonClientError'; +import { + explainLostConnectionAsync, + observeServingDaemonAsync, + type IServingDaemon +} from './DaemonDisconnect'; /** * The maximum number of successors a single request follows. Each restart serves at least one other @@ -27,6 +32,8 @@ const DEFAULT_STARTUP_TIMEOUT_MS: number = 15000; * Restarts are retried with jittered backoff inside the request's admission deadline; once the * retries or the deadline are exhausted, a `fallback` outcome lets the caller run in-process instead. * The connection options must select the request's expected daemon and startup environment. + * A connection lost before the result is reported as a `disconnected` error that says whether the daemon + * process exited, what its launcher log recorded and how to recover, unless the request was aborted first. * @beta */ export async function executeWithDaemonRestartAsync( @@ -41,7 +48,10 @@ export async function executeWithDaemonRestartAsync( : (execution.abortSignal ?? connection.abortSignal); const waitTimeoutMs: number | undefined = execution.request.admission?.waitTimeoutMs; let owner: IDaemonLockfile | undefined = await attestOwnerAsync(client, connection); - let outcome: DaemonClientOutcome = await client.executeAsync({ ...execution, abortSignal }); + let outcome: DaemonClientOutcome = await executeOnDaemonAsync(client, connection, { + ...execution, + abortSignal + }); let previous: DaemonClient | undefined; try { for (let retry: number = 1; outcome.kind === 'result' && outcome.result.retryAfterRestart; retry++) { @@ -97,7 +107,7 @@ export async function executeWithDaemonRestartAsync( const remainingMs: number | undefined = getRemainingMs(); if (isExpired(remainingMs)) return restartExhaustedOutcome(retry); owner = await attestOwnerAsync(successor, connection); - outcome = await successor.executeAsync({ + outcome = await executeOnDaemonAsync(successor, connection, { ...execution, abortSignal, request: @@ -115,6 +125,22 @@ export async function executeWithDaemonRestartAsync( } } +/** Executes one attempt; a lost connection is explained by what happened to the daemon that served it. */ +async function executeOnDaemonAsync( + client: DaemonClient, + connection: IConnectOrStartDaemonOptions, + execution: IDaemonClientExecuteOptions +): Promise { + const daemon: IServingDaemon | undefined = await observeServingDaemonAsync(client, connection.paths); + try { + return await client.executeAsync(execution); + } catch (error) { + // After cancellation the caller reports the cancellation, whatever the connection did afterwards. + if (execution.abortSignal?.aborted) throw error; + throw await explainLostConnectionAsync(error, daemon, execution.request); + } +} + function isExpired(remainingMs: number | undefined): boolean { return remainingMs !== undefined && remainingMs <= 0; } diff --git a/libraries/rush-client-core/src/index.ts b/libraries/rush-client-core/src/index.ts index 7597de72ab..6f3fbd5e3c 100644 --- a/libraries/rush-client-core/src/index.ts +++ b/libraries/rush-client-core/src/index.ts @@ -20,10 +20,17 @@ export { type IDaemonArtifactResetOptions, type IDaemonArtifactResetResult } from './DaemonOwnership'; +export { assertDaemonRuntimeFolderIsPrivate } from './DaemonRuntimeFolder'; +export { + inspectDaemonStartupReservation, + type DaemonStartupHelperState, + type IDaemonStartupReservationInfo +} from './DaemonStartupReservation'; export { executeWithDaemonRestartAsync } from './executeWithDaemonRestart'; export { connectOrStartDaemonAsync, requestDaemonShutdownAsync, + resolveDaemonStartupReservationAsync, type IConnectOrStartDaemonOptions, type IDaemonStartCommand } from './connectOrStartDaemon'; diff --git a/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts b/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts new file mode 100644 index 0000000000..39d6bde652 --- /dev/null +++ b/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts @@ -0,0 +1,121 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawnSync } from 'node:child_process'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; +import { DaemonTransportError, DaemonTransportErrorCode } from '@rushstack/rush-daemon-transport'; + +import { captureDaemonRequest } from '../captureDaemonRequest'; +import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from '../DaemonClientError'; +import { explainLostConnectionAsync, findLoggedFatalError, type IServingDaemon } from '../DaemonDisconnect'; +import { isProcessDefunct } from '../ProcessStartTime'; +import { withUnreapedChildAsync } from './UnreapedChildProcess'; + +const linuxIt: typeof it = process.platform === 'linux' ? it : it.skip; + +/** The lines that this Node.js version writes to stderr when `script` fails with an uncaught error. */ +function getCrashReport(script: string): string[] { + const { status, stderr } = spawnSync(process.execPath, ['-e', script], { encoding: 'utf8' }); + expect(status).not.toBe(0); + return stderr.split(/\r?\n/); +} + +describe(findLoggedFatalError.name, () => { + it('returns the message of an uncaught error without its stack', () => { + const report: string[] = getCrashReport("setImmediate(() => { throw new Error('first\\nsecond'); })"); + expect(findLoggedFatalError(['rushd started', ...report])).toBe('Error: first second'); + }); + + it('returns the message of an unhandled rejection', () => { + expect(findLoggedFatalError(getCrashReport("Promise.reject(new TypeError('rejected'))"))).toBe( + 'TypeError: rejected' + ); + expect(findLoggedFatalError(getCrashReport("Promise.reject('reason')"))).toMatch( + /^UnhandledPromiseRejection: .* "reason"\.$/ + ); + }); + + it('returns a thrown value that is not an error', () => { + expect(findLoggedFatalError(getCrashReport("throw 'a plain string'"))).toBe('a plain string'); + }); + + it("returns V8's fatal error line", () => { + const lines: string[] = [ + '<--- JS stacktrace --->', + '', + 'FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory', + '----- Native stack trace -----' + ]; + expect(findLoggedFatalError(lines)).toBe( + 'FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory' + ); + }); + + it('returns the first report', () => { + const lines: string[] = [ + ...getCrashReport("throw new Error('first')"), + ...getCrashReport("throw new Error('second')") + ]; + expect(findLoggedFatalError(lines)).toBe('Error: first'); + }); + + it('returns undefined without a complete report', () => { + const report: string[] = getCrashReport("throw new Error('boom')"); + const trailer: number = report.findIndex((line) => line.startsWith('Node.js v')); + expect(trailer).toBeGreaterThan(0); + expect(findLoggedFatalError(report.slice(0, trailer))).toBeUndefined(); + expect(findLoggedFatalError(report.slice(3))).toBeUndefined(); + expect(findLoggedFatalError(['rushd started', 'Node.js v22.0.0'])).toBeUndefined(); + }); +}); + +describe(explainLostConnectionAsync.name, () => { + const request: IDaemonRequestEnvelope = captureDaemonRequest({ + argv: ['build'], + commandName: 'build', + commandOrigin: 'built-in', + cwd: os.tmpdir(), + environment: {}, + terminal: { isTTY: false, supportsColor: false } + }); + + function getServingDaemon(pid: number): IServingDaemon { + return { + pid, + startedAt: undefined, + logFilePath: path.join(os.tmpdir(), 'missing.log'), + logOffset: undefined + }; + } + + it('returns other failures, and failures without a known daemon, unchanged', async () => { + const timeout: DaemonClientError = new DaemonClientError('timeout', 'The daemon did not answer.'); + expect(await explainLostConnectionAsync(timeout, getServingDaemon(process.pid), request)).toBe(timeout); + const lost: DaemonClientError = new DaemonClientError('disconnected', DAEMON_DISCONNECTED_MESSAGE); + expect(await explainLostConnectionAsync(lost, undefined, request)).toBe(lost); + }); + + linuxIt('treats a daemon that exited but is not reaped yet as exited', async () => { + await withUnreapedChildAsync(async (child) => { + const deadline: number = Date.now() + 5000; + while (!isProcessDefunct(child) && Date.now() < deadline) await delayAsync(20); + const closed: DaemonTransportError = new DaemonTransportError( + DaemonTransportErrorCode.transportClosed, + 'The daemon connection closed.' + ); + const startedAt: number = Date.now(); + const explained: unknown = await explainLostConnectionAsync(closed, getServingDaemon(child), request); + expect(Date.now() - startedAt).toBeLessThan(500); + expect(explained).toBeInstanceOf(DaemonClientError); + expect(explained).toMatchObject({ + code: 'disconnected', + cause: closed, + message: `${DAEMON_DISCONNECTED_MESSAGE} rushd (PID ${child}) exited while it ran the command; "rush-client daemon logs" may show why. Run the command again; if the daemon exits again, run the command with "rush-client --no-daemon".` + }); + }); + }); +}); diff --git a/libraries/rush-client-core/src/test/ProcessStartTime.test.ts b/libraries/rush-client-core/src/test/ProcessStartTime.test.ts index a7a9ff4418..f2d1db45cb 100644 --- a/libraries/rush-client-core/src/test/ProcessStartTime.test.ts +++ b/libraries/rush-client-core/src/test/ProcessStartTime.test.ts @@ -4,8 +4,10 @@ import { spawn, type ChildProcess } from 'node:child_process'; import { once } from 'node:events'; import { performance } from 'node:perf_hooks'; +import { setTimeout as delayAsync } from 'node:timers/promises'; -import { isProcessStartedAfter, tryGetProcessStartTimeMs } from '../ProcessStartTime'; +import { isProcessDefunct, isProcessStartedAfter, tryGetProcessStartTimeMs } from '../ProcessStartTime'; +import { withUnreapedChildAsync } from './UnreapedChildProcess'; const linuxIt: typeof it = process.platform === 'linux' ? it : it.skip; @@ -28,4 +30,14 @@ describe('process start time', () => { expect(isProcessStartedAfter(exited.pid!, new Date(0).toISOString())).toBe(false); expect(isProcessStartedAfter(process.pid, 'not a timestamp')).toBe(false); }); + + linuxIt('detects an exited process that its parent has not reaped', async () => { + await withUnreapedChildAsync(async (child, parent) => { + const deadline: number = Date.now() + 5000; + while (!isProcessDefunct(child) && Date.now() < deadline) await delayAsync(20); + expect(isProcessDefunct(child)).toBe(true); + expect(isProcessDefunct(parent)).toBe(false); + expect(isProcessDefunct(process.pid)).toBe(false); + }); + }); }); diff --git a/libraries/rush-client-core/src/test/UnreapedChildProcess.ts b/libraries/rush-client-core/src/test/UnreapedChildProcess.ts new file mode 100644 index 0000000000..b5767015fb --- /dev/null +++ b/libraries/rush-client-core/src/test/UnreapedChildProcess.ts @@ -0,0 +1,26 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; + +/** + * Runs `callbackAsync` with the PID of a process that exits after 200 ms but stays unreaped, because its parent + * never waits for it, and the parent's PID. POSIX only. + */ +export async function withUnreapedChildAsync( + callbackAsync: (pid: number, parentPid: number) => Promise +): Promise { + // The shell starts a short-lived child, then becomes a process that never reaps it. + const parent: ChildProcess = spawn('/bin/sh', ['-c', 'sleep 0.2 & echo $!; exec sleep 10'], { + stdio: ['ignore', 'pipe', 'ignore'] + }); + const closed: Promise = once(parent, 'close'); + try { + const [output] = await once(parent.stdout!, 'data'); + await callbackAsync(Number(String(output).trim()), parent.pid!); + } finally { + parent.kill(); + await closed; + } +} diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index 027d2e22bf..8d7d89b397 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import { spawn, type ChildProcess } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; import { once } from 'node:events'; import * as fs from 'node:fs'; import * as net from 'node:net'; @@ -10,19 +11,28 @@ import * as path from 'node:path'; import { setTimeout as delayAsync } from 'node:timers/promises'; import { FileSystem } from '@rushstack/node-core-library'; +import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; import { readDaemonLockfile, type IDaemonLockfile, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; -import { DaemonClient } from '../DaemonClient'; +import { DaemonClient, type DaemonClientOutcome } from '../DaemonClient'; +import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from '../DaemonClientError'; import { captureDaemonRequest } from '../captureDaemonRequest'; import { getDaemonLogFilePath } from '../DaemonLogFile'; import { resetDaemonArtifactsAsync } from '../DaemonOwnership'; -import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from '../connectOrStartDaemon'; +import { + connectOrStartDaemonAsync, + requestDaemonShutdownAsync, + resolveDaemonStartupReservationAsync, + type IConnectOrStartDaemonOptions +} from '../connectOrStartDaemon'; import { executeWithDaemonRestartAsync } from '../executeWithDaemonRestart'; -import { getDaemonStartupFilePath } from '../DaemonStartup'; +import { getDaemonStartupFilePath, releaseDaemonStartup, reserveDaemonStartup } from '../DaemonStartup'; +import { inspectDaemonStartupReservation } from '../DaemonStartupReservation'; +import { tryAcquireStartupLockAsync, type IStartupLock } from '../StartupLock'; import { removeTestFolderAsync, waitForTestProcessExitAsync } from './TestProcessExit'; describe('detached daemon startup', () => { @@ -134,12 +144,36 @@ describe('detached daemon startup', () => { return daemonPid; } + async function getExitedPidAsync(): Promise { + const exited: ChildProcess = spawn(process.execPath, ['-e', ''], { stdio: 'ignore' }); + await once(exited, 'close'); + return exited.pid!; + } + + function writeReservation(helperPid: number, helperStartedAt: string = new Date().toISOString()): string { + const contents: string = JSON.stringify({ token: randomUUID(), helperPid, helperStartedAt }); + fs.writeFileSync(getDaemonStartupFilePath(paths), contents); + return contents; + } + + async function startFixtureDaemonAsync(): Promise { + const client: DaemonClient = await connectOrStartDaemonAsync(options); + const { pid } = await client.status; + await client.closeAsync(); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + return pid!; + } + it('fails closed for successor starters while the original detached daemon remains pre-bind', async () => { const daemonPid: number = await killStarterBeforeBindAsync(); const results = await Promise.all( Array.from({ length: 4 }, () => startClient({ ...options, startupTimeoutMs: 700 }).result) ); expect(results.every(({ code }) => code !== 0)).toBe(true); + // The helper is alive, so the client that holds the start lock waits for it until its own deadline. + expect( + results.some(({ stderr }) => stderr.includes('is still waiting for the daemon to become ready')) + ).toBe(true); expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n`); expect(fs.existsSync(paths.lockfilePath)).toBe(false); @@ -214,6 +248,218 @@ describe('detached daemon startup', () => { expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); }); + it('refuses another launch at once when the startup helper exited without releasing its reservation', async () => { + const startupPath: string = getDaemonStartupFilePath(paths); + const failing: IConnectOrStartDaemonOptions = { + ...options, + startCommand: { ...options.startCommand!, args: [path.join(folder, 'missing-entry.js')] } + }; + await expect(connectOrStartDaemonAsync(failing)).rejects.toThrow('Unable to start'); + const contents: string = fs.readFileSync(startupPath, 'utf8'); + const { helperPid } = JSON.parse(contents); + expect(inspectDaemonStartupReservation(paths)).toEqual({ + path: startupPath, + helperPid, + helperState: 'exited' + }); + const started: number = Date.now(); + const error: Error = await connectOrStartDaemonAsync({ ...options, startupTimeoutMs: 20000 }).then( + () => new Error('Expected startup to be refused.'), + (refusal: Error) => refusal + ); + expect(Date.now() - started).toBeLessThan(3000); + expect(error.message).toContain( + `unresolved startup handoff at ${startupPath}: its startup helper (PID ${helperPid}) exited before the daemon became ready; refusing another launch.` + ); + expect(error.message).toContain('daemon stop --force'); + expect(error.message).not.toContain('..'); + expect(fs.readFileSync(startupPath, 'utf8')).toBe(contents); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }); + + it.each(['legacy', 'exited helper', 'no auto-start'])( + 'uses and resolves a ready daemon next to a retained startup reservation (%s)', + async (kind) => { + const daemonPid: number = await startFixtureDaemonAsync(); + if (kind === 'legacy') { + fs.writeFileSync(getDaemonStartupFilePath(paths), randomUUID()); + } else { + writeReservation(await getExitedPidAsync()); + } + const started: number = Date.now(); + const client: DaemonClient = await connectOrStartDaemonAsync( + kind === 'no auto-start' ? { paths, expectedDaemonVersion: 'fixture' } : options + ); + try { + expect((await client.status).pid).toBe(daemonPid); + } finally { + await client.closeAsync(); + } + expect(Date.now() - started).toBeLessThan(3000); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n`); + } + ); + + it('resolves a retained startup reservation before replacing a mismatched ready daemon', async () => { + const daemonPid: number = await startFixtureDaemonAsync(); + fs.writeFileSync(getDaemonStartupFilePath(paths), randomUUID()); + const replacement: DaemonClient = await connectOrStartDaemonAsync({ + ...options, + expectedDaemonVersion: 'replacement', + startCommand: { ...options.startCommand!, args: [...options.startCommand!.args, 'replacement'] } + }); + try { + const status = await replacement.status; + expect(status.daemonVersion).toBe('replacement'); + expect(status.pid).not.toBe(daemonPid); + } finally { + await replacement.closeAsync(); + } + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(2); + }); + + it('keeps a startup reservation next to a ready daemon that is not the attested owner', async () => { + await startFixtureDaemonAsync(); + const owner: string = fs.readFileSync(paths.lockfilePath, 'utf8'); + fs.writeFileSync(paths.lockfilePath, JSON.stringify({ ...JSON.parse(owner), pid: process.pid })); + const contents: string = writeReservation(await getExitedPidAsync()); + try { + await expect(connectOrStartDaemonAsync({ ...options, startupTimeoutMs: 2000 })).rejects.toThrow( + 'unresolved startup handoff' + ); + await expect(connectOrStartDaemonAsync({ paths, expectedDaemonVersion: 'fixture' })).rejects.toThrow( + 'auto-start is disabled' + ); + expect(fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8')).toBe(contents); + } finally { + fs.writeFileSync(paths.lockfilePath, owner); + } + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + }); + + it('resolves a retained startup reservation before shutdown so that a successor can start', async () => { + const daemonPid: number = await startFixtureDaemonAsync(); + writeReservation(await getExitedPidAsync()); + const running: DaemonClient = await DaemonClient.connectAsync({ socketPath: paths.socketPath }); + const previousDaemon = await requestDaemonShutdownAsync(running, paths).finally(() => + running.closeAsync() + ); + expect(previousDaemon.pid).toBe(daemonPid); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + const successor: DaemonClient = await connectOrStartDaemonAsync({ ...options, previousDaemon }); + try { + expect((await successor.status).pid).not.toBe(daemonPid); + } finally { + await successor.closeAsync(); + } + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(2); + }); + + it('does not request shutdown while the start lock keeps a startup reservation unresolved', async () => { + const daemonPid: number = await startFixtureDaemonAsync(); + const contents: string = writeReservation(await getExitedPidAsync()); + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + expect(lock).toBeDefined(); + const running: DaemonClient = await DaemonClient.connectAsync({ socketPath: paths.socketPath }); + try { + await expect(requestDaemonShutdownAsync(running, paths, 300)).rejects.toThrow('shutdown was not sent'); + } finally { + await running.closeAsync(); + await lock!.releaseAsync(); + } + expect(fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8')).toBe(contents); + const client: DaemonClient = await connectOrStartDaemonAsync(options); + try { + expect((await client.status).pid).toBe(daemonPid); + } finally { + await client.closeAsync(); + } + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + }); + + it('resolves a startup reservation before a plain stop only for the attested ready daemon', async () => { + const daemonPid: number = await startFixtureDaemonAsync(); + const running: DaemonClient = await DaemonClient.connectAsync({ socketPath: paths.socketPath }); + try { + await expect(resolveDaemonStartupReservationAsync(running, paths)).resolves.toBe(true); + const contents: string = writeReservation(await getExitedPidAsync()); + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + expect(lock).toBeDefined(); + try { + await expect(resolveDaemonStartupReservationAsync(running, paths, 300)).resolves.toBe(false); + } finally { + await lock!.releaseAsync(); + } + const owner: string = fs.readFileSync(paths.lockfilePath, 'utf8'); + fs.writeFileSync(paths.lockfilePath, JSON.stringify({ ...JSON.parse(owner), pid: process.pid })); + try { + await expect(resolveDaemonStartupReservationAsync(running, paths)).resolves.toBe(false); + } finally { + fs.writeFileSync(paths.lockfilePath, owner); + } + expect(fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8')).toBe(contents); + await expect(resolveDaemonStartupReservationAsync(running, paths)).resolves.toBe(true); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + await running.shutdownAsync(); + } finally { + await running.closeAsync(); + } + // With the reservation resolved, a later start launches a new daemon instead of being refused. + await waitForTestProcessExitAsync(daemonPid); + const successor: DaemonClient = await connectOrStartDaemonAsync(options); + try { + expect((await successor.status).pid).not.toBe(daemonPid); + } finally { + await successor.closeAsync(); + } + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(2); + }); + + it('reports a startup reservation and whether its helper can still release it', async () => { + const startupPath: string = getDaemonStartupFilePath(paths); + expect(inspectDaemonStartupReservation(paths)).toBeUndefined(); + fs.writeFileSync(startupPath, randomUUID()); + expect(inspectDaemonStartupReservation(paths)).toEqual({ path: startupPath, helperState: 'unknown' }); + writeReservation(process.pid); + expect(inspectDaemonStartupReservation(paths)).toEqual({ + path: startupPath, + helperPid: process.pid, + helperState: 'running' + }); + const exitedPid: number = await getExitedPidAsync(); + writeReservation(exitedPid); + expect(inspectDaemonStartupReservation(paths)).toEqual({ + path: startupPath, + helperPid: exitedPid, + helperState: 'exited' + }); + if (process.platform === 'linux') { + // This process started after the recorded helper, so it merely reuses the PID. + writeReservation(process.pid, new Date(Date.now() - 3600000).toISOString()); + expect(inspectDaemonStartupReservation(paths)).toMatchObject({ helperState: 'exited' }); + } + fs.unlinkSync(startupPath); + fs.mkdirSync(startupPath); + expect(inspectDaemonStartupReservation(paths)).toEqual({ path: startupPath, helperState: 'unknown' }); + fs.rmdirSync(startupPath); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }); + + it('lets a startup helper release only its own reservation, tolerating one already resolved', () => { + const startupPath: string = getDaemonStartupFilePath(paths); + const helper = { pid: process.pid, startedAt: new Date().toISOString() }; + const token: string = reserveDaemonStartup(paths, helper); + expect(() => reserveDaemonStartup(paths, helper)).toThrow('EEXIST'); + expect(() => releaseDaemonStartup(paths, randomUUID())).toThrow('changed ownership'); + expect(fs.existsSync(startupPath)).toBe(true); + releaseDaemonStartup(paths, token); + expect(fs.existsSync(startupPath)).toBe(false); + releaseDaemonStartup(paths, token); + expect(fs.existsSync(startupPath)).toBe(false); + }); + it.each([false, true])( 'preserves an explicit launcher and environment (relative cwd: %s)', async (relative) => { @@ -623,6 +869,122 @@ describe('detached daemon startup', () => { } }); + describe('when the connection is lost before the result', () => { + const exitedMessage = (pid: number, logged: boolean, client: string = 'rush-client'): string => + `${DAEMON_DISCONNECTED_MESSAGE} rushd (PID ${pid}) exited while it ran the command; ` + + `"rush-client daemon logs" ${logged ? 'shows' : 'may show'} why. Run the command again; ` + + `if the daemon exits again, run the command with "${client} --no-daemon".`; + + function withMode(mode: string): IConnectOrStartDaemonOptions { + return { + ...options, + startCommand: { ...options.startCommand!, args: [...options.startCommand!.args, 'fixture', mode] } + }; + } + + function captureRequest(invocationKind?: 'rushx'): IDaemonRequestEnvelope { + return captureDaemonRequest({ + argv: ['test'], + commandName: 'test', + commandOrigin: 'custom', + cwd: folder, + environment: {}, + terminal: { isTTY: false, supportsColor: false }, + invocationKind + }); + } + + async function waitForRequestAsync(): Promise { + const deadline: number = Date.now() + 5000; + while (!fs.existsSync(path.join(folder, 'requests')) && Date.now() < deadline) await delayAsync(20); + expect(fs.existsSync(path.join(folder, 'requests'))).toBe(true); + } + + it('says that rushd exited and quotes the error it logged, without retrying', async () => { + const connection: IConnectOrStartDaemonOptions = withMode('crash-on-request'); + const client: DaemonClient = await connectOrStartDaemonAsync(connection); + const { pid } = await client.status; + const error: unknown = await executeWithDaemonRestartAsync(client, connection, { + request: captureRequest() + }).catch((failure: unknown) => failure); + expect(error).toBeInstanceOf(DaemonClientError); + expect(error).toMatchObject({ + code: 'disconnected', + message: + `${exitedMessage(pid!, true)}\n` + + 'The daemon log reports: Error: fixture daemon crash while running the request' + }); + expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8')).toBe('fixture\n'); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${pid}\n`); + }); + + it('ignores log output from before the request and names the rushx client', async () => { + const earlierCrash: string = [ + '/earlier/daemon.js:1', + "throw new Error('an earlier crash');", + '^', + '', + 'Error: an earlier crash', + ' at /earlier/daemon.js:1:7', + '', + 'Node.js v22.0.0', + '' + ].join('\n'); + fs.writeFileSync(getDaemonLogFilePath(paths), earlierCrash); + const connection: IConnectOrStartDaemonOptions = withMode('kill-on-request'); + const client: DaemonClient = await connectOrStartDaemonAsync(connection); + const { pid } = await client.status; + await expect( + executeWithDaemonRestartAsync(client, connection, { request: captureRequest('rushx') }) + ).rejects.toMatchObject({ code: 'disconnected', message: exitedMessage(pid!, false, 'rushx-client') }); + expect(fs.readFileSync(getDaemonLogFilePath(paths), 'utf8')).toContain(earlierCrash); + }); + + it('says the connection closed while rushd still runs', async () => { + const connection: IConnectOrStartDaemonOptions = withMode('close-on-request'); + const client: DaemonClient = await connectOrStartDaemonAsync(connection); + const { pid } = await client.status; + await expect( + executeWithDaemonRestartAsync(client, connection, { request: captureRequest() }) + ).rejects.toMatchObject({ + code: 'disconnected', + message: `${DAEMON_DISCONNECTED_MESSAGE} The connection to rushd (PID ${pid}) closed, but the daemon is still running; run the command again.` + }); + expect(fs.existsSync(path.join(folder, `stopped-${pid}`))).toBe(false); + }); + + it('keeps the plain text after the request was cancelled', async () => { + const connection: IConnectOrStartDaemonOptions = withMode('crash-on-cancel'); + const client: DaemonClient = await connectOrStartDaemonAsync(connection); + const abort: AbortController = new AbortController(); + const pending: Promise = executeWithDaemonRestartAsync(client, connection, { + request: captureRequest(), + abortSignal: abort.signal + }); + await waitForRequestAsync(); + abort.abort(); + await expect(pending).rejects.toMatchObject({ + code: 'disconnected', + message: DAEMON_DISCONNECTED_MESSAGE + }); + }); + + it('keeps the aborted result that an orderly shutdown sends', async () => { + const connection: IConnectOrStartDaemonOptions = withMode('hold-until-shutdown'); + const client: DaemonClient = await connectOrStartDaemonAsync(connection); + const pending: Promise = executeWithDaemonRestartAsync(client, connection, { + request: captureRequest() + }); + await waitForRequestAsync(); + const stopping: DaemonClient = await connectOrStartDaemonAsync(connection); + await stopping.shutdownAsync(); + expect(await pending).toMatchObject({ + kind: 'result', + result: { exitCode: 130, outcome: 'aborted', aborted: true } + }); + }); + }); + it('waits through published ownership handoff even without a captured predecessor', async () => { const running = await connectOrStartDaemonAsync({ ...options, @@ -717,6 +1079,38 @@ describe('detached daemon startup', () => { expect(fs.readFileSync(logFilePath, 'utf8')).toContain('Cannot find module'); }); + it('tells the daemon it starts which runtime folder its clients look in', async () => { + const client = await connectOrStartDaemonAsync(options); + await client.closeAsync(); + expect(fs.readFileSync(path.join(folder, 'runtime-base'), 'utf8')).toBe(path.dirname(folder)); + }); + + (process.platform === 'win32' ? it.skip : it)( + 'reports a linked runtime folder as a startup failure without starting a daemon', + async () => { + const link: string = `${folder}-link`; + fs.symlinkSync(folder, link); + try { + const failure: Promise = connectOrStartDaemonAsync({ + ...options, + paths: { + runtimeDir: link, + socketPath: path.join(link, 'd.sock'), + lockfilePath: path.join(link, 'daemon.pid.json') + } + }); + await expect(failure).rejects.toBeInstanceOf(DaemonClientError); + await expect(failure).rejects.toMatchObject({ code: 'startupFailed' }); + await expect(failure).rejects.toThrow( + `The daemon runtime folder ${link} is unsafe: it is a symbolic link` + ); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + } finally { + fs.unlinkSync(link); + } + } + ); + (process.platform === 'win32' ? it.skip : it)( 'refuses linked log destinations without changing their target', async () => { diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index 5d04da0f96..6256888197 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -20,11 +20,14 @@ async function mainAsync(): Promise { const paths: IDaemonPaths = JSON.parse(process.argv[2]); const folder: string = path.dirname(paths.lockfilePath); const daemonVersion: string = process.argv[3] ?? 'fixture'; - const restartMode: string | undefined = process.argv[4]; + const mode: string | undefined = process.argv[4]; + const restartMode: string | undefined = mode?.startsWith('restart-') ? mode : undefined; const connections: Set = new Set(); let closing: Promise | undefined; + let heldRequest: { connection: DaemonFrameConnection; requestId: string } | undefined; fs.appendFileSync(path.join(folder, 'starts'), `${process.pid}\n`); fs.appendFileSync(path.join(folder, 'parents'), `${process.ppid}\n`); + fs.writeFileSync(path.join(folder, 'runtime-base'), process.env.RUSHD_RUNTIME_DIR ?? '(unset)'); process.stdout.write('launcher stdout\n'); process.stderr.write('launcher stderr\n'); if (fs.existsSync(path.join(folder, 'hold-prebind'))) { @@ -65,6 +68,18 @@ async function mainAsync(): Promise { }); } else if (message.kind === 'requestStart') { fs.appendFileSync(path.join(folder, 'requests'), `${daemonVersion}\n`); + if (mode === 'crash-on-request' || mode === 'kill-on-request') { + exitAbruptly(); + return; + } + if (mode === 'close-on-request') { + await connection.closeAsync(); + return; + } + if (mode === 'hold-until-shutdown' || mode === 'crash-on-cancel') { + heldRequest = { connection, requestId: message.payload.requestId }; + return; + } if (message.payload.admission?.waitTimeoutMs !== undefined) { fs.appendFileSync(path.join(folder, 'waits'), `${message.payload.admission.waitTimeoutMs}\n`); } @@ -92,7 +107,24 @@ async function mainAsync(): Promise { fs.appendFileSync(path.join(folder, 'restarted'), 'r'); await stopAsync(); } + } else if (message.kind === 'requestCancel' && mode === 'crash-on-cancel') { + exitAbruptly(); } else if (message.kind === 'shutdown') { + if (heldRequest) { + // Like rushd, an orderly shutdown ends a running request with an aborted result first. + await heldRequest.connection.sendFrameAsync({ + kind: DaemonFrameType.controlJson, + payload: encodeDaemonControlMessage({ + kind: 'requestResult', + payload: { + requestId: heldRequest.requestId, + exitCode: 130, + outcome: 'aborted', + aborted: true + } + }) + }); + } await connection.sendFrameAsync({ kind: DaemonFrameType.controlJson, payload: encodeDaemonControlMessage({ kind: 'shutdownAck', payload: {} }) @@ -116,6 +148,15 @@ async function mainAsync(): Promise { return closing; } + /** Exits like a crashed daemon, without cleanup. The test expects this exit, so it also counts as stopped. */ + function exitAbruptly(): void { + fs.writeFileSync(path.join(folder, `stopped-${process.pid}`), ''); + if (mode === 'kill-on-request') process.kill(process.pid, 'SIGKILL'); + setImmediate(() => { + throw new Error('fixture daemon crash\nwhile running the request'); + }); + } + async function closeOnceAsync(): Promise { clearInterval(timer); const stopped: Promise = listener.stopAcceptingAsync(); diff --git a/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts b/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts index b35d1d6883..37a661c978 100644 --- a/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts +++ b/libraries/rush-daemon-protocol/src/DaemonProtocolVersion.ts @@ -28,6 +28,13 @@ export const DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR: number = 10; /** The first additive protocol minor whose shutdown acknowledgement reports the active request count. @beta */ export const DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR: number = 11; +/** + * The first protocol minor whose clients and daemons meet in `/tmp/rushd-` (or in the folder that + * `RUSHD_RUNTIME_DIR` names) rather than in one chosen by `XDG_RUNTIME_DIR` or `TMPDIR`, and whose workspace + * identity ignores temporary and runtime folder variables. @beta + */ +export const DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR: number = 12; + /** * A rushd wire protocol version. * @@ -61,7 +68,7 @@ export interface IDaemonProtocolVersion { */ export const DAEMON_PROTOCOL_VERSION: IDaemonProtocolVersion = { major: 0, - minor: DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR + minor: DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR }; /** diff --git a/libraries/rush-daemon-protocol/src/index.ts b/libraries/rush-daemon-protocol/src/index.ts index 1537163377..575a87e8e3 100644 --- a/libraries/rush-daemon-protocol/src/index.ts +++ b/libraries/rush-daemon-protocol/src/index.ts @@ -29,10 +29,10 @@ export { DAEMON_INVOCATION_KIND_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_LIFECYCLE_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_REQUEST_ADMISSION_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_REQUEST_LIFECYCLE_PROTOCOL_MINOR, DAEMON_PROTOCOL_VERSION } from './DaemonProtocolVersion'; +export { DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_SHUTDOWN_ACTIVE_REQUESTS_PROTOCOL_MINOR } from './DaemonProtocolVersion'; export { DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR } from './DaemonProtocolVersion'; -export { isDaemonProtocolCompatible } from './DaemonProtocolVersion'; -export type { IDaemonProtocolVersion } from './DaemonProtocolVersion'; +export { isDaemonProtocolCompatible, type IDaemonProtocolVersion } from './DaemonProtocolVersion'; export type { IDaemonClientCaps } from './DaemonClientCaps'; export { DAEMON_CONTROL_MESSAGE_KINDS, isDaemonControlMessageKind } from './DaemonControlKinds'; export type { DaemonControlMessageKind } from './DaemonControlKinds'; diff --git a/libraries/rush-daemon-transport/README.md b/libraries/rush-daemon-transport/README.md index 4abb9d30ef..f0322bf8cf 100644 --- a/libraries/rush-daemon-transport/README.md +++ b/libraries/rush-daemon-transport/README.md @@ -7,8 +7,13 @@ The workspace-keyed socket/pipe **transport** for the Rush daemon (`rushd`): - **Workspace keys** — `sha256(canonicalRepoRoot + rushVersion + startupOptions)`, so distinct workspaces, Rush versions, or startup options resolve to distinct daemon endpoints while the same workspace stays stable across runs. -- **Per-user path derivation** — `$XDG_RUNTIME_DIR`-aware Unix domain sockets on POSIX and - `\\.\pipe\rushd-` named pipes on Windows. +- **Per-user path derivation** — Unix domain sockets in `/tmp/rushd-/` on POSIX, or in + `$RUSHD_RUNTIME_DIR/rushd-/` when that variable is an absolute path, and + `\\.\pipe\rushd-` named pipes on Windows. `TMPDIR` and `XDG_RUNTIME_DIR` are not + consulted, because they differ between the shells, jobs and services of one user. A daemon + resolves its paths with the same rule, and a client that starts one passes the folder it + chose as `RUSHD_RUNTIME_DIR`. The folder must be a directory (not a symbolic link) that the + user owns; one that others can open is made owner-only (`0700`). - **`net` listener and connector** — framed with [`@rushstack/rush-daemon-protocol`](https://www.npmjs.com/package/@rushstack/rush-daemon-protocol), with backpressure-aware writes and serialized async frame handlers for inbound flow control. @@ -18,6 +23,10 @@ The workspace-keyed socket/pipe **transport** for the Rush daemon (`rushd`): retaining ownership. Hosts release the endpoint with `closeAsync()` after their resources have finished disposing. A live owner prevents rebinding even when its socket has already closed; repeated closes cannot remove a successor's endpoint. +- **Owner-safe publication** — a listener binds a private name and hard-links it to the socket + path, so it never replaces a live peer's socket; the runtime folder's file system must support + hard links. On close it removes the socket and ownership record only while they are still the + files it created, so a predecessor that shuts down late never removes a successor's endpoint. Part of the Rush 6 / rushd re-architecture: [microsoft/rushstack#5894](https://github.com/microsoft/rushstack/issues/5894). diff --git a/libraries/rush-daemon-transport/src/DaemonFileIdentity.ts b/libraries/rush-daemon-transport/src/DaemonFileIdentity.ts new file mode 100644 index 0000000000..2d2c896b67 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonFileIdentity.ts @@ -0,0 +1,60 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +const WINDOWS_PLATFORM: NodeJS.Platform = 'win32'; +const READ_ONLY: string = 'r'; + +/** The device and inode of a file that this process created. */ +export interface IDaemonFileIdentity { + readonly dev: number; + readonly ino: number; + /** An open descriptor that keeps the inode in use, so no later file can get its number (POSIX only). */ + readonly fd?: number; +} + +/** The identity of the file (not a link target) at `filePath`, or `undefined` when nothing is there. */ +export function readFileIdentity(filePath: string): IDaemonFileIdentity | undefined { + const stats: fs.Stats | undefined = fs.lstatSync(filePath, { throwIfNoEntry: false }); + return stats && { dev: stats.dev, ino: stats.ino }; +} + +/** + * Returns the identity of the file that this process just wrote at `filePath`, and holds it open on POSIX. + * + * @remarks + * XFS and ext4 give a deleted file's inode number to the next file created, which could then pass for this one + * in {@link removeOwnFile}. The open descriptor keeps the inode in use until {@link removeOwnFile} releases it. + * An NTFS file id includes a sequence number that changes on reuse, so Windows needs no descriptor. + */ +export function pinFileIdentity(filePath: string): IDaemonFileIdentity { + const fd: number = fs.openSync(filePath, READ_ONLY); + const { dev, ino } = fs.fstatSync(fd); + if (process.platform !== WINDOWS_PLATFORM) return { dev, ino, fd }; + fs.closeSync(fd); + return { dev, ino }; +} + +function isSameFile(current: IDaemonFileIdentity | undefined, expected: IDaemonFileIdentity): boolean { + if (!current) return false; + return current.dev === expected.dev && current.ino === expected.ino; +} + +function release(identity: IDaemonFileIdentity): void { + if (identity.fd !== undefined) fs.closeSync(identity.fd); +} + +/** + * Deletes `filePath` only while it is still the file this process created, then releases `identity`. Call it + * once for each identity. + * + * @remarks + * Once a daemon's socket or lockfile has been replaced, for example after someone deleted it and a successor + * started, the name belongs to the successor, which must stay reachable when this daemon exits. + */ +export function removeOwnFile(filePath: string, identity: IDaemonFileIdentity | undefined): void { + if (!identity) return; + if (isSameFile(readFileIdentity(filePath), identity)) fs.rmSync(filePath, { force: true }); + release(identity); +} diff --git a/libraries/rush-daemon-transport/src/DaemonListener.ts b/libraries/rush-daemon-transport/src/DaemonListener.ts index 083ba3ae50..6a8e18b76c 100644 --- a/libraries/rush-daemon-transport/src/DaemonListener.ts +++ b/libraries/rush-daemon-transport/src/DaemonListener.ts @@ -5,14 +5,18 @@ import * as net from 'node:net'; import type { IDaemonProtocolVersion } from '@rushstack/rush-daemon-protocol'; +import { pinFileIdentity } from './DaemonFileIdentity'; +import type { IDaemonFileIdentity } from './DaemonFileIdentity'; import { DaemonFrameConnection } from './DaemonFrameConnection'; import { listenWithReclaimAsync } from './DaemonListenerBinding'; import { DaemonListenerLifetime } from './DaemonListenerLifetime'; -import { ensureDaemonRuntimeDir, writeDaemonLockfile } from './DaemonLockfile'; +import type { IDaemonListenerFiles } from './DaemonListenerLifetime'; +import { writeDaemonLockfile } from './DaemonLockfile'; import { startOperationGroupRecording } from './DaemonOperationGroupRecorder'; import { getOperationGroupsFolder } from './DaemonOperationGroups'; import { assertDaemonOwnershipAvailable } from './DaemonOwnership'; import type { IDaemonPaths } from './DaemonPaths'; +import { ensureDaemonRuntimeDir } from './DaemonRuntimeDir'; /** Options for {@link DaemonFrameListener.listenAsync}. @beta */ export interface IDaemonListenerOptions { @@ -32,11 +36,11 @@ export interface IDaemonListenerOptions { * @beta */ export class DaemonFrameListener { readonly #lifetime: DaemonListenerLifetime; - private constructor(server: net.Server, paths: IDaemonPaths) { + private constructor(server: net.Server, paths: IDaemonPaths, files: IDaemonListenerFiles) { // Record detached operation groups for as long as this process owns the lockfile, so a successor can // reap them if this daemon dies uncleanly. const folder: string = getOperationGroupsFolder(paths.lockfilePath, process.pid); - this.#lifetime = new DaemonListenerLifetime(server, paths, startOperationGroupRecording(folder)); + this.#lifetime = new DaemonListenerLifetime(server, paths, files, startOperationGroupRecording(folder)); } /** Binds the socket/pipe path and writes the PID lockfile. */ public static async listenAsync( @@ -48,19 +52,20 @@ export class DaemonFrameListener { options.onConnection(new DaemonFrameConnection(socket)); }); ensureDaemonRuntimeDir(paths); - await listenWithReclaimAsync(server, paths); + const socket: IDaemonFileIdentity | undefined = await listenWithReclaimAsync(server, paths); // Lockfile after bind: a pre-existing stale record must read as dead, not // as a live owner that would make reclaim refuse. + let lockfile: IDaemonFileIdentity | undefined; try { - writeListenerLockfile(paths, options); + lockfile = writeListenerLockfile(paths, options); } catch (error) { - await new DaemonListenerLifetime(server, paths).stopAcceptingAsync(); + await new DaemonListenerLifetime(server, paths, { socket }).stopAcceptingAsync(); throw error; } - return new DaemonFrameListener(server, paths); + return new DaemonFrameListener(server, paths, { socket, lockfile }); } - /** Stops accepting connections and releases the socket/pipe and lockfile. */ + /** Stops accepting connections and releases the socket/pipe and lockfile, unless a successor replaced them. */ public closeAsync(): Promise { return this.#lifetime.closeAsync(); } @@ -70,11 +75,12 @@ export class DaemonFrameListener { } } -function writeListenerLockfile(paths: IDaemonPaths, options: IDaemonListenerOptions): void { +function writeListenerLockfile(paths: IDaemonPaths, options: IDaemonListenerOptions): IDaemonFileIdentity { writeDaemonLockfile(paths.lockfilePath, { pid: process.pid, protocolVersion: options.protocolVersion, startedAt: options.startedAt ?? new Date().toISOString(), socketPath: paths.socketPath }); + return pinFileIdentity(paths.lockfilePath); } diff --git a/libraries/rush-daemon-transport/src/DaemonListenerBinding.ts b/libraries/rush-daemon-transport/src/DaemonListenerBinding.ts index 77885fa935..a10afacb56 100644 --- a/libraries/rush-daemon-transport/src/DaemonListenerBinding.ts +++ b/libraries/rush-daemon-transport/src/DaemonListenerBinding.ts @@ -3,15 +3,31 @@ import type * as net from 'node:net'; +import type { IDaemonFileIdentity } from './DaemonFileIdentity'; import { ADDRESS_IN_USE, listenOrErrorAsync, toListenTransportError } from './DaemonListenerNet'; import type { INetError } from './DaemonListenerNet'; import type { IDaemonPaths } from './DaemonPaths'; import { reclaimStaleDaemonAsync } from './DaemonReclaim'; +import { listenPublishedAsync } from './DaemonSocketPublication'; const FIRST_ATTEMPT: number = 0; const RECLAIM_ATTEMPT: number = 1; +const WINDOWS_PLATFORM: NodeJS.Platform = 'win32'; -export async function listenWithReclaimAsync(server: net.Server, paths: IDaemonPaths): Promise { +/** + * Binds the listener, reclaiming its path once from a dead daemon. Returns the identity of a POSIX socket; + * a Windows named pipe has no file and disappears with its server. + */ +export async function listenWithReclaimAsync( + server: net.Server, + paths: IDaemonPaths +): Promise { + if (process.platform !== WINDOWS_PLATFORM) return listenPublishedAsync(server, paths); + await listenPipeWithReclaimAsync(server, paths); + return undefined; +} + +async function listenPipeWithReclaimAsync(server: net.Server, paths: IDaemonPaths): Promise { for (let attempt: number = FIRST_ATTEMPT; attempt <= RECLAIM_ATTEMPT; attempt++) { const bound: boolean = await tryListenOnceAsync(server, paths, attempt); if (bound) { diff --git a/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts b/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts index 90f79a48b6..a7a81b2493 100644 --- a/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts +++ b/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts @@ -3,15 +3,23 @@ import type * as net from 'node:net'; -import { removeDaemonArtifacts } from './DaemonLockfile'; +import { removeOwnFile } from './DaemonFileIdentity'; +import type { IDaemonFileIdentity } from './DaemonFileIdentity'; import type { StopOperationGroupRecording } from './DaemonOperationGroupRecorder'; import type { IDaemonPaths } from './DaemonPaths'; +/** The files a listener created: its published POSIX socket and its lockfile. */ +export interface IDaemonListenerFiles { + readonly socket?: IDaemonFileIdentity; + readonly lockfile?: IDaemonFileIdentity; +} + function keepNoRecords(): void { // Nothing was recorded before the lockfile was written. } export class DaemonListenerLifetime { + readonly #files: IDaemonListenerFiles; readonly #paths: IDaemonPaths; readonly #server: net.Server; readonly #stopRecording: StopOperationGroupRecording; @@ -21,15 +29,17 @@ export class DaemonListenerLifetime { public constructor( server: net.Server, paths: IDaemonPaths, + files: IDaemonListenerFiles, stopRecording: StopOperationGroupRecording = keepNoRecords ) { this.#server = server; this.#paths = paths; + this.#files = files; this.#stopRecording = stopRecording; } public stopAcceptingAsync(): Promise { - this.#stopPromise ??= new Promise((resolve) => this.#server.close(() => resolve())); + this.#stopPromise ??= this.#stopOnceAsync(); return this.#stopPromise; } @@ -38,9 +48,16 @@ export class DaemonListenerLifetime { return this.#closePromise; } + #stopOnceAsync(): Promise { + // Unlink before closing, as libuv does for the path it bound, but only this listener's own socket: the + // name may belong to a successor by now. + removeOwnFile(this.#paths.socketPath, this.#files.socket); + return new Promise((resolve: () => void) => this.#server.close(() => resolve())); + } + async #closeOnceAsync(): Promise { await this.stopAcceptingAsync(); this.#stopRecording(); - removeDaemonArtifacts(this.#paths.lockfilePath, this.#paths.socketPath); + removeOwnFile(this.#paths.lockfilePath, this.#files.lockfile); } } diff --git a/libraries/rush-daemon-transport/src/DaemonLockfile.ts b/libraries/rush-daemon-transport/src/DaemonLockfile.ts index f81fbd1ef7..a8db247a39 100644 --- a/libraries/rush-daemon-transport/src/DaemonLockfile.ts +++ b/libraries/rush-daemon-transport/src/DaemonLockfile.ts @@ -6,8 +6,6 @@ import * as path from 'node:path'; import type { IDaemonProtocolVersion } from '@rushstack/rush-daemon-protocol'; -import type { IDaemonPaths } from './DaemonPaths'; - const UTF8: BufferEncoding = 'utf8'; const NO_SIGNAL: number = 0; const DIR_MODE: number = 0o700; @@ -25,14 +23,6 @@ export interface IDaemonLockfile { readonly socketPath: string; } -/** Creates the per-user runtime directory (mode `0700`) when the platform has one. - * Must be called before binding a POSIX socket inside it. @beta */ -export function ensureDaemonRuntimeDir(paths: IDaemonPaths): void { - if (paths.runtimeDir !== undefined) { - fs.mkdirSync(paths.runtimeDir, { recursive: true, mode: DIR_MODE }); - } -} - /** Returns `true` when a process with `pid` exists and is signalable. @beta */ export function isDaemonProcessAlive(pid: number): boolean { try { @@ -68,7 +58,9 @@ export function writeDaemonLockfile(lockfilePath: string, lockfile: IDaemonLockf fs.writeFileSync(lockfilePath, JSON.stringify(lockfile), { encoding: UTF8, mode: FILE_MODE }); } -/** Removes the daemon lockfile and (on POSIX) the stale socket file; idempotent. @beta */ +/** Removes the daemon lockfile and (on POSIX) the stale socket file; idempotent. + * Only for a dead owner's files: a daemon removes its own with an identity check, since a successor may + * already have replaced them. @beta */ export function removeDaemonArtifacts(lockfilePath: string, socketPath: string): void { for (const filePath of [lockfilePath, socketPath]) { try { diff --git a/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts b/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts index f1d4a45068..98fbb7c589 100644 --- a/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts +++ b/libraries/rush-daemon-transport/src/DaemonOperationGroupReaper.ts @@ -9,12 +9,14 @@ import { removeOperationGroupRecords } from './DaemonOperationGroups'; import type { IOperationGroupRecord } from './DaemonOperationGroups'; +import { isOwnedEntry } from './DaemonOwnedEntry'; import type { IProcessStat } from './DaemonProcessStat'; import { createReapContext, isSignalableGroup } from './DaemonReapOptions'; import type { IDaemonOrphanReaperOptions, IReapContext } from './DaemonReapOptions'; const NO_MEMBERS: number = 0; const LIST_SEPARATOR: string = ', '; +const NOTHING_REAPED: DaemonOrphanReapOutcome = 'none'; function isSameLeader(leader: IProcessStat, record: IOperationGroupRecord): boolean { const { groupId } = record; @@ -47,6 +49,19 @@ async function terminateAndLogAsync( return outcome; } +async function reapRecordedGroupsAsync( + folder: string, + context: IReapContext +): Promise { + const groupIds: number[] = readOperationGroupRecords(folder) + .filter((record: IOperationGroupRecord) => isProvenOperationGroup(record, context)) + .map((record: IOperationGroupRecord) => record.groupId); + const outcome: DaemonOrphanReapOutcome = + groupIds.length > NO_MEMBERS ? await terminateAndLogAsync(context, groupIds) : NOTHING_REAPED; + removeOperationGroupRecords(folder); + return outcome; +} + /** * Terminates the detached operation process groups that dead daemon `deadPid` recorded while it ran * (see `startOperationGroupRecording`), then deletes the records. @@ -56,6 +71,7 @@ async function terminateAndLogAsync( * the recorded start time and still leads group and session `groupId` (a reused pid has another start * time), or its leader has exited and every live member of the group is in session `groupId`. Unproven * records are dropped without a signal. Records survive a failed reap, so the next reclaim retries. + * A record folder that is a symbolic link, or that another user owns, is left alone without a signal. * Call only under the reclaim mutex, after the daemon has been proven dead. */ export async function reapDeadDaemonOperationGroupsAsync( @@ -65,11 +81,7 @@ export async function reapDeadDaemonOperationGroupsAsync( ): Promise { const context: IReapContext = createReapContext(deadPid, options); const folder: string = getOperationGroupsFolder(lockfilePath, deadPid); - const groupIds: number[] = readOperationGroupRecords(folder) - .filter((record: IOperationGroupRecord) => isProvenOperationGroup(record, context)) - .map((record: IOperationGroupRecord) => record.groupId); - const outcome: DaemonOrphanReapOutcome = - groupIds.length > NO_MEMBERS ? await terminateAndLogAsync(context, groupIds) : 'none'; - removeOperationGroupRecords(folder); - return outcome; + return isOwnedEntry(folder, 'directory', context.uid) + ? reapRecordedGroupsAsync(folder, context) + : NOTHING_REAPED; } diff --git a/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts b/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts index 9de74cb9d3..e1433d0f4c 100644 --- a/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts +++ b/libraries/rush-daemon-transport/src/DaemonOrphanReaper.ts @@ -5,7 +5,8 @@ import { terminateProcessGroupsAsync } from './DaemonGroupTermination'; import type { DaemonOrphanReapOutcome } from './DaemonGroupTermination'; import type { IDaemonLockfile } from './DaemonLockfile'; import { reapDeadDaemonOperationGroupsAsync } from './DaemonOperationGroupReaper'; -import { createReapContext, isSignalableGroup } from './DaemonReapOptions'; +import { isOwnedEntry } from './DaemonOwnedEntry'; +import { createReapContext, isSignalableGroup, resolveCallerUid } from './DaemonReapOptions'; import type { IDaemonOrphanReaperOptions, IReapContext } from './DaemonReapOptions'; function isOrphanedDaemonGroup(context: IReapContext): boolean { @@ -36,13 +37,24 @@ export async function reapDeadDaemonProcessGroupAsync( return outcome; } -/** Reaps the orphaned processes of a reclaimed daemon's recorded owner, if there is one. */ +function isOwnRecord( + lockfilePath: string, + owner: IDaemonLockfile | undefined, + options: IDaemonOrphanReaperOptions +): owner is IDaemonLockfile { + return owner !== undefined && isOwnedEntry(lockfilePath, 'file', resolveCallerUid(options)); +} + +/** + * Reaps the orphaned processes of a reclaimed daemon's recorded owner, if there is one. A lockfile that is a + * symbolic link, or that another user owns, names no daemon of this user, so nothing is signaled. + */ export async function reapOrphansOfDeadOwnerAsync( lockfilePath: string, owner: IDaemonLockfile | undefined, options: IDaemonOrphanReaperOptions = {} ): Promise { - if (!owner) return; + if (!isOwnRecord(lockfilePath, owner, options)) return; await reapDeadDaemonProcessGroupAsync(owner.pid, options); await reapDeadDaemonOperationGroupsAsync(lockfilePath, owner.pid, options); } diff --git a/libraries/rush-daemon-transport/src/DaemonOwnedEntry.ts b/libraries/rush-daemon-transport/src/DaemonOwnedEntry.ts new file mode 100644 index 0000000000..a7d82d8801 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonOwnedEntry.ts @@ -0,0 +1,28 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +/** What a record must be: a regular file or a directory, never a symbolic link. */ +export type DaemonOwnedEntryKind = 'file' | 'directory'; + +const DIRECTORY: DaemonOwnedEntryKind = 'directory'; + +function isKind(stats: fs.Stats, kind: DaemonOwnedEntryKind): boolean { + return kind === DIRECTORY ? stats.isDirectory() : stats.isFile(); +} + +function isOwner(stats: fs.Stats, uid: number | undefined): boolean { + // Windows has no user ids to compare. + return uid === undefined || stats.uid === uid; +} + +/** `true` when `entryPath` itself (not a link target) is a `kind` that user `uid` owns. */ +export function isOwnedEntry( + entryPath: string, + kind: DaemonOwnedEntryKind, + uid: number | undefined +): boolean { + const stats: fs.Stats | undefined = fs.lstatSync(entryPath, { throwIfNoEntry: false }); + return stats !== undefined && isOwner(stats, uid) && isKind(stats, kind); +} diff --git a/libraries/rush-daemon-transport/src/DaemonPaths.ts b/libraries/rush-daemon-transport/src/DaemonPaths.ts index fb1f4e558a..6e3b78134d 100644 --- a/libraries/rush-daemon-transport/src/DaemonPaths.ts +++ b/libraries/rush-daemon-transport/src/DaemonPaths.ts @@ -8,7 +8,16 @@ const PIPE_PREFIX: string = '\\\\.\\pipe\\'; const SOCKET_SUFFIX: string = '.sock'; const LOCKFILE_SUFFIX: string = '.pid.json'; const RUNTIME_DIR_NAME: string = 'rushd'; -const XDG_RUNTIME_DIR_ENV: string = 'XDG_RUNTIME_DIR'; +// Unlike TMPDIR or XDG_RUNTIME_DIR, /tmp is the same folder for every process of a user, whatever its +// environment (sudo, cron, env -i, ssh without PAM, a service unit), so all of them meet at one daemon. +const SHARED_POSIX_TEMP_DIR: string = '/tmp'; + +/** + * The environment variable that moves the per-user runtime directory on POSIX. Only an absolute path is used. + * + * @beta + */ +export const DAEMON_RUNTIME_DIR_ENV_VAR: 'RUSHD_RUNTIME_DIR' = 'RUSHD_RUNTIME_DIR'; /** * The platform facts {@link resolveDaemonPaths} needs, injectable for tests. @@ -40,18 +49,18 @@ export interface IDaemonPaths { readonly lockfilePath: string; } -/** - * Resolves the per-user socket/pipe and lockfile paths for a workspace key. - * - * @remarks - * POSIX: `$XDG_RUNTIME_DIR/rushd-/` (falling back to `/rushd-/`), - * with the socket at `rushd-.sock` inside it. Windows: the named pipe - * `\\.\pipe\rushd-`; the lockfile lives in `/rushd/` (the - * temporary directory is already per-user on Windows). - * - * @beta - */ -export function resolveDaemonPaths(environment: IDaemonPathEnvironment, workspaceKey: string): IDaemonPaths { +/** The base folder that `RUSHD_RUNTIME_DIR` names, when it is an absolute path. */ +function getConfiguredRuntimeBase(environment: IDaemonPathEnvironment): string | undefined { + const value: string | undefined = environment.env[DAEMON_RUNTIME_DIR_ENV_VAR]; + return value !== undefined && path.posix.isAbsolute(value) ? value : undefined; +} + +/** Resolves the paths of `workspaceKey` in the runtime directory `/rushd-` (POSIX). */ +function resolveDaemonPathsInBase( + environment: IDaemonPathEnvironment, + workspaceKey: string, + base: string +): IDaemonPaths { if (environment.platform === WINDOWS_PLATFORM) { return { runtimeDir: undefined, @@ -59,7 +68,6 @@ export function resolveDaemonPaths(environment: IDaemonPathEnvironment, workspac lockfilePath: path.win32.join(environment.tmpdir, RUNTIME_DIR_NAME, `${workspaceKey}${LOCKFILE_SUFFIX}`) }; } - const base: string = environment.env[XDG_RUNTIME_DIR_ENV] ?? environment.tmpdir; const runtimeDir: string = path.posix.join(base, `${RUNTIME_DIR_NAME}-${environment.uid}`); return { runtimeDir, @@ -67,3 +75,23 @@ export function resolveDaemonPaths(environment: IDaemonPathEnvironment, workspac lockfilePath: path.posix.join(runtimeDir, `${workspaceKey}${LOCKFILE_SUFFIX}`) }; } + +/** + * Resolves the per-user socket/pipe and lockfile paths at which a client finds the daemon of a workspace key. + * + * @remarks + * POSIX: `/tmp/rushd-/`, or `$RUSHD_RUNTIME_DIR/rushd-/` when that variable is an absolute path, + * with the socket at `rushd-.sock` inside it. `TMPDIR` and `XDG_RUNTIME_DIR` are deliberately not + * consulted: they differ between the shells, services and tools of one user, which would give one checkout + * one daemon per environment. Windows: the named pipe `\\.\pipe\rushd-`; the lockfile lives in + * `/rushd/` (the temporary directory is already per-user on Windows). + * A daemon resolves the paths it listens at with this same rule, and a client that starts one passes the + * base it chose as `RUSHD_RUNTIME_DIR`, so a daemon, its successor and every client of a checkout meet at + * one endpoint. + * + * @beta + */ +export function resolveDaemonPaths(environment: IDaemonPathEnvironment, workspaceKey: string): IDaemonPaths { + const base: string = getConfiguredRuntimeBase(environment) ?? SHARED_POSIX_TEMP_DIR; + return resolveDaemonPathsInBase(environment, workspaceKey, base); +} diff --git a/libraries/rush-daemon-transport/src/DaemonReapOptions.ts b/libraries/rush-daemon-transport/src/DaemonReapOptions.ts index 550183106c..68e6468be3 100644 --- a/libraries/rush-daemon-transport/src/DaemonReapOptions.ts +++ b/libraries/rush-daemon-transport/src/DaemonReapOptions.ts @@ -16,6 +16,8 @@ export interface IDaemonOrphanReaperOptions { readonly selfPid?: number; /** How long SIGTERM'd (and then SIGKILL'd) processes get to exit. */ readonly graceMs?: number; + /** The caller's user id (`process.getuid()`); only records that this user owns are acted on. */ + readonly uid?: number; } /** {@link IDaemonOrphanReaperOptions} with every default applied, for the dead daemon `deadPid`. */ @@ -25,19 +27,26 @@ export interface IReapContext { readonly selfPid: number; readonly deadPid: number; readonly graceMs: number; + readonly uid: number | undefined; } function resolveCaller(options: IDaemonOrphanReaperOptions): Pick { return { platform: options.platform ?? process.platform, selfPid: options.selfPid ?? process.pid }; } +/** The user whose records may be acted on. */ +export function resolveCallerUid(options: IDaemonOrphanReaperOptions): number | undefined { + return options.uid ?? process.getuid?.(); +} + /** Applies the defaults of {@link IDaemonOrphanReaperOptions}. */ export function createReapContext(deadPid: number, options: IDaemonOrphanReaperOptions): IReapContext { return { ...resolveCaller(options), ops: options.ops ?? POSIX_PROCESS_GROUP_OPS, deadPid, - graceMs: options.graceMs ?? DEFAULT_GRACE_MS + graceMs: options.graceMs ?? DEFAULT_GRACE_MS, + uid: resolveCallerUid(options) }; } diff --git a/libraries/rush-daemon-transport/src/DaemonReclaim.ts b/libraries/rush-daemon-transport/src/DaemonReclaim.ts index 5ce5a8d0e4..779b343008 100644 --- a/libraries/rush-daemon-transport/src/DaemonReclaim.ts +++ b/libraries/rush-daemon-transport/src/DaemonReclaim.ts @@ -10,6 +10,7 @@ import { reapOrphansOfDeadOwnerAsync } from './DaemonOrphanReaper'; import type { IDaemonPaths } from './DaemonPaths'; import { tryAcquireReclaimLock } from './DaemonReclaimLock'; import type { DaemonReclaimLockOutcome } from './DaemonReclaimLock'; +import { assertDaemonRuntimeDirIsPrivate } from './DaemonRuntimeDir'; import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTransportError'; /** @@ -23,15 +24,18 @@ import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTranspor * unlink the socket path, so a concurrent starter cannot delete a socket that * another process just bound. Operation processes still running in the dead * daemon's process group, or in the operation process groups it recorded, are - * terminated first (see `DaemonOrphanReaper`). + * terminated first (see `DaemonOrphanReaper`). Nothing is read, reaped or + * removed unless the runtime directory is a private directory of this user. * * @throws {@link DaemonTransportError} with code `daemonAlreadyRunning` when a * live (or plausibly live) daemon owns the path, or when another starter holds - * the reclaim lock. + * the reclaim lock, and with code `unsafeRuntimeDirectory` for an unsafe + * runtime directory. * * @beta */ export async function reclaimStaleDaemonAsync(paths: IDaemonPaths): Promise { + assertDaemonRuntimeDirIsPrivate(paths); // The mutex lives beside the lockfile (never the same file): the lockfile // records the *running* daemon's live PID, while the mutex only ever records // a reclaimer's pid. So a live daemon is "locked" (its PID alive), while a diff --git a/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts b/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts new file mode 100644 index 0000000000..914b5240e4 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts @@ -0,0 +1,52 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +import type { IDaemonPaths } from './DaemonPaths'; +import { verifyRuntimeFolder } from './DaemonRuntimeFolderCheck'; + +const DIR_MODE: number = 0o700; + +function getCurrentUid(): number | undefined { + return process.getuid?.(); +} + +function tryCreateFolder(folder: string): unknown { + try { + fs.mkdirSync(folder, { recursive: true, mode: DIR_MODE }); + return undefined; + } catch (error) { + return error; + } +} + +/** + * Checks the per-user runtime directory, when it exists, before a client trusts the socket inside it. + * + * @remarks + * The folder must be a real directory (not a symbolic link) that the current user owns; one that is also + * open to others is changed to mode `0700`. Otherwise another user could have created it first, for example + * in `/tmp`, and could listen at the socket path or plant the records that reclaim acts on. + * + * @throws {@link DaemonTransportError} with code `unsafeRuntimeDirectory`. + * + * @beta + */ +export function assertDaemonRuntimeDirIsPrivate(paths: IDaemonPaths): void { + if (paths.runtimeDir !== undefined) verifyRuntimeFolder(paths.runtimeDir, getCurrentUid()); +} + +/** + * Creates the per-user runtime directory (mode `0700`) when the platform has one, and checks it as + * {@link assertDaemonRuntimeDirIsPrivate} does. Must be called before binding a POSIX socket inside it. + * + * @beta + */ +export function ensureDaemonRuntimeDir(paths: IDaemonPaths): void { + if (paths.runtimeDir === undefined) return; + const failure: unknown = tryCreateFolder(paths.runtimeDir); + // An entry in the way (such as a file or a dangling link) is explained by the check, not by mkdir. + verifyRuntimeFolder(paths.runtimeDir, getCurrentUid()); + if (failure !== undefined) throw failure; +} diff --git a/libraries/rush-daemon-transport/src/DaemonRuntimeFolderCheck.ts b/libraries/rush-daemon-transport/src/DaemonRuntimeFolderCheck.ts new file mode 100644 index 0000000000..a9cce175ec --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonRuntimeFolderCheck.ts @@ -0,0 +1,64 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +import { DAEMON_RUNTIME_DIR_ENV_VAR } from './DaemonPaths'; +import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTransportError'; + +const DIR_MODE: number = 0o700; +// The remainder by 0o100 is the group and other permission bits; by 0o10000, all permission bits. +const GROUP_AND_OTHER_MODE_MODULUS: number = 0o100; +const PERMISSION_MODE_MODULUS: number = 0o10000; +const NO_MODE_BITS: number = 0; +const OCTAL_RADIX: number = 8; + +interface IUnsafeFolderCheck { + readonly isUnsafe: (stats: fs.Stats, uid: number | undefined) => boolean; + readonly reason: string; +} + +const UNSAFE_FOLDER_CHECKS: readonly IUnsafeFolderCheck[] = [ + { isUnsafe: (stats: fs.Stats) => stats.isSymbolicLink(), reason: 'it is a symbolic link' }, + { isUnsafe: (stats: fs.Stats) => !stats.isDirectory(), reason: 'it is not a directory' }, + { + // Windows has no uid to compare. + isUnsafe: (stats: fs.Stats, uid: number | undefined) => uid !== undefined && stats.uid !== uid, + reason: 'another user owns it' + } +]; + +function createUnsafeFolderError(folder: string, stats: fs.Stats, reason: string): DaemonTransportError { + const mode: string = (stats.mode % PERMISSION_MODE_MODULUS).toString(OCTAL_RADIX); + return new DaemonTransportError( + DaemonTransportErrorCode.unsafeRuntimeDirectory, + `The daemon runtime folder ${folder} is unsafe: ${reason} (owner uid ${stats.uid}, mode ${mode}). ` + + `Remove it, or set ${DAEMON_RUNTIME_DIR_ENV_VAR} to an absolute path of a folder that only you can write.` + ); +} + +function restrictToOwner(folder: string, stats: fs.Stats, uid: number | undefined): void { + // Windows has no POSIX permission bits to tighten. + if (uid !== undefined && stats.mode % GROUP_AND_OTHER_MODE_MODULUS !== NO_MODE_BITS) { + fs.chmodSync(folder, DIR_MODE); + } +} + +function assertSafeFolder(folder: string, stats: fs.Stats, uid: number | undefined): void { + const unsafe: IUnsafeFolderCheck | undefined = UNSAFE_FOLDER_CHECKS.find((check: IUnsafeFolderCheck) => + check.isUnsafe(stats, uid) + ); + if (unsafe) throw createUnsafeFolderError(folder, stats, unsafe.reason); + restrictToOwner(folder, stats, uid); +} + +/** + * When `folder` exists, throws unless it is a directory (not a symbolic link) that user `uid` owns, and makes + * it owner-only (mode `0700`) when others have any access. Without a `uid` (Windows), only the kind is checked. + * + * @throws {@link DaemonTransportError} with code `unsafeRuntimeDirectory`. + */ +export function verifyRuntimeFolder(folder: string, uid: number | undefined): void { + const stats: fs.Stats | undefined = fs.lstatSync(folder, { throwIfNoEntry: false }); + if (stats) assertSafeFolder(folder, stats, uid); +} diff --git a/libraries/rush-daemon-transport/src/DaemonSocketPublication.ts b/libraries/rush-daemon-transport/src/DaemonSocketPublication.ts new file mode 100644 index 0000000000..d3e72e3c8c --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonSocketPublication.ts @@ -0,0 +1,88 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as crypto from 'node:crypto'; +import * as fs from 'node:fs'; +import type * as net from 'node:net'; +import * as path from 'node:path'; + +import type { IDaemonFileIdentity } from './DaemonFileIdentity'; +import { listenOrErrorAsync, toListenTransportError } from './DaemonListenerNet'; +import type { INetError } from './DaemonListenerNet'; +import type { IDaemonPaths } from './DaemonPaths'; +import { reclaimStaleDaemonAsync } from './DaemonReclaim'; +import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTransportError'; + +const PRIVATE_NAME_PREFIX: string = '.bind-'; +const PRIVATE_NAME_SEPARATOR: string = '-'; +const PRIVATE_NAME_RANDOM_BYTES: number = 4; +const HEX: BufferEncoding = 'hex'; +const SOCKET_MODE: number = 0o600; +const ALREADY_EXISTS: string = 'EEXIST'; + +function getPrivateSocketPath(socketPath: string): string { + const suffix: string = crypto.randomBytes(PRIVATE_NAME_RANDOM_BYTES).toString(HEX); + const name: string = `${PRIVATE_NAME_PREFIX}${process.pid}${PRIVATE_NAME_SEPARATOR}${suffix}`; + return path.join(path.dirname(socketPath), name); +} + +function closeServerAsync(server: net.Server): Promise { + return new Promise((resolve: () => void) => server.close(() => resolve())); +} + +/** Gives the private socket the name `socketPath`; `false` when that name exists (link(2) never replaces). */ +function tryLink(privatePath: string, socketPath: string): boolean { + try { + fs.linkSync(privatePath, socketPath); + return true; + } catch (error) { + if ((error as INetError).code !== ALREADY_EXISTS) throw error; + return false; + } +} + +function stillInUse(socketPath: string): DaemonTransportError { + return new DaemonTransportError( + DaemonTransportErrorCode.daemonAlreadyRunning, + `The daemon path ${socketPath} is still in use after reclaim.` + ); +} + +async function publishAsync(privatePath: string, paths: IDaemonPaths): Promise { + fs.chmodSync(privatePath, SOCKET_MODE); + const { dev, ino } = fs.lstatSync(privatePath); + if (!tryLink(privatePath, paths.socketPath)) { + // Reclaim a dead daemon's leftovers once (this throws while their owner lives), then try again. + await reclaimStaleDaemonAsync(paths); + if (!tryLink(privatePath, paths.socketPath)) throw stillInUse(paths.socketPath); + } + return { dev, ino }; +} + +/** + * Binds a POSIX socket under a private name and then publishes it at `paths.socketPath`. + * + * @remarks + * When a server closes, libuv unlinks the path it bound, whichever file has that name by then. The published + * name may by then belong to a successor (after someone deleted this daemon's socket and another daemon + * started), so only the private name is ever bound, and it is deleted as soon as the socket is published. + * Publishing uses link(2), which, unlike bind or rename, never replaces an existing name. The socket is made + * owner-only before it is published. Returns the identity of the published socket; the listening socket keeps + * its inode in use, so the identity needs no open descriptor. + */ +export async function listenPublishedAsync( + server: net.Server, + paths: IDaemonPaths +): Promise { + const privatePath: string = getPrivateSocketPath(paths.socketPath); + const error: INetError | undefined = await listenOrErrorAsync(server, privatePath); + if (error) throw toListenTransportError(error, privatePath); + try { + return await publishAsync(privatePath, paths); + } catch (publishError) { + await closeServerAsync(server); + throw publishError; + } finally { + fs.rmSync(privatePath, { force: true }); + } +} diff --git a/libraries/rush-daemon-transport/src/DaemonTransportError.ts b/libraries/rush-daemon-transport/src/DaemonTransportError.ts index 7a7efb9310..48e9dfb121 100644 --- a/libraries/rush-daemon-transport/src/DaemonTransportError.ts +++ b/libraries/rush-daemon-transport/src/DaemonTransportError.ts @@ -14,7 +14,9 @@ export enum DaemonTransportErrorCode { /** The connection attempt exceeded the configured timeout. */ connectionTimeout = 'connectionTimeout', /** The transport was closed while an operation was in flight. */ - transportClosed = 'transportClosed' + transportClosed = 'transportClosed', + /** The per-user runtime directory is a symbolic link, is not a directory, or another user owns it. */ + unsafeRuntimeDirectory = 'unsafeRuntimeDirectory' } /** diff --git a/libraries/rush-daemon-transport/src/index.ts b/libraries/rush-daemon-transport/src/index.ts index 4674d34b02..7623e62198 100644 --- a/libraries/rush-daemon-transport/src/index.ts +++ b/libraries/rush-daemon-transport/src/index.ts @@ -18,7 +18,6 @@ export { connectDaemonAsync, type IDaemonConnectorOptions } from './DaemonConnec export { DaemonFrameConnection } from './DaemonFrameConnection'; export { DaemonFrameListener, type IDaemonListenerOptions } from './DaemonListener'; export { - ensureDaemonRuntimeDir, isDaemonProcessAlive, readDaemonLockfile, removeDaemonArtifacts, @@ -26,9 +25,15 @@ export { type IDaemonLockfile } from './DaemonLockfile'; export { tryAcquireReclaimLock, type DaemonReclaimLockOutcome } from './DaemonReclaimLock'; -export { resolveDaemonPaths, type IDaemonPathEnvironment, type IDaemonPaths } from './DaemonPaths'; +export { + DAEMON_RUNTIME_DIR_ENV_VAR, + resolveDaemonPaths, + type IDaemonPathEnvironment, + type IDaemonPaths +} from './DaemonPaths'; export { resolveDaemonPathsFromProcess } from './DaemonPathsFromProcess'; export { reclaimStaleDaemonAsync } from './DaemonReclaim'; +export { assertDaemonRuntimeDirIsPrivate, ensureDaemonRuntimeDir } from './DaemonRuntimeDir'; export { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTransportError'; export { computeDaemonWorkspaceKey, diff --git a/libraries/rush-daemon-transport/src/test/DaemonPaths.test.ts b/libraries/rush-daemon-transport/src/test/DaemonPaths.test.ts index aee4c54bce..ffe6bfdbd6 100644 --- a/libraries/rush-daemon-transport/src/test/DaemonPaths.test.ts +++ b/libraries/rush-daemon-transport/src/test/DaemonPaths.test.ts @@ -1,32 +1,38 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import type { IDaemonPathEnvironment } from '../DaemonPaths'; +import type { IDaemonPathEnvironment, IDaemonPaths } from '../DaemonPaths'; import { resolveDaemonPaths } from '../DaemonPaths'; const TEST_UID: number = 1000; const KEY: string = 'rushd-deadbeef'; +const IGNORED_VALUES: readonly string[] = ['', 'relative/folder']; function posixEnv(env: Readonly>): IDaemonPathEnvironment { - return { platform: 'linux', env, tmpdir: '/tmp', uid: TEST_UID }; + return { platform: 'linux', env, tmpdir: '/var/tmp/per-session', uid: TEST_UID }; } -it('uses XDG_RUNTIME_DIR on POSIX when set', () => { - const paths: ReturnType = resolveDaemonPaths( - posixEnv({ XDG_RUNTIME_DIR: '/run/user/1000' }), - KEY - ); - expect(paths.socketPath).toBe('/run/user/1000/rushd-1000/rushd-deadbeef.sock'); - expect(paths.lockfilePath).toBe('/run/user/1000/rushd-1000/rushd-deadbeef.pid.json'); +it('meets in /tmp on POSIX, whatever TMPDIR and XDG_RUNTIME_DIR say', () => { + const plain: IDaemonPaths = resolveDaemonPaths(posixEnv({}), KEY); + expect(plain.socketPath).toBe('/tmp/rushd-1000/rushd-deadbeef.sock'); + expect(plain.lockfilePath).toBe('/tmp/rushd-1000/rushd-deadbeef.pid.json'); + const session: IDaemonPathEnvironment = posixEnv({ XDG_RUNTIME_DIR: '/run/user/1000', TMPDIR: '/scratch' }); + expect(resolveDaemonPaths(session, KEY)).toEqual(plain); +}); + +it('moves the runtime directory to an absolute RUSHD_RUNTIME_DIR', () => { + const paths: IDaemonPaths = resolveDaemonPaths(posixEnv({ RUSHD_RUNTIME_DIR: '/run/rush' }), KEY); + expect(paths.runtimeDir).toBe('/run/rush/rushd-1000'); + expect(paths.socketPath).toBe('/run/rush/rushd-1000/rushd-deadbeef.sock'); }); -it('falls back to the temp dir on POSIX without XDG_RUNTIME_DIR', () => { - const paths: ReturnType = resolveDaemonPaths(posixEnv({}), KEY); - expect(paths.socketPath).toBe('/tmp/rushd-1000/rushd-deadbeef.sock'); +it.each(IGNORED_VALUES)('ignores RUSHD_RUNTIME_DIR=%j, which is not an absolute path', (value: string) => { + const paths: IDaemonPaths = resolveDaemonPaths(posixEnv({ RUSHD_RUNTIME_DIR: value }), KEY); + expect(paths.runtimeDir).toBe('/tmp/rushd-1000'); }); it('uses a named pipe on Windows', () => { - const paths: ReturnType = resolveDaemonPaths( + const paths: IDaemonPaths = resolveDaemonPaths( { platform: 'win32', env: {}, tmpdir: 'C:\\Users\\u\\AppData\\Local\\Temp', uid: undefined }, KEY ); @@ -36,7 +42,7 @@ it('uses a named pipe on Windows', () => { }); it('derives distinct paths for distinct keys', () => { - const first: ReturnType = resolveDaemonPaths(posixEnv({}), KEY); - const second: ReturnType = resolveDaemonPaths(posixEnv({}), 'rushd-00000000'); + const first: IDaemonPaths = resolveDaemonPaths(posixEnv({}), KEY); + const second: IDaemonPaths = resolveDaemonPaths(posixEnv({}), 'rushd-00000000'); expect(first.socketPath).not.toBe(second.socketPath); }); diff --git a/libraries/rush-daemon-transport/src/test/FileIdentity.test.ts b/libraries/rush-daemon-transport/src/test/FileIdentity.test.ts new file mode 100644 index 0000000000..3ad643c1e0 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/FileIdentity.test.ts @@ -0,0 +1,65 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import type { IDaemonFileIdentity } from '../DaemonFileIdentity'; +import { pinFileIdentity, readFileIdentity, removeOwnFile } from '../DaemonFileIdentity'; +import { DaemonFrameListener } from '../DaemonListener'; +import type { IDaemonPaths } from '../DaemonPaths'; + +import { createIsolatedTestDaemonPaths, removeIsolatedBase } from './TestDaemonFixture'; + +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +const UTF8: BufferEncoding = 'utf8'; +const SOCKET_MODE: number = 0o600; +const PERMISSION_MODULUS: number = 0o1000; +const PRIVATE_NAME_PREFIX: string = '.bind-'; + +function createFile(paths: IDaemonPaths, content: string): string { + const filePath: string = path.join(paths.runtimeDir ?? '', 'file'); + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, content); + return filePath; +} + +posixIt('keeps a pinned inode in use, so a replacement never passes for the file it replaced', () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const filePath: string = createFile(paths, 'first'); + const pinned: IDaemonFileIdentity = pinFileIdentity(filePath); + fs.rmSync(filePath); + fs.writeFileSync(filePath, 'second'); + expect(readFileIdentity(filePath)?.ino).not.toBe(pinned.ino); + removeOwnFile(filePath, pinned); + expect(fs.readFileSync(filePath, UTF8)).toBe('second'); + expect(() => fs.fstatSync(pinned.fd ?? Number.NaN)).toThrow(); + removeIsolatedBase(paths); +}); + +posixIt('removes a file that is still the pinned one', () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const filePath: string = createFile(paths, 'only'); + removeOwnFile(filePath, pinFileIdentity(filePath)); + expect(fs.existsSync(filePath)).toBe(false); + removeIsolatedBase(paths); +}); + +posixIt('publishes an owner-only socket and deletes the private name it bound', async () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const listener: DaemonFrameListener = await DaemonFrameListener.listenAsync(paths, { + protocolVersion: DAEMON_PROTOCOL_VERSION, + onConnection: () => undefined + }); + try { + expect(fs.lstatSync(paths.socketPath).isSocket()).toBe(true); + expect(fs.lstatSync(paths.socketPath).mode % PERMISSION_MODULUS).toBe(SOCKET_MODE); + const names: string[] = fs.readdirSync(paths.runtimeDir ?? ''); + expect(names.filter((name: string) => name.startsWith(PRIVATE_NAME_PREFIX))).toEqual([]); + } finally { + await listener.closeAsync(); + removeIsolatedBase(paths); + } +}); diff --git a/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts b/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts new file mode 100644 index 0000000000..abd43150e2 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts @@ -0,0 +1,71 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import { connectDaemonAsync } from '../DaemonConnector'; +import type { DaemonFrameConnection } from '../DaemonFrameConnection'; +import { DaemonFrameListener } from '../DaemonListener'; +import { readDaemonLockfile } from '../DaemonLockfile'; +import type { IDaemonPaths } from '../DaemonPaths'; + +import { createDeferred, createIsolatedTestDaemonPaths, removeIsolatedBase } from './TestDaemonFixture'; +import type { IDeferred } from './TestDaemonFixture'; + +// A named pipe can't be deleted while its server runs, so only POSIX has a successor in this sense. +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +const SUCCESSOR: string = 'successor'; + +function listenAsync(paths: IDaemonPaths, reached?: IDeferred): Promise { + return DaemonFrameListener.listenAsync(paths, { + protocolVersion: DAEMON_PROTOCOL_VERSION, + onConnection: () => reached?.resolve(SUCCESSOR) + }); +} + +async function expectSuccessorReachableAsync(paths: IDaemonPaths, reached: IDeferred): Promise { + expect(readDaemonLockfile(paths.lockfilePath)?.socketPath).toBe(paths.socketPath); + const client: DaemonFrameConnection = await connectDaemonAsync(paths.socketPath); + await expect(reached.promise).resolves.toBe(SUCCESSOR); + await client.closeAsync(); +} + +async function expectPredecessorCloseSparesSuccessorAsync( + deleteFiles: (paths: IDaemonPaths) => void +): Promise { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const predecessor: DaemonFrameListener = await listenAsync(paths); + deleteFiles(paths); + const reached: IDeferred = createDeferred(); + const successor: DaemonFrameListener = await listenAsync(paths, reached); + try { + await predecessor.closeAsync(); + await expectSuccessorReachableAsync(paths, reached); + } finally { + await successor.closeAsync(); + removeIsolatedBase(paths); + } +} + +posixIt('keeps a successor reachable after a predecessor whose socket and lockfile were deleted closes', () => + expectPredecessorCloseSparesSuccessorAsync((paths: IDaemonPaths) => { + fs.rmSync(paths.socketPath); + fs.rmSync(paths.lockfilePath); + }) +); + +posixIt('keeps a successor reachable after a predecessor whose runtime folder was deleted closes', () => + expectPredecessorCloseSparesSuccessorAsync((paths: IDaemonPaths) => { + fs.rmSync(paths.runtimeDir ?? paths.socketPath, { recursive: true }); + }) +); + +posixIt('removes its own socket and lockfile when it closes', async () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const listener: DaemonFrameListener = await listenAsync(paths); + await listener.closeAsync(); + expect(fs.readdirSync(paths.runtimeDir ?? paths.socketPath)).toEqual([]); + removeIsolatedBase(paths); +}); diff --git a/libraries/rush-daemon-transport/src/test/RecordOwnership.test.ts b/libraries/rush-daemon-transport/src/test/RecordOwnership.test.ts new file mode 100644 index 0000000000..9cc68e6d56 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/RecordOwnership.test.ts @@ -0,0 +1,97 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import type { IDaemonLockfile } from '../DaemonLockfile'; +import { writeDaemonLockfile } from '../DaemonLockfile'; +import { reapDeadDaemonOperationGroupsAsync } from '../DaemonOperationGroupReaper'; +import { getOperationGroupsFolder } from '../DaemonOperationGroups'; +import { reapOrphansOfDeadOwnerAsync } from '../DaemonOrphanReaper'; +import type { IDaemonOrphanReaperOptions } from '../DaemonReapOptions'; + +import { OPERATION_GROUP, operationTree, recordGroups, recordsRemain } from './OperationGroupFixture'; +import { DEAD_PID, createFakeGroup } from './OrphanReaperFixture'; +import type { IFakeGroup } from './OrphanReaperFixture'; + +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +const NO_UID: number = 0; +const FIRST_INDEX: number = 0; +const EPOCH_MS: number = 0; +const OTHER_USER_OFFSET: number = 1; +const LINK_TARGET_SUFFIX: string = '.target'; +const OWNER: IDaemonLockfile = { + pid: DEAD_PID, + protocolVersion: DAEMON_PROTOCOL_VERSION, + startedAt: new Date(EPOCH_MS).toISOString(), + socketPath: 'unused' +}; + +const createdEntries: string[] = []; + +afterEach(() => { + for (const entry of createdEntries.splice(FIRST_INDEX)) fs.rmSync(entry, { recursive: true, force: true }); +}); + +function recordOwnGroups(): string { + const lockfilePath: string = recordGroups([OPERATION_GROUP]); + for (const entry of [lockfilePath, getOperationGroupsFolder(lockfilePath, DEAD_PID)]) { + createdEntries.push(entry, `${entry}${LINK_TARGET_SUFFIX}`); + } + return lockfilePath; +} + +function createFake(): IFakeGroup { + return createFakeGroup({ exitsOn: 'SIGTERM', processes: operationTree(OPERATION_GROUP) }); +} + +function asAnotherUser(options: IDaemonOrphanReaperOptions): IDaemonOrphanReaperOptions { + return { ...options, uid: (process.getuid?.() ?? NO_UID) + OTHER_USER_OFFSET }; +} + +function moveBehindLink(entryPath: string): void { + fs.renameSync(entryPath, `${entryPath}${LINK_TARGET_SUFFIX}`); + fs.symlinkSync(`${entryPath}${LINK_TARGET_SUFFIX}`, entryPath); +} + +function recordDeadOwner(): string { + const lockfilePath: string = recordOwnGroups(); + writeDaemonLockfile(lockfilePath, OWNER); + return lockfilePath; +} + +posixIt('never signals the groups in a record folder that is a symbolic link', async () => { + const fake: IFakeGroup = createFake(); + const lockfilePath: string = recordOwnGroups(); + moveBehindLink(getOperationGroupsFolder(lockfilePath, DEAD_PID)); + await expect(reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, fake.options)).resolves.toBe( + 'none' + ); + expect(fake.signals).toEqual([]); + expect(recordsRemain(lockfilePath)).toBe(true); +}); + +posixIt('never signals the groups in a record folder of another user', async () => { + const fake: IFakeGroup = createFake(); + const lockfilePath: string = recordOwnGroups(); + const options: IDaemonOrphanReaperOptions = asAnotherUser(fake.options); + await expect(reapDeadDaemonOperationGroupsAsync(lockfilePath, DEAD_PID, options)).resolves.toBe('none'); + expect(fake.signals).toEqual([]); +}); + +posixIt('never signals the owner named by a lockfile of another user, or behind a link', async () => { + const fake: IFakeGroup = createFake(); + const lockfilePath: string = recordDeadOwner(); + await reapOrphansOfDeadOwnerAsync(lockfilePath, OWNER, asAnotherUser(fake.options)); + moveBehindLink(lockfilePath); + await reapOrphansOfDeadOwnerAsync(lockfilePath, OWNER, fake.options); + expect(fake.signals).toEqual([]); +}); + +it('signals the group of the owner named by a lockfile of this user', async () => { + const fake: IFakeGroup = createFake(); + await reapOrphansOfDeadOwnerAsync(recordDeadOwner(), OWNER, fake.options); + expect(fake.targets).toEqual([DEAD_PID]); +}); diff --git a/libraries/rush-daemon-transport/src/test/RuntimeFolder.test.ts b/libraries/rush-daemon-transport/src/test/RuntimeFolder.test.ts new file mode 100644 index 0000000000..a0f65eb349 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/RuntimeFolder.test.ts @@ -0,0 +1,98 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import { DaemonFrameListener } from '../DaemonListener'; +import type { IDaemonPaths } from '../DaemonPaths'; +import { DAEMON_RUNTIME_DIR_ENV_VAR } from '../DaemonPaths'; +import { reclaimStaleDaemonAsync } from '../DaemonReclaim'; +import { assertDaemonRuntimeDirIsPrivate, ensureDaemonRuntimeDir } from '../DaemonRuntimeDir'; +import { verifyRuntimeFolder } from '../DaemonRuntimeFolderCheck'; +import { DaemonTransportErrorCode } from '../DaemonTransportError'; + +import { createIsolatedTestDaemonPaths, removeIsolatedBase } from './TestDaemonFixture'; + +// Windows has no runtime folder: its named pipes live in the kernel's pipe namespace. +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +const PRIVATE_MODE: number = 0o700; +const OPEN_MODE: number = 0o755; +const PERMISSION_MODULUS: number = 0o1000; +const NO_UID: number = 0; +const OTHER_USER_OFFSET: number = 1; +const LINK_TARGET_SUFFIX: string = '.target'; + +type Planter = (folder: string) => void; + +const PLANTERS: readonly [string, Planter][] = [ + [ + 'it is a symbolic link', + (folder: string) => { + fs.mkdirSync(`${folder}${LINK_TARGET_SUFFIX}`, { mode: PRIVATE_MODE }); + fs.symlinkSync(`${folder}${LINK_TARGET_SUFFIX}`, folder); + } + ], + ['it is not a directory', (folder: string) => fs.writeFileSync(folder, '')] +]; + +function getMode(folder: string): number { + return fs.statSync(folder).mode % PERMISSION_MODULUS; +} + +function getOtherUid(): number { + return (process.getuid?.() ?? NO_UID) + OTHER_USER_OFFSET; +} + +function captureError(action: () => void): unknown { + try { + action(); + } catch (error) { + return error; + } +} + +posixIt('creates the runtime folder owner-only, and tightens one that others can enter', () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const folder: string = paths.runtimeDir ?? ''; + ensureDaemonRuntimeDir(paths); + expect(getMode(folder)).toBe(PRIVATE_MODE); + fs.chmodSync(folder, OPEN_MODE); + assertDaemonRuntimeDirIsPrivate(paths); + expect(getMode(folder)).toBe(PRIVATE_MODE); + removeIsolatedBase(paths); +}); + +posixIt.each(PLANTERS)('refuses a runtime folder when %s', async (reason: string, plant: Planter) => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + plant(paths.runtimeDir ?? ''); + const expected: object = { + code: DaemonTransportErrorCode.unsafeRuntimeDirectory, + message: expect.stringContaining(reason) + }; + const error: unknown = captureError(() => assertDaemonRuntimeDirIsPrivate(paths)); + expect(error).toMatchObject(expected); + expect((error as Error).message).toContain(DAEMON_RUNTIME_DIR_ENV_VAR); + expect(captureError(() => ensureDaemonRuntimeDir(paths))).toMatchObject(expected); + await expect(reclaimStaleDaemonAsync(paths)).rejects.toMatchObject(expected); + const listening: Promise = DaemonFrameListener.listenAsync(paths, { + protocolVersion: DAEMON_PROTOCOL_VERSION, + onConnection: () => undefined + }); + await expect(listening).rejects.toMatchObject(expected); + removeIsolatedBase(paths); +}); + +posixIt('refuses a runtime folder of another user without changing it', () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const folder: string = paths.runtimeDir ?? ''; + fs.mkdirSync(folder, { mode: OPEN_MODE }); + fs.chmodSync(folder, OPEN_MODE); + expect(captureError(() => verifyRuntimeFolder(folder, getOtherUid()))).toMatchObject({ + code: DaemonTransportErrorCode.unsafeRuntimeDirectory, + message: expect.stringContaining('another user owns it') + }); + expect(getMode(folder)).toBe(OPEN_MODE); + removeIsolatedBase(paths); +}); diff --git a/libraries/rush-daemon-transport/src/test/TestDaemonFixture.ts b/libraries/rush-daemon-transport/src/test/TestDaemonFixture.ts index b04a4f0f85..08585b307a 100644 --- a/libraries/rush-daemon-transport/src/test/TestDaemonFixture.ts +++ b/libraries/rush-daemon-transport/src/test/TestDaemonFixture.ts @@ -1,7 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as fs from 'node:fs'; import * as os from 'node:os'; +import * as path from 'node:path'; import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; @@ -9,21 +11,44 @@ import { connectDaemonAsync } from '../DaemonConnector'; import type { DaemonFrameConnection } from '../DaemonFrameConnection'; import { DaemonFrameListener } from '../DaemonListener'; import type { IDaemonPaths } from '../DaemonPaths'; -import { resolveDaemonPaths } from '../DaemonPaths'; +import { DAEMON_RUNTIME_DIR_ENV_VAR, resolveDaemonPaths } from '../DaemonPaths'; let testKeyCounter: number = 0; const COUNTER_START: number = 1; +const ISOLATED_BASE_PREFIX: string = 'rushd-test-base-'; +// Short, as a socket path must be (104 bytes on macOS); the daemon's own default base is /tmp too. +const POSIX_TEMP_FOLDER: string = '/tmp'; -/** Creates unique daemon paths for the current platform in the temp dir. */ -export function createTestDaemonPaths(): IDaemonPaths { +function resolveTestDaemonPaths(env: Readonly>): IDaemonPaths { testKeyCounter += COUNTER_START; const workspaceKey: string = `rushd-test-${process.pid}-${testKeyCounter}`; return resolveDaemonPaths( - { platform: process.platform, env: {}, tmpdir: os.tmpdir(), uid: process.getuid?.() }, + { platform: process.platform, env, tmpdir: os.tmpdir(), uid: process.getuid?.() }, workspaceKey ); } +/** Creates unique daemon paths for the current platform in the user's shared runtime directory. */ +export function createTestDaemonPaths(): IDaemonPaths { + return resolveTestDaemonPaths({}); +} + +/** + * Creates unique daemon paths in a new runtime base of their own (see {@link removeIsolatedBase}), for tests + * that delete or replace the runtime directory. Real daemons use the shared one. + */ +export function createIsolatedTestDaemonPaths(): IDaemonPaths { + const parent: string = process.platform === 'win32' ? os.tmpdir() : POSIX_TEMP_FOLDER; + const base: string = fs.mkdtempSync(path.join(parent, ISOLATED_BASE_PREFIX)); + return resolveTestDaemonPaths({ [DAEMON_RUNTIME_DIR_ENV_VAR]: base }); +} + +/** Deletes the runtime base of paths from {@link createIsolatedTestDaemonPaths}. */ +export function removeIsolatedBase(paths: IDaemonPaths): void { + if (paths.runtimeDir !== undefined) + fs.rmSync(path.dirname(paths.runtimeDir), { recursive: true, force: true }); +} + /** A minimal deferred promise for crossing the callback/async boundary. */ export interface IDeferred { readonly promise: Promise; diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index a28234b3df..5b121b1c42 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -86,7 +86,9 @@ The host uses stable fingerprints to classify native requests: Configuration fingerprints use contents rather than timestamps. Runtime content hashes are cached only behind file identity/size/mtime/ctime checks; touching unchanged content does not itself change a fingerprint. Native dispatch first copies the envelope and normalizes only engine-owned `_RUSH_LIB_PATH` to this daemon's -real engine, preventing false restarts or wrong SDK selection from a foreign client path. Environment +own engine, preventing false restarts or wrong SDK selection from a foreign client path. The daemon keeps the +spelling that its engine chose when it loaded: a source-built rush-lib, such as one in a `rush deploy` output, +is spelled through the daemon's own `node_modules/@microsoft/rush-lib` link so plugins can resolve it by name. Environment comparisons (the tier-2 fingerprint and the production resolver's startup-environment check) both use rush-lib's `getWorkspaceFingerprintEnvironmentEntries()`, which omits `workspaceFingerprintIgnoredEnvironmentVariables`: volatile per-shell, terminal, session and client-routing variables such as `PWD`, `OLDPWD`, `SHLVL`, `_`, diff --git a/libraries/rush-daemon/package.json b/libraries/rush-daemon/package.json index c89c2059a4..663f3128ca 100644 --- a/libraries/rush-daemon/package.json +++ b/libraries/rush-daemon/package.json @@ -56,6 +56,9 @@ }, "devDependencies": { "@rushstack/heft": "workspace:*", + "@rushstack/rush-amazon-s3-build-cache-plugin": "workspace:*", + "@rushstack/rush-azure-storage-build-cache-plugin": "workspace:*", + "@rushstack/rush-http-build-cache-plugin": "workspace:*", "eslint": "~9.37.0", "local-node-rig": "workspace:*" }, diff --git a/libraries/rush-daemon/src/RushLibPathHandoff.ts b/libraries/rush-daemon/src/RushLibPathHandoff.ts new file mode 100644 index 0000000000..14a924e5ac --- /dev/null +++ b/libraries/rush-daemon/src/RushLibPathHandoff.ts @@ -0,0 +1,30 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; + +/** + * Returns the daemon's own `_RUSH_LIB_PATH` value for the rush-lib at `entryPoint`. + * + * @remarks + * When rush-lib loads, it may spell its entry point through a `node_modules` link, so that plugins can + * resolve it by name (for example in a `rush deploy` output). That spelling is kept whenever it names the + * same file as `entryPoint`. Anything else, such as a foreign client's engine, is replaced by `entryPoint`. + */ +export function getRushLibPathHandoff(entryPoint: string, currentValue: string | undefined): string { + if (currentValue === undefined || currentValue === entryPoint) { + return entryPoint; + } + const realEntryPoint: string | undefined = tryGetRealPath(entryPoint); + return realEntryPoint !== undefined && tryGetRealPath(currentValue) === realEntryPoint + ? currentValue + : entryPoint; +} + +function tryGetRealPath(filePath: string): string | undefined { + try { + return fs.realpathSync.native(filePath); + } catch { + return undefined; + } +} diff --git a/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts b/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts index ee8d7b1e97..f39cbd2717 100644 --- a/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts +++ b/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts @@ -13,6 +13,7 @@ import { type IDaemonInstallationMetadata, type IInstalledDaemonLauncher } from './DaemonInstallation'; +import { getRushLibPathHandoff } from './RushLibPathHandoff'; import { installWindowsHideDefault } from './WindowsSubprocessConsoles'; async function mainAsync(): Promise { @@ -30,7 +31,8 @@ async function mainAsync(): Promise { throw new Error(`Rush runtime and installed metadata disagree for ${metadata.launcherPath}.`); } // This is the native Rush SDK handoff, pointing at the actual selected engine, not the caller's engine. - process.env._RUSH_LIB_PATH = rushLibEntryPoint; + // Loading the selected rush-lib already set it; its spelling is kept if it names that engine. + process.env._RUSH_LIB_PATH = getRushLibPathHandoff(rushLibEntryPoint, process.env._RUSH_LIB_PATH); const protocol: { DAEMON_PROTOCOL_VERSION?: IDaemonProtocolVersion } = selectedRequire( '@rushstack/rush-daemon-protocol' ); diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 96349a42b2..9bc35cba8e 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -45,6 +45,7 @@ import { } from './WorkspaceRequestAdmission'; import { WorkspaceEngineRecreationRequiredError } from './WorkspaceEngineComponentFactory'; import { getDaemonShutdownReason } from './DaemonShutdownError'; +import { getRushLibPathHandoff } from './RushLibPathHandoff'; import type { IWorkspaceSession } from './WorkspaceSession'; import type { WorkspaceSessionProvider } from './WorkspaceSessionProvider'; import { assertWorkspaceRequestResourcesHealthy } from './WorkspaceRequestResources'; @@ -184,7 +185,10 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { ...request, environment: { ...request.environment, - [EnvironmentVariableNames._RUSH_LIB_PATH]: require.resolve('@microsoft/rush-lib') + [EnvironmentVariableNames._RUSH_LIB_PATH]: getRushLibPathHandoff( + require.resolve('@microsoft/rush-lib'), + process.env[EnvironmentVariableNames._RUSH_LIB_PATH] + ) } }; if (this.#restartPending) { diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index c9c31ac580..0667da53eb 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -525,6 +525,48 @@ describe('native production daemon engine', () => { } }); + it('keeps its own linked spelling of the native SDK handoff', async () => { + // A deployed daemon spells rush-lib through its own node_modules link, so plugins can resolve it by name. + const originalRushLibPath: string | undefined = process.env._RUSH_LIB_PATH; + const linkRoot: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-sdk-link-')); + const rushLibFolder: string = path.dirname(require.resolve('@microsoft/rush-lib/package.json')); + const rushLibLink: string = path.join(linkRoot, 'node_modules', '@microsoft', 'rush-lib'); + fs.mkdirSync(path.dirname(rushLibLink), { recursive: true }); + fs.symlinkSync(rushLibFolder, rushLibLink, 'junction'); + const linkedEntryPoint: string = path.join( + rushLibLink, + path.relative(rushLibFolder, require.resolve('@microsoft/rush-lib')) + ); + process.env._RUSH_LIB_PATH = linkedEntryPoint; + try { + const fixture: IFixture = await createFixtureAsync(); + try { + expect((await runAsync(fixture, 'linked-sdk', ['build', '--only', 'a'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0 } + }); + const environment: Record = { + ...requestEnvironment(), + _RUSH_LIB_PATH: path.join(fixture.repoRoot, 'foreign-client-engine.js') + }; + expect( + (await runAsync(fixture, 'foreign-linked-sdk', ['build', '--only', 'a'], { environment })).terminal + ).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0, scheduled: false } }); + expect(process.env._RUSH_LIB_PATH).toBe(linkedEntryPoint); + expect(runs(fixture)).toEqual(['a:one:']); + } finally { + await fixture[Symbol.asyncDispose](); + } + } finally { + if (originalRushLibPath === undefined) { + delete process.env._RUSH_LIB_PATH; + } else { + process.env._RUSH_LIB_PATH = originalRushLibPath; + } + fs.rmSync(linkRoot, { recursive: true, force: true }); + } + }); + it('replaces configuration and command shape in-process without using a disposed generation', async () => { const fixture: IFixture = await createFixtureAsync(); try { diff --git a/libraries/rush-daemon/src/test/RushLibPathHandoff.test.ts b/libraries/rush-daemon/src/test/RushLibPathHandoff.test.ts new file mode 100644 index 0000000000..6a0a3b9d11 --- /dev/null +++ b/libraries/rush-daemon/src/test/RushLibPathHandoff.test.ts @@ -0,0 +1,47 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { getRushLibPathHandoff } from '../RushLibPathHandoff'; + +describe(getRushLibPathHandoff.name, () => { + let folder: string; + let entryPoint: string; + let linkedEntryPoint: string; + + beforeEach(() => { + folder = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-lib-path-'))); + const rushLibFolder: string = path.join(folder, 'libraries', 'rush-lib'); + fs.mkdirSync(path.join(rushLibFolder, 'lib-commonjs'), { recursive: true }); + entryPoint = path.join(rushLibFolder, 'lib-commonjs', 'index.js'); + fs.writeFileSync(entryPoint, ''); + const rushLibLink: string = path.join(folder, 'daemon', 'node_modules', '@microsoft', 'rush-lib'); + fs.mkdirSync(path.dirname(rushLibLink), { recursive: true }); + fs.symlinkSync(rushLibFolder, rushLibLink, 'junction'); + linkedEntryPoint = path.join(rushLibLink, 'lib-commonjs', 'index.js'); + }); + + afterEach(() => { + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it('uses the entry point when nothing is set', () => { + expect(getRushLibPathHandoff(entryPoint, undefined)).toBe(entryPoint); + }); + + it('keeps a linked spelling of the same entry point', () => { + expect(getRushLibPathHandoff(entryPoint, linkedEntryPoint)).toBe(linkedEntryPoint); + }); + + it('replaces a different or missing engine', () => { + const otherEntryPoint: string = path.join(folder, 'other', 'index.js'); + fs.mkdirSync(path.dirname(otherEntryPoint), { recursive: true }); + fs.writeFileSync(otherEntryPoint, ''); + + expect(getRushLibPathHandoff(entryPoint, otherEntryPoint)).toBe(entryPoint); + expect(getRushLibPathHandoff(entryPoint, path.join(folder, 'missing.js'))).toBe(entryPoint); + }); +}); diff --git a/libraries/rush-daemon/src/test/SuccessfulMutationFixture.ts b/libraries/rush-daemon/src/test/SuccessfulMutationFixture.ts index 3e2b004672..c6a00d72b3 100644 --- a/libraries/rush-daemon/src/test/SuccessfulMutationFixture.ts +++ b/libraries/rush-daemon/src/test/SuccessfulMutationFixture.ts @@ -104,7 +104,7 @@ export class SuccessfulMutationFixture implements AsyncDisposable { ...process.env, HOME: home, USERPROFILE: home, - XDG_RUNTIME_DIR: path.join(this.folder, 'runtime'), + RUSHD_RUNTIME_DIR: path.join(this.folder, 'runtime'), RUSH_GLOBAL_FOLDER: path.join(this.folder, 'rush-global'), RUSH_PNPM_STORE_PATH: path.join(this.folder, 'store'), RUSH_TEMP_FOLDER: undefined, diff --git a/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts b/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts index 1187628df5..5458cc0ad3 100644 --- a/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts +++ b/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts @@ -56,6 +56,7 @@ describe('version-selected daemon launcher', () => { RUSH_GLOBAL_FOLDER: path.join(repoRoot, 'global'), RUSH_PREVIEW_VERSION: undefined, NPM_CONFIG_CACHE: path.join(repoRoot, 'npm-cache'), + RUSHD_RUNTIME_DIR: path.join(repoRoot, 'runtime'), XDG_RUNTIME_DIR: path.join(repoRoot, 'runtime'), TMPDIR: path.join(repoRoot, 'runtime'), TMP: path.join(repoRoot, 'runtime'), @@ -99,13 +100,20 @@ describe('version-selected daemon launcher', () => { COPILOT_AGENT_SESSION_ID: 'session-1', RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', RUSHD_OUTPUT: 'agent', + // The first client's session folders may disappear while the daemon lives on. + TMPDIR: '/tmp/session-1', + XDG_RUNTIME_DIR: '/run/user/1000', + TEMP: '/tmp/temp-1', + RUSHD_RUNTIME_DIR: '/var/rushd', UNSET: undefined } }); expect(command.environment).toEqual({ HOME: '/home/user', RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', - RUSHD_OUTPUT: 'agent' + RUSHD_OUTPUT: 'agent', + TEMP: '/tmp/temp-1', + RUSHD_RUNTIME_DIR: '/var/rushd' }); expect(Object.isFrozen(command.environment)).toBe(true); }); diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index b8914cfab9..0505b2633f 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -66,6 +66,8 @@ export interface IWorkspaceInputFingerprintOptions { * `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` is sent as each request's admission deadline, `RUSHD_OUTPUT` selects * the client's output mode, and `RUSH_DAEMON_EXPERIMENTAL` is read from each request rather than from the process * - `RUSH_PARALLELISM`, which a long-lived host applies to each request as its `--parallelism` default + * - temporary and runtime folders, which are often set per session, job or sandbox: `TMPDIR`, `TMP`, `TEMP` and + * `XDG_RUNTIME_DIR`, and `RUSHD_RUNTIME_DIR`, which only selects the folder where a client meets its daemon * * Every other variable remains a process-bound input, including the remaining `RUSH_*` settings (such as * `RUSH_BUILD_CACHE_*` and the daemon's own `RUSH_DAEMON_*` resource settings), `NODE_*`, npm/pnpm @@ -142,7 +144,12 @@ export const workspaceFingerprintIgnoredEnvironmentVariables: ReadonlySet = new Set([ 'RUSH_PARALLELISM', - 'COPILOT_AGENT_SESSION_ID' + 'COPILOT_AGENT_SESSION_ID', + 'TMPDIR', + 'XDG_RUNTIME_DIR' ]); /** diff --git a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts index 90f646331c..400cc891eb 100644 --- a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts +++ b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts @@ -221,7 +221,9 @@ describe('workspace input fingerprints', () => { WT_SESSION: 'w' }, { PATH: `${base.PATH}${path.delimiter}${base.PATH}` }, - { TERM: undefined, PWD: undefined } + { TERM: undefined, PWD: undefined }, + { TMPDIR: '/scratch/job-1', XDG_RUNTIME_DIR: '/run/user/1000' }, + { TMP: 'C:\\Temp\\2', TEMP: 'C:\\Temp\\2', RUSHD_RUNTIME_DIR: '/run/rush' } ]) { expect(await getHashAsync({ ...base, ...volatile })).toBe(baseHash); } @@ -262,12 +264,18 @@ describe('workspace input fingerprints', () => { COPILOT_AGENT_SESSION_ID: 'session-1', RUSHD_OUTPUT: 'agent', RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', + TMPDIR: '/scratch/job-1', + XDG_RUNTIME_DIR: '/run/user/1000', + TEMP: 'C:\\Temp', + RUSHD_RUNTIME_DIR: '/run/rush', UNSET: undefined }; expect(getWorkspaceHostEnvironment(environment)).toEqual({ HOME: '/home/user', RUSHD_OUTPUT: 'agent', - RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400' + RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', + TEMP: 'C:\\Temp', + RUSHD_RUNTIME_DIR: '/run/rush' }); for (const name of workspaceRequestScopedEnvironmentVariables) { expect(workspaceFingerprintIgnoredEnvironmentVariables.has(name)).toBe(true); diff --git a/libraries/rush-lib/src/pluginFramework/PluginManager.ts b/libraries/rush-lib/src/pluginFramework/PluginManager.ts index 2322d6f071..e5c2223cdc 100644 --- a/libraries/rush-lib/src/pluginFramework/PluginManager.ts +++ b/libraries/rush-lib/src/pluginFramework/PluginManager.ts @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { FileSystem, Import, InternalError } from '@rushstack/node-core-library'; +import { FileSystem, Import, InternalError, type IPackageJson } from '@rushstack/node-core-library'; import type { ITerminal } from '@rushstack/terminal'; import type { CommandLineConfiguration } from '../api/CommandLineConfiguration'; @@ -13,6 +13,8 @@ import { _createRushSessionForPlugin, type RushSession } from './RushSession'; import type { PluginLoaderBase, IRushPluginManifest } from './PluginLoader/PluginLoaderBase'; import { Rush } from '../api/Rush'; import type { RushGlobalFolder } from '../api/RushGlobalFolder'; +import { findNodeModulesPackageFolder } from '../utilities/RushLibPathHandoff'; +import { rushLibPathHandoff } from '../utilities/SetRushLibPath'; export interface IPluginManagerOptions { terminal: ITerminal; @@ -61,19 +63,38 @@ export class PluginManager { // "publishOnlyDependencies" which gets moved into "dependencies" during publishing. const builtInPluginConfigurations: IBuiltInPluginConfiguration[] = options.builtInPluginConfigurations; - const ownPackageJsonDependencies: Record = Rush._rushLibPackageJson.dependencies || {}; + const ownPackageJson: IPackageJson & { publishOnlyDependencies?: Record } = + Rush._rushLibPackageJson; + const ownPackageJsonDependencies: Record = ownPackageJson.dependencies || {}; + const publishOnlyDependencies: Record = ownPackageJson.publishOnlyDependencies || {}; function tryAddBuiltInPlugin(builtInPluginName: string, pluginPackageName?: string): void { if (!pluginPackageName) { pluginPackageName = `@rushstack/${builtInPluginName}`; } + if ( + builtInPluginConfigurations.some( + ({ packageName, pluginName }) => packageName === pluginPackageName && pluginName === builtInPluginName + ) + ) { + // The host already provides this plugin, as apps/rush/src/start-dev.ts does. + return; + } + let pluginPackageFolder: string | undefined; if (ownPackageJsonDependencies[pluginPackageName]) { + pluginPackageFolder = Import.resolvePackage({ + packageName: pluginPackageName, + baseFolderPath: __dirname + }); + } else if (publishOnlyDependencies[pluginPackageName] && rushLibPathHandoff) { + // An unpublished rush-lib, such as one in a "rush deploy" output, uses the plugins that its host + // installed next to the rush-lib link that _RUSH_LIB_PATH goes through. + pluginPackageFolder = findNodeModulesPackageFolder(rushLibPathHandoff.packageFolder, pluginPackageName); + } + if (pluginPackageFolder) { builtInPluginConfigurations.push({ packageName: pluginPackageName, pluginName: builtInPluginName, - pluginPackageFolder: Import.resolvePackage({ - packageName: pluginPackageName, - baseFolderPath: __dirname - }) + pluginPackageFolder }); } } diff --git a/libraries/rush-lib/src/pluginFramework/test/PluginManager.test.ts b/libraries/rush-lib/src/pluginFramework/test/PluginManager.test.ts new file mode 100644 index 0000000000..443e90a293 --- /dev/null +++ b/libraries/rush-lib/src/pluginFramework/test/PluginManager.test.ts @@ -0,0 +1,129 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +let mockRushLibPathHandoff: IRushLibPathHandoff | undefined; +jest.mock('../../utilities/SetRushLibPath', () => ({ + get rushLibPathHandoff(): IRushLibPathHandoff | undefined { + return mockRushLibPathHandoff; + } +})); + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import type { IPackageJson } from '@rushstack/node-core-library'; +import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; + +import { Rush } from '../../api/Rush'; +import type { RushConfiguration } from '../../api/RushConfiguration'; +import type { RushGlobalFolder } from '../../api/RushGlobalFolder'; +import type { IRushLibPathHandoff } from '../../utilities/RushLibPathHandoff'; +import type { IBuiltInPluginConfiguration } from '../PluginLoader/BuiltInPluginLoader'; +import { PluginManager } from '../PluginManager'; +import type { RushSession } from '../RushSession'; + +const S3_PACKAGE: string = '@rushstack/rush-amazon-s3-build-cache-plugin'; +const AZURE_PACKAGE: string = '@rushstack/rush-azure-storage-build-cache-plugin'; +const HTTP_PACKAGE: string = '@rushstack/rush-http-build-cache-plugin'; + +describe(PluginManager.name, () => { + let folder: string; + let hostNodeModules: string; + + function addBuiltInPlugins( + builtInPluginConfigurations: IBuiltInPluginConfiguration[] + ): IBuiltInPluginConfiguration[] { + // The constructor adds Rush's own built-in plugins to the host's list. + new PluginManager({ + terminal: new Terminal(new StringBufferTerminalProvider()), + rushConfiguration: undefined as unknown as RushConfiguration, + rushSession: {} as RushSession, + builtInPluginConfigurations, + restrictConsoleOutput: false, + rushGlobalFolder: {} as RushGlobalFolder + }); + return builtInPluginConfigurations; + } + + beforeEach(() => { + folder = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-plugin-manager-'))); + hostNodeModules = path.join(folder, 'deploy', 'apps', 'host', 'node_modules'); + const rushLibLink: string = path.join(hostNodeModules, '@microsoft', 'rush-lib'); + fs.mkdirSync(rushLibLink, { recursive: true }); + fs.writeFileSync(path.join(rushLibLink, 'package.json'), '{}'); + for (const packageName of [S3_PACKAGE, AZURE_PACKAGE]) { + fs.mkdirSync(path.join(hostNodeModules, packageName), { recursive: true }); + fs.writeFileSync(path.join(hostNodeModules, packageName, 'package.json'), '{}'); + } + mockRushLibPathHandoff = { + entryPoint: path.join(rushLibLink, 'lib-commonjs', 'index.js'), + packageFolder: rushLibLink + }; + jest.spyOn(Rush, '_rushLibPackageJson', 'get').mockReturnValue({ + name: '@microsoft/rush-lib', + version: '1.0.0', + dependencies: {}, + publishOnlyDependencies: { + [S3_PACKAGE]: 'workspace:*', + [AZURE_PACKAGE]: 'workspace:*', + [HTTP_PACKAGE]: 'workspace:*' + } + } as IPackageJson); + }); + + afterEach(() => { + jest.restoreAllMocks(); + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it('registers publish-only built-in plugins that the host installed next to its rush-lib link', () => { + expect(addBuiltInPlugins([])).toEqual([ + { + packageName: S3_PACKAGE, + pluginName: 'rush-amazon-s3-build-cache-plugin', + pluginPackageFolder: path.join(hostNodeModules, S3_PACKAGE) + }, + { + packageName: AZURE_PACKAGE, + pluginName: 'rush-azure-storage-build-cache-plugin', + pluginPackageFolder: path.join(hostNodeModules, AZURE_PACKAGE) + }, + { + packageName: AZURE_PACKAGE, + pluginName: 'rush-azure-interactive-auth-plugin', + pluginPackageFolder: path.join(hostNodeModules, AZURE_PACKAGE) + } + ]); + }); + + it('does not add a built-in plugin that the host already provides', () => { + const hostAzurePlugin: IBuiltInPluginConfiguration = { + packageName: AZURE_PACKAGE, + pluginName: 'rush-azure-storage-build-cache-plugin', + pluginPackageFolder: path.join(folder, 'dev', AZURE_PACKAGE) + }; + + expect( + addBuiltInPlugins([hostAzurePlugin]).map(({ pluginName, pluginPackageFolder }) => [ + pluginName, + pluginPackageFolder + ]) + ).toEqual([ + ['rush-azure-storage-build-cache-plugin', path.join(folder, 'dev', AZURE_PACKAGE)], + ['rush-amazon-s3-build-cache-plugin', path.join(hostNodeModules, S3_PACKAGE)], + ['rush-azure-interactive-auth-plugin', path.join(hostNodeModules, AZURE_PACKAGE)] + ]); + }); + + it('registers no publish-only plugin when rush-lib is not linked next to it', () => { + const localRushLib: string = path.join(folder, 'deploy', 'libraries', 'rush-lib'); + fs.mkdirSync(localRushLib, { recursive: true }); + mockRushLibPathHandoff = { + entryPoint: path.join(localRushLib, 'lib-commonjs', 'index.js'), + packageFolder: localRushLib + }; + + expect(addBuiltInPlugins([])).toEqual([]); + }); +}); diff --git a/libraries/rush-lib/src/utilities/RushLibPathHandoff.ts b/libraries/rush-lib/src/utilities/RushLibPathHandoff.ts new file mode 100644 index 0000000000..60136347c8 --- /dev/null +++ b/libraries/rush-lib/src/utilities/RushLibPathHandoff.ts @@ -0,0 +1,106 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as path from 'node:path'; + +import { FileSystem } from '@rushstack/node-core-library'; + +const RUSH_LIB_PACKAGE_NAME: string = '@microsoft/rush-lib'; + +/** + * The rush-lib entry point that Rush hands to plugins and child processes as `_RUSH_LIB_PATH`. + */ +export interface IRushLibPathHandoff { + /** The entry point handed off as `_RUSH_LIB_PATH`. */ + readonly entryPoint: string; + /** The rush-lib package folder, spelled the same way as `entryPoint`. */ + readonly packageFolder: string; +} + +export interface IGetRushLibPathHandoffOptions { + /** The folder of the loaded rush-lib package. */ + readonly packageFolder: string; + /** The loaded rush-lib entry point, inside `packageFolder`. */ + readonly entryPoint: string; + /** Scripts whose own `node_modules` folders may link rush-lib, such as `process.argv[1]`. */ + readonly hostScriptPaths: ReadonlyArray; + /** The inherited `_RUSH_LIB_PATH` value, if any. */ + readonly inheritedEntryPoint: string | undefined; +} + +/** + * Returns the first `node_modules/` folder that Node.js module lookup finds from `startFolder`. + * Symbolic links are not followed, so the result keeps the spelling of `startFolder`. + */ +export function findNodeModulesPackageFolder(startFolder: string, packageName: string): string | undefined { + let folder: string = path.resolve(startFolder); + for (;;) { + if (path.basename(folder) !== 'node_modules') { + const packageFolder: string = path.join(folder, 'node_modules', packageName); + if (FileSystem.exists(path.join(packageFolder, 'package.json'))) { + return packageFolder; + } + } + const parentFolder: string = path.dirname(folder); + if (parentFolder === folder) { + return undefined; + } + folder = parentFolder; + } +} + +/** + * Chooses how to spell the rush-lib entry point in `_RUSH_LIB_PATH`. + * + * @remarks + * Plugins resolve `@microsoft/rush-lib` and its dependencies by name from `_RUSH_LIB_PATH`. That needs a + * `node_modules/@microsoft/rush-lib` folder above the path. Installed packages always have one, so their real + * path is used unchanged. A local project folder, such as one in a `rush deploy` output, doesn't. Its entry + * point is spelled through the `node_modules` link of the host script that loaded it, or else of an inherited + * `_RUSH_LIB_PATH` that points at the same file. + */ +export function getRushLibPathHandoff(options: IGetRushLibPathHandoffOptions): IRushLibPathHandoff { + const { packageFolder, entryPoint, hostScriptPaths, inheritedEntryPoint } = options; + const realPackageFolder: string | undefined = tryGetRealPath(packageFolder); + const findRushLibLink: (fromPath: string) => string | undefined = (fromPath: string) => { + const linkFolder: string | undefined = findNodeModulesPackageFolder( + path.dirname(fromPath), + RUSH_LIB_PACKAGE_NAME + ); + return linkFolder !== undefined && tryGetRealPath(linkFolder) === realPackageFolder + ? linkFolder + : undefined; + }; + + if (realPackageFolder === undefined || findRushLibLink(entryPoint) !== undefined) { + return { entryPoint, packageFolder }; + } + + const relativeEntryPoint: string = path.relative(packageFolder, entryPoint); + for (const hostScriptPath of hostScriptPaths) { + const realHostScriptPath: string | undefined = hostScriptPath ? tryGetRealPath(hostScriptPath) : undefined; + const linkFolder: string | undefined = realHostScriptPath ? findRushLibLink(realHostScriptPath) : undefined; + if (linkFolder !== undefined) { + return { entryPoint: path.join(linkFolder, relativeEntryPoint), packageFolder: linkFolder }; + } + } + + if (inheritedEntryPoint && tryGetRealPath(inheritedEntryPoint) === tryGetRealPath(entryPoint)) { + const linkFolder: string | undefined = findRushLibLink(inheritedEntryPoint); + const spelledEntryPoint: string | undefined = + linkFolder !== undefined ? path.join(linkFolder, relativeEntryPoint) : undefined; + if (linkFolder !== undefined && spelledEntryPoint === path.resolve(inheritedEntryPoint)) { + return { entryPoint: spelledEntryPoint, packageFolder: linkFolder }; + } + } + + return { entryPoint, packageFolder }; +} + +function tryGetRealPath(fileOrFolderPath: string): string | undefined { + try { + return FileSystem.getRealPath(fileOrFolderPath); + } catch { + return undefined; + } +} diff --git a/libraries/rush-lib/src/utilities/SetRushLibPath.ts b/libraries/rush-lib/src/utilities/SetRushLibPath.ts index ffc527ff22..f032e043fd 100644 --- a/libraries/rush-lib/src/utilities/SetRushLibPath.ts +++ b/libraries/rush-lib/src/utilities/SetRushLibPath.ts @@ -4,10 +4,26 @@ import { PackageJsonLookup } from '@rushstack/node-core-library'; import { EnvironmentVariableNames } from '../api/EnvironmentConfiguration'; +import { getRushLibPathHandoff, type IRushLibPathHandoff } from './RushLibPathHandoff'; -const rootDir: string | undefined = PackageJsonLookup.instance.tryGetPackageFolderFor(__dirname); -if (rootDir) { +function setRushLibPath(): IRushLibPathHandoff | undefined { + const rootDir: string | undefined = PackageJsonLookup.instance.tryGetPackageFolderFor(__dirname); + if (!rootDir) { + return undefined; + } // Route to the 'main' field of package.json const rushLibIndex: string = require.resolve(rootDir, { paths: [] }); - process.env[EnvironmentVariableNames._RUSH_LIB_PATH] = rushLibIndex; + const handoff: IRushLibPathHandoff = getRushLibPathHandoff({ + packageFolder: rootDir, + entryPoint: rushLibIndex, + hostScriptPaths: [process.argv[1]], + inheritedEntryPoint: process.env[EnvironmentVariableNames._RUSH_LIB_PATH] + }); + process.env[EnvironmentVariableNames._RUSH_LIB_PATH] = handoff.entryPoint; + return handoff; } + +/** + * The `_RUSH_LIB_PATH` value that this rush-lib set when it was loaded. + */ +export const rushLibPathHandoff: IRushLibPathHandoff | undefined = setRushLibPath(); diff --git a/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts b/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts new file mode 100644 index 0000000000..0b3437cb10 --- /dev/null +++ b/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts @@ -0,0 +1,200 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as childProcess from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + findNodeModulesPackageFolder, + getRushLibPathHandoff, + type IRushLibPathHandoff +} from '../RushLibPathHandoff'; + +const ENTRY_POINT_SUBPATH: string = path.join('lib-commonjs', 'index.js'); + +describe(getRushLibPathHandoff.name, () => { + let folder: string; + let localRushLib: string; + let hostScript: string; + let hostLink: string; + + function writePackage(packageFolder: string, name: string): void { + fs.mkdirSync(path.join(packageFolder, 'lib-commonjs'), { recursive: true }); + fs.writeFileSync( + path.join(packageFolder, 'package.json'), + JSON.stringify({ name, version: '1.0.0', main: './lib-commonjs/index.js' }) + ); + fs.writeFileSync(path.join(packageFolder, ENTRY_POINT_SUBPATH), 'module.exports = {};'); + } + + function link(target: string, newLinkPath: string): void { + fs.mkdirSync(path.dirname(newLinkPath), { recursive: true }); + fs.symlinkSync(target, newLinkPath, 'junction'); + } + + // Resolves the way plugins do, from a process outside of any rush-lib package scope. + function resolveByName(request: string, fromPath: string): string { + return childProcess + .execFileSync( + process.execPath, + ['-e', 'process.stdout.write(require.resolve(process.argv[1], { paths: [process.argv[2]] }))'].concat( + request, + fromPath + ), + { cwd: folder, encoding: 'utf8' } + ) + .toString(); + } + + beforeEach(() => { + folder = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-lib-path-handoff-'))); + // A "rush deploy" layout: rush-lib is a local project folder, and each host links it. + localRushLib = path.join(folder, 'deploy', 'libraries', 'rush-lib'); + writePackage(localRushLib, '@microsoft/rush-lib'); + writePackage(path.join(folder, 'deploy', 'libraries', 'node-core-library'), '@rushstack/node-core-library'); + link( + path.join(folder, 'deploy', 'libraries', 'node-core-library'), + path.join(localRushLib, 'node_modules', '@rushstack', 'node-core-library') + ); + hostScript = path.join(folder, 'deploy', 'apps', 'host', 'bin', 'host'); + fs.mkdirSync(path.dirname(hostScript), { recursive: true }); + fs.writeFileSync(hostScript, ''); + hostLink = path.join(folder, 'deploy', 'apps', 'host', 'node_modules', '@microsoft', 'rush-lib'); + link(localRushLib, hostLink); + }); + + afterEach(() => { + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it('keeps the real path of an installed rush-lib', () => { + const installedRushLib: string = path.join(folder, 'install', 'node_modules', '@microsoft', 'rush-lib'); + writePackage(installedRushLib, '@microsoft/rush-lib'); + const entryPoint: string = path.join(installedRushLib, ENTRY_POINT_SUBPATH); + + expect( + getRushLibPathHandoff({ + packageFolder: installedRushLib, + entryPoint, + hostScriptPaths: [hostScript], + inheritedEntryPoint: path.join(hostLink, ENTRY_POINT_SUBPATH) + }) + ).toEqual({ entryPoint, packageFolder: installedRushLib }); + }); + + it('spells a local rush-lib through the node_modules link of the host script', () => { + const handoff: IRushLibPathHandoff = getRushLibPathHandoff({ + packageFolder: localRushLib, + entryPoint: path.join(localRushLib, ENTRY_POINT_SUBPATH), + hostScriptPaths: [undefined, hostScript], + inheritedEntryPoint: undefined + }); + + expect(handoff).toEqual({ entryPoint: path.join(hostLink, ENTRY_POINT_SUBPATH), packageFolder: hostLink }); + // This is how plugins find rush-lib and its dependencies from _RUSH_LIB_PATH. + expect(fs.realpathSync.native(resolveByName('@microsoft/rush-lib/package.json', handoff.entryPoint))).toBe( + path.join(localRushLib, 'package.json') + ); + expect( + fs.realpathSync.native(resolveByName('@rushstack/node-core-library/package.json', handoff.packageFolder)) + ).toBe(path.join(folder, 'deploy', 'libraries', 'node-core-library', 'package.json')); + }); + + it('follows a symbolic link to the host script before looking for its node_modules folder', () => { + const scriptAlias: string = path.join(folder, 'bin', 'host'); + fs.mkdirSync(path.dirname(scriptAlias), { recursive: true }); + fs.symlinkSync(hostScript, scriptAlias); + + expect( + getRushLibPathHandoff({ + packageFolder: localRushLib, + entryPoint: path.join(localRushLib, ENTRY_POINT_SUBPATH), + hostScriptPaths: [scriptAlias], + inheritedEntryPoint: undefined + }).packageFolder + ).toBe(hostLink); + }); + + it('ignores a host whose own rush-lib is a different package', () => { + const otherHostScript: string = path.join(folder, 'other', 'bin', 'host'); + fs.mkdirSync(path.dirname(otherHostScript), { recursive: true }); + fs.writeFileSync(otherHostScript, ''); + writePackage(path.join(folder, 'other', 'node_modules', '@microsoft', 'rush-lib'), '@microsoft/rush-lib'); + // A link further up would be shadowed by the host's own rush-lib, so it must not be used either. + link(localRushLib, path.join(folder, 'node_modules', '@microsoft', 'rush-lib')); + const entryPoint: string = path.join(localRushLib, ENTRY_POINT_SUBPATH); + + expect( + getRushLibPathHandoff({ + packageFolder: localRushLib, + entryPoint, + hostScriptPaths: [otherHostScript, path.join(folder, 'missing', 'script')], + inheritedEntryPoint: undefined + }) + ).toEqual({ entryPoint, packageFolder: localRushLib }); + }); + + it('keeps an inherited spelling of the same entry point', () => { + const inheritedEntryPoint: string = path.join(hostLink, ENTRY_POINT_SUBPATH); + + expect( + getRushLibPathHandoff({ + packageFolder: localRushLib, + entryPoint: path.join(localRushLib, ENTRY_POINT_SUBPATH), + hostScriptPaths: [path.join(folder, 'missing', 'script')], + inheritedEntryPoint + }) + ).toEqual({ entryPoint: inheritedEntryPoint, packageFolder: hostLink }); + }); + + it('replaces an inherited path that names a different engine', () => { + const otherRushLib: string = path.join(folder, 'other', 'node_modules', '@microsoft', 'rush-lib'); + writePackage(otherRushLib, '@microsoft/rush-lib'); + const entryPoint: string = path.join(localRushLib, ENTRY_POINT_SUBPATH); + + expect( + getRushLibPathHandoff({ + packageFolder: localRushLib, + entryPoint, + hostScriptPaths: [], + inheritedEntryPoint: path.join(otherRushLib, ENTRY_POINT_SUBPATH) + }) + ).toEqual({ entryPoint, packageFolder: localRushLib }); + }); +}); + +describe(findNodeModulesPackageFolder.name, () => { + let folder: string; + + beforeEach(() => { + folder = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-node-modules-lookup-'))); + }); + + afterEach(() => { + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it('finds a package next to a linked package without following the link', () => { + const target: string = path.join(folder, 'target'); + fs.mkdirSync(target, { recursive: true }); + fs.writeFileSync(path.join(target, 'package.json'), '{}'); + const plugin: string = path.join(folder, 'host', 'node_modules', '@scope', 'plugin'); + fs.mkdirSync(plugin, { recursive: true }); + fs.writeFileSync(path.join(plugin, 'package.json'), '{}'); + const linkedPackage: string = path.join(folder, 'host', 'node_modules', '@scope', 'linked'); + fs.symlinkSync(target, linkedPackage, 'junction'); + + expect(findNodeModulesPackageFolder(linkedPackage, '@scope/plugin')).toBe(plugin); + expect(findNodeModulesPackageFolder(target, '@scope/plugin')).toBeUndefined(); + }); + + it('does not look for node_modules inside a node_modules folder', () => { + const nested: string = path.join(folder, 'node_modules', 'node_modules', 'pkg'); + fs.mkdirSync(nested, { recursive: true }); + fs.writeFileSync(path.join(nested, 'package.json'), '{}'); + + expect(findNodeModulesPackageFolder(path.join(folder, 'node_modules', 'a'), 'pkg')).toBeUndefined(); + }); +}); From c5158102b26b1f4b6ee0717c01a83026ded675d1 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 19:29:42 +0000 Subject: [PATCH 034/265] [rush-client-core] Wait for a daemon a live process is still starting instead of running Rush in-process Swarm integration step 19; original commit a81a00520d (merge of o07/startup-fallback at 668753ad4d). Scope: task 95. Brings 647a8bf902 [rush-client-core] and 668753ad4d [rush-cli-client]. Gate: ch01 GATE OK board 1731 (tree e4a00622d4); burst: t01 board 1716. Commits folded into this step (2): - 647a8bf902 [rush-client-core] Wait for a daemon that a live process is still starting before permitting in-process Rush - 668753ad4d [rush-cli-client] Don't run Rush in-process while daemon startup is still pending Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 14 + apps/rush-cli-client/src/launchClient.ts | 19 +- .../src/test/startupPendingFallback.test.ts | 109 ++++++ ...tup-pending-fallback_2026-09-28-16-58.json | 11 + ...tup-pending-fallback_2026-09-28-16-58.json | 11 + common/reviews/api/rush-client-core.api.md | 13 + docs/rush/dogfooding-rush-daemon.md | 8 +- libraries/rush-client-core/README.md | 15 + .../src/connectOrAwaitDaemonStartup.ts | 136 ++++++++ libraries/rush-client-core/src/index.ts | 5 + .../test/connectOrAwaitDaemonStartup.test.ts | 311 ++++++++++++++++++ .../src/test/fixtures/awaitingStarter.ts | 58 ++++ .../src/test/fixtures/startLockHolder.ts | 28 ++ 13 files changed, 733 insertions(+), 5 deletions(-) create mode 100644 apps/rush-cli-client/src/test/startupPendingFallback.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/startup-pending-fallback_2026-09-28-16-58.json create mode 100644 common/changes/@rushstack/rush-client-core/startup-pending-fallback_2026-09-28-16-58.json create mode 100644 libraries/rush-client-core/src/connectOrAwaitDaemonStartup.ts create mode 100644 libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts create mode 100644 libraries/rush-client-core/src/test/fixtures/awaitingStarter.ts create mode 100644 libraries/rush-client-core/src/test/fixtures/startLockHolder.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 96c9d681da..fcbfaa9d06 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -42,6 +42,20 @@ Routing precedence: 3. `RUSH_DAEMON` overrides `rush.json`'s `daemon.enabled`; the default is false. 4. Auto-start is considered only after selecting daemon execution. +When the selected daemon cannot be reached or started, an ordinary invocation prints the reason and runs +in-process (`rush-client: Using in-process Rush.`). A startup failure while a live process can +still make the daemon ready is the exception: a process that listens at the endpoint but does not +complete hello/ping in time, a startup helper that still waits for its daemon, or another client that +holds the start mutex, as in a burst of clients that all find no daemon. In-process Rush would take the +repository lock, and the daemon would then reject the requests it serves with "Another Rush command is +already running in this repository." Instead, the client keeps trying for one more startup deadline +(15 seconds, so about 30 seconds in all) and uses the daemon once it is ready. It says so when it starts +waiting (`rush-client: The daemon is not ready yet. , so this command waits up to 15 s more +for it instead of running Rush in-process.`; agent output shows "rushd is still starting; waiting for it" +as the progress phase). If the daemon is still not ready, the command exits with code 1. The message +gives the startup error with its `--no-daemon` hint, then the process that is still live, "so Rush was +not run in-process", and a pointer to `rush-client daemon status`. + `--no-wait` fails immediately when daemon admission is unavailable. `--wait-timeout SECONDS` (or `--wait-timeout=SECONDS`) overrides the configured queue timeout; finite nonnegative decimal seconds up to 2147483.647 are accepted and diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 6ae500af55..00cd328d1e 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -13,7 +13,7 @@ import { JsonFile } from '@rushstack/node-core-library'; import { DaemonClientError, captureDaemonRequest, - connectOrStartDaemonAsync, + connectOrAwaitDaemonStartupAsync, executeWithDaemonRestartAsync, type DaemonClient, type DaemonClientOutcome, @@ -141,7 +141,22 @@ export async function launchClientAsync( verbosity } }; - client = await connectOrStartDaemonAsync(connection); + // While a live daemon or starter can still make the daemon ready, a startup failure rejects with a + // DaemonStartupPendingError, which is not a DaemonClientError, so Rush does not run in-process next to it. + client = await connectOrAwaitDaemonStartupAsync({ + ...connection, + onAwaitStartup: (owner: string, waitMs: number): void => { + if (agentRenderer) { + agentRenderer.setPhase('rushd is still starting; waiting for it'); + return; + } + const seconds: number = Math.round(waitMs / 1000); + process.stderr.write( + `rush-client: The daemon is not ready yet. ${owner}, so this command waits up to ${seconds} s ` + + 'more for it instead of running Rush in-process.\n' + ); + } + }); } catch (error) { if ( !(error instanceof DaemonClientError) && diff --git a/apps/rush-cli-client/src/test/startupPendingFallback.test.ts b/apps/rush-cli-client/src/test/startupPendingFallback.test.ts new file mode 100644 index 0000000000..934fad1ace --- /dev/null +++ b/apps/rush-cli-client/src/test/startupPendingFallback.test.ts @@ -0,0 +1,109 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('@microsoft/rush/lib/start', () => ({})); +jest.mock('@rushstack/rush-client-core', () => ({ + ...jest.requireActual('@rushstack/rush-client-core'), + connectOrAwaitDaemonStartupAsync: jest.fn() +})); + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + DaemonClientError, + DaemonStartupPendingError, + connectOrAwaitDaemonStartupAsync +} from '@rushstack/rush-client-core'; + +import { AgentProgressRenderer } from '../AgentProgressRenderer'; +import * as connectionOptions from '../daemonConnectionOptions'; +import { launchClientAsync } from '../launchClient'; + +describe('a daemon startup failure', () => { + let folder: string; + let originalArgv: string[]; + let originalEnvironment: NodeJS.ProcessEnv; + let output: jest.SpyInstance; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-client-startup-pending-')); + originalArgv = process.argv; + originalEnvironment = process.env; + fs.writeFileSync( + path.join(folder, 'rush.json'), + JSON.stringify({ rushVersion: '5.178.1', pnpmVersion: '10.27.0', projects: [] }) + ); + jest.spyOn(connectionOptions, 'getDaemonConnectionOptionsAsync').mockResolvedValue({ + paths: { + runtimeDir: folder, + socketPath: path.join(folder, 'd.sock'), + lockfilePath: path.join(folder, 'daemon.pid.json') + } + }); + output = jest.spyOn(process.stderr, 'write').mockReturnValue(true); + jest.spyOn(process, 'cwd').mockReturnValue(folder); + process.argv = [process.execPath, 'rush-client', 'build', '--to', 'project']; + process.env = { ...originalEnvironment, CI: 'false', RUSH_DAEMON: '1', RUSH_REPORTER: 'legacy' }; + delete process.env.RUSH_PREVIEW_VERSION; + }); + + afterEach(() => { + process.argv = originalArgv; + process.env = originalEnvironment; + jest.restoreAllMocks(); + jest.mocked(connectOrAwaitDaemonStartupAsync).mockReset(); + fs.rmSync(folder, { recursive: true }); + }); + + it('fails the command instead of running Rush in-process while daemon startup is pending', async () => { + jest + .mocked(connectOrAwaitDaemonStartupAsync) + .mockRejectedValue(new DaemonStartupPendingError('Another client is still starting the daemon.')); + await expect(launchClientAsync(false)).rejects.toThrow('Another client is still starting the daemon.'); + expect(connectOrAwaitDaemonStartupAsync).toHaveBeenCalledTimes(1); + expect(process.argv).toEqual([process.execPath, 'rush-client', 'build', '--to', 'project']); + expect(output).not.toHaveBeenCalledWith(expect.stringContaining('Using in-process Rush')); + }); + + it('says why it keeps waiting for a daemon that is still starting', async () => { + const pending: DaemonStartupPendingError = new DaemonStartupPendingError('Still starting.'); + jest.mocked(connectOrAwaitDaemonStartupAsync).mockImplementation(async (options) => { + options.onAwaitStartup?.('Its startup helper (PID 42) is still waiting for the daemon', 15000); + throw pending; + }); + await expect(launchClientAsync(false)).rejects.toBe(pending); + expect(output).toHaveBeenCalledWith( + 'rush-client: The daemon is not ready yet. Its startup helper (PID 42) is still waiting for the daemon, ' + + 'so this command waits up to 15 s more for it instead of running Rush in-process.\n' + ); + + // Agent output shows it as the phase of its progress line instead. + output.mockClear(); + const write: jest.Mock = jest.fn(); + const renderer: AgentProgressRenderer = new AgentProgressRenderer({ + commandName: 'build', + isTTY: false, + columns: 80, + write + }); + await expect(launchClientAsync(false, renderer)).rejects.toBe(pending); + expect(write).toHaveBeenCalledWith(expect.stringContaining('rushd is still starting; waiting for it')); + expect(output).not.toHaveBeenCalledWith(expect.stringContaining('The daemon is not ready yet')); + }); + + it('still runs Rush in-process after a startup failure that no live process owns', async () => { + jest + .mocked(connectOrAwaitDaemonStartupAsync) + .mockRejectedValue(new DaemonClientError('startupFailed', 'No ready daemon; auto-start is disabled.')); + await launchClientAsync(false); + expect(process.argv[1]).toBe( + path.join(path.dirname(require.resolve('@microsoft/rush/package.json')), 'bin/rush') + ); + expect(process.argv.slice(2)).toEqual(['build', '--to', 'project']); + expect(output).toHaveBeenCalledWith( + 'rush-client: No ready daemon; auto-start is disabled. Using in-process Rush.\n' + ); + }); +}); diff --git a/common/changes/@rushstack/rush-cli-client/startup-pending-fallback_2026-09-28-16-58.json b/common/changes/@rushstack/rush-cli-client/startup-pending-fallback_2026-09-28-16-58.json new file mode 100644 index 0000000000..d815203aa5 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/startup-pending-fallback_2026-09-28-16-58.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "When daemon startup fails while a listener, a startup helper, or another starting client can still make the daemon ready, a command says so and keeps trying for one more startup deadline, and then fails with exit code 1, instead of running Rush in-process, where it would take the repository lock from the requests that the daemon serves.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/startup-pending-fallback_2026-09-28-16-58.json b/common/changes/@rushstack/rush-client-core/startup-pending-fallback_2026-09-28-16-58.json new file mode 100644 index 0000000000..0252cfc12d --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/startup-pending-fallback_2026-09-28-16-58.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Add `connectOrAwaitDaemonStartupAsync()` and `DaemonStartupPendingError` for callers that run Rush in-process when no daemon is available: after a startup failure, they keep trying for one more startup deadline while a listener, a running startup helper, or another starting client can still make the daemon ready, and then get an error that does not permit running in-process. The optional `onAwaitStartup` callback names that process when the extra wait begins.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 4a07758b95..c28959d376 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -22,6 +22,9 @@ export function assertDaemonRuntimeFolderIsPrivate(paths: IDaemonPaths): void; // @beta export function captureDaemonRequest(options: ICaptureDaemonRequestOptions): IDaemonRequestEnvelope; +// @beta +export function connectOrAwaitDaemonStartupAsync(options: IConnectOrAwaitDaemonStartupOptions): Promise; + // @beta export function connectOrStartDaemonAsync(options: IConnectOrStartDaemonOptions): Promise; @@ -63,6 +66,11 @@ export type DaemonClientOutcome = { // @beta export type DaemonStartupHelperState = 'running' | 'exited' | 'unknown'; +// @beta +export class DaemonStartupPendingError extends Error { + constructor(message: string, options?: ErrorOptions); +} + // @beta export function executeWithDaemonRestartAsync(client: DaemonClient, connection: IConnectOrStartDaemonOptions, execution: IDaemonClientExecuteOptions): Promise; @@ -77,6 +85,11 @@ export interface ICaptureDaemonRequestOptions extends Omit void; +} + // @beta export interface IConnectOrStartDaemonOptions extends Omit { readonly abortSignal?: AbortSignal; diff --git a/docs/rush/dogfooding-rush-daemon.md b/docs/rush/dogfooding-rush-daemon.md index 9a5c3d282d..77a9fb7106 100644 --- a/docs/rush/dogfooding-rush-daemon.md +++ b/docs/rush/dogfooding-rush-daemon.md @@ -155,9 +155,11 @@ node common/scripts/install-run-rush.js deploy --scenario rush-daemon-dogfood -- ``` On Windows, the first daemon start from a new or refreshed snapshot can take longer than the client's 15-second -startup deadline while Windows scans the newly written files. That request then uses native Rush (with a -`rush-client: ... Using in-process Rush.` message), but the daemon finishes starting in the background and the next -request uses it. +startup deadline while Windows scans the newly written files. Because the startup helper is still waiting for the +daemon, the client says so and keeps trying for another 15 seconds, then uses the daemon once it is ready. If it is +still not ready, the request fails with exit code 1 and a `rush-client: ...` message that says Rush was not run +in-process, rather than running native Rush next to the starting daemon. The daemon finishes starting in the +background, so rerun the command (`rush-client daemon status` shows when it is ready). The daemon's identity is the canonical repository root plus the selected Rush version, so each checkout or worktree has its own daemon, and `rush-client daemon ...` commands address the daemon for the checkout that diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 3d3c57504a..b611c372fc 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -119,6 +119,21 @@ Socket resets and broken pipes during hello/ping readiness are retried within th a closing host may still retain its listener while joining owned resources. This applies only before `requestStart`; a connection failure after execution begins still never permits replay. +`connectOrAwaitDaemonStartupAsync()` wraps `connectOrStartDaemonAsync()` for a caller that runs Rush +in-process when no daemon is available, as the CLI client does. A `startupFailed` or `timeout` error +alone does not make that safe while a live process can still make the daemon ready: a listener at the +endpoint that did not complete hello/ping in time, a recorded startup helper that is still running, or +another client that holds the start mutex. In-process Rush would take the repository lock that the +daemon's requests need. The wrapper instead keeps connecting (and starting, through the same mutex and +reservation) until one more startup deadline has passed. If the daemon is still not ready, it rejects +with `DaemonStartupPendingError`, which is not a `DaemonClientError`: the message is the startup error +followed by the process that is still live, and `cause` is that error. When none remains, it rejects at +once with the original `DaemonClientError`, for example when auto-start is disabled and nothing listens, +or when the helper has exited. An ownership record alone does not count, because a daemon publishes it +only after binding. Other errors, such as `versionMismatch`, pass through unchanged. Before it keeps +waiting, it calls the optional `onAwaitStartup(owner, waitMs)` once, with the live process and the +remaining wait, so that the caller can say why the command has not started yet. + `getDaemonLogFilePath(paths)` is the shared stable path used by both the launcher and the CLI's local `daemon logs` reader. Child stdout/stderr are appended across restarts, including startup failures; the parent always closes its descriptor diff --git a/libraries/rush-client-core/src/connectOrAwaitDaemonStartup.ts b/libraries/rush-client-core/src/connectOrAwaitDaemonStartup.ts new file mode 100644 index 0000000000..7364009944 --- /dev/null +++ b/libraries/rush-client-core/src/connectOrAwaitDaemonStartup.ts @@ -0,0 +1,136 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import type { DaemonClient } from './DaemonClient'; +import { DaemonClientError } from './DaemonClientError'; +import { isEndpointUnboundAsync } from './DaemonOwnership'; +import { readDaemonStartupReservation, type IDaemonStartupReservation } from './DaemonStartup'; +import { getStartupHelperState } from './DaemonStartupReservation'; +import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from './connectOrStartDaemon'; +import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; + +/** Matches the default of {@link IConnectOrStartDaemonOptions.startupTimeoutMs}. */ +const DEFAULT_STARTUP_TIMEOUT_MS: number = 15000; +/** Keeps a retry that fails at once from spinning until the deadline. */ +const RETRY_DELAY_MS: number = 100; + +/** + * Daemon startup did not finish in time, but a live process can still make the daemon ready. Unlike a + * {@link DaemonClientError}, this does not mean that running Rush in-process is safe: it would compete with + * that daemon for the repository. + * @beta + */ +export class DaemonStartupPendingError extends Error { + public constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = 'DaemonStartupPendingError'; + } +} + +/** + * Options for {@link connectOrAwaitDaemonStartupAsync}. + * @beta + */ +export interface IConnectOrAwaitDaemonStartupOptions extends IConnectOrStartDaemonOptions { + /** + * Called at most once, when startup did not finish in time but a live process can still make the daemon + * ready, just before this waits up to `waitMs` more for it. `owner` describes that process, for example + * "Its startup helper (PID 123) is still waiting for the daemon". A caller can say why it is still waiting. + */ + onAwaitStartup?: (owner: string, waitMs: number) => void; +} + +/** + * {@link connectOrStartDaemonAsync} for a caller that runs Rush in-process when the daemon is unavailable. + * @remarks When startup fails while a live process can still make the daemon ready (a listener at the endpoint, + * a startup helper that still waits for its daemon, or another client that holds the start mutex), this retries + * for one more startup deadline. If the daemon is still not ready, it rejects with a + * {@link DaemonStartupPendingError}. It rejects with a {@link DaemonClientError} only when no such process + * remains, so that the caller can run Rush in-process without competing with a daemon for the repository. + * @beta + */ +export async function connectOrAwaitDaemonStartupAsync( + options: IConnectOrAwaitDaemonStartupOptions +): Promise { + try { + return await connectOrStartDaemonAsync(options); + } catch (error) { + if (!isStartupFailure(error)) throw error; + return await awaitLiveStartupAsync(options, error); + } +} + +async function awaitLiveStartupAsync( + options: IConnectOrAwaitDaemonStartupOptions, + firstError: DaemonClientError +): Promise { + const deadline: number = Date.now() + (options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS); + let lastError: DaemonClientError = firstError; + for (let attempt: number = 0; ; attempt++) { + const owner: string | undefined = await findLiveStartupOwnerAsync(options.paths); + if (owner === undefined) throw lastError; + if (Date.now() >= deadline) { + throw new DaemonStartupPendingError( + `${lastError.message} ${owner}, so Rush was not run in-process, where it would compete with that daemon for the repository. "rush-client daemon status" reports when the daemon is ready.`, + { cause: lastError } + ); + } + if (attempt === 0) { + options.onAwaitStartup?.(owner, deadline - Date.now()); + } else { + await delayAsync(Math.min(RETRY_DELAY_MS, Math.max(1, deadline - Date.now())), undefined, { + signal: options.abortSignal + }); + } + try { + return await connectOrStartDaemonAsync({ + ...options, + startupTimeoutMs: Math.max(1, deadline - Date.now()) + }); + } catch (error) { + if (!isStartupFailure(error)) throw error; + lastError = error; + } + } +} + +function isStartupFailure(error: unknown): error is DaemonClientError { + return error instanceof DaemonClientError && (error.code === 'startupFailed' || error.code === 'timeout'); +} + +/** + * Describes a live process that can still make this workspace's daemon ready, or returns `undefined`. + * An ownership record alone is not such evidence: a daemon publishes it only after it binds, so a record + * next to an unbound endpoint belongs to a daemon that is shutting down or to a reused PID. + */ +async function findLiveStartupOwnerAsync(paths: IDaemonPaths): Promise { + if (!fs.existsSync(path.dirname(paths.lockfilePath))) return undefined; + if (!(await isEndpointUnboundAsync(paths.socketPath))) { + return `A process listens at ${paths.socketPath} but was not ready in time`; + } + let reservation: IDaemonStartupReservation | undefined; + try { + reservation = readDaemonStartupReservation(paths); + } catch { + // An unreadable reservation is no evidence of a live helper. + } + if (reservation?.helper && getStartupHelperState(reservation) === 'running') { + return `Its startup helper (PID ${reservation.helper.pid}) is still waiting for the daemon`; + } + let lock: IStartupLock | undefined; + try { + lock = await tryAcquireStartupLockAsync(paths); + } catch { + // A start mutex that cannot be checked is no evidence of a live starter. + return undefined; + } + if (!lock) return 'Another client is still starting the daemon'; + await lock.releaseAsync(); + return undefined; +} diff --git a/libraries/rush-client-core/src/index.ts b/libraries/rush-client-core/src/index.ts index 6f3fbd5e3c..c1ae540535 100644 --- a/libraries/rush-client-core/src/index.ts +++ b/libraries/rush-client-core/src/index.ts @@ -27,6 +27,11 @@ export { type IDaemonStartupReservationInfo } from './DaemonStartupReservation'; export { executeWithDaemonRestartAsync } from './executeWithDaemonRestart'; +export { + connectOrAwaitDaemonStartupAsync, + DaemonStartupPendingError, + type IConnectOrAwaitDaemonStartupOptions +} from './connectOrAwaitDaemonStartup'; export { connectOrStartDaemonAsync, requestDaemonShutdownAsync, diff --git a/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts b/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts new file mode 100644 index 0000000000..c120d3416b --- /dev/null +++ b/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts @@ -0,0 +1,311 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import * as fs from 'node:fs'; +import * as net from 'node:net'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { captureDaemonRequest } from '../captureDaemonRequest'; +import { DaemonClientError } from '../DaemonClientError'; +import { getDaemonStartupFilePath } from '../DaemonStartup'; +import { DaemonStartupPendingError, connectOrAwaitDaemonStartupAsync } from '../connectOrAwaitDaemonStartup'; +import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from '../connectOrStartDaemon'; +import { removeTestFolderAsync, waitForTestProcessExitAsync } from './TestProcessExit'; + +interface IAwaitingResult { + readonly kind: 'connected' | 'pending' | 'fallback' | 'error'; + readonly pid?: number; + readonly message?: string; + readonly elapsedMs: number; + readonly notices: { owner: string; waitMs: number }[]; +} + +describe('connectOrAwaitDaemonStartupAsync', () => { + let folder: string; + let paths: IDaemonPaths; + let options: IConnectOrStartDaemonOptions; + let children: ChildProcess[]; + + beforeEach(() => { + children = []; + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-client-await-')); + paths = { + runtimeDir: folder, + socketPath: + process.platform === 'win32' + ? `\\\\.\\pipe\\rush-client-await-${path.basename(folder)}` + : path.join(folder, 'd.sock'), + lockfilePath: path.join(folder, 'daemon.pid.json') + }; + const environment = captureDaemonRequest({ + argv: [], + commandName: 'test', + commandOrigin: 'custom', + cwd: folder, + environment: { PATH: process.env.PATH, SystemRoot: process.env.SystemRoot }, + terminal: { isTTY: false, supportsColor: false } + }).environment; + options = { + paths, + expectedDaemonVersion: 'fixture', + startupTimeoutMs: 7000, + startCommand: { + command: process.execPath, + args: [path.join(__dirname, 'fixtures/daemon.js'), JSON.stringify(paths)], + cwd: folder, + environment + } + }; + }); + + afterEach(async () => { + for (const child of children) { + if (child.exitCode === null && child.signalCode === null) { + const closed: Promise = once(child, 'close'); + child.kill('SIGKILL'); + await closed; + } + } + if (fs.existsSync(path.join(folder, 'starts'))) { + fs.writeFileSync(path.join(folder, 'stop'), ''); + const pids: string[] = fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n'); + await Promise.all(pids.map((pid) => waitForTestProcessExitAsync(Number(pid)))); + } + if (fs.existsSync(path.join(folder, 'parents'))) { + const parents = new Set(fs.readFileSync(path.join(folder, 'parents'), 'utf8').trim().split('\n')); + await Promise.all([...parents].map((pid) => waitForTestProcessExitAsync(Number(pid)))); + } + await removeTestFolderAsync(folder); + }); + + function run( + fixture: string, + args: string[] + ): { child: ChildProcess; result: Promise<{ code: number | null; stdout: string; stderr: string }> } { + const child: ChildProcess = spawn( + process.execPath, + [path.join(__dirname, 'fixtures', fixture), ...args], + { + stdio: ['ignore', 'pipe', 'pipe'] + } + ); + children.push(child); + let stdout: string = ''; + let stderr: string = ''; + child.stdout!.on('data', (chunk: Buffer) => { + stdout += chunk.toString(); + }); + child.stderr!.on('data', (chunk: Buffer) => { + stderr += chunk.toString(); + }); + return { child, result: once(child, 'close').then(([code]) => ({ code, stdout, stderr })) }; + } + + async function runAwaitingAsync( + startOptions: IConnectOrStartDaemonOptions, + signalFolder?: string + ): Promise { + const { code, stdout, stderr } = await run( + 'awaitingStarter.js', + signalFolder ? [JSON.stringify(startOptions), signalFolder] : [JSON.stringify(startOptions)] + ).result; + expect({ code, stderr }).toEqual({ code: 0, stderr: '' }); + return JSON.parse(stdout); + } + + async function waitForFileAsync(filePath: string): Promise { + const deadline: number = Date.now() + 10000; + while (!fs.existsSync(filePath) && Date.now() < deadline) await delayAsync(20); + return fs.readFileSync(filePath, 'utf8'); + } + + async function waitForLinesAsync(filePath: string, count: number): Promise { + const deadline: number = Date.now() + 30000; + const countLines = (): number => + fs.existsSync(filePath) ? fs.readFileSync(filePath, 'utf8').trim().split('\n').length : 0; + while (countLines() < count && Date.now() < deadline) await delayAsync(20); + } + + it('keeps every client on the daemon when it becomes ready only after their first startup deadline', async () => { + const clientCount: number = 8; + // On a loaded host, the first attempts end up to about 2 seconds apart, and the daemon takes up to about + // 3 seconds after its release to serve every client, so the earliest client's second deadline needs room. + const startupTimeoutMs: number = 10000; + fs.writeFileSync(path.join(folder, 'hold-prebind'), ''); + const burst: IConnectOrStartDaemonOptions = { ...options, startupTimeoutMs }; + // A client that does not wait shows that the first deadline expires before the daemon listens. + const control = run('starter.js', [JSON.stringify(burst)]); + const results: Promise = Promise.all( + Array.from({ length: clientCount }, () => runAwaitingAsync(burst, folder)) + ); + // The clients start together, so that their first deadlines expire together even on a loaded host. + await waitForLinesAsync(path.join(folder, 'clients'), clientCount); + fs.writeFileSync(path.join(folder, 'go'), ''); + const daemonPid: number = Number(await waitForFileAsync(path.join(folder, 'prebind'))); + // A client's first deadline has expired once it says that it keeps waiting. + await waitForLinesAsync(path.join(folder, 'notices'), clientCount); + const controlResult = await control.result; + fs.unlinkSync(path.join(folder, 'hold-prebind')); + + expect(controlResult.code).toBe(1); + expect(controlResult.stderr).toContain('Daemon startup'); + for (const result of await results) { + expect(result).toMatchObject({ kind: 'connected', pid: daemonPid }); + expect(result.elapsedMs).toBeGreaterThan(startupTimeoutMs); + // Each client said once why it kept waiting: the daemon binds only after every client has done so. + expect(result.notices).toHaveLength(1); + expect(result.notices[0].owner).toMatch( + /^(Its startup helper \(PID \d+\) is still waiting for the daemon|Another client is still starting the daemon)$/ + ); + expect(result.notices[0].waitMs).toBeGreaterThan(0); + expect(result.notices[0].waitMs).toBeLessThanOrEqual(startupTimeoutMs); + } + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n`); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + }, 60000); + + it('fails without permitting in-process Rush while the startup helper still waits after a second deadline', async () => { + fs.writeFileSync(path.join(folder, 'hold-prebind'), ''); + const result: IAwaitingResult = await runAwaitingAsync({ ...options, startupTimeoutMs: 2000 }); + const daemonPid: number = Number(fs.readFileSync(path.join(folder, 'prebind'), 'utf8')); + const helperPid: number = Number(fs.readFileSync(path.join(folder, 'parents'), 'utf8')); + expect(result.kind).toBe('pending'); + expect(result.elapsedMs).toBeGreaterThanOrEqual(4000); + expect(result.message).toContain( + `Its startup helper (PID ${helperPid}) is still waiting for the daemon, so Rush was not run in-process` + ); + expect(result.message).toContain('use --no-daemon'); + expect(result.message).toContain('"rush-client daemon status"'); + expect(result.notices).toEqual([ + { + owner: `Its startup helper (PID ${helperPid}) is still waiting for the daemon`, + waitMs: expect.any(Number) + } + ]); + + fs.unlinkSync(path.join(folder, 'hold-prebind')); + const client = await connectOrStartDaemonAsync(options); + expect((await client.status).pid).toBe(daemonPid); + await client.closeAsync(); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n`); + }, 30000); + + it('fails without permitting in-process Rush while a listener does not become ready', async () => { + // Accepts connections but never answers, like a daemon whose event loop is blocked. + const sockets: Set = new Set(); + const listener: net.Server = net.createServer((socket) => { + sockets.add(socket); + socket.on('close', () => sockets.delete(socket)); + }); + await new Promise((resolve) => listener.listen(paths.socketPath, resolve)); + const onAwaitStartup: jest.Mock = jest.fn(); + try { + const error: unknown = await connectOrAwaitDaemonStartupAsync({ + ...options, + startupTimeoutMs: 2000, + onAwaitStartup + }).then( + () => undefined, + (rejection: unknown) => rejection + ); + expect(error).toBeInstanceOf(DaemonStartupPendingError); + expect(error).not.toBeInstanceOf(DaemonClientError); + expect((error as Error).message).toContain( + `A process listens at ${paths.socketPath} but was not ready in time, so Rush was not run in-process` + ); + expect(onAwaitStartup.mock.calls).toEqual([ + [`A process listens at ${paths.socketPath} but was not ready in time`, expect.any(Number)] + ]); + } finally { + for (const socket of sockets) socket.destroy(); + await new Promise((resolve) => listener.close(() => resolve())); + } + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }, 30000); + + it('fails without permitting in-process Rush while another client holds the start mutex', async () => { + const holder = run('startLockHolder.js', [JSON.stringify(paths)]); + await waitForFileAsync(path.join(folder, 'lock-held')); + const started: number = Date.now(); + const onAwaitStartup: jest.Mock = jest.fn(); + const error: unknown = await connectOrAwaitDaemonStartupAsync({ + ...options, + startupTimeoutMs: 2000, + onAwaitStartup + }).then( + () => undefined, + (rejection: unknown) => rejection + ); + expect(Date.now() - started).toBeGreaterThanOrEqual(4000); + expect(error).toBeInstanceOf(DaemonStartupPendingError); + expect((error as Error).message).toContain( + 'Another client is still starting the daemon, so Rush was not run in-process' + ); + expect(error).toHaveProperty('cause.code', 'startupFailed'); + expect(onAwaitStartup).toHaveBeenCalledTimes(1); + expect(onAwaitStartup.mock.calls[0][0]).toBe('Another client is still starting the daemon'); + expect(onAwaitStartup.mock.calls[0][1]).toBeGreaterThan(0); + expect(onAwaitStartup.mock.calls[0][1]).toBeLessThanOrEqual(2000); + + fs.writeFileSync(path.join(folder, 'release-lock'), ''); + expect(await holder.result).toEqual({ code: 0, stdout: '', stderr: '' }); + // Once the starter is gone, the same failure permits in-process Rush again, without a notice. + await expect( + connectOrAwaitDaemonStartupAsync({ + paths, + expectedDaemonVersion: 'fixture', + startupTimeoutMs: 500, + onAwaitStartup + }) + ).rejects.toBeInstanceOf(DaemonClientError); + expect(onAwaitStartup).toHaveBeenCalledTimes(1); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }, 30000); + + it('rejects at once with the startup error when nothing live owns the workspace', async () => { + const started: number = Date.now(); + const onAwaitStartup: jest.Mock = jest.fn(); + await expect( + connectOrAwaitDaemonStartupAsync({ paths, expectedDaemonVersion: 'fixture', onAwaitStartup }) + ).rejects.toThrow( + new DaemonClientError( + 'startupFailed', + `No ready daemon at ${paths.socketPath}; auto-start is disabled.` + ) + ); + + // A launcher that exits before readiness leaves a reservation whose helper has exited. + const failing: IConnectOrStartDaemonOptions = { + ...options, + startCommand: { ...options.startCommand!, args: [path.join(folder, 'missing-entry.js')] } + }; + const failure: unknown = await connectOrAwaitDaemonStartupAsync({ ...failing, onAwaitStartup }).catch( + (error: unknown) => error + ); + expect(failure).toBeInstanceOf(DaemonClientError); + expect((failure as Error).message).toContain('Unable to start'); + const refusal: unknown = await connectOrAwaitDaemonStartupAsync({ ...options, onAwaitStartup }).catch( + (error: unknown) => error + ); + expect(refusal).toBeInstanceOf(DaemonClientError); + expect((refusal as Error).message).toContain('exited before the daemon became ready'); + expect(Date.now() - started).toBeLessThan(options.startupTimeoutMs!); + expect(onAwaitStartup).not.toHaveBeenCalled(); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }, 30000); + + it('leaves a version mismatch to the caller as before', async () => { + const running = await connectOrStartDaemonAsync(options); + await running.closeAsync(); + await expect( + connectOrAwaitDaemonStartupAsync({ paths, expectedDaemonVersion: 'replacement' }) + ).rejects.toMatchObject({ code: 'versionMismatch' }); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + }); +}); diff --git a/libraries/rush-client-core/src/test/fixtures/awaitingStarter.ts b/libraries/rush-client-core/src/test/fixtures/awaitingStarter.ts new file mode 100644 index 0000000000..10dad54b93 --- /dev/null +++ b/libraries/rush-client-core/src/test/fixtures/awaitingStarter.ts @@ -0,0 +1,58 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { DaemonClientError } from '../../DaemonClientError'; +import type { IConnectOrStartDaemonOptions } from '../../connectOrStartDaemon'; +import { + DaemonStartupPendingError, + connectOrAwaitDaemonStartupAsync +} from '../../connectOrAwaitDaemonStartup'; + +/** + * Prints one JSON line: whether the client connected, may run Rush in-process, or must fail, and every + * onAwaitStartup call. Given a folder as the second argument, it first appends its PID to "clients" there and + * waits for a "go" file, and it appends its PID to "notices" there on each onAwaitStartup call. + */ +async function mainAsync(): Promise { + const options: IConnectOrStartDaemonOptions = JSON.parse(process.argv[2]); + const signalFolder: string | undefined = process.argv[3]; + if (signalFolder) { + fs.appendFileSync(path.join(signalFolder, 'clients'), `${process.pid}\n`); + while (!fs.existsSync(path.join(signalFolder, 'go'))) await delayAsync(10); + } + const startedAt: number = Date.now(); + const notices: { owner: string; waitMs: number }[] = []; + try { + const client = await connectOrAwaitDaemonStartupAsync({ + ...options, + onAwaitStartup: (owner: string, waitMs: number) => { + notices.push({ owner, waitMs }); + if (signalFolder) fs.appendFileSync(path.join(signalFolder, 'notices'), `${process.pid}\n`); + } + }); + const elapsedMs: number = Date.now() - startedAt; + const { pid } = await client.status; + await client.closeAsync(); + process.stdout.write(`${JSON.stringify({ kind: 'connected', pid, elapsedMs, notices })}\n`); + } catch (error) { + const kind: string = + error instanceof DaemonStartupPendingError + ? 'pending' + : error instanceof DaemonClientError + ? 'fallback' + : 'error'; + const elapsedMs: number = Date.now() - startedAt; + process.stdout.write( + `${JSON.stringify({ kind, message: (error as Error).message, elapsedMs, notices })}\n` + ); + } +} + +mainAsync().catch((error: Error) => { + process.stderr.write(`${error.stack}\n`); + process.exitCode = 1; +}); diff --git a/libraries/rush-client-core/src/test/fixtures/startLockHolder.ts b/libraries/rush-client-core/src/test/fixtures/startLockHolder.ts new file mode 100644 index 0000000000..08d5140af7 --- /dev/null +++ b/libraries/rush-client-core/src/test/fixtures/startLockHolder.ts @@ -0,0 +1,28 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { tryAcquireStartupLockAsync, type IStartupLock } from '../../StartupLock'; + +/** Holds the start mutex, like a client that has not reserved startup yet, until a "release-lock" file exists. */ +async function mainAsync(): Promise { + const paths: IDaemonPaths = JSON.parse(process.argv[2]); + const folder: string = path.dirname(paths.lockfilePath); + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + if (!lock) throw new Error('The start mutex is already held.'); + fs.writeFileSync(path.join(folder, 'lock-held'), String(process.pid)); + const expiry: number = Date.now() + 30000; + while (!fs.existsSync(path.join(folder, 'release-lock')) && Date.now() < expiry) { + await new Promise((resolve) => setTimeout(resolve, 20)); + } + await lock.releaseAsync(); +} + +mainAsync().catch((error: Error) => { + process.stderr.write(`${error.stack}\n`); + process.exitCode = 1; +}); From adf57d5899acef39a810edd704a3d51a0e796670 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:24:09 +0000 Subject: [PATCH 035/265] [rush-lib] Reuse workspace input digests and project configurations while their files are unchanged Swarm integration step 20; original commit 0f987a5b62 (merge of swarm/r07-t93-on-s10 at 18900fe380). Scope: task 93. Brings 143f21a891 and 18900fe380 [rush-lib]. Gate: ch01 GATE OK board 1869 (tree 8892a71708); m01 CONFIRMED board 1738 and board 1818; t04 board 1794. Commits folded into this step (2): - 143f21a891 [rush-lib] Reuse workspace input digests and project configurations while their files are unchanged - 18900fe380 [rush-lib] Test that a project configuration that failed to load is loaded afresh once it is fixed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...use-unchanged-inputs_2026-09-28-18-30.json | 10 + common/reviews/api/rush-lib.api.md | 2 + .../src/api/RushProjectConfiguration.ts | 337 +++++++++++++++++- .../src/api/WorkspaceInputFingerprint.ts | 124 +++++-- .../api/test/RushProjectConfiguration.test.ts | 310 ++++++++++++++++ .../test/WorkspaceInputFingerprint.test.ts | 86 +++++ .../src/utilities/FileContentStamp.ts | 36 ++ .../utilities/test/FileContentStamp.test.ts | 46 +++ 8 files changed, 903 insertions(+), 48 deletions(-) create mode 100644 common/changes/@microsoft/rush/reuse-unchanged-inputs_2026-09-28-18-30.json create mode 100644 libraries/rush-lib/src/utilities/FileContentStamp.ts create mode 100644 libraries/rush-lib/src/utilities/test/FileContentStamp.test.ts diff --git a/common/changes/@microsoft/rush/reuse-unchanged-inputs_2026-09-28-18-30.json b/common/changes/@microsoft/rush/reuse-unchanged-inputs_2026-09-28-18-30.json new file mode 100644 index 0000000000..4f59def32d --- /dev/null +++ b/common/changes/@microsoft/rush/reuse-unchanged-inputs_2026-09-28-18-30.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Long-lived hosts such as the Rush daemon now reuse the content digests of workspace definition and installation files, and the merged rush-project.json configuration of each project, while the files they were computed from are unchanged. A file that changed within the last few seconds is always read again, and a configuration is reloaded if any file of its rig or \"extends\" chain changes or resolves to another file.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 35b81910c8..9e61e8f19d 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -2150,6 +2150,8 @@ export const workspaceRequestScopedEnvironmentVariables: ReadonlySet; // @alpha export class WorkspaceRuntimeFingerprintCache { get changedPaths(): ReadonlyArray; + // @internal + _hashInputFilesAsync(filenames: Iterable): Promise; // @internal (undocumented) _hashPaths(paths: ReadonlyArray): string; } diff --git a/libraries/rush-lib/src/api/RushProjectConfiguration.ts b/libraries/rush-lib/src/api/RushProjectConfiguration.ts index 9ce3da893d..17dc406e30 100644 --- a/libraries/rush-lib/src/api/RushProjectConfiguration.ts +++ b/libraries/rush-lib/src/api/RushProjectConfiguration.ts @@ -2,8 +2,9 @@ // See LICENSE in the project root for license information. import * as path from 'node:path'; +import * as fs from 'node:fs'; -import { AlreadyReportedError, Async, FileSystem, JsonFile, Path } from '@rushstack/node-core-library'; +import { AlreadyReportedError, Async, FileSystem, Import, JsonFile, Path } from '@rushstack/node-core-library'; import type { ITerminal } from '@rushstack/terminal'; import { ProjectConfigurationFile, InheritanceType } from '@rushstack/heft-config-file'; import { @@ -22,6 +23,12 @@ import anythingSchemaJson from '../schemas/anything.schema.json'; import { HotlinkManager } from '../utilities/HotlinkManager'; import type { RushConfiguration } from './RushConfiguration'; import { PhasedCommandEngineProjectConfigurationError } from './PhasedCommandEngineProjectConfigurationError'; +import { + getFileStamp, + getSettledBeforeNs, + isFileStatSettled, + MISSING_FILE_STAMP +} from '../utilities/FileContentStamp'; /** * Describes the file structure for the `/config/rush-project.json` config file. @@ -311,11 +318,52 @@ const OLD_RUSH_PROJECT_CONFIGURATION_FILE: ProjectConfigurationFile = new Map(); -interface IIsolatedProjectConfigurationLoaders { +interface IProjectConfigurationLoaders { readonly configurationFile: ProjectConfigurationFile; readonly oldConfigurationFile: ProjectConfigurationFile; } +/** A configuration file that a load read, or found missing. */ +interface IConfigurationFileInput { + /** The path the loader reads. */ + readonly filePath: string; + readonly stamp: string; + /** The file's `extends` value and the path it resolved to. */ + readonly parent: { readonly specifier: string; readonly filePath: string } | undefined; +} + +/** The rig profile that a project without its own configuration file loads it from. */ +interface IRigProfileInput { + /** `/node_modules//package.json`, the first path that the rig package can resolve to. */ + readonly packageJsonPath: string; + /** `/node_modules//profiles/` */ + readonly profileFolderPath: string; + /** The profile folder's real path, which the loader reads the rig's configuration file from. */ + readonly realProfileFolderPath: string; +} + +interface IProjectConfigurationInputs { + readonly rigJsonStamp: string; + readonly ownFileStamp: string; + readonly rigProfile: IRigProfileInput | undefined; + /** The loaded file and its `extends` chain, or the missing rig configuration file. */ + readonly files: ReadonlyArray; +} + +interface IProjectConfigurationCacheEntry { + readonly rushProjectJson: IRushProjectJson | undefined; + readonly jsonForFingerprint: string | undefined; + /** Undefined if the load can't be reused, for example because one of its files changed too recently. */ + readonly inputs: IProjectConfigurationInputs | undefined; +} + +/** + * The last configuration that {@link RushProjectConfiguration._tryLoadForProjectsUncachedAsync} loaded for a + * project, with the inputs it was loaded from. + */ +const _currentConfigurationCache: WeakMap = + new WeakMap(); + /** * Use this class to load the "config/rush-project.json" config file. * @@ -343,10 +391,11 @@ export class RushProjectConfiguration { private constructor( project: RushConfigurationProject, rushProjectJson: IRushProjectJson, - operationSettingsByOperationName: ReadonlyMap + operationSettingsByOperationName: ReadonlyMap, + jsonForFingerprint: string = JSON.stringify(rushProjectJson) ) { this.project = project; - this.#jsonForFingerprint = JSON.stringify(rushProjectJson); + this.#jsonForFingerprint = jsonForFingerprint; this.incrementalBuildIgnoredGlobs = rushProjectJson.incrementalBuildIgnoredGlobs || []; this.disableBuildCacheForProject = rushProjectJson.disableBuildCacheForProject || false; this.operationSettingsByOperationName = operationSettingsByOperationName; @@ -571,45 +620,60 @@ export class RushProjectConfiguration { } /** - * Loads a fresh native configuration snapshot without reading or modifying process-wide - * project, inherited-file, or rig caches. The loaders are owned only by this invocation. + * Loads a native configuration snapshot of the current files without reading or modifying the process-wide + * project, inherited-file, or rig caches of other loads. The loaders are owned only by this invocation. * * @remarks * Throws a {@link PhasedCommandEngineProjectConfigurationError} that names a project whose * configuration could not be loaded. + * + * A project's merged configuration is reused from an earlier call only if every input of its load is + * unchanged: the stamps (identity, size, mtime and ctime) of its `config/rig.json`, its own + * `config/rush-project.json`, and each file of the `extends` chain that was loaded; the path each `extends` + * value resolves to; and, for a configuration that comes from a rig, the real path of the rig profile folder + * that the project's `node_modules` reaches. A configuration is recorded only if each of its files had + * already been unchanged for a few seconds when it was examined, and only after a load that succeeded; the + * warnings are reported again by every call. * @internal */ public static async _tryLoadForProjectsUncachedAsync( projects: Iterable, terminal: ITerminal ): Promise> { - const loaders: IIsolatedProjectConfigurationLoaders = { + const loaders: IProjectConfigurationLoaders = { configurationFile: createProjectConfigurationFile(), oldConfigurationFile: new ProjectConfigurationFile({ projectRelativeFilePath: RUSH_PROJECT_CONFIGURATION_FILE.projectRelativeFilePath, jsonSchemaObject: anythingSchemaJson }) }; + const view: ConfigurationInputView = _createConfigurationInputView(); const result: Map = new Map(); await Async.forEachAsync( projects, async (project) => { try { - const rushProjectJson: IRushProjectJson | undefined = await _tryLoadJsonForProjectAsync( + const entry: IProjectConfigurationCacheEntry = await _getCurrentConfigurationEntryAsync( project, terminal, - loaders + loaders, + view ); + const { rushProjectJson } = entry; if (rushProjectJson) { result.set( project, new RushProjectConfiguration( project, rushProjectJson, - _getRushProjectConfiguration(project, rushProjectJson, terminal) + _getRushProjectConfiguration(project, rushProjectJson, terminal), + entry.jsonForFingerprint ) ); } + if (entry.inputs) { + _currentConfigurationCache.set(project, entry); + } } catch (error) { throw new PhasedCommandEngineProjectConfigurationError(project.packageName, error); } @@ -667,17 +731,26 @@ export class RushProjectConfiguration { async function _tryLoadJsonForProjectAsync( project: RushConfigurationProject, - terminal: ITerminal, - loaders?: IIsolatedProjectConfigurationLoaders + terminal: ITerminal ): Promise { - const configurationFile: ProjectConfigurationFile = - loaders?.configurationFile ?? RUSH_PROJECT_CONFIGURATION_FILE; - const oldConfigurationFile: ProjectConfigurationFile = - loaders?.oldConfigurationFile ?? OLD_RUSH_PROJECT_CONFIGURATION_FILE; - const rigConfig: IRigConfig | undefined = loaders - ? await loadIsolatedRigConfigAsync(project.projectFolder) - : await RigConfig.loadForProjectFolderAsync({ projectFolderPath: project.projectFolder }); + return await _tryLoadJsonForProjectWithRigAsync( + project, + terminal, + { + configurationFile: RUSH_PROJECT_CONFIGURATION_FILE, + oldConfigurationFile: OLD_RUSH_PROJECT_CONFIGURATION_FILE + }, + await RigConfig.loadForProjectFolderAsync({ projectFolderPath: project.projectFolder }) + ); +} +async function _tryLoadJsonForProjectWithRigAsync( + project: RushConfigurationProject, + terminal: ITerminal, + loaders: IProjectConfigurationLoaders, + rigConfig: IRigConfig | undefined +): Promise { + const { configurationFile, oldConfigurationFile } = loaders; try { return await configurationFile.tryLoadConfigurationFileForProjectAsync( terminal, @@ -791,6 +864,228 @@ async function loadIsolatedRigConfigAsync(projectFolder: string): Promise { + const entry: IProjectConfigurationCacheEntry | undefined = _currentConfigurationCache.get(project); + if (entry && view.isCurrent(project, entry)) { + return entry; + } + _currentConfigurationCache.delete(project); + // Examined before the loader reads them. + const rigJson: IConfigurationFileStat | undefined = view.getStat(getRigJsonPath(project)); + const ownFile: IConfigurationFileStat | undefined = view.getStat(getOwnConfigurationFilePath(project)); + const rigConfig: IRigConfig | undefined = await loadIsolatedRigConfigAsync(project.projectFolder); + const inputs: IProjectConfigurationInputs | undefined = + rigJson?.settled && ownFile?.settled + ? await view.tryGetInputsAsync(project, rigConfig, rigJson.stamp, ownFile.stamp) + : undefined; + const rushProjectJson: IRushProjectJson | undefined = await _tryLoadJsonForProjectWithRigAsync( + project, + terminal, + loaders, + rigConfig + ); + return { + rushProjectJson, + jsonForFingerprint: rushProjectJson && JSON.stringify(rushProjectJson), + inputs + }; +} + +function getRigJsonPath(project: RushConfigurationProject): string { + return path.join(project.projectFolder, 'config', 'rig.json'); +} + +function getOwnConfigurationFilePath(project: RushConfigurationProject): string { + // The path that the loader reads. + return path.resolve(project.projectFolder, RUSH_PROJECT_CONFIGURATION_FILE.projectRelativeFilePath); +} + +interface IConfigurationFileStat { + readonly stamp: string; + /** Whether the file had been unchanged long enough for its stamp to identify its content. */ + readonly settled: boolean; +} + +const getNativeRealPath: (folderPath: string) => string = + // As in the "resolve" package, which Windows network paths make fall back to the JavaScript implementation. + process.platform === 'win32' ? fs.realpathSync : fs.realpathSync.native; + +/** + * The file system state that one {@link RushProjectConfiguration._tryLoadForProjectsUncachedAsync} call compares + * recorded inputs with. Each fact is examined at most once per call, and a file is always examined before the + * call's loader reads it. + */ +class ConfigurationInputView { + readonly #settledBeforeNs: bigint = getSettledBeforeNs(); + readonly #stats: Map = new Map(); + readonly #realPaths: Map = new Map(); + readonly #resolutions: Map = new Map(); + readonly #fileInputs: Map | undefined>> = new Map(); + + /** Returns undefined for a path that isn't a regular file or can't be examined. */ + public getStat(filePath: string): IConfigurationFileStat | undefined { + if (!this.#stats.has(filePath)) { + let result: IConfigurationFileStat | undefined; + try { + // statSync follows links, so dev and ino identify the file whose content is loaded. + const stat: fs.BigIntStats | undefined = fs.statSync(filePath, { bigint: true, throwIfNoEntry: false }); + if (!stat) { + result = { stamp: MISSING_FILE_STAMP, settled: true }; + } else if (stat.isFile()) { + result = { stamp: getFileStamp(stat), settled: isFileStatSettled(stat, this.#settledBeforeNs) }; + } + } catch (error) { + if (FileSystem.isNotExistError(error as Error)) { + result = { stamp: MISSING_FILE_STAMP, settled: true }; + } + } + this.#stats.set(filePath, result); + } + return this.#stats.get(filePath); + } + + public isCurrent(project: RushConfigurationProject, entry: IProjectConfigurationCacheEntry): boolean { + const { inputs } = entry; + if ( + !inputs || + this.getStat(getRigJsonPath(project))?.stamp !== inputs.rigJsonStamp || + this.getStat(getOwnConfigurationFilePath(project))?.stamp !== inputs.ownFileStamp || + (inputs.rigProfile && !this.#isRigProfileCurrent(inputs.rigProfile)) + ) { + return false; + } + return inputs.files.every( + ({ filePath, stamp, parent }) => + this.getStat(filePath)?.stamp === stamp && + (!parent || this.#resolveExtends(parent.specifier, filePath) === parent.filePath) + ); + } + + /** + * Records what a load of the project's configuration depends on, before the loader reads any of it. Returns + * undefined if the load can't be reused. + */ + public async tryGetInputsAsync( + project: RushConfigurationProject, + rigConfig: IRigConfig | undefined, + rigJsonStamp: string, + ownFileStamp: string + ): Promise { + let rigProfile: IRigProfileInput | undefined; + let filePath: string | undefined; + if (ownFileStamp !== MISSING_FILE_STAMP) { + filePath = getOwnConfigurationFilePath(project); + } else if (rigConfig instanceof RealProfileFolderRigConfig) { + const rigFolderPath: string = path.join(project.projectFolder, 'node_modules', rigConfig.rigPackageName); + rigProfile = { + packageJsonPath: path.join(rigFolderPath, 'package.json'), + profileFolderPath: path.join(rigFolderPath, rigConfig.relativeProfileFolderPath), + realProfileFolderPath: rigConfig.getResolvedProfileFolder() + }; + // The shortcut that a reuse checks must agree with the loader's own resolution of the rig. + if (!this.#isRigProfileCurrent(rigProfile)) return undefined; + filePath = path.resolve( + rigProfile.realProfileFolderPath, + RUSH_PROJECT_CONFIGURATION_FILE.projectRelativeFilePath + ); + } else if (rigConfig?.rigFound) { + // The rig package can't be resolved, which the loader reports if it needs the rig. + return undefined; + } + const files: ReadonlyArray | undefined = filePath + ? await this.#getFileInputsAsync(filePath) + : []; + return files && { rigJsonStamp, ownFileStamp, rigProfile, files }; + } + + #isRigProfileCurrent(rigProfile: IRigProfileInput): boolean { + // When this file exists, the rig package resolves to it before any other candidate. + const packageJson: IConfigurationFileStat | undefined = this.getStat(rigProfile.packageJsonPath); + return ( + packageJson !== undefined && + packageJson.stamp !== MISSING_FILE_STAMP && + this.#getRealPath(rigProfile.profileFolderPath) === rigProfile.realProfileFolderPath + ); + } + + #getRealPath(folderPath: string): string | undefined { + if (!this.#realPaths.has(folderPath)) { + let realPath: string | undefined; + try { + realPath = getNativeRealPath(folderPath); + } catch { + // A missing profile folder is reported by the loader. + } + this.#realPaths.set(folderPath, realPath); + } + return this.#realPaths.get(folderPath); + } + + #resolveExtends(specifier: string, configurationFilePath: string): string | undefined { + const baseFolderPath: string = path.dirname(configurationFilePath); + const key: string = `${baseFolderPath}\0${specifier}`; + if (!this.#resolutions.has(key)) { + let resolvedPath: string | undefined; + try { + // As the configuration file loader resolves "extends". + resolvedPath = Import.resolveModule({ modulePath: specifier, baseFolderPath }); + } catch { + // The loader reports the failure. + } + this.#resolutions.set(key, resolvedPath); + } + return this.#resolutions.get(key); + } + + #getFileInputsAsync(filePath: string): Promise | undefined> { + let result: Promise | undefined> | undefined = + this.#fileInputs.get(filePath); + if (!result) { + result = this.#readFileInputsAsync(filePath); + this.#fileInputs.set(filePath, result); + } + return result; + } + + async #readFileInputsAsync( + firstFilePath: string + ): Promise | undefined> { + const files: IConfigurationFileInput[] = []; + const visited: Set = new Set(); + for (let filePath: string | undefined = firstFilePath; filePath !== undefined; ) { + const stat: IConfigurationFileStat | undefined = this.getStat(filePath); + if (!stat?.settled || visited.has(filePath)) return undefined; + visited.add(filePath); + let specifier: unknown; + if (stat.stamp !== MISSING_FILE_STAMP) { + try { + specifier = JsonFile.parseString(await FileSystem.readFileAsync(filePath))?.extends; + } catch { + // The loader reports the failure. + return undefined; + } + } + if (specifier && typeof specifier !== 'string') return undefined; + const parentPath: string | undefined = specifier + ? this.#resolveExtends(specifier as string, filePath) + : undefined; + if (specifier && parentPath === undefined) return undefined; + files.push({ + filePath, + stamp: stat.stamp, + parent: parentPath !== undefined ? { specifier: specifier as string, filePath: parentPath } : undefined + }); + filePath = parentPath; + } + return files; + } +} + /** * Parses and validates the operation settings from the rush-project.json data. Returns the * validated `operationSettingsByOperationName` map used to construct a {@link RushProjectConfiguration}. @@ -858,3 +1153,7 @@ function _getRushProjectConfiguration( return operationSettingsByOperationName; } + +function _createConfigurationInputView(): ConfigurationInputView { + return new ConfigurationInputView(); +} diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index 0505b2633f..47d5df524c 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -14,6 +14,7 @@ import type { RushConfigurationProject } from './RushConfigurationProject'; import { RushProjectConfiguration } from './RushProjectConfiguration'; import { getDaemonIpcImplementationIdentityAsync } from '../logic/operations/DaemonIpcConfiguration'; import { AutoinstallerPluginLoader } from '../pluginFramework/PluginLoader/AutoinstallerPluginLoader'; +import { getFileStamp, getSettledBeforeNs, isFileStatSettled } from '../utilities/FileContentStamp'; /** Stable inputs which distinguish reusable, reloadable, and process-bound workspace state. @alpha */ export interface IWorkspaceInputFingerprint { @@ -34,7 +35,11 @@ export interface IWorkspaceInputFingerprintOptions { readonly environment: Readonly>; /** Additional implementation files/folders owned by the embedding host. */ readonly runtimePaths?: ReadonlyArray; - /** Invocation-owner cache for implementation files; workspace definitions are always read by content. */ + /** + * Invocation-owner cache for file digests. Workspace definitions are always compared by content: a cached + * digest is reused only for a file that had stopped changing before it was read (see + * {@link WorkspaceRuntimeFingerprintCache}). + */ readonly runtimeCache?: WorkspaceRuntimeFingerprintCache; } @@ -251,6 +256,11 @@ function removeRepeatedPathEntries(value: string): string { return Array.from(new Set(value.split(path.delimiter))).join(path.delimiter); } +interface IFileDigest { + readonly stamp: string; + readonly entry: ReadonlyArray; +} + /** * Memoizes runtime content digests behind file identity, size, nanosecond mtime and ctime checks. * Changes to metadata alone still produce the same content fingerprint. @@ -261,10 +271,22 @@ function removeRepeatedPathEntries(value: string): string { * updates the cache; hosts can inspect {@link WorkspaceRuntimeFingerprintCache.changedPaths} * when reporting why a process restart is required. * + * The cache also memoizes the digests of workspace definition and installation files, which users edit while + * a host is running. Such a digest is recorded only if the file's ctime and mtime were at least 3 seconds old + * when the file was examined, and it is reused only while the file's identity, size, mtime and ctime are + * unchanged. A file that changed more recently is read again by every capture. Every write updates a file's + * ctime, which userspace can't set, so a later write can't keep the recorded stamp even on a filesystem whose + * timestamps are coarse, provided that the filesystem's clock agrees with the host's to within that margin. + * + * Like the runtime digests, a memoized entry keeps the file's resolved path while the identity of the file it + * reaches is unchanged. A symbolic link that is retargeted to another hard link of the same file, or a parent + * folder that is moved without changing the file, keeps the previous resolved path. + * * @alpha */ export class WorkspaceRuntimeFingerprintCache { - private readonly _files: Map }> = new Map(); + private readonly _files: Map = new Map(); + private readonly _inputFiles: Map = new Map(); private _baseline: ReadonlyMap | undefined; private _changedPaths: ReadonlyArray = []; @@ -289,8 +311,8 @@ export class WorkspaceRuntimeFingerprintCache { // statSync follows links, so dev and ino identify the file that is loaded. Its resolved path is // recomputed whenever that identity changes, which avoids a costly realpath for every unchanged file. const stat: fsSync.BigIntStats = fsSync.statSync(filename, { bigint: true }); - const stamp: string = `${stat.dev}:${stat.ino}:${stat.size}:${stat.mtimeNs}:${stat.ctimeNs}`; - let cached: { stamp: string; entry: ReadonlyArray } | undefined = this._files.get(filename); + const stamp: string = getFileStamp(stat); + let cached: IFileDigest | undefined = this._files.get(filename); if (cached?.stamp !== stamp) { cached = { stamp, @@ -318,6 +340,58 @@ export class WorkspaceRuntimeFingerprintCache { ); return hashText(JSON.stringify(entries)); } + + /** + * Hashes workspace definition or installation files by content, as `[filename, realpath, sha256]` entries or + * `[filename, 'missing']`. See the remarks of {@link WorkspaceRuntimeFingerprintCache} for when a digest is reused. + * @internal + */ + public async _hashInputFilesAsync(filenames: Iterable): Promise { + const settledBeforeNs: bigint = getSettledBeforeNs(); + const sortedFilenames: string[] = Array.from(filenames).sort(); + const entries: ReadonlyArray[] = new Array(sortedFilenames.length); + const misses: { index: number; stat: fsSync.BigIntStats | undefined }[] = []; + for (let index: number = 0; index < sortedFilenames.length; index++) { + const filename: string = sortedFilenames[index]; + let stat: fsSync.BigIntStats | undefined; + try { + // statSync follows links, so dev and ino identify the file whose content is hashed. + stat = fsSync.statSync(filename, { bigint: true, throwIfNoEntry: false }); + } catch { + // Hashing the file reports the error, or its absence, as an uncached capture does. + misses.push({ index, stat: undefined }); + continue; + } + if (!stat) { + this._inputFiles.delete(filename); + entries[index] = [filename, 'missing']; + } else if (!stat.isFile()) { + misses.push({ index, stat: undefined }); + } else { + const cached: IFileDigest | undefined = this._inputFiles.get(filename); + if (cached?.stamp === getFileStamp(stat)) { + entries[index] = cached.entry; + } else { + misses.push({ index, stat }); + } + } + } + await Async.forEachAsync( + misses, + async ({ index, stat }) => { + const filename: string = sortedFilenames[index]; + const entry: ReadonlyArray = await hashFileAsync(filename); + entries[index] = entry; + if (stat && entry.length === 3 && isFileStatSettled(stat, settledBeforeNs)) { + this._inputFiles.set(filename, { stamp: getFileStamp(stat), entry }); + } else { + this._inputFiles.delete(filename); + } + }, + { concurrency: 3 } + ); + return hashText(JSON.stringify(entries)); + } } /** The strongest action required by a workspace input change. @alpha */ @@ -420,13 +494,12 @@ export async function captureWorkspaceInputFingerprintAsync( for (const pluginConfiguration of rushConfiguration._rushPluginsConfiguration.configuration.plugins) { runtimePaths.push(AutoinstallerPluginLoader.getPluginPackageFolder(rushConfiguration, pluginConfiguration)); } - const runtimeHash: string = (options.runtimeCache ?? new WorkspaceRuntimeFingerprintCache())._hashPaths( - runtimePaths - ); + const cache: WorkspaceRuntimeFingerprintCache = options.runtimeCache ?? new WorkspaceRuntimeFingerprintCache(); + const runtimeHash: string = cache._hashPaths(runtimePaths); return { - configurationHash: await hashFilesAsync(definitions), + configurationHash: await cache._hashInputFilesAsync(definitions), environmentHash: hashText(JSON.stringify(getWorkspaceFingerprintEnvironmentEntries(environment))), - installationHash: await hashFilesAsync(installation), + installationHash: await cache._hashInputFilesAsync(installation), runtimeHash: hashText(JSON.stringify([process.execPath, process.version, runtimeHash])), selectedRushVersion: environment.RUSH_PREVIEW_VERSION ?? rushJson.rushVersion }; @@ -461,26 +534,19 @@ function hashText(text: string): string { return createHash('sha256').update(text).digest('hex'); } -async function hashFilesAsync(filenames: Iterable): Promise { - const entries: string[][] = await Async.mapAsync( - Array.from(filenames).sort(), - async (filename) => { - try { - return [ - filename, - await fs.realpath(filename), - createHash('sha256') - .update(await fs.readFile(filename)) - .digest('hex') - ]; - } catch (error) { - if (!FileSystem.isNotExistError(error as Error)) throw error; - return [filename, 'missing']; - } - }, - { concurrency: 3 } - ); - return hashText(JSON.stringify(entries)); +async function hashFileAsync(filename: string): Promise> { + try { + return [ + filename, + await fs.realpath(filename), + createHash('sha256') + .update(await fs.readFile(filename)) + .digest('hex') + ]; + } catch (error) { + if (!FileSystem.isNotExistError(error as Error)) throw error; + return [filename, 'missing']; + } } /** diff --git a/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts b/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts index 8db1a1fb44..bdb3fe6bdb 100644 --- a/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts @@ -226,6 +226,8 @@ describe(RushProjectConfiguration.name, () => { }); const projects: RushConfigurationProject[] = names.map(project); + // Files that changed moments ago aren't recorded for reuse, which would read them once more. + const dateNow: jest.SpyInstance = jest.spyOn(Date, 'now').mockReturnValue(Date.now()); const readFileAsync: jest.SpyInstance = jest.spyOn(FileSystem, 'readFileAsync'); let configurations: ReadonlyMap; let readPaths: string[]; @@ -234,6 +236,7 @@ describe(RushProjectConfiguration.name, () => { readPaths = readFileAsync.mock.calls.map(([filePath]) => Path.convertToSlashes(filePath)); } finally { readFileAsync.mockRestore(); + dateNow.mockRestore(); } expect(readPaths.filter((p) => p.endsWith('/profiles/default/config/rush-project.json'))).toHaveLength(1); expect(readPaths.filter((p) => p.endsWith('/shared/rush-project.json'))).toHaveLength(1); @@ -255,6 +258,313 @@ describe(RushProjectConfiguration.name, () => { expect(configurations.get(linked)!._getJsonForFingerprint()).toBe(native!._getJsonForFingerprint()); } }); + + describe('reuse of unchanged configurations', () => { + const realDateNow: () => number = Date.now; + let dateNow: jest.SpyInstance; + let readFileAsync: jest.SpyInstance; + + /** Makes every file look as if it had been unchanged for several seconds. */ + const settleFiles = (): void => { + dateNow.mockImplementation(() => realDateNow() + 10_000); + }; + /** Makes the files that exist look as if they had just changed, however long the test takes. */ + const stopClock = (): void => { + dateNow.mockReturnValue(realDateNow()); + }; + const takeReadCount = (): number => { + const count: number = readFileAsync.mock.calls.length; + readFileAsync.mockClear(); + return count; + }; + /** Waits until a write gets a later ctime than the file has, which a coarse clock can delay. */ + const waitForLaterCtime = (relativePath: string): void => { + const { ctimeNs } = fs.statSync(path.join(folder, relativePath), { bigint: true }); + const probePath: string = path.join(folder, 'probe'); + do { + fs.writeFileSync(probePath, ''); + } while (fs.statSync(probePath, { bigint: true }).ctimeNs <= ctimeNs); + }; + const link = (linkPath: string, targetPath: string): void => { + const fullLinkPath: string = path.join(folder, linkPath); + fs.mkdirSync(path.dirname(fullLinkPath), { recursive: true }); + fs.rmSync(fullLinkPath, { force: true }); + fs.symlinkSync(path.join(folder, targetPath), fullLinkPath, 'junction'); + }; + const operationSettings = (operationName: string, outputFolderName: string): object => ({ + operationSettings: [{ operationName, outputFolderNames: [outputFolderName] }] + }); + const getOutputFolders = ( + configuration: RushProjectConfiguration | undefined + ): Record => + Object.fromEntries( + Array.from(configuration?.operationSettingsByOperationName ?? [], ([name, settings]) => [ + name, + [...(settings.outputFolderNames ?? [])] + ]) + ); + /** Checks the configuration that is loaded for the project, and for a project that has never been loaded. */ + const expectOutputFoldersAsync = async ( + rushProject: RushConfigurationProject, + expected: Record + ): Promise => { + const unloaded: RushConfigurationProject = project(rushProject.packageName); + expect(getOutputFolders((await loadAsync(rushProject)).get(rushProject))).toEqual(expected); + expect(getOutputFolders((await loadAsync(unloaded)).get(unloaded))).toEqual(expected); + }; + + beforeEach(() => { + dateNow = jest.spyOn(Date, 'now'); + readFileAsync = jest.spyOn(FileSystem, 'readFileAsync'); + }); + + afterEach(() => { + jest.restoreAllMocks(); + }); + + it('reuses unchanged configurations without reading any file', async () => { + settleFiles(); + write('plain/package.json', { name: 'plain', version: '1.0.0' }); + const projects: RushConfigurationProject[] = ['rigged', 'own-file', 'plain'].map(project); + const first: ReadonlyMap = await loadAsync( + ...projects + ); + expect(takeReadCount()).toBeGreaterThan(0); + const second: ReadonlyMap = await loadAsync( + ...projects + ); + expect(takeReadCount()).toBe(0); + expect([...second.keys()]).toEqual(projects.slice(0, 2)); + for (const rushProject of projects.slice(0, 2)) { + expect(second.get(rushProject)!._getJsonForFingerprint()).toBe( + first.get(rushProject)!._getJsonForFingerprint() + ); + expect(getOutputFolders(second.get(rushProject))).toEqual(getOutputFolders(first.get(rushProject))); + } + }); + + it('loads a configuration again after any file that it was loaded from changes', async () => { + settleFiles(); + const rigged: RushConfigurationProject = project('rigged'); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-rig'] }); + write( + 'rigged/node_modules/example-rig/profiles/default/config/rush-project.json', + operationSettings('_phase:build', 'from-rig-2') + ); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-rig-2'] }); + + // The project's own file takes precedence over the rig's while it exists. + write('rigged/config/rush-project.json', operationSettings('_phase:build', 'from-project')); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-project'] }); + // An edit that keeps the file's identity, size and modification time + const ownFilePath: string = path.join(folder, 'rigged/config/rush-project.json'); + fs.utimesSync(ownFilePath, 1_000_000, 1_000_000); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-project'] }); + waitForLaterCtime('rigged/config/rush-project.json'); + write('rigged/config/rush-project.json', operationSettings('_phase:build', 'from-PROJECT')); + fs.utimesSync(ownFilePath, 1_000_000, 1_000_000); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-PROJECT'] }); + fs.rmSync(ownFilePath); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-rig-2'] }); + + write( + 'rigged/node_modules/example-rig/profiles/other/config/rush-project.json', + operationSettings('_phase:build', 'from-other') + ); + write('rigged/config/rig.json', { rigPackageName: 'example-rig', rigProfile: 'other' }); + await expectOutputFoldersAsync(rigged, { '_phase:build': ['from-other'] }); + + write('rigged/config/base.json', operationSettings('_phase:test', 'from-base')); + write('rigged/config/rush-project.json', { + extends: './base.json', + ...operationSettings('_phase:build', 'from-project') + }); + await expectOutputFoldersAsync(rigged, { + '_phase:build': ['from-project'], + '_phase:test': ['from-base'] + }); + write('rigged/config/base.json', operationSettings('_phase:test', 'from-base-2')); + await expectOutputFoldersAsync(rigged, { + '_phase:build': ['from-project'], + '_phase:test': ['from-base-2'] + }); + }); + + it('loads a configuration again after its rig or an extended package resolves to another folder', async () => { + settleFiles(); + for (const version of ['1', '2']) { + write(`store/shared-config@${version}/package.json`, { name: 'shared-config', version }); + write( + `store/shared-config@${version}/rush-project.json`, + operationSettings('_phase:test', `from-shared-${version}`) + ); + write(`store/example-rig@${version}/package.json`, { name: 'example-rig', version }); + write(`store/example-rig@${version}/profiles/default/config/rush-project.json`, { + extends: 'shared-config/rush-project.json', + ...operationSettings('_phase:build', `from-rig-${version}`) + }); + link(`store/example-rig@${version}/node_modules/shared-config`, 'store/shared-config@1'); + } + write('linked/package.json', { name: 'linked', version: '1.0.0' }); + write('linked/config/rig.json', { rigPackageName: 'example-rig' }); + link('linked/node_modules/example-rig', 'store/example-rig@1'); + const linked: RushConfigurationProject = project('linked'); + await expectOutputFoldersAsync(linked, { + '_phase:build': ['from-rig-1'], + '_phase:test': ['from-shared-1'] + }); + link('linked/node_modules/example-rig', 'store/example-rig@2'); + await expectOutputFoldersAsync(linked, { + '_phase:build': ['from-rig-2'], + '_phase:test': ['from-shared-1'] + }); + link('store/example-rig@2/node_modules/shared-config', 'store/shared-config@2'); + await expectOutputFoldersAsync(linked, { + '_phase:build': ['from-rig-2'], + '_phase:test': ['from-shared-2'] + }); + }); + + it('loads a configuration again after the rig package leaves the project node_modules folder', async () => { + settleFiles(); + write('nested/package.json', { name: 'nested', version: '1.0.0' }); + write('nested/config/rig.json', { rigPackageName: 'example-rig' }); + for (const [rigFolderPath, outputFolderName] of [ + ['nested/node_modules/example-rig', 'from-nested'], + ['node_modules/example-rig', 'from-hoisted'] + ]) { + write(`${rigFolderPath}/package.json`, { name: 'example-rig', version: '1.0.0' }); + write( + `${rigFolderPath}/profiles/default/config/rush-project.json`, + operationSettings('_phase:build', outputFolderName) + ); + } + const nested: RushConfigurationProject = project('nested'); + await expectOutputFoldersAsync(nested, { '_phase:build': ['from-nested'] }); + // Without its package.json, the folder no longer provides the rig package, even though the profile + // folder is still there. + fs.rmSync(path.join(folder, 'nested/node_modules/example-rig/package.json')); + await expectOutputFoldersAsync(nested, { '_phase:build': ['from-hoisted'] }); + }); + + it('reads a configuration again while any of its files may still be changing', async () => { + const rigged: RushConfigurationProject = project('rigged'); + const loadTwiceAsync = async (): Promise => { + await loadAsync(rigged); + takeReadCount(); + await loadAsync(rigged); + return takeReadCount(); + }; + stopClock(); + expect(await loadTwiceAsync()).toBeGreaterThan(0); + + settleFiles(); + const rigFilePath: string = path.join( + folder, + 'rigged/node_modules/example-rig/profiles/default/config/rush-project.json' + ); + const future: number = realDateNow() / 1000 + 60; + fs.utimesSync(rigFilePath, future, future); + expect(await loadTwiceAsync()).toBeGreaterThan(0); + fs.utimesSync(rigFilePath, 1_000_000, 1_000_000); + expect(await loadTwiceAsync()).toBe(0); + }); + + it('reports errors and warnings on every call', async () => { + settleFiles(); + write('deprecated/package.json', { name: 'deprecated', version: '1.0.0' }); + write('deprecated/config/rush-project.json', { + operationSettings: [ + { operationName: '_phase:build', sharding: { count: 2, shardOperationSettings: {} } } + ] + }); + write('duplicate/package.json', { name: 'duplicate', version: '1.0.0' }); + write('duplicate/config/rush-project.json', { + operationSettings: [{ operationName: '_phase:build' }, { operationName: '_phase:build' }] + }); + const [deprecated, duplicate, missingProfile] = ['deprecated', 'duplicate', 'missing-profile'].map( + project + ); + const deprecatedReadCounts: number[] = []; + for (let i: number = 0; i < 2; i++) { + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const terminal: Terminal = new Terminal(terminalProvider); + takeReadCount(); + await RushProjectConfiguration._tryLoadForProjectsUncachedAsync([deprecated], terminal); + deprecatedReadCounts.push(takeReadCount()); + expect(terminalProvider.getWarningOutput()).toContain( + 'DEPRECATED: The "sharding.shardOperationSettings" field is deprecated.' + ); + await expect( + RushProjectConfiguration._tryLoadForProjectsUncachedAsync([duplicate], terminal) + ).rejects.toThrow(); + expect(terminalProvider.getErrorOutput()).toContain( + 'The operation "_phase:build" appears multiple times' + ); + await expect( + RushProjectConfiguration._tryLoadForProjectsUncachedAsync([missingProfile], terminal) + ).rejects.toThrow('The rig profile "missing" is not defined by the rig package "example-rig"'); + } + // The second warning comes from a reused configuration. + expect(deprecatedReadCounts[0]).toBeGreaterThan(0); + expect(deprecatedReadCounts[1]).toBe(0); + write( + 'missing-profile/node_modules/example-rig/profiles/missing/config/rush-project.json', + operationSettings('_phase:build', 'from-missing') + ); + await expectOutputFoldersAsync(missingProfile, { '_phase:build': ['from-missing'] }); + }); + + it('loads a configuration afresh after a load that failed is fixed', async () => { + settleFiles(); + const rigged: RushConfigurationProject = project('rigged'); + const loadWithReadCountAsync = async (): Promise<[Record, number]> => { + takeReadCount(); + const configurations: ReadonlyMap = + await loadAsync(rigged); + return [getOutputFolders(configurations.get(rigged)), takeReadCount()]; + }; + const expectProjectErrorAsync = async (cause: object): Promise => { + const error: unknown = await loadAsync(rigged).catch((e: unknown) => e); + expect(error).toBeInstanceOf(PhasedCommandEngineProjectConfigurationError); + expect(error).toMatchObject({ projectName: 'rigged', cause }); + }; + const ownFilePath: string = 'rigged/config/rush-project.json'; + write(ownFilePath, operationSettings('_phase:build', 'from-project')); + const recordedContent: string = fs.readFileSync(path.join(folder, ownFilePath)).toString(); + expect((await loadWithReadCountAsync())[0]).toEqual({ '_phase:build': ['from-project'] }); + expect(await loadWithReadCountAsync()).toEqual([{ '_phase:build': ['from-project'] }, 0]); + + waitForLaterCtime(ownFilePath); + fs.writeFileSync(path.join(folder, ownFilePath), recordedContent.slice(0, -1)); + await expectProjectErrorAsync({}); + await expectProjectErrorAsync({}); + // Even with the content that it had when its configuration was recorded, the file is read again. + waitForLaterCtime(ownFilePath); + fs.writeFileSync(path.join(folder, ownFilePath), recordedContent); + const [fixedOutputFolders, fixedReadCount] = await loadWithReadCountAsync(); + expect(fixedOutputFolders).toEqual({ '_phase:build': ['from-project'] }); + expect(fixedReadCount).toBeGreaterThan(0); + expect(await loadWithReadCountAsync()).toEqual([{ '_phase:build': ['from-project'] }, 0]); + + fs.rmSync(path.join(folder, ownFilePath)); + write('rigged/config/rig.json', { rigPackageName: 'uninstalled-rig' }); + await expectProjectErrorAsync({ code: 'MODULE_NOT_FOUND' }); + await expectProjectErrorAsync({ code: 'MODULE_NOT_FOUND' }); + write('rigged/node_modules/uninstalled-rig/package.json', { + name: 'uninstalled-rig', + version: '1.0.0' + }); + write( + 'rigged/node_modules/uninstalled-rig/profiles/default/config/rush-project.json', + operationSettings('_phase:build', 'from-installed-rig') + ); + const [installedOutputFolders, installedReadCount] = await loadWithReadCountAsync(); + expect(installedOutputFolders).toEqual({ '_phase:build': ['from-installed-rig'] }); + expect(installedReadCount).toBeGreaterThan(0); + expect(await loadWithReadCountAsync()).toEqual([{ '_phase:build': ['from-installed-rig'] }, 0]); + }); + }); }); describe('operationSettingsByOperationName', () => { diff --git a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts index 400cc891eb..f31d382118 100644 --- a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts +++ b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts @@ -445,4 +445,90 @@ describe('workspace input fingerprints', () => { fs.rmSync(folder, { recursive: true, force: true }); } }); + + it('reuses the digests of definition and installation files only once they have stopped changing', async () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-fingerprint-')); + const realDateNow: () => number = Date.now; + const readFile: jest.SpyInstance = jest.spyOn(fs.promises, 'readFile'); + try { + const write = (relativePath: string, content: string): void => { + const filename: string = path.join(folder, relativePath); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, content); + }; + write( + 'rush.json', + JSON.stringify({ + rushVersion: '5.179.0', + pnpmVersion: '10.27.0', + projects: [{ packageName: 'a', projectFolder: 'a' }] + }) + ); + write('a/package.json', '{"name":"a","version":"1.0.0"}'); + write('common/config/rush/pnpm-lock.yaml', 'lockfileVersion: 1'); + write('npmrc/a', 'registry=https://a.example/'); + write('npmrc/b', 'registry=https://b.example/'); + fs.symlinkSync(path.join(folder, 'npmrc/a'), path.join(folder, '.npmrc')); + const rushConfiguration: RushConfiguration = RushConfiguration.loadFromConfigurationFile( + path.join(folder, 'rush.json') + ); + const packageJsonPath: string = path.join(rushConfiguration.rushJsonFolder, 'a', 'package.json'); + const runtimeCache: WorkspaceRuntimeFingerprintCache = new WorkspaceRuntimeFingerprintCache(); + /** Returns the fingerprint and the number of times that the capture read the project's package.json. */ + const captureAsync = async (): Promise<[IWorkspaceInputFingerprint, number]> => { + readFile.mockClear(); + const fingerprint: IWorkspaceInputFingerprint = await captureWorkspaceInputFingerprintAsync({ + rushConfiguration, + runtimeCache, + environment: {} + }); + return [fingerprint, readFile.mock.calls.filter(([filename]) => filename === packageJsonPath).length]; + }; + + // Files that changed moments ago are read by every capture. + const dateNow: jest.SpyInstance = jest.spyOn(Date, 'now').mockReturnValue(realDateNow()); + const [first, firstReadCount] = await captureAsync(); + expect(firstReadCount).toBe(1); + expect(await captureAsync()).toEqual([first, 1]); + // Once they have been unchanged for a few seconds, a capture records digests that later captures reuse. + dateNow.mockImplementation(() => realDateNow() + 10_000); + expect(await captureAsync()).toEqual([first, 1]); + expect(await captureAsync()).toEqual([first, 0]); + + // An edit that keeps the file's identity, size and modification time + fs.utimesSync(packageJsonPath, 1_000_000, 1_000_000); + expect(await captureAsync()).toEqual([first, 1]); + const { ctimeNs } = fs.statSync(packageJsonPath, { bigint: true }); + const probePath: string = path.join(folder, 'probe'); + do { + // A coarse clock can give a write the same ctime as the previous one. + fs.writeFileSync(probePath, ''); + } while (fs.statSync(probePath, { bigint: true }).ctimeNs <= ctimeNs); + write('a/package.json', '{"name":"a","version":"1.0.1"}'); + fs.utimesSync(packageJsonPath, 1_000_000, 1_000_000); + const [edited, editedReadCount] = await captureAsync(); + expect(editedReadCount).toBe(1); + expect(classifyWorkspaceInputChange(first, edited)).toBe(WorkspaceInputChangeTier.Reload); + expect(await captureAsync()).toEqual([edited, 0]); + + // A link that reaches another file + fs.rmSync(path.join(folder, '.npmrc')); + fs.symlinkSync(path.join(folder, 'npmrc/b'), path.join(folder, '.npmrc')); + const [retargeted] = await captureAsync(); + expect(classifyWorkspaceInputChange(edited, retargeted)).toBe(WorkspaceInputChangeTier.Reload); + // A file that is removed, and then created again + fs.rmSync(packageJsonPath); + const [removed] = await captureAsync(); + expect(classifyWorkspaceInputChange(retargeted, removed)).toBe(WorkspaceInputChangeTier.Reload); + write('a/package.json', '{"name":"a","version":"1.0.1"}'); + expect(await captureAsync()).toEqual([retargeted, 1]); + // An installation file + write('common/config/rush/pnpm-lock.yaml', 'lockfileVersion: 10'); + const [installed] = await captureAsync(); + expect(classifyWorkspaceInputChange(retargeted, installed)).toBe(WorkspaceInputChangeTier.Restart); + } finally { + jest.restoreAllMocks(); + fs.rmSync(folder, { recursive: true, force: true }); + } + }); }); diff --git a/libraries/rush-lib/src/utilities/FileContentStamp.ts b/libraries/rush-lib/src/utilities/FileContentStamp.ts new file mode 100644 index 0000000000..646ed7b627 --- /dev/null +++ b/libraries/rush-lib/src/utilities/FileContentStamp.ts @@ -0,0 +1,36 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type * as fs from 'node:fs'; + +/** + * A file whose ctime or mtime is newer than this many milliseconds must not be memoized by its stamp. + * + * @remarks + * Every write updates a file's ctime, and userspace can't set it. A stamp recorded for a file whose last change + * was already this old can't hide a later write, because that write gets a newer ctime even on a filesystem + * whose timestamps have a coarse granularity (a jiffy on Linux, 2 seconds on FAT). The argument assumes that the + * filesystem's clock agrees with this process's clock to within this margin, as a local filesystem's does. + */ +export const SETTLED_FILE_AGE_MS: number = 3000; + +/** The stamp recorded for a path that doesn't exist. */ +export const MISSING_FILE_STAMP: string = 'missing'; + +/** + * The ctime and mtime a file must be older than for its stamp to be memoized. Take it before examining any of + * the files, so that a write made after a file's examination gets a later ctime. + */ +export function getSettledBeforeNs(): bigint { + return BigInt(Date.now() - SETTLED_FILE_AGE_MS) * BigInt(1e6); +} + +/** Identifies a file's content by the identity, size, nanosecond mtime and ctime of the file it reaches. */ +export function getFileStamp(stat: fs.BigIntStats): string { + return `${stat.dev}:${stat.ino}:${stat.size}:${stat.mtimeNs}:${stat.ctimeNs}`; +} + +/** Whether a file examined after `settledBeforeNs` was taken may be memoized by its stamp. */ +export function isFileStatSettled(stat: fs.BigIntStats, settledBeforeNs: bigint): boolean { + return stat.ctimeNs < settledBeforeNs && stat.mtimeNs < settledBeforeNs; +} diff --git a/libraries/rush-lib/src/utilities/test/FileContentStamp.test.ts b/libraries/rush-lib/src/utilities/test/FileContentStamp.test.ts new file mode 100644 index 0000000000..cd2a3288a4 --- /dev/null +++ b/libraries/rush-lib/src/utilities/test/FileContentStamp.test.ts @@ -0,0 +1,46 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type * as fs from 'node:fs'; + +import { + getFileStamp, + getSettledBeforeNs, + isFileStatSettled, + SETTLED_FILE_AGE_MS +} from '../FileContentStamp'; + +const MS: bigint = BigInt(1e6); + +function createStat(mtimeMs: number, ctimeMs: number): fs.BigIntStats { + return { + dev: BigInt(1), + ino: BigInt(2), + size: BigInt(3), + mtimeNs: BigInt(mtimeMs) * MS + BigInt(4), + ctimeNs: BigInt(ctimeMs) * MS + BigInt(5) + } as fs.BigIntStats; +} + +describe('FileContentStamp', () => { + afterEach(() => { + jest.restoreAllMocks(); + }); + + it('identifies a file by identity, size and nanosecond times', () => { + expect(getFileStamp(createStat(10, 20))).toBe('1:2:3:10000004:20000005'); + }); + + it('settles files whose ctime and mtime are both older than the margin', () => { + jest.spyOn(Date, 'now').mockReturnValue(100_000); + const settledBeforeNs: bigint = getSettledBeforeNs(); + expect(settledBeforeNs).toBe(BigInt(100_000 - SETTLED_FILE_AGE_MS) * MS); + + const boundaryMs: number = 100_000 - SETTLED_FILE_AGE_MS; + expect(isFileStatSettled(createStat(boundaryMs - 1, boundaryMs - 1), settledBeforeNs)).toBe(true); + // Any time in the same millisecond as the bound is too recent. + expect(isFileStatSettled(createStat(boundaryMs - 1, boundaryMs), settledBeforeNs)).toBe(false); + // A future mtime set with utimes() doesn't settle a file whose ctime is old. + expect(isFileStatSettled(createStat(200_000, boundaryMs - 1), settledBeforeNs)).toBe(false); + }); +}); From 9000351f8336623d0347b5640ccb05cfdad94715 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:28:45 +0000 Subject: [PATCH 036/265] [rush-cli-client] The daemon's admission and restart-drain reasons in full in agent output, and rushd's startup wait as a pipe line Swarm integration step 21; original commit 478f4d3af0 (merge of swarm/r04-t83-a81 at 4d1a045fca). Scope: task 83. Brings task 83's chain as resolved onto a81a00520d: 244ed8b1f5 (merge of swarm/r04-t83-clip @9d71dbe9bb, the clip fix for board 1539) and 4d1a045fca, with board 711 (1886d80936), tasks 79, 71 and 50 and the restart-drain timeout. Gate: ch01 GATE OK board 1872 (lifts board 1778); combined with task 93, tree d0e8df0156, full suite board 1888. Commits folded into this step (21): - 2e7d278d8f [rush-daemon] Report the full log path of failed and warning operations - 3b91eec856 [rush-cli-client] Keep spawned test clients on the default output when tests run in an agent shell - 1ae03fc555 [rush-cli-client] Agent output that holds at repository scale - 862ed5e676 [rush-cli-client] Do not report a daemon shutdown as a user cancellation - 39cae5f22b [rush-cli-client] Send no terminal width for a pseudo-terminal without a size - 5f3c36fe17 [rush-daemon] Describe a rejected request without the debug messages of the graph load (task 71) - 563e541fdf [rush-cli-client] Cap a multi-line error message in agent output (task 71) - 2377740ab9 [rush-daemon] Keep the loaded graph when a request only selects an unknown project (task 71) - e382d2ac80 [rush-lib] Run the initial script for every command outside watch mode (task 50, board 115) - b7bba00fb3 [rush-cli-client] Never leave agent output silent on a pipe (task 79) - feacc54bf7 [rush-daemon] Report a restart drain as a queue position (task 79) - 1886d80936 [rush-cli-client] Report Ctrl+C during graph preparation as cancelled (board 711) - ae31023aad [rush-cli-client] Report an empty selection in agent output (board 634) - bacfa3d5af [rush-daemon] Do not apply the client-default queue timeout to a restart drain (task 83) - 5f1c7b4f1b [rush-cli-client] Keep "daemon admission failed ()" in the agent summary; word an empty selection like native Rush (board 785, board 634) - 3b5dc578d6 [rush-daemon] Document the restart drain and its client-default timeout (task 83) - cbf8a3a45e [rush-daemon] Apply the client-default timeout to a restart drain while a rushx script runs or for later requests (task 83) - 6cd768c6b3 [rush-daemon] Suggest stopping the rushx script when a restart drain times out behind one (task 83) - b2f9d78f1f [rush-daemon] Name the waived time when a restart drain times out (task 83) - 9d71dbe9bb Write the daemon's reason for an admission failure in full in agent output (task 83, board 1539) - 4d1a045fca Write rushd's startup wait as a pipe line in agent output (task 83 on task 95) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 61 +- .../src/AgentOperationTracker.ts | 246 ++++++ .../src/AgentProgressRenderer.ts | 464 ++++++++---- .../src/OperationOutputExcerpt.ts | 224 ++++++ .../rush-cli-client/src/clientCancellation.ts | 12 +- apps/rush-cli-client/src/launchClient.ts | 27 +- apps/rush-cli-client/src/outputSelection.ts | 4 + apps/rush-cli-client/src/start.ts | 9 +- apps/rush-cli-client/src/terminalColumns.ts | 12 + .../src/test/AgentProgressRenderer.test.ts | 699 ++++++++++++++++-- .../src/test/GraphGenerationWire.test.ts | 5 +- .../src/test/NativeBuildTestFixture.test.ts | 30 + .../src/test/NativeBuildTestFixture.ts | 4 +- .../src/test/OperationOutputExcerpt.test.ts | 255 +++++++ .../src/test/RushXDaemonTestFixture.ts | 4 +- .../src/test/TestProcessEnvironment.ts | 18 + .../src/test/daemonGraph.test.ts | 3 +- .../src/test/daemonLogs.test.ts | 11 +- .../src/test/launchClient.test.ts | 39 +- .../src/test/nativeMutation.test.ts | 4 +- .../src/test/pipedInput.test.ts | 4 +- .../src/test/terminalColumns.test.ts | 18 + ...mission-failure-code_2026-09-28-15-08.json | 11 + ...gent-empty-selection_2026-09-28-14-45.json | 11 + .../agent-error-detail_2026-09-28-13-30.json | 11 + .../agent-output-scale_2026-09-28-13-00.json | 11 + .../agent-pipe-status_2026-09-28-14-20.json | 11 + ...l-before-engine-init_2026-09-28-14-45.json | 11 + ...utdown-not-cancelled_2026-09-28-13-00.json | 11 + ...t83-admission-reason_2026-09-28-18-55.json | 10 + ...83-startup-wait-line_2026-09-28-20-10.json | 10 + .../zero-size-pty_2026-09-28-13-00.json | 11 + ...rain-default-timeout_2026-09-28-15-08.json | 11 + ...gent-output-log-path_2026-09-28-13-00.json | 11 + ...rain-default-timeout_2026-09-28-15-08.json | 11 + ...gent-output-log-path_2026-09-28-13-00.json | 11 + ...ejection-keeps-graph_2026-09-28-13-45.json | 11 + ...-without-debug-lines_2026-09-28-13-30.json | 11 + ...rain-default-timeout_2026-09-28-15-08.json | 11 + ...drain-queue-position_2026-09-28-14-20.json | 11 + .../reviews/api/rush-daemon-protocol.api.md | 1 + .../src/executeWithDaemonRestart.ts | 20 +- .../src/test/connectOrStartDaemon.test.ts | 38 + .../src/test/fixtures/daemon.ts | 11 + .../src/DaemonOperationPayloads.ts | 5 + .../src/DaemonRequestAdmission.ts | 6 +- libraries/rush-daemon/README.md | 18 +- .../rush-daemon/src/EngineTerminalProvider.ts | 31 +- .../rush-daemon/src/PhasedRequestEventSink.ts | 16 +- .../src/WorkspaceRequestAdmission.ts | 34 +- .../src/WorkspaceRequestLifecycle.ts | 38 +- .../src/WorkspaceRestartArbiter.ts | 143 +++- .../src/test/EngineTerminalProvider.test.ts | 34 + .../src/test/PhasedRequestEventSink.test.ts | 42 +- .../ProductionDaemonRequestResolver.test.ts | 27 + .../src/test/RestartDrainAdmission.test.ts | 131 ++++ .../src/test/WorkspaceRestartArbiter.test.ts | 198 ++++- .../test/WorkspaceRestartArbitration.test.ts | 15 + 58 files changed, 2840 insertions(+), 317 deletions(-) create mode 100644 apps/rush-cli-client/src/AgentOperationTracker.ts create mode 100644 apps/rush-cli-client/src/OperationOutputExcerpt.ts create mode 100644 apps/rush-cli-client/src/terminalColumns.ts create mode 100644 apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts create mode 100644 apps/rush-cli-client/src/test/TestProcessEnvironment.ts create mode 100644 apps/rush-cli-client/src/test/terminalColumns.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/agent-admission-failure-code_2026-09-28-15-08.json create mode 100644 common/changes/@rushstack/rush-cli-client/agent-empty-selection_2026-09-28-14-45.json create mode 100644 common/changes/@rushstack/rush-cli-client/agent-error-detail_2026-09-28-13-30.json create mode 100644 common/changes/@rushstack/rush-cli-client/agent-output-scale_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-cli-client/agent-pipe-status_2026-09-28-14-20.json create mode 100644 common/changes/@rushstack/rush-cli-client/cancel-before-engine-init_2026-09-28-14-45.json create mode 100644 common/changes/@rushstack/rush-cli-client/daemon-shutdown-not-cancelled_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t83-admission-reason_2026-09-28-18-55.json create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t83-startup-wait-line_2026-09-28-20-10.json create mode 100644 common/changes/@rushstack/rush-cli-client/zero-size-pty_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-client-core/restart-drain-default-timeout_2026-09-28-15-08.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/agent-output-log-path_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/restart-drain-default-timeout_2026-09-28-15-08.json create mode 100644 common/changes/@rushstack/rush-daemon/agent-output-log-path_2026-09-28-13-00.json create mode 100644 common/changes/@rushstack/rush-daemon/rejection-keeps-graph_2026-09-28-13-45.json create mode 100644 common/changes/@rushstack/rush-daemon/rejection-without-debug-lines_2026-09-28-13-30.json create mode 100644 common/changes/@rushstack/rush-daemon/restart-drain-default-timeout_2026-09-28-15-08.json create mode 100644 common/changes/@rushstack/rush-daemon/restart-drain-queue-position_2026-09-28-14-20.json create mode 100644 libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index fcbfaa9d06..790c03a04d 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -52,7 +52,7 @@ already running in this repository." Instead, the client keeps trying for one mo (15 seconds, so about 30 seconds in all) and uses the daemon once it is ready. It says so when it starts waiting (`rush-client: The daemon is not ready yet. , so this command waits up to 15 s more for it instead of running Rush in-process.`; agent output shows "rushd is still starting; waiting for it" -as the progress phase). If the daemon is still not ready, the command exits with code 1. The message +as the progress phase, and on a pipe writes it as a progress line). If the daemon is still not ready, the command exits with code 1. The message gives the startup error with its `--no-daemon` hint, then the process that is still live, "so Rush was not run in-process", and a pointer to `rush-client daemon status`. @@ -72,13 +72,19 @@ while the first build after startup loads the graph runs once the load finishes. That wait fails after 10 times the timeout (5 minutes with the default), so a load that never finishes does not hold other requests forever. The request's own routing and execution do not count either. A configured or per-invocation timeout also -limits waiting for a running build that the request could not join; the built-in -default does not. With the default, a build that arrives while a compatible build is -already running waits for it to finish and then runs, instead of failing after 30 -seconds. `--no-wait` fails wherever the request would wait. On a timeout, the client -exits with code 1 and suggests `--wait-timeout`. It does not suggest exporting -`RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, because Rush versions that do not recognize a -`RUSH_` environment variable fail every command while it is set. +limits waiting for a running build that the request could not join, and waiting for +the requests that the daemon is serving to finish before it restarts for the +request's environment. The built-in default does not: with it, a build that arrives +while a compatible build is already running waits for it to finish and then runs, +instead of failing after 30 seconds, and a request that needs a restart waits for +the requests that were running when it arrived to finish and then runs on the +restarted daemon. The default still limits a restart wait while the daemon runs a +`rushx` script, such as a dev server, which may not exit until it is stopped, and +while it serves requests that arrived later. `--no-wait` fails wherever the request +would wait. On a timeout, the client exits with code 1 and suggests +`--wait-timeout`. It does not suggest exporting `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, +because Rush versions that do not recognize a `RUSH_` environment variable fail +every command while it is set. Admission controls also apply to experimental graph requests, but not `start|stop|restart|status|logs`. They affect daemon admission only; native fallback @@ -109,15 +115,34 @@ agent mode writes nothing ahead of it. Otherwise, selection precedence is: Agent mode is plain text for humans and agents, not the AI reporter's JSON record format; use `--reporter=ai` for machine-parsed records. It writes a first status line before -`@microsoft/rush-lib` is loaded, then at most three live rows on a TTY (append-only lines -throttled to one per 2 seconds on a pipe), the queue position when waiting for admission, -and always one final summary line (`rush build: SUCCESS 12/12 operations (...) in 3.1s`, or -`up to date (no operations needed)`). Warnings and errors that Rush or a Rush plugin writes -outside any operation (for example a plugin that continues without the cloud build cache) -precede the summary line, at most three lines of them. On failure, it lists failed operations and a -bounded tail (10 lines) of their stderr, or of their stdout when they wrote no stderr. -Operation logs are otherwise not printed; use `RUSHD_OUTPUT=legacy` for full logs. When -a request falls back to in-process Rush, agent mode stops and native output follows. +`@microsoft/rush-lib` is loaded, then at most three live rows on a TTY. On a pipe it writes +at most three progress lines in total, however many operations run and however long they +take: a wait for a daemon that is still starting (`rushd is still starting; waiting for it (up to 15s more)`), +the start, the first wait for admission (`queued behind another request (position N)`), +the start of execution, and the first failure, in that order until three were written. It +always ends with one summary line, for example +`rush build: SUCCESS 772/772 operations (12 success, 760 from cache) in 3.1s`, or +`up to date (no operations needed)`, or, when the selection parameters matched no projects, +`rush build: SUCCESS 0 operations in 0.5s · the selection parameters did not match any projects`. +The counts follow the native summary: silent operations +(such as phases a project does not define) are not counted unless they fail, and operations +that did not need to run (`SKIPPED` or `NO OP` in native output) are counted as `up to date`. The verdict is +`SUCCESS`, `FAILURE` or `CANCELLED` (Ctrl+C or a termination signal); a request that a daemon +shutdown aborted is a `FAILURE` whose summary line gives the reason, with exit code 1. When the +daemon did not admit the request in time, or at once with `--no-wait`, the reason on the summary +line starts with `daemon admission failed (wait-timeout)` or `daemon admission failed (no-wait)`, +as in legacy output, followed by the daemon's reason in full. Warnings and errors that Rush or a +Rush plugin writes outside any operation (for example a plugin that continues without the cloud +build cache) precede the summary line, and any failure report, at most three lines of them. + +When the request fails, a report comes before the summary line. It covers up to three failed +operations, or, if none failed, the operations whose warnings failed the request. Each one gets a +`failed: · full log: ` line (`warnings: …` for warnings) and a short excerpt +of its output: error lines with the line that follows them first, then the last lines. Stack +frames, `Require stack:` lists and progress noise are left out. The summary line names up to five +failed (or warning) operations. Every operation's full output is in its project's `rush-logs/` +folder, whether or not it was printed. When a request falls back to in-process Rush, agent mode +stops and native output follows. Positively identified built-in `install` and `update` follow the same opt-in routing precedence as workspace builds and require protocol **0.10** @@ -245,7 +270,7 @@ keys and unknown `RUSH_DAEMON*` variables fail validation. | `enabled` | `RUSH_DAEMON` | false | Client routing | | `autoStart` | `RUSH_DAEMON_AUTO_START` | true | Only after opt-in | | `idleTimeoutSeconds` | `RUSH_DAEMON_IDLE_TIMEOUT_SECONDS` | 900 | Host idle shutdown after request/output/cleanup drain | -| `queueTimeoutSeconds` | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | 30 | Admission wait limit. Time behind another request's graph load (up to 10 times the limit) and the request's own work do not count. The default does not limit waiting behind a running compatible build; an explicit value does | +| `queueTimeoutSeconds` | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | 30 | Admission wait limit. Time behind another request's graph load (up to 10 times the limit) and the request's own work do not count. The default does not limit waiting behind a running compatible build, or, for a daemon restart, behind requests that were already running (while no `rushx` script is running); an explicit value does | | `watch` | `RUSH_DAEMON_WATCH` | false | Persistent host observation of requested warm projects; false keeps root/config guards only. Never schedules builds | | `usePersistentIpcRunners` | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | false | Enables explicit per-operation `daemonIpc` Node launchers for unsharded incremental daemon builds | | `warmIdleTimeoutSeconds` | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | 300 | Idle runner, project-watcher and retained-result eviction | diff --git a/apps/rush-cli-client/src/AgentOperationTracker.ts b/apps/rush-cli-client/src/AgentOperationTracker.ts new file mode 100644 index 0000000000..e2381da3b4 --- /dev/null +++ b/apps/rush-cli-client/src/AgentOperationTracker.ts @@ -0,0 +1,246 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// Keep this module free of heavy imports: start.ts loads it (through AgentProgressRenderer) before +// @microsoft/rush-lib. + +import { OperationOutputExcerpt } from './OperationOutputExcerpt'; + +const FAILURE: string = 'FAILURE'; +const SUCCESS_WITH_WARNINGS: string = 'SUCCESS WITH WARNINGS'; +const EXECUTING: string = 'EXECUTING'; + +/** The order in which status counts are listed in the summary: problems first. */ +const STATUS_ORDER: ReadonlyArray = [ + FAILURE, + 'BLOCKED', + 'ABORTED', + SUCCESS_WITH_WARNINGS, + 'SUCCESS', + 'FROM CACHE', + 'SKIPPED', + 'NO OP' +]; +const TERMINAL_STATUSES: ReadonlySet = new Set(STATUS_ORDER); + +export interface IAgentOperationStatusUpdate { + readonly operationId: string; + readonly status: string; + /** The operation's full log file, when the daemon reports one (failures and warnings). */ + readonly logFilePath?: string; +} + +/** An operation's final status, as reported in the daemon's result. */ +export interface IAgentOperationResult { + readonly operationId: string; + readonly status: string; + readonly errorMessage?: string; +} + +/** An operation that explains a failed request, with the excerpt and log file to print for it. */ +export interface IAgentProblemOperation { + readonly operationId: string; + readonly logFilePath: string | undefined; + readonly excerpt: OperationOutputExcerpt | undefined; + /** The engine's error for the operation, when the daemon's result reported one. */ + readonly errorMessage: string | undefined; +} + +/** + * Tracks one request's operations for agent output: progress counters, running and failed operations, + * and short output excerpts for the operations that can explain a failure. + * + * @remarks + * Silent operations (for example phases that a project does not define) are registered but are not part of + * the progress totals, matching the native `x of y` operation headers; only a silent failure is counted. + * Counters follow status transitions, so an operation that is reset and runs again is counted once. + * Output that belongs to no registered operation (a global command's byte stream) is kept as one excerpt. + */ +export class AgentOperationTracker { + readonly #registered: Set = new Set(); + readonly #silent: Set = new Set(); + readonly #statuses: Map = new Map(); + readonly #running: Set = new Set(); + readonly #counts: Map = new Map(); + readonly #failed: Set = new Set(); + readonly #warned: Set = new Set(); + readonly #excerpts: Map = new Map(); + readonly #logFilePaths: Map = new Map(); + readonly #errorMessages: Map = new Map(); + readonly #globalExcerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + #headerTotal: number = 0; + #done: number = 0; + + /** The number of non-silent operations that reached a terminal status. */ + public get done(): number { + return this.#done; + } + + /** The number of non-silent operations in the request, once known. */ + public get total(): number { + return Math.max(this.#headerTotal, this.#registered.size); + } + + /** Operations currently executing, oldest first. */ + public get running(): ReadonlyArray { + return [...this.#running]; + } + + /** Failed operations, in the order they failed. */ + public get failed(): ReadonlyArray { + return [...this.#failed]; + } + + /** Operations that succeeded with warnings, in the order they finished. */ + public get warned(): ReadonlyArray { + return [...this.#warned]; + } + + /** Whether any output arrived that belongs to no operation (for example a global command's output). */ + public get hasGlobalOutput(): boolean { + return this.#globalExcerpt.lineCount > 0; + } + + /** Whether the daemon announced any operation for the request, including silent ones. */ + public get hasOperations(): boolean { + return this.#registered.size > 0 || this.#silent.size > 0; + } + + public get globalExcerpt(): OperationOutputExcerpt { + return this.#globalExcerpt; + } + + /** Status counts in summary order, for example `[['FAILURE', 1], ['BLOCKED', 3]]`. */ + public getCounts(): ReadonlyArray<[string, number]> { + return STATUS_ORDER.filter((status) => (this.#counts.get(status) ?? 0) > 0).map((status) => [ + status, + this.#counts.get(status) ?? 0 + ]); + } + + public register(operationId: string, silent: boolean): void { + if (silent && !this.#registered.has(operationId)) { + this.#silent.add(operationId); + } else if (!silent) { + this.#silent.delete(operationId); + this.#registered.add(operationId); + } + } + + /** Records the engine's per-request operation total (from the operation header extension). */ + public setHeaderTotal(total: number): void { + this.#headerTotal = Math.max(this.#headerTotal, total); + } + + /** Applies a status change. Returns true when it is the request's first failure. */ + public updateStatus(update: IAgentOperationStatusUpdate): boolean { + const { operationId, status } = update; + if (this.#silent.has(operationId)) { + if (status !== FAILURE) { + return false; + } + // Failed operations are reported even if silent, as in the native summary. + this.register(operationId, false); + } + // An operation that was never registered still counts, so `done` never exceeds `total`. + this.#registered.add(operationId); + const firstFailure: boolean = status === FAILURE && this.#failed.size === 0; + const previous: string | undefined = this.#statuses.get(operationId); + this.#statuses.set(operationId, status); + if (previous !== undefined && TERMINAL_STATUSES.has(previous)) { + this.#done--; + this.#counts.set(previous, (this.#counts.get(previous) ?? 1) - 1); + this.#failed.delete(operationId); + this.#warned.delete(operationId); + } + if (status === EXECUTING) { + this.#running.add(operationId); + } else { + this.#running.delete(operationId); + } + if (TERMINAL_STATUSES.has(status)) { + this.#onTerminalStatus(update); + } + return firstFailure; + } + + /** + * Applies the daemon's final operation results to operations whose final status this client did not see, + * for example an operation still executing when the daemon shut down, which the result reports as aborted. + * + * @remarks + * The results also list silent operations without saying so, so an operation that no event announced is + * only added when it failed. + */ + public reconcile(results: ReadonlyArray): void { + for (const { operationId, status, errorMessage } of results) { + const current: string | undefined = this.#statuses.get(operationId); + const known: boolean = this.#registered.has(operationId) || this.#silent.has(operationId); + if ( + TERMINAL_STATUSES.has(status) && + (known || status === FAILURE) && + (current === undefined || !TERMINAL_STATUSES.has(current)) + ) { + this.updateStatus({ operationId, status }); + } + if (errorMessage && this.#statuses.get(operationId) === FAILURE) { + this.#errorMessages.set(operationId, errorMessage); + } + } + } + + /** Records output. Output of operations that already succeeded without warnings is discarded. */ + public appendLog(operationId: string, text: string, stream: 'stdout' | 'stderr'): void { + if (this.#silent.has(operationId)) { + return; + } + const status: string | undefined = this.#statuses.get(operationId); + if (status === undefined && !this.#registered.has(operationId)) { + this.#globalExcerpt.append(text, stream); + return; + } + if (status !== undefined && TERMINAL_STATUSES.has(status) && !this.#isProblemStatus(status)) { + return; + } + let excerpt: OperationOutputExcerpt | undefined = this.#excerpts.get(operationId); + if (!excerpt) { + excerpt = new OperationOutputExcerpt(); + this.#excerpts.set(operationId, excerpt); + } + excerpt.append(text, stream); + } + + /** + * The operations that explain a failed request: failed operations, or, when none failed, operations that + * reported warnings (warnings fail a build unless the command allows them). + */ + public getProblemOperations(): ReadonlyArray { + const operationIds: ReadonlyArray = this.#failed.size ? this.failed : this.warned; + return operationIds.map((operationId) => ({ + operationId, + logFilePath: this.#logFilePaths.get(operationId), + excerpt: this.#excerpts.get(operationId), + errorMessage: this.#errorMessages.get(operationId) + })); + } + + #isProblemStatus(status: string): boolean { + return status === FAILURE || status === SUCCESS_WITH_WARNINGS; + } + + #onTerminalStatus(update: IAgentOperationStatusUpdate): void { + const { operationId, status, logFilePath } = update; + this.#done++; + this.#counts.set(status, (this.#counts.get(status) ?? 0) + 1); + if (!this.#isProblemStatus(status)) { + this.#excerpts.delete(operationId); + return; + } + // Record unterminated last lines now, so a kept excerpt holds no partial-line buffer. + this.#excerpts.get(operationId)?.flush(); + (status === FAILURE ? this.#failed : this.#warned).add(operationId); + if (logFilePath) { + this.#logFilePaths.set(operationId, logFilePath); + } + } +} diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index 87e8a41d97..6195aac9f9 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -4,25 +4,56 @@ // Keep this module free of heavy imports: start.ts loads it before @microsoft/rush-lib // so that the first line can be written within a few milliseconds. -import type { IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; +import type { DaemonRequestAdmissionErrorCode, IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; import { AgentNotices } from './AgentNotices'; +import { + AgentOperationTracker, + type IAgentOperationResult, + type IAgentProblemOperation +} from './AgentOperationTracker'; +import { clipLine } from './OperationOutputExcerpt'; const SPINNER_FRAMES: readonly string[] = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']; -const TERMINAL_STATUSES: ReadonlySet = new Set([ - 'SUCCESS', - 'SUCCESS WITH WARNINGS', - 'SKIPPED', - 'FROM CACHE', - 'FAILURE', - 'BLOCKED', - 'NO OP', - 'ABORTED' -]); -const MAX_ERROR_LINES: number = 10; -const PIPE_MIN_INTERVAL_MS: number = 2000; -const PIPE_HEARTBEAT_MS: number = 10000; +/** On a pipe, the most milestone lines written before the summary, however long the request runs. */ +const MAX_PIPE_PROGRESS_LINES: number = 3; +/** On a pipe, how long the first line waits for the connection, so that a fast connect costs one line, not two. */ +const PIPE_FIRST_LINE_DELAY_MS: number = 1000; +/** + * On a pipe, a status line is written whenever nothing was written for this long, so that a reader can tell a slow + * request from a hung one. Agent shells return partial output after 30 s. These lines are not milestones. + */ +const PIPE_STATUS_INTERVAL_MS: number = 25_000; +const SENT_PHASE: string = 'sent to rushd; preparing the workspace graph'; +const STARTING_PHASE: string = 'rushd is still starting; waiting for it'; const TTY_INTERVAL_MS: number = 100; +/** The most failed (or warning) operations whose output excerpt is printed. */ +const MAX_REPORTED_OPERATIONS: number = 3; +/** Excerpt lines for the first reported operation, which is most often the root cause. */ +const FIRST_OPERATION_EXCERPT_LINES: number = 8; +const OTHER_OPERATION_EXCERPT_LINES: number = 3; +const GLOBAL_OUTPUT_EXCERPT_LINES: number = 8; +/** The most operation names listed in the summary line, and in a live row. */ +const MAX_SUMMARY_NAMES: number = 5; +const MAX_LIVE_NAMES: number = 3; +const MAX_MESSAGE_LENGTH: number = 300; +/** + * Lines of a multi-line error message printed after its first line: the first and last lines of the rest, since a + * message that ends with the error after the diagnostics written before it can be thousands of lines long. + */ +const ERROR_DETAIL_HEAD_LINES: number = 2; +const ERROR_DETAIL_TAIL_LINES: number = 5; +/** + * Summary labels that differ from the native status name. A daemon reports an operation that is unchanged since + * it last ran as `NO OP` (when no operation in the iteration had to run) or as `SKIPPED`; both mean up to date. + */ +const STATUS_LABELS: ReadonlyMap = new Map([ + ['SKIPPED', 'up to date'], + ['NO OP', 'up to date'] +]); + +type PipeMilestone = 'starting' | 'sent' | 'queued' | 'running' | 'failure'; +type Verdict = 'SUCCESS' | 'FAILURE' | 'CANCELLED'; export interface IAgentProgressRendererOptions { readonly commandName: string; @@ -33,38 +64,68 @@ export interface IAgentProgressRendererOptions { readonly startTimeMs?: number; } -interface IAgentFinalResult { +export interface IAgentFinalResult { readonly exitCode: number; readonly errorMessage?: string; + /** Whether the command was cancelled (for example with Ctrl+C); reported as `CANCELLED`, not `FAILURE`. */ + readonly cancelled?: boolean; + /** The daemon's final operation results, which may report statuses that no event carried. */ + readonly operationResults?: ReadonlyArray; + /** Why the daemon did not admit the request, if it did not. */ + readonly admissionErrorCode?: DaemonRequestAdmissionErrorCode; +} + +function formatNames(names: ReadonlyArray, maxNames: number): string { + const shown: string = names.slice(0, maxNames).join(', '); + return names.length > maxNames ? `${shown} +${names.length - maxNames} more` : shown; +} + +function getErrorDetail(lines: ReadonlyArray): string[] { + const omitted: number = lines.length - ERROR_DETAIL_HEAD_LINES - ERROR_DETAIL_TAIL_LINES; + if (omitted <= 1) { + return [...lines]; + } + return [ + ...lines.slice(0, ERROR_DETAIL_HEAD_LINES), + `… ${omitted} more lines …`, + ...lines.slice(-ERROR_DETAIL_TAIL_LINES) + ]; } /** - * Compact progress for agents on the daemon path: an immediate first line, at most - * three live rows (TTY) or throttled append-only lines (pipes), and a guaranteed - * bounded final summary line, even when no operation ran. + * Compact progress for agents on the daemon path: an immediate first line, at most three live rows (TTY) or + * at most three progress lines in total (pipes), and a guaranteed one-line summary, even when no + * operation ran. On failure, the summary is preceded by each failed operation's log file and a short + * excerpt of its output. + * + * @remarks + * On a pipe, a line is written at a milestone: when the client waits for a daemon that is still starting, when the + * request is sent to the daemon, the first time it waits for admission, the start of execution, and the first + * failure; once three lines were written, later milestones are left to the summary. A connection that takes + * longer than a second gets a line of its own first, unless a milestone came first. Whenever + * nothing was written for 25 s, a status line with the counts and the running operations follows, so a reader + * never sees more than 25 s of silence, and a request shorter than that costs no status lines at all. */ export class AgentProgressRenderer { readonly #options: IAgentProgressRendererOptions; readonly #now: () => number; readonly #startTimeMs: number; - readonly #registered: Set = new Set(); - readonly #statuses: Map = new Map(); - readonly #running: Set = new Set(); - readonly #counts: Map = new Map(); - readonly #failed: string[] = []; - readonly #stderrTails: Map = new Map(); - readonly #stdoutTails: Map = new Map(); + readonly #tracker: AgentOperationTracker = new AgentOperationTracker(); + readonly #milestones: Set = new Set(); readonly #notices: AgentNotices = new AgentNotices(); - #total: number = 0; - #done: number = 0; #lastActivity: string = ''; #phase: string = 'connecting to rushd (auto-starts if needed)'; #painted: number = 0; #frame: number = 0; - #lastLineKey: string = ''; - #lastLineAtMs: number = -Infinity; + #pipeLines: number = 0; #timer: ReturnType | undefined; + #firstLineTimer: ReturnType | undefined; + #statusTimer: ReturnType | undefined; + /** The last queue position and when it was reported, until operations start. */ + #queued: { readonly position: number; readonly elapsed: string } | undefined; #stopped: boolean = false; + /** The error message that the summary line contains in full, once written. */ + #reportedErrorMessage: string | undefined; public constructor(options: IAgentProgressRendererOptions) { this.#options = options; @@ -72,13 +133,19 @@ export class AgentProgressRenderer { this.#startTimeMs = options.startTimeMs ?? this.#now(); } - /** Writes the first line and starts the spinner / heartbeat. */ + /** + * On a TTY, paints the first line and starts the spinner. On a pipe, writes the first line after a second + * unless another line came first, and starts the status lines. + */ public start(): void { - this.#render(true); - this.#timer = setInterval( - () => this.#render(false), - this.#options.isTTY ? TTY_INTERVAL_MS : PIPE_MIN_INTERVAL_MS - ); + if (!this.#options.isTTY) { + this.#firstLineTimer = setTimeout(() => this.#writePipeLine(this.#rows()[0]), PIPE_FIRST_LINE_DELAY_MS); + this.#firstLineTimer.unref?.(); + this.#scheduleStatusLine(); + return; + } + this.#paint(); + this.#timer = setInterval(() => this.#paint(), TTY_INTERVAL_MS); this.#timer.unref?.(); } @@ -87,45 +154,46 @@ export class AgentProgressRenderer { return; } this.#phase = phase; - this.#render(true); + if (this.#options.isTTY) { + this.#paint(); + } + } + + /** + * The daemon is not ready yet, but a live process can still make it ready, so the client waits up to `waitMs` + * more for it instead of running Rush in-process. On a pipe, says so once. + */ + public onAwaitStartup(waitMs: number): void { + this.setPhase(STARTING_PHASE); + this.#writeMilestone('starting', ` (up to ${Math.round(waitMs / 1000)}s more)`); + } + + /** The daemon has the request. On a pipe, says so, and that the next line can take a while. */ + public onRequestSent(): void { + this.setPhase(SENT_PHASE); + this.#writeMilestone('sent', ` (status at least every ${PIPE_STATUS_INTERVAL_MS / 1000}s)`); } public onQueuePosition(position: number): void { + this.#queued = { position, elapsed: this.#elapsed() }; this.setPhase(`queued behind another request (position ${position})`); + this.#writeMilestone('queued'); } public onEvent(event: IDaemonEventEnvelope): void { + if (this.#stopped) { + return; + } const payload: Record = (event.payload ?? {}) as Record; switch (event.type) { case 'operationRegistered': { - if (!payload.silent && typeof payload.operationId === 'string') { - this.#registered.add(payload.operationId); + if (typeof payload.operationId === 'string') { + this.#tracker.register(payload.operationId, !!payload.silent); } break; } case 'operationStatusChanged': { - const operationId: unknown = payload.operationId; - const status: unknown = payload.status; - if (typeof operationId !== 'string' || typeof status !== 'string') { - break; - } - this.#phase = 'running'; - const previous: string | undefined = this.#statuses.get(operationId); - this.#statuses.set(operationId, status); - if (status === 'EXECUTING') { - this.#running.add(operationId); - } - if (TERMINAL_STATUSES.has(status) && (previous === undefined || !TERMINAL_STATUSES.has(previous))) { - this.#running.delete(operationId); - this.#done++; - this.#counts.set(status, (this.#counts.get(status) ?? 0) + 1); - if (status === 'FAILURE') { - this.#failed.push(operationId); - } else { - this.#stdoutTails.delete(operationId); - this.#stderrTails.delete(operationId); - } - } + this.#onStatusChanged(payload); break; } case 'extension': { @@ -133,7 +201,7 @@ export class AgentProgressRenderer { | { totalOperations?: unknown } | undefined; if (data && typeof data.totalOperations === 'number') { - this.#total = Math.max(this.#total, data.totalOperations); + this.#tracker.setHeaderTotal(data.totalOperations); } break; } @@ -141,41 +209,20 @@ export class AgentProgressRenderer { this.#notices.add(payload, event.scope?.operationId); if (typeof payload.text === 'string' && payload.text.trim()) { this.#lastActivity = payload.text.trim().split('\n')[0]; - if (this.#phase !== 'running') { - this.#phase = 'running'; - } + this.#phase = 'running'; + this.#queued = undefined; } break; } } - this.#render(false); } /** - * Keeps bounded per-operation stderr and stdout tails (the last lines of each). They are only - * printed for failed operations; stdout is used when an operation reported its diagnostics - * there (tsc, eslint, jest) and wrote nothing to stderr. + * Keeps short excerpts of operation output. They are only printed for operations that explain a failed + * request (failed operations, or operations with warnings when warnings failed the request). */ public onLog(bytes: Uint8Array, operationId: string, stream: 'stdout' | 'stderr'): void { - const status: string | undefined = this.#statuses.get(operationId); - if (status !== undefined && TERMINAL_STATUSES.has(status) && status !== 'FAILURE') { - return; - } - const tails: Map = stream === 'stderr' ? this.#stderrTails : this.#stdoutTails; - for (const line of Buffer.from(bytes).toString('utf8').split('\n')) { - if (!line.trim()) { - continue; - } - let tail: string[] | undefined = tails.get(operationId); - if (!tail) { - tail = []; - tails.set(operationId, tail); - } - tail.push(line.trim()); - if (tail.length > MAX_ERROR_LINES) { - tail.shift(); - } - } + this.#tracker.appendLog(operationId, Buffer.from(bytes).toString('utf8'), stream); } /** Stops rendering without a summary (e.g. the request is handed to in-process Rush). */ @@ -184,47 +231,137 @@ export class AgentProgressRenderer { } /** - * Stops the live region and writes the final summary line, at most once. Warnings and errors that Rush or a - * plugin wrote outside any operation precede it. + * Stops the live region and writes the failure report (on failure) and the final summary line, at most once. + * Warnings and errors that Rush or a plugin wrote outside any operation precede them. The summary line carries + * the first line of the error message; the further lines of a multi-line message precede it, at most eight, + * with the middle ones elided. Returns true when the error message was reported, now or by an earlier call, so + * callers need not repeat it: only a single line too long for the summary line is not, unless it gives the + * reason for an admission failure, which is never clipped. */ - public finish(result: IAgentFinalResult | undefined): void { + public finish(result: IAgentFinalResult | undefined): boolean { + const errorMessage: string | undefined = result?.errorMessage?.trim(); if (!this.#stop()) { - return; + return errorMessage !== undefined && errorMessage === this.#reportedErrorMessage; } for (const notice of this.#notices.getLines()) { this.#options.write(`${notice}\n`); } - const succeeded: boolean = result !== undefined && result.exitCode === 0; - const total: number = this.#getTotal(); - const parts: string[] = [...this.#counts].map(([status, count]) => `${count} ${status.toLowerCase()}`); - const scope: string = - total === 0 && succeeded - ? 'up to date (no operations needed)' - : `${this.#done}/${total} operations${parts.length ? ` (${parts.join(', ')})` : ''}`; - let line: string = `rush ${this.#options.commandName}: ${succeeded ? 'SUCCESS' : 'FAILURE'} ${scope} in ${this.#elapsed()}`; - if (this.#failed.length) { - line += ` · failed: ${this.#failed.join(', ')}`; + if (result?.operationResults) { + this.#tracker.reconcile(result.operationResults); } - if (result?.errorMessage) { - line += ` · ${result.errorMessage}`; - } - this.#options.write(`${line}\n`); - if (!succeeded) { - for (const errorLine of this.#getFailureLines().slice(0, MAX_ERROR_LINES)) { - this.#options.write(` ${errorLine}\n`); + const verdict: Verdict = result?.cancelled + ? 'CANCELLED' + : result !== undefined && result.exitCode === 0 + ? 'SUCCESS' + : 'FAILURE'; + const lines: string[] = verdict === 'SUCCESS' ? [] : this.#getFailureReport(verdict); + // A phased result without operations, for which the daemon announced none, had an empty selection. + const emptySelection: boolean = + verdict === 'SUCCESS' && result?.operationResults?.length === 0 && !this.#tracker.hasOperations; + let summary: string = this.#getSummaryLine(verdict, emptySelection); + if (errorMessage) { + const [firstLine, ...detail] = errorMessage.split('\n').filter((line) => line.trim()); + const admissionErrorCode: string | undefined = + verdict === 'FAILURE' && + (result?.admissionErrorCode === 'no-wait' || result?.admissionErrorCode === 'wait-timeout') + ? result.admissionErrorCode + : undefined; + // Keep the string that legacy output prints for a busy workspace, which guidance tells agents to look for. + const prefix: string = admissionErrorCode ? `daemon admission failed (${admissionErrorCode}): ` : ''; + // The daemon's reason for an admission failure says what the request waited for and what to do instead, and + // the caller has no better text for it, so it is never clipped. + const summaryMessage: string = admissionErrorCode ? firstLine : clipLine(firstLine, MAX_MESSAGE_LENGTH); + summary += ` · ${prefix}${summaryMessage}`; + if (detail.length) { + lines.push(...getErrorDetail(detail).map((line) => ` ${clipLine(line, MAX_MESSAGE_LENGTH)}`)); } + if (detail.length || summaryMessage === firstLine) { + this.#reportedErrorMessage = errorMessage; + } + } + lines.push(summary); + this.#options.write(lines.map((line) => `${line}\n`).join('')); + return this.#reportedErrorMessage !== undefined; + } + + #onStatusChanged(payload: Record): void { + const { operationId, status, logFilePath } = payload; + if (typeof operationId !== 'string' || typeof status !== 'string') { + return; + } + const firstFailure: boolean = this.#tracker.updateStatus({ + operationId, + status, + logFilePath: typeof logFilePath === 'string' ? logFilePath : undefined + }); + this.#phase = 'running'; + this.#queued = undefined; + this.#writeMilestone('running'); + if (firstFailure) { + this.#writeMilestone('failure', ` · first failure: ${operationId}`); + } + } + + #getSummaryLine(verdict: Verdict, emptySelection: boolean): string { + const tracker: AgentOperationTracker = this.#tracker; + const { total, done } = tracker; + const countsByLabel: Map = new Map(); + for (const [status, count] of tracker.getCounts()) { + const label: string = STATUS_LABELS.get(status) ?? status.toLowerCase(); + countsByLabel.set(label, (countsByLabel.get(label) ?? 0) + count); + } + const counts: string[] = [...countsByLabel].map(([label, count]) => `${count} ${label}`); + const matchedNothing: boolean = total === 0 && done === 0 && emptySelection && !tracker.hasGlobalOutput; + let scope: string = ''; + if (total > 0 || done > 0) { + scope = ` ${done}/${total} operations${counts.length ? ` (${counts.join(', ')})` : ''}`; + } else if (matchedNothing) { + scope = ' 0 operations'; + } else if (verdict === 'SUCCESS' && !tracker.hasGlobalOutput) { + scope = ' up to date (no operations needed)'; + } + let line: string = `rush ${this.#options.commandName}: ${verdict}${scope} in ${this.#elapsed()}`; + if (tracker.failed.length) { + line += ` · failed: ${formatNames(tracker.failed, MAX_SUMMARY_NAMES)}`; + } else if (verdict === 'FAILURE' && tracker.warned.length) { + line += ` · warnings: ${formatNames(tracker.warned, MAX_SUMMARY_NAMES)}`; + } else if (matchedNothing) { + // Worded like native Rush's "The command line selection parameters did not match any projects." + line += ' · the selection parameters did not match any projects'; } + return line; } - /** Failed operations' tails, or every operation's stderr tail when no operation failed. */ - #getFailureLines(): string[] { - const operationIds: Iterable = this.#failed.length ? this.#failed : this.#stderrTails.keys(); + /** + * Each failed operation's log file and output excerpt. Without failed operations: operations with warnings + * (they fail a build unless the command allows warnings), or else output that belongs to no operation. A + * cancelled command reports only failed operations. + */ + #getFailureReport(verdict: Verdict): string[] { + const tracker: AgentOperationTracker = this.#tracker; + if (verdict === 'CANCELLED' && !tracker.failed.length) { + return []; + } + const problems: ReadonlyArray = tracker.getProblemOperations(); + if (!problems.length) { + return tracker.globalExcerpt.getExcerpt(GLOBAL_OUTPUT_EXCERPT_LINES).map((line) => ` ${line}`); + } + const label: string = tracker.failed.length ? 'failed' : 'warnings'; const lines: string[] = []; - for (const operationId of operationIds) { - const tail: string[] = this.#stderrTails.get(operationId) ?? this.#stdoutTails.get(operationId) ?? []; - for (const line of tail) { - lines.push(`${operationId}: ${line}`); + for (const [index, problem] of problems.slice(0, MAX_REPORTED_OPERATIONS).entries()) { + const logFile: string = problem.logFilePath ? ` · full log: ${problem.logFilePath}` : ''; + lines.push(`${label}: ${problem.operationId}${logFile}`); + const maxLines: number = index === 0 ? FIRST_OPERATION_EXCERPT_LINES : OTHER_OPERATION_EXCERPT_LINES; + const excerpt: string[] = problem.excerpt?.getExcerpt(maxLines) ?? []; + if (!excerpt.length && problem.errorMessage) { + excerpt.push(clipLine(problem.errorMessage.trim().split('\n')[0], MAX_MESSAGE_LENGTH)); } + lines.push(...(excerpt.length ? excerpt : ['(no output)']).map((line) => ` ${line}`)); + } + const hidden: number = problems.length - MAX_REPORTED_OPERATIONS; + if (hidden > 0) { + const what: string = label === 'failed' ? 'failed operations' : 'operations with warnings'; + lines.push(`+${hidden} more ${what}; their logs are in each project's rush-logs folder`); } return lines; } @@ -235,6 +372,9 @@ export class AgentProgressRenderer { return false; } this.#stopped = true; + for (const timer of [this.#firstLineTimer, this.#statusTimer]) { + clearTimeout(timer); + } if (this.#timer) { clearInterval(this.#timer); this.#timer = undefined; @@ -243,50 +383,86 @@ export class AgentProgressRenderer { return true; } - #getTotal(): number { - return Math.max(this.#total, this.#registered.size, this.#done); - } - #elapsed(): string { return `${((this.#now() - this.#startTimeMs) / 1000).toFixed(1)}s`; } #rows(): [string, string, string] { - const total: number = this.#getTotal(); - const running: string[] = [...this.#running]; - const shown: string = - running.slice(0, 3).join(', ') + (running.length > 3 ? ` +${running.length - 3} more` : ''); - const counter: string = total ? ` ${this.#done}/${total}` : ''; + const { total, done, running, failed } = this.#tracker; + const counter: string = total ? ` ${done}/${total}` : ''; return [ `rush ${this.#options.commandName}${counter} · ${this.#elapsed()} · ${this.#phase}`, - running.length ? `running: ${shown}` : '', - this.#lastActivity + running.length ? `running: ${formatNames(running, MAX_LIVE_NAMES)}` : '', + failed.length ? `failed: ${formatNames(failed, MAX_LIVE_NAMES)}` : this.#lastActivity ]; } - #render(force: boolean): void { - if (this.#stopped) { + #writeMilestone(milestone: PipeMilestone, suffix: string = ''): void { + if (this.#options.isTTY || this.#stopped || this.#milestones.has(milestone)) { return; } - const rows: [string, string, string] = this.#rows(); - if (this.#options.isTTY) { - const width: number = Math.max(20, this.#options.columns || 80) - 1; - const clip = (row: string): string => (row.length > width ? `${row.slice(0, width - 1)}…` : row); - const frame: string = SPINNER_FRAMES[this.#frame++ % SPINNER_FRAMES.length]; - const text: string = [`${frame} ${rows[0]}`, rows[1], rows[2]].map(clip).join('\n'); - this.#options.write(`${this.#painted ? `\x1b[${this.#painted}A\x1b[0J` : '\x1b[?25l'}${text}\n`); - this.#painted = 3; - return; + this.#milestones.add(milestone); + this.#writePipeLine(`${this.#rows()[0]}${suffix}`); + } + + #writePipeLine(line: string): void { + if (this.#pipeLines < MAX_PIPE_PROGRESS_LINES) { + this.#pipeLines++; + this.#writeStatus(line); + } + } + + /** Writes a line on a pipe; the first line and the next status line are then due later. */ + #writeStatus(line: string): void { + clearTimeout(this.#firstLineTimer); + this.#firstLineTimer = undefined; + this.#options.write(`${line}\n`); + if (this.#statusTimer) { + this.#scheduleStatusLine(); + } + } + + #scheduleStatusLine(): void { + clearTimeout(this.#statusTimer); + this.#statusTimer = setTimeout(() => { + if (!this.#stopped) { + this.#writeStatus(this.#getStatusLine()); + } + }, PIPE_STATUS_INTERVAL_MS); + this.#statusTimer.unref?.(); + } + + /** + * A pipe status line: the counts, then the running operations, or else what the request waits for. The daemon + * does not report admission, so after a queue position this says when the position was reported instead of + * claiming that the request is still queued. + */ + #getStatusLine(): string { + const { total, done, running, failed } = this.#tracker; + let activity: string = this.#phase; + if (running.length) { + activity = `running: ${formatNames(running, MAX_LIVE_NAMES)}`; + } else if (this.#queued) { + activity = + `waiting for admission or the workspace graph ` + + `(queue position ${this.#queued.position} at ${this.#queued.elapsed})`; } - const key: string = `${this.#phase}|${this.#done}`; - const nowMs: number = this.#now(); - const sinceLast: number = nowMs - this.#lastLineAtMs; - if (!force && (sinceLast < PIPE_MIN_INTERVAL_MS || (key === this.#lastLineKey && sinceLast < PIPE_HEARTBEAT_MS))) { + const failures: string = failed.length ? ` · failed: ${formatNames(failed, MAX_LIVE_NAMES)}` : ''; + const counter: string = total ? ` ${done}/${total}` : ''; + return `rush ${this.#options.commandName}${counter} · ${this.#elapsed()} · ${activity}${failures}`; + } + + #paint(): void { + if (this.#stopped) { return; } - this.#lastLineKey = key; - this.#lastLineAtMs = nowMs; - this.#options.write(`${rows[0]}${rows[1] ? ` · ${rows[1]}` : ''}\n`); + const rows: [string, string, string] = this.#rows(); + const width: number = Math.max(20, this.#options.columns || 80) - 1; + const clip = (row: string): string => (row.length > width ? `${row.slice(0, width - 1)}…` : row); + const frame: string = SPINNER_FRAMES[this.#frame++ % SPINNER_FRAMES.length]; + const text: string = [`${frame} ${rows[0]}`, rows[1], rows[2]].map(clip).join('\n'); + this.#options.write(`${this.#painted ? `\x1b[${this.#painted}A\x1b[0J` : '\x1b[?25l'}${text}\n`); + this.#painted = 3; } #clear(): void { diff --git a/apps/rush-cli-client/src/OperationOutputExcerpt.ts b/apps/rush-cli-client/src/OperationOutputExcerpt.ts new file mode 100644 index 0000000000..98671349c3 --- /dev/null +++ b/apps/rush-cli-client/src/OperationOutputExcerpt.ts @@ -0,0 +1,224 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// Keep this module free of heavy imports: start.ts loads it (through AgentProgressRenderer) before +// @microsoft/rush-lib. + +const ESC: string = String.fromCharCode(27); +const BEL: string = String.fromCharCode(7); +/** Matches ANSI CSI sequences (colors, cursor movement) and OSC sequences (hyperlinks, titles). */ +const ANSI_ESCAPE_PATTERN: RegExp = new RegExp( + `${ESC}(?:\\[[0-?]*[ -/]*[@-~]|\\][^${BEL}${ESC}]*(?:${BEL}|${ESC}\\\\))`, + 'g' +); +/** A Heft task prefix such as `[build:typescript]`, followed by the line's content. */ +const TASK_PREFIX_PATTERN: RegExp = /^(\[[^\]\s]+\])\s*(.*)$/; +/** + * Lines that carry no diagnostic value in a short excerpt: JavaScript stack frames, `Require stack:` and its + * module paths, code frames (`> 113 | code`, ` | ^`), decorative banners (`---- build finished ----`), + * and the lines Rush itself writes into every operation's output (the command it invokes and the build cache + * status). A failed operation is always a cache miss, and "not found" would otherwise make that line look + * like an error. + */ +const NOISE_PATTERNS: ReadonlyArray = [ + /^at\s.*(?::\d+(?::\d+)?\)?|\((?:index \d+|native|)\))$/, + /^Require stack:$/, + /^-\s+(?:\/|[A-Za-z]:[\\/]|\\\\)\S*$/, + /^(?:>\s*)?\d*\s*\|/, + /^[-=]{3,}(?:\s.*\s[-=]{3,})?$/, + /^Invoking(?: \((?:initial|incremental)\))?: /, + /^(?:This project was not found in the build cache|Build cache hit|Successfully set cache entry)\.$/, + /^Caching build output folders: / +]; +const ERROR_PATTERN: RegExp = + /\b(?:errors?|exception|fatal|failed|failure|FAIL|cannot|could not|unable to|not found|TS\d{4,5})\b|\bERR(?:!|_[A-Z0-9_]+)|[●✖✕]/i; +const NO_ERRORS_PATTERN: RegExp = /\b(?:0|no) errors?\b/i; +/** Lines without any letter (bare exit codes, progress percentages, caret markers) explain nothing. */ +const LETTER_PATTERN: RegExp = /\p{L}/u; + +/** Lines longer than this keep their start and end, joined by an ellipsis. */ +const MAX_LINE_LENGTH: number = 300; +const CLIPPED_LINE_TAIL_LENGTH: number = 80; +/** Raw lines are clipped to this length before they are classified, so huge lines cost little. */ +const MAX_RAW_LINE_LENGTH: number = 4096; +/** A partial line longer than this (for example a progress bar without newlines) is recorded as a line. */ +const MAX_PARTIAL_LINE_LENGTH: number = 65536; +const HEAD_LINES: number = 8; +const TAIL_LINES: number = 3; +const ERROR_LINES: number = 8; +/** Excerpt rows reserved for the last lines, which usually contain a tool's own error summary. */ +const TAIL_RESERVE: number = 2; + +type OutputStream = 'stdout' | 'stderr'; + +interface IExcerptLine { + /** The arrival order across both streams, used to print the excerpt in output order. */ + readonly index: number; + readonly text: string; +} + +interface IErrorLine extends IExcerptLine { + /** The line that followed this error line, which often continues its message. */ + context?: IExcerptLine; +} + +interface IStreamLines { + readonly head: IExcerptLine[]; + readonly tail: IExcerptLine[]; + partial: string; +} + +/** + * Normalizes one line of operation output for an excerpt: removes ANSI escapes and carriage-return overwrites, + * and collapses whitespace after a Heft task prefix. Returns undefined for lines that are empty or noise. + */ +export function normalizeExcerptLine(rawLine: string): string | undefined { + let line: string = clipLine(rawLine, MAX_RAW_LINE_LENGTH); + if (line.includes(ESC)) { + line = line.replace(ANSI_ESCAPE_PATTERN, ''); + } + if (line.endsWith('\r')) { + line = line.slice(0, -1); + } + line = line.slice(line.lastIndexOf('\r') + 1).trim(); + const prefixMatch: RegExpMatchArray | null = TASK_PREFIX_PATTERN.exec(line); + const content: string = prefixMatch ? prefixMatch[2].trim() : line; + if (!LETTER_PATTERN.test(content) || NOISE_PATTERNS.some((pattern) => pattern.test(content))) { + return undefined; + } + return clipLine(prefixMatch ? `${prefixMatch[1]} ${content}` : content, MAX_LINE_LENGTH); +} + +/** Returns whether a normalized line looks like an error message. */ +export function isErrorLine(line: string): boolean { + return ERROR_PATTERN.test(line) && !NO_ERRORS_PATTERN.test(line); +} + +/** Shortens a line to at most `maxLength` characters, keeping its start and its end. */ +export function clipLine(line: string, maxLength: number): string { + if (line.length <= maxLength) { + return line; + } + const tailLength: number = Math.min(CLIPPED_LINE_TAIL_LENGTH, Math.floor(maxLength / 3)); + return `${line.slice(0, maxLength - tailLength - 1)}…${line.slice(line.length - tailLength)}`; +} + +/** + * Keeps a short excerpt of one operation's output, to explain a failure in a few lines. + * + * @remarks + * Memory use does not grow with the output: only the first and last lines of each stream are kept, plus the + * first distinct error-looking lines of either stream, each with the next line of the same stream. The + * excerpt prefers error lines, then the end of the output (where tools print their error summary), then its + * beginning. Like the native `StdioSummarizer`, head and tail lines come from stderr when the operation wrote + * any, otherwise from stdout; error lines come from both, because tools such as tsc and eslint report errors + * on stdout. Stack frames, `Require stack:` paths and code frames are dropped, so they cannot crowd out the + * cause. + */ +export class OperationOutputExcerpt { + readonly #streams: Record = { + stdout: { head: [], tail: [], partial: '' }, + stderr: { head: [], tail: [], partial: '' } + }; + readonly #errors: IErrorLine[] = []; + readonly #errorTexts: Set = new Set(); + /** + * Per stream, the error line whose context is the stream's next line. The streams are separate pipes, so the + * next line of the other stream is unrelated to the error. + */ + readonly #pendingContext: Record = { + stdout: undefined, + stderr: undefined + }; + #lineCount: number = 0; + + /** The number of non-empty, non-noise lines seen so far. */ + public get lineCount(): number { + return this.#lineCount; + } + + /** Records a chunk of output; a line split across chunks is joined. */ + public append(text: string, stream: OutputStream): void { + const lines: IStreamLines = this.#streams[stream]; + const pieces: string[] = (lines.partial + text).split('\n'); + lines.partial = pieces.pop() ?? ''; + if (lines.partial.length > MAX_PARTIAL_LINE_LENGTH) { + pieces.push(lines.partial); + lines.partial = ''; + } + for (const piece of pieces) { + this.#addLine(piece, stream); + } + } + + /** Records any unterminated last lines. Further output is still accepted. */ + public flush(): void { + for (const stream of ['stdout', 'stderr'] as const) { + const lines: IStreamLines = this.#streams[stream]; + if (lines.partial) { + const partial: string = lines.partial; + lines.partial = ''; + this.#addLine(partial, stream); + } + } + } + + /** Returns at most `maxLines` lines that best explain the output, in output order. */ + public getExcerpt(maxLines: number): string[] { + this.flush(); + const preferred: IStreamLines = this.#streams.stderr.head.length + ? this.#streams.stderr + : this.#streams.stdout; + const chosen: Map = new Map(); + const texts: Set = new Set(); + const add = (line: IExcerptLine | undefined, limit: number): void => { + if (line && chosen.size < limit && !chosen.has(line.index) && !texts.has(line.text)) { + chosen.set(line.index, line); + texts.add(line.text); + } + }; + const errorLimit: number = Math.max(1, maxLines - TAIL_RESERVE); + for (const errorLine of this.#errors) { + add(errorLine, errorLimit); + add(errorLine.context, errorLimit); + } + // Newest first, so the last line (often the tool's own error summary) is always kept. + for (let i: number = preferred.tail.length - 1; i >= 0; i--) { + add(preferred.tail[i], maxLines); + } + for (const errorLine of this.#errors) { + add(errorLine, maxLines); + } + for (const line of preferred.head) { + add(line, maxLines); + } + return [...chosen.values()].sort((a, b) => a.index - b.index).map((line) => line.text); + } + + #addLine(rawLine: string, stream: OutputStream): void { + const text: string | undefined = normalizeExcerptLine(rawLine); + if (text === undefined) { + return; + } + const line: IExcerptLine = { index: this.#lineCount++, text }; + const lines: IStreamLines = this.#streams[stream]; + if (lines.head.length < HEAD_LINES) { + lines.head.push(line); + } + lines.tail.push(line); + if (lines.tail.length > TAIL_LINES) { + lines.tail.shift(); + } + const pendingContext: IErrorLine | undefined = this.#pendingContext[stream]; + if (pendingContext) { + pendingContext.context = line; + this.#pendingContext[stream] = undefined; + } + if (this.#errors.length < ERROR_LINES && !this.#errorTexts.has(text) && isErrorLine(text)) { + const errorLine: IErrorLine = { ...line }; + this.#errors.push(errorLine); + this.#errorTexts.add(text); + this.#pendingContext[stream] = errorLine; + } + } +} diff --git a/apps/rush-cli-client/src/clientCancellation.ts b/apps/rush-cli-client/src/clientCancellation.ts index 4c60a964eb..f3a2102bff 100644 --- a/apps/rush-cli-client/src/clientCancellation.ts +++ b/apps/rush-cli-client/src/clientCancellation.ts @@ -25,15 +25,19 @@ export function formatCancellationMessage(commandName: string): string { /** * Returns whether a daemon outcome represents a cancelled command. A result is cancelled when the daemon reports it - * as aborted, even if an operation failure determines its semantic outcome. A completed (non-aborted) result wins - * over a late signal, and a rejection is never reported as a cancellation. + * as aborted, even if an operation failure determines its semantic outcome, unless the daemon aborted it for its own + * reason (such as a daemon shutdown), which the result's error message carries and a signal did not cause. A + * completed (non-aborted) result wins over a late signal. A rejection is only a cancellation when the client was + * signalled and the daemon could not route the request: that is how the daemon answers a request cancelled before + * engine initialization. A rejection of the request itself (for example an invalid or unsupported request) is + * always reported. */ export function isCancelledOutcome(outcome: DaemonClientOutcome, signalled: boolean): boolean { switch (outcome.kind) { case 'result': - return outcome.result.aborted; + return outcome.result.aborted && (signalled || !outcome.result.errorMessage); case 'rejected': - return false; + return signalled && outcome.rejection.code === 'routingFailed'; default: return signalled; } diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 00cd328d1e..7ff5cd2572 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -36,6 +36,7 @@ import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; import { getResultDiagnostic } from './resultDiagnostics'; +import { getTerminalColumns } from './terminalColumns'; import { writeStreamAsync } from './writeStreamAsync'; import { getBundledRushVersion, @@ -112,7 +113,7 @@ export async function launchClientAsync( terminal: { isTTY: !!process.stdout.isTTY, supportsColor: terminal.supportsColor, - columns: process.stdout.columns, + columns: getTerminalColumns(process.stdout), acceptsStdin: true }, admission: @@ -147,7 +148,7 @@ export async function launchClientAsync( ...connection, onAwaitStartup: (owner: string, waitMs: number): void => { if (agentRenderer) { - agentRenderer.setPhase('rushd is still starting; waiting for it'); + agentRenderer.onAwaitStartup(waitMs); return; } const seconds: number = Math.round(waitMs / 1000); @@ -182,7 +183,7 @@ export async function launchClientAsync( verbosity, terminal: { get columns() { - return process.stdout.columns ?? 80; + return getTerminalColumns(process.stdout) ?? 80; }, get isTTY() { return !!process.stdout.isTTY; @@ -205,7 +206,7 @@ export async function launchClientAsync( ); } await renderer.initializeAsync(); - agentRenderer?.setPhase('request submitted; preparing the workspace graph'); + agentRenderer?.onRequestSent(); outcome = await executeWithDaemonRestartAsync(client, connection, { request, abortSignal: abort.signal, @@ -254,17 +255,24 @@ export async function launchClientAsync( } if (outcome === undefined || isCancelledOutcome(outcome, abort.signal.aborted)) { const exitCode: number = getSignalExitCode(cancellationSignal ?? 'SIGINT'); - agentRenderer?.finish({ exitCode, errorMessage: 'cancelled' }); + agentRenderer?.finish( + outcome?.kind === 'result' + ? { ...outcome.result, exitCode, cancelled: true } + : { exitCode, cancelled: true } + ); process.exitCode = exitCode; // After SIGHUP the terminal may be gone; the exit code is what matters. await writeStreamAsync(process.stderr, Buffer.from(formatCancellationMessage(route.commandName))).catch( () => undefined ); } else if (outcome.kind === 'result') { - agentRenderer?.finish(outcome.result); + // In agent mode the summary line may already carry the complete error message; do not repeat it. + const reportedByAgent: boolean = agentRenderer?.finish(outcome.result) ?? false; process.exitCode = outcome.result.exitCode; const diagnostic: string | undefined = getResultDiagnostic(outcome.result); - if (diagnostic) { + if (reportedByAgent) { + // The summary line already explains the failure. + } else if (diagnostic) { await writeStreamAsync(process.stderr, Buffer.from(diagnostic)); } else if (outcome.result.admissionErrorCode) { await writeStreamAsync( @@ -273,8 +281,9 @@ export async function launchClientAsync( ); } } else if (outcome.kind === 'rejected') { - agentRenderer?.finish({ exitCode: 1, errorMessage: `daemon rejected the request (${outcome.rejection.code})` }); - throw new Error(`Daemon rejected the request (${outcome.rejection.code}): ${outcome.rejection.message}`); + const message: string = `Daemon rejected the request (${outcome.rejection.code}): ${outcome.rejection.message}`; + agentRenderer?.finish({ exitCode: 1, errorMessage: message }); + throw new Error(message); } else { agentRenderer?.dispose(); process.stderr.write(`rush-client: ${outcome.message ?? outcome.reason}; using in-process Rush.\n`); diff --git a/apps/rush-cli-client/src/outputSelection.ts b/apps/rush-cli-client/src/outputSelection.ts index cc5bce489c..5378bdf1d8 100644 --- a/apps/rush-cli-client/src/outputSelection.ts +++ b/apps/rush-cli-client/src/outputSelection.ts @@ -14,6 +14,10 @@ export const RUSHD_OUTPUT_ENV_VAR: 'RUSHD_OUTPUT' = 'RUSHD_OUTPUT'; * `detectAgent()` in `@rushstack/reporter` (libraries/reporter/src/config/AgentDetection.ts). */ const AGENT_MARKERS: readonly string[] = ['COPILOT_CLI']; + +/** Every environment variable that `selectClientOutputMode()` reads to choose between agent and legacy output. */ +export const CLIENT_OUTPUT_SELECTION_ENV_VARS: readonly string[] = [RUSHD_OUTPUT_ENV_VAR, ...AGENT_MARKERS]; + const INACTIVE_VALUES: ReadonlySet = new Set(['', '0', 'false', 'no', 'off']); const NATIVE_REPORTER_FLAGS: readonly string[] = ['--reporter', '--output', '--log-level']; diff --git a/apps/rush-cli-client/src/start.ts b/apps/rush-cli-client/src/start.ts index 989adb4e9f..ca87ecc13d 100644 --- a/apps/rush-cli-client/src/start.ts +++ b/apps/rush-cli-client/src/start.ts @@ -23,7 +23,8 @@ const agentRenderer: AgentProgressRenderer | undefined = commandName !== undefined ? new AgentProgressRenderer({ commandName, - isTTY: !!process.stdout.isTTY && process.env.TERM !== 'dumb', + // A pty without a size (e.g. `script` run without a terminal) cannot be repainted; treat it as a pipe. + isTTY: !!process.stdout.isTTY && process.env.TERM !== 'dumb' && !!process.stdout.columns, columns: process.stdout.columns || 80, write: (text: string) => process.stdout.write(text), startTimeMs @@ -34,7 +35,9 @@ agentRenderer?.start(); const { launchClientAsync } = require('./launchClient') as typeof import('./launchClient'); launchClientAsync(false, agentRenderer).catch((error: Error) => { - agentRenderer?.finish({ exitCode: 1, errorMessage: error.message }); - process.stderr.write(`rush-client: ${error.message}\n`); + // In agent mode the summary line may already carry the complete message; do not repeat it. + if (!agentRenderer?.finish({ exitCode: 1, errorMessage: error.message })) { + process.stderr.write(`rush-client: ${error.message}\n`); + } process.exitCode = 1; }); diff --git a/apps/rush-cli-client/src/terminalColumns.ts b/apps/rush-cli-client/src/terminalColumns.ts new file mode 100644 index 0000000000..9ab79e6b1a --- /dev/null +++ b/apps/rush-cli-client/src/terminalColumns.ts @@ -0,0 +1,12 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * Returns the stream's terminal width, or undefined when it has none. A pseudo-terminal that was never sized + * (for example `script` started from a non-terminal) reports 0 columns, which the daemon rejects as request + * metadata; treat it like a stream without a width. + */ +export function getTerminalColumns(stream: { readonly columns?: number }): number | undefined { + const { columns } = stream; + return columns !== undefined && Number.isSafeInteger(columns) && columns > 0 ? columns : undefined; +} diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts index 6a23fc2c5c..3fdee47959 100644 --- a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -26,45 +26,84 @@ function event(type: DaemonEventType, payload: unknown): IDaemonEventEnvelope { const ANSI_ESCAPE: RegExp = new RegExp(`${String.fromCharCode(27)}\\[[0-9;?]*[A-Za-z]`, 'g'); -function createRenderer(isTTY: boolean): { renderer: AgentProgressRenderer; output: string[]; clock: { ms: number } } { +interface ITestRenderer { + renderer: AgentProgressRenderer; + output: string[]; + clock: { ms: number }; + /** All output, split into lines. */ + lines(): string[]; +} + +function createRenderer(isTTY: boolean, commandName: string = 'build'): ITestRenderer { const output: string[] = []; const clock: { ms: number } = { ms: 0 }; const renderer: AgentProgressRenderer = new AgentProgressRenderer({ - commandName: 'build', + commandName, isTTY, columns: 60, write: (text: string) => output.push(text), now: () => clock.ms, startTimeMs: 0 }); - return { renderer, output, clock }; + return { renderer, output, clock, lines: () => output.join('').split('\n').slice(0, -1) }; +} + +function registered(operationId: string, silent: boolean = false): IDaemonEventEnvelope { + return event('operationRegistered', { operationId, silent }); +} + +function status(operationId: string, value: string, logFilePath?: string): IDaemonEventEnvelope { + return event('operationStatusChanged', { + operationId, + previousStatus: 'READY', + status: value, + logFilePath + }); +} + +function header( + operationId: string, + completedOperations: number, + totalOperations: number +): IDaemonEventEnvelope { + return event('extension', { + name: 'rushd.operation-header', + data: { operationId, completedOperations, totalOperations } + }); } -function status(operationId: string, value: string): IDaemonEventEnvelope { - return event('operationStatusChanged', { operationId, previousStatus: 'READY', status: value }); +function fail(renderer: AgentProgressRenderer, operationId: string, errorLines: ReadonlyArray): void { + renderer.onEvent(status(operationId, 'EXECUTING')); + renderer.onLog(Buffer.from(errorLines.map((line) => `${line}\n`).join('')), operationId, 'stderr'); + renderer.onEvent(status(operationId, 'FAILURE', `/repo/${operationId.split(' ')[0]}/rush-logs/x.log`)); } describe(AgentProgressRenderer.name, () => { - it('writes a first line immediately and a bounded summary for a successful build (pipe)', () => { - const { renderer, output, clock } = createRenderer(false); + it('writes one line when the request is sent, then milestones and a summary for a successful build (pipe)', () => { + const { renderer, output, clock, lines } = createRenderer(false); renderer.start(); - expect(output).toEqual(['rush build · 0.0s · connecting to rushd (auto-starts if needed)\n']); - renderer.onEvent(event('operationRegistered', { operationId: 'a (build)', silent: false })); - renderer.onEvent(event('operationRegistered', { operationId: 'b (build)', silent: false })); - renderer.onEvent(event('operationRegistered', { operationId: 'hidden', silent: true })); + expect(output).toEqual([]); + renderer.onRequestSent(); + expect(output).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)\n' + ]); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(registered('b (build)')); + renderer.onEvent(registered('hidden', true)); renderer.onEvent(status('a (build)', 'EXECUTING')); renderer.onLog(Buffer.from('noise\n'), 'a (build)', 'stdout'); clock.ms = 2500; renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.onEvent(status('hidden', 'SUCCESS')); renderer.onEvent(status('b (build)', 'SKIPPED')); clock.ms = 3000; - renderer.finish({ exitCode: 0 }); + expect(renderer.finish({ exitCode: 0 })).toBe(false); renderer.dispose(); - expect(output.join('')).not.toContain('noise'); - expect(output[output.length - 1]).toBe( - 'rush build: SUCCESS 2/2 operations (1 success, 1 skipped) in 3.0s\n' - ); - expect(output.length).toBeLessThanOrEqual(4); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush build 0/2 · 0.0s · running', + 'rush build: SUCCESS 2/2 operations (1 success, 1 up to date) in 3.0s' + ]); }); it('reports an up-to-date request instead of printing nothing', () => { @@ -73,68 +112,278 @@ describe(AgentProgressRenderer.name, () => { expect(output).toEqual(['rush build: SUCCESS up to date (no operations needed) in 0.0s\n']); }); - it('lists failed operations and a bounded stderr tail on failure', () => { - const { renderer, output } = createRenderer(false); - renderer.onEvent(status('p05 (build)', 'EXECUTING')); - for (let i = 0; i < 20; i++) { - renderer.onLog(Buffer.from(`error ${i}\n`), 'p05 (build)', 'stderr'); - } - renderer.onEvent(status('p05 (build)', 'FAILURE')); + it('tells an empty selection apart from a request whose operations were all up to date (#634)', () => { + const hot: ITestRenderer = createRenderer(false); + // The daemon announces retained operations as silent and reports their retained results. + hot.renderer.onEvent(registered('a (build)', true)); + hot.renderer.onEvent(registered('b (build)', true)); + hot.renderer.finish({ + exitCode: 0, + operationResults: [ + { operationId: 'a (build)', status: 'SUCCESS' }, + { operationId: 'b (build)', status: 'FROM CACHE' } + ] + }); + expect(hot.output).toEqual(['rush build: SUCCESS up to date (no operations needed) in 0.0s\n']); + + const empty: ITestRenderer = createRenderer(false); + empty.renderer.finish({ exitCode: 0, operationResults: [] }); + expect(empty.output).toEqual([ + 'rush build: SUCCESS 0 operations in 0.0s · the selection parameters did not match any projects\n' + ]); + }); + + it('reports a failed operation with its log file and an excerpt, before the summary line', () => { + const { renderer, lines } = createRenderer(false); + fail( + renderer, + 'p05 (build)', + Array.from({ length: 20 }, (unused, i) => `error ${i}`) + ); renderer.onEvent(status('p06 (build)', 'BLOCKED')); renderer.finish({ exitCode: 1 }); - const text: string = output.join(''); - expect(text).toContain('rush build: FAILURE 2/2 operations (1 failure, 1 blocked) in 0.0s · failed: p05 (build)\n'); - expect(text).toContain(' p05 (build): error 10\n'); - expect(text).toContain(' p05 (build): error 19\n'); - expect(text).not.toContain('error 9\n'); + expect(lines()).toEqual([ + 'rush build 0/1 · 0.0s · running', + 'rush build 1/1 · 0.0s · running · first failure: p05 (build)', + 'failed: p05 (build) · full log: /repo/p05/rush-logs/x.log', + ' error 0', + ' error 1', + ' error 2', + ' error 3', + ' error 4', + ' error 5', + ' error 18', + ' error 19', + 'rush build: FAILURE 2/2 operations (1 failure, 1 blocked) in 0.0s · failed: p05 (build)' + ]); }); it('keeps failure diagnostics when successful operations wrote stderr first', () => { const { renderer, output } = createRenderer(false); renderer.onEvent(status('noisy (build)', 'EXECUTING')); - for (let i = 0; i < 20; i++) { + for (let i: number = 0; i < 20; i++) { renderer.onLog(Buffer.from(`warning ${i}\n`), 'noisy (build)', 'stderr'); } renderer.onEvent(status('noisy (build)', 'SUCCESS WITH WARNINGS')); - renderer.onEvent(status('broken (build)', 'EXECUTING')); - renderer.onLog(Buffer.from('the real error\n'), 'broken (build)', 'stderr'); - renderer.onEvent(status('broken (build)', 'FAILURE')); + fail(renderer, 'broken (build)', ['the real error']); renderer.finish({ exitCode: 1 }); const text: string = output.join(''); - expect(text).toContain(' broken (build): the real error\n'); - expect(text).not.toContain('noisy (build): warning'); + expect(text).toContain( + 'failed: broken (build) · full log: /repo/broken/rush-logs/x.log\n the real error\n' + ); + expect(text).not.toContain('warning 1'); + expect(text).toMatch(/· failed: broken \(build\)\n$/); + }); + + it('reports unchanged operations as up to date, whether the daemon says SKIPPED or NO OP', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(status('a (build)', 'NO OP')); + renderer.onEvent(status('b (build)', 'SKIPPED')); + renderer.onEvent(status('c (build)', 'FROM CACHE')); + renderer.finish({ exitCode: 0 }); + expect(output[output.length - 1]).toBe( + 'rush build: SUCCESS 3/3 operations (1 from cache, 2 up to date) in 0.0s\n' + ); }); it('counts ABORTED operations as finished', () => { const { renderer, output } = createRenderer(false); - renderer.onEvent(event('operationRegistered', { operationId: 'a (build)', silent: false })); + renderer.onEvent(registered('a (build)')); renderer.onEvent(status('a (build)', 'EXECUTING')); renderer.onEvent(status('a (build)', 'ABORTED')); renderer.finish({ exitCode: 1 }); expect(output[output.length - 1]).toBe('rush build: FAILURE 1/1 operations (1 aborted) in 0.0s\n'); }); + it('reports a cancelled command as CANCELLED, without the output of the interrupted operations', () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(registered('b (build)')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('error: interrupted by SIGINT\n'), 'a (build)', 'stderr'); + renderer.finish({ + exitCode: 130, + cancelled: true, + operationResults: [ + { operationId: 'a (build)', status: 'ABORTED' }, + { operationId: 'b (build)', status: 'ABORTED' } + ] + }); + expect(lines()).toEqual([ + 'rush build 0/2 · 0.0s · running', + 'rush build: CANCELLED 2/2 operations (2 aborted) in 0.0s' + ]); + }); + + it('still reports the failures of a cancelled command', () => { + const { renderer, lines } = createRenderer(false); + fail(renderer, 'a (build)', ['src/a.ts(1,1): error TS2322: bad']); + renderer.finish({ exitCode: 130, cancelled: true }); + expect(lines().slice(-3)).toEqual([ + 'failed: a (build) · full log: /repo/a/rush-logs/x.log', + ' src/a.ts(1,1): error TS2322: bad', + 'rush build: CANCELLED 1/1 operations (1 failure) in 0.0s · failed: a (build)' + ]); + }); + + it('applies final statuses from the daemon result that no event reported', () => { + const { renderer, lines } = createRenderer(false); + const reason: string = + 'The Rush daemon was shut down (idle timeout) while this request was running; re-run the command.'; + renderer.onEvent(registered('a (build)')); + renderer.onEvent(registered('b (build)')); + renderer.onEvent(registered('silent (build)', true)); + renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.onEvent(status('b (build)', 'EXECUTING')); + const reported: boolean = renderer.finish({ + exitCode: 1, + errorMessage: reason, + operationResults: [ + { operationId: 'a (build)', status: 'SUCCESS' }, + { operationId: 'b (build)', status: 'ABORTED' }, + { operationId: 'silent (build)', status: 'ABORTED' }, + // The result lists silent operations too; one that no event announced is not counted. + { operationId: 'unannounced (build)', status: 'NO OP' } + ] + }); + expect(reported).toBe(true); + expect(lines().slice(-1)).toEqual([ + `rush build: FAILURE 2/2 operations (1 aborted, 1 success) in 0.0s · ${reason}` + ]); + }); + + it("prints a failed operation's error from the daemon result when the operation wrote no output", () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.finish({ + exitCode: 1, + operationResults: [{ operationId: 'a (build)', status: 'FAILURE', errorMessage: 'spawn heft ENOENT' }] + }); + expect(lines().slice(-3)).toEqual([ + 'failed: a (build)', + ' spawn heft ENOENT', + 'rush build: FAILURE 1/1 operations (1 failure) in 0.0s · failed: a (build)' + ]); + }); + + it('tells a repeated finish whether its error message was already reported in full', () => { + const { renderer, output } = createRenderer(false); + const message: string = 'Daemon rejected the request (routingFailed): Another Rush command is running.'; + expect(renderer.finish({ exitCode: 1, errorMessage: message })).toBe(true); + expect(renderer.finish({ exitCode: 1, errorMessage: message })).toBe(true); + expect(renderer.finish({ exitCode: 1, errorMessage: 'another message' })).toBe(false); + expect(output).toEqual([`rush build: FAILURE in 0.0s · ${message}\n`]); + }); + + it('keeps the legacy "daemon admission failed ()" string in the summary line (#785)', () => { + const timeout: ITestRenderer = createRenderer(false); + const message: string = + 'The request was not admitted within 30000ms while waiting for workspace admission. ' + + 'Use --wait-timeout or RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS to wait longer.'; + expect( + timeout.renderer.finish({ exitCode: 1, admissionErrorCode: 'wait-timeout', errorMessage: message }) + ).toBe(true); + expect(timeout.output).toEqual([ + `rush build: FAILURE in 0.0s · daemon admission failed (wait-timeout): ${message}\n` + ]); + + const noWait: ITestRenderer = createRenderer(false); + noWait.renderer.finish({ + exitCode: 1, + admissionErrorCode: 'no-wait', + errorMessage: 'The workspace is busy.' + }); + expect(noWait.output).toEqual([ + 'rush build: FAILURE in 0.0s · daemon admission failed (no-wait): The workspace is busy.\n' + ]); + + // A request aborted while it waited, for example by a daemon shutdown, is not a busy workspace. + const aborted: ITestRenderer = createRenderer(false); + aborted.renderer.finish({ + exitCode: 1, + admissionErrorCode: 'aborted', + errorMessage: 'The Rush daemon was shut down.' + }); + expect(aborted.output).toEqual(['rush build: FAILURE in 0.0s · The Rush daemon was shut down.\n']); + }); + + it('writes the reason for an admission failure in full however long it is, so the caller adds nothing', () => { + // A restart drain that timed out behind a rushx script after it waived time: 350 characters. + const message: string = + 'The request was not admitted before the daemon could restart for its environment, which waits for the ' + + 'requests that the daemon is serving to finish, including a rushx script that may not exit until it is ' + + 'stopped; 61.8s spent waiting for requests that were already running did not count. Stop the script, ' + + 'or use --wait-timeout to wait longer.'; + expect(message.length).toBeGreaterThan(300); + const { renderer, output } = createRenderer(false); + expect(renderer.finish({ exitCode: 1, admissionErrorCode: 'wait-timeout', errorMessage: message })).toBe( + true + ); + expect(output).toEqual([ + `rush build: FAILURE in 0.0s · daemon admission failed (wait-timeout): ${message}\n` + ]); + }); + + it('clips any other single-line error message that is too long, and leaves it to the caller', () => { + const message: string = `Rush failed: ${'x'.repeat(400)} (end)`; + const { renderer, output } = createRenderer(false); + expect(renderer.finish({ exitCode: 1, errorMessage: message })).toBe(false); + expect(output).toHaveLength(1); + expect(output[0]).toMatch(/^rush build: FAILURE in 0\.0s · Rush failed: x+…x+ \(end\)\n$/); + expect(output[0].length).toBeLessThan(message.length); + }); + it('writes the final line at most once and nothing after it', () => { const { renderer, output } = createRenderer(false); - renderer.finish({ exitCode: 1, errorMessage: 'daemon rejected the request (x)' }); - renderer.finish({ exitCode: 1, errorMessage: 'again' }); + expect(renderer.finish({ exitCode: 1, errorMessage: 'daemon rejected the request (x)' })).toBe(true); + expect(renderer.finish({ exitCode: 1, errorMessage: 'again' })).toBe(false); renderer.onQueuePosition(3); + renderer.onEvent(status('a (build)', 'FAILURE')); renderer.dispose(); - expect(output).toEqual(['rush build: FAILURE 0/0 operations in 0.0s · daemon rejected the request (x)\n']); + expect(output).toEqual(['rush build: FAILURE in 0.0s · daemon rejected the request (x)\n']); }); - it('does not repeat an unchanged queue position', () => { + it('writes the further lines of a multi-line error message before the summary line', () => { + const { renderer, output } = createRenderer(false); + expect(renderer.finish({ exitCode: 1, errorMessage: 'first line\n\nsecond line\n' })).toBe(true); + expect(output).toEqual([' second line\nrush build: FAILURE in 0.0s · first line\n']); + }); + + it('elides the middle of an error message with thousands of lines and keeps its last lines', () => { + const { renderer, output } = createRenderer(false); + const diagnostics: string[] = Array.from({ length: 1745 }, (unused, index) => `debug line ${index}`); + const message: string = [ + 'Daemon rejected the request (invalidRequest): Incremental strategy: cache restoration', + ...diagnostics, + 'The project name "@x/nope" passed to "--to" does not exist in rush.json.', + 'An error occurred.' + ].join('\n'); + expect(renderer.finish({ exitCode: 1, errorMessage: message })).toBe(true); + expect(output.join('').split('\n')).toEqual([ + ' debug line 0', + ' debug line 1', + ' … 1740 more lines …', + ' debug line 1742', + ' debug line 1743', + ' debug line 1744', + ' The project name "@x/nope" passed to "--to" does not exist in rush.json.', + ' An error occurred.', + 'rush build: FAILURE in 0.0s · Daemon rejected the request (invalidRequest): Incremental strategy: cache restoration', + '' + ]); + }); + + it('writes the queue milestone once, however often the position changes', () => { const { renderer, output } = createRenderer(false); renderer.onQueuePosition(2); renderer.onQueuePosition(2); - renderer.onQueuePosition(2); - expect(output).toHaveLength(1); renderer.onQueuePosition(1); - expect(output).toHaveLength(2); + expect(output).toEqual(['rush build · 0.0s · queued behind another request (position 2)\n']); renderer.dispose(); }); - it('shows the stdout tail of a failed operation that reported errors on stdout', () => { + it('shows the excerpt of a failed operation that reported errors on stdout', () => { const { renderer, output } = createRenderer(false); renderer.onEvent(status('ok (build)', 'EXECUTING')); renderer.onLog(Buffer.from('ok noise\n'), 'ok (build)', 'stdout'); @@ -144,10 +393,21 @@ describe(AgentProgressRenderer.name, () => { renderer.onEvent(status('tsc (build)', 'FAILURE')); renderer.finish({ exitCode: 1 }); const text: string = output.join(''); - expect(text).toContain(' tsc (build): src/x.ts(1,1): error TS1005: stdout-error\n'); + expect(text).toContain('failed: tsc (build)\n src/x.ts(1,1): error TS1005: stdout-error\n'); expect(text).not.toContain('ok noise'); }); + it('says so when a failed operation wrote no output', () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(status('quiet (build)', 'FAILURE')); + renderer.finish({ exitCode: 1 }); + expect(lines().slice(-3)).toEqual([ + 'failed: quiet (build)', + ' (no output)', + 'rush build: FAILURE 1/1 operations (1 failure) in 0.0s · failed: quiet (build)' + ]); + }); + it('shows queue position immediately', () => { const { renderer, output } = createRenderer(false); renderer.onQueuePosition(2); @@ -163,21 +423,310 @@ describe(AgentProgressRenderer.name, () => { expect(output[output.length - 1]).toBe('rush build: SUCCESS up to date (no operations needed) in 4.0s\n'); }); - it('throttles progress lines on a pipe', () => { - const { renderer, output, clock } = createRenderer(false); + it('writes at most three progress lines and one summary line on a pipe at odsp-web scale', () => { + const { renderer, clock, lines } = createRenderer(false); renderer.start(); - for (let i = 0; i < 50; i++) { - clock.ms = i * 10; + renderer.onRequestSent(); + renderer.onQueuePosition(1); + const operationIds: string[] = Array.from({ length: 772 }, (unused, i) => `p${i} (build)`); + for (const operationId of operationIds) { + renderer.onEvent(registered(operationId)); + } + for (let i: number = 0; i < 1200; i++) { + renderer.onEvent(registered(`p${i} (tool-build)`, true)); + } + for (const [index, operationId] of operationIds.entries()) { + clock.ms = index * 270; + renderer.onEvent(status(operationId, 'EXECUTING')); + renderer.onLog(Buffer.from(`building ${operationId}\n`), operationId, 'stdout'); + renderer.onEvent(status(operationId, index % 2 ? 'FROM CACHE' : 'SUCCESS')); + renderer.onEvent(header(operationId, index + 1, 772)); + renderer.onEvent(event('activityChanged', { text: `${index + 1} of 772 operations complete` })); + } + for (let i: number = 0; i < 1200; i++) { + renderer.onEvent(status(`p${i} (tool-build)`, 'NO OP')); + } + clock.ms = 210_000; + renderer.finish({ exitCode: 0 }); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush build · 0.0s · queued behind another request (position 1)', + 'rush build 0/772 · 0.0s · running', + 'rush build: SUCCESS 772/772 operations (386 success, 386 from cache) in 210.0s' + ]); + }); + + it('keeps a failure at odsp-web scale to the progress lines, the failure report and one summary line', () => { + const { renderer, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + for (let i: number = 0; i < 772; i++) { + renderer.onEvent(registered(`p${i} (build)`)); + } + for (let i: number = 0; i < 700; i++) { renderer.onEvent(status(`p${i} (build)`, 'EXECUTING')); + renderer.onLog(Buffer.from(`${'noise '.repeat(20)}\n`.repeat(50)), `p${i} (build)`, 'stdout'); + renderer.onEvent(status(`p${i} (build)`, 'SUCCESS')); } - expect(output).toHaveLength(1); - clock.ms = 2500; - renderer.onEvent(status('p0 (build)', 'SUCCESS')); - expect(output).toHaveLength(2); - clock.ms = 3000; - renderer.onEvent(status('p1 (build)', 'SUCCESS')); - expect(output).toHaveLength(2); - renderer.dispose(); + fail(renderer, 'p700 (build)', [ + 'src/x.ts:1:1 - error TS2304: Cannot find name "y".', + 'Encountered 1 error' + ]); + for (let i: number = 701; i < 772; i++) { + renderer.onEvent(status(`p${i} (build)`, 'BLOCKED')); + } + renderer.finish({ exitCode: 1 }); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush build 0/772 · 0.0s · running', + 'rush build 701/772 · 0.0s · running · first failure: p700 (build)', + 'failed: p700 (build) · full log: /repo/p700/rush-logs/x.log', + ' src/x.ts:1:1 - error TS2304: Cannot find name "y".', + ' Encountered 1 error', + 'rush build: FAILURE 772/772 operations (1 failure, 71 blocked, 700 success) in 0.0s · failed: p700 (build)' + ]); + }); + + it('ignores silent operations in the counters unless they fail', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(registered('s1 (tool-build)', true)); + renderer.onEvent(registered('s2 (tool-build)', true)); + renderer.onLog(Buffer.from('silent output\n'), 's1 (tool-build)', 'stderr'); + renderer.onEvent(status('s1 (tool-build)', 'NO OP')); + renderer.onEvent(status('s2 (tool-build)', 'FAILURE')); + renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.finish({ exitCode: 1 }); + const text: string = output.join(''); + expect(text).not.toContain('silent output'); + expect(output[output.length - 1]).toMatch( + /rush build: FAILURE 2\/2 operations \(1 failure, 1 success\) in 0\.0s · failed: s2 \(tool-build\)\n$/ + ); + }); + + it('uses the per-request total from the operation header', () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(header('a (build)', 1, 772)); + renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.finish({ exitCode: 0 }); + expect(lines()).toEqual([ + 'rush build 1/772 · 0.0s · running', + 'rush build: SUCCESS 1/772 operations (1 success) in 0.0s' + ]); + }); + + it('counts an operation that runs again once', () => { + const { renderer, output } = createRenderer(false); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(status('a (build)', 'FAILURE')); + renderer.onEvent(status('a (build)', 'READY')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.finish({ exitCode: 0 }); + expect(output[output.length - 1]).toBe('rush build: SUCCESS 1/1 operations (1 success) in 0.0s\n'); + }); + + it('reports operations with warnings when the warnings failed the request', () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(status('w (build)', 'EXECUTING')); + renderer.onLog( + Buffer.from('[build:lint] Warning: src/x.ts:1:1 - (rule) message\n'), + 'w (build)', + 'stderr' + ); + renderer.onEvent(status('w (build)', 'SUCCESS WITH WARNINGS', '/repo/w/rush-logs/w._phase_build.log')); + renderer.finish({ exitCode: 1 }); + expect(lines().slice(-3)).toEqual([ + 'warnings: w (build) · full log: /repo/w/rush-logs/w._phase_build.log', + ' [build:lint] Warning: src/x.ts:1:1 - (rule) message', + 'rush build: FAILURE 1/1 operations (1 success with warnings) in 0.0s · warnings: w (build)' + ]); + }); + + it('does not report warnings when the request succeeded', () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(status('w (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('Warning: x\n'), 'w (build)', 'stderr'); + renderer.onEvent(status('w (build)', 'SUCCESS WITH WARNINGS')); + renderer.finish({ exitCode: 0 }); + expect(lines().slice(-1)).toEqual([ + 'rush build: SUCCESS 1/1 operations (1 success with warnings) in 0.0s' + ]); + expect(lines().join('\n')).not.toContain('Warning: x'); + }); + + it('caps the reported operations, their excerpts and the names in the summary', () => { + const { renderer, lines } = createRenderer(false); + for (let i: number = 0; i < 8; i++) { + fail( + renderer, + `f${i} (build)`, + Array.from({ length: 12 }, (unused, j) => `f${i} error ${j}`) + ); + } + renderer.finish({ exitCode: 1 }); + const report: string[] = lines().slice(2); + expect(report.filter((line) => line.startsWith('failed: '))).toEqual([ + 'failed: f0 (build) · full log: /repo/f0/rush-logs/x.log', + 'failed: f1 (build) · full log: /repo/f1/rush-logs/x.log', + 'failed: f2 (build) · full log: /repo/f2/rush-logs/x.log' + ]); + expect(report.filter((line) => line.startsWith(' f0 '))).toHaveLength(8); + expect(report.filter((line) => line.startsWith(' f1 '))).toHaveLength(3); + expect(report.filter((line) => line.startsWith(' f2 '))).toHaveLength(3); + expect(report.slice(-2)).toEqual([ + "+5 more failed operations; their logs are in each project's rush-logs folder", + 'rush build: FAILURE 8/8 operations (8 failure) in 0.0s · failed: f0 (build), f1 (build), f2 (build), ' + + 'f3 (build), f4 (build) +3 more' + ]); + }); + + it('shows the output of a command that failed without running operations', () => { + const { renderer, lines } = createRenderer(false, 'install'); + renderer.onLog(Buffer.from('Installing packages\n'), 'request-id', 'stdout'); + renderer.onLog( + Buffer.from('ERR_PNPM_FETCH_401 GET https://registry.example/pkg: Unauthorized\n'), + 'request-id', + 'stderr' + ); + renderer.finish({ exitCode: 1 }); + expect(lines()).toEqual([ + ' ERR_PNPM_FETCH_401 GET https://registry.example/pkg: Unauthorized', + 'rush install: FAILURE in 0.0s' + ]); + }); + + it('does not claim a successful global command was up to date', () => { + const { renderer, output } = createRenderer(false, 'install'); + renderer.onLog(Buffer.from('Installing packages\n'), 'request-id', 'stdout'); + renderer.finish({ exitCode: 0 }); + expect(output).toEqual(['rush install: SUCCESS in 0.0s\n']); + }); + + describe('on a pipe, with timers', () => { + beforeEach(() => jest.useFakeTimers()); + afterEach(() => jest.useRealTimers()); + + function advance(clock: { ms: number }, ms: number): void { + clock.ms += ms; + jest.advanceTimersByTime(ms); + } + + it('writes the connecting line only when the connection takes more than a second', () => { + const { renderer, output, clock, lines } = createRenderer(false); + renderer.start(); + advance(clock, 999); + expect(output).toEqual([]); + advance(clock, 1); + renderer.onRequestSent(); + renderer.dispose(); + expect(lines()).toEqual([ + 'rush build · 1.0s · connecting to rushd (auto-starts if needed)', + 'rush build · 1.0s · sent to rushd; preparing the workspace graph (status at least every 25s)' + ]); + }); + + it('writes one line when the client waits for a daemon that is still starting (task 95)', () => { + const { renderer, output, clock, lines } = createRenderer(false); + renderer.start(); + advance(clock, 300); + renderer.onAwaitStartup(15_000); + expect(lines()).toEqual([ + 'rush build · 0.3s · rushd is still starting; waiting for it (up to 15s more)' + ]); + advance(clock, 1_000); + renderer.onAwaitStartup(15_000); + expect(output).toHaveLength(1); + advance(clock, 11_000); + renderer.onRequestSent(); + renderer.dispose(); + expect(lines()).toEqual([ + 'rush build · 0.3s · rushd is still starting; waiting for it (up to 15s more)', + 'rush build · 12.3s · sent to rushd; preparing the workspace graph (status at least every 25s)' + ]); + }); + + it('writes nothing for a request that is handed to in-process Rush within a second', () => { + const { renderer, output, clock } = createRenderer(false); + renderer.start(); + advance(clock, 500); + renderer.dispose(); + advance(clock, 60_000); + expect(output).toEqual([]); + }); + + it('writes a status line after 25 s of silence, with the running and failed operations', () => { + const { renderer, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + for (const name of ['a', 'b', 'c', 'd', 'e']) { + renderer.onEvent(registered(`${name} (build)`)); + } + advance(clock, 20_000); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onEvent(status('b (build)', 'EXECUTING')); + advance(clock, 24_999); + expect(lines()).toHaveLength(2); + advance(clock, 1); + renderer.onEvent(status('c (build)', 'EXECUTING')); + renderer.onEvent(status('d (build)', 'EXECUTING')); + renderer.onEvent(status('e (build)', 'EXECUTING')); + fail(renderer, 'a (build)', ['error TS2322']); + advance(clock, 25_000); + renderer.onEvent(status('b (build)', 'SUCCESS')); + advance(clock, 25_000); + clock.ms += 1000; + renderer.finish({ exitCode: 1 }); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush build 0/5 · 20.0s · running', + 'rush build 0/5 · 45.0s · running: a (build), b (build)', + // The first failure is a milestone, so it is written although status lines were written before it. + 'rush build 1/5 · 45.0s · running · first failure: a (build)', + 'rush build 1/5 · 70.0s · running: b (build), c (build), d (build) +1 more · failed: a (build)', + 'rush build 2/5 · 95.0s · running: c (build), d (build), e (build) · failed: a (build)', + 'failed: a (build) · full log: /repo/a/rush-logs/x.log', + ' error TS2322', + 'rush build: FAILURE 2/5 operations (1 failure, 1 success) in 96.0s · failed: a (build)' + ]); + }); + + it('does not claim that a request is still queued once it may have been admitted', () => { + const { renderer, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + advance(clock, 2000); + renderer.onQueuePosition(1); + advance(clock, 25_000); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + advance(clock, 25_000); + renderer.dispose(); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush build · 2.0s · queued behind another request (position 1)', + 'rush build · 27.0s · waiting for admission or the workspace graph (queue position 1 at 2.0s)', + 'rush build 0/1 · 27.0s · running', + 'rush build 0/1 · 52.0s · running: a (build)' + ]); + }); + + it('writes status lines after the milestone lines ran out, and none after the summary', () => { + const { renderer, output, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + renderer.onQueuePosition(1); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + fail(renderer, 'a (build)', ['error']); + expect(lines()).toHaveLength(3); + advance(clock, 25_000); + expect(lines()[3]).toBe('rush build 1/1 · 25.0s · running · failed: a (build)'); + renderer.finish({ exitCode: 1 }); + const written: number = output.length; + advance(clock, 100_000); + expect(output).toHaveLength(written); + }); }); it('renders at most three live rows on a TTY and clears them before the summary', () => { @@ -196,4 +745,40 @@ describe(AgentProgressRenderer.name, () => { expect(output[output.length - 2]).toBe('\x1b[3A\x1b[0J\x1b[?25h'); expect(output[output.length - 1]).toContain('rush build: SUCCESS'); }); + + it('shows the wait for a daemon that is still starting as the phase on a TTY', () => { + const { renderer, output } = createRenderer(true); + renderer.start(); + renderer.onAwaitStartup(15_000); + renderer.dispose(); + expect(output).toHaveLength(3); + // The row is clipped to the 60 columns of the test terminal. + expect(output[1].replace(ANSI_ESCAPE, '').split('\n')[0]).toMatch( + /^. rush build · 0\.0s · rushd is still starting; waiting for…$/ + ); + }); + + it('repaints a TTY on its timer rather than on every event, and shows failures in the last row', () => { + jest.useFakeTimers(); + try { + const { renderer, output } = createRenderer(true); + renderer.start(); + for (let i: number = 0; i < 100; i++) { + renderer.onEvent(status(`p${i} (build)`, 'EXECUTING')); + } + renderer.onEvent(status('p0 (build)', 'FAILURE')); + expect(output).toHaveLength(1); + jest.advanceTimersByTime(100); + expect(output).toHaveLength(2); + const rows: string[] = output[1].replace(ANSI_ESCAPE, '').split('\n'); + expect(rows[0]).toMatch(/^. rush build 1\/100 · 0\.0s · running$/); + expect(rows[1]).toBe('running: p1 (build), p2 (build), p3 (build) +96 more'); + expect(rows[2]).toBe('failed: p0 (build)'); + renderer.dispose(); + jest.advanceTimersByTime(1000); + expect(output).toHaveLength(3); + } finally { + jest.useRealTimers(); + } + }); }); diff --git a/apps/rush-cli-client/src/test/GraphGenerationWire.test.ts b/apps/rush-cli-client/src/test/GraphGenerationWire.test.ts index f4ec40eb3a..36c934b8ac 100644 --- a/apps/rush-cli-client/src/test/GraphGenerationWire.test.ts +++ b/apps/rush-cli-client/src/test/GraphGenerationWire.test.ts @@ -30,6 +30,7 @@ import { import { DaemonFrameListener, type DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; import { getDaemonConnectionOptions } from '../daemonConnectionOptions'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; describe('generation-aware graph reference client', () => { it.each([ @@ -146,7 +147,7 @@ describe('generation-aware graph reference client', () => { ], { cwd: folder, - env: { ...process.env, RUSH_DAEMON_EXPERIMENTAL: '1' }, + env: { ...getTestProcessEnvironment(), RUSH_DAEMON_EXPERIMENTAL: '1' }, stdio: ['ignore', 'pipe', 'pipe'] } ); @@ -384,7 +385,7 @@ describe('generation-aware graph reference client', () => { { cwd: folder, env: { - ...process.env, + ...getTestProcessEnvironment(), RUSH_DAEMON_EXPERIMENTAL: '1', RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: mode === 'environment-timeout' ? '0.25' : undefined }, diff --git a/apps/rush-cli-client/src/test/NativeBuildTestFixture.test.ts b/apps/rush-cli-client/src/test/NativeBuildTestFixture.test.ts index 79670f2790..e6e812b3ce 100644 --- a/apps/rush-cli-client/src/test/NativeBuildTestFixture.test.ts +++ b/apps/rush-cli-client/src/test/NativeBuildTestFixture.test.ts @@ -9,6 +9,36 @@ import * as path from 'node:path'; import { createNativeBuildTestFixture } from './NativeBuildTestFixture'; describe('native build fixture lifetime', () => { + it('gives spawned clients the default output when the tests run in an agent shell', async () => { + const saved: Record = { + COPILOT_CLI: process.env.COPILOT_CLI, + RUSHD_OUTPUT: process.env.RUSHD_OUTPUT + }; + process.env.COPILOT_CLI = '1'; + process.env.RUSHD_OUTPUT = 'agent'; + const fixture = createNativeBuildTestFixture(); + try { + expect(fixture.environment.COPILOT_CLI).toBeUndefined(); + expect(fixture.environment.RUSHD_OUTPUT).toBeUndefined(); + await fixture.runAsync(async ({ invokeAsync }) => { + const result = await invokeAsync(['build', '--to', 'a', '--verbose']); + expect(result.code).toBe(0); + expect(result.stderr).not.toMatch(/using in-process/i); + expect(result.stdout).toContain('built-a-one'); + expect(result.stdout).not.toContain('connecting to rushd'); + }); + } finally { + for (const [name, value] of Object.entries(saved)) { + if (value === undefined) { + delete process.env[name]; + } else { + process.env[name] = value; + } + } + await fixture.closeAsync(); + } + }, 60000); + it('joins the whole old callback without rebinding it to a later fixture', async () => { const old = createNativeBuildTestFixture(); const next = createNativeBuildTestFixture(); diff --git a/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts b/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts index ace2ae4152..659e65647a 100644 --- a/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts +++ b/apps/rush-cli-client/src/test/NativeBuildTestFixture.ts @@ -17,6 +17,8 @@ import { type IDaemonPaths } from '@rushstack/rush-daemon-transport'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; + export interface INativeBuildResult { readonly code: number | undefined; readonly stdout: string; @@ -41,7 +43,7 @@ export interface INativeBuildTestFixture { export function createNativeBuildTestFixture(): INativeBuildTestFixture { const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-client-native-')); const environment: NodeJS.ProcessEnv = { - ...process.env, + ...getTestProcessEnvironment(), RUSH_DAEMON: '1', RUSH_REPORTER: 'legacy', CI: 'false', diff --git a/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts b/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts new file mode 100644 index 0000000000..4e241e65b0 --- /dev/null +++ b/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts @@ -0,0 +1,255 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { + OperationOutputExcerpt, + clipLine, + isErrorLine, + normalizeExcerptLine +} from '../OperationOutputExcerpt'; + +const ESC: string = String.fromCharCode(27); + +// Shaped like a Heft plugin that fails to load: the cause is on one line, followed by a long require stack. +const REQUIRE_STACK_FAILURE: string = [ + '', + 'Internal Error: Could not load plugin from "/repo/tools/example-plugin/lib/ExamplePlugin.js": ' + + "Error: Cannot find module '/repo/tools/example-plugin/lib/ExamplePlugin.js'", + 'Require stack:', + ...Array.from( + { length: 8 }, + (unused, i) => + `- /repo/common/temp/node_modules/.pnpm/@rushstack+heft@1.2.3/node_modules/@rushstack/heft/lib/m${i}.js` + ), + '', + 'You have encountered a software defect. Please consider reporting the issue to the maintainers of this application.', + '' +].join('\n'); + +// Shaped like Heft's Jest plugin output: prefixed lines, a message, a code frame and stack frames per test. +function jestFailure(testNames: ReadonlyArray): string { + const lines: string[] = []; + for (const [index, testName] of testNames.entries()) { + lines.push( + `[test:jest] ● ${testName}`, + '[test:jest] ', + '[test:jest] 1', + '[test:jest] ', + '[test:jest] ERROR: The following environment variables were found with the "RUSH_" prefix,', + '[test:jest] but they are not recognized by this version of Rush: RUSH_EXAMPLE', + '[test:jest] ', + '[test:jest] 11 | if (result.status !== 0) {', + '[test:jest] > 12 | throw new Error(`${result.status}`);', + '[test:jest] | ^', + '[test:jest] 13 | }', + '[test:jest] ', + `[test:jest] at runStep (src/test/example.test.ts:12:11)`, + `[test:jest] at Object.runStep (src/test/example.test.ts:${40 + index}:3)`, + '[test:jest] ' + ); + } + lines.push(`[test:jest] Error: ${testNames.length} Jest tests failed`, 'Encountered 1 error', ''); + return lines.join('\n'); +} + +describe(normalizeExcerptLine.name, () => { + it('removes colors, carriage-return overwrites and the whitespace after a task prefix', () => { + expect(normalizeExcerptLine(`${ESC}[31merror${ESC}[39m TS2304: x`)).toBe('error TS2304: x'); + expect(normalizeExcerptLine('progress 10%\rprogress 100%\r')).toBe('progress 100%'); + expect(normalizeExcerptLine('[build:lint] Warning: src/x.ts:1:1')).toBe( + '[build:lint] Warning: src/x.ts:1:1' + ); + expect(normalizeExcerptLine(`${ESC}]8;;https://example.com${String.fromCharCode(7)}link`)).toBe('link'); + }); + + it('drops empty, prefix-only and noise lines', () => { + for (const line of [ + '', + ' ', + '[test:jest] ', + ' at Object. (src/x.test.ts:12:19)', + ' at async Promise.all (index 0)', + ' at new Promise ()', + 'Require stack:', + '- /repo/node_modules/x/lib/index.js', + '- C:\\repo\\node_modules\\x\\lib\\index.js', + '[test:jest] > 12 | throw new Error();', + '[test:jest] | ^', + ' ---- build started ---- ', + '-------------------- Finished (12.407s) --------------------', + 'Invoking: heft run --only build -- --clean ', + 'Invoking (incremental): heft run --only build --', + 'This project was not found in the build cache.', + 'Build cache hit.', + 'Caching build output folders: lib, lib-commonjs', + 'Successfully set cache entry.' + ]) { + expect(normalizeExcerptLine(line)).toBeUndefined(); + } + }); + + it('keeps English lines that merely start with "at"', () => { + expect(normalizeExcerptLine('at least one project failed')).toBe('at least one project failed'); + }); + + it('clips very long lines, keeping their start and end', () => { + const line: string | undefined = normalizeExcerptLine( + `start ${'x'.repeat(10000)} Cannot find module 'y'` + ); + expect(line?.length).toBe(300); + expect(line?.startsWith('start ')).toBe(true); + expect(line?.endsWith("Cannot find module 'y'")).toBe(true); + }); +}); + +describe(isErrorLine.name, () => { + it.each([ + 'src/x.ts(1,1): error TS1005: expected', + 'src/x.ts:1:1 - error TS2304: Cannot find name', + '[test:jest] ● suite › test', + 'Error: 3 Jest tests failed', + 'npm ERR! code ELIFECYCLE', + 'ERR_PNPM_FETCH_401 GET https://registry.example/pkg: Unauthorized', + 'Encountered 1 error', + 'FAIL src/x.test.ts', + 'Could not load plugin' + ])('recognizes %s', (line) => { + expect(isErrorLine(line)).toBe(true); + }); + + it.each([ + 'Found 0 errors. Watching for file changes.', + 'Linting finished with no errors', + 'Compiling 12 files' + ])('ignores %s', (line) => { + expect(isErrorLine(line)).toBe(false); + }); +}); + +describe(clipLine.name, () => { + it('leaves short lines alone', () => { + expect(clipLine('short', 10)).toBe('short'); + }); + + it('keeps the start and end of long lines', () => { + expect(clipLine('abcdefghijklmnopqrstuvwxyz', 10)).toBe('abcdef…xyz'); + }); +}); + +describe(OperationOutputExcerpt.name, () => { + it('shows the cause of a plugin load failure instead of its require stack', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append(REQUIRE_STACK_FAILURE, 'stderr'); + const lines: string[] = excerpt.getExcerpt(8); + expect(lines).toHaveLength(2); + expect(lines[0]).toMatch(/^Internal Error: Could not load plugin/); + expect(lines[0]).toContain("Cannot find module '"); + expect(lines[1]).toMatch(/^You have encountered a software defect/); + }); + + it('shows the failing tests, the message and the summary of a Jest failure', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append('[test:jest] PASS src/test/other.test.ts\n', 'stdout'); + excerpt.append(jestFailure(['test one', 'test two', 'test three']), 'stderr'); + expect(excerpt.getExcerpt(8)).toEqual([ + '[test:jest] ● test one', + '[test:jest] ERROR: The following environment variables were found with the "RUSH_" prefix,', + '[test:jest] but they are not recognized by this version of Rush: RUSH_EXAMPLE', + '[test:jest] ● test two', + '[test:jest] ● test three', + '[test:jest] Error: 3 Jest tests failed', + 'Encountered 1 error' + ]); + expect(excerpt.getExcerpt(3)).toEqual([ + '[test:jest] ● test one', + '[test:jest] Error: 3 Jest tests failed', + 'Encountered 1 error' + ]); + }); + + it('finds errors that tools report on stdout', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append('[build:typescript] Using TypeScript version 5.8.2\n', 'stdout'); + excerpt.append('[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".\n', 'stdout'); + excerpt.append('[build:typescript] Encountered 1 error\n', 'stderr'); + expect(excerpt.getExcerpt(8)).toEqual([ + '[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".', + '[build:typescript] Encountered 1 error' + ]); + }); + + it("does not spend the error lines of a short excerpt on Rush's own cache and invocation lines", () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append('This project was not found in the build cache.\n', 'stdout'); + excerpt.append('Invoking (initial): heft run --only build -- --clean \n', 'stdout'); + excerpt.append('[build:typescript] Using TypeScript version 5.8.2\n', 'stdout'); + excerpt.append('[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".\n', 'stderr'); + excerpt.append('[build:lint] Using ESLint version 9.37.0\n', 'stdout'); + excerpt.append('Error: Encountered 1 error\n', 'stderr'); + expect(excerpt.getExcerpt(3)).toEqual([ + '[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".', + 'Error: Encountered 1 error' + ]); + }); + + it('takes the line after an error from the same stream, not from the interleaved other stream', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append( + "src/x.ts(3,7): error TS2322: Type 'string' is not assignable to type 'number'.\n", + 'stderr' + ); + excerpt.append('[build:lint] still linting file 4\n', 'stdout'); + excerpt.append(' The expected type comes from property "x".\n', 'stderr'); + excerpt.append("src/y.ts(1,1): error TS2304: Cannot find name 'z'.\n", 'stderr'); + excerpt.append('[build:lint] still linting file 5\n', 'stdout'); + excerpt.append(' Did you mean "y"?\n', 'stderr'); + excerpt.append('Encountered 2 errors\n', 'stderr'); + expect(excerpt.getExcerpt(5)).toEqual([ + "src/x.ts(3,7): error TS2322: Type 'string' is not assignable to type 'number'.", + 'The expected type comes from property "x".', + "src/y.ts(1,1): error TS2304: Cannot find name 'z'.", + 'Did you mean "y"?', + 'Encountered 2 errors' + ]); + }); + + it('always keeps the last line, which usually is the tool summary', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + for (let i: number = 0; i < 20; i++) { + excerpt.append(`error ${i}\n`, 'stderr'); + } + const lines: string[] = excerpt.getExcerpt(8); + expect(lines).toHaveLength(8); + expect(lines[0]).toBe('error 0'); + expect(lines[lines.length - 1]).toBe('error 19'); + }); + + it('joins lines split across chunks and records an unterminated last line', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append('src/x.ts(1,1): err', 'stdout'); + excerpt.append('or TS1005: expected\nlast line without newline', 'stdout'); + expect(excerpt.getExcerpt(8)).toEqual([ + 'src/x.ts(1,1): error TS1005: expected', + 'last line without newline' + ]); + }); + + it('does not repeat a line that was written to both streams', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append('Error: boom\n', 'stdout'); + excerpt.append('Error: boom\n', 'stderr'); + expect(excerpt.getExcerpt(8)).toEqual(['Error: boom']); + }); + + it('keeps memory use flat for huge output', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + for (let i: number = 0; i < 100000; i++) { + excerpt.append(`line ${i} ${'x'.repeat(100)}\n`, 'stdout'); + } + excerpt.append('x'.repeat(200000), 'stdout'); + expect(excerpt.lineCount).toBe(100001); + const lines: string[] = excerpt.getExcerpt(8); + expect(lines).toHaveLength(8); + expect(lines[lines.length - 1]).toMatch(/^x+…x+$/); + }); +}); diff --git a/apps/rush-cli-client/src/test/RushXDaemonTestFixture.ts b/apps/rush-cli-client/src/test/RushXDaemonTestFixture.ts index 389a3133f8..0f5a87b511 100644 --- a/apps/rush-cli-client/src/test/RushXDaemonTestFixture.ts +++ b/apps/rush-cli-client/src/test/RushXDaemonTestFixture.ts @@ -22,6 +22,8 @@ import { } from '@rushstack/rush-daemon'; import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; + export interface IScriptResult { readonly exitCode: number | undefined; readonly stdout: Buffer; @@ -155,7 +157,7 @@ setInterval(() => {}, 1000); public environment(overrides: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { return { - ...process.env, + ...getTestProcessEnvironment(), HOME: this.home, USERPROFILE: this.home, CLIENT_MARKER: 'client', diff --git a/apps/rush-cli-client/src/test/TestProcessEnvironment.ts b/apps/rush-cli-client/src/test/TestProcessEnvironment.ts new file mode 100644 index 0000000000..df8bac55b2 --- /dev/null +++ b/apps/rush-cli-client/src/test/TestProcessEnvironment.ts @@ -0,0 +1,18 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { CLIENT_OUTPUT_SELECTION_ENV_VARS } from '../outputSelection'; + +/** + * Returns a copy of `base` (by default `process.env`) for the clients that tests spawn, without the variables + * that select the client's output mode (`RUSHD_OUTPUT` and agent markers such as `COPILOT_CLI`). Spawned + * clients then use the default output even when the tests run in an agent's shell. Tests of agent output set + * those variables explicitly. + */ +export function getTestProcessEnvironment(base: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv { + const environment: NodeJS.ProcessEnv = { ...base }; + for (const name of CLIENT_OUTPUT_SELECTION_ENV_VARS) { + delete environment[name]; + } + return environment; +} diff --git a/apps/rush-cli-client/src/test/daemonGraph.test.ts b/apps/rush-cli-client/src/test/daemonGraph.test.ts index 4305314b83..fd843d4e95 100644 --- a/apps/rush-cli-client/src/test/daemonGraph.test.ts +++ b/apps/rush-cli-client/src/test/daemonGraph.test.ts @@ -21,6 +21,7 @@ import { import { DaemonFrameListener, type DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; import { getDaemonConnectionOptions } from '../daemonConnectionOptions'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; describe('graph client fails closed over the public wire', () => { it.each(['unsupported', 'missing-snapshot', 'invalid-snapshot', 'old-protocol'])( @@ -108,7 +109,7 @@ describe('graph client fails closed over the public wire', () => { [path.resolve(__dirname, '../../bin/rush-client'), 'daemon', 'graph', 'show'], { cwd: folder, - env: { ...process.env, RUSH_DAEMON_EXPERIMENTAL: '1' }, + env: { ...getTestProcessEnvironment(), RUSH_DAEMON_EXPERIMENTAL: '1' }, stdio: ['ignore', 'pipe', 'pipe'] } ); diff --git a/apps/rush-cli-client/src/test/daemonLogs.test.ts b/apps/rush-cli-client/src/test/daemonLogs.test.ts index a098d9276d..630984b13a 100644 --- a/apps/rush-cli-client/src/test/daemonLogs.test.ts +++ b/apps/rush-cli-client/src/test/daemonLogs.test.ts @@ -15,6 +15,7 @@ import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; import { getDaemonConnectionOptions } from '../daemonConnectionOptions'; import { printDaemonLogAsync } from '../daemonLogs'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; async function waitUntilAsync(predicate: () => boolean): Promise { const deadline: number = Date.now() + 5000; @@ -159,7 +160,7 @@ describe('daemon launcher log following', () => { ], { cwd: folder, - env: { ...process.env, RUSH_DAEMON: '1' }, + env: { ...getTestProcessEnvironment(), RUSH_DAEMON: '1' }, stdio: windowsSignal ? ['ignore', 'pipe', 'pipe', 'ipc'] : ['ignore', 'pipe', 'pipe'] } ); @@ -213,7 +214,11 @@ describe('daemon launcher log following', () => { const child = spawn( process.execPath, [path.resolve(__dirname, '../../bin/rush-client'), 'daemon', 'logs', '--follow'], - { cwd: folder, env: { ...process.env, RUSH_DAEMON: '1' }, stdio: ['ignore', 'pipe', 'pipe'] } + { + cwd: folder, + env: { ...getTestProcessEnvironment(), RUSH_DAEMON: '1' }, + stdio: ['ignore', 'pipe', 'pipe'] + } ); const closed = once(child, 'close'); const readable = once(child.stdout, 'readable'); @@ -268,7 +273,7 @@ describe('daemon launcher log following', () => { ], { cwd: folder, - env: { ...process.env, RUSH_DAEMON: '1' }, + env: { ...getTestProcessEnvironment(), RUSH_DAEMON: '1' }, stdio: windowsSignal ? ['ignore', outputFd, 'pipe', 'ipc'] : ['ignore', outputFd, 'pipe'] } ); diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index e8f70172b5..eadfc26bdc 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -27,6 +27,7 @@ import { getSignalExitCode, isCancelledOutcome } from '../clientCancellation'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; interface IInvocationResult { readonly code: number | undefined; @@ -106,7 +107,7 @@ describe('standalone rushx fallback', () => { const child: ChildProcess = spawn(process.execPath, args, { cwd: project, env: { - ...process.env, + ...getTestProcessEnvironment(), CLIENT_MARKER: 'script-output', RUSH_DAEMON: optIn ? '1' : '0', CI: managementArgs ? 'true' : 'false', @@ -661,6 +662,22 @@ describe('daemon client cancellation exit codes', () => { expect(isCancelledOutcome(cancelledWithFailure, true)).toBe(true); }); + it('does not report a request that a daemon shutdown aborted as cancelled, unless the client was signalled', () => { + const shutDown: DaemonClientOutcome = { + kind: 'result', + result: { + requestId: 'r', + outcome: 'failure', + exitCode: 1, + aborted: true, + errorMessage: + 'The Rush daemon was shut down (idle timeout) while this request was running; re-run the command.' + } + }; + expect(isCancelledOutcome(shutDown, false)).toBe(false); + expect(isCancelledOutcome(shutDown, true)).toBe(true); + }); + it('keeps completed results and rejections when a signal arrives late', () => { const succeeded: DaemonClientOutcome = { kind: 'result', @@ -670,9 +687,29 @@ describe('daemon client cancellation exit codes', () => { kind: 'rejected', rejection: { requestId: 'r', code: 'unsupportedProtocolVersion', message: 'no' } } as unknown as DaemonClientOutcome; + const invalid: DaemonClientOutcome = { + kind: 'rejected', + rejection: { requestId: 'r', code: 'invalidRequest', message: 'Unknown operation id.' } + }; expect(isCancelledOutcome(succeeded, true)).toBe(false); expect(isCancelledOutcome(rejected, true)).toBe(false); + expect(isCancelledOutcome(invalid, true)).toBe(false); expect(isCancelledOutcome({ kind: 'fallback', reason: 'unsupported' }, true)).toBe(true); expect(isCancelledOutcome({ kind: 'fallback', reason: 'unsupported' }, false)).toBe(false); }); + + it('treats a request that a signal cancelled before engine initialization as cancelled (#711)', () => { + // The daemon rejects a request that was cancelled while it prepared the workspace graph. + const cancelledBeforeEngine: DaemonClientOutcome = { + kind: 'rejected', + rejection: { + requestId: 'r', + code: 'routingFailed', + message: 'The request was cancelled before engine initialization.' + } + }; + expect(isCancelledOutcome(cancelledBeforeEngine, true)).toBe(true); + // Without a signal, the daemon failed to route the request for its own reason, such as a shutdown. + expect(isCancelledOutcome(cancelledBeforeEngine, false)).toBe(false); + }); }); diff --git a/apps/rush-cli-client/src/test/nativeMutation.test.ts b/apps/rush-cli-client/src/test/nativeMutation.test.ts index 44158a49ca..e7db540889 100644 --- a/apps/rush-cli-client/src/test/nativeMutation.test.ts +++ b/apps/rush-cli-client/src/test/nativeMutation.test.ts @@ -11,6 +11,8 @@ import { Rush } from '@microsoft/rush-lib'; import { SuccessfulMutationFixture } from '@rushstack/rush-daemon/lib/test/SuccessfulMutationFixture'; import { computeDaemonWorkspaceKey, resolveDaemonPaths } from '@rushstack/rush-daemon-transport'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; + interface IClientResult { readonly exitCode: number | undefined; readonly stdout: string; @@ -23,7 +25,7 @@ async function invokeAsync( ): Promise { const child = spawn(process.execPath, [path.resolve(__dirname, '../../bin/rush-client'), ...argv], { cwd: fixture.repoRoot, - env: { ...fixture.environment, RUSH_DAEMON: '1', RUSH_REPORTER: 'legacy' }, + env: { ...getTestProcessEnvironment(fixture.environment), RUSH_DAEMON: '1', RUSH_REPORTER: 'legacy' }, stdio: ['ignore', 'pipe', 'pipe'] }); let stdout: string = ''; diff --git a/apps/rush-cli-client/src/test/pipedInput.test.ts b/apps/rush-cli-client/src/test/pipedInput.test.ts index 0de06c1cc3..2b3e29a03b 100644 --- a/apps/rush-cli-client/src/test/pipedInput.test.ts +++ b/apps/rush-cli-client/src/test/pipedInput.test.ts @@ -16,6 +16,8 @@ import { type IDaemonRequestResolver } from '@rushstack/rush-daemon'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; + const SCRIPT: string = "const chunks=[];process.stdin.on('data',c=>{chunks.push(c);process.stdin.pause();" + 'setTimeout(()=>process.stdin.resume(),1);});' + @@ -86,7 +88,7 @@ describe('standalone client piped input', () => { const child = spawn(process.execPath, [entry, 'sample', ...admissionArgs], { cwd: project, env: { - ...process.env, + ...getTestProcessEnvironment(), RUSH_DAEMON: '1', RUSH_REPORTER: 'legacy', RUSH_QUIET_MODE: '1', diff --git a/apps/rush-cli-client/src/test/terminalColumns.test.ts b/apps/rush-cli-client/src/test/terminalColumns.test.ts new file mode 100644 index 0000000000..ea2fc00e57 --- /dev/null +++ b/apps/rush-cli-client/src/test/terminalColumns.test.ts @@ -0,0 +1,18 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { getTerminalColumns } from '../terminalColumns'; + +describe(getTerminalColumns.name, () => { + it('keeps a real terminal width', () => { + expect(getTerminalColumns({ columns: 123 })).toBe(123); + }); + + it.each([0, -1, 1.5, Number.NaN])('treats %p columns as unknown', (columns) => { + expect(getTerminalColumns({ columns })).toBeUndefined(); + }); + + it('treats a stream without a width as unknown', () => { + expect(getTerminalColumns({})).toBeUndefined(); + }); +}); diff --git a/common/changes/@rushstack/rush-cli-client/agent-admission-failure-code_2026-09-28-15-08.json b/common/changes/@rushstack/rush-cli-client/agent-admission-failure-code_2026-09-28-15-08.json new file mode 100644 index 0000000000..7bc9a6e9d8 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-admission-failure-code_2026-09-28-15-08.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, keep \"daemon admission failed ()\" in the summary line when the daemon did not admit a request, and word an empty selection like native Rush: \"the selection parameters did not match any projects\".", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/agent-empty-selection_2026-09-28-14-45.json b/common/changes/@rushstack/rush-cli-client/agent-empty-selection_2026-09-28-14-45.json new file mode 100644 index 0000000000..ce0d099f74 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-empty-selection_2026-09-28-14-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, report a selection that matched no projects as such, instead of as an up-to-date build.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/agent-error-detail_2026-09-28-13-30.json b/common/changes/@rushstack/rush-cli-client/agent-error-detail_2026-09-28-13-30.json new file mode 100644 index 0000000000..63291c8b1c --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-error-detail_2026-09-28-13-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, print at most eight further lines of a multi-line error message before the summary line, instead of the whole message on stderr.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/agent-output-scale_2026-09-28-13-00.json b/common/changes/@rushstack/rush-cli-client/agent-output-scale_2026-09-28-13-00.json new file mode 100644 index 0000000000..ffc75bbe1a --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-output-scale_2026-09-28-13-00.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Make agent output hold at repository scale: at most three progress lines on a pipe; name the failed (or warning) operations with an error-first excerpt and the path of their full log; count skipped and no-op operations as up to date; report Ctrl+C as CANCELLED.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/agent-pipe-status_2026-09-28-14-20.json b/common/changes/@rushstack/rush-cli-client/agent-pipe-status_2026-09-28-14-20.json new file mode 100644 index 0000000000..1509e2f741 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/agent-pipe-status_2026-09-28-14-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "On a pipe, agent output reports when the request was sent to rushd and prints a status line with the running operations at least every 25 seconds, so a long build or wait is never silent. A pseudo-terminal without a size is treated as a pipe.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/cancel-before-engine-init_2026-09-28-14-45.json b/common/changes/@rushstack/rush-cli-client/cancel-before-engine-init_2026-09-28-14-45.json new file mode 100644 index 0000000000..ee59cebc6e --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/cancel-before-engine-init_2026-09-28-14-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Report a command that Ctrl+C cancelled while the daemon prepared the workspace graph as cancelled (exit code 130), not as a daemon routing failure.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/daemon-shutdown-not-cancelled_2026-09-28-13-00.json b/common/changes/@rushstack/rush-cli-client/daemon-shutdown-not-cancelled_2026-09-28-13-00.json new file mode 100644 index 0000000000..a389a1c7e1 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/daemon-shutdown-not-cancelled_2026-09-28-13-00.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Report a request that a daemon shutdown aborted as a failure with the shutdown reason and exit code 1, instead of as a user cancellation with exit code 130.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t83-admission-reason_2026-09-28-18-55.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t83-admission-reason_2026-09-28-18-55.json new file mode 100644 index 0000000000..c6a497636f --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t83-admission-reason_2026-09-28-18-55.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, write the daemon's reason for an admission failure in full on the summary line, instead of shortening a long reason and then adding a generic wait-timeout line.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t83-startup-wait-line_2026-09-28-20-10.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t83-startup-wait-line_2026-09-28-20-10.json new file mode 100644 index 0000000000..ea5ac3b32d --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t83-startup-wait-line_2026-09-28-20-10.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output on a pipe, write a line when the client waits for a daemon that is still starting, as agent output on a TTY and legacy output already say.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-cli-client/zero-size-pty_2026-09-28-13-00.json b/common/changes/@rushstack/rush-cli-client/zero-size-pty_2026-09-28-13-00.json new file mode 100644 index 0000000000..13b18d2f36 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/zero-size-pty_2026-09-28-13-00.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Do not send a terminal width of 0 when stdout is a pseudo-terminal without a size; the daemon rejected every such request.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/restart-drain-default-timeout_2026-09-28-15-08.json b/common/changes/@rushstack/rush-client-core/restart-drain-default-timeout_2026-09-28-15-08.json new file mode 100644 index 0000000000..e9f7953174 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/restart-drain-default-timeout_2026-09-28-15-08.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Follow a daemon restart that takes longer than the client-default queue timeout; each successor daemon applies that default to its own admission. An explicit timeout is still one deadline across restarts.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/agent-output-log-path_2026-09-28-13-00.json b/common/changes/@rushstack/rush-daemon-protocol/agent-output-log-path_2026-09-28-13-00.json new file mode 100644 index 0000000000..d1ef986d9a --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/agent-output-log-path_2026-09-28-13-00.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add an optional `logFilePath` to operation status events, which points at the full log of a failed operation or an operation with warnings.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/restart-drain-default-timeout_2026-09-28-15-08.json b/common/changes/@rushstack/rush-daemon-protocol/restart-drain-default-timeout_2026-09-28-15-08.json new file mode 100644 index 0000000000..fee233931f --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/restart-drain-default-timeout_2026-09-28-15-08.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Document that a client-default admission timeout does not apply while a request that waits for the daemon to restart for its environment waits for requests that were already running while no rushx script is running.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/agent-output-log-path_2026-09-28-13-00.json b/common/changes/@rushstack/rush-daemon/agent-output-log-path_2026-09-28-13-00.json new file mode 100644 index 0000000000..87ef240c06 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/agent-output-log-path_2026-09-28-13-00.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Report the full log file path of failed operations and operations with warnings in their status events.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rejection-keeps-graph_2026-09-28-13-45.json b/common/changes/@rushstack/rush-daemon/rejection-keeps-graph_2026-09-28-13-45.json new file mode 100644 index 0000000000..f224b43ff3 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rejection-keeps-graph_2026-09-28-13-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Keep the workspace graph that a request loaded when only its project selection is invalid, so that the next request does not load the workspace again.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rejection-without-debug-lines_2026-09-28-13-30.json b/common/changes/@rushstack/rush-daemon/rejection-without-debug-lines_2026-09-28-13-30.json new file mode 100644 index 0000000000..51ecd71e72 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rejection-without-debug-lines_2026-09-28-13-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Leave the verbose and debug messages written while loading the workspace out of a rejected request's error message, and do not append \"An error occurred.\" to the error lines that already describe the failure.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/restart-drain-default-timeout_2026-09-28-15-08.json b/common/changes/@rushstack/rush-daemon/restart-drain-default-timeout_2026-09-28-15-08.json new file mode 100644 index 0000000000..c0b81bc376 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/restart-drain-default-timeout_2026-09-28-15-08.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Do not apply the client-default queue timeout while a request that waits for rushd to restart for its environment waits for requests that were already running while no rushx script is running, and do not count that wait against it; the default still limits the wait while a rushx script runs and waiting for later requests, and an explicit --wait-timeout or --no-wait applies to the whole wait. A restart wait that times out names the time that did not count, and suggests stopping a rushx script that it waits for.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/restart-drain-queue-position_2026-09-28-14-20.json b/common/changes/@rushstack/rush-daemon/restart-drain-queue-position_2026-09-28-14-20.json new file mode 100644 index 0000000000..7255dde156 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/restart-drain-queue-position_2026-09-28-14-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Report a request that waits for other requests to finish before rushd restarts for its environment as a queue position, so the client can say what it is waiting for.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index 1747837712..c7ec3d5863 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -410,6 +410,7 @@ export interface IDaemonOperationRegisteredPayload { // @beta export interface IDaemonOperationStatusChangedPayload { + readonly logFilePath?: string; readonly operationId: string; readonly previousStatus?: string; readonly status: string; diff --git a/libraries/rush-client-core/src/executeWithDaemonRestart.ts b/libraries/rush-client-core/src/executeWithDaemonRestart.ts index d63c6d3ce7..87f260e61f 100644 --- a/libraries/rush-client-core/src/executeWithDaemonRestart.ts +++ b/libraries/rush-client-core/src/executeWithDaemonRestart.ts @@ -3,7 +3,10 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; -import { DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR } from '@rushstack/rush-daemon-protocol'; +import { + DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR, + type IDaemonRequestAdmissionOptions +} from '@rushstack/rush-daemon-protocol'; import { readDaemonLockfile, type IDaemonLockfile } from '@rushstack/rush-daemon-transport'; import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from './connectOrStartDaemon'; @@ -29,8 +32,9 @@ const DEFAULT_STARTUP_TIMEOUT_MS: number = 15000; /** * Executes on a ready client, retrying only for a typed pre-execution restart. * Preserves the original request, callbacks and unread input; never retries connection loss. - * Restarts are retried with jittered backoff inside the request's admission deadline; once the - * retries or the deadline are exhausted, a `fallback` outcome lets the caller run in-process instead. + * Restarts are retried with jittered backoff inside the request's explicit admission deadline, if any (a + * client-default timeout applies to each daemon separately); once the retries or the deadline are exhausted, + * a `fallback` outcome lets the caller run in-process instead. * The connection options must select the request's expected daemon and startup environment. * A connection lost before the result is reported as a `disconnected` error that says whether the daemon * process exited, what its launcher log recorded and how to recover, unless the request was aborted first. @@ -46,7 +50,13 @@ export async function executeWithDaemonRestartAsync( execution.abortSignal && connection.abortSignal ? AbortSignal.any([execution.abortSignal, connection.abortSignal]) : (execution.abortSignal ?? connection.abortSignal); - const waitTimeoutMs: number | undefined = execution.request.admission?.waitTimeoutMs; + const admission: IDaemonRequestAdmissionOptions | undefined = execution.request.admission; + // A client-default timeout applies to each daemon's own admission, not to following its restarts: a daemon + // restarts only after the requests it serves finish, which is progress, so each successor gets the original + // request. An explicit timeout is one deadline across restarts. + const waitTimeoutMs: number | undefined = admission?.waitTimeoutIsDefault + ? undefined + : admission?.waitTimeoutMs; let owner: IDaemonLockfile | undefined = await attestOwnerAsync(client, connection); let outcome: DaemonClientOutcome = await executeOnDaemonAsync(client, connection, { ...execution, @@ -183,4 +193,4 @@ function abortedOutcome(execution: IDaemonClientExecuteOptions): DaemonClientOut kind: 'result', result: { requestId: execution.request.requestId, exitCode: 130, outcome: 'aborted', aborted: true } }; -} \ No newline at end of file +} diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index 8d7d89b397..324e7b01be 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -841,6 +841,44 @@ describe('detached daemon startup', () => { expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n')).toHaveLength(1); }); + it.each([ + ['a client-default timeout, which each successor applies afresh', true], + ['not an explicit timeout', false] + ])('follows a restart that takes longer than %s', async (name, isDefault) => { + // The first daemon answers only after the timeout, like one that restarts after a long build. + fs.writeFileSync(path.join(folder, 'drain-ms'), '800'); + const connection: IConnectOrStartDaemonOptions = { + ...options, + startCommand: { + ...options.startCommand!, + args: [...options.startCommand!.args, 'fixture', 'restart-once'] + } + }; + const client = await connectOrStartDaemonAsync(connection); + const request = captureDaemonRequest({ + argv: ['build'], + commandName: 'build', + commandOrigin: 'built-in', + cwd: folder, + environment: {}, + terminal: { isTTY: false, supportsColor: false }, + admission: isDefault ? { waitTimeoutMs: 300, waitTimeoutIsDefault: true } : { waitTimeoutMs: 300 } + }); + const outcome = await executeWithDaemonRestartAsync(client, connection, { request }); + const starts: string[] = fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n'); + const waits: string[] = fs.readFileSync(path.join(folder, 'waits'), 'utf8').trim().split('\n'); + if (isDefault) { + expect(outcome).toMatchObject({ kind: 'result', result: { exitCode: 0 } }); + expect(starts).toHaveLength(2); + expect(waits).toEqual(['300', '300']); + expect(fs.readFileSync(path.join(folder, 'default-waits'), 'utf8').trim().split('\n')).toHaveLength(2); + } else { + expect(outcome).toMatchObject({ kind: 'fallback', reason: 'restartRetriesExhausted' }); + expect(starts).toHaveLength(1); + expect(waits).toEqual(['300']); + } + }); + it('refuses restart retry if ownership was not attested before submitting', async () => { const connection: IConnectOrStartDaemonOptions = { ...options, diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index 6256888197..d4d05b7cab 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -83,6 +83,12 @@ async function mainAsync(): Promise { if (message.payload.admission?.waitTimeoutMs !== undefined) { fs.appendFileSync(path.join(folder, 'waits'), `${message.payload.admission.waitTimeoutMs}\n`); } + if (message.payload.admission?.waitTimeoutIsDefault) { + fs.appendFileSync( + path.join(folder, 'default-waits'), + `${message.payload.admission.waitTimeoutMs}\n` + ); + } const restartCount: number = fs.existsSync(path.join(folder, 'restarted')) ? fs.readFileSync(path.join(folder, 'restarted'), 'utf8').length : 0; @@ -90,6 +96,11 @@ async function mainAsync(): Promise { restartMode !== undefined && (restartMode !== 'restart-once' || restartCount < 1) && (restartMode !== 'restart-twice' || restartCount < 2); + const drainMsPath: string = path.join(folder, 'drain-ms'); + if (restart && fs.existsSync(drainMsPath)) { + // Like a daemon that restarts only after the requests it serves finish. + await new Promise((resolve) => setTimeout(resolve, Number(fs.readFileSync(drainMsPath, 'utf8')))); + } await connection.sendFrameAsync({ kind: DaemonFrameType.controlJson, payload: encodeDaemonControlMessage({ diff --git a/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts b/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts index 9f700d26c7..261a5a8c7d 100644 --- a/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts +++ b/libraries/rush-daemon-protocol/src/DaemonOperationPayloads.ts @@ -32,6 +32,11 @@ export interface IDaemonOperationStatusChangedPayload { readonly status: string; /** The previous raw engine status string, when known. */ readonly previousStatus?: string; + /** + * The absolute path of the operation's full text log, when the operation failed or succeeded with + * warnings and wrote a log. Clients that summarize output print it so the full output can be read later. + */ + readonly logFilePath?: string; } /** diff --git a/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts b/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts index 6be5d0c380..2320840043 100644 --- a/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts +++ b/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts @@ -13,8 +13,10 @@ export interface IDaemonRequestAdmissionOptions { /** Fail immediately when the request cannot be admitted. */ readonly noWait?: boolean; /** - * True when `waitTimeoutMs` is a client default rather than an explicit user choice. A default timeout bounds - * workspace admission only, not waiting behind running compatible shared builds. + * True when `waitTimeoutMs` is a client default rather than an explicit user choice. A default timeout applies + * to each daemon's workspace admission only: not to waiting behind running compatible shared builds, nor, when + * the daemon restarts for the request's environment, to waiting for the requests that it was already serving + * while it serves no rushx script. */ readonly waitTimeoutIsDefault?: boolean; /** Maximum queue wait in milliseconds. Omission means no timeout. */ diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 5b121b1c42..dc809142de 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -176,6 +176,17 @@ Successor startup reuses `connectOrStartDaemonAsync`: acknowledged old ownership resources finish, startup is serialized with ordinary clients, and hello/ping readiness attests a different PID. `restartCompleted` reports completion or failure. +A request whose environment needs another process does not restart the daemon while it serves other requests. It +first waits for the requests that this process is serving to finish (the restart drain), and its queue position is the +number of those requests. Like the graph-execution gate, waiting for the requests that were already being served when +the drain began is progress rather than contention: while one of them is still being served and no rushx script is, a +client-default `waitTimeoutMs` (`waitTimeoutIsDefault`) does not limit the drain and is not spent, and the client +sends the request to the successor with its default again. The default still limits the drain while a rushx script is +served, since a script may not exit until it is stopped, and while it waits for requests that arrived during the +drain, which could otherwise keep it waiting for as long as they keep arriving. An explicit `noWait` or +`waitTimeoutMs` limits the whole drain, and only its remaining time carries over to the successor. When a drain times +out, its message names the time that did not count. + Protocol 0.10 (`DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR`) provides bounded, typed retry authorization. Only a pre-execution command result may carry `retryAfterRestart: true`. During a planned restart, accepted queued requests drain those typed results before disconnect rather than being reduced to ambiguous connection @@ -475,9 +486,10 @@ build, still time out. Routing and executing an admitted request do not spend th such as the graph-execution gate apply the remaining budget they receive, and a request that re-enters workspace admission to reload the graph after its inputs changed starts again from the budget it had when it was admitted; time it spent at those boundaries is not charged again. The default and an explicit value differ only at the -graph-execution gate. When the client marks `waitTimeoutMs` as its default (`waitTimeoutIsDefault`), a `SHARED-BUILD` -request that arrives after the current batch has closed waits there without a deadline, because it is queued only -behind running compatible shared builds, and then runs in the next batch. An explicit value still limits that wait. +graph-execution gate and at a restart drain (see "Process restart and isolated install/update"). When the client marks +`waitTimeoutMs` as its default (`waitTimeoutIsDefault`), a `SHARED-BUILD` request that arrives after the current batch +has closed waits at the graph-execution gate without a deadline, because it is queued only behind running compatible +shared builds, and then runs in the next batch. An explicit value still limits that wait. Cancellation, disconnect, or queue-output failure removes queued work before it can execute. A requesting client receives only its enabled dependency closure's WS1 raw chunks and structured events through backpressured, ordered callbacks, followed exactly once by a typed final command result after all preceding output diff --git a/libraries/rush-daemon/src/EngineTerminalProvider.ts b/libraries/rush-daemon/src/EngineTerminalProvider.ts index cb13c5d782..403a5c51b8 100644 --- a/libraries/rush-daemon/src/EngineTerminalProvider.ts +++ b/libraries/rush-daemon/src/EngineTerminalProvider.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import type { IOperationGraph, _IOperationGraphEventSink } from '@microsoft/rush-lib'; +import { AlreadyReportedError } from '@rushstack/node-core-library'; import { TerminalProviderSeverity, type ITerminalProvider } from '@rushstack/terminal'; import { getEngineActivityOptions } from './EngineActivityOptions'; @@ -22,13 +23,23 @@ export class EngineTerminalProvider implements ITerminalProvider { /** * Drains buffered diagnostics into the failure description, so that they belong to the failing request - * and are never replayed into a later request. + * and are never replayed into a later request. Like the request output, the description omits verbose and + * debug messages unless the graph runs in debug mode: loading a large workspace writes thousands of them. */ public describeError(error: unknown): string { - return [ - ...this.#messages.splice(0).map(({ text }) => text), - error instanceof Error ? error.message : String(error) - ].join('\n'); + const lines: string[] = []; + let hasErrorLine: boolean = false; + for (const { text, severity } of this.#messages.splice(0)) { + const line: string = text.replace(/\r?\n$/, ''); + if (this.#isHidden(severity) || !line.trim()) continue; + hasErrorLine ||= severity === TerminalProviderSeverity.error; + lines.push(line); + } + // An AlreadyReportedError only says "An error occurred."; the error lines written before it are the report. + if (!(hasErrorLine && error instanceof Error && error instanceof AlreadyReportedError)) { + lines.push(error instanceof Error ? error.message : String(error)); + } + return lines.join('\n'); } public get hasBufferedMessages(): boolean { @@ -89,11 +100,15 @@ export class EngineTerminalProvider implements ITerminalProvider { ); } - #emit(text: string, severity: TerminalProviderSeverity): void { - if ( + #isHidden(severity: TerminalProviderSeverity): boolean { + return ( !this.#graph?.debugMode && (severity === TerminalProviderSeverity.verbose || severity === TerminalProviderSeverity.debug) - ) { + ); + } + + #emit(text: string, severity: TerminalProviderSeverity): void { + if (this.#isHidden(severity)) { return; } this.#graph?.eventSink?.onActivity?.(text, getEngineActivityOptions(severity)); diff --git a/libraries/rush-daemon/src/PhasedRequestEventSink.ts b/libraries/rush-daemon/src/PhasedRequestEventSink.ts index 4a849617c8..e726b07ccb 100644 --- a/libraries/rush-daemon/src/PhasedRequestEventSink.ts +++ b/libraries/rush-daemon/src/PhasedRequestEventSink.ts @@ -14,7 +14,8 @@ import type { DaemonEventType, IDaemonActivityPayload, IDaemonEventEnvelope, - IDaemonEventScope + IDaemonEventScope, + IDaemonOperationStatusChangedPayload } from '@rushstack/rush-daemon-protocol'; import { TerminalChunkKind } from '@rushstack/terminal'; import type { ITerminalChunk } from '@rushstack/terminal'; @@ -186,11 +187,18 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { executionResult: result, status: result.status }); - this.#emitEvent('operationStatusChanged', { + // Summarizing clients (agent output) point at the full log of the operations that explain a failure. + const logFilePath: string | undefined = + result.status === OperationStatus.Failure || result.status === OperationStatus.SuccessWithWarning + ? result.logFilePaths?.text + : undefined; + const payload: IDaemonOperationStatusChangedPayload = { operationId, previousStatus, - status: result.status - }); + status: result.status, + ...(logFilePath ? { logFilePath } : {}) + }; + this.#emitEvent('operationStatusChanged', payload); } public onOperationHeader(operationId: string): void { diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index 40e49c264e..c5bf80e37e 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -390,17 +390,37 @@ export class RequestAdmissionController { } } - /** Waits, within the same admission budget, until a restart would not preempt another request. */ + /** + * Waits until a restart would not preempt another request. The client is told how many requests it waits for, + * as a queue position, since a restart waits as long as their builds. + * + * @remarks + * Like the per-graph execution gate, waiting for the requests that this process was already serving when the wait + * began is progress rather than contention. A client-default timeout therefore does not apply while one of them is + * still being served and no rushx script is, and that time does not count against it. The default still limits + * the wait while a rushx script is served, since a script may not exit until it is stopped, and waiting for + * requests that arrived later, which could otherwise keep the request waiting for as long as they keep arriving. + * An explicit `noWait` or `waitTimeoutMs` applies to the whole wait, using the same absolute deadline as workspace + * admission. + */ public async waitForRestartDrainAsync( arbiter: WorkspaceRestartArbiter, ticket: IWorkspaceRestartTicket ): Promise { - // The arbiter reports its own admission errors, so this does not depend on the scheduler error mapping. - await arbiter.waitForDrainAsync(ticket, { - abortSignal: this.#abortController.signal, - noWait: this.#admission?.noWait, - waitTimeoutMs: this.#getRemainingWaitTimeoutMs() - }); + const writer: QueuePositionWriter | undefined = this.#writer; + try { + // The arbiter reports its own admission errors, so this does not depend on the scheduler error mapping. + const waivedMs: number = await arbiter.waitForDrainAsync(ticket, { + abortSignal: this.#abortController.signal, + noWait: this.#admission?.noWait, + waitTimeoutMs: this.#getRemainingWaitTimeoutMs(), + waivesTimeoutForServedWork: this.#admission?.waitTimeoutIsDefault === true, + onServingCountChanged: writer ? (servingCount: number) => writer.enqueue(servingCount) : undefined + }); + if (this.#deadlineMs !== undefined) this.#deadlineMs += waivedMs; + } finally { + await writer?.flushAsync(); + } } public dispose(): void { diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 9bc35cba8e..030ec677f2 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -216,8 +216,11 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { client, requestId: envelope.requestId }); - // Long-lived observers are cancelled by a transition, so they never delay a restart. - const ticket: IWorkspaceRestartTicket | undefined = observer ? undefined : this.#restartArbiter.enter(); + // Long-lived observers are cancelled by a transition, so they never delay a restart. A rushx script does delay + // one until it exits, so a client-default timeout still limits waiting for it. + const ticket: IWorkspaceRestartTicket | undefined = observer + ? undefined + : this.#restartArbiter.enter({ runsScript: isRushxInvocation(envelope) }); let generation: IPreparedGeneration | undefined; try { for (let attempt: number = 0; ; attempt++) { @@ -548,6 +551,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { workspaceLease.release(); throw new PhasedCommandEngineBusyError(); } + let selectionRejection: DaemonRequestDispatchError | undefined; try { const before: IWorkspaceInputFingerprint = await this.#captureAsync(session, envelope); let expectedFingerprint: IWorkspaceInputFingerprint = before; @@ -590,11 +594,18 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { const replacementSession: IWorkspaceSession = await this.#options.provider.reloadAsync(); validationContext.session = replacementSession; session = replacementSession; - await resolver.resolveRequestAsync({ - envelope, - workspaceSession: session, - abortSignal: client.abortSignal - }); + try { + await resolver.resolveRequestAsync({ + envelope, + workspaceSession: session, + abortSignal: client.abortSignal + }); + } catch (error) { + // An invalid selection (such as an unknown project) fails after the new graph was bound. Keep that + // generation, so that the next request does not load the whole workspace again. + if (!isSelectionRejection(error, session)) throw error; + selectionRejection = error; + } const after: IWorkspaceInputFingerprint = await this.#captureAsync(session, envelope); if (classifyWorkspaceInputChange(before, after) !== WorkspaceInputChangeTier.Reuse) { this.#forceReload = true; @@ -621,6 +632,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { nativeLock.release(); workspaceLease.release(); } + if (selectionRejection) throw selectionRejection; this.#gate.downgradeExclusiveLease(lease, RequestExclusivityClass.SharedBuild); return { session, @@ -883,6 +895,18 @@ function getResolverLifecycle(resolver: IDaemonRequestResolver): IWorkspaceResol return resolver.workspaceLifecycle; } +/** A request rejected as invalid after the resolver bound a graph to the session: its selection failed. */ +function isSelectionRejection( + error: unknown, + session: IWorkspaceSession +): error is DaemonRequestDispatchError { + return ( + error instanceof DaemonRequestDispatchError && + error.code === 'invalidRequest' && + session.operationGraph !== undefined + ); +} + function isGraphWatch(envelope: IDaemonRequestEnvelope): boolean { return isGraphRequest(envelope) && envelope.argv[1] === 'graph' && envelope.argv[2] === 'watch'; } diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts index b1228eaca7..ee76274a88 100644 --- a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -4,15 +4,34 @@ import { RequestSchedulerError, RequestSchedulerErrorCode } from './RequestScheduler'; const MAX_TIMER_DELAY_MS: number = 0x7fffffff; +const SCRIPT_TIMEOUT_CLAUSE: string = ', including a rushx script that may not exit until it is stopped'; +// Waiting longer may not help behind a script, such as a dev server, that runs until it is stopped. +const SCRIPT_TIMEOUT_REMEDY: string = 'Stop the script, or use --wait-timeout to wait longer.'; +const TIMEOUT_REMEDY: string = 'Use --wait-timeout to wait longer.'; + +/** Names the waived time, since a drain that waived time times out that much later than the wait timeout. */ +function formatWaivedTime(waivedMs: number): string { + const seconds: number = Math.round(waivedMs / 100) / 10; + return seconds === 0 + ? '' + : `; ${seconds}s spent waiting for requests that were already running did not count`; +} /** One dispatched request tracked by a {@link WorkspaceRestartArbiter}. */ export interface IWorkspaceRestartTicket { readonly waitingForDrain: boolean; } +/** Options for {@link WorkspaceRestartArbiter.enter}. */ +export interface IWorkspaceRestartTicketOptions { + /** The request runs a rushx script, which can run until it is stopped (for example, a dev server). */ + readonly runsScript?: boolean; +} + interface IMutableTicket { waitingForDrain: boolean; left: boolean; + readonly runsScript: boolean; } /** Options for {@link WorkspaceRestartArbiter.waitForDrainAsync}, supplied by request admission. */ @@ -20,6 +39,18 @@ export interface IWorkspaceRestartDrainOptions { readonly abortSignal: AbortSignal; readonly noWait: boolean | undefined; readonly waitTimeoutMs: number | undefined; + /** + * Do not spend `waitTimeoutMs` while a request that was already being served when the wait began is still being + * served and no rushx script is. The timeout then limits waiting while a rushx script is served, since a script + * may not exit until it is stopped, and waiting for requests that arrived later, which could otherwise keep the + * request waiting for as long as they keep arriving. + */ + readonly waivesTimeoutForServedWork?: boolean; + /** + * Called while the request waits with the number of other requests that it waits for, when the wait begins and + * whenever that number changes, so that the client can report the wait as a queue position. + */ + readonly onServingCountChanged?: (servingCount: number) => void; } /** @@ -29,16 +60,21 @@ export interface IWorkspaceRestartDrainOptions { */ export class WorkspaceRestartArbiter { readonly #listeners: Set<() => void> = new Set(); - #servingCount: number = 0; + readonly #countListeners: Set<(servingCount: number) => void> = new Set(); + readonly #serving: Set = new Set(); /** The number of tracked requests that are not waiting for a restart. */ public get servingCount(): number { - return this.#servingCount; + return this.#serving.size; } - public enter(): IWorkspaceRestartTicket { - this.#servingCount++; - const ticket: IMutableTicket = { waitingForDrain: false, left: false }; + public enter(options?: IWorkspaceRestartTicketOptions): IWorkspaceRestartTicket { + const ticket: IMutableTicket = { + waitingForDrain: false, + left: false, + runsScript: options?.runsScript === true + }; + this.#serve(ticket); return ticket; } @@ -46,47 +82,100 @@ export class WorkspaceRestartArbiter { const state: IMutableTicket = ticket as IMutableTicket; if (state.left) return; state.left = true; - if (!state.waitingForDrain) this.#decrement(); + if (!state.waitingForDrain) this.#stopServing(state); } /** * Waits until no other tracked request is still being served by this process, then counts the ticket - * as served again so concurrent restart candidates proceed one at a time. + * as served again so concurrent restart candidates proceed one at a time. Returns how many milliseconds of the + * wait did not spend `waitTimeoutMs` (see {@link IWorkspaceRestartDrainOptions.waivesTimeoutForServedWork}). */ public async waitForDrainAsync( ticket: IWorkspaceRestartTicket, options: IWorkspaceRestartDrainOptions - ): Promise { + ): Promise { const state: IMutableTicket = ticket as IMutableTicket; if (state.left || state.waitingForDrain) throw new Error('The restart ticket is not being served.'); state.waitingForDrain = true; - this.#decrement(); + this.#stopServing(state); + const waivedFor: IMutableTicket[] = options.waivesTimeoutForServedWork ? Array.from(this.#serving) : []; + let remainingMs: number | undefined = options.waitTimeoutMs; + let waivedMs: number = 0; + let reported: number | undefined; + const report = (servingCount: number): void => { + if (servingCount > 0 && servingCount !== reported) { + reported = servingCount; + options.onServingCountChanged?.(servingCount); + } + }; try { - const deadline: number | undefined = - options.waitTimeoutMs === undefined ? undefined : Date.now() + options.waitTimeoutMs; - while (this.#servingCount > 0) { + while (this.#serving.size > 0) { if (options.noWait) { throw new RequestSchedulerError( RequestSchedulerErrorCode.NoWait, 'Another environment is still being served; the request did not wait for a restart.' ); } - await this.#waitForChangeAsync(options.abortSignal, deadline); + report(this.#serving.size); + const waived: boolean = + !this.#isServingScript() && waivedFor.some((served: IMutableTicket) => this.#serving.has(served)); + const startedAt: number = Date.now(); + this.#countListeners.add(report); + try { + await this.#waitForChangeAsync( + options.abortSignal, + waived || remainingMs === undefined ? undefined : startedAt + remainingMs, + waivedMs + ); + } finally { + this.#countListeners.delete(report); + const elapsedMs: number = Date.now() - startedAt; + if (waived) waivedMs += elapsedMs; + else if (remainingMs !== undefined) remainingMs -= elapsedMs; + } } + return waivedMs; } finally { state.waitingForDrain = false; - this.#servingCount++; + this.#serve(state); } } - #decrement(): void { - this.#servingCount--; - if (this.#servingCount === 0) { - for (const listener of Array.from(this.#listeners)) listener(); - } + #serve(ticket: IMutableTicket): void { + this.#serving.add(ticket); + this.#notifyChange(); + } + + #stopServing(ticket: IMutableTicket): void { + this.#serving.delete(ticket); + this.#notifyChange(); + } + + /** Reports the new count, and wakes every waiting candidate to re-check what it waits for. */ + #notifyChange(): void { + for (const listener of Array.from(this.#countListeners)) listener(this.#serving.size); + for (const listener of Array.from(this.#listeners)) listener(); } - #waitForChangeAsync(abortSignal: AbortSignal, deadline: number | undefined): Promise { + #isServingScript(): boolean { + return Array.from(this.#serving).some((served: IMutableTicket) => served.runsScript); + } + + #createTimeoutError(waivedMs: number): RequestSchedulerError { + const script: boolean = this.#isServingScript(); + return new RequestSchedulerError( + RequestSchedulerErrorCode.WaitTimeout, + 'The request was not admitted before the daemon could restart for its environment, which waits for the ' + + `requests that the daemon is serving to finish${script ? SCRIPT_TIMEOUT_CLAUSE : ''}` + + `${formatWaivedTime(waivedMs)}. ${script ? SCRIPT_TIMEOUT_REMEDY : TIMEOUT_REMEDY}` + ); + } + + #waitForChangeAsync( + abortSignal: AbortSignal, + deadline: number | undefined, + waivedMs: number + ): Promise { return new Promise((resolve, reject) => { let timer: ReturnType | undefined; const unsubscribe: AbortController = new AbortController(); @@ -99,7 +188,10 @@ export class WorkspaceRestartArbiter { }; const settleAborted = (): void => settle( - new RequestSchedulerError(RequestSchedulerErrorCode.Aborted, 'The request was aborted before execution.') + new RequestSchedulerError( + RequestSchedulerErrorCode.Aborted, + 'The request was aborted before execution.' + ) ); if (abortSignal.aborted) { settleAborted(); @@ -109,14 +201,7 @@ export class WorkspaceRestartArbiter { abortSignal.addEventListener('abort', settleAborted, { once: true, signal: unsubscribe.signal }); if (deadline !== undefined) { timer = setTimeout( - () => - settle( - new RequestSchedulerError( - RequestSchedulerErrorCode.WaitTimeout, - 'The request was not admitted before the daemon could restart for its environment. ' + - 'Use --wait-timeout to wait longer.' - ) - ), + () => settle(this.#createTimeoutError(waivedMs)), Math.min(MAX_TIMER_DELAY_MS, Math.max(0, deadline - Date.now())) ); } diff --git a/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts b/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts index 2c6fcbfdf7..549b8dd99c 100644 --- a/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts +++ b/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import { AlreadyReportedError } from '@rushstack/node-core-library'; import { TerminalProviderSeverity } from '@rushstack/terminal'; import { EngineTerminalProvider } from '../EngineTerminalProvider'; @@ -41,6 +42,39 @@ describe(EngineTerminalProvider.name, () => { expect(terminal.hasBufferedMessages).toBe(false); }); + it('describes a failure without the verbose and debug messages that loading a workspace writes', () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + terminal.write('Incremental strategy: cache restoration\n', TerminalProviderSeverity.verbose); + for (let index: number = 0; index < 1000; index++) { + terminal.write( + `Configuration file "p${index}/config/rush-project.json" not found.\n`, + TerminalProviderSeverity.debug + ); + } + terminal.write('\n', TerminalProviderSeverity.log); + terminal.write('Project "a" has no "build" script.\n', TerminalProviderSeverity.warning); + expect(terminal.describeError(new Error('selection failed'))).toBe( + 'Project "a" has no "build" script.\nselection failed' + ); + }); + + it('describes an already reported error by the error lines written before it', () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + terminal.write('Incremental strategy: cache restoration\n', TerminalProviderSeverity.verbose); + terminal.write( + 'The project name "@x/nope" passed to "--to" does not exist in rush.json.\n', + TerminalProviderSeverity.error + ); + expect(terminal.describeError(new AlreadyReportedError())).toBe( + 'The project name "@x/nope" passed to "--to" does not exist in rush.json.' + ); + + terminal.write('No error line was written.\n', TerminalProviderSeverity.warning); + expect(terminal.describeError(new AlreadyReportedError())).toBe( + 'No error line was written.\nAn error occurred.' + ); + }); + it('drops diagnostics when the engine must be recreated', async () => { const terminal: EngineTerminalProvider = new EngineTerminalProvider(); const recreate: WorkspaceEngineRecreationRequiredError = new WorkspaceEngineRecreationRequiredError(); diff --git a/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts b/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts index bb2bcb191d..78c41e666a 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts @@ -93,4 +93,44 @@ it('does not report settlement when an active operation was aborted', () => { sink.onOperationCompleted(createRecord(SECOND_ACTIVE_OPERATION, OperationStatus.Success)); expect(onSettled).not.toHaveBeenCalled(); -}); \ No newline at end of file +}); +it('points at the full log of failed operations and operations with warnings, and only those', async () => { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + const sink: PhasedRequestEventSink = createSink(client); + const logFilePaths = { text: '/repo/project-a/rush-logs/project-a._phase_test.log' }; + for (const status of [ + OperationStatus.Executing, + OperationStatus.Failure, + OperationStatus.SuccessWithWarning, + OperationStatus.Success + ]) { + const record: IOperationExecutionResult = { + ...createRecord(ACTIVE_OPERATION, status), + logFilePaths + } as unknown as IOperationExecutionResult; + sink.onOperationStatusChanged(record, OperationStatus.Ready); + } + + await sink.flushAsync(); + + const payloads: unknown[] = client.writes + .map(({ event }) => event) + .filter((event) => event?.type === 'operationStatusChanged') + .map((event) => event?.payload); + expect(payloads).toEqual([ + { operationId: ACTIVE_OPERATION, previousStatus: 'READY', status: 'EXECUTING' }, + { + operationId: ACTIVE_OPERATION, + previousStatus: 'READY', + status: 'FAILURE', + logFilePath: logFilePaths.text + }, + { + operationId: ACTIVE_OPERATION, + previousStatus: 'READY', + status: 'SUCCESS WITH WARNINGS', + logFilePath: logFilePaths.text + }, + { operationId: ACTIVE_OPERATION, previousStatus: 'READY', status: 'SUCCESS' } + ]); +}); diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 0667da53eb..b4fb7b3e08 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -460,6 +460,33 @@ describe('native production daemon engine', () => { expect(events.at(-1)).toBe('disposed:2'); }); + it('rejects an unknown project with only the error line and keeps the graph it loaded for later requests', async () => { + const fixture: IFixture = await createFixtureAsync(); + try { + const rejection: { kind: string; payload: { code: string; message: string } } = { + kind: 'requestRejected', + payload: { + code: 'invalidRequest', + message: 'The project name "nope" passed to "--to" does not exist in rush.json.' + } + }; + expect((await runAsync(fixture, 'cold', ['build', '--to', 'nope'])).terminal).toMatchObject(rejection); + const session: WorkspaceSession = fixture.session; + const graph: IOperationGraph | undefined = session.operationGraph; + expect(graph).toBeDefined(); + expect((await runAsync(fixture, 'again', ['build', '--to', 'nope'])).terminal).toMatchObject(rejection); + expect((await runAsync(fixture, 'valid', ['build', '--only', 'a'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0 } + }); + expect(fixture.session).toBe(session); + expect(fixture.session.operationGraph).toBe(graph); + expect(runs(fixture)).toEqual(['a:one:']); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('keeps tier0 session and graph identity for unchanged content, including metadata touches', async () => { const fixture: IFixture = await createFixtureAsync(); try { diff --git a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts new file mode 100644 index 0000000000..a91e129d94 --- /dev/null +++ b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts @@ -0,0 +1,131 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; + +import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestScheduler'; +import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; +import { + WorkspaceRestartArbiter, + type IWorkspaceRestartTicket, + type IWorkspaceRestartTicketOptions +} from '../WorkspaceRestartArbiter'; + +interface IDrainTest { + readonly admission: RequestAdmissionController; + readonly arbiter: WorkspaceRestartArbiter; + readonly serving: IWorkspaceRestartTicket; + readonly ticket: IWorkspaceRestartTicket; +} + +/** A restart candidate with the given admission options, and one other request that the daemon is serving. */ +function createDrainTest( + options: IDaemonRequestAdmissionOptions, + servingOptions?: IWorkspaceRestartTicketOptions +): IDrainTest { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(servingOptions); + const ticket: IWorkspaceRestartTicket = arbiter.enter(); + const admission: RequestAdmissionController = new RequestAdmissionController({ + admission: options, + client: { abortSignal: new AbortController().signal }, + requestId: 'restart-candidate' + }); + return { admission, arbiter, serving, ticket }; +} + +async function isSettledAsync(promise: Promise): Promise { + let settled: boolean = false; + void promise.then( + () => (settled = true), + () => (settled = true) + ); + await new Promise((resolve) => setImmediate(resolve)); + return settled; +} + +describe('RequestAdmissionController.waitForRestartDrainAsync', () => { + it('waits past a client-default timeout, and the wait does not count against it', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest({ + waitTimeoutMs: 50, + waitTimeoutIsDefault: true + }); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); + await delayAsync(250); + expect(await isSettledAsync(draining)).toBe(false); + arbiter.leave(serving); + await draining; + // The later admission steps keep what was left of the default when the drain began. + expect(admission.remainingAdmission).toMatchObject({ waitTimeoutIsDefault: true }); + expect(admission.remainingAdmission?.waitTimeoutMs).toBeGreaterThan(0); + arbiter.leave(ticket); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); + + it('still applies a client-default timeout while it waits for a rushx script, and says why', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest( + { waitTimeoutMs: 50, waitTimeoutIsDefault: true }, + { runsScript: true } + ); + const error: unknown = await admission + .waitForRestartDrainAsync(arbiter, ticket) + .catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + expect((error as Error).message).toContain( + 'including a rushx script that may not exit until it is stopped' + ); + arbiter.leave(ticket); + arbiter.leave(serving); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); + + it('applies a client-default timeout to requests that arrive during the drain, once earlier ones finish', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest({ + waitTimeoutMs: 50, + waitTimeoutIsDefault: true + }); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); + const late: IWorkspaceRestartTicket = arbiter.enter(); + await delayAsync(150); + expect(await isSettledAsync(draining)).toBe(false); + arbiter.leave(serving); + const error: unknown = await draining.catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + arbiter.leave(ticket); + arbiter.leave(late); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); + + it('applies an explicit timeout to the drain, and suggests only --wait-timeout', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest({ waitTimeoutMs: 50 }); + const error: unknown = await admission + .waitForRestartDrainAsync(arbiter, ticket) + .catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(RequestSchedulerError); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + // Setting the environment variable instead would itself restart the daemon. + expect((error as Error).message).toContain('--wait-timeout '); + expect((error as Error).message).not.toContain('RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS'); + arbiter.leave(ticket); + arbiter.leave(serving); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); + + it('applies no-wait to the drain', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest({ noWait: true }); + const error: unknown = await admission + .waitForRestartDrainAsync(arbiter, ticket) + .catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.NoWait); + arbiter.leave(ticket); + arbiter.leave(serving); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts index 8ee1001739..10a86e2314 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts @@ -1,8 +1,14 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import { setTimeout as delayAsync } from 'node:timers/promises'; + import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestScheduler'; -import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from '../WorkspaceRestartArbiter'; +import { + WorkspaceRestartArbiter, + type IWorkspaceRestartDrainOptions, + type IWorkspaceRestartTicket +} from '../WorkspaceRestartArbiter'; const WAIT: { abortSignal: AbortSignal; noWait: undefined; waitTimeoutMs: undefined } = { abortSignal: new AbortController().signal, @@ -35,8 +41,8 @@ describe(WorkspaceRestartArbiter.name, () => { const serving: IWorkspaceRestartTicket = arbiter.enter(); const first: IWorkspaceRestartTicket = arbiter.enter(); const second: IWorkspaceRestartTicket = arbiter.enter(); - const firstWait: Promise = arbiter.waitForDrainAsync(first, WAIT); - const secondWait: Promise = arbiter.waitForDrainAsync(second, WAIT); + const firstWait: Promise = arbiter.waitForDrainAsync(first, WAIT); + const secondWait: Promise = arbiter.waitForDrainAsync(second, WAIT); const late: IWorkspaceRestartTicket = arbiter.enter(); arbiter.leave(serving); expect(await isSettledAsync(firstWait)).toBe(false); @@ -58,7 +64,7 @@ describe(WorkspaceRestartArbiter.name, () => { const serving: IWorkspaceRestartTicket = arbiter.enter(); const ticket: IWorkspaceRestartTicket = arbiter.enter(); const abort: AbortController = new AbortController(); - const waiting: Promise = arbiter.waitForDrainAsync(ticket, { + const waiting: Promise = arbiter.waitForDrainAsync(ticket, { abortSignal: abort.signal, noWait: mode === 'no-wait' ? true : undefined, waitTimeoutMs: mode === 'timeout' ? 10 : undefined @@ -72,4 +78,186 @@ describe(WorkspaceRestartArbiter.name, () => { arbiter.leave(serving); expect(arbiter.servingCount).toBe(0); }); -}); \ No newline at end of file + + it('reports how many requests a restart candidate waits for, whenever that number changes', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const first: IWorkspaceRestartTicket = arbiter.enter(); + const second: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const counts: number[] = []; + const waiting: Promise = arbiter.waitForDrainAsync(candidate, { + ...WAIT, + onServingCountChanged: (count: number) => counts.push(count) + }); + expect(counts).toEqual([2]); + const late: IWorkspaceRestartTicket = arbiter.enter(); + arbiter.leave(first); + arbiter.leave(second); + expect(counts).toEqual([2, 3, 2, 1]); + // Another restart candidate stops counting once it waits too; it then waits for the first candidate. + const other: IWorkspaceRestartTicket = arbiter.enter(); + const otherCounts: number[] = []; + const otherWaiting: Promise = arbiter.waitForDrainAsync(other, { + ...WAIT, + onServingCountChanged: (count: number) => otherCounts.push(count) + }); + arbiter.leave(late); + await waiting; + expect(counts).toEqual([2, 3, 2, 1, 2, 1]); + expect(await isSettledAsync(otherWaiting)).toBe(false); + expect(otherCounts).toEqual([1]); + arbiter.leave(candidate); + await otherWaiting; + arbiter.leave(other); + expect(counts).toHaveLength(6); + expect(arbiter.servingCount).toBe(0); + }); + + it('reports no count for a candidate that does not wait', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const ticket: IWorkspaceRestartTicket = arbiter.enter(); + const counts: number[] = []; + await arbiter.waitForDrainAsync(ticket, { + ...WAIT, + onServingCountChanged: (count: number) => counts.push(count) + }); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const noWait: IWorkspaceRestartTicket = arbiter.enter(); + const error: unknown = await arbiter + .waitForDrainAsync(noWait, { + ...WAIT, + noWait: true, + onServingCountChanged: (count: number) => counts.push(count) + }) + .catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.NoWait); + expect(counts).toEqual([]); + for (const served of [ticket, serving, noWait]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + describe('with waivesTimeoutForServedWork', () => { + const WAIVED: IWorkspaceRestartDrainOptions = { + ...WAIT, + waitTimeoutMs: 50, + waivesTimeoutForServedWork: true + }; + + function expectWaitTimeout(error: unknown): RequestSchedulerError { + expect(error).toBeInstanceOf(RequestSchedulerError); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + return error as RequestSchedulerError; + } + + it('does not spend the timeout on requests served when the wait began, and returns that time', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); + await delayAsync(150); + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(earlier); + expect(await waiting).toBeGreaterThanOrEqual(140); + arbiter.leave(candidate); + expect(arbiter.servingCount).toBe(0); + }); + + it('spends the timeout on requests that arrived after the wait began, once the earlier ones finish', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); + const late: IWorkspaceRestartTicket = arbiter.enter(); + await delayAsync(150); + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(earlier); + const error: RequestSchedulerError = expectWaitTimeout( + await waiting.catch((caught: unknown) => caught) + ); + expect(error.message).not.toContain('rushx'); + expect(error.message).toMatch( + /to finish; \d+(\.\d)?s spent waiting for requests that were already running did not count\. Use --wait-timeout to wait longer\.$/ + ); + for (const served of [candidate, late]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('spends the timeout on a rushx script, and says that the script may not exit', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const error: RequestSchedulerError = expectWaitTimeout( + await arbiter.waitForDrainAsync(candidate, WAIVED).catch((caught: unknown) => caught) + ); + expect(error.message).toContain( + ', including a rushx script that may not exit until it is stopped. Stop the script, or use --wait-timeout' + ); + for (const served of [candidate, script]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('spends the timeout while a rushx script is served, even next to an earlier build', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const outcome: Promise = arbiter + .waitForDrainAsync(candidate, WAIVED) + .catch((caught: unknown) => caught); + await delayAsync(150); + expect(await isSettledAsync(outcome)).toBe(true); + expect(expectWaitTimeout(await outcome).message).toContain('including a rushx script'); + for (const served of [candidate, earlier, script]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('stops spending the timeout when the rushx script exits while an earlier build is served', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); + arbiter.leave(script); + await delayAsync(150); + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(earlier); + expect(await waiting).toBeGreaterThanOrEqual(140); + arbiter.leave(candidate); + expect(arbiter.servingCount).toBe(0); + }); + + it('spends the timeout once a rushx script arrives during the wait', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const outcome: Promise = arbiter + .waitForDrainAsync(candidate, WAIVED) + .catch((caught: unknown) => caught); + await delayAsync(100); + expect(await isSettledAsync(outcome)).toBe(false); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + await delayAsync(150); + expect(await isSettledAsync(outcome)).toBe(true); + expect(expectWaitTimeout(await outcome).message).toMatch( + /, including a rushx script that may not exit until it is stopped; \d+(\.\d)?s spent waiting for requests that were already running did not count\. Stop the script, or use --wait-timeout/ + ); + for (const served of [candidate, earlier, script]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('returns 0 when the timeout is not waived', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, { + ...WAIT, + waitTimeoutMs: 5_000 + }); + await delayAsync(20); + arbiter.leave(earlier); + expect(await waiting).toBe(0); + arbiter.leave(candidate); + expect(arbiter.servingCount).toBe(0); + }); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts index 557c54ddc5..ade38105f6 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbitration.test.ts @@ -6,6 +6,8 @@ import * as path from 'node:path'; import { setTimeout as delayAsync } from 'node:timers/promises'; import { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; +import { DaemonFrameType, decodeDaemonControlMessage } from '@rushstack/rush-daemon-protocol'; +import type { DaemonControlMessage, IDaemonFrame } from '@rushstack/rush-daemon-protocol'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; @@ -15,6 +17,15 @@ import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; jest.setTimeout(60_000); +function getQueuePositions(exchange: ITerminalExchange): number[] { + return exchange.frames + .filter((frame: IDaemonFrame) => frame.kind === DaemonFrameType.controlJson) + .map((frame: IDaemonFrame) => decodeDaemonControlMessage(frame.payload)) + .flatMap((message: DaemonControlMessage) => + message.kind === 'queuePosition' ? [message.payload.position] : [] + ); +} + it('queues a mismatched-environment restart until matching queued and in-flight requests drain', async () => { const fixture = await DaemonGraphTestFixture.createAsync((created) => { setDaemonPolicy(created, {}); @@ -64,6 +75,10 @@ it('queues a mismatched-environment restart until matching queued and in-flight payload: { exitCode: 1, retryAfterRestart: true } }); expect(order[order.length - 1]).toBe('mismatched'); + // The client is told how many requests the restart waits for: 2 while both matching requests are open, then 1. + const positions: number[] = getQueuePositions(restart); + expect(positions).toContain(2); + expect(positions[positions.length - 1]).toBe(1); const restarted = await fixture.host.restartCompleted; expect(restarted?.pid).not.toBe(before.pid); From 96a7e496dd0e4da49d0e9ed7bd56a882cf6f2375 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 21:00:00 +0000 Subject: [PATCH 037/265] [rush-cli-client] Report failed operations as they fail in agent output Swarm integration step 22; original commit 6bc6dd8f83 (merge of swarm/r04-t108-a81 at 14faa4e5be). Scope: task 108. Brings task 108 items 1 and 3 onto task 83: agent output prints a failed operation's first diagnostics as soon as that operation finishes, no `running` line before 10 s for quick requests, and each TypeScript error once (t07 board 1930: L4 17 of 41 on 4d1a045fca against 24 of 41 on s11; 38 of 41 with items 1 and 3 on their old base, board 1692). Gate: ch01 GATE OK board 1980 (tree 9079647277) Commits folded into this step (1): - 14faa4e5be Report failed operations as they fail in agent output (task 108 items 1 and 3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 42 ++-- .../src/AgentOperationTracker.ts | 17 +- .../src/AgentProgressRenderer.ts | 196 ++++++++++------ .../src/OperationOutputExcerpt.ts | 63 ++++- .../src/test/AgentProgressRenderer.test.ts | 216 +++++++++++++----- .../src/test/OperationOutputExcerpt.test.ts | 49 +++- ...-t108-early-failures_2026-09-28-18-14.json | 10 + 7 files changed, 420 insertions(+), 173 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t108-early-failures_2026-09-28-18-14.json diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 790c03a04d..3a2bd0f840 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -114,13 +114,17 @@ agent mode writes nothing ahead of it. Otherwise, selection precedence is: 3. Otherwise `legacy`: the unchanged collated operation stream. Agent mode is plain text for humans and agents, not the AI reporter's JSON record format; -use `--reporter=ai` for machine-parsed records. It writes a first status line before -`@microsoft/rush-lib` is loaded, then at most three live rows on a TTY. On a pipe it writes -at most three progress lines in total, however many operations run and however long they -take: a wait for a daemon that is still starting (`rushd is still starting; waiting for it (up to 15s more)`), -the start, the first wait for admission (`queued behind another request (position N)`), -the start of execution, and the first failure, in that order until three were written. It -always ends with one summary line, for example +use `--reporter=ai` for machine-parsed records. On a TTY it paints at most three live rows, the +first before `@microsoft/rush-lib` is loaded. On a pipe it writes one progress line when the +daemon has the request (`rush build · 0.1s · sent to rushd; preparing the workspace graph +(status at least every 25s)`), however many operations run. A longer request also gets status +lines, so that it does not look hung: one whenever nothing was written for 25 s, with the counts +and the running operations, and one when connecting to the daemon takes more than 10 s. A +wait for a daemon that is still starting gets a line of its own +(`rushd is still starting; waiting for it (up to 15s more)`). A +request that waited for admission says so at the end of its summary line +(`· queued behind another request (position 1 at 0.2s)`). It always ends with one summary +line, for example `rush build: SUCCESS 772/772 operations (12 success, 760 from cache) in 3.1s`, or `up to date (no operations needed)`, or, when the selection parameters matched no projects, `rush build: SUCCESS 0 operations in 0.5s · the selection parameters did not match any projects`. @@ -133,16 +137,20 @@ daemon did not admit the request in time, or at once with `--no-wait`, the reaso line starts with `daemon admission failed (wait-timeout)` or `daemon admission failed (no-wait)`, as in legacy output, followed by the daemon's reason in full. Warnings and errors that Rush or a Rush plugin writes outside any operation (for example a plugin that continues without the cloud -build cache) precede the summary line, and any failure report, at most three lines of them. - -When the request fails, a report comes before the summary line. It covers up to three failed -operations, or, if none failed, the operations whose warnings failed the request. Each one gets a -`failed: · full log: ` line (`warnings: …` for warnings) and a short excerpt -of its output: error lines with the line that follows them first, then the last lines. Stack -frames, `Require stack:` lists and progress noise are left out. The summary line names up to five -failed (or warning) operations. Every operation's full output is in its project's `rush-logs/` -folder, whether or not it was printed. When a request falls back to in-process Rush, agent mode -stops and native output follows. +build cache) are written at the end, at most three lines of them, before the summary line and any +operations reported with it. + +A failed operation is reported as soon as it fails, while the rest of the request runs on: a +`failed: · full log: ` line and a short excerpt of its output, error lines +with the line that follows them first, then the last lines. Stack frames, `Require stack:` lists +and progress noise are left out, and so are a message that a tool repeats in its summary, an +error count that the shown errors account for, and, when the first error shown names a source +location, the lines before it. Up to three operations are reported. Two kinds are reported just +before the summary line instead: a failed operation that wrote no output, with the error from the +daemon's result, and, when no operation failed, the operations whose warnings failed the request +(`warnings: …`). The summary line names up to five failed (or warning) operations. Every +operation's full output is in its project's `rush-logs/` folder, whether or not it was printed. +When a request falls back to in-process Rush, agent mode stops and native output follows. Positively identified built-in `install` and `update` follow the same opt-in routing precedence as workspace builds and require protocol **0.10** diff --git a/apps/rush-cli-client/src/AgentOperationTracker.ts b/apps/rush-cli-client/src/AgentOperationTracker.ts index e2381da3b4..93fae6c77c 100644 --- a/apps/rush-cli-client/src/AgentOperationTracker.ts +++ b/apps/rush-cli-client/src/AgentOperationTracker.ts @@ -132,19 +132,18 @@ export class AgentOperationTracker { this.#headerTotal = Math.max(this.#headerTotal, total); } - /** Applies a status change. Returns true when it is the request's first failure. */ - public updateStatus(update: IAgentOperationStatusUpdate): boolean { + /** Applies a status change. */ + public updateStatus(update: IAgentOperationStatusUpdate): void { const { operationId, status } = update; if (this.#silent.has(operationId)) { if (status !== FAILURE) { - return false; + return; } // Failed operations are reported even if silent, as in the native summary. this.register(operationId, false); } // An operation that was never registered still counts, so `done` never exceeds `total`. this.#registered.add(operationId); - const firstFailure: boolean = status === FAILURE && this.#failed.size === 0; const previous: string | undefined = this.#statuses.get(operationId); this.#statuses.set(operationId, status); if (previous !== undefined && TERMINAL_STATUSES.has(previous)) { @@ -161,7 +160,6 @@ export class AgentOperationTracker { if (TERMINAL_STATUSES.has(status)) { this.#onTerminalStatus(update); } - return firstFailure; } /** @@ -216,12 +214,17 @@ export class AgentOperationTracker { */ public getProblemOperations(): ReadonlyArray { const operationIds: ReadonlyArray = this.#failed.size ? this.failed : this.warned; - return operationIds.map((operationId) => ({ + return operationIds.map((operationId) => this.getProblemOperation(operationId)); + } + + /** An operation's log file, output excerpt and error, to report it. */ + public getProblemOperation(operationId: string): IAgentProblemOperation { + return { operationId, logFilePath: this.#logFilePaths.get(operationId), excerpt: this.#excerpts.get(operationId), errorMessage: this.#errorMessages.get(operationId) - })); + }; } #isProblemStatus(status: string): boolean { diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index 6195aac9f9..2f3dc95820 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -15,17 +15,20 @@ import { import { clipLine } from './OperationOutputExcerpt'; const SPINNER_FRAMES: readonly string[] = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']; -/** On a pipe, the most milestone lines written before the summary, however long the request runs. */ -const MAX_PIPE_PROGRESS_LINES: number = 3; -/** On a pipe, how long the first line waits for the connection, so that a fast connect costs one line, not two. */ -const PIPE_FIRST_LINE_DELAY_MS: number = 1000; +/** + * On a pipe, the connecting line is written only when the connection takes longer than this. Like a status line, + * it is never written in the first 10 s, so a connection that is fast, or a daemon that starts within that time, + * costs no line. + */ +const PIPE_CONNECTING_LINE_DELAY_MS: number = 10_000; /** * On a pipe, a status line is written whenever nothing was written for this long, so that a reader can tell a slow - * request from a hung one. Agent shells return partial output after 30 s. These lines are not milestones. + * request from a hung one. Agent shells return partial output after 30 s. A request that ends sooner writes none. */ const PIPE_STATUS_INTERVAL_MS: number = 25_000; const SENT_PHASE: string = 'sent to rushd; preparing the workspace graph'; const STARTING_PHASE: string = 'rushd is still starting; waiting for it'; +const FAILURE_STATUS: string = 'FAILURE'; const TTY_INTERVAL_MS: number = 100; /** The most failed (or warning) operations whose output excerpt is printed. */ const MAX_REPORTED_OPERATIONS: number = 3; @@ -52,9 +55,14 @@ const STATUS_LABELS: ReadonlyMap = new Map([ ['NO OP', 'up to date'] ]); -type PipeMilestone = 'starting' | 'sent' | 'queued' | 'running' | 'failure'; type Verdict = 'SUCCESS' | 'FAILURE' | 'CANCELLED'; +/** A queue position that the daemon reported, and when. */ +interface IQueuePosition { + readonly position: number; + readonly elapsed: string; +} + export interface IAgentProgressRendererOptions { readonly commandName: string; readonly isTTY: boolean; @@ -93,36 +101,40 @@ function getErrorDetail(lines: ReadonlyArray): string[] { } /** - * Compact progress for agents on the daemon path: an immediate first line, at most three live rows (TTY) or - * at most three progress lines in total (pipes), and a guaranteed one-line summary, even when no - * operation ran. On failure, the summary is preceded by each failed operation's log file and a short - * excerpt of its output. + * Compact progress for agents on the daemon path: at most three live rows (TTY) or one line when the request is + * sent (pipes), each failed operation's log file and a short excerpt of its output as soon as it fails, and a + * guaranteed one-line summary, even when no operation ran. * * @remarks - * On a pipe, a line is written at a milestone: when the client waits for a daemon that is still starting, when the - * request is sent to the daemon, the first time it waits for admission, the start of execution, and the first - * failure; once three lines were written, later milestones are left to the summary. A connection that takes - * longer than a second gets a line of its own first, unless a milestone came first. Whenever - * nothing was written for 25 s, a status line with the counts and the running operations follows, so a reader - * never sees more than 25 s of silence, and a request shorter than that costs no status lines at all. + * On a pipe, a request that takes less than 25 s writes the line that says it was sent, its failures and its + * summary line, and nothing else. Status lines keep a longer request from looking hung: whenever nothing was + * written for 25 s, a status line with the counts and the running operations follows, and a connection that + * takes longer than 10 s gets one. A wait for a daemon that is still starting also gets a line, once. Only the + * first three failed operations are reported. Whether warnings fail the request is only known at its end, so + * operations with warnings are reported before the summary line; so is a failed operation that wrote no output, + * whose error only the daemon's result carries. */ export class AgentProgressRenderer { readonly #options: IAgentProgressRendererOptions; readonly #now: () => number; readonly #startTimeMs: number; readonly #tracker: AgentOperationTracker = new AgentOperationTracker(); - readonly #milestones: Set = new Set(); readonly #notices: AgentNotices = new AgentNotices(); + /** The operations whose log file and excerpt were written, in that order. */ + readonly #reported: Set = new Set(); #lastActivity: string = ''; #phase: string = 'connecting to rushd (auto-starts if needed)'; #painted: number = 0; #frame: number = 0; - #pipeLines: number = 0; + #startingLineWritten: boolean = false; + #sentLineWritten: boolean = false; #timer: ReturnType | undefined; - #firstLineTimer: ReturnType | undefined; + #connectingTimer: ReturnType | undefined; #statusTimer: ReturnType | undefined; - /** The last queue position and when it was reported, until operations start. */ - #queued: { readonly position: number; readonly elapsed: string } | undefined; + /** The last queue position, until operations start. */ + #queued: IQueuePosition | undefined; + /** The first queue position, for the summary line. */ + #firstQueued: IQueuePosition | undefined; #stopped: boolean = false; /** The error message that the summary line contains in full, once written. */ #reportedErrorMessage: string | undefined; @@ -134,13 +146,16 @@ export class AgentProgressRenderer { } /** - * On a TTY, paints the first line and starts the spinner. On a pipe, writes the first line after a second - * unless another line came first, and starts the status lines. + * On a TTY, paints the first line and starts the spinner. On a pipe, starts the status lines, and writes the + * connecting line after 10 s unless the request was sent by then. */ public start(): void { if (!this.#options.isTTY) { - this.#firstLineTimer = setTimeout(() => this.#writePipeLine(this.#rows()[0]), PIPE_FIRST_LINE_DELAY_MS); - this.#firstLineTimer.unref?.(); + this.#connectingTimer = setTimeout( + () => this.#writePipeLine(this.#rows()[0]), + PIPE_CONNECTING_LINE_DELAY_MS + ); + this.#connectingTimer.unref?.(); this.#scheduleStatusLine(); return; } @@ -161,23 +176,34 @@ export class AgentProgressRenderer { /** * The daemon is not ready yet, but a live process can still make it ready, so the client waits up to `waitMs` - * more for it instead of running Rush in-process. On a pipe, says so once. + * more for it instead of running Rush in-process. On a pipe, says so once; the connecting line is then not due. */ public onAwaitStartup(waitMs: number): void { this.setPhase(STARTING_PHASE); - this.#writeMilestone('starting', ` (up to ${Math.round(waitMs / 1000)}s more)`); + clearTimeout(this.#connectingTimer); + this.#connectingTimer = undefined; + if (!this.#options.isTTY && !this.#stopped && !this.#startingLineWritten) { + this.#startingLineWritten = true; + this.#writePipeLine(`${this.#rows()[0]} (up to ${Math.round(waitMs / 1000)}s more)`); + } } - /** The daemon has the request. On a pipe, says so, and that the next line can take a while. */ + /** The daemon has the request. On a pipe, says so once, and that the next line can take a while. */ public onRequestSent(): void { this.setPhase(SENT_PHASE); - this.#writeMilestone('sent', ` (status at least every ${PIPE_STATUS_INTERVAL_MS / 1000}s)`); + clearTimeout(this.#connectingTimer); + this.#connectingTimer = undefined; + if (!this.#options.isTTY && !this.#stopped && !this.#sentLineWritten) { + this.#sentLineWritten = true; + this.#writePipeLine(`${this.#rows()[0]} (status at least every ${PIPE_STATUS_INTERVAL_MS / 1000}s)`); + } } + /** The request waits for admission. The status lines and the summary line say so. */ public onQueuePosition(position: number): void { this.#queued = { position, elapsed: this.#elapsed() }; + this.#firstQueued ??= this.#queued; this.setPhase(`queued behind another request (position ${position})`); - this.#writeMilestone('queued'); } public onEvent(event: IDaemonEventEnvelope): void { @@ -259,6 +285,11 @@ export class AgentProgressRenderer { const emptySelection: boolean = verdict === 'SUCCESS' && result?.operationResults?.length === 0 && !this.#tracker.hasOperations; let summary: string = this.#getSummaryLine(verdict, emptySelection); + // An admission failure says that the request waited, and why it stopped waiting. + if (this.#firstQueued && !result?.admissionErrorCode) { + const { position, elapsed } = this.#firstQueued; + summary += ` · queued behind another request (position ${position} at ${elapsed})`; + } if (errorMessage) { const [firstLine, ...detail] = errorMessage.split('\n').filter((line) => line.trim()); const admissionErrorCode: string | undefined = @@ -289,16 +320,39 @@ export class AgentProgressRenderer { if (typeof operationId !== 'string' || typeof status !== 'string') { return; } - const firstFailure: boolean = this.#tracker.updateStatus({ + this.#tracker.updateStatus({ operationId, status, logFilePath: typeof logFilePath === 'string' ? logFilePath : undefined }); this.#phase = 'running'; this.#queued = undefined; - this.#writeMilestone('running'); - if (firstFailure) { - this.#writeMilestone('failure', ` · first failure: ${operationId}`); + if (status === FAILURE_STATUS) { + this.#reportFailure(operationId); + } + } + + /** + * Writes a failed operation's log file and output excerpt as soon as it fails, while the rest of the request + * runs on. The operation's output all arrived before its status. An operation that wrote nothing is left to + * the failure report, which has the error from the daemon's result. + */ + #reportFailure(operationId: string): void { + if (this.#stopped || this.#reported.has(operationId) || this.#reported.size >= MAX_REPORTED_OPERATIONS) { + return; + } + const problem: IAgentProblemOperation = this.#tracker.getProblemOperation(operationId); + if (!problem.excerpt?.lineCount) { + return; + } + const lines: string[] = this.#getProblemLines('failed', problem); + const text: string = lines.map((line) => `${line}\n`).join(''); + if (this.#options.isTTY) { + this.#clear(); + this.#options.write(text); + this.#paint(); + } else { + this.#writePipeText(text); } } @@ -333,9 +387,10 @@ export class AgentProgressRenderer { } /** - * Each failed operation's log file and output excerpt. Without failed operations: operations with warnings - * (they fail a build unless the command allows warnings), or else output that belongs to no operation. A - * cancelled command reports only failed operations. + * The log file and output excerpt of each failed operation not yet reported. Without failed operations: + * operations with warnings (they fail a build unless the command allows warnings), or else output that belongs + * to no operation. A cancelled command reports only failed operations. At most three operations are reported + * in all, with the operations reported as they failed. */ #getFailureReport(verdict: Verdict): string[] { const tracker: AgentOperationTracker = this.#tracker; @@ -348,17 +403,17 @@ export class AgentProgressRenderer { } const label: string = tracker.failed.length ? 'failed' : 'warnings'; const lines: string[] = []; - for (const [index, problem] of problems.slice(0, MAX_REPORTED_OPERATIONS).entries()) { - const logFile: string = problem.logFilePath ? ` · full log: ${problem.logFilePath}` : ''; - lines.push(`${label}: ${problem.operationId}${logFile}`); - const maxLines: number = index === 0 ? FIRST_OPERATION_EXCERPT_LINES : OTHER_OPERATION_EXCERPT_LINES; - const excerpt: string[] = problem.excerpt?.getExcerpt(maxLines) ?? []; - if (!excerpt.length && problem.errorMessage) { - excerpt.push(clipLine(problem.errorMessage.trim().split('\n')[0], MAX_MESSAGE_LENGTH)); + let hidden: number = 0; + for (const problem of problems) { + if (this.#reported.has(problem.operationId)) { + continue; + } + if (this.#reported.size < MAX_REPORTED_OPERATIONS) { + lines.push(...this.#getProblemLines(label, problem)); + } else { + hidden++; } - lines.push(...(excerpt.length ? excerpt : ['(no output)']).map((line) => ` ${line}`)); } - const hidden: number = problems.length - MAX_REPORTED_OPERATIONS; if (hidden > 0) { const what: string = label === 'failed' ? 'failed operations' : 'operations with warnings'; lines.push(`+${hidden} more ${what}; their logs are in each project's rush-logs folder`); @@ -366,13 +421,33 @@ export class AgentProgressRenderer { return lines; } + /** + * An operation's report: its log file, then its output excerpt, which is longer for the first reported + * operation (most often the root cause). Records that the operation was reported. + */ + #getProblemLines(label: string, problem: IAgentProblemOperation): string[] { + const maxLines: number = this.#reported.size + ? OTHER_OPERATION_EXCERPT_LINES + : FIRST_OPERATION_EXCERPT_LINES; + this.#reported.add(problem.operationId); + const excerpt: string[] = problem.excerpt?.getExcerpt(maxLines) ?? []; + if (!excerpt.length && problem.errorMessage) { + excerpt.push(clipLine(problem.errorMessage.trim().split('\n')[0], MAX_MESSAGE_LENGTH)); + } + const logFile: string = problem.logFilePath ? ` · full log: ${problem.logFilePath}` : ''; + return [ + `${label}: ${problem.operationId}${logFile}`, + ...(excerpt.length ? excerpt : ['(no output)']).map((line) => ` ${line}`) + ]; + } + /** Returns false if rendering had already stopped; after stopping, nothing more is written. */ #stop(): boolean { if (this.#stopped) { return false; } this.#stopped = true; - for (const timer of [this.#firstLineTimer, this.#statusTimer]) { + for (const timer of [this.#connectingTimer, this.#statusTimer]) { clearTimeout(timer); } if (this.#timer) { @@ -397,26 +472,15 @@ export class AgentProgressRenderer { ]; } - #writeMilestone(milestone: PipeMilestone, suffix: string = ''): void { - if (this.#options.isTTY || this.#stopped || this.#milestones.has(milestone)) { - return; - } - this.#milestones.add(milestone); - this.#writePipeLine(`${this.#rows()[0]}${suffix}`); - } - #writePipeLine(line: string): void { - if (this.#pipeLines < MAX_PIPE_PROGRESS_LINES) { - this.#pipeLines++; - this.#writeStatus(line); + if (!this.#stopped) { + this.#writePipeText(`${line}\n`); } } - /** Writes a line on a pipe; the first line and the next status line are then due later. */ - #writeStatus(line: string): void { - clearTimeout(this.#firstLineTimer); - this.#firstLineTimer = undefined; - this.#options.write(`${line}\n`); + /** Writes to a pipe; the next status line is then due 25 s later. */ + #writePipeText(text: string): void { + this.#options.write(text); if (this.#statusTimer) { this.#scheduleStatusLine(); } @@ -424,11 +488,7 @@ export class AgentProgressRenderer { #scheduleStatusLine(): void { clearTimeout(this.#statusTimer); - this.#statusTimer = setTimeout(() => { - if (!this.#stopped) { - this.#writeStatus(this.#getStatusLine()); - } - }, PIPE_STATUS_INTERVAL_MS); + this.#statusTimer = setTimeout(() => this.#writePipeLine(this.#getStatusLine()), PIPE_STATUS_INTERVAL_MS); this.#statusTimer.unref?.(); } diff --git a/apps/rush-cli-client/src/OperationOutputExcerpt.ts b/apps/rush-cli-client/src/OperationOutputExcerpt.ts index 98671349c3..cb5b66d54a 100644 --- a/apps/rush-cli-client/src/OperationOutputExcerpt.ts +++ b/apps/rush-cli-client/src/OperationOutputExcerpt.ts @@ -35,6 +35,13 @@ const ERROR_PATTERN: RegExp = const NO_ERRORS_PATTERN: RegExp = /\b(?:0|no) errors?\b/i; /** Lines without any letter (bare exit codes, progress percentages, caret markers) explain nothing. */ const LETTER_PATTERN: RegExp = /\p{L}/u; +/** A severity word that some tools put before a diagnostic that they also print without it. */ +const SEVERITY_PREFIX_PATTERN: RegExp = /^(?:error|warning)\s*:\s*/i; +const WHITESPACE_PATTERN: RegExp = /\s+/g; +/** A tool's error count, such as Heft's `Encountered 2 errors` or tsc's `Found 1 error.` */ +const ERROR_COUNT_PATTERN: RegExp = /^(?:encountered|found) (\d+) errors?\b/i; +/** A source location such as `src/x.ts:3:7` or `src/x.ts(3,7)`. */ +const SOURCE_LOCATION_PATTERN: RegExp = /[\w-]\.[A-Za-z]\w{0,5}(?::\d+|\(\d+,\d+\))/; /** Lines longer than this keep their start and end, joined by an ellipsis. */ const MAX_LINE_LENGTH: number = 300; @@ -55,6 +62,8 @@ interface IExcerptLine { /** The arrival order across both streams, used to print the excerpt in output order. */ readonly index: number; readonly text: string; + /** Equal for lines that repeat one message; see {@link getRepeatKey}. */ + readonly key: string; } interface IErrorLine extends IExcerptLine { @@ -94,6 +103,40 @@ export function isErrorLine(line: string): boolean { return ERROR_PATTERN.test(line) && !NO_ERRORS_PATTERN.test(line); } +/** + * A normalized line's message without its task prefix, a leading `Error:` or `Warning:`, case and repeated + * whitespace. Tools such as Heft print each diagnostic when it occurs and again in their final summary, once with + * and once without these, so lines with the same key repeat one message. + */ +function getRepeatKey(line: string): string { + const prefixMatch: RegExpMatchArray | null = TASK_PREFIX_PATTERN.exec(line); + const content: string = prefixMatch ? prefixMatch[2] : line; + return content.replace(SEVERITY_PREFIX_PATTERN, '').replace(WHITESPACE_PATTERN, ' ').toLowerCase(); +} + +/** The number of errors that a tool's error count line reports, or undefined if the line is no such count. */ +function getReportedErrorCount(line: IExcerptLine): number | undefined { + const match: RegExpMatchArray | null = ERROR_COUNT_PATTERN.exec(line.key); + return match ? Number(match[1]) : undefined; +} + +/** + * Leaves out of chosen lines (in output order) the error counts that the chosen errors account for, and, when + * the first chosen error names a source location, the lines before it. + */ +function trimExcerpt(lines: ReadonlyArray): IExcerptLine[] { + const errors: IExcerptLine[] = lines.filter( + (line) => getReportedErrorCount(line) === undefined && isErrorLine(line.text) + ); + const kept: IExcerptLine[] = lines.filter((line) => { + const count: number | undefined = getReportedErrorCount(line); + return count === undefined || count === 0 || count > errors.length; + }); + return errors.length && SOURCE_LOCATION_PATTERN.test(errors[0].text) + ? kept.slice(kept.indexOf(errors[0])) + : kept; +} + /** Shortens a line to at most `maxLength` characters, keeping its start and its end. */ export function clipLine(line: string, maxLength: number): string { if (line.length <= maxLength) { @@ -113,7 +156,9 @@ export function clipLine(line: string, maxLength: number): string { * beginning. Like the native `StdioSummarizer`, head and tail lines come from stderr when the operation wrote * any, otherwise from stdout; error lines come from both, because tools such as tsc and eslint report errors * on stdout. Stack frames, `Require stack:` paths and code frames are dropped, so they cannot crowd out the - * cause. + * cause. A message that a tool repeats in its summary is shown once, and so is an error count that the shown + * errors already account for; when the first error shown names a source location, the lines before it (a + * tool's banner and progress) are left out. */ export class OperationOutputExcerpt { readonly #streams: Record = { @@ -121,7 +166,7 @@ export class OperationOutputExcerpt { stderr: { head: [], tail: [], partial: '' } }; readonly #errors: IErrorLine[] = []; - readonly #errorTexts: Set = new Set(); + readonly #errorKeys: Set = new Set(); /** * Per stream, the error line whose context is the stream's next line. The streams are separate pipes, so the * next line of the other stream is unrelated to the error. @@ -170,11 +215,11 @@ export class OperationOutputExcerpt { ? this.#streams.stderr : this.#streams.stdout; const chosen: Map = new Map(); - const texts: Set = new Set(); + const keys: Set = new Set(); const add = (line: IExcerptLine | undefined, limit: number): void => { - if (line && chosen.size < limit && !chosen.has(line.index) && !texts.has(line.text)) { + if (line && chosen.size < limit && !chosen.has(line.index) && !keys.has(line.key)) { chosen.set(line.index, line); - texts.add(line.text); + keys.add(line.key); } }; const errorLimit: number = Math.max(1, maxLines - TAIL_RESERVE); @@ -192,7 +237,7 @@ export class OperationOutputExcerpt { for (const line of preferred.head) { add(line, maxLines); } - return [...chosen.values()].sort((a, b) => a.index - b.index).map((line) => line.text); + return trimExcerpt([...chosen.values()].sort((a, b) => a.index - b.index)).map((line) => line.text); } #addLine(rawLine: string, stream: OutputStream): void { @@ -200,7 +245,7 @@ export class OperationOutputExcerpt { if (text === undefined) { return; } - const line: IExcerptLine = { index: this.#lineCount++, text }; + const line: IExcerptLine = { index: this.#lineCount++, text, key: getRepeatKey(text) }; const lines: IStreamLines = this.#streams[stream]; if (lines.head.length < HEAD_LINES) { lines.head.push(line); @@ -214,10 +259,10 @@ export class OperationOutputExcerpt { pendingContext.context = line; this.#pendingContext[stream] = undefined; } - if (this.#errors.length < ERROR_LINES && !this.#errorTexts.has(text) && isErrorLine(text)) { + if (this.#errors.length < ERROR_LINES && !this.#errorKeys.has(line.key) && isErrorLine(text)) { const errorLine: IErrorLine = { ...line }; this.#errors.push(errorLine); - this.#errorTexts.add(text); + this.#errorKeys.add(line.key); this.#pendingContext[stream] = errorLine; } } diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts index 3fdee47959..9d6921e648 100644 --- a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -34,13 +34,13 @@ interface ITestRenderer { lines(): string[]; } -function createRenderer(isTTY: boolean, commandName: string = 'build'): ITestRenderer { +function createRenderer(isTTY: boolean, commandName: string = 'build', columns: number = 60): ITestRenderer { const output: string[] = []; const clock: { ms: number } = { ms: 0 }; const renderer: AgentProgressRenderer = new AgentProgressRenderer({ commandName, isTTY, - columns: 60, + columns, write: (text: string) => output.push(text), now: () => clock.ms, startTimeMs: 0 @@ -79,7 +79,7 @@ function fail(renderer: AgentProgressRenderer, operationId: string, errorLines: } describe(AgentProgressRenderer.name, () => { - it('writes one line when the request is sent, then milestones and a summary for a successful build (pipe)', () => { + it('writes one line when the request is sent and a summary line for a successful build (pipe)', () => { const { renderer, output, clock, lines } = createRenderer(false); renderer.start(); expect(output).toEqual([]); @@ -101,7 +101,6 @@ describe(AgentProgressRenderer.name, () => { renderer.dispose(); expect(lines()).toEqual([ 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', - 'rush build 0/2 · 0.0s · running', 'rush build: SUCCESS 2/2 operations (1 success, 1 up to date) in 3.0s' ]); }); @@ -133,18 +132,17 @@ describe(AgentProgressRenderer.name, () => { ]); }); - it('reports a failed operation with its log file and an excerpt, before the summary line', () => { + it('reports a failed operation with its log file and an excerpt as soon as it fails', () => { const { renderer, lines } = createRenderer(false); fail( renderer, 'p05 (build)', Array.from({ length: 20 }, (unused, i) => `error ${i}`) ); + expect(lines()).toHaveLength(9); renderer.onEvent(status('p06 (build)', 'BLOCKED')); renderer.finish({ exitCode: 1 }); expect(lines()).toEqual([ - 'rush build 0/1 · 0.0s · running', - 'rush build 1/1 · 0.0s · running · first failure: p05 (build)', 'failed: p05 (build) · full log: /repo/p05/rush-logs/x.log', ' error 0', ' error 1', @@ -209,10 +207,7 @@ describe(AgentProgressRenderer.name, () => { { operationId: 'b (build)', status: 'ABORTED' } ] }); - expect(lines()).toEqual([ - 'rush build 0/2 · 0.0s · running', - 'rush build: CANCELLED 2/2 operations (2 aborted) in 0.0s' - ]); + expect(lines()).toEqual(['rush build: CANCELLED 2/2 operations (2 aborted) in 0.0s']); }); it('still reports the failures of a cancelled command', () => { @@ -374,13 +369,30 @@ describe(AgentProgressRenderer.name, () => { ]); }); - it('writes the queue milestone once, however often the position changes', () => { - const { renderer, output } = createRenderer(false); - renderer.onQueuePosition(2); + it('reports the queue in the summary line rather than in a line of its own (pipe)', () => { + const { renderer, output, clock } = createRenderer(false); + clock.ms = 100; renderer.onQueuePosition(2); + clock.ms = 5000; renderer.onQueuePosition(1); - expect(output).toEqual(['rush build · 0.0s · queued behind another request (position 2)\n']); - renderer.dispose(); + expect(output).toEqual([]); + clock.ms = 15_000; + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onEvent(status('a (build)', 'SUCCESS')); + renderer.finish({ exitCode: 0 }); + expect(output).toEqual([ + 'rush build: SUCCESS 1/1 operations (1 success) in 15.0s · ' + + 'queued behind another request (position 2 at 0.1s)\n' + ]); + }); + + it('leaves the queue to the message of a request that was not admitted', () => { + const { renderer, output } = createRenderer(false); + renderer.onQueuePosition(1); + renderer.finish({ exitCode: 1, admissionErrorCode: 'no-wait', errorMessage: 'The workspace is busy.' }); + expect(output).toEqual([ + 'rush build: FAILURE in 0.0s · daemon admission failed (no-wait): The workspace is busy.\n' + ]); }); it('shows the excerpt of a failed operation that reported errors on stdout', () => { @@ -408,10 +420,11 @@ describe(AgentProgressRenderer.name, () => { ]); }); - it('shows queue position immediately', () => { - const { renderer, output } = createRenderer(false); + it('shows the queue position immediately on a TTY', () => { + const { renderer, output } = createRenderer(true, 'build', 120); renderer.onQueuePosition(2); expect(output[0]).toContain('queued behind another request (position 2)'); + renderer.dispose(); }); it('writes a final summary line after a queued request completes', () => { @@ -420,10 +433,13 @@ describe(AgentProgressRenderer.name, () => { renderer.onQueuePosition(1); clock.ms = 4000; renderer.finish({ exitCode: 0 }); - expect(output[output.length - 1]).toBe('rush build: SUCCESS up to date (no operations needed) in 4.0s\n'); + expect(output[output.length - 1]).toBe( + 'rush build: SUCCESS up to date (no operations needed) in 4.0s · ' + + 'queued behind another request (position 1 at 0.0s)\n' + ); }); - it('writes at most three progress lines and one summary line on a pipe at odsp-web scale', () => { + it('writes one progress line and one summary line on a pipe at odsp-web scale', () => { const { renderer, clock, lines } = createRenderer(false); renderer.start(); renderer.onRequestSent(); @@ -450,13 +466,12 @@ describe(AgentProgressRenderer.name, () => { renderer.finish({ exitCode: 0 }); expect(lines()).toEqual([ 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', - 'rush build · 0.0s · queued behind another request (position 1)', - 'rush build 0/772 · 0.0s · running', - 'rush build: SUCCESS 772/772 operations (386 success, 386 from cache) in 210.0s' + 'rush build: SUCCESS 772/772 operations (386 success, 386 from cache) in 210.0s · ' + + 'queued behind another request (position 1 at 0.0s)' ]); }); - it('keeps a failure at odsp-web scale to the progress lines, the failure report and one summary line', () => { + it('keeps a failure at odsp-web scale to the sent line, the failure report and one summary line', () => { const { renderer, lines } = createRenderer(false); renderer.start(); renderer.onRequestSent(); @@ -478,11 +493,8 @@ describe(AgentProgressRenderer.name, () => { renderer.finish({ exitCode: 1 }); expect(lines()).toEqual([ 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', - 'rush build 0/772 · 0.0s · running', - 'rush build 701/772 · 0.0s · running · first failure: p700 (build)', 'failed: p700 (build) · full log: /repo/p700/rush-logs/x.log', ' src/x.ts:1:1 - error TS2304: Cannot find name "y".', - ' Encountered 1 error', 'rush build: FAILURE 772/772 operations (1 failure, 71 blocked, 700 success) in 0.0s · failed: p700 (build)' ]); }); @@ -509,10 +521,7 @@ describe(AgentProgressRenderer.name, () => { renderer.onEvent(header('a (build)', 1, 772)); renderer.onEvent(status('a (build)', 'SUCCESS')); renderer.finish({ exitCode: 0 }); - expect(lines()).toEqual([ - 'rush build 1/772 · 0.0s · running', - 'rush build: SUCCESS 1/772 operations (1 success) in 0.0s' - ]); + expect(lines()).toEqual(['rush build: SUCCESS 1/772 operations (1 success) in 0.0s']); }); it('counts an operation that runs again once', () => { @@ -565,7 +574,7 @@ describe(AgentProgressRenderer.name, () => { ); } renderer.finish({ exitCode: 1 }); - const report: string[] = lines().slice(2); + const report: string[] = lines(); expect(report.filter((line) => line.startsWith('failed: '))).toEqual([ 'failed: f0 (build) · full log: /repo/f0/rush-logs/x.log', 'failed: f1 (build) · full log: /repo/f1/rush-logs/x.log', @@ -581,6 +590,65 @@ describe(AgentProgressRenderer.name, () => { ]); }); + it('reports a failed operation that wrote no output with the error from the daemon result', () => { + const { renderer, lines } = createRenderer(false); + fail(renderer, 'a (build)', ['src/a.ts:1:1 - error TS2322: a']); + renderer.onEvent(status('quiet (build)', 'FAILURE')); + fail(renderer, 'b (build)', ['src/b.ts:1:1 - error TS2322: b']); + expect(lines()).toHaveLength(4); + renderer.finish({ + exitCode: 1, + operationResults: [ + { operationId: 'a (build)', status: 'FAILURE' }, + { operationId: 'quiet (build)', status: 'FAILURE', errorMessage: 'spawn heft ENOENT' }, + { operationId: 'b (build)', status: 'FAILURE' } + ] + }); + expect(lines()).toEqual([ + 'failed: a (build) · full log: /repo/a/rush-logs/x.log', + ' src/a.ts:1:1 - error TS2322: a', + 'failed: b (build) · full log: /repo/b/rush-logs/x.log', + ' src/b.ts:1:1 - error TS2322: b', + 'failed: quiet (build)', + ' spawn heft ENOENT', + 'rush build: FAILURE 3/3 operations (3 failure) in 0.0s · failed: a (build), quiet (build), b (build)' + ]); + }); + + it('reports at most three operations in all, as they failed or before the summary line', () => { + const { renderer, lines } = createRenderer(false); + fail(renderer, 'a (build)', ['a error']); + for (const name of ['q1', 'q2', 'q3']) { + renderer.onEvent(status(`${name} (build)`, 'FAILURE')); + } + fail(renderer, 'b (build)', ['b error']); + renderer.finish({ exitCode: 1 }); + expect(lines()).toEqual([ + 'failed: a (build) · full log: /repo/a/rush-logs/x.log', + ' a error', + 'failed: b (build) · full log: /repo/b/rush-logs/x.log', + ' b error', + 'failed: q1 (build)', + ' (no output)', + "+2 more failed operations; their logs are in each project's rush-logs folder", + 'rush build: FAILURE 5/5 operations (5 failure) in 0.0s · ' + + 'failed: a (build), q1 (build), q2 (build), q3 (build), b (build)' + ]); + }); + + it('writes a failure report above the live rows on a TTY when the operation fails', () => { + const { renderer, output } = createRenderer(true); + renderer.start(); + fail(renderer, 'a (build)', ['src/a.ts:1:1 - error TS2322: a']); + renderer.dispose(); + expect(output[1]).toBe('\x1b[3A\x1b[0J\x1b[?25h'); + expect(output[2]).toBe( + 'failed: a (build) · full log: /repo/a/rush-logs/x.log\n src/a.ts:1:1 - error TS2322: a\n' + ); + expect(output[3].replace(ANSI_ESCAPE, '').split('\n')[2]).toBe('failed: a (build)'); + expect(output.slice(4)).toEqual(['\x1b[3A\x1b[0J\x1b[?25h']); + }); + it('shows the output of a command that failed without running operations', () => { const { renderer, lines } = createRenderer(false, 'install'); renderer.onLog(Buffer.from('Installing packages\n'), 'request-id', 'stdout'); @@ -612,44 +680,70 @@ describe(AgentProgressRenderer.name, () => { jest.advanceTimersByTime(ms); } - it('writes the connecting line only when the connection takes more than a second', () => { + it('writes the connecting line only when the connection takes more than 10 s', () => { + const fast: ITestRenderer = createRenderer(false); + fast.renderer.start(); + advance(fast.clock, 3000); + fast.renderer.onRequestSent(); + advance(fast.clock, 10_000); + fast.renderer.dispose(); + expect(fast.lines()).toEqual([ + 'rush build · 3.0s · sent to rushd; preparing the workspace graph (status at least every 25s)' + ]); + const { renderer, output, clock, lines } = createRenderer(false); renderer.start(); - advance(clock, 999); + advance(clock, 9999); expect(output).toEqual([]); advance(clock, 1); renderer.onRequestSent(); renderer.dispose(); expect(lines()).toEqual([ - 'rush build · 1.0s · connecting to rushd (auto-starts if needed)', - 'rush build · 1.0s · sent to rushd; preparing the workspace graph (status at least every 25s)' + 'rush build · 10.0s · connecting to rushd (auto-starts if needed)', + 'rush build · 10.0s · sent to rushd; preparing the workspace graph (status at least every 25s)' ]); }); it('writes one line when the client waits for a daemon that is still starting (task 95)', () => { - const { renderer, output, clock, lines } = createRenderer(false); - renderer.start(); - advance(clock, 300); - renderer.onAwaitStartup(15_000); - expect(lines()).toEqual([ + const early: ITestRenderer = createRenderer(false); + early.renderer.start(); + advance(early.clock, 300); + early.renderer.onAwaitStartup(15_000); + expect(early.lines()).toEqual([ 'rush build · 0.3s · rushd is still starting; waiting for it (up to 15s more)' ]); - advance(clock, 1_000); + advance(early.clock, 1_000); + early.renderer.onAwaitStartup(15_000); + // The connecting line is not due after this line. + advance(early.clock, 11_000); + expect(early.output).toHaveLength(1); + early.renderer.onRequestSent(); + early.renderer.dispose(); + expect(early.lines()).toEqual([ + 'rush build · 0.3s · rushd is still starting; waiting for it (up to 15s more)', + 'rush build · 12.3s · sent to rushd; preparing the workspace graph (status at least every 25s)' + ]); + + // As usual, the first startup deadline (15 s) comes after the connecting line. + const { renderer, clock, lines } = createRenderer(false); + renderer.start(); + advance(clock, 10_000); + advance(clock, 5_000); renderer.onAwaitStartup(15_000); - expect(output).toHaveLength(1); - advance(clock, 11_000); + advance(clock, 6_000); renderer.onRequestSent(); renderer.dispose(); expect(lines()).toEqual([ - 'rush build · 0.3s · rushd is still starting; waiting for it (up to 15s more)', - 'rush build · 12.3s · sent to rushd; preparing the workspace graph (status at least every 25s)' + 'rush build · 10.0s · connecting to rushd (auto-starts if needed)', + 'rush build · 15.0s · rushd is still starting; waiting for it (up to 15s more)', + 'rush build · 21.0s · sent to rushd; preparing the workspace graph (status at least every 25s)' ]); }); - it('writes nothing for a request that is handed to in-process Rush within a second', () => { + it('writes nothing for a request that is handed to in-process Rush within 10 s', () => { const { renderer, output, clock } = createRenderer(false); renderer.start(); - advance(clock, 500); + advance(clock, 9000); renderer.dispose(); advance(clock, 60_000); expect(output).toEqual([]); @@ -665,9 +759,11 @@ describe(AgentProgressRenderer.name, () => { advance(clock, 20_000); renderer.onEvent(status('a (build)', 'EXECUTING')); renderer.onEvent(status('b (build)', 'EXECUTING')); - advance(clock, 24_999); - expect(lines()).toHaveLength(2); + advance(clock, 4999); + expect(lines()).toHaveLength(1); advance(clock, 1); + expect(lines()).toHaveLength(2); + advance(clock, 20_000); renderer.onEvent(status('c (build)', 'EXECUTING')); renderer.onEvent(status('d (build)', 'EXECUTING')); renderer.onEvent(status('e (build)', 'EXECUTING')); @@ -679,14 +775,12 @@ describe(AgentProgressRenderer.name, () => { renderer.finish({ exitCode: 1 }); expect(lines()).toEqual([ 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', - 'rush build 0/5 · 20.0s · running', - 'rush build 0/5 · 45.0s · running: a (build), b (build)', - // The first failure is a milestone, so it is written although status lines were written before it. - 'rush build 1/5 · 45.0s · running · first failure: a (build)', - 'rush build 1/5 · 70.0s · running: b (build), c (build), d (build) +1 more · failed: a (build)', - 'rush build 2/5 · 95.0s · running: c (build), d (build), e (build) · failed: a (build)', + 'rush build 0/5 · 25.0s · running: a (build), b (build)', + // The failure report is written when the operation fails, and the next status line is due 25 s later. 'failed: a (build) · full log: /repo/a/rush-logs/x.log', ' error TS2322', + 'rush build 1/5 · 70.0s · running: b (build), c (build), d (build) +1 more · failed: a (build)', + 'rush build 2/5 · 95.0s · running: c (build), d (build), e (build) · failed: a (build)', 'rush build: FAILURE 2/5 operations (1 failure, 1 success) in 96.0s · failed: a (build)' ]); }); @@ -697,21 +791,19 @@ describe(AgentProgressRenderer.name, () => { renderer.onRequestSent(); advance(clock, 2000); renderer.onQueuePosition(1); - advance(clock, 25_000); + advance(clock, 23_000); renderer.onEvent(registered('a (build)')); renderer.onEvent(status('a (build)', 'EXECUTING')); advance(clock, 25_000); renderer.dispose(); expect(lines()).toEqual([ 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', - 'rush build · 2.0s · queued behind another request (position 1)', - 'rush build · 27.0s · waiting for admission or the workspace graph (queue position 1 at 2.0s)', - 'rush build 0/1 · 27.0s · running', - 'rush build 0/1 · 52.0s · running: a (build)' + 'rush build · 25.0s · waiting for admission or the workspace graph (queue position 1 at 2.0s)', + 'rush build 0/1 · 50.0s · running: a (build)' ]); }); - it('writes status lines after the milestone lines ran out, and none after the summary', () => { + it('writes status lines after a failure report, and none after the summary', () => { const { renderer, output, clock, lines } = createRenderer(false); renderer.start(); renderer.onRequestSent(); diff --git a/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts b/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts index 4e241e65b0..bc139a88fd 100644 --- a/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts +++ b/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts @@ -157,13 +157,11 @@ describe(OperationOutputExcerpt.name, () => { '[test:jest] but they are not recognized by this version of Rush: RUSH_EXAMPLE', '[test:jest] ● test two', '[test:jest] ● test three', - '[test:jest] Error: 3 Jest tests failed', - 'Encountered 1 error' + '[test:jest] Error: 3 Jest tests failed' ]); expect(excerpt.getExcerpt(3)).toEqual([ '[test:jest] ● test one', - '[test:jest] Error: 3 Jest tests failed', - 'Encountered 1 error' + '[test:jest] Error: 3 Jest tests failed' ]); }); @@ -173,8 +171,7 @@ describe(OperationOutputExcerpt.name, () => { excerpt.append('[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".\n', 'stdout'); excerpt.append('[build:typescript] Encountered 1 error\n', 'stderr'); expect(excerpt.getExcerpt(8)).toEqual([ - '[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".', - '[build:typescript] Encountered 1 error' + '[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".' ]); }); @@ -187,8 +184,7 @@ describe(OperationOutputExcerpt.name, () => { excerpt.append('[build:lint] Using ESLint version 9.37.0\n', 'stdout'); excerpt.append('Error: Encountered 1 error\n', 'stderr'); expect(excerpt.getExcerpt(3)).toEqual([ - '[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".', - 'Error: Encountered 1 error' + '[build:typescript] src/x.ts:1:1 - error TS2304: Cannot find name "y".' ]); }); @@ -208,11 +204,44 @@ describe(OperationOutputExcerpt.name, () => { "src/x.ts(3,7): error TS2322: Type 'string' is not assignable to type 'number'.", 'The expected type comes from property "x".', "src/y.ts(1,1): error TS2304: Cannot find name 'z'.", - 'Did you mean "y"?', - 'Encountered 2 errors' + 'Did you mean "y"?' ]); }); + it('shows a diagnostic that a tool repeats in its summary once, without the error count', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + const message: string = "src/index.ts:3:7 - (TS2322) Type 'string' is not assignable to type 'number'."; + excerpt.append(`[build:typescript] Error: ${message}\n`, 'stdout'); + excerpt.append('Encountered 1 error\n', 'stderr'); + excerpt.append(` [build:typescript] ${message}\n`, 'stderr'); + expect(excerpt.getExcerpt(8)).toEqual([`[build:typescript] Error: ${message}`]); + }); + + it('keeps an error count that reports more errors than the excerpt shows', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + for (let i: number = 0; i < 12; i++) { + excerpt.append(`src/x${i}.ts:1:1 - error TS2304: Cannot find name "y${i}".\n`, 'stdout'); + } + excerpt.append('Encountered 12 errors\n', 'stderr'); + expect(excerpt.getExcerpt(3)).toEqual([ + 'src/x0.ts:1:1 - error TS2304: Cannot find name "y0".', + 'src/x1.ts:1:1 - error TS2304: Cannot find name "y1".', + 'Encountered 12 errors' + ]); + }); + + it('leaves out the lines before an error that names a source location, but not before other errors', () => { + const located: OperationOutputExcerpt = new OperationOutputExcerpt(); + located.append('[build] @x/a: start\n', 'stdout'); + located.append("ERROR in src/index.ts:4:1: Unexpected token '}'\n", 'stdout'); + expect(located.getExcerpt(8)).toEqual(["ERROR in src/index.ts:4:1: Unexpected token '}'"]); + + const unlocated: OperationOutputExcerpt = new OperationOutputExcerpt(); + unlocated.append('src/a.ts(1,1): something unexpected\n', 'stdout'); + unlocated.append('Build failed\n', 'stdout'); + expect(unlocated.getExcerpt(8)).toEqual(['src/a.ts(1,1): something unexpected', 'Build failed']); + }); + it('always keeps the last line, which usually is the tool summary', () => { const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); for (let i: number = 0; i < 20; i++) { diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-early-failures_2026-09-28-18-14.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-early-failures_2026-09-28-18-14.json new file mode 100644 index 0000000000..555f3a7a98 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-early-failures_2026-09-28-18-14.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, report a failed operation's log file and output excerpt as soon as it fails; on a pipe, write only the line that says the request was sent, plus status lines after 25 s of silence, and report the queue in the summary line. Excerpts no longer repeat a diagnostic that a tool prints again in its summary, or an error count that the shown errors account for, and start at the first error when it names a source location.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client" +} From 52e592c3db59d50f809c4218b7ff06184763bdb3 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 22:04:14 +0000 Subject: [PATCH 038/265] [rush-lib] Keep results that a plugin found up to date in long-lived operation graphs Swarm integration step 23; original commit e16ed01d3a (merge of swarm/r01-t85-a81 at 4066e8f5d7). Scope: task 85. Brings task 85 onto task 108 items 1+3: the daemon keeps a result that a plugin such as fstrace found up to date across requests, and checks a Skipped result again when it was marked unverifiable. m01 CONFIRMED the gated tip (board 2024: 72 of 72 hot requests "1066 no op") and measured 24 of 24 hot B6 requests as "no operations needed" (board 2073); ch04 CONFIRMED 6e39ea6527 (board 2003) and posted the product-default A/B (board 2132: -0.24 s [-0.57, +0.09]). ch01's combined suite on this tree passes (board 2150: rush-lib 1169/1, a timing flake in InstallRunScripts.test that task 85 doesn't touch, 5 of 5 on rerun; rush-daemon 515/0, rush-cli-client 367/0). Gate: ch01 GATE OK board 1955, combined suite board 2150 (tree 1c6a3bfa10) Commits folded into this step (2): - 95c8ece420 [rush-lib] Retain results that a plugin found up to date in long-lived operation graphs - 4066e8f5d7 [rush-lib] Check a Skipped result again if it was marked unverifiable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...ugin-skipped-results_2026-09-28-18-39.json | 11 ++ ...ugin-skipped-results_2026-09-28-18-39.json | 11 ++ .../src/OperationOutputFingerprints.ts | 12 +- .../ProductionDaemonRequestResolver.test.ts | 49 ++++++ .../src/logic/operations/IOperationGraph.ts | 4 +- .../src/logic/operations/OperationGraph.ts | 9 +- .../logic/operations/PhasedOperationPlugin.ts | 40 ++++- .../operations/RetainedResultVerification.ts | 10 +- ...ableOperationPluginRetainedResults.test.ts | 72 +++++++- ...asedOperationPluginRetainedResults.test.ts | 154 +++++++++++++++++- 10 files changed, 348 insertions(+), 24 deletions(-) create mode 100644 common/changes/@microsoft/rush/retain-plugin-skipped-results_2026-09-28-18-39.json create mode 100644 common/changes/@rushstack/rush-daemon/retain-plugin-skipped-results_2026-09-28-18-39.json diff --git a/common/changes/@microsoft/rush/retain-plugin-skipped-results_2026-09-28-18-39.json b/common/changes/@microsoft/rush/retain-plugin-skipped-results_2026-09-28-18-39.json new file mode 100644 index 0000000000..beb64848e6 --- /dev/null +++ b/common/changes/@microsoft/rush/retain-plugin-skipped-results_2026-09-28-18-39.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "In long-lived operation graphs such as the Rush daemon, a selected operation that a plugin or the legacy skip detection found up to date now keeps that result while its state hash is unchanged, so later builds no longer check it again.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/retain-plugin-skipped-results_2026-09-28-18-39.json b/common/changes/@rushstack/rush-daemon/retain-plugin-skipped-results_2026-09-28-18-39.json new file mode 100644 index 0000000000..786d960c38 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/retain-plugin-skipped-results_2026-09-28-18-39.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "If the declared output folders of an operation that a plugin or the legacy skip detection found up to date are deleted, the next request runs that operation again instead of reusing its retained result.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/OperationOutputFingerprints.ts b/libraries/rush-daemon/src/OperationOutputFingerprints.ts index 7536718113..25836885af 100644 --- a/libraries/rush-daemon/src/OperationOutputFingerprints.ts +++ b/libraries/rush-daemon/src/OperationOutputFingerprints.ts @@ -14,11 +14,14 @@ import { const PLUGIN_NAME: 'DaemonOperationOutputFingerprints' = 'DaemonOperationOutputFingerprints'; /** - * Retained results that allow the warm graph to skip an operation. Other statuses always re-run. + * Retained results that allow the warm graph to skip an operation. Other statuses always re-run. The graph only + * retains a `Skipped` result for an operation that it selected, when a plugin (e.g. change detection) found its + * outputs up to date. */ const TRACKED_STATUSES: ReadonlySet = new Set([ OperationStatus.Success, - OperationStatus.FromCache + OperationStatus.FromCache, + OperationStatus.Skipped ]); interface IOutputFingerprint { @@ -27,7 +30,8 @@ interface IOutputFingerprint { } /** - * Detects retained successful operations whose declared output folders were changed outside the daemon. + * Detects retained successful or up-to-date operations whose declared output folders were changed outside + * the daemon. * * @remarks * Build outputs are normally git-ignored, so they do not contribute to any operation state hash. Without @@ -52,7 +56,7 @@ export class OperationOutputFingerprints { } /** - * Returns retained successful operations whose output folders no longer match the recorded fingerprint. + * Returns retained operations whose output folders no longer match the recorded fingerprint. * * @remarks * Fingerprints of changed operations are forgotten only after all cleanup succeeded, so a failed diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index b4fb7b3e08..b2d718df00 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -1569,6 +1569,55 @@ process.exit(23); } }); + it('reuses results that legacy skip detection found up to date until their declared outputs are deleted', async () => { + const fixture: IFixture = await createFixtureAsync(); + try { + await runAsync(fixture, 'initial', ['build']); + expect(runs(fixture)).toHaveLength(3); + const graph: IOperationGraph | undefined = fixture.session.operationGraph; + // Reload the configuration without changing the inputs of any operation, so the new graph has no results. + const commandLineFile: string = path.join(fixture.repoRoot, 'common/config/rush/command-line.json'); + const commandLine: { commands: object[] } = JSON.parse(fs.readFileSync(commandLineFile, 'utf8')); + commandLine.commands.push({ + commandKind: 'global', + name: 'unrelated', + summary: 'Unrelated', + shellCommand: 'node --version' + }); + fs.writeFileSync(commandLineFile, JSON.stringify(commandLine)); + const reloaded: ITerminalExchange = await runAsync(fixture, 'reloaded', ['build']); + expect(reloaded.terminal).toMatchObject({ + kind: 'requestResult', + payload: { + exitCode: 0, + scheduled: true, + operationResults: [ + { operationId: 'a (compile)', status: 'SKIPPED' }, + { operationId: 'b (compile)', status: 'SKIPPED' }, + { operationId: 'c (compile)', status: 'SKIPPED' } + ] + } + }); + expect(fixture.session.operationGraph).not.toBe(graph); + + // The skipped results are current, so they are not checked again. + expect((await runAsync(fixture, 'warm', ['build'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: false } + }); + + fs.rmSync(path.join(fixture.repoRoot, 'projects/c/lib'), { recursive: true }); + expect((await runAsync(fixture, 'deleted', ['build'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: true } + }); + expect(runs(fixture).slice(3)).toEqual(['c:one:']); + expect(fs.readFileSync(path.join(fixture.repoRoot, 'projects/c/lib/output.txt'), 'utf8')).toBe('one'); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('restores deleted outputs of a warm operation from the native build cache', async () => { const fixture: IFixture = await createFixtureAsync(true); try { diff --git a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts index 9b72d55383..7da9225e19 100644 --- a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts @@ -54,7 +54,9 @@ export interface IOperationGraph { * A map from each `Operation` in the graph to its current result record. * The map is updated in real time as operations execute during an iteration. * Only statuses representing a completed execution (e.g. `Success`, `Failure`, - * `SuccessWithWarning`) write to this map; statuses such as `Skipped` or `Aborted` — + * `SuccessWithWarning`) write to this map, as does `Skipped` for an operation that was + * selected to execute and whose outputs a plugin (e.g. change detection) found up to date. + * Statuses such as `Aborted`, or `Skipped` for an operation that was not selected — * which indicate that an operation did not actually run — do not update it. * For operations that have not yet run in the current iteration, the map retains the * result from whichever prior iteration the operation last ran in. diff --git a/libraries/rush-lib/src/logic/operations/OperationGraph.ts b/libraries/rush-lib/src/logic/operations/OperationGraph.ts index 09da8be3a0..68cc1ed4c1 100644 --- a/libraries/rush-lib/src/logic/operations/OperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/OperationGraph.ts @@ -1400,8 +1400,13 @@ function _handleOperationFromCache( * Handle skipped operation. */ function _handleOperationSkipped(record: OperationExecutionRecord, context: IStatefulExecutionContext): void { - // Do not set resultByOperation here. "Skipped" means the operation was not executed, - // so it should not be considered the last *execution* result. + if (record.enabled) { + // The operation was selected to execute, and a plugin (e.g. change detection) reported that its outputs are + // already up to date for its current state hash. Keep this as its last result, so that a long-lived graph + // (e.g. the Rush daemon) can reuse it while the state hash is unchanged instead of checking it again. + context.resultByOperation.set(record.operation, record); + } + // Otherwise, the operation was not executed, so it should not be considered the last *execution* result. if (!record.silent) { record.eventSink?.onActivity?.(`"${record.name}" was skipped.`, { operationId: record.name }); record.collatedWriter.terminal.writeStdoutLine(Colorize.green(`"${record.name}" was skipped.`)); diff --git a/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts index 093ce3fc7f..105cb23b26 100644 --- a/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/PhasedOperationPlugin.ts @@ -26,6 +26,16 @@ const PLUGIN_NAME: 'PhasedOperationPlugin' = 'PhasedOperationPlugin'; // unverifiable. const VERIFY_RESULT_STAGE: number = 1; +/** + * The statuses of results retained by an earlier iteration of the graph that are reused while the state hash of the + * operation is unchanged. The graph only retains a `Skipped` result for an operation that was selected to execute, + * when a plugin (e.g. change detection) found its outputs up to date. + */ +const RETAINED_RESULT_STATUSES: ReadonlySet = new Set([ + ...SUCCESS_STATUSES, + OperationStatus.Skipped +]); + /** * Core phased command plugin that provides the functionality for generating a base operation graph * from the set of selected projects and phases. @@ -151,7 +161,12 @@ function configureExecutionManager(graph: IOperationGraph, context: IOperationGr if (iterationOptions.inputsSnapshot) { // A retained result that is current by state hash can still have been built against outputs of a // dependency that were not current, e.g. by an `--only` request. - enableUnverifiedRetainedOperations(currentStates, lastStates, verifiedStateHashByOperation); + enableUnverifiedRetainedOperations( + currentStates, + lastStates, + verifiedStateHashByOperation, + RETAINED_RESULT_STATUSES + ); } } ); @@ -195,8 +210,20 @@ function updateVerifiedStateHash( const { operation } = record; switch (record.status) { case OperationStatus.Skipped: { - // The outputs were left as they were. - return; + if (!record.enabled) { + // The operation was not selected, so its outputs were left as they were. + return; + } + // A plugin (e.g. change detection) found the outputs of the selected operation up to date for its state + // hash, which verifies them in the same way as executing it. + if ( + !isResultUnverifiable(record) && + areDependenciesVerified(operation, records, verifiedStateHashByOperation) + ) { + verifiedStateHashByOperation.set(operation, record.getStateHash()); + return; + } + break; } case OperationStatus.FromCache: { @@ -234,7 +261,10 @@ function areDependenciesVerified( ): boolean { for (const dependency of operation.dependencies) { const dependencyRecord: IOperationExecutionResult | undefined = records.get(dependency); - if (!dependencyRecord || verifiedStateHashByOperation.get(dependency) !== dependencyRecord.getStateHash()) { + if ( + !dependencyRecord || + verifiedStateHashByOperation.get(dependency) !== dependencyRecord.getStateHash() + ) { return false; } } @@ -250,7 +280,7 @@ function shouldEnableOperation( return true; } - if (!SUCCESS_STATUSES.has(lastState.status)) { + if (!RETAINED_RESULT_STATUSES.has(lastState.status)) { return true; } diff --git a/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts b/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts index f15be001ff..5cadda9523 100644 --- a/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts +++ b/libraries/rush-lib/src/logic/operations/RetainedResultVerification.ts @@ -3,7 +3,7 @@ import type { Operation } from './Operation'; import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; -import { SUCCESS_STATUSES } from './OperationStatus'; +import { type OperationStatus, SUCCESS_STATUSES } from './OperationStatus'; const unverifiableResults: WeakSet = new WeakSet(); @@ -29,7 +29,7 @@ export function isResultUnverifiable(result: IOperationExecutionResult): boolean } /** - * Re-enables selected operations whose successful result retained by a previous iteration of a long-lived graph + * Re-enables selected operations whose result retained by a previous iteration of a long-lived graph * (e.g. the Rush daemon) is current by state hash, but not verified at that state hash, so that they are restored * from the build cache or executed instead of being skipped. Such a result was produced while the outputs of one * of its dependencies were not verified (e.g. that dependency was not selected and had changed), so its outputs @@ -43,11 +43,13 @@ export function isResultUnverifiable(result: IOperationExecutionResult): boolean * @param records - The records of the iteration that is being configured * @param lastStates - The results retained by previous iterations of the graph * @param verifiedStateHashByOperation - The state hash at which the retained result of each operation is verified + * @param retainedResultStatuses - The statuses of retained results that running the operation again can verify */ export function enableUnverifiedRetainedOperations( records: ReadonlyMap, lastStates: ReadonlyMap, - verifiedStateHashByOperation: ReadonlyMap + verifiedStateHashByOperation: ReadonlyMap, + retainedResultStatuses: ReadonlySet = SUCCESS_STATUSES ): void { // Whether the result of each operation will still be unverified at the end of this iteration. const remainsUnverifiedByOperation: Map = new Map(); @@ -79,7 +81,7 @@ export function enableUnverifiedRetainedOperations( if ( operation.enabled === true && lastState && - SUCCESS_STATUSES.has(lastState.status) && + retainedResultStatuses.has(lastState.status) && lastState.getStateHash() === stateHash ) { record.enabled = true; diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts index 4581d7c702..3694a67830 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts @@ -58,7 +58,7 @@ import { OperationGraph } from '../OperationGraph'; import { Operation } from '../Operation'; import { OperationStatus } from '../OperationStatus'; import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; -import type { IExecutionResult } from '../IOperationExecutionResult'; +import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; import type { OperationExecutionRecord } from '../OperationExecutionRecord'; const mockPhase: IPhase = { @@ -74,7 +74,7 @@ const mockPhase: IPhase = { class CacheableMockRunner implements IOperationRunner { public readonly reportTiming: boolean = true; public readonly silent: boolean = false; - public readonly cacheable: boolean = true; + public cacheable: boolean = true; public readonly warningsAreAllowed: boolean = false; public readonly isNoOp: boolean = false; public readonly name: string; @@ -102,6 +102,10 @@ interface ITestGraph { executions: string[]; cacheWrites: string[]; cacheRestores: string[]; + /** + * The operations that the emulated change detection plugin checked, if enabled by `upToDate`. + */ + checks: string[]; executeAsync(): Promise; } @@ -111,6 +115,11 @@ interface ITestGraphOptions { */ dependencies?: Record; cacheWriteEnabled?: boolean; + /** + * If set, emulates a plugin with its own change detection (e.g. by tracing the files that each operation reads), + * which reports a selected operation as skipped if its name is in this set, because its outputs are up to date. + */ + upToDate?: ReadonlySet; } /** @@ -118,8 +127,9 @@ interface ITestGraphOptions { * The mock build cache stores an entry per operation and state hash, and restores it if it exists. */ async function createTestGraphAsync(names: string[], options: ITestGraphOptions = {}): Promise { - const { dependencies, cacheWriteEnabled = true } = options; + const { dependencies, cacheWriteEnabled = true, upToDate } = options; const executions: string[] = []; + const checks: string[] = []; const cacheWrites: string[] = []; const cacheRestores: string[] = []; const cacheEntries: Set = new Set(); @@ -201,6 +211,32 @@ async function createTestGraphAsync(names: string[], options: ITestGraphOptions isIncrementalBuildAllowed: true, projectConfigurations } as unknown as IOperationGraphContext); + if (upToDate) { + graph.hooks.beforeExecuteIterationAsync.tap('TestChangeDetectionPlugin', (records) => { + for (const operation of records.keys()) { + (operation.runner as CacheableMockRunner).cacheable = true; + } + }); + graph.hooks.beforeExecuteOperationAsync.tapPromise( + // Before the build cache is read + { name: 'TestChangeDetectionPlugin', stage: -200 }, + async ( + record: IOperationRunnerContext & IOperationExecutionResult + ): Promise => { + if (record.silent) { + return; + } + const { name } = record.operation; + checks.push(name); + if (!upToDate.has(name)) { + return; + } + // The build cache does not handle operations that another plugin skipped. + (record.operation.runner as CacheableMockRunner).cacheable = false; + return OperationStatus.Skipped; + } + ); + } const inputsSnapshot: IInputsSnapshot = { hashes: new Map(), @@ -217,10 +253,12 @@ async function createTestGraphAsync(names: string[], options: ITestGraphOptions executions, cacheWrites, cacheRestores, + checks, executeAsync: async () => { executions.length = 0; cacheWrites.length = 0; cacheRestores.length = 0; + checks.length = 0; return await graph.executeAsync({ inputsSnapshot }); } }; @@ -398,6 +436,34 @@ describe(`${CacheableOperationPlugin.name} retained results`, () => { expect(testGraph.executions).toEqual([]); }); + it('does not check results that a plugin found up to date again while their state hashes are unchanged', async () => { + const upToDate: Set = new Set(['lib', 'tool', 'app']); + const testGraph: ITestGraph = await createTestGraphAsync(['lib', 'tool', 'app'], { upToDate }); + + // The outputs were built before this graph was created. + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['lib', 'tool', 'app']); + expect(testGraph.executions).toEqual([]); + + // Checking a skipped result again cannot make it trusted, so it is not re-enabled. + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.checks).toEqual([]); + + // Edit "app": its dependencies are not trusted, so its cache entry is not written. + testGraph.localHashes.set('app', 'app-v2'); + upToDate.delete('app'); + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['app']); + expect(testGraph.executions).toEqual(['app']); + expect(testGraph.cacheWrites).toEqual([]); + + const secondHotResult: IExecutionResult = await testGraph.executeAsync(); + expect(secondHotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.checks).toEqual([]); + expect(testGraph.executions).toEqual([]); + }); + it('does not re-enable operations that another plugin disabled', async () => { const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); // Like a plugin that performs the work itself. This tap runs after PhasedOperationPlugin's. diff --git a/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts index d748569f95..da2f64382e 100644 --- a/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/PhasedOperationPluginRetainedResults.test.ts @@ -82,6 +82,10 @@ interface ITestGraph { operations: Map; localHashes: Map; executions: string[]; + /** + * The operations that the emulated change detection plugin checked, if enabled by `upToDate`. + */ + checks: string[]; executeAsync(): Promise; } @@ -94,6 +98,11 @@ interface ITestGraphOptions { * If set, the incremental state files of the legacy skip detection are stored in this folder. */ legacySkipFolder?: string; + /** + * If set, emulates a plugin with its own change detection (e.g. by tracing the files that each operation reads), + * which reports a selected operation as skipped if its name is in this set, because its outputs are up to date. + */ + upToDate?: ReadonlySet; } /** @@ -103,8 +112,9 @@ async function createTestGraphAsync( dependencies: Record, options: ITestGraphOptions = {} ): Promise { - const { noOps, legacySkipFolder } = options; + const { noOps, legacySkipFolder, upToDate } = options; const executions: string[] = []; + const checks: string[] = []; const localHashes: Map = new Map(); const operations: Map = new Map(); @@ -153,6 +163,21 @@ async function createTestGraphAsync( isIncrementalBuildAllowed: true, projectConfigurations: new Map() } as unknown as IOperationGraphContext); + if (upToDate) { + graph.hooks.beforeExecuteOperationAsync.tapPromise( + { name: 'TestChangeDetectionPlugin', stage: -200 }, + async ( + record: IOperationRunnerContext & IOperationExecutionResult + ): Promise => { + if (record.silent) { + return; + } + const { name } = record.operation; + checks.push(name); + return upToDate.has(name) ? OperationStatus.Skipped : undefined; + } + ); + } const inputsSnapshot: IInputsSnapshot = { hashes: new Map(), @@ -168,13 +193,27 @@ async function createTestGraphAsync( operations, localHashes, executions, + checks, executeAsync: async () => { executions.length = 0; + checks.length = 0; return await graph.executeAsync({ inputsSnapshot }); } }; } +/** + * Returns the status of each operation in the result of an iteration, or "silent" if it was not selected. + */ +function getStatuses(testGraph: ITestGraph, result: IExecutionResult): Record { + const statuses: Record = {}; + for (const [name, operation] of testGraph.operations) { + const record: IOperationExecutionResult = result.operationResults.get(operation)!; + statuses[name] = record.silent ? 'silent' : record.status; + } + return statuses; +} + // How results retained by earlier iterations of a long-lived graph (e.g. the Rush daemon) are verified, // whether or not the build cache is in use. describe(`${PhasedOperationPlugin.name} retained results`, () => { @@ -374,6 +413,87 @@ describe(`${PhasedOperationPlugin.name} retained results`, () => { expect(testGraph.executions).toEqual([]); }); + it('reuses the result of a selected operation that a plugin found up to date while its state hash is unchanged', async () => { + const upToDate: Set = new Set(['a', 'b']); + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { upToDate }); + + // The outputs were built before this graph was created. + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['a', 'b']); + expect(testGraph.executions).toEqual([]); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.checks).toEqual([]); + + // Edit "b" + testGraph.localHashes.set('b', 'b-v2'); + upToDate.delete('b'); + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['b']); + expect(testGraph.executions).toEqual(['b']); + + const secondHotResult: IExecutionResult = await testGraph.executeAsync(); + expect(secondHotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.checks).toEqual([]); + expect(testGraph.executions).toEqual([]); + }); + + it('checks a result that a plugin found up to date against outputs of a dependency that were not current again', async () => { + const upToDate: Set = new Set(['a', 'b']); + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { upToDate }); + const a: Operation = testGraph.operations.get('a')!; + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['a', 'b']); + + // --only b, after editing "a": "b" is found up to date with the outputs of the previous "a". + testGraph.localHashes.set('a', 'a-v2'); + upToDate.delete('a'); + a.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['b']); + expect(testGraph.executions).toEqual([]); + + // --to b: "b" has the same state hash as its retained result, but "a" is rebuilt, which changes the inputs of "b". + a.enabled = true; + upToDate.delete('b'); + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['a', 'b']); + expect(testGraph.executions).toEqual(['a', 'b']); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.checks).toEqual([]); + }); + + it('checks a result that a plugin found up to date again if it was marked unverifiable, and the results checked against it', async () => { + const upToDate: Set = new Set(['a', 'b']); + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { upToDate }); + const a: Operation = testGraph.operations.get('a')!; + let isMarkingA: boolean = true; + // Like CacheableOperationPlugin when input files of "a" changed during the iteration + testGraph.graph.hooks.afterExecuteOperationAsync.tap( + 'TestPlugin', + (record: IOperationExecutionResult) => { + if (isMarkingA && record.operation === a) { + markResultUnverifiable(record); + } + } + ); + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['a', 'b']); + + // The same state hashes + isMarkingA = false; + await testGraph.executeAsync(); + expect(testGraph.checks).toEqual(['a', 'b']); + expect(testGraph.executions).toEqual([]); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.checks).toEqual([]); + }); + describe('with legacy skip detection', () => { let legacySkipFolder: string; @@ -429,17 +549,41 @@ describe(`${PhasedOperationPlugin.name} retained results`, () => { // Build once in another process, then start a long-lived graph. await (await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder })).executeAsync(); const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder }); - await testGraph.executeAsync(); - expect(testGraph.executions).toEqual([]); + const a: Operation = testGraph.operations.get('a')!; - // --to b, after editing "b": "a" is skipped, so it is not verified in this graph. + // --only b, after editing "b": "a" is not selected, so it is not verified in this graph. testGraph.localHashes.set('b', 'b-v2'); + a.enabled = false; await testGraph.executeAsync(); expect(testGraph.executions).toEqual(['b']); - // --to b: the legacy skip detection still skips "b". + // --to b: the legacy skip detection skips "a", and still skips "b". + a.enabled = true; await testGraph.executeAsync(); expect(testGraph.executions).toEqual([]); + + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + }); + + it('reuses results that the legacy skip detection found up to date while their state hashes are unchanged', async () => { + // Build once in another process, then start a long-lived graph. + await (await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder })).executeAsync(); + const testGraph: ITestGraph = await createTestGraphAsync({ a: [], b: ['a'] }, { legacySkipFolder }); + const result: IExecutionResult = await testGraph.executeAsync(); + expect(getStatuses(testGraph, result)).toEqual({ a: 'SKIPPED', b: 'SKIPPED' }); + expect(testGraph.executions).toEqual([]); + + // No iteration is scheduled. + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + + // Edit "b": "a" is not checked again. + testGraph.localHashes.set('b', 'b-v2'); + const changedResult: IExecutionResult = await testGraph.executeAsync(); + expect(getStatuses(testGraph, changedResult)).toEqual({ a: 'silent', b: 'SUCCESS' }); + expect(testGraph.executions).toEqual(['b']); }); }); }); From d2dc4375142c1bbd79a36e1a260703442e501b23 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 22:04:59 +0000 Subject: [PATCH 039/265] [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks (and 7 more) Swarm integration step 24; original commit 31a8db3e65 (merge of swarm/r01-t85-x106b at 4835a8700e). Scope: tasks 85 and 106. Brings task 106 (guarded :incremental builds in the daemon) and task 84 at de9a503a50 through r01's 85x106 resolution. This is step 1 of the path in board 2046 and board 2161; step 2 brings task 106's test-only tip b767c64dac. Merging b767c64dac straight onto task 85 conflicts in CacheableOperationPluginRetainedResults.test.ts. Gate: ch01 GATE OK board 2161 (tree 8250469c70) Commits folded into this step (8): - c7400b3f9c [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks - e07159a38f [rush-lib] Let the daemon run :incremental scripts when a guard allows it - 90a76234e5 [rush-lib] Register RUSH_DAEMON_INCREMENTAL_BUILDS as a known environment variable - b1647a952a [rush-lib] Test that --only consumers of a retained incremental result skip cache writes - aed31544c2 [rush-lib] Test that a terminated command drops its incremental base - 69e33c03a8 [rush-lib] Format the incremental build changes with Prettier - c3afda3587 [rush-lib] Test that a run whose inputs changed while it ran leaves no incremental base - de9a503a50 [rush-lib] Check dependencies reached through same-project phases in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 1 + .../rushd-incremental-builds_2026-09-28.json | 11 + .../rushd-iteration-hook-env_2026-09-28.json | 11 + .../rushd-incremental-builds_2026-09-28.json | 11 + .../rushd-incremental-builds_2026-09-28.json | 11 + common/reviews/api/rush-lib.api.md | 2 + docs/rush/dogfooding-rush-daemon.md | 8 + docs/rush/environment-variables.md | 1 + .../src/WorkspaceEngineComponentFactory.ts | 2 + .../ProductionDaemonRequestResolver.test.ts | 67 +- .../rush-lib/src/api/DaemonConfiguration.ts | 18 +- .../src/api/EnvironmentConfiguration.ts | 3 + .../src/api/test/DaemonConfiguration.test.ts | 16 +- .../api/test/EnvironmentConfiguration.test.ts | 8 + .../cli/scriptActions/PhasedScriptAction.ts | 11 + .../operations/CacheableOperationPlugin.ts | 63 +- .../IncrementalExecutionGuardPlugin.ts | 573 +++++++++++++++ .../operations/IncrementalExecutionState.ts | 83 +++ .../src/logic/operations/LegacySkipPlugin.ts | 7 + .../src/logic/operations/OperationGraph.ts | 3 +- .../operations/OperationOutputManifest.ts | 218 ++++++ .../PhasedCommandEngineExecution.ts | 3 +- .../logic/operations/ShellOperationRunner.ts | 305 +++++--- .../operations/ShellOperationRunnerPlugin.ts | 29 +- ...ableOperationPluginRetainedResults.test.ts | 114 ++- .../IncrementalExecutionGuardPlugin.test.ts | 691 ++++++++++++++++++ ...OperationGraphOperationEnvironment.test.ts | 29 + .../test/OperationOutputManifest.test.ts | 125 ++++ ...ellOperationRunnerIncrementalGuard.test.ts | 213 ++++++ .../test/ShellOperationRunnerPlugin.test.ts | 116 +++ .../rush-lib/src/schemas/rush.schema.json | 5 + 31 files changed, 2622 insertions(+), 136 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-incremental-builds_2026-09-28.json create mode 100644 common/changes/@microsoft/rush/rushd-iteration-hook-env_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-cli-client/rushd-incremental-builds_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-incremental-builds_2026-09-28.json create mode 100644 libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts create mode 100644 libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts create mode 100644 libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerIncrementalGuard.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 3a2bd0f840..676cae2822 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -281,6 +281,7 @@ keys and unknown `RUSH_DAEMON*` variables fail validation. | `queueTimeoutSeconds` | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | 30 | Admission wait limit. Time behind another request's graph load (up to 10 times the limit) and the request's own work do not count. The default does not limit waiting behind a running compatible build, or, for a daemon restart, behind requests that were already running (while no `rushx` script is running); an explicit value does | | `watch` | `RUSH_DAEMON_WATCH` | false | Persistent host observation of requested warm projects; false keeps root/config guards only. Never schedules builds | | `usePersistentIpcRunners` | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | false | Enables explicit per-operation `daemonIpc` Node launchers for unsharded incremental daemon builds | +| `incrementalBuilds` | `RUSH_DAEMON_INCREMENTAL_BUILDS` | true | Runs an operation's `:incremental` script instead of its initial script when only files it builds were edited since its last successful run in the daemon and its output folders are unchanged. Additions, deletions, renames, configuration, tool, environment and command-line changes, bundled outputs, cache restores and native Rush commands run the initial script. Incremental results are never written to the build cache | | `warmIdleTimeoutSeconds` | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | 300 | Idle runner, project-watcher and retained-result eviction | | `warmMemoryBudgetMB` | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | 512 | Best-effort sampled RSS budget in MiB, not a hard ceiling. Compared against whole-daemon RSS plus measured child RSS, so keep it above the daemon baseline (~130-190 MiB) | | `warmSetMaxProjects` | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | 20 | Best-effort limit on projects holding warm resources (active runners, watchers); retained results of resource-free projects do not count. Never trims requested execution | diff --git a/common/changes/@microsoft/rush/rushd-incremental-builds_2026-09-28.json b/common/changes/@microsoft/rush/rushd-incremental-builds_2026-09-28.json new file mode 100644 index 0000000000..fa138c064c --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-incremental-builds_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Let the Rush daemon run an operation's `:incremental` script outside watch mode, on top of the outputs of its last successful run in the daemon, when only files that the operation builds were edited since then and its output folders are unchanged. Additions, deletions and renames of input files, changes to configuration files, build tools, dependsOnEnvVars values or the command line, bundled outputs, build cache restores and native Rush commands make it run the initial script, and the operation log says why. Results of the incremental script, and of operations built against them, are never written to the build cache. Set `daemon.incrementalBuilds` to false in rush.json, or `RUSH_DAEMON_INCREMENTAL_BUILDS=0`, to always run the initial script.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@microsoft/rush/rushd-iteration-hook-env_2026-09-28.json b/common/changes/@microsoft/rush/rushd-iteration-hook-env_2026-09-28.json new file mode 100644 index 0000000000..965ac454d4 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-iteration-hook-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Pass `getOperationEnvironment` to the `beforeExecuteIterationAsync` and `afterExecuteIterationAsync` hooks of an operation graph, so that they receive the same iteration options as `configureIteration`.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/rushd-incremental-builds_2026-09-28.json b/common/changes/@rushstack/rush-cli-client/rushd-incremental-builds_2026-09-28.json new file mode 100644 index 0000000000..55bdda953b --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/rushd-incremental-builds_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Document the `incrementalBuilds` daemon setting and `RUSH_DAEMON_INCREMENTAL_BUILDS`.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-incremental-builds_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-incremental-builds_2026-09-28.json new file mode 100644 index 0000000000..6df532751c --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-incremental-builds_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Test that daemon builds run an operation's `:incremental` script after an edit of a file that it builds, and never write its result to the build cache.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 9e61e8f19d..55724ef1d0 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -321,6 +321,7 @@ export const EnvironmentVariableNames: { readonly RUSH_DAEMON_AUTO_START: "RUSH_DAEMON_AUTO_START"; readonly RUSH_DAEMON_WATCH: "RUSH_DAEMON_WATCH"; readonly RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: "RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS"; + readonly RUSH_DAEMON_INCREMENTAL_BUILDS: "RUSH_DAEMON_INCREMENTAL_BUILDS"; readonly RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: "RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS"; readonly RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS: "RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS"; readonly RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: "RUSH_DAEMON_WARM_MEMORY_BUDGET_MB"; @@ -528,6 +529,7 @@ export interface IDaemonConfigurationJson { readonly compatiblePlugins?: ReadonlyArray; readonly enabled?: boolean; readonly idleTimeoutSeconds?: number; + readonly incrementalBuilds?: boolean; readonly queueTimeoutSeconds?: number; readonly usePersistentIpcRunners?: boolean; readonly warmIdleTimeoutSeconds?: number; diff --git a/docs/rush/dogfooding-rush-daemon.md b/docs/rush/dogfooding-rush-daemon.md index 77a9fb7106..ef9af40482 100644 --- a/docs/rush/dogfooding-rush-daemon.md +++ b/docs/rush/dogfooding-rush-daemon.md @@ -191,6 +191,14 @@ Remove the snapshot with `rm -rf common/temp/rush-daemon-dogfood` (or `rush purg - **No persistent Heft or TypeScript workers.** Each operation still starts its Heft process; the daemon saves Rush startup and graph construction, not compilation. The `usePersistentIpcRunners`/`daemonIpc` mode requires a bundled, self-contained worker entry point, and Heft is not packaged that way. +- **`:incremental` scripts.** With `daemon.incrementalBuilds` (on by default), an operation whose project + defines a `_phase::incremental` script runs it instead of the initial script when only files that the + operation builds were edited since its last successful run in the daemon. The operation log then says + `Invoking (incremental): ...`. An added, deleted or renamed input, a configuration, tool, environment or + command-line change, a change to its output folders, bundled outputs, a cache restore or a native `rush` + command make it run the initial script, and the log says why (`Not using the incremental command because + ...`). Incremental results are never written to the build cache, and neither are the results of operations + built against them. Set `RUSH_DAEMON_INCREMENTAL_BUILDS=0` to always run the initial script. - **Plugins.** This repository's only configured plugin, `@rushstack/rush-published-versions-json-plugin`, is associated only with `record-published-versions` and is inert for builds. A plugin without `associatedCommands`, a plugin associated with `build` or `rebuild`, or a plugin command-line that defines diff --git a/docs/rush/environment-variables.md b/docs/rush/environment-variables.md index 590a0932a3..09e141ae94 100644 --- a/docs/rush/environment-variables.md +++ b/docs/rush/environment-variables.md @@ -27,6 +27,7 @@ variables and invalid values are errors, not ignored settings. | `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS` | `30` | Maximum request admission wait. Nonnegative seconds, at most 2147483.647; converted to whole milliseconds by rounding down. Time behind another request's load of the workspace graph (up to 10 times this value) does not count. Per-invocation `--wait-timeout` or `--no-wait` takes precedence. Overrides `queueTimeoutSeconds`. | | `RUSH_DAEMON_WATCH` | `0` | Observe requested warm projects between requests. Never schedules builds and does not enable `--watch` mode. Root/config guards and request-time input reconciliation remain active when disabled. Overrides `watch`. | | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | `0` | Enable explicitly configured `operationSettings[].daemonIpc` Node workers for supported incremental daemon builds. Does not convert arbitrary shell scripts into persistent workers. Overrides `usePersistentIpcRunners`. | +| `RUSH_DAEMON_INCREMENTAL_BUILDS` | `1` | Let daemon builds run an operation's `:incremental` script on top of the outputs of its last successful run in the daemon, when only files that it builds were edited and its output folders are unchanged. Otherwise the initial script runs, as it does for native Rush. `0` always runs the initial script. Incremental results are never written to the build cache. Overrides `incrementalBuilds`. | | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | `300` | Idle expiration for retained runners and project watchers, together with those projects' results. Results of resource-free (shell/null) projects do not expire. Positive seconds, at most 2147483.647. Overrides `warmIdleTimeoutSeconds`. | | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | `512` | Best-effort sampled RSS budget in MiB. Positive number, at most 9007199254740991. Not a hard process-tree memory ceiling; active/protected work and results of resource-free projects are exempt. Overrides `warmMemoryBudgetMB`. | | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | `20` | Best-effort retained-project limit. Positive safe integer, at most 9007199254740991; never trims the requested execution set. Overrides `warmSetMaxProjects`. | diff --git a/libraries/rush-daemon/src/WorkspaceEngineComponentFactory.ts b/libraries/rush-daemon/src/WorkspaceEngineComponentFactory.ts index 1f2686cacd..9b34180ea6 100644 --- a/libraries/rush-daemon/src/WorkspaceEngineComponentFactory.ts +++ b/libraries/rush-daemon/src/WorkspaceEngineComponentFactory.ts @@ -23,6 +23,8 @@ import type { WorkspaceInvalidationTracker } from './WorkspaceInvalidationTracker'; +// Rush's incremental execution guard keeps the base of an operation that is invalidated with this reason, because it +// compares the operation's inputs itself (INPUTS_CHANGED_INVALIDATION_REASON in rush-lib). Keep the two in sync. const INVALIDATION_REASON: 'workspace-inputs-changed' = 'workspace-inputs-changed'; /** diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index b2d718df00..03381a9d62 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -77,8 +77,10 @@ interface IFixtureOptions { readonly getSuccessorLaunchAsync?: GetWorkspaceSuccessorLaunchAsync; readonly onSessionCreated?: (session: WorkspaceSession) => void; readonly resolver?: IDaemonRequestResolver; - /** Adds the watch-only `_phase:compile:incremental` script, which passes `--incremental` to build.cjs. */ + /** Adds the `_phase:compile:incremental` script, which passes `--incremental` to build.cjs. */ readonly incrementalScript?: boolean; + /** Sets `daemon.incrementalBuilds` in rush.json. */ + readonly incrementalBuilds?: boolean; /** Uses PNPM, which installs a dependency file (shrinkwrap-deps.json) that change detection hashes per project. */ readonly pnpm?: boolean; } @@ -141,7 +143,10 @@ async function createFixtureAsync( rushVersion: RUSH_VERSION, ...(options.pnpm ? { pnpmVersion: '9.15.9' } : { npmVersion: '10.0.0' }), // Retention assertions must not depend on the surrounding Jest worker's accumulated RSS. - daemon: { warmMemoryBudgetMB: 100_000 }, + daemon: { + warmMemoryBudgetMB: 100_000, + ...(options.incrementalBuilds === undefined ? {} : { incrementalBuilds: options.incrementalBuilds }) + }, projectFolderMinDepth: 2, projectFolderMaxDepth: 2, projects: ['a', 'b', 'c'].map((name) => ({ @@ -220,7 +225,7 @@ async function createFixtureAsync( const fs = require('node:fs'); const path = require('node:path'); const name = require('./package.json').name; -const input = fs.readFileSync('input.txt', 'utf8'); +const input = fs.readFileSync(fs.existsSync('src/input.txt') ? 'src/input.txt' : 'input.txt', 'utf8'); (async () => { const gateFile = path.resolve('../../common/temp/gate-' + name + '.json'); if (fs.existsSync(gateFile)) { @@ -1507,8 +1512,11 @@ process.exit(23); } }); - it('runs the initial script for every warm build request, as native rush build does, not the watch-only one', async () => { - const fixture: IFixture = await createFixtureAsync(true, 'direct', { incrementalScript: true }); + it('runs the initial script for every warm build request, as native rush build does, if incremental builds are off', async () => { + const fixture: IFixture = await createFixtureAsync(true, 'direct', { + incrementalScript: true, + incrementalBuilds: false + }); try { for (const [requestId, input] of [ ['initial-script-1', 'one'], @@ -1531,6 +1539,55 @@ process.exit(23); } }); + it('runs the incremental script for an edit of a built file, and never caches its result', async () => { + const fixture: IFixture = await createFixtureAsync(true, 'direct', { incrementalScript: true }); + const inputPath: string = path.join(fixture.repoRoot, 'projects/a/src/input.txt'); + const buildAsync = async (requestId: string, input: string, status: string): Promise => { + fs.mkdirSync(path.dirname(inputPath), { recursive: true }); + fs.writeFileSync(inputPath, input); + const exchange: ITerminalExchange = await runAsync(fixture, requestId, ['build', '--only', 'a']); + expect(exchange.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, operationResults: [{ operationId: 'a (compile)', status }] } + }); + return logText(exchange); + }; + try { + expect(await buildAsync('incremental-1', 'one', 'SUCCESS')).toContain( + 'Invoking (initial): node build.cjs' + ); + const incremental: string = await buildAsync('incremental-2', 'two', 'SUCCESS'); + expect(incremental).toContain('Invoking (incremental): node build.cjs --incremental'); + expect(incremental).toContain( + 'This operation ran its incremental command; not writing a build cache entry.' + ); + await buildAsync('incremental-3', 'three', 'SUCCESS'); + // The result of the incremental script was not cached, so it runs again. + await buildAsync('incremental-4', 'two', 'SUCCESS'); + // The result of the initial script was cached. + await buildAsync('incremental-5', 'one', 'FROM CACHE'); + // The incremental script never runs on top of outputs restored from the build cache. + expect(await buildAsync('incremental-6', 'three', 'SUCCESS')).toContain( + 'Not using the incremental command because its outputs were not built by a successful run of its own' + + ' command in this process.' + ); + fs.writeFileSync(path.join(fixture.repoRoot, 'projects/a/input.txt'), 'root'); + expect(await buildAsync('incremental-7', 'three', 'SUCCESS')).toContain( + 'Not using the incremental command because a configuration file changed ("projects/a/input.txt").' + ); + expect(runs(fixture)).toEqual([ + 'a:one:', + 'a:two:--incremental', + 'a:three:--incremental', + 'a:two:--incremental', + 'a:three:', + 'a:three:' + ]); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('re-runs only the operation whose declared outputs were deleted after a warm build', async () => { const fixture: IFixture = await createFixtureAsync(); try { diff --git a/libraries/rush-lib/src/api/DaemonConfiguration.ts b/libraries/rush-lib/src/api/DaemonConfiguration.ts index 09c17f999d..8edf596f7e 100644 --- a/libraries/rush-lib/src/api/DaemonConfiguration.ts +++ b/libraries/rush-lib/src/api/DaemonConfiguration.ts @@ -13,6 +13,13 @@ export interface IDaemonConfigurationJson { readonly watch?: boolean; /** Enables explicit operationSettings[].daemonIpc Node runners for daemon builds. Defaults to false. */ readonly usePersistentIpcRunners?: boolean; + /** + * Lets daemon builds run an operation's `:incremental` script, instead of its initial script, on top of the + * outputs of its last successful run in the daemon, when only files that it builds were edited since then and its + * output folders are unchanged. Otherwise the initial script runs, as it does for native Rush. Results of an + * incremental script are not written to the build cache. Defaults to true. + */ + readonly incrementalBuilds?: boolean; /** Maximum admission queue wait in seconds. Defaults to 30. */ readonly queueTimeoutSeconds?: number; /** @@ -45,6 +52,7 @@ const defaults: Required = { autoStart: true, watch: false, usePersistentIpcRunners: false, + incrementalBuilds: true, queueTimeoutSeconds: 30, warmIdleTimeoutSeconds: 300, warmMemoryBudgetMB: 512, @@ -61,6 +69,7 @@ export const daemonEnvironmentVariables: Readonly> ): boolean { diff --git a/libraries/rush-lib/src/api/EnvironmentConfiguration.ts b/libraries/rush-lib/src/api/EnvironmentConfiguration.ts index 00357ce7d2..926237dd35 100644 --- a/libraries/rush-lib/src/api/EnvironmentConfiguration.ts +++ b/libraries/rush-lib/src/api/EnvironmentConfiguration.ts @@ -276,6 +276,8 @@ export const EnvironmentVariableNames = { RUSH_DAEMON_WATCH: 'RUSH_DAEMON_WATCH', /** Enables explicitly configured persistent Node IPC operations in the daemon. */ RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: 'RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS', + /** Lets daemon builds run an operation's guarded `:incremental` script outside watch mode. */ + RUSH_DAEMON_INCREMENTAL_BUILDS: 'RUSH_DAEMON_INCREMENTAL_BUILDS', /** Overrides the request admission queue timeout. */ RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: 'RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS', /** Overrides idle eviction in an attached daemon warm set. */ @@ -694,6 +696,7 @@ export class EnvironmentConfiguration { case EnvironmentVariableNames.RUSH_DAEMON_AUTO_START: case EnvironmentVariableNames.RUSH_DAEMON_WATCH: case EnvironmentVariableNames.RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: + case EnvironmentVariableNames.RUSH_DAEMON_INCREMENTAL_BUILDS: case EnvironmentVariableNames.RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: case EnvironmentVariableNames.RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS: case EnvironmentVariableNames.RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: diff --git a/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts b/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts index fb16ad2fd5..af95f8278d 100644 --- a/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts @@ -11,6 +11,7 @@ describe('daemon configuration', () => { enabled: false, autoStart: true, usePersistentIpcRunners: false, + incrementalBuilds: true, idleTimeoutSeconds: 900 }); expect( @@ -35,7 +36,8 @@ describe('daemon configuration', () => { { RUSH_DAEMON_WARM_SET_MAX_PROJECTS: '1.5' }, { RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY: '' }, { RUSH_DAEMON_EXPERIMENTAL: 'yes' }, - { RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: 'yes' } + { RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: 'yes' }, + { RUSH_DAEMON_INCREMENTAL_BUILDS: 'off' } ])('rejects invalid overrides %j', (environment) => { expect(() => resolveDaemonConfiguration({}, environment)).toThrow(); }); @@ -53,6 +55,7 @@ describe('daemon configuration', () => { { warmSetMaxProjects: 0.5 }, { autoWarmByTelemetry: 1 }, { usePersistentIpcRunners: 'true' }, + { incrementalBuilds: 'false' }, { compatiblePlugins: 'rush-example-plugin' }, { compatiblePlugins: [''] }, { compatiblePlugins: [' rush-example-plugin'] }, @@ -68,6 +71,17 @@ describe('daemon configuration', () => { ).toThrow(); }); + it('runs :incremental scripts in the daemon unless the environment or configuration turns it off', () => { + expect(resolveDaemonConfiguration({ incrementalBuilds: false }, {}).incrementalBuilds).toBe(false); + expect( + resolveDaemonConfiguration({ incrementalBuilds: false }, { RUSH_DAEMON_INCREMENTAL_BUILDS: '1' }) + .incrementalBuilds + ).toBe(true); + expect(resolveDaemonConfiguration({}, { RUSH_DAEMON_INCREMENTAL_BUILDS: '0' }).incrementalBuilds).toBe( + false + ); + }); + it('resolves compatible plugin names from the environment, then configuration, then no plugins', () => { expect(resolveDaemonConfiguration({}, {}).compatiblePlugins).toEqual([]); const configured: string[] = ['rush-a-plugin', 'rush-b-plugin']; diff --git a/libraries/rush-lib/src/api/test/EnvironmentConfiguration.test.ts b/libraries/rush-lib/src/api/test/EnvironmentConfiguration.test.ts index 0dc982bb2e..ca75e513a1 100644 --- a/libraries/rush-lib/src/api/test/EnvironmentConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/EnvironmentConfiguration.test.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import * as path from 'node:path'; +import { daemonEnvironmentVariables } from '../DaemonConfiguration'; import { EnvironmentConfiguration } from '../EnvironmentConfiguration'; describe(EnvironmentConfiguration.name, () => { @@ -37,6 +38,13 @@ describe(EnvironmentConfiguration.name, () => { expect(process.env.RUSH_LOG_LEVEL).toBe(env.RUSH_LOG_LEVEL); }); + it('allows every daemon environment variable', () => { + for (const name of Object.values(daemonEnvironmentVariables)) { + process.env[name] = '1'; + } + expect(EnvironmentConfiguration.validate).not.toThrow(); + }); + it('does not allow unknown environment variables', () => { process.env['rush_foobar'] = 'asdf'; // eslint-disable-line dot-notation expect(EnvironmentConfiguration.validate).toThrow(); diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index 0e6b9ba4a1..da402638e6 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -660,6 +660,17 @@ export class PhasedScriptAction extends BaseScriptAction i ); new DaemonIpcOperationRunnerPlugin().apply(this.hooks); } + if ( + onEngine && + this.#isIncrementalBuildAllowed && + this.rushConfiguration.daemon.incrementalBuilds && + !cobuildConfiguration?.cobuildFeatureEnabled + ) { + const { IncrementalExecutionGuardPlugin } = await import( + /* webpackChunkName: 'IncrementalExecutionGuardPlugin' */ '../../logic/operations/IncrementalExecutionGuardPlugin' + ); + new IncrementalExecutionGuardPlugin().apply(this.hooks); + } if (isWatch && this.#noIPCParameter?.value === false) { new ( await import( diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index 5f96073993..0a5de47256 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -51,6 +51,7 @@ import type { BuildCacheConfiguration } from '../../api/BuildCacheConfiguration' import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; import type { OperationExecutionRecord } from './OperationExecutionRecord'; import { enableUnverifiedRetainedOperations, markResultUnverifiable } from './RetainedResultVerification'; +import { wasExecutedIncrementally } from './IncrementalExecutionState'; const PLUGIN_NAME: 'CacheablePhasedOperationPlugin' = 'CacheablePhasedOperationPlugin'; const PERIODIC_CALLBACK_INTERVAL_IN_SECONDS: number = 10; @@ -83,6 +84,12 @@ export interface IOperationBuildCacheContext { cacheRestored: boolean; isCacheReadAttempted: boolean; + // True if the outputs of this operation were produced by its incremental command in this iteration (see + // IncrementalExecutionGuardPlugin), or if it executed against outputs of a dependency that were. Such outputs + // can differ from those of the initial command, so neither they nor the outputs of their consumers are written + // to the build cache. Unlike a blocked cache write, this does not stop a long-lived graph from trusting them. + isIncrementalResult: boolean; + // The on-disk state of the tracked input files whose hashes produced the cache key, captured right after // the iteration's inputs snapshot. Used to refuse cache writes, and to keep a long-lived graph from skipping // the operation later, if the inputs changed while the snapshot was being taken or while the operation was @@ -192,10 +199,14 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { // The state hash at which each operation last completed successfully in an iteration of this graph // in which cache writes were allowed for it (i.e. no dependency had an unknown state). const trustedStateHashByOperation: Map = new Map(); + // The trusted state hash of each operation whose trusted outputs are an incremental result, which must not + // be written to the build cache, nor may the outputs of its consumers. + const incrementalStateHashByOperation: Map = new Map(); graph.hooks.beforeDeleteResults.tap(PLUGIN_NAME, (operations: ReadonlySet) => { for (const operation of operations) { trustedStateHashByOperation.delete(operation); + incrementalStateHashByOperation.delete(operation); } // Terminals and cobuild callbacks can retain the entire completed iteration, including other // projects' records. All of this scratch state is rebuilt by beforeExecuteIterationAsync. @@ -300,6 +311,7 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { }), cacheRestored: false, isCacheReadAttempted: false, + isIncrementalResult: false, inputFilesState, inputFileHashes: inputFilesState ? fileHashes : undefined }; @@ -576,8 +588,21 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { return; } - const { cobuildLock, operationBuildCache, isCacheWriteAllowed, buildCacheTerminal, cacheRestored } = - buildCacheContext; + const ranIncrementalCommand: boolean = + !buildCacheContext.cacheRestored && wasExecutedIncrementally(record); + if (ranIncrementalCommand) { + buildCacheContext.isIncrementalResult = true; + } + + const { + cobuildLock, + operationBuildCache, + isCacheWriteAllowed: isCacheWriteAllowedForOperation, + isIncrementalResult, + buildCacheTerminal, + cacheRestored + } = buildCacheContext; + const isCacheWriteAllowed: boolean = isCacheWriteAllowedForOperation && !isIncrementalResult; try { if (!cacheRestored) { @@ -607,6 +632,12 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { throw new InternalError(`Build Cache Terminal is not created`); } + if (ranIncrementalCommand && isCacheWriteAllowedForOperation) { + buildCacheTerminal.writeLine( + 'This operation ran its incremental command; not writing a build cache entry.' + ); + } + let setCompletedStatePromiseFunction: (() => Promise | undefined) | undefined; let setCacheEntryPromise: (() => Promise | undefined) | undefined; if (cobuildLock && isCacheWriteAllowed) { @@ -712,6 +743,8 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { this.#buildCacheContextByOperation.get(operation); // Status changes to direct dependents let blockCacheWrite: boolean = !buildCacheContext?.isCacheWriteAllowed; + // Whether the outputs that consumers execute against are an incremental result + let isIncrementalResult: boolean = false; switch (record.status) { case OperationStatus.Skipped: { @@ -724,30 +757,50 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { if (blockCacheWrite || trustedStateHashByOperation.get(operation) !== record.getStateHash()) { blockCacheWrite = true; trustedStateHashByOperation.delete(operation); + incrementalStateHashByOperation.delete(operation); + } else { + isIncrementalResult = + incrementalStateHashByOperation.get(operation) === record.getStateHash(); } break; } default: { + // Outputs restored from the build cache are those of the initial command. + isIncrementalResult = + record.status !== OperationStatus.FromCache && + (!!buildCacheContext?.isIncrementalResult || wasExecutedIncrementally(record)); if (!blockCacheWrite && buildCacheContext && SUCCESS_STATUSES.has(record.status)) { // The outputs of this operation were produced (or restored) in an iteration where cache // writes were allowed, so they can be trusted by consumers in later iterations as long as - // the state hash is unchanged. + // the state hash is unchanged. An incremental result is trusted too, but it still blocks the + // cache writes of consumers. trustedStateHashByOperation.set(operation, record.getStateHash()); + if (isIncrementalResult) { + incrementalStateHashByOperation.set(operation, record.getStateHash()); + } else { + incrementalStateHashByOperation.delete(operation); + } } else { trustedStateHashByOperation.delete(operation); + incrementalStateHashByOperation.delete(operation); } break; } } // Apply status changes to direct dependents - if (blockCacheWrite) { + if (blockCacheWrite || isIncrementalResult) { for (const consumer of operation.consumers) { const consumerBuildCacheContext: IOperationBuildCacheContext | undefined = this.#getBuildCacheContextByOperation(consumer); if (consumerBuildCacheContext) { - consumerBuildCacheContext.isCacheWriteAllowed = false; + if (blockCacheWrite) { + consumerBuildCacheContext.isCacheWriteAllowed = false; + } + if (isIncrementalResult) { + consumerBuildCacheContext.isIncrementalResult = true; + } } } } diff --git a/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts b/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts new file mode 100644 index 0000000000..f643f8b135 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts @@ -0,0 +1,573 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { createHash, type Hash } from 'node:crypto'; +import * as path from 'node:path'; + +import { FileSystem, InternalError, Path } from '@rushstack/node-core-library'; + +import type { RushConfigurationProject } from '../../api/RushConfigurationProject'; +import type { IInputsSnapshot } from '../incremental/InputsSnapshot'; +import type { + IOperationGraphContext, + IPhasedCommandPlugin, + PhasedCommandHooks +} from '../../pluginFramework/PhasedCommandHooks'; +import type { IOperationExecutionResult } from './IOperationExecutionResult'; +import type { IOperationGraph, IOperationGraphIterationOptions } from './IOperationGraph'; +import type { IOperationRunnerContext } from './IOperationRunner'; +import type { Operation } from './Operation'; +import { OperationStatus } from './OperationStatus'; +import { + getCommandExecution, + INPUTS_CHANGED_INVALIDATION_REASON, + NATIVE_COMMAND_INVALIDATION_REASON, + setIncrementalExecutionGuard, + type ICommandExecution, + type IIncrementalExecutionGuard +} from './IncrementalExecutionState'; +import { + describeOutputFileChanges, + readOperationOutputManifestAsync, + type IOperationOutputManifest +} from './OperationOutputManifest'; +import { isResultUnverifiable } from './RetainedResultVerification'; + +const PLUGIN_NAME: 'IncrementalExecutionGuardPlugin' = 'IncrementalExecutionGuardPlugin'; + +// Runs after the default-stage taps, e.g. CacheableOperationPlugin's input file checks, which can mark a result as +// unverifiable. +const RECORD_RESULT_STAGE: number = 1; + +const MAX_EXAMPLE_PATHS: number = 3; + +// Project files that configure a build rather than being built by it. +const CONFIG_INPUT_FILE_NAME_REGEXP: RegExp = + /^(?:package\.json|tsconfig[^/]*\.json|\.eslintrc[^/]*|eslint\.config\.[^/]+|\.eslintignore|\.babelrc[^/]*|babel\.config\.[^/]+|[^/]+\.config\.(?:[cm]?[jt]s|json)|\.npmrc|\.browserslistrc|\.gitignore)$/; +const CONFIG_INPUT_FOLDER_REGEXP: RegExp = /^(?:config|\.rush)\//; +// Dependencies that are part of the build toolchain, even when they are production dependencies. +const TOOLING_PACKAGE_NAME_REGEXP: RegExp = + /(?:^|[/-])(?:eslint-(?:config|plugin)|tsconfig|prettier-config|babel-(?:preset|plugin))(?:-|$)/; + +/** + * The inputs of an operation that decide whether it may run its incremental command, as of one inputs snapshot. + */ +export interface IIncrementalInputState { + /** + * The path of the operation's project folder relative to the root of the inputs snapshot, with a trailing `/`, + * or an empty string for a project at the root. + */ + readonly projectPrefix: string; + /** + * The configuration hash of the operation's runner, e.g. its command line. + */ + readonly configHash: string; + /** + * The values of the environment variables that the operation depends on. + */ + readonly environment: string; + /** + * The state hash of each dependency of the operation, including the dependencies that it has through operations + * of its own project, by operation name. + */ + readonly dependencyStateHashes: ReadonlyMap; + /** + * Covers the paths of the operation's input files. + */ + readonly inputPathsDigest: string; + /** + * Covers the paths and hashes of the operation's configuration input files. + */ + readonly configInputsDigest: string; + /** + * The operation's input files and their hashes, used to name changed files while the snapshot is in memory. + */ + readonly inputFiles: WeakRef>; +} + +/** + * Captures the inputs of an operation that decide whether it may run its incremental command. + * + * @param record - The operation's result or execution record in the iteration that uses `inputsSnapshot` + * @param inputsSnapshot - The inputs snapshot of the iteration + * @param environment - The environment that the operation runs with in the iteration + * @param records - The results or execution records of the iteration, by operation + */ +export function captureIncrementalInputState( + record: IOperationExecutionResult, + inputsSnapshot: IInputsSnapshot, + environment: Readonly>, + records: ReadonlyMap +): IIncrementalInputState { + const { operation } = record; + const { associatedProject: project, associatedPhase: phase, settings } = operation; + const { config } = record.getStateHashComponents(); + + const inputFiles: ReadonlyMap = inputsSnapshot.getTrackedFileHashesForOperation( + project, + phase.name + ); + const projectPrefix: string = getProjectPrefix(inputsSnapshot, project); + const inputPathsHash: Hash = createHash('sha1'); + const configInputsHash: Hash = createHash('sha1'); + for (const filePath of Array.from(inputFiles.keys()).sort()) { + inputPathsHash.update(`${filePath}\n`); + if (isConfigInput(filePath, projectPrefix)) { + configInputsHash.update(`${filePath}\0${inputFiles.get(filePath)}\n`); + } + } + + const dependencyStateHashes: Map = new Map(); + for (const [name, dependency] of getGuardedDependencies(operation)) { + const dependencyRecord: IOperationExecutionResult | undefined = records.get(dependency); + if (!dependencyRecord) { + throw new InternalError(`The iteration has no record of the operation "${name}".`); + } + dependencyStateHashes.set(name, dependencyRecord.getStateHash()); + } + + return { + projectPrefix, + configHash: config, + environment: (settings?.dependsOnEnvVars ?? []) + .map((name: string) => `${name}=${environment[name] || ''}`) + .join('\n'), + dependencyStateHashes, + inputPathsDigest: inputPathsHash.digest('hex'), + configInputsDigest: configInputsHash.digest('hex'), + inputFiles: new WeakRef(inputFiles) + }; +} + +/** + * Returns why an operation must not run its incremental command on top of the outputs of its last successful run, + * judging by how its inputs changed since that run, or `undefined` if the inputs allow it. + * + * @remarks + * The incremental command is only trusted to handle edits of the files that the operation builds. It may keep the + * outputs of deleted or renamed files, and it may not notice a change to the configuration or the tools of the + * build. The inputs therefore only allow it if these are unchanged: + * + * - the operation's command line and the environment variables that it depends on; + * - the set of its input files, so that no file was added, deleted or renamed; + * - its configuration input files: any file outside of the project folder (e.g. the shrinkwrap file or a global + * additional file), any file in the project's root folder, `config/` or `.rush/` folders, and configuration files + * such as `tsconfig*.json`, `.eslintrc*` or `*.config.js` anywhere in the project; + * - the set of its dependencies, and each dependency that belongs to another project and is not a production + * dependency (`dependencies`, `optionalDependencies` or `peerDependencies` in package.json), or that is a build + * tool such as a rig, a Heft plugin or a lint configuration. This includes the dependencies that the operation has + * through operations of its own project, e.g. through a phase without a script that only orders its build after + * the builds of other projects. + * + * @param operation - The operation + * @param lastState - The inputs of the operation's last successful run + * @param currentState - The inputs of the run that is about to start + */ +export function getIncrementalInputChangeReason( + operation: Operation, + lastState: IIncrementalInputState, + currentState: IIncrementalInputState +): string | undefined { + if (lastState.configHash !== currentState.configHash) { + return 'its command line changed'; + } + if (lastState.environment !== currentState.environment) { + return 'an environment variable that it depends on changed'; + } + if (lastState.inputPathsDigest !== currentState.inputPathsDigest) { + return `input files were added, deleted or renamed${describeInputFileChanges( + lastState, + currentState, + (lastHash: string | undefined, currentHash: string | undefined) => + (lastHash === undefined) !== (currentHash === undefined) + )}`; + } + if (lastState.configInputsDigest !== currentState.configInputsDigest) { + return `a configuration file changed${describeInputFileChanges( + lastState, + currentState, + (lastHash: string | undefined, currentHash: string | undefined, filePath: string) => + lastHash !== currentHash && isConfigInput(filePath, currentState.projectPrefix) + )}`; + } + + const { dependencyStateHashes: lastDependencies } = lastState; + const { dependencyStateHashes: currentDependencies } = currentState; + if (lastDependencies.size !== currentDependencies.size) { + return 'its dependencies changed'; + } + const dependencyByName: ReadonlyMap = getGuardedDependencies(operation); + for (const [name, stateHash] of currentDependencies) { + const lastStateHash: string | undefined = lastDependencies.get(name); + const dependency: Operation | undefined = dependencyByName.get(name); + if (lastStateHash === undefined || !dependency) { + return 'its dependencies changed'; + } + if (lastStateHash !== stateHash) { + const reason: string | undefined = getToolingDependencyReason(operation, dependency); + if (reason) { + return reason; + } + } + } + + return undefined; +} + +/** + * Lets operations of a long-lived host, such as the Rush daemon, run their `:incremental` command outside watch mode + * when doing so builds the same outputs as their initial command. + * + * @remarks + * An operation runs its incremental command only if all of these hold, and its initial command otherwise: + * + * 1. Its last result in this graph is a success of its own command, not a result restored from the build cache, a + * failure, or a run that was interrupted or whose input files changed while it ran. A native Rush command that + * ran in the workspace since then forgets every such result. + * 2. Its inputs changed as {@link getIncrementalInputChangeReason} allows. + * 3. Its declared output folders hold the same files and folders as at the end of that run, and none of the folders + * was recreated or had an entry added, removed or replaced since then. + * 4. Its outputs do not include bundles (JavaScript or CSS in a `dist` or `release` folder, or with a content hash in + * the name). A bundler can replace a chunk with a differently named one and leave the old one behind. + * + * When the incremental command succeeds, the operation's output files must be the files that it had before, or the + * initial command runs as well. Results of the incremental command are never written to the build cache (see + * `CacheableOperationPlugin`). + */ +export class IncrementalExecutionGuardPlugin implements IPhasedCommandPlugin { + public apply(hooks: PhasedCommandHooks): void { + hooks.onGraphCreatedAsync.tap(PLUGIN_NAME, (graph: IOperationGraph, context: IOperationGraphContext) => { + if (context.isIncrementalBuildAllowed && !context.isWatch) { + applyToGraph(graph); + } + }); + } +} + +interface IOutputState { + readonly signature: string; + readonly cleanOnlyReason: string | undefined; +} + +interface IIncrementalBase { + readonly inputs: IIncrementalInputState; + // Rejects if the output folders could not be read. + readonly outputsPromise: Promise; +} + +interface IRecordState { + readonly records: ReadonlyMap; + readonly inputsSnapshot: IInputsSnapshot; + readonly getOperationEnvironment: IOperationGraphIterationOptions['getOperationEnvironment']; + preRunOutputs?: IOperationOutputManifest; + verifiedOutputs?: IOutputState; +} + +function applyToGraph(graph: IOperationGraph): void { + const baseByOperation: Map = new Map(); + const cleanOnlyReasonByOperation: Map = new Map(); + const stateByRecord: WeakMap = new WeakMap(); + + graph.hooks.beforeExecuteIterationAsync.tap( + PLUGIN_NAME, + ( + records: ReadonlyMap, + iterationOptions: IOperationGraphIterationOptions + ): void => { + const { inputsSnapshot, getOperationEnvironment } = iterationOptions; + if (!inputsSnapshot) { + // Without a snapshot the inputs of later runs cannot be compared with those of this one. + baseByOperation.clear(); + return; + } + for (const record of records.values()) { + const recordState: IRecordState = { records, inputsSnapshot, getOperationEnvironment }; + stateByRecord.set(record, recordState); + const guard: IIncrementalExecutionGuard = { + getBlockReasonAsync: () => getBlockReasonAsync(record, recordState), + verifyIncrementalResultAsync: () => verifyIncrementalResultAsync(record, recordState) + }; + setIncrementalExecutionGuard(record, guard); + } + } + ); + + async function getBlockReasonAsync( + record: IOperationExecutionResult, + recordState: IRecordState + ): Promise { + const { operation } = record; + const cleanOnlyReason: string | undefined = cleanOnlyReasonByOperation.get(operation); + if (cleanOnlyReason) { + return cleanOnlyReason; + } + const base: IIncrementalBase | undefined = baseByOperation.get(operation); + if (!base) { + return 'its outputs were not built by a successful run of its own command in this process'; + } + + const inputChangeReason: string | undefined = getIncrementalInputChangeReason( + operation, + base.inputs, + captureIncrementalInputState( + record, + recordState.inputsSnapshot, + getEnvironment(record, recordState), + recordState.records + ) + ); + if (inputChangeReason) { + return inputChangeReason; + } + + const outputFolderNames: ReadonlyArray | undefined = operation.settings?.outputFolderNames; + if (!outputFolderNames?.length) { + return 'it declares no output folders'; + } + let baseOutputs: IOutputState; + try { + baseOutputs = await base.outputsPromise; + } catch (error) { + return `its output folders could not be read after its last run: ${error}`; + } + if (baseOutputs.cleanOnlyReason) { + cleanOnlyReasonByOperation.set(operation, baseOutputs.cleanOnlyReason); + return baseOutputs.cleanOnlyReason; + } + const outputs: IOperationOutputManifest = await readOperationOutputManifestAsync( + operation.associatedProject.projectFolder, + outputFolderNames + ); + if (outputs.signature !== baseOutputs.signature) { + return 'its output folders changed since its last successful run'; + } + // eslint-disable-next-line require-atomic-updates -- The runner of the execution record calls the guard sequentially. + recordState.preRunOutputs = outputs; + return undefined; + } + + async function verifyIncrementalResultAsync( + record: IOperationExecutionResult, + recordState: IRecordState + ): Promise { + const { operation } = record; + const { preRunOutputs } = recordState; + const outputFolderNames: ReadonlyArray | undefined = operation.settings?.outputFolderNames; + if (!preRunOutputs || !outputFolderNames) { + return 'the outputs of the incremental command could not be compared with the outputs before it'; + } + const outputs: IOperationOutputManifest = await readOperationOutputManifestAsync( + operation.associatedProject.projectFolder, + outputFolderNames + ); + const changes: string | undefined = describeOutputFileChanges(preRunOutputs.files, outputs.files); + if (changes) { + // Its outputs may be named after their content, so a later incremental run could leave stale files behind. + cleanOnlyReasonByOperation.set( + operation, + `its incremental command changed which output files it has in an earlier run: ${changes}` + ); + return `the incremental command changed which output files it has: ${changes}`; + } + if (outputs.cleanOnlyReason) { + cleanOnlyReasonByOperation.set(operation, outputs.cleanOnlyReason); + return outputs.cleanOnlyReason; + } + // eslint-disable-next-line require-atomic-updates -- The runner of the execution record calls the guard sequentially. + recordState.verifiedOutputs = { signature: outputs.signature, cleanOnlyReason: undefined }; + return undefined; + } + + graph.hooks.afterExecuteOperationAsync.tapPromise( + { name: PLUGIN_NAME, stage: RECORD_RESULT_STAGE }, + async (record: IOperationRunnerContext & IOperationExecutionResult): Promise => { + const { operation, status } = record; + const recordState: IRecordState | undefined = stateByRecord.get(record); + stateByRecord.delete(record); + + const execution: ICommandExecution | undefined = getCommandExecution(record); + if (!execution) { + switch (status) { + case OperationStatus.Skipped: + case OperationStatus.NoOp: + case OperationStatus.Blocked: + case OperationStatus.Aborted: + // No command ran, so the outputs are those of the last run. + return; + default: + // E.g. restored from the build cache, or executed by a runner that does not report its command. + baseByOperation.delete(operation); + return; + } + } + + const outputFolderNames: ReadonlyArray | undefined = operation.settings?.outputFolderNames; + if ( + !recordState || + !execution.hasIncrementalCommand || + !outputFolderNames?.length || + cleanOnlyReasonByOperation.has(operation) || + (status !== OperationStatus.Success && status !== OperationStatus.SuccessWithWarning) || + isResultUnverifiable(record) + ) { + baseByOperation.delete(operation); + return; + } + + const { verifiedOutputs } = recordState; + const outputsPromise: Promise = verifiedOutputs + ? Promise.resolve(verifiedOutputs) + : readOperationOutputManifestAsync(operation.associatedProject.projectFolder, outputFolderNames).then( + ({ signature, cleanOnlyReason }: IOperationOutputManifest) => ({ signature, cleanOnlyReason }) + ); + baseByOperation.set(operation, { + inputs: captureIncrementalInputState( + record, + recordState.inputsSnapshot, + getEnvironment(record, recordState), + recordState.records + ), + outputsPromise + }); + // Finish reading the outputs before anything else can change them, e.g. an operation that depends on this one. + // A failure is reported by the next check of this operation. + await outputsPromise.catch(() => undefined); + } + ); + + graph.hooks.onInvalidateOperations.tap( + PLUGIN_NAME, + (operations: Iterable, reason: string | undefined): void => { + if (reason === INPUTS_CHANGED_INVALIDATION_REASON) { + return; + } + if (reason === NATIVE_COMMAND_INVALIDATION_REASON) { + baseByOperation.clear(); + return; + } + for (const operation of operations) { + baseByOperation.delete(operation); + } + } + ); + + graph.hooks.beforeDeleteResults.tap(PLUGIN_NAME, (operations: ReadonlySet): void => { + for (const operation of operations) { + baseByOperation.delete(operation); + } + }); +} + +function getEnvironment( + record: IOperationExecutionResult, + { getOperationEnvironment }: IRecordState +): Readonly> { + return getOperationEnvironment?.(record.operation) ?? process.env; +} + +function getProjectPrefix(inputsSnapshot: IInputsSnapshot, project: RushConfigurationProject): string { + const relativePath: string = Path.convertToSlashes( + path.relative(inputsSnapshot.rootDirectory, project.projectFolder) + ); + return relativePath === '' ? '' : `${relativePath}/`; +} + +function isConfigInput(filePath: string, projectPrefix: string): boolean { + if (!filePath.startsWith(projectPrefix) || path.isAbsolute(filePath)) { + return true; + } + const projectRelativePath: string = filePath.slice(projectPrefix.length); + if (!projectRelativePath.includes('/') || CONFIG_INPUT_FOLDER_REGEXP.test(projectRelativePath)) { + return true; + } + return CONFIG_INPUT_FILE_NAME_REGEXP.test( + projectRelativePath.slice(projectRelativePath.lastIndexOf('/') + 1) + ); +} + +function describeInputFileChanges( + lastState: IIncrementalInputState, + currentState: IIncrementalInputState, + isChanged: (lastHash: string | undefined, currentHash: string | undefined, filePath: string) => boolean +): string { + const lastFiles: ReadonlyMap | undefined = lastState.inputFiles.deref(); + const currentFiles: ReadonlyMap | undefined = currentState.inputFiles.deref(); + if (!lastFiles || !currentFiles) { + return ''; + } + const changedFiles: string[] = []; + for (const filePath of new Set([...lastFiles.keys(), ...currentFiles.keys()])) { + if (isChanged(lastFiles.get(filePath), currentFiles.get(filePath), filePath)) { + changedFiles.push(filePath); + } + } + if (changedFiles.length === 0) { + return ''; + } + changedFiles.sort(); + const examples: string = changedFiles + .slice(0, MAX_EXAMPLE_PATHS) + .map((filePath: string) => JSON.stringify(filePath)) + .join(', '); + return ` (${examples}${changedFiles.length > MAX_EXAMPLE_PATHS ? ', ...' : ''})`; +} + +/** + * Returns the dependencies of an operation by name, including the dependencies that it has through operations of its + * own project, e.g. through a phase without a script that only orders its build after the builds of other projects. + */ +function getGuardedDependencies(operation: Operation): ReadonlyMap { + const { associatedProject: project } = operation; + const dependencyByName: Map = new Map(); + const dependents: Operation[] = [operation]; + for (let i: number = 0; i < dependents.length; i++) { + for (const dependency of dependents[i].dependencies) { + if (dependencyByName.has(dependency.name)) { + continue; + } + dependencyByName.set(dependency.name, dependency); + if (dependency.associatedProject === project) { + dependents.push(dependency); + } + } + } + return dependencyByName; +} + +function getToolingDependencyReason(operation: Operation, dependency: Operation): string | undefined { + const { associatedProject: project } = operation; + const { associatedProject: dependencyProject } = dependency; + if (dependencyProject === project) { + // Another phase of the same project, e.g. the build that a test operation runs against. + return undefined; + } + const { packageName } = dependencyProject; + const { dependencies, optionalDependencies, peerDependencies } = project.packageJson; + if ( + !dependencies?.[packageName] && + !optionalDependencies?.[packageName] && + !peerDependencies?.[packageName] + ) { + return `its dependency "${packageName}" changed, and it is not a production dependency`; + } + if (isToolingProject(dependencyProject)) { + return `its dependency "${packageName}" changed, and it is a build tool`; + } + return undefined; +} + +const isToolingProjectByProject: WeakMap = new WeakMap(); + +function isToolingProject(project: RushConfigurationProject): boolean { + let isTooling: boolean | undefined = isToolingProjectByProject.get(project); + if (isTooling === undefined) { + const { packageName, projectFolder } = project; + isTooling = + TOOLING_PACKAGE_NAME_REGEXP.test(packageName) || + // A rig package + FileSystem.exists(`${projectFolder}/profiles`) || + FileSystem.exists(`${projectFolder}/heft-plugin.json`); + isToolingProjectByProject.set(project, isTooling); + } + return isTooling; +} diff --git a/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts b/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts new file mode 100644 index 0000000000..76e0520809 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts @@ -0,0 +1,83 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * The reason with which the Rush daemon invalidates operations whose inputs changed. The incremental execution + * guard compares the inputs itself, so such an invalidation keeps the last successful run of an operation as the + * base for its next run. Every other invalidation forgets the base of the operations that it invalidates. + * + * @remarks + * The Rush daemon only imports types from Rush, so it repeats this value. + */ +export const INPUTS_CHANGED_INVALIDATION_REASON: 'workspace-inputs-changed' = 'workspace-inputs-changed'; + +/** + * The reason with which a long-lived host invalidates every operation after a native Rush command ran in the + * workspace. That command may have replaced the outputs of any operation, including those that were already + * invalidated, so the incremental execution guard forgets every base. + */ +export const NATIVE_COMMAND_INVALIDATION_REASON: 'native-command-completed' = 'native-command-completed'; + +/** + * Decides whether an operation may run its `:incremental` command outside watch mode. + * Registered per execution record by {@link IncrementalExecutionGuardPlugin}. + */ +export interface IIncrementalExecutionGuard { + /** + * Returns `undefined` if the incremental command may run, otherwise why it may not, as a clause that completes + * "Not using the incremental command because ...", e.g. `its command line changed`. + */ + getBlockReasonAsync(): Promise; + /** + * Called after the incremental command succeeded. Returns `undefined` if its outputs can be kept, otherwise why + * the initial command must run as well, as a clause that completes "Running the initial command, because ...". + */ + verifyIncrementalResultAsync(): Promise; +} + +/** + * Which command an operation runner executed for an operation in one iteration. + */ +export interface ICommandExecution { + /** + * The command that produced the final outputs. + */ + readonly kind: 'initial' | 'incremental'; + /** + * Whether the runner has an incremental command that a later iteration could use. + */ + readonly hasIncrementalCommand: boolean; +} + +// Both maps are keyed by the execution record, which is the runner's context and the hooks' argument. +const guardByRecord: WeakMap = new WeakMap(); +const commandExecutionByRecord: WeakMap = new WeakMap(); + +export function setIncrementalExecutionGuard(record: object, guard: IIncrementalExecutionGuard): void { + guardByRecord.set(record, guard); +} + +export function getIncrementalExecutionGuard(record: object): IIncrementalExecutionGuard | undefined { + return guardByRecord.get(record); +} + +/** + * Records which command a runner is about to execute for an execution record. Call it before starting the command, + * so that the outputs of a command that fails or is aborted are attributed to it too. + */ +export function setCommandExecution(record: object, execution: ICommandExecution): void { + commandExecutionByRecord.set(record, execution); +} + +export function getCommandExecution(record: object): ICommandExecution | undefined { + return commandExecutionByRecord.get(record); +} + +/** + * Returns true if the outputs of this execution record were produced by an operation's incremental command. + * Such outputs are never written to the build cache or recorded for legacy skip detection, and neither are the + * outputs of consumers that were built against them. + */ +export function wasExecutedIncrementally(record: object): boolean { + return commandExecutionByRecord.get(record)?.kind === 'incremental'; +} diff --git a/libraries/rush-lib/src/logic/operations/LegacySkipPlugin.ts b/libraries/rush-lib/src/logic/operations/LegacySkipPlugin.ts index 2b2ea20896..b7eb963fe8 100644 --- a/libraries/rush-lib/src/logic/operations/LegacySkipPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/LegacySkipPlugin.ts @@ -12,6 +12,7 @@ import type { IPhasedCommandPlugin, PhasedCommandHooks } from '../../pluginFrame import type { IOperationGraphIterationOptions } from './IOperationGraph'; import type { IOperationRunnerContext } from './IOperationRunner'; import type { IOperationExecutionResult } from './IOperationExecutionResult'; +import { wasExecutedIncrementally } from './IncrementalExecutionState'; const PLUGIN_NAME: 'LegacySkipPlugin' = 'LegacySkipPlugin'; @@ -245,6 +246,12 @@ export class LegacySkipPlugin implements IPhasedCommandPlugin { const { packageDeps, packageDepsPath } = skipRecord; + if (wasExecutedIncrementally(record)) { + // The outputs of an incremental command can differ from those of the initial command, so a later + // command must not skip the operation. + return; + } + if ( status === OperationStatus.NoOp || (packageDeps && diff --git a/libraries/rush-lib/src/logic/operations/OperationGraph.ts b/libraries/rush-lib/src/logic/operations/OperationGraph.ts index 68cc1ed4c1..b0408511bb 100644 --- a/libraries/rush-lib/src/logic/operations/OperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/OperationGraph.ts @@ -910,7 +910,8 @@ export class OperationGraph implements IOperationGraph { const iterationOptions: IOperationGraphIterationOptions = { inputsSnapshot: iterationContext.inputsSnapshot, - startTime: iterationContext.startTime + startTime: iterationContext.startTime, + getOperationEnvironment: iterationContext.getOperationEnvironment }; const executionQueue: AsyncOperationQueue = new AsyncOperationQueue( diff --git a/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts b/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts new file mode 100644 index 0000000000..99932a9350 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts @@ -0,0 +1,218 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { createHash, type Hash } from 'node:crypto'; +import type * as fs from 'node:fs'; +import { lstat, readdir } from 'node:fs/promises'; +import * as path from 'node:path'; + +/** + * The files in an operation's declared output folders. + */ +export interface IOperationOutputManifest { + /** + * Covers the path of every file and folder in the output folders, and the identity, modification time and status + * change time of every folder. It changes if an output is added, deleted or renamed, if a file is replaced by a + * rename (as atomic writes do), or if a folder is recreated. It does not change if a file is rewritten in place. + */ + readonly signature: string; + /** + * Project-relative paths, with `/` separators, of the files and symbolic links in the output folders. + */ + readonly files: ReadonlySet; + /** + * If defined, the incremental command must never be used for this operation, and this says why. + */ + readonly cleanOnlyReason: string | undefined; +} + +const MAX_CONCURRENT_READS: number = 8; + +// A content or chunk hash in a JavaScript or CSS file name, e.g. `chunk.main_1a2b3c4d.js` or `0dd8cf755e5195a5.js`. +// The hash must contain a digit, so that words such as `facade` are not mistaken for one. +const HASHED_BUNDLE_FILE_REGEXP: RegExp = + /(?:^|[._-])([0-9a-f]{8,})(?:\.min)?\.(?:js|mjs|cjs|css)(?:\.map|\.LICENSE\.txt)?$/i; +const BUNDLE_FILE_REGEXP: RegExp = /\.(?:js|mjs|cjs|css)$/i; +const BUNDLE_FOLDER_REGEXP: RegExp = /^(?:dist|release)(?:[-_.][^/]*)?(?:\/|$)/i; +// A path with a long hexadecimal run names a cache entry by the hash of its content, e.g. Jest's transform cache +// `temp/test/jest/jest-transform-cache--/7f/index_`. A run adds such entries without changing +// what it builds. +const CONTENT_ADDRESSED_PATH_REGEXP: RegExp = /[0-9a-f]{16,}/i; + +/** + * Lists the files in the output folders of an operation. + * + * @param projectFolder - The absolute path of the project folder + * @param outputFolderNames - The project-relative output folders of the operation + */ +export async function readOperationOutputManifestAsync( + projectFolder: string, + outputFolderNames: ReadonlyArray +): Promise { + const entries: string[] = []; + const files: Set = new Set(); + const limitAsync: (fn: () => Promise) => Promise = createConcurrencyLimiter(MAX_CONCURRENT_READS); + + const readFolderAsync = async (relativeFolder: string, stats: fs.Stats): Promise => { + entries.push(`${relativeFolder}/ ${stats.ino} ${stats.mtimeMs} ${stats.ctimeMs}`); + const children: fs.Dirent[] = await limitAsync(() => + tryReaddirAsync(path.resolve(projectFolder, relativeFolder)) + ); + const subfolderPromises: Promise[] = []; + for (const child of children) { + const relativePath: string = `${relativeFolder}/${child.name}`; + if (child.isDirectory()) { + subfolderPromises.push( + limitAsync(() => tryLstatAsync(path.resolve(projectFolder, relativePath))).then( + async (childStats: fs.Stats | undefined) => { + if (childStats?.isDirectory()) { + await readFolderAsync(relativePath, childStats); + } else { + // It was deleted or replaced after its parent was read. + entries.push(`${relativePath}/ replaced`); + } + } + ) + ); + } else { + entries.push(relativePath); + files.add(relativePath); + } + } + await Promise.all(subfolderPromises); + }; + + await Promise.all( + outputFolderNames.map(async (folderName: string) => { + const relativePath: string = folderName.replace(/\\/g, '/').replace(/\/+$/, ''); + const stats: fs.Stats | undefined = await limitAsync(() => + tryLstatAsync(path.resolve(projectFolder, relativePath)) + ); + if (!stats) { + entries.push(`missing ${relativePath}`); + } else if (stats.isDirectory()) { + await readFolderAsync(relativePath, stats); + } else { + entries.push(`${relativePath} ${stats.ino} ${stats.mtimeMs} ${stats.ctimeMs} ${stats.size}`); + files.add(relativePath); + } + }) + ); + + entries.sort(); + const hash: Hash = createHash('sha1'); + for (const entry of entries) { + hash.update(entry); + hash.update('\n'); + } + + return { + signature: hash.digest('hex'), + files, + cleanOnlyReason: getCleanOnlyReason(files) + }; +} + +/** + * Returns why an operation with these output files must always run its initial command, if it must. + * + * @remarks + * The incremental command does not delete outputs that a run no longer produces. Bundlers name chunks after + * their content or their place in the module graph, so an edit can replace a chunk with a differently named one + * and leave the old one behind, which a full build would not produce. + */ +export function getCleanOnlyReason(files: Iterable): string | undefined { + let bundleFile: string | undefined; + for (const file of files) { + const baseName: string = file.slice(file.lastIndexOf('/') + 1); + const match: RegExpExecArray | null = HASHED_BUNDLE_FILE_REGEXP.exec(baseName); + if (match && /\d/.test(match[1])) { + return `its outputs include the content-hashed file "${file}"`; + } + if (bundleFile === undefined && BUNDLE_FILE_REGEXP.test(baseName) && BUNDLE_FOLDER_REGEXP.test(file)) { + bundleFile = file; + } + } + return bundleFile === undefined ? undefined : `its outputs include the bundle "${bundleFile}"`; +} + +/** + * Describes how two sets of output files differ, e.g. `2 added ("lib/a.js", ...), 1 removed ("lib/b.js")`. + * Files whose paths contain a content hash, such as cache entries, are ignored. + */ +export function describeOutputFileChanges( + before: ReadonlySet, + after: ReadonlySet +): string | undefined { + const added: string[] = []; + const removed: string[] = []; + for (const file of after) { + if (!before.has(file) && !CONTENT_ADDRESSED_PATH_REGEXP.test(file)) { + added.push(file); + } + } + for (const file of before) { + if (!after.has(file) && !CONTENT_ADDRESSED_PATH_REGEXP.test(file)) { + removed.push(file); + } + } + if (added.length === 0 && removed.length === 0) { + return undefined; + } + const describe = (files: string[], verb: string): string => + `${files.length} ${verb} (${files + .sort() + .slice(0, 3) + .map((file: string) => JSON.stringify(file)) + .join(', ')}${files.length > 3 ? ', ...' : ''})`; + return [added.length ? describe(added, 'added') : '', removed.length ? describe(removed, 'removed') : ''] + .filter(Boolean) + .join(', '); +} + +function createConcurrencyLimiter(maxConcurrency: number): (fn: () => Promise) => Promise { + let active: number = 0; + const waiting: (() => void)[] = []; + return async (fn: () => Promise): Promise => { + if (active < maxConcurrency) { + active++; + } else { + // The slot of a finishing call is handed over directly, so `active` is not incremented here. + await new Promise((resolve: () => void) => waiting.push(resolve)); + } + try { + return await fn(); + } finally { + const next: (() => void) | undefined = waiting.shift(); + if (next) { + next(); + } else { + active--; + } + } + }; +} + +async function tryLstatAsync(filePath: string): Promise { + try { + return await lstat(filePath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') { + return undefined; + } + throw error; + } +} + +async function tryReaddirAsync(folderPath: string): Promise { + try { + return await readdir(folderPath, { withFileTypes: true }); + } catch (error) { + const { code } = error as NodeJS.ErrnoException; + // The folder was deleted or replaced by a file after its parent was read; the signature still changes. + if (code === 'ENOENT' || code === 'ENOTDIR') { + return []; + } + throw error; + } +} diff --git a/libraries/rush-lib/src/logic/operations/PhasedCommandEngineExecution.ts b/libraries/rush-lib/src/logic/operations/PhasedCommandEngineExecution.ts index 4fe099a0b9..1d3c77be19 100644 --- a/libraries/rush-lib/src/logic/operations/PhasedCommandEngineExecution.ts +++ b/libraries/rush-lib/src/logic/operations/PhasedCommandEngineExecution.ts @@ -5,6 +5,7 @@ import { LockFile } from '@rushstack/node-core-library'; import type { IPhasedCommandEngine } from '../../api/PhasedCommandEngine'; import { PhasedCommandEngineBusyError } from '../../api/PhasedCommandEngineBusyError'; +import { NATIVE_COMMAND_INVALIDATION_REASON } from './IncrementalExecutionState'; /** * Owns the native process lock only while a host is reconciling or executing one graph iteration. @@ -34,7 +35,7 @@ export class PhasedCommandEngineExecution implements AsyncDisposable { // Native CLI actions leave their process lock file behind on exit. Their work may have // changed ignored outputs, so do not trust the previous in-memory success records. if (lock.dirtyWhenAcquired) { - this._engine.operationGraph.invalidateOperations(undefined, 'native-command-completed'); + this._engine.operationGraph.invalidateOperations(undefined, NATIVE_COMMAND_INVALIDATION_REASON); } } catch (error) { lock.release(); diff --git a/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts b/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts index 627236e23b..9283810d9c 100644 --- a/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts +++ b/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts @@ -17,6 +17,12 @@ import type { IOperationChildProcessReporter } from './OperationEventSink'; import { HeftChildReporterNonFatalError } from './HeftChildProcessReporter'; import { OperationError } from './OperationError'; import { OperationStatus } from './OperationStatus'; +import { + getIncrementalExecutionGuard, + setCommandExecution, + type ICommandExecution, + type IIncrementalExecutionGuard +} from './IncrementalExecutionState'; export interface IShellOperationRunnerOptions { phase: IPhase; @@ -24,10 +30,23 @@ export interface IShellOperationRunnerOptions { displayName: string; initialCommand: string; incrementalCommand: string | undefined; + /** + * If true, the incremental command runs only if a plugin registered an incremental execution guard for the + * operation's execution record and the guard allows it, and the guard checks the outputs of the incremental command + * afterwards. Otherwise the incremental command runs whenever the operation has a last state, as in watch mode. + * Defaults to false. + */ + incrementalCommandRequiresGuard?: boolean; commandForHash: string; ignoredParameterValues: ReadonlyArray; } +interface ICommandTerminals { + readonly terminal: ITerminal; + readonly terminalProvider: ITerminalProvider; + readonly structuredChildOutputTerminalProvider: ITerminalProvider; +} + /** * An `IOperationRunner` implementation that performs an operation via a shell command. * Currently contains the build cache logic, pending extraction as separate operations. @@ -48,6 +67,7 @@ export class ShellOperationRunner implements IOperationRunner { readonly #commandForHash: string; readonly #initialCommand: string; readonly #incrementalCommand: string | undefined; + readonly #incrementalCommandRequiresGuard: boolean; readonly #rushProject: RushConfigurationProject; @@ -60,6 +80,7 @@ export class ShellOperationRunner implements IOperationRunner { rushProject, initialCommand, incrementalCommand, + incrementalCommandRequiresGuard = false, commandForHash, ignoredParameterValues } = options; @@ -70,6 +91,7 @@ export class ShellOperationRunner implements IOperationRunner { this.#rushProject = rushProject; this.#initialCommand = initialCommand; this.#incrementalCommand = incrementalCommand; + this.#incrementalCommandRequiresGuard = incrementalCommandRequiresGuard; this.#commandForHash = commandForHash; this.#ignoredParameterValues = ignoredParameterValues; } @@ -84,130 +106,53 @@ export class ShellOperationRunner implements IOperationRunner { terminalProvider: ITerminalProvider, structuredChildOutputTerminalProvider: ITerminalProvider ) => { - let hasWarningOrError: boolean = false; - // Log any ignored parameters if (this.#ignoredParameterValues.length > 0) { terminal.writeLine( `These parameters were ignored for this operation by project-level configuration: ${this.#ignoredParameterValues.join(' ')}` ); } - const incrementalCommand: string | undefined = - lastState && this.#incrementalCommand ? this.#incrementalCommand : undefined; - const commandToRun: string = incrementalCommand ?? this.#initialCommand; - - // Run the operation - terminal.writeLine( - `Invoking (${incrementalCommand !== undefined ? 'incremental' : 'initial'}): ${commandToRun}` - ); - const { rushConfiguration, projectFolder } = this.#rushProject; - - const { environment: initialEnvironment, abortSignal } = context; - const childProcessReporter: IOperationChildProcessReporter | undefined = - !IS_WINDOWS && isHeftCommand(commandToRun) ? context.createChildProcessReporter() : undefined; - - const subProcess: child_process.ChildProcess = Utilities.executeLifecycleCommandAsync(commandToRun, { - rushConfiguration: rushConfiguration, - workingDirectory: projectFolder, - initCwd: rushConfiguration.commonTempFolder, - handleOutput: true, - environmentPathOptions: { - includeProjectBin: true - }, - initialEnvironment, - additionalEnvironment: childProcessReporter?.environment, - stdio: childProcessReporter?.stdio, - // Isolate the process tree so that a hard abort can terminate it. - connectSubprocessTerminator: abortSignal !== undefined - }); - const terminateProcessTree: () => void = () => { - try { - if (!IS_WINDOWS && subProcess.pid !== undefined && typeof subProcess.exitCode === 'number') { - // The shell already exited, but descendants in its process group may still hold its stdio open. - // killProcessTree() is a no-op in that state, so signal the process group directly. - killExitedProcessGroup(subProcess.pid); - } else { - SubprocessTerminator.killProcessTree(subProcess, SubprocessTerminator.RECOMMENDED_OPTIONS); - } - } catch (error) { - terminal.writeErrorLine(`Failed to terminate the operation process tree: ${error}`); - } + const terminals: ICommandTerminals = { + terminal, + terminalProvider, + structuredChildOutputTerminalProvider }; - if (abortSignal?.aborted) { - terminateProcessTree(); - } else { - abortSignal?.addEventListener('abort', terminateProcessTree, { once: true }); + const incrementalCommand: string | undefined = lastState ? this.#incrementalCommand : undefined; + if (incrementalCommand === undefined) { + return await this.#invokeCommandAsync(context, terminals, 'initial', this.#initialCommand); + } + if (!this.#incrementalCommandRequiresGuard) { + return await this.#invokeCommandAsync(context, terminals, 'incremental', incrementalCommand); } - let reporterError: Error | undefined; - const reporterDrainPromise: Promise = childProcessReporter - ? childProcessReporter - .attachAsync(subProcess, structuredChildOutputTerminalProvider) - .catch((error) => { - reporterError = - error instanceof Error ? error : new Error('The Heft child reporter channel failed.'); - }) - : Promise.resolve(); - - // Hook into events, in order to get live streaming of the log - subProcess.stdout?.on('data', (data: Buffer) => { - const text: string = data.toString(); - terminalProvider.write(text, TerminalProviderSeverity.log); - }); - subProcess.stderr?.on('data', (data: Buffer) => { - const text: string = data.toString(); - terminalProvider.write(text, TerminalProviderSeverity.error); - hasWarningOrError = true; - }); - const closePromise: Promise<{ - readonly exitCode: number | null; - readonly signal: NodeJS.Signals | null; - }> = new Promise( - ( - resolve: (result: { - readonly exitCode: number | null; - readonly signal: NodeJS.Signals | null; - }) => void, - reject: (error: OperationError) => void - ) => { - subProcess.on('close', (exitCode: number | null, signal: NodeJS.Signals | null) => { - try { - resolve({ exitCode, signal }); - } catch (error) { - context.error = error as OperationError; - reject(error as OperationError); - } - }); - } - ); - const [{ exitCode, signal }]: [ - { readonly exitCode: number | null; readonly signal: NodeJS.Signals | null }, - void - ] = await Promise.all([closePromise, reporterDrainPromise]).finally(() => { - abortSignal?.removeEventListener('abort', terminateProcessTree); - }); + const guard: IIncrementalExecutionGuard | undefined = getIncrementalExecutionGuard(context); + if (!guard) { + return await this.#invokeCommandAsync(context, terminals, 'initial', this.#initialCommand); + } + const blockReason: string | undefined = await getGuardResultAsync(() => guard.getBlockReasonAsync()); + if (blockReason !== undefined) { + terminal.writeLine(`Not using the incremental command because ${blockReason}.`); + return await this.#invokeCommandAsync(context, terminals, 'initial', this.#initialCommand); + } - if (abortSignal?.aborted) { - terminal.writeLine('Terminated because the operation was aborted.'); - return OperationStatus.Aborted; - } else if (signal) { - // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. - context.error = new OperationError('error', `Terminated by signal: ${signal}`); - return OperationStatus.Failure; - } else if (exitCode !== 0) { - // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. - context.error = new OperationError('error', `Returned error code: ${exitCode}`); - return OperationStatus.Failure; - } else if (reporterError && !(reporterError instanceof HeftChildReporterNonFatalError)) { - // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. - context.error = new OperationError('error', reporterError.message); - return OperationStatus.Failure; - } else if (hasWarningOrError || childProcessReporter?.hasWarningOrError) { - return OperationStatus.SuccessWithWarning; - } else { - return OperationStatus.Success; + const status: OperationStatus = await this.#invokeCommandAsync( + context, + terminals, + 'incremental', + incrementalCommand + ); + if (status !== OperationStatus.Success && status !== OperationStatus.SuccessWithWarning) { + return status; } + const rerunReason: string | undefined = await getGuardResultAsync(() => + guard.verifyIncrementalResultAsync() + ); + if (rerunReason === undefined) { + return status; + } + terminal.writeLine(`Running the initial command, because ${rerunReason}.`); + return await this.#invokeCommandAsync(context, terminals, 'initial', this.#initialCommand); }, { createLogFile: true @@ -218,6 +163,142 @@ export class ShellOperationRunner implements IOperationRunner { public getConfigHash(): string { return this.#commandForHash; } + + async #invokeCommandAsync( + context: IOperationRunnerContext, + { terminal, terminalProvider, structuredChildOutputTerminalProvider }: ICommandTerminals, + kind: ICommandExecution['kind'], + commandToRun: string + ): Promise { + let hasWarningOrError: boolean = false; + + if (this.#incrementalCommandRequiresGuard) { + // Recorded before the command starts, so that outputs of a command that fails or is aborted are attributed to it. + setCommandExecution(context, { kind, hasIncrementalCommand: this.#incrementalCommand !== undefined }); + } + + // Run the operation + terminal.writeLine(`Invoking (${kind}): ${commandToRun}`); + + const { rushConfiguration, projectFolder } = this.#rushProject; + + const { environment: initialEnvironment, abortSignal } = context; + const childProcessReporter: IOperationChildProcessReporter | undefined = + !IS_WINDOWS && isHeftCommand(commandToRun) ? context.createChildProcessReporter() : undefined; + + const subProcess: child_process.ChildProcess = Utilities.executeLifecycleCommandAsync(commandToRun, { + rushConfiguration: rushConfiguration, + workingDirectory: projectFolder, + initCwd: rushConfiguration.commonTempFolder, + handleOutput: true, + environmentPathOptions: { + includeProjectBin: true + }, + initialEnvironment, + additionalEnvironment: childProcessReporter?.environment, + stdio: childProcessReporter?.stdio, + // Isolate the process tree so that a hard abort can terminate it. + connectSubprocessTerminator: abortSignal !== undefined + }); + const terminateProcessTree: () => void = () => { + try { + if (!IS_WINDOWS && subProcess.pid !== undefined && typeof subProcess.exitCode === 'number') { + // The shell already exited, but descendants in its process group may still hold its stdio open. + // killProcessTree() is a no-op in that state, so signal the process group directly. + killExitedProcessGroup(subProcess.pid); + } else { + SubprocessTerminator.killProcessTree(subProcess, SubprocessTerminator.RECOMMENDED_OPTIONS); + } + } catch (error) { + terminal.writeErrorLine(`Failed to terminate the operation process tree: ${error}`); + } + }; + if (abortSignal?.aborted) { + terminateProcessTree(); + } else { + abortSignal?.addEventListener('abort', terminateProcessTree, { once: true }); + } + let reporterError: Error | undefined; + const reporterDrainPromise: Promise = childProcessReporter + ? childProcessReporter.attachAsync(subProcess, structuredChildOutputTerminalProvider).catch((error) => { + reporterError = + error instanceof Error ? error : new Error('The Heft child reporter channel failed.'); + }) + : Promise.resolve(); + + // Hook into events, in order to get live streaming of the log + subProcess.stdout?.on('data', (data: Buffer) => { + const text: string = data.toString(); + terminalProvider.write(text, TerminalProviderSeverity.log); + }); + subProcess.stderr?.on('data', (data: Buffer) => { + const text: string = data.toString(); + terminalProvider.write(text, TerminalProviderSeverity.error); + hasWarningOrError = true; + }); + + const closePromise: Promise<{ + readonly exitCode: number | null; + readonly signal: NodeJS.Signals | null; + }> = new Promise( + ( + resolve: (result: { + readonly exitCode: number | null; + readonly signal: NodeJS.Signals | null; + }) => void, + reject: (error: OperationError) => void + ) => { + subProcess.on('close', (exitCode: number | null, signal: NodeJS.Signals | null) => { + try { + resolve({ exitCode, signal }); + } catch (error) { + context.error = error as OperationError; + reject(error as OperationError); + } + }); + } + ); + const [{ exitCode, signal }]: [ + { readonly exitCode: number | null; readonly signal: NodeJS.Signals | null }, + void + ] = await Promise.all([closePromise, reporterDrainPromise]).finally(() => { + abortSignal?.removeEventListener('abort', terminateProcessTree); + }); + + if (abortSignal?.aborted) { + terminal.writeLine('Terminated because the operation was aborted.'); + return OperationStatus.Aborted; + } else if (signal) { + // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. + context.error = new OperationError('error', `Terminated by signal: ${signal}`); + return OperationStatus.Failure; + } else if (exitCode !== 0) { + // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. + context.error = new OperationError('error', `Returned error code: ${exitCode}`); + return OperationStatus.Failure; + } else if (reporterError && !(reporterError instanceof HeftChildReporterNonFatalError)) { + // eslint-disable-next-line require-atomic-updates -- This operation context has one active runner. + context.error = new OperationError('error', reporterError.message); + return OperationStatus.Failure; + } else if (hasWarningOrError || childProcessReporter?.hasWarningOrError) { + return OperationStatus.SuccessWithWarning; + } else { + return OperationStatus.Success; + } + } +} + +/** + * Returns what an incremental execution guard returned, or why it failed. + */ +async function getGuardResultAsync( + getResultAsync: () => Promise +): Promise { + try { + return await getResultAsync(); + } catch (error) { + return `its incremental execution guard failed: ${error}`; + } } function killExitedProcessGroup(pid: number): void { diff --git a/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts b/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts index 3330ac8e32..89f489a632 100644 --- a/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/ShellOperationRunnerPlugin.ts @@ -54,14 +54,18 @@ export class ShellOperationRunnerPlugin implements IPhasedCommandPlugin { // For execution of non-initial watch iterations, prefer the `:incremental` script if it exists. // However, the `shellCommand` value still takes precedence per the spec for that feature. - // Outside watch mode, every command runs the initial script, as a single `rush build` does, even - // when a long-lived host (rushd) executes it on a graph that already ran the operation. The - // incremental script may keep outputs of deleted inputs, and only watch mode disables cache writes. + // Outside watch mode, the `:incremental` script only runs where a long-lived host (rushd) registers an + // incremental execution guard for the operation (IncrementalExecutionGuardPlugin). Without one, every + // command runs the initial script, as a single `rush build` does: the incremental script may keep the + // outputs of deleted inputs, and outside watch mode its results could be written to the build cache. const initialCommand: string | undefined = shellCommand ?? scripts?.[phaseName]; - const incrementalCommand: string | undefined = - isIncrementalBuildAllowed && isWatch + const incrementalCommand: string | undefined = !isIncrementalBuildAllowed + ? undefined + : isWatch ? (shellCommand ?? scripts?.[`${phaseName}:incremental`]) - : undefined; + : shellCommand === undefined + ? scripts?.[`${phaseName}:incremental`] + : undefined; operation.runner = initializeShellOperationRunner({ phase, @@ -70,6 +74,7 @@ export class ShellOperationRunnerPlugin implements IPhasedCommandPlugin { commandForHash, initialCommand, incrementalCommand, + incrementalCommandRequiresGuard: !isWatch, customParameterValues, ignoredParameterValues, rushConfiguration @@ -90,6 +95,10 @@ export function initializeShellOperationRunner(options: { rushConfiguration: RushConfiguration; initialCommand: string | undefined; incrementalCommand: string | undefined; + /** + * See `IShellOperationRunnerOptions.incrementalCommandRequiresGuard`. Defaults to false. + */ + incrementalCommandRequiresGuard?: boolean; commandForHash?: string; customParameterValues: ReadonlyArray; ignoredParameterValues: ReadonlyArray; @@ -99,6 +108,7 @@ export function initializeShellOperationRunner(options: { project, initialCommand: rawInitialCommand, incrementalCommand: rawIncrementalCommand, + incrementalCommandRequiresGuard = false, displayName, ignoredParameterValues } = options; @@ -113,9 +123,13 @@ export function initializeShellOperationRunner(options: { const { commandForHash: rawCommandForHash, customParameterValues } = options; const initialCommand: string = formatCommand(rawInitialCommand, customParameterValues); - const incrementalCommand: string | undefined = rawIncrementalCommand + let incrementalCommand: string | undefined = rawIncrementalCommand ? formatCommand(rawIncrementalCommand, customParameterValues) : undefined; + if (incrementalCommandRequiresGuard && incrementalCommand === initialCommand) { + // Running it as the incremental command would only prevent its results from being cached. + incrementalCommand = undefined; + } const commandForHash: string = rawCommandForHash ? formatCommand(rawCommandForHash, customParameterValues) : initialCommand; @@ -123,6 +137,7 @@ export function initializeShellOperationRunner(options: { return new ShellOperationRunner({ initialCommand, incrementalCommand, + incrementalCommandRequiresGuard, commandForHash, displayName, phase, diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts index 3694a67830..172a2e76e1 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts @@ -60,6 +60,7 @@ import { OperationStatus } from '../OperationStatus'; import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; import type { OperationExecutionRecord } from '../OperationExecutionRecord'; +import { setCommandExecution } from '../IncrementalExecutionState'; const mockPhase: IPhase = { name: 'phase', @@ -79,14 +80,22 @@ class CacheableMockRunner implements IOperationRunner { public readonly isNoOp: boolean = false; public readonly name: string; readonly #executions: string[]; + readonly #incrementalNames: ReadonlySet; - public constructor(name: string, executions: string[]) { + public constructor(name: string, executions: string[], incrementalNames: ReadonlySet) { this.name = name; this.#executions = executions; + this.#incrementalNames = incrementalNames; } public async executeAsync(context: IOperationRunnerContext): Promise { - this.#executions.push(this.name); + if (this.#incrementalNames.has(this.name)) { + // Like a ShellOperationRunner whose incremental execution guard allowed its incremental command + setCommandExecution(context, { kind: 'incremental', hasIncrementalCommand: true }); + this.#executions.push(`${this.name}:incremental`); + } else { + this.#executions.push(this.name); + } return OperationStatus.Success; } @@ -106,6 +115,10 @@ interface ITestGraph { * The operations that the emulated change detection plugin checked, if enabled by `upToDate`. */ checks: string[]; + /** + * The names of the operations whose runner executes its incremental command + */ + incrementalNames: Set; executeAsync(): Promise; } @@ -132,6 +145,7 @@ async function createTestGraphAsync(names: string[], options: ITestGraphOptions const checks: string[] = []; const cacheWrites: string[] = []; const cacheRestores: string[] = []; + const incrementalNames: Set = new Set(); const cacheEntries: Set = new Set(); const localHashes: Map = new Map(); const operations: Map = new Map(); @@ -147,7 +161,7 @@ async function createTestGraphAsync(names: string[], options: ITestGraphOptions getCacheDisabledReason: () => undefined } as unknown as RushProjectConfiguration); const operation: Operation = new Operation({ - runner: new CacheableMockRunner(name, executions), + runner: new CacheableMockRunner(name, executions, incrementalNames), logFilenameIdentifier: name, phase: mockPhase, project @@ -254,6 +268,7 @@ async function createTestGraphAsync(names: string[], options: ITestGraphOptions cacheWrites, cacheRestores, checks, + incrementalNames, executeAsync: async () => { executions.length = 0; cacheWrites.length = 0; @@ -464,6 +479,99 @@ describe(`${CacheableOperationPlugin.name} retained results`, () => { expect(testGraph.executions).toEqual([]); }); + it('trusts an incremental result, but writes neither it nor the results built against it to the build cache', async () => { + // "a" <- "b" <- "c" + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); + await testGraph.executeAsync(); + expect(testGraph.cacheWrites).toEqual(['a', 'b', 'c']); + + // Edit "a", which runs its incremental command. + testGraph.localHashes.set('a', 'a-v2'); + testGraph.incrementalNames.add('a'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a:incremental', 'b', 'c']); + expect(testGraph.cacheWrites).toEqual([]); + + // Nothing changed, so nothing runs again. + const hotResult: IExecutionResult = await testGraph.executeAsync(); + expect(hotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + + // Edit "c": it is built against the retained outputs of "b", which were built against the incremental result. + testGraph.localHashes.set('c', 'c-v2'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + expect(testGraph.cacheWrites).toEqual([]); + + const secondHotResult: IExecutionResult = await testGraph.executeAsync(); + expect(secondHotResult.status).toBe(OperationStatus.NoOp); + expect(testGraph.executions).toEqual([]); + + // Edit "a", which runs its initial command: every result can be written again. + testGraph.localHashes.set('a', 'a-v3'); + testGraph.incrementalNames.delete('a'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a', 'b', 'c']); + expect(testGraph.cacheWrites).toEqual(['a', 'b', 'c']); + }); + + it('does not write results built against an incremental result that the request did not select', async () => { + // "a" <- "b" <- "c" + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); + const { operations } = testGraph; + await testGraph.executeAsync(); + + testGraph.localHashes.set('a', 'a-v2'); + testGraph.incrementalNames.add('a'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a:incremental', 'b', 'c']); + expect(testGraph.cacheWrites).toEqual([]); + + // --only b: "b" is built against the retained incremental result of "a". + testGraph.localHashes.set('b', 'b-v2'); + operations.get('a')!.enabled = false; + operations.get('c')!.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual([]); + + // --only c: "c" is built against "b", which was built against the incremental result of "a". + testGraph.localHashes.set('c', 'c-v2'); + operations.get('b')!.enabled = false; + operations.get('c')!.enabled = true; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + expect(testGraph.cacheWrites).toEqual([]); + }); + + it('restores a consumer of an incremental result from the build cache and trusts it as a cacheable result', async () => { + // "a" <- "b" <- "c" + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); + await testGraph.executeAsync(); + + // Edit "a", which runs its incremental command, and "c". + testGraph.localHashes.set('a', 'a-v2'); + testGraph.localHashes.set('c', 'c-v2'); + testGraph.incrementalNames.add('a'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a:incremental', 'b', 'c']); + expect(testGraph.cacheWrites).toEqual([]); + + // Revert both: every operation has an entry from the first iteration. + testGraph.localHashes.set('a', 'a-v1'); + testGraph.localHashes.set('c', 'c-v1'); + testGraph.incrementalNames.delete('a'); + await testGraph.executeAsync(); + expect(testGraph.cacheRestores).toEqual(['a', 'b', 'c']); + expect(testGraph.executions).toEqual([]); + + // Edit "c": it is built against restored outputs, so its result is written. + testGraph.localHashes.set('c', 'c-v3'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['c']); + expect(testGraph.cacheWrites).toEqual(['c']); + }); + it('does not re-enable operations that another plugin disabled', async () => { const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); // Like a plugin that performs the work itself. This tap runs after PhasedOperationPlugin's. diff --git a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts new file mode 100644 index 0000000000..02b33ea377 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts @@ -0,0 +1,691 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('../ProjectLogWritable', () => { + const actual = jest.requireActual('../ProjectLogWritable'); + const { TerminalWritable } = jest.requireActual('@rushstack/terminal'); + class MockTerminalWritable extends TerminalWritable { + protected onWriteChunk(): void { + /* noop */ + } + protected onClose(): void { + /* noop */ + } + } + return { + ...actual, + initializeProjectLogFilesAsync: jest.fn(async () => new MockTerminalWritable()) + }; +}); +jest.mock('../OperationMetadataManager', () => { + class MockOperationMetadataManager { + public readonly logFilenameIdentifier: string; + public readonly metadataFolderPath: string = '.rush/temp/operation/mock'; + public readonly stateFile: { state: undefined } = { state: undefined }; + public constructor({ operation }: { operation: { logFilenameIdentifier: string } }) { + this.logFilenameIdentifier = operation.logFilenameIdentifier; + } + public async saveAsync(): Promise { + /* noop */ + } + public async tryRestoreAsync(): Promise { + /* noop */ + } + public tryRestoreStopwatch(originalStopwatch: T): T { + return originalStopwatch; + } + } + return { OperationMetadataManager: MockOperationMetadataManager }; +}); + +import type * as childProcess from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { EventEmitter } from 'node:events'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { PassThrough } from 'node:stream'; + +import { LookupByPath } from '@rushstack/lookup-by-path'; +import { SubprocessTerminator } from '@rushstack/node-core-library'; +import { MockWritable } from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import type { IOperationSettings, RushProjectConfiguration } from '../../../api/RushProjectConfiguration'; +import { PhasedCommandHooks, type IOperationGraphContext } from '../../../pluginFramework/PhasedCommandHooks'; +import { Utilities } from '../../../utilities/Utilities'; +import { InputsSnapshot, type IInputsSnapshotProjectMetadata } from '../../incremental/InputsSnapshot'; +import { IncrementalExecutionGuardPlugin } from '../IncrementalExecutionGuardPlugin'; +import { + INPUTS_CHANGED_INVALIDATION_REASON, + NATIVE_COMMAND_INVALIDATION_REASON +} from '../IncrementalExecutionState'; +import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; +import { NullOperationRunner } from '../NullOperationRunner'; +import { Operation } from '../Operation'; +import type { OperationExecutionRecord } from '../OperationExecutionRecord'; +import { OperationGraph } from '../OperationGraph'; +import { OperationStatus } from '../OperationStatus'; +import { PhasedOperationPlugin } from '../PhasedOperationPlugin'; +import { markResultUnverifiable } from '../RetainedResultVerification'; +import { ShellOperationRunner } from '../ShellOperationRunner'; + +const PHASE_NAME: string = '_phase:build'; +const INITIAL_COMMAND: string = 'node build.js'; +const INCREMENTAL_COMMAND: string = 'node build.js --incremental'; + +const buildPhase: IPhase = { + name: PHASE_NAME, + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + dependencies: { self: new Set(), upstream: new Set() }, + isSynthetic: false, + logFilenameIdentifier: '_phase_build', + missingScriptBehavior: 'error' +}; + +const liteBuildPhase: IPhase = { + ...buildPhase, + name: '_phase:lite-build', + logFilenameIdentifier: '_phase_lite-build', + missingScriptBehavior: 'silent' +}; + +interface IProjectSpec { + readonly name: string; + readonly dependencies?: ReadonlyArray; + readonly devDependencies?: ReadonlyArray; + /** + * If set, the build writes a single bundle `dist/main.js` instead of a `lib` file per source file. + */ + readonly isBundle?: boolean; + readonly dependsOnEnvVars?: ReadonlyArray; + /** + * If set, the project has a `profiles` folder, like a rig package. + */ + readonly isRig?: boolean; +} + +interface IWorkspaceOptions { + /** + * If set, like the phases of the rushstack repo, the build of each project depends only on a `_phase:lite-build` + * operation of its own project, which has no script and depends on the builds of the project's dependencies. + */ + readonly hasPassThroughPhase?: boolean; +} + +interface ITestIteration { + readonly result: IExecutionResult; + /** + * Each command that ran, as `:initial` or `:incremental` + */ + readonly commands: ReadonlyArray; + readonly output: string; + getStatus(name: string): OperationStatus; +} + +interface ITestWorkspace { + readonly rootFolder: string; + readonly graph: OperationGraph; + readonly operations: ReadonlyMap; + writeFile(relativePath: string, content: string): void; + deleteFile(relativePath: string): void; + executeAsync(environment?: Readonly>): Promise; + /** + * Resolves when the next command that hangs has written its outputs. It runs until it is terminated. + */ + waitForHangAsync(): Promise; +} + +const workspaceFolders: string[] = []; + +afterEach(() => { + jest.restoreAllMocks(); + for (const folder of workspaceFolders.splice(0)) { + fs.rmSync(folder, { recursive: true, force: true }); + } +}); + +function listFiles(folder: string, exclude: ReadonlySet = new Set()): string[] { + const files: string[] = []; + const visit = (relativeFolder: string): void => { + for (const entry of fs.readdirSync(path.join(folder, relativeFolder), { withFileTypes: true })) { + const relativePath: string = relativeFolder ? `${relativeFolder}/${entry.name}` : entry.name; + if (exclude.has(relativePath)) { + continue; + } + if (entry.isDirectory()) { + visit(relativePath); + } else { + files.push(relativePath); + } + } + }; + visit(''); + return files.sort(); +} + +// Like a compiler: writes a file per source file, and its incremental mode neither cleans the output folder nor +// deletes the outputs of deleted source files. A source containing "emit:" also emits ".js", a +// source containing "error" fails the build, and after the output of a source containing "hang", the build runs +// until it is terminated (it returns undefined). +function build(projectFolder: string, isBundle: boolean, isIncremental: boolean): number | undefined { + const outputFolder: string = `${projectFolder}/${isBundle ? 'dist' : 'lib'}`; + if (!isIncremental) { + fs.rmSync(outputFolder, { recursive: true, force: true }); + } + fs.mkdirSync(outputFolder, { recursive: true }); + const bundle: string[] = []; + for (const sourcePath of listFiles(`${projectFolder}/src`)) { + const source: string = fs.readFileSync(`${projectFolder}/src/${sourcePath}`, 'utf8'); + if (source.includes('error')) { + return 1; + } + bundle.push(source); + if (!isBundle) { + const outputPath: string = `${outputFolder}/${sourcePath.replace(/\.ts$/, '.js')}`; + fs.mkdirSync(path.dirname(outputPath), { recursive: true }); + fs.writeFileSync(outputPath, source); + } + const emitted: RegExpExecArray | null = /emit:(\w+)/.exec(source); + if (emitted) { + fs.writeFileSync(`${outputFolder}/${emitted[1]}.js`, ''); + } + if (source.includes('hang')) { + return undefined; + } + } + if (isBundle) { + fs.writeFileSync(`${outputFolder}/main.js`, bundle.join('\n')); + } + return 0; +} + +async function createWorkspaceAsync( + projectSpecs: ReadonlyArray, + { hasPassThroughPhase }: IWorkspaceOptions = {} +): Promise { + const rootFolder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-incremental-guard-')); + workspaceFolders.push(rootFolder); + + const writeFile = (relativePath: string, content: string): void => { + const filePath: string = `${rootFolder}/${relativePath}`; + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, content); + }; + + const commands: string[] = []; + const hangWaiters: (() => void)[] = []; + const specByFolder: Map = new Map(); + const close = (child: childProcess.ChildProcess, exitCode: number | null, signal: string | null): void => { + queueMicrotask(() => { + (child.stdout as PassThrough).end(); + (child.stderr as PassThrough).end(); + child.emit('close', exitCode, signal); + }); + }; + jest + .spyOn(Utilities, 'executeLifecycleCommandAsync') + .mockImplementation((command: string, { workingDirectory }: { workingDirectory: string }) => { + const spec: IProjectSpec = specByFolder.get(workingDirectory)!; + const isIncremental: boolean = command === INCREMENTAL_COMMAND; + commands.push(`${spec.name}:${isIncremental ? 'incremental' : 'initial'}`); + const exitCode: number | undefined = build(workingDirectory, !!spec.isBundle, isIncremental); + const child: childProcess.ChildProcess = Object.assign(new EventEmitter(), { + stdout: new PassThrough(), + stderr: new PassThrough(), + stdio: [] + }) as unknown as childProcess.ChildProcess; + if (exitCode === undefined) { + for (const resolve of hangWaiters.splice(0)) { + resolve(); + } + } else { + close(child, exitCode, null); + } + return child; + }); + jest + .spyOn(SubprocessTerminator, 'killProcessTree') + .mockImplementation((child: childProcess.ChildProcess) => close(child, null, 'SIGTERM')); + + const operations: Map = new Map(); + const passThroughOperations: Operation[] = []; + const projectMap: Map = new Map(); + const projectConfigurations: Map = new Map(); + const lookupByPath: LookupByPath = new LookupByPath(); + const outputFolderByPrefix: Map = new Map(); + for (const spec of projectSpecs) { + const { name, dependencies = [], devDependencies = [], isBundle, dependsOnEnvVars, isRig } = spec; + const projectFolder: string = `${rootFolder}/${name}`; + const toVersions = (names: ReadonlyArray): Record => + Object.fromEntries(names.map((dependencyName: string) => [dependencyName, 'workspace:*'])); + const packageJson: RushConfigurationProject['packageJson'] = { + name, + version: '1.0.0', + dependencies: toVersions(dependencies), + devDependencies: toVersions(devDependencies) + }; + writeFile(`${name}/package.json`, JSON.stringify(packageJson)); + writeFile(`${name}/tsconfig.json`, '{}'); + writeFile(`${name}/src/one.ts`, 'one'); + writeFile(`${name}/src/sub/two.ts`, 'two'); + if (isRig) { + writeFile(`${name}/profiles/default/config/heft.json`, '{}'); + } + + const project: RushConfigurationProject = { + packageName: name, + projectFolder, + projectRelativeFolder: name, + packageJson, + rushConfiguration: { commonTempFolder: `${rootFolder}/common/temp` } + } as unknown as RushConfigurationProject; + const outputFolderName: string = isBundle ? 'dist' : 'lib'; + const settings: IOperationSettings = { + operationName: PHASE_NAME, + outputFolderNames: [outputFolderName], + dependsOnEnvVars: dependsOnEnvVars ? [...dependsOnEnvVars] : undefined + }; + const projectConfiguration: RushProjectConfiguration = { + operationSettingsByOperationName: new Map([[PHASE_NAME, settings]]), + getCacheDisabledReason: () => undefined + } as unknown as RushProjectConfiguration; + projectConfigurations.set(project, projectConfiguration); + projectMap.set(project, { projectConfig: projectConfiguration }); + lookupByPath.setItem(name, project); + outputFolderByPrefix.set(name, outputFolderName); + specByFolder.set(projectFolder, spec); + + const operation: Operation = new Operation({ + phase: buildPhase, + project, + settings, + logFilenameIdentifier: '_phase_build', + runner: new ShellOperationRunner({ + phase: buildPhase, + rushProject: project, + displayName: name, + initialCommand: INITIAL_COMMAND, + incrementalCommand: INCREMENTAL_COMMAND, + incrementalCommandRequiresGuard: true, + commandForHash: INITIAL_COMMAND, + ignoredParameterValues: [] + }) + }); + let dependent: Operation = operation; + if (hasPassThroughPhase) { + dependent = new Operation({ + phase: liteBuildPhase, + project, + logFilenameIdentifier: liteBuildPhase.logFilenameIdentifier, + runner: new NullOperationRunner({ + name: `${name} (lite-build)`, + result: OperationStatus.NoOp, + silent: true + }) + }); + operation.addDependency(dependent); + passThroughOperations.push(dependent); + } + for (const dependencyName of [...dependencies, ...devDependencies]) { + dependent.addDependency(operations.get(dependencyName)!); + } + operations.set(name, operation); + } + + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new PhasedOperationPlugin().apply(hooks); + new IncrementalExecutionGuardPlugin().apply(hooks); + const destination: MockWritable = new MockWritable(); + const graphOperations: Set = new Set([...operations.values(), ...passThroughOperations]); + const graph: OperationGraph = new OperationGraph(graphOperations, { + quietMode: false, + debugMode: false, + parallelism: 1, + allowOversubscription: true, + destinations: [destination], + abortController: new AbortController(), + // Like the graphs of the Rush daemon + supportsTerminateRunning: true + }); + await hooks.onGraphCreatedAsync.promise(graph, { + isIncrementalBuildAllowed: true, + isWatch: false, + projectConfigurations + } as unknown as IOperationGraphContext); + + // Like `git hash-object` for each file, except the outputs, which are ignored by git. + const createInputsSnapshot = (environment: Readonly>): InputsSnapshot => { + const hashes: Map = new Map(); + for (const [prefix, outputFolderName] of outputFolderByPrefix) { + for (const file of listFiles(`${rootFolder}/${prefix}`, new Set([outputFolderName]))) { + const content: Buffer = fs.readFileSync(`${rootFolder}/${prefix}/${file}`); + hashes.set(`${prefix}/${file}`, createHash('sha1').update(content).digest('hex')); + } + } + return new InputsSnapshot({ + rootDir: rootFolder, + hashes, + hasUncommittedChanges: false, + lookupByPath, + projectMap, + environment: { ...environment } + }); + }; + + return { + rootFolder, + graph, + operations, + writeFile, + deleteFile: (relativePath: string) => fs.rmSync(`${rootFolder}/${relativePath}`), + executeAsync: async (environment: Readonly> = {}): Promise => { + commands.length = 0; + destination.reset(); + const result: IExecutionResult = await graph.executeAsync({ + inputsSnapshot: createInputsSnapshot(environment), + getOperationEnvironment: () => environment + }); + return { + result, + commands: [...commands], + output: destination.getAllOutput(), + getStatus: (name: string) => + (result.operationResults.get(operations.get(name)!) as OperationExecutionRecord).status + }; + }, + waitForHangAsync: () => new Promise((resolve: () => void) => hangWaiters.push(resolve)) + }; +} + +describe(IncrementalExecutionGuardPlugin.name, () => { + it('runs the incremental command for edits of built files, and the initial command otherwise', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + expect((await workspace.executeAsync()).commands).toEqual(['a:initial']); + + workspace.writeFile('a/src/one.ts', 'one 2'); + const edited: ITestIteration = await workspace.executeAsync(); + expect(edited.commands).toEqual(['a:incremental']); + expect(edited.getStatus('a')).toBe(OperationStatus.Success); + expect(edited.output).toContain(`Invoking (incremental): ${INCREMENTAL_COMMAND}`); + + // The result of the incremental command is the base of the next one. + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + + // An unchanged workspace runs nothing. + expect((await workspace.executeAsync()).commands).toEqual([]); + }); + + interface IInputChangeCase { + readonly change: (workspace: ITestWorkspace) => void; + readonly reason: string; + /** + * A source file that exists after the change + */ + readonly sourceFile: string; + } + + it.each<[string, IInputChangeCase]>([ + [ + 'a file is added', + { + change: (workspace: ITestWorkspace) => workspace.writeFile('a/src/three.ts', 'three'), + reason: 'input files were added, deleted or renamed ("a/src/three.ts")', + sourceFile: 'a/src/three.ts' + } + ], + [ + 'a file is deleted', + { + change: (workspace: ITestWorkspace) => workspace.deleteFile('a/src/sub/two.ts'), + reason: 'input files were added, deleted or renamed ("a/src/sub/two.ts")', + sourceFile: 'a/src/one.ts' + } + ], + [ + 'a file is renamed', + { + change: (workspace: ITestWorkspace) => + fs.renameSync(`${workspace.rootFolder}/a/src/one.ts`, `${workspace.rootFolder}/a/src/uno.ts`), + reason: 'input files were added, deleted or renamed ("a/src/one.ts", "a/src/uno.ts")', + sourceFile: 'a/src/uno.ts' + } + ], + [ + 'a configuration file changes', + { + change: (workspace: ITestWorkspace) => { + workspace.writeFile('a/src/one.ts', 'one 2'); + workspace.writeFile('a/tsconfig.json', '{ "compilerOptions": {} }'); + }, + reason: 'a configuration file changed ("a/tsconfig.json")', + sourceFile: 'a/src/one.ts' + } + ] + ])('runs the initial command if %s', async (description: string, { change, reason, sourceFile }) => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + change(workspace); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain(`Not using the incremental command because ${reason}.`); + + // The result of the initial command is the new base. + workspace.writeFile(sourceFile, 'edited'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + }); + + it('runs the initial command if an environment variable that the operation depends on changes', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a', dependsOnEnvVars: ['MODE'] }]); + await workspace.executeAsync({ MODE: 'debug' }); + + const changed: ITestIteration = await workspace.executeAsync({ MODE: 'ship' }); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain( + 'Not using the incremental command because an environment variable that it depends on changed.' + ); + }); + + it('runs the incremental command of a project whose production dependency changed', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([ + { name: 'a' }, + { name: 'b', dependencies: ['a'] } + ]); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'one 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental', 'b:incremental']); + }); + + it.each([ + [ + 'a dev dependency', + [{ name: 'a' }, { name: 'b', devDependencies: ['a'] }], + 'its dependency "a" changed, and it is not a production dependency' + ], + [ + 'a rig', + [ + { name: 'a', isRig: true }, + { name: 'b', dependencies: ['a'] } + ], + 'its dependency "a" changed, and it is a build tool' + ] + ])( + 'runs the initial command of a project if %s changed', + async (description, projectSpecs: IProjectSpec[], reason: string) => { + const workspace: ITestWorkspace = await createWorkspaceAsync(projectSpecs); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'one 2'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental', 'b:initial']); + expect(changed.output).toContain(`Not using the incremental command because ${reason}.`); + } + ); + + it('checks the dependencies that an operation has through a phase of its own project without a script', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync( + [{ name: 'a' }, { name: 'b', devDependencies: ['a'] }, { name: 'c', dependencies: ['a'] }], + { hasPassThroughPhase: true } + ); + expect([...(await workspace.executeAsync()).commands].sort()).toEqual([ + 'a:initial', + 'b:initial', + 'c:initial' + ]); + + workspace.writeFile('a/src/one.ts', 'one 2'); + const changed: ITestIteration = await workspace.executeAsync(); + expect([...changed.commands].sort()).toEqual(['a:incremental', 'b:initial', 'c:incremental']); + expect(changed.output).toContain( + 'Not using the incremental command because its dependency "a" changed, and it is not a production dependency.' + ); + + // A change in the project of the phase without a script is judged by the operation's own inputs. + workspace.writeFile('b/src/one.ts', 'one 2'); + workspace.writeFile('c/src/one.ts', 'one 2'); + expect([...(await workspace.executeAsync()).commands].sort()).toEqual(['b:incremental', 'c:incremental']); + }); + + it('runs the initial command if the output folders changed since the last run', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + // E.g. another tool atomically replaced an output file. + workspace.writeFile('a/lib/sub/two.js.tmp', 'tampered'); + fs.renameSync(`${workspace.rootFolder}/a/lib/sub/two.js.tmp`, `${workspace.rootFolder}/a/lib/sub/two.js`); + workspace.writeFile('a/src/one.ts', 'one 2'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain( + 'Not using the incremental command because its output folders changed since its last successful run.' + ); + }); + + it('always runs the initial command of an operation that builds a bundle', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a', isBundle: true }]); + await workspace.executeAsync(); + + for (const content of ['one 2', 'one 3']) { + workspace.writeFile('a/src/one.ts', content); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain( + 'Not using the incremental command because its outputs include the bundle "dist/main.js".' + ); + } + }); + + it('runs the initial command after an incremental command that changed which output files exist', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'one emit:chunk'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental', 'a:initial']); + expect(changed.getStatus('a')).toBe(OperationStatus.Success); + expect(changed.output).toContain( + 'Running the initial command, because the incremental command changed which output files it has: 1 added ("lib/chunk.js").' + ); + + // Its outputs may be named after their content, so it never runs its incremental command again. + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + const next: ITestIteration = await workspace.executeAsync(); + expect(next.commands).toEqual(['a:initial']); + expect(next.output).toContain( + 'Not using the incremental command because its incremental command changed which output files it has in an earlier run: 1 added ("lib/chunk.js").' + ); + }); + + it('runs the initial command after a failure', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'error'); + const failed: ITestIteration = await workspace.executeAsync(); + expect(failed.commands).toEqual(['a:incremental']); + expect(failed.getStatus('a')).toBe(OperationStatus.Failure); + + workspace.writeFile('a/src/one.ts', 'one 2'); + const fixed: ITestIteration = await workspace.executeAsync(); + expect(fixed.commands).toEqual(['a:initial']); + expect(fixed.output).toContain( + 'Not using the incremental command because its outputs were not built by a successful run of its own command in this process.' + ); + }); + + it('runs the initial command after a command that was terminated, but not for operations that did not start', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([ + { name: 'a' }, + { name: 'b', dependencies: ['a'] } + ]); + await workspace.executeAsync(); + + // Its incremental command rewrites "lib/one.js" in place, so its output folders list the same files. + workspace.writeFile('a/src/one.ts', 'one hang'); + const hung: Promise = workspace.waitForHangAsync(); + const execution: Promise = workspace.executeAsync(); + await hung; + await workspace.graph.abortCurrentIterationAsync({ terminateRunning: true }); + const aborted: ITestIteration = await execution; + expect(aborted.commands).toEqual(['a:incremental']); + expect(aborted.getStatus('a')).toBe(OperationStatus.Aborted); + expect(aborted.getStatus('b')).toBe(OperationStatus.Aborted); + + // An aborted operation keeps its last successful result, but not its base if its command started. + workspace.writeFile('a/src/one.ts', 'one 3'); + const next: ITestIteration = await workspace.executeAsync(); + expect(next.commands).toEqual(['a:initial', 'b:incremental']); + expect(next.output).toContain( + 'Not using the incremental command because its outputs were not built by a successful run of its own command in this process.' + ); + }); + + it('runs the initial command after a run whose input files changed while it ran', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([ + { name: 'a' }, + { name: 'b', dependencies: ['a'] } + ]); + // Like CacheableOperationPlugin, e.g. when the command of "a" rewrites an API report that is one of its inputs. + let unverifiableProjectName: string | undefined = 'a'; + workspace.graph.hooks.afterExecuteOperationAsync.tap('test', (record: IOperationExecutionResult) => { + if (record.operation.associatedProject.packageName === unverifiableProjectName) { + markResultUnverifiable(record); + } + }); + expect((await workspace.executeAsync()).commands).toEqual(['a:initial', 'b:initial']); + + unverifiableProjectName = undefined; + workspace.writeFile('a/src/one.ts', 'one 2'); + const next: ITestIteration = await workspace.executeAsync(); + expect(next.commands).toEqual(['a:initial', 'b:incremental']); + expect(next.output).toContain( + 'Not using the incremental command because its outputs were not built by a successful run of its own command in this process.' + ); + + workspace.writeFile('a/src/one.ts', 'one 3'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental', 'b:incremental']); + }); + + it('forgets every base after a native command, but not after an input change', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }, { name: 'b' }]); + await workspace.executeAsync(); + + workspace.graph.invalidateOperations( + [workspace.operations.get('a')!], + INPUTS_CHANGED_INVALIDATION_REASON + ); + workspace.writeFile('a/src/one.ts', 'one 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + + workspace.graph.invalidateOperations(undefined, NATIVE_COMMAND_INVALIDATION_REASON); + workspace.writeFile('a/src/one.ts', 'one 3'); + workspace.writeFile('b/src/one.ts', 'one 3'); + expect([...(await workspace.executeAsync()).commands].sort()).toEqual(['a:initial', 'b:initial']); + }); +}); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts index 653c027eb2..061de0a2b8 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts @@ -167,4 +167,33 @@ describe('OperationGraph operation environment', () => { ]) ); }); + + it('gives every iteration hook the environment lookup that the iteration was scheduled with', async () => { + const graph: OperationGraph = createGraph([new EnvironmentRecordingRunner('operation')]); + type GetOperationEnvironment = IOperationGraphIterationOptions['getOperationEnvironment']; + const received: [string, GetOperationEnvironment][] = []; + graph.hooks.configureIteration.tap('test', (records, lastResults, options) => { + received.push(['configureIteration', options.getOperationEnvironment]); + }); + graph.hooks.beforeExecuteIterationAsync.tapPromise('test', async (records, options) => { + received.push(['beforeExecuteIterationAsync', options.getOperationEnvironment]); + }); + graph.hooks.afterExecuteIterationAsync.tapPromise('test', async (status, records, options) => { + received.push(['afterExecuteIterationAsync', options.getOperationEnvironment]); + return status; + }); + const getOperationEnvironment: GetOperationEnvironment = () => ({ [SESSION_VARIABLE]: 'A' }); + + expect((await graph.executeAsync({ getOperationEnvironment })).status).toBe(OperationStatus.Success); + expect((await graph.executeAsync({})).status).toBe(OperationStatus.Success); + + expect(received).toEqual([ + ['configureIteration', getOperationEnvironment], + ['beforeExecuteIterationAsync', getOperationEnvironment], + ['afterExecuteIterationAsync', getOperationEnvironment], + ['configureIteration', undefined], + ['beforeExecuteIterationAsync', undefined], + ['afterExecuteIterationAsync', undefined] + ]); + }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts new file mode 100644 index 0000000000..83582351ea --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts @@ -0,0 +1,125 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + describeOutputFileChanges, + getCleanOnlyReason, + readOperationOutputManifestAsync, + type IOperationOutputManifest +} from '../OperationOutputManifest'; + +describe(getCleanOnlyReason.name, () => { + it.each([ + [ + 'lib/chunk.main_1a2b3c4d.js', + 'its outputs include the content-hashed file "lib/chunk.main_1a2b3c4d.js"' + ], + ['lib/0dd8cf755e5195a5.js', 'its outputs include the content-hashed file "lib/0dd8cf755e5195a5.js"'], + ['lib/app-3f9a1c7b.min.css', 'its outputs include the content-hashed file "lib/app-3f9a1c7b.min.css"'], + [ + 'lib/vendor.4e5f6a7b.js.map', + 'its outputs include the content-hashed file "lib/vendor.4e5f6a7b.js.map"' + ], + ['dist/main.js', 'its outputs include the bundle "dist/main.js"'], + ['dist-cjs/index.cjs', 'its outputs include the bundle "dist-cjs/index.cjs"'], + ['release/styles.css', 'its outputs include the bundle "release/styles.css"'] + ])('treats %s as a clean-only output', (file: string, reason: string) => { + expect(getCleanOnlyReason(['lib/index.js', file])).toBe(reason); + }); + + it.each(['lib/index.js', 'lib/facade.js', 'lib/deadbeefcafe.js', 'lib-commonjs/a.js', 'dist/index.d.ts'])( + 'allows incremental builds of %s', + (file: string) => { + expect(getCleanOnlyReason([file])).toBeUndefined(); + } + ); +}); + +describe(describeOutputFileChanges.name, () => { + it('describes added and removed files, and ignores content-addressed files', () => { + expect(describeOutputFileChanges(new Set(['lib/a.js']), new Set(['lib/a.js']))).toBeUndefined(); + expect( + describeOutputFileChanges( + new Set(['lib/a.js', 'lib/b.js']), + new Set(['lib/a.js', 'lib/e.js', 'lib/d.js', 'lib/c.js', 'lib/f.js']) + ) + ).toBe('4 added ("lib/c.js", "lib/d.js", "lib/e.js", ...), 1 removed ("lib/b.js")'); + expect( + describeOutputFileChanges( + new Set(['temp/jest-transform-cache-0123456789abcdef-0123456789abcdef/7f/a_1']), + new Set(['temp/jest-transform-cache-0123456789abcdef-0123456789abcdef/8e/b_2']) + ) + ).toBeUndefined(); + }); +}); + +describe(readOperationOutputManifestAsync.name, () => { + let projectFolder: string; + + beforeEach(() => { + projectFolder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-output-manifest-')); + fs.mkdirSync(`${projectFolder}/lib/sub`, { recursive: true }); + fs.writeFileSync(`${projectFolder}/lib/index.js`, 'index'); + fs.writeFileSync(`${projectFolder}/lib/sub/util.js`, 'util'); + fs.writeFileSync(`${projectFolder}/tsconfig.tsbuildinfo`, '{}'); + }); + + afterEach(() => { + fs.rmSync(projectFolder, { recursive: true, force: true }); + }); + + async function readAsync(): Promise { + return await readOperationOutputManifestAsync(projectFolder, ['lib/', 'tsconfig.tsbuildinfo', 'lib-esm']); + } + + it('lists the output files', async () => { + const manifest: IOperationOutputManifest = await readAsync(); + expect(Array.from(manifest.files).sort()).toEqual([ + 'lib/index.js', + 'lib/sub/util.js', + 'tsconfig.tsbuildinfo' + ]); + expect(manifest.cleanOnlyReason).toBeUndefined(); + expect((await readAsync()).signature).toBe(manifest.signature); + }); + + it('does not change if a file is rewritten in place', async () => { + const { signature } = await readAsync(); + fs.writeFileSync(`${projectFolder}/lib/sub/util.js`, 'util 2'); + expect((await readAsync()).signature).toBe(signature); + }); + + it.each([ + [ + 'a file is added to a subfolder', + (folder: string) => fs.writeFileSync(`${folder}/lib/sub/new.js`, 'new') + ], + ['a file is deleted', (folder: string) => fs.rmSync(`${folder}/lib/sub/util.js`)], + [ + 'a file in a subfolder is replaced by a rename', + (folder: string) => { + fs.writeFileSync(`${folder}/lib/sub/util.js.tmp`, 'util 2'); + fs.renameSync(`${folder}/lib/sub/util.js.tmp`, `${folder}/lib/sub/util.js`); + } + ], + [ + 'a folder is recreated', + (folder: string) => { + fs.rmSync(`${folder}/lib/sub`, { recursive: true }); + fs.mkdirSync(`${folder}/lib/sub`); + fs.writeFileSync(`${folder}/lib/sub/util.js`, 'util'); + } + ], + ['a missing output folder is created', (folder: string) => fs.mkdirSync(`${folder}/lib-esm`)] + ])('changes if %s', async (description: string, change: (folder: string) => void) => { + const { signature } = await readAsync(); + // Folder modification times can have a coarse resolution. + await new Promise((resolve) => setTimeout(resolve, 20)); + change(projectFolder); + expect((await readAsync()).signature).not.toBe(signature); + }); +}); diff --git a/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerIncrementalGuard.test.ts b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerIncrementalGuard.test.ts new file mode 100644 index 0000000000..2ad766f703 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerIncrementalGuard.test.ts @@ -0,0 +1,213 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type * as childProcess from 'node:child_process'; +import { EventEmitter } from 'node:events'; +import { PassThrough } from 'node:stream'; + +import { + StringBufferTerminalProvider, + Terminal, + type ITerminal, + type ITerminalProvider +} from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import { Utilities } from '../../../utilities/Utilities'; +import type { IOperationRunnerContext } from '../IOperationRunner'; +import { OperationStatus } from '../OperationStatus'; +import { ShellOperationRunner } from '../ShellOperationRunner'; +import { + getCommandExecution, + setIncrementalExecutionGuard, + type IIncrementalExecutionGuard +} from '../IncrementalExecutionState'; + +const INITIAL_COMMAND: string = 'node build.js'; +const INCREMENTAL_COMMAND: string = 'node build.js --incremental'; + +interface ITestRun { + readonly status: OperationStatus; + readonly commands: ReadonlyArray; + readonly output: string; + readonly context: IOperationRunnerContext; +} + +interface ITestRunOptions { + readonly requiresGuard: boolean; + readonly guard?: IIncrementalExecutionGuard; + readonly hasLastState?: boolean; + readonly exitCodeByCommand?: Readonly>; +} + +async function runAsync(options: ITestRunOptions): Promise { + const { requiresGuard, guard, hasLastState = true, exitCodeByCommand = {} } = options; + const commands: string[] = []; + const executeSpy: jest.SpyInstance = jest + .spyOn(Utilities, 'executeLifecycleCommandAsync') + .mockImplementation((command: string) => { + commands.push(command); + const stdout: PassThrough = new PassThrough(); + const stderr: PassThrough = new PassThrough(); + const child: childProcess.ChildProcess = Object.assign(new EventEmitter(), { + stdout, + stderr, + stdio: [] + }) as unknown as childProcess.ChildProcess; + queueMicrotask(() => { + stdout.end(); + stderr.end(); + child.emit('close', exitCodeByCommand[command] ?? 0, null); + }); + return child; + }); + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const context: IOperationRunnerContext = { + environment: undefined, + createChildProcessReporter: jest.fn(), + async runWithTerminalAsync( + callback: ( + terminal: ITerminal, + operationTerminalProvider: ITerminalProvider, + structuredChildOutputTerminalProvider: ITerminalProvider + ) => Promise + ): Promise { + return await callback(new Terminal(terminalProvider), terminalProvider, terminalProvider); + } + } as unknown as IOperationRunnerContext; + if (guard) { + setIncrementalExecutionGuard(context, guard); + } + const runner: ShellOperationRunner = new ShellOperationRunner({ + phase: { allowWarningsOnSuccess: false } as IPhase, + rushProject: { + projectFolder: process.cwd(), + rushConfiguration: { commonTempFolder: process.cwd() } + } as RushConfigurationProject, + displayName: 'a (build)', + initialCommand: INITIAL_COMMAND, + incrementalCommand: INCREMENTAL_COMMAND, + incrementalCommandRequiresGuard: requiresGuard, + commandForHash: INITIAL_COMMAND, + ignoredParameterValues: [] + }); + try { + const status: OperationStatus = await runner.executeAsync( + context, + hasLastState ? { status: OperationStatus.Success } : undefined + ); + return { status, commands, output: terminalProvider.getOutput(), context }; + } finally { + executeSpy.mockRestore(); + } +} + +function createGuard( + blockReason: string | undefined, + rerunReason: string | undefined = undefined +): IIncrementalExecutionGuard & { + getBlockReasonAsync: jest.Mock; + verifyIncrementalResultAsync: jest.Mock; +} { + return { + getBlockReasonAsync: jest.fn(async () => blockReason), + verifyIncrementalResultAsync: jest.fn(async () => rerunReason) + }; +} + +describe(`${ShellOperationRunner.name} with an incremental command that requires a guard`, () => { + it('runs the initial command if no guard is registered', async () => { + const { status, commands, output, context } = await runAsync({ requiresGuard: true }); + expect(status).toBe(OperationStatus.Success); + expect(commands).toEqual([INITIAL_COMMAND]); + expect(output).toContain(`Invoking (initial): ${INITIAL_COMMAND}`); + expect(getCommandExecution(context)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + }); + + it('runs the initial command without asking the guard if the operation has no last state', async () => { + const guard: ReturnType = createGuard(undefined); + const { commands } = await runAsync({ requiresGuard: true, guard, hasLastState: false }); + expect(commands).toEqual([INITIAL_COMMAND]); + expect(guard.getBlockReasonAsync).not.toHaveBeenCalled(); + }); + + it('runs the initial command and says why if the guard blocks the incremental command', async () => { + const guard: ReturnType = createGuard('input files were added, deleted or renamed'); + const { status, commands, output, context } = await runAsync({ requiresGuard: true, guard }); + expect(status).toBe(OperationStatus.Success); + expect(commands).toEqual([INITIAL_COMMAND]); + expect(output).toContain( + 'Not using the incremental command because input files were added, deleted or renamed.' + ); + expect(guard.verifyIncrementalResultAsync).not.toHaveBeenCalled(); + expect(getCommandExecution(context)?.kind).toBe('initial'); + }); + + it('runs only the incremental command if the guard allows it and its outputs are verified', async () => { + const guard: ReturnType = createGuard(undefined); + const { status, commands, output, context } = await runAsync({ requiresGuard: true, guard }); + expect(status).toBe(OperationStatus.Success); + expect(commands).toEqual([INCREMENTAL_COMMAND]); + expect(output).toContain(`Invoking (incremental): ${INCREMENTAL_COMMAND}`); + expect(guard.verifyIncrementalResultAsync).toHaveBeenCalledTimes(1); + expect(getCommandExecution(context)).toEqual({ kind: 'incremental', hasIncrementalCommand: true }); + }); + + it('runs the initial command after the incremental command if its outputs are not verified', async () => { + const guard: ReturnType = createGuard( + undefined, + 'the incremental command changed which output files it has: added "lib/b.js"' + ); + const { status, commands, output, context } = await runAsync({ requiresGuard: true, guard }); + expect(status).toBe(OperationStatus.Success); + expect(commands).toEqual([INCREMENTAL_COMMAND, INITIAL_COMMAND]); + expect(output).toContain( + 'Running the initial command, because the incremental command changed which output files it has: added "lib/b.js".' + ); + expect(getCommandExecution(context)?.kind).toBe('initial'); + }); + + it('does not verify or repeat a failed incremental command', async () => { + const guard: ReturnType = createGuard(undefined); + const { status, commands, context } = await runAsync({ + requiresGuard: true, + guard, + exitCodeByCommand: { [INCREMENTAL_COMMAND]: 1 } + }); + expect(status).toBe(OperationStatus.Failure); + expect(commands).toEqual([INCREMENTAL_COMMAND]); + expect(guard.verifyIncrementalResultAsync).not.toHaveBeenCalled(); + expect(getCommandExecution(context)?.kind).toBe('incremental'); + }); + + it('runs the initial command if the guard fails', async () => { + const guard: IIncrementalExecutionGuard = { + getBlockReasonAsync: async () => { + throw new Error('EACCES: permission denied'); + }, + verifyIncrementalResultAsync: async () => undefined + }; + const { status, commands, output } = await runAsync({ requiresGuard: true, guard }); + expect(status).toBe(OperationStatus.Success); + expect(commands).toEqual([INITIAL_COMMAND]); + expect(output).toContain( + 'Not using the incremental command because its incremental execution guard failed: Error: EACCES: permission denied.' + ); + }); +}); + +describe(`${ShellOperationRunner.name} with an incremental command that does not require a guard`, () => { + it('runs the incremental command for an operation with a last state, as in watch mode, without recording it', async () => { + const guard: ReturnType = createGuard('its command line changed'); + const { commands, context } = await runAsync({ requiresGuard: false, guard }); + expect(commands).toEqual([INCREMENTAL_COMMAND]); + expect(guard.getBlockReasonAsync).not.toHaveBeenCalled(); + expect(getCommandExecution(context)).toBeUndefined(); + }); + + it('runs the initial command for an operation without a last state', async () => { + const { commands } = await runAsync({ requiresGuard: false, hasLastState: false }); + expect(commands).toEqual([INITIAL_COMMAND]); + }); +}); diff --git a/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts index b93ecb3d91..78237670cb 100644 --- a/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/ShellOperationRunnerPlugin.test.ts @@ -38,6 +38,7 @@ import { import { RushProjectConfiguration } from '../../../api/RushProjectConfiguration'; import { defineCustomParameters } from '../../../cli/parsing/defineCustomParameters'; import { associateParametersByPhase } from '../../../cli/parsing/associateParametersByPhase'; +import { getCommandExecution, setIncrementalExecutionGuard } from '../IncrementalExecutionState'; interface ISerializedOperation { name: string; @@ -349,4 +350,119 @@ describe(ShellOperationRunnerPlugin.name, () => { expect(commands).toEqual(expectedCommands); } ); + + describe('outside watch mode', () => { + async function runTwiceAsync(options: { + scripts: Record; + shellCommand?: string; + guarded: boolean; + }): Promise<{ commands: string[]; hasIncrementalCommand: boolean | undefined }> { + const phase: IPhase = { + name: '_phase:build', + isSynthetic: false, + missingScriptBehavior: 'error', + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + shellCommand: options.shellCommand + } as unknown as IPhase; + const project: RushConfigurationProject = { + packageName: 'a', + projectFolder: process.cwd(), + packageJson: { scripts: options.scripts }, + rushConfiguration: { commonTempFolder: process.cwd() } + } as unknown as RushConfigurationProject; + const operation: Operation = new Operation({ phase, project, logFilenameIdentifier: 'a' }); + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new ShellOperationRunnerPlugin().apply(hooks); + await hooks.createOperationsAsync.promise(new Set([operation]), { + isIncrementalBuildAllowed: true, + isWatch: false + } as unknown as ICreateOperationsContext); + + const commands: string[] = []; + const executeSpy = jest + .spyOn(Utilities, 'executeLifecycleCommandAsync') + .mockImplementation((command) => { + commands.push(command.trim()); + const stdout: PassThrough = new PassThrough(); + const stderr: PassThrough = new PassThrough(); + const child: childProcess.ChildProcess = Object.assign(new EventEmitter(), { + stdout, + stderr, + stdio: [] + }) as unknown as childProcess.ChildProcess; + queueMicrotask(() => { + stdout.end(); + stderr.end(); + child.emit('close', 0, null); + }); + return child; + }); + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const context: IOperationRunnerContext = { + environment: undefined, + async runWithTerminalAsync( + callback: ( + terminal: ITerminal, + operationTerminalProvider: ITerminalProvider, + structuredChildOutputTerminalProvider: ITerminalProvider + ) => Promise + ): Promise { + return await callback(new Terminal(terminalProvider), terminalProvider, terminalProvider); + } + } as unknown as IOperationRunnerContext; + if (options.guarded) { + setIncrementalExecutionGuard(context, { + getBlockReasonAsync: async () => undefined, + verifyIncrementalResultAsync: async () => undefined + }); + } + try { + await expect(operation.runner!.executeAsync(context)).resolves.toBe(OperationStatus.Success); + await expect( + operation.runner!.executeAsync(context, { status: OperationStatus.Success }) + ).resolves.toBe(OperationStatus.Success); + } finally { + executeSpy.mockRestore(); + } + return { commands, hasIncrementalCommand: getCommandExecution(context)?.hasIncrementalCommand }; + } + + it('runs the :incremental script for a repeated operation when its guard allows it', async () => { + const { commands, hasIncrementalCommand } = await runTwiceAsync({ + scripts: { + '_phase:build': 'node build.js', + '_phase:build:incremental': 'node build.js --incremental' + }, + guarded: true + }); + expect(commands).toEqual(['node build.js', 'node build.js --incremental']); + expect(hasIncrementalCommand).toBe(true); + }); + + it('does not treat an :incremental script that equals the initial script as an incremental command', async () => { + const { commands, hasIncrementalCommand } = await runTwiceAsync({ + scripts: { + '_phase:build': 'node build.js', + '_phase:build:incremental': 'node build.js' + }, + guarded: true + }); + expect(commands).toEqual(['node build.js', 'node build.js']); + expect(hasIncrementalCommand).toBe(false); + }); + + it('does not use the :incremental script of an operation with a shellCommand', async () => { + const { commands, hasIncrementalCommand } = await runTwiceAsync({ + scripts: { + '_phase:build': 'node build.js', + '_phase:build:incremental': 'node build.js --incremental' + }, + shellCommand: 'node custom.js', + guarded: true + }); + expect(commands).toEqual(['node custom.js', 'node custom.js']); + expect(hasIncrementalCommand).toBe(false); + }); + }); }); diff --git a/libraries/rush-lib/src/schemas/rush.schema.json b/libraries/rush-lib/src/schemas/rush.schema.json index b702606300..ed1b151020 100644 --- a/libraries/rush-lib/src/schemas/rush.schema.json +++ b/libraries/rush-lib/src/schemas/rush.schema.json @@ -277,6 +277,11 @@ "default": false, "description": "Enable explicit operationSettings[].daemonIpc Node launchers for incremental daemon builds. Never enables watch mode or unrequested execution. RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS overrides." }, + "incrementalBuilds": { + "type": "boolean", + "default": true, + "description": "Let daemon builds run an operation's `:incremental` script on top of the outputs of its last successful run in the daemon, when only files that it builds were edited since then and its output folders are unchanged. Otherwise the initial script runs, as it does for native Rush. Results of an incremental script are not written to the build cache. RUSH_DAEMON_INCREMENTAL_BUILDS overrides." + }, "queueTimeoutSeconds": { "type": "number", "minimum": 0, From 9b5b7ea4b0ee3fb97b9063a43bc838daf76df425 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 22:05:08 +0000 Subject: [PATCH 040/265] [rush-lib] Let the daemon run :incremental scripts when a guard allows it Swarm integration step 25; original commit 7d0ffb955b (merge of swarm/r06-t106 at b767c64dac). Scope: task 106. Brings task 106's test-only tip onto step 1 (31a8db3e65). With task 106 the daemon runs an operation's :incremental script, on top of the outputs of its last successful run in the daemon, when only files that the operation builds were edited since then and its output folders are unchanged; anything else runs the initial command. A result built on an incremental base is never written to the build cache. Kill switch: daemon.incrementalBuilds false in rush.json, or RUSH_DAEMON_INCREMENTAL_BUILDS=0. Task 84 passes getOperationEnvironment to the before/after iteration hooks. Second agents: t03 CONFIRMED (board 2067, 37 of 39; TAMPERI is the declared limitation) and t04 CONFIRMED (board 2100). ch01 on this tree (board 2161): rush-lib 1228/0, rush-daemon 516/0, rush-cli-client 367/0; 50 mutants, 42 killed; 33 of 33 end-to-end rows against a native cache-off oracle. Known costs, per ch01's rulings in board 2161: revisiting a state restores less from the cache (Rule 2), and an in-place rewrite of an output file by another writer is not detected (TAMPERI; task 139, r06's 641c39f120 in board 2171). Gate: ch01 GATE OK board 2161 (tree b1c0c2e690) Commits folded into this step (1): - b767c64dac [rush-lib] Test that an incremental result taints cache writes through an operation without a script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...ableOperationPluginRetainedResults.test.ts | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts index 172a2e76e1..ef643be028 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts @@ -61,6 +61,7 @@ import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRun import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; import type { OperationExecutionRecord } from '../OperationExecutionRecord'; import { setCommandExecution } from '../IncrementalExecutionState'; +import { NullOperationRunner } from '../NullOperationRunner'; const mockPhase: IPhase = { name: 'phase', @@ -123,6 +124,10 @@ interface ITestGraph { } interface ITestGraphOptions { + /** + * The names of the operations that have no script, like a phase whose script is missing + */ + noOpNames?: ReadonlySet; /** * The names of the dependencies of each operation. By default, each operation depends on the previous one. */ @@ -161,7 +166,9 @@ async function createTestGraphAsync(names: string[], options: ITestGraphOptions getCacheDisabledReason: () => undefined } as unknown as RushProjectConfiguration); const operation: Operation = new Operation({ - runner: new CacheableMockRunner(name, executions, incrementalNames), + runner: options.noOpNames?.has(name) + ? new NullOperationRunner({ name, result: OperationStatus.NoOp, silent: true }) + : new CacheableMockRunner(name, executions, incrementalNames), logFilenameIdentifier: name, phase: mockPhase, project @@ -544,6 +551,30 @@ describe(`${CacheableOperationPlugin.name} retained results`, () => { expect(testGraph.cacheWrites).toEqual([]); }); + it('does not write results built against an incremental result through an operation without a script', async () => { + // "a" <- "b-lite" <- "b", like the phases of the rushstack repo: the build of a project depends only on a phase of + // its own project that has no script and depends on the builds of upstream projects. + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b-lite', 'b'], { + noOpNames: new Set(['b-lite']) + }); + await testGraph.executeAsync(); + expect(testGraph.cacheWrites).toEqual(['a', 'b']); + + testGraph.localHashes.set('a', 'a-v2'); + testGraph.incrementalNames.add('a'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a:incremental', 'b']); + expect(testGraph.cacheWrites).toEqual([]); + + // --only b: "b" is built against the retained incremental result of "a", through the retained "b-lite". + testGraph.localHashes.set('b', 'b-v2'); + testGraph.operations.get('a')!.enabled = false; + testGraph.operations.get('b-lite')!.enabled = false; + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['b']); + expect(testGraph.cacheWrites).toEqual([]); + }); + it('restores a consumer of an incremental result from the build cache and trusts it as a cacheable result', async () => { // "a" <- "b" <- "c" const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); From 060e7eea939428dc62bce29769469973202e1d7c Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 22:07:54 +0000 Subject: [PATCH 041/265] [rush-lib] Test that a configuration is read again while only its rig.json may still be changing Swarm integration step 26; original commit 8a1d1ee8dd (merge of swarm/r07-t93-nit at f5826544d1). Scope: task 93 NIT. Test-only: 7 lines in RushProjectConfiguration.test.ts, no source change. It kills ch01's task 93 mutant M13 (board 1869): with the mutant the new assertion fails, 25/1, and 18900fe380's test file lets it survive, 26/0. Full rush-lib on the cherry-pick onto b1c0c2e690: 1228/0 (board 2178). The deployed daemon files don't change, so snapshot s14 (7d0ffb955b) is unaffected. Gate: ch01 GATE OK board 2178 (tree 09d90522de) Commits folded into this step (1): - f5826544d1 [rush-lib] Test that a configuration is read again while only its rig.json may still be changing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../rush-lib/src/api/test/RushProjectConfiguration.test.ts | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts b/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts index bdb3fe6bdb..6232aa0e0d 100644 --- a/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/RushProjectConfiguration.test.ts @@ -468,6 +468,13 @@ describe(RushProjectConfiguration.name, () => { expect(await loadTwiceAsync()).toBeGreaterThan(0); fs.utimesSync(rigFilePath, 1_000_000, 1_000_000); expect(await loadTwiceAsync()).toBe(0); + + // The same holds when rig.json is the only file that may still be changing. + const rigJsonPath: string = path.join(folder, 'rigged/config/rig.json'); + fs.utimesSync(rigJsonPath, future, future); + expect(await loadTwiceAsync()).toBeGreaterThan(0); + fs.utimesSync(rigJsonPath, 1_000_000, 1_000_000); + expect(await loadTwiceAsync()).toBe(0); }); it('reports errors and warnings on every call', async () => { From e671434a19fa0337971a77e35d199191d2746eeb Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 22:21:42 +0000 Subject: [PATCH 042/265] [rush-lib] The incremental guard detects an in-place rewrite of an output file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swarm integration step 27; original commit 6eae8fadad (merge of swarm/r06-t106 at 641c39f120). Scope: task 139. Brings r06's fold-in on task 106 (board 2171), in place of b767c64dac. The output signature of an operation now records each output file's inode, size and mtime as well as the folder listing, so when another writer appends to or rewrites an output file in place, the next build runs the initial command ("its output folders changed since its last successful run") instead of an incremental run on the stale file. This closes t03's TAMPERI. A touch that changes only mtime now also costs one initial run. r06 measured about 4 µs per output file per read. ch01 on tree 1eb87abdd2 (board 2211): rush-lib 1231/0, rush-daemon 516/0, rush-cli-client 367/0; b767c64dac's manifest fails 4 of the new tests; 10 mutants, 6 killed (N1, N3 and N6 survive; task 138 adds their tests); tamper end-to-end rows 9/9 against 2/9 before, full rows 33/33, cache-off rows 7/7. On this branch the tree is 1eb87abdd2 plus r07's test-only 8a1d1ee8dd change, byte for byte. Gate: ch01 GATE OK board 2211 (tree 1eb87abdd2) Commits folded into this step (1): - 641c39f120 [rush-lib] Detect in-place rewrites of output files in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../operations/OperationOutputManifest.ts | 62 +++++++++---------- .../IncrementalExecutionGuardPlugin.test.ts | 14 +++++ .../test/OperationOutputManifest.test.ts | 23 ++++++- 3 files changed, 63 insertions(+), 36 deletions(-) diff --git a/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts b/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts index 99932a9350..006b1d6c05 100644 --- a/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts +++ b/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts @@ -2,8 +2,8 @@ // See LICENSE in the project root for license information. import { createHash, type Hash } from 'node:crypto'; -import type * as fs from 'node:fs'; -import { lstat, readdir } from 'node:fs/promises'; +import * as fs from 'node:fs'; +import { readdir } from 'node:fs/promises'; import * as path from 'node:path'; /** @@ -11,9 +11,10 @@ import * as path from 'node:path'; */ export interface IOperationOutputManifest { /** - * Covers the path of every file and folder in the output folders, and the identity, modification time and status - * change time of every folder. It changes if an output is added, deleted or renamed, if a file is replaced by a - * rename (as atomic writes do), or if a folder is recreated. It does not change if a file is rewritten in place. + * Covers the path of every file and folder in the output folders, the identity, size and modification time of + * every file, and the identity, modification time and status change time of every folder. It changes if an output + * is added, deleted, renamed or rewritten, including in place, or if a folder is recreated. It does not change if + * a hard link to a file is created or removed elsewhere. */ readonly signature: string; /** @@ -55,27 +56,21 @@ export async function readOperationOutputManifestAsync( const readFolderAsync = async (relativeFolder: string, stats: fs.Stats): Promise => { entries.push(`${relativeFolder}/ ${stats.ino} ${stats.mtimeMs} ${stats.ctimeMs}`); - const children: fs.Dirent[] = await limitAsync(() => - tryReaddirAsync(path.resolve(projectFolder, relativeFolder)) - ); + const folderPath: string = path.resolve(projectFolder, relativeFolder); + const children: fs.Dirent[] = await limitAsync(() => tryReaddirAsync(folderPath)); const subfolderPromises: Promise[] = []; for (const child of children) { const relativePath: string = `${relativeFolder}/${child.name}`; + const childStats: fs.Stats | undefined = lstatIfExists(`${folderPath}${path.sep}${child.name}`); if (child.isDirectory()) { - subfolderPromises.push( - limitAsync(() => tryLstatAsync(path.resolve(projectFolder, relativePath))).then( - async (childStats: fs.Stats | undefined) => { - if (childStats?.isDirectory()) { - await readFolderAsync(relativePath, childStats); - } else { - // It was deleted or replaced after its parent was read. - entries.push(`${relativePath}/ replaced`); - } - } - ) - ); + if (childStats?.isDirectory()) { + subfolderPromises.push(readFolderAsync(relativePath, childStats)); + } else { + // It was deleted or replaced after its parent was read. + entries.push(`${relativePath}/ replaced`); + } } else { - entries.push(relativePath); + entries.push(getFileEntry(relativePath, childStats)); files.add(relativePath); } } @@ -85,15 +80,13 @@ export async function readOperationOutputManifestAsync( await Promise.all( outputFolderNames.map(async (folderName: string) => { const relativePath: string = folderName.replace(/\\/g, '/').replace(/\/+$/, ''); - const stats: fs.Stats | undefined = await limitAsync(() => - tryLstatAsync(path.resolve(projectFolder, relativePath)) - ); + const stats: fs.Stats | undefined = lstatIfExists(path.resolve(projectFolder, relativePath)); if (!stats) { entries.push(`missing ${relativePath}`); } else if (stats.isDirectory()) { await readFolderAsync(relativePath, stats); } else { - entries.push(`${relativePath} ${stats.ino} ${stats.mtimeMs} ${stats.ctimeMs} ${stats.size}`); + entries.push(getFileEntry(relativePath, stats)); files.add(relativePath); } }) @@ -170,6 +163,12 @@ export function describeOutputFileChanges( .join(', '); } +// The status change time of a file is left out, because it also changes when a hard link to the file is created or +// removed elsewhere, e.g. by a tool that links outputs into another folder. +function getFileEntry(relativePath: string, stats: fs.Stats | undefined): string { + return stats ? `${relativePath} ${stats.ino} ${stats.size} ${stats.mtimeMs}` : `${relativePath} deleted`; +} + function createConcurrencyLimiter(maxConcurrency: number): (fn: () => Promise) => Promise { let active: number = 0; const waiting: (() => void)[] = []; @@ -193,15 +192,10 @@ function createConcurrencyLimiter(maxConcurrency: number): (fn: () => Promise }; } -async function tryLstatAsync(filePath: string): Promise { - try { - return await lstat(filePath); - } catch (error) { - if ((error as NodeJS.ErrnoException).code === 'ENOENT') { - return undefined; - } - throw error; - } +// Synchronous, because every output file is stat'ed and an lstat call takes a few microseconds, far less than a +// round trip through the thread pool. The event loop still runs between folders, which are read asynchronously. +function lstatIfExists(filePath: string): fs.Stats | undefined { + return fs.lstatSync(filePath, { throwIfNoEntry: false }); } async function tryReaddirAsync(folderPath: string): Promise { diff --git a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts index 02b33ea377..9e954a0013 100644 --- a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts @@ -567,6 +567,20 @@ describe(IncrementalExecutionGuardPlugin.name, () => { ); }); + it('runs the initial command if an output file was rewritten in place since the last run', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + // E.g. a debugging edit through the symbolic link to the project in node_modules, which keeps the inode. + fs.appendFileSync(`${workspace.rootFolder}/a/lib/sub/two.js`, '\nconsole.log("debug");'); + workspace.writeFile('a/src/one.ts', 'one 2'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain( + 'Not using the incremental command because its output folders changed since its last successful run.' + ); + }); + it('always runs the initial command of an operation that builds a bundle', async () => { const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a', isBundle: true }]); await workspace.executeAsync(); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts index 83582351ea..6001b6ee28 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts @@ -87,9 +87,28 @@ describe(readOperationOutputManifestAsync.name, () => { expect((await readAsync()).signature).toBe(manifest.signature); }); - it('does not change if a file is rewritten in place', async () => { + it.each([ + ['with the same size', (filePath: string) => fs.writeFileSync(filePath, 'utiL')], + ['by an append', (filePath: string) => fs.appendFileSync(filePath, ' 2')] + ])( + 'changes if a file is rewritten in place %s', + async (description: string, rewrite: (filePath: string) => void) => { + const filePath: string = `${projectFolder}/lib/sub/util.js`; + const { ino } = fs.statSync(filePath); + const { signature } = await readAsync(); + // File modification times can have a coarse resolution. + await new Promise((resolve) => setTimeout(resolve, 20)); + rewrite(filePath); + expect(fs.statSync(filePath).ino).toBe(ino); + expect((await readAsync()).signature).not.toBe(signature); + } + ); + + it('does not change if a hard link to a file is created elsewhere', async () => { const { signature } = await readAsync(); - fs.writeFileSync(`${projectFolder}/lib/sub/util.js`, 'util 2'); + await new Promise((resolve) => setTimeout(resolve, 20)); + fs.linkSync(`${projectFolder}/lib/sub/util.js`, `${projectFolder}/util.js`); + fs.linkSync(`${projectFolder}/tsconfig.tsbuildinfo`, `${projectFolder}/tsbuildinfo.json`); expect((await readAsync()).signature).toBe(signature); }); From 851e3f2c0271dbcc41f75e9e7a53586e7c9855fb Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 23:23:39 +0000 Subject: [PATCH 043/265] [rush-lib] Tests for the incremental guard's configuration inputs, content-hashed outputs, invalidation and skip record Swarm integration step 28; original commit 0d30a5164c (merge of swarm/r06-t138 at dbd5d99147). Scope: task 138 and t106d. Brings r06's task 138 tests (board 2302) and t106d (4c9f23e650, board 2233). Only 3 test files change (+204/-18): ProductionDaemonRequestResolver.test.ts, IncrementalExecutionGuardPlugin.test.ts and OperationOutputManifest.test.ts. No product code changes, so no snapshot is needed. ch01 on tree 45c2c5edd5 (board 2364): rush-lib 1241/0 (1 skipped); rush-daemon 516/0 with t106d. With the new tests, each of the 10 guard and manifest mutants (G9-G12, G19, G24, L1, N1, N3, N6) fails its intended test; all 10 pass the old tests. t106d's kill-switch test fails against a product mutant that drops the daemon.incrementalBuilds check at PhasedScriptAction.ts:666, and 6eae8fadad's version of it passed that mutant. Gate: ch01 GATE OK board 2364 (tree 45c2c5edd5) Commits folded into this step (2): - 4c9f23e650 [rush-daemon] Make the test of daemon.incrementalBuilds: false edit a source file - dbd5d99147 [rush-lib] Test the incremental guard's configuration inputs, content-hashed outputs, invalidation and skip record Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../ProductionDaemonRequestResolver.test.ts | 6 +- .../IncrementalExecutionGuardPlugin.test.ts | 180 ++++++++++++++++-- .../test/OperationOutputManifest.test.ts | 36 ++++ 3 files changed, 204 insertions(+), 18 deletions(-) diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 03381a9d62..f307a3a75d 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -1517,19 +1517,23 @@ process.exit(23); incrementalScript: true, incrementalBuilds: false }); + // With incremental builds on, an edit of this source file runs the incremental script (see the next test). + const inputPath: string = path.join(fixture.repoRoot, 'projects/a/src/input.txt'); try { + fs.mkdirSync(path.dirname(inputPath), { recursive: true }); for (const [requestId, input] of [ ['initial-script-1', 'one'], ['initial-script-2', 'two'], ['initial-script-3', 'three'] ]) { - fs.writeFileSync(path.join(fixture.repoRoot, 'projects/a/input.txt'), input); + fs.writeFileSync(inputPath, input); const exchange: ITerminalExchange = await runAsync(fixture, requestId, ['build', '--only', 'a']); expect(exchange.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0, operationResults: [{ operationId: 'a (compile)', status: 'SUCCESS' }] } }); expect(logText(exchange)).toContain('Invoking (initial): node build.cjs'); + expect(logText(exchange)).not.toContain('Not using the incremental command'); } // A watch-only incremental script can keep outputs of deleted inputs, and its output would be cached // under the key of the initial script that native Rush runs. diff --git a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts index 9e954a0013..425705774f 100644 --- a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts @@ -48,7 +48,7 @@ import { PassThrough } from 'node:stream'; import { LookupByPath } from '@rushstack/lookup-by-path'; import { SubprocessTerminator } from '@rushstack/node-core-library'; -import { MockWritable } from '@rushstack/terminal'; +import { MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import type { IPhase } from '../../../api/CommandLineConfiguration'; import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; @@ -62,6 +62,7 @@ import { NATIVE_COMMAND_INVALIDATION_REASON } from '../IncrementalExecutionState'; import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; +import { LegacySkipPlugin } from '../LegacySkipPlugin'; import { NullOperationRunner } from '../NullOperationRunner'; import { Operation } from '../Operation'; import type { OperationExecutionRecord } from '../OperationExecutionRecord'; @@ -105,6 +106,10 @@ interface IProjectSpec { * If set, the project has a `profiles` folder, like a rig package. */ readonly isRig?: boolean; + /** + * Files outside of the project that its build depends on, like `dependsOnAdditionalFiles` in rush-project.json + */ + readonly additionalFiles?: ReadonlyArray; } interface IWorkspaceOptions { @@ -113,6 +118,10 @@ interface IWorkspaceOptions { * operation of its own project, which has no script and depends on the builds of the project's dependencies. */ readonly hasPassThroughPhase?: boolean; + /** + * If set, applies the skip detection that Rush uses when the build cache is not enabled. + */ + readonly hasLegacySkipDetection?: boolean; } interface ITestIteration { @@ -204,7 +213,7 @@ function build(projectFolder: string, isBundle: boolean, isIncremental: boolean) async function createWorkspaceAsync( projectSpecs: ReadonlyArray, - { hasPassThroughPhase }: IWorkspaceOptions = {} + { hasPassThroughPhase, hasLegacySkipDetection }: IWorkspaceOptions = {} ): Promise { const rootFolder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-incremental-guard-')); workspaceFolders.push(rootFolder); @@ -256,8 +265,17 @@ async function createWorkspaceAsync( const projectConfigurations: Map = new Map(); const lookupByPath: LookupByPath = new LookupByPath(); const outputFolderByPrefix: Map = new Map(); + const additionalFiles: Set = new Set(); for (const spec of projectSpecs) { - const { name, dependencies = [], devDependencies = [], isBundle, dependsOnEnvVars, isRig } = spec; + const { + name, + dependencies = [], + devDependencies = [], + isBundle, + dependsOnEnvVars, + isRig, + additionalFiles: projectAdditionalFiles = [] + } = spec; const projectFolder: string = `${rootFolder}/${name}`; const toVersions = (names: ReadonlyArray): Record => Object.fromEntries(names.map((dependencyName: string) => [dependencyName, 'workspace:*'])); @@ -274,11 +292,17 @@ async function createWorkspaceAsync( if (isRig) { writeFile(`${name}/profiles/default/config/heft.json`, '{}'); } + for (const file of projectAdditionalFiles) { + writeFile(file, '{}'); + additionalFiles.add(file); + } const project: RushConfigurationProject = { packageName: name, projectFolder, projectRelativeFolder: name, + // Outside of the project folder, so that its files are not inputs + projectRushTempFolder: `${rootFolder}/common/temp/projects/${name}`, packageJson, rushConfiguration: { commonTempFolder: `${rootFolder}/common/temp` } } as unknown as RushConfigurationProject; @@ -293,7 +317,10 @@ async function createWorkspaceAsync( getCacheDisabledReason: () => undefined } as unknown as RushProjectConfiguration; projectConfigurations.set(project, projectConfiguration); - projectMap.set(project, { projectConfig: projectConfiguration }); + projectMap.set(project, { + projectConfig: projectConfiguration, + additionalFilesByOperationName: new Map([[PHASE_NAME, new Set(projectAdditionalFiles)]]) + }); lookupByPath.setItem(name, project); outputFolderByPrefix.set(name, outputFolderName); specByFolder.set(projectFolder, spec); @@ -338,6 +365,13 @@ async function createWorkspaceAsync( const hooks: PhasedCommandHooks = new PhasedCommandHooks(); new PhasedOperationPlugin().apply(hooks); new IncrementalExecutionGuardPlugin().apply(hooks); + if (hasLegacySkipDetection) { + new LegacySkipPlugin({ + terminal: new Terminal(new StringBufferTerminalProvider()), + changedProjectsOnly: false, + isIncrementalBuildAllowed: true + }).apply(hooks); + } const destination: MockWritable = new MockWritable(); const graphOperations: Set = new Set([...operations.values(), ...passThroughOperations]); const graph: OperationGraph = new OperationGraph(graphOperations, { @@ -359,12 +393,18 @@ async function createWorkspaceAsync( // Like `git hash-object` for each file, except the outputs, which are ignored by git. const createInputsSnapshot = (environment: Readonly>): InputsSnapshot => { const hashes: Map = new Map(); + const hashFile = (file: string): void => { + const content: Buffer = fs.readFileSync(`${rootFolder}/${file}`); + hashes.set(file, createHash('sha1').update(content).digest('hex')); + }; for (const [prefix, outputFolderName] of outputFolderByPrefix) { for (const file of listFiles(`${rootFolder}/${prefix}`, new Set([outputFolderName]))) { - const content: Buffer = fs.readFileSync(`${rootFolder}/${prefix}/${file}`); - hashes.set(`${prefix}/${file}`, createHash('sha1').update(content).digest('hex')); + hashFile(`${prefix}/${file}`); } } + for (const file of additionalFiles) { + hashFile(file); + } return new InputsSnapshot({ rootDir: rootFolder, hashes, @@ -420,6 +460,14 @@ describe(IncrementalExecutionGuardPlugin.name, () => { }); interface IInputChangeCase { + /** + * The project, `{ name: 'a' }` by default + */ + readonly spec?: IProjectSpec; + /** + * Runs before the first build + */ + readonly prepare?: (workspace: ITestWorkspace) => void; readonly change: (workspace: ITestWorkspace) => void; readonly reason: string; /** @@ -464,20 +512,63 @@ describe(IncrementalExecutionGuardPlugin.name, () => { reason: 'a configuration file changed ("a/tsconfig.json")', sourceFile: 'a/src/one.ts' } + ], + [ + 'a configuration file in a subfolder changes', + { + prepare: (workspace: ITestWorkspace) => workspace.writeFile('a/test/tsconfig.json', '{}'), + change: (workspace: ITestWorkspace) => + workspace.writeFile('a/test/tsconfig.json', '{ "compilerOptions": {} }'), + reason: 'a configuration file changed ("a/test/tsconfig.json")', + sourceFile: 'a/src/one.ts' + } + ], + [ + 'a file in the root folder of the project changes', + { + prepare: (workspace: ITestWorkspace) => workspace.writeFile('a/build.js', '// build'), + change: (workspace: ITestWorkspace) => workspace.writeFile('a/build.js', '// build 2'), + reason: 'a configuration file changed ("a/build.js")', + sourceFile: 'a/src/one.ts' + } + ], + [ + 'a file in the config folder of the project changes', + { + prepare: (workspace: ITestWorkspace) => workspace.writeFile('a/config/heft.json', '{}'), + change: (workspace: ITestWorkspace) => + workspace.writeFile('a/config/heft.json', '{ "phasesByName": {} }'), + reason: 'a configuration file changed ("a/config/heft.json")', + sourceFile: 'a/src/one.ts' + } + ], + [ + 'a file outside of the project that it depends on changes', + { + spec: { name: 'a', additionalFiles: ['tools/shared/data.json'] }, + change: (workspace: ITestWorkspace) => + workspace.writeFile('tools/shared/data.json', '{ "edited": true }'), + reason: 'a configuration file changed ("tools/shared/data.json")', + sourceFile: 'a/src/one.ts' + } ] - ])('runs the initial command if %s', async (description: string, { change, reason, sourceFile }) => { - const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); - await workspace.executeAsync(); + ])( + 'runs the initial command if %s', + async (description: string, { spec = { name: 'a' }, prepare, change, reason, sourceFile }) => { + const workspace: ITestWorkspace = await createWorkspaceAsync([spec]); + prepare?.(workspace); + await workspace.executeAsync(); - change(workspace); - const changed: ITestIteration = await workspace.executeAsync(); - expect(changed.commands).toEqual(['a:initial']); - expect(changed.output).toContain(`Not using the incremental command because ${reason}.`); + change(workspace); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain(`Not using the incremental command because ${reason}.`); - // The result of the initial command is the new base. - workspace.writeFile(sourceFile, 'edited'); - expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); - }); + // The result of the initial command is the new base. + workspace.writeFile(sourceFile, 'edited'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + } + ); it('runs the initial command if an environment variable that the operation depends on changes', async () => { const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a', dependsOnEnvVars: ['MODE'] }]); @@ -616,6 +707,46 @@ describe(IncrementalExecutionGuardPlugin.name, () => { ); }); + it('runs the initial command after an incremental command that emitted a content-hashed file', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + // Like a chunk with webpack's default hash length. Comparing the output files ignores such names. + workspace.writeFile('a/src/one.ts', 'one emit:chunk_0123456789abcdef0123'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental', 'a:initial']); + expect(changed.getStatus('a')).toBe(OperationStatus.Success); + expect(changed.output).toContain( + 'Running the initial command, because its outputs include the content-hashed file "lib/chunk_0123456789abcdef0123.js".' + ); + + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + const next: ITestIteration = await workspace.executeAsync(); + expect(next.commands).toEqual(['a:initial']); + expect(next.output).toContain( + 'Not using the incremental command because its outputs include the content-hashed file "lib/chunk_0123456789abcdef0123.js".' + ); + }); + + it('does not let a later command skip an operation after its incremental command', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }], { + hasLegacySkipDetection: true + }); + const packageDepsPath: string = `${workspace.rootFolder}/common/temp/projects/a/package-deps__phase_build.json`; + expect((await workspace.executeAsync()).commands).toEqual(['a:initial']); + expect(fs.existsSync(packageDepsPath)).toBe(true); + + // The outputs of the incremental command can differ from those of the initial command, so a later command, + // e.g. one that does not use the Rush daemon, must not skip the operation. + workspace.writeFile('a/src/one.ts', 'one 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + expect(fs.existsSync(packageDepsPath)).toBe(false); + + workspace.graph.invalidateOperations(undefined, NATIVE_COMMAND_INVALIDATION_REASON); + expect((await workspace.executeAsync()).commands).toEqual(['a:initial']); + expect(fs.existsSync(packageDepsPath)).toBe(true); + }); + it('runs the initial command after a failure', async () => { const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); await workspace.executeAsync(); @@ -702,4 +833,19 @@ describe(IncrementalExecutionGuardPlugin.name, () => { workspace.writeFile('b/src/one.ts', 'one 3'); expect([...(await workspace.executeAsync()).commands].sort()).toEqual(['a:initial', 'b:initial']); }); + + it('forgets the base of an operation that is invalidated for another reason', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }, { name: 'b' }]); + await workspace.executeAsync(); + + // E.g. a client of the Rush daemon that invalidates the operation. + workspace.graph.invalidateOperations([workspace.operations.get('a')!], 'daemon graph invalidate'); + workspace.writeFile('a/src/one.ts', 'one 2'); + workspace.writeFile('b/src/one.ts', 'one 2'); + const next: ITestIteration = await workspace.executeAsync(); + expect([...next.commands].sort()).toEqual(['a:initial', 'b:incremental']); + expect(next.output).toContain( + 'Not using the incremental command because its outputs were not built by a successful run of its own command in this process.' + ); + }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts index 6001b6ee28..ae29e6be3c 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts @@ -58,6 +58,8 @@ describe(describeOutputFileChanges.name, () => { }); describe(readOperationOutputManifestAsync.name, () => { + // Whole seconds, so that the time is set exactly + const KEPT_MODIFICATION_TIME_SECONDS: number = 1_700_000_000; let projectFolder: string; beforeEach(() => { @@ -112,12 +114,46 @@ describe(readOperationOutputManifestAsync.name, () => { expect((await readAsync()).signature).toBe(signature); }); + // E.g. `cp -p`, `rsync -t` or `tar -x`, which give the files that they write the modification time of their source. + it.each([ + [ + 'a file is rewritten in place by an append', + (folder: string) => fs.appendFileSync(`${folder}/lib/sub/util.js`, ' 2') + ], + [ + 'an output file that is not in a folder is replaced by a rename with the same size', + (folder: string) => { + fs.writeFileSync(`${folder}/tsconfig.tsbuildinfo.tmp`, '[]'); + fs.renameSync(`${folder}/tsconfig.tsbuildinfo.tmp`, `${folder}/tsconfig.tsbuildinfo`); + } + ] + ])( + 'changes if %s and its modification time is kept', + async (description: string, change: (folder: string) => void) => { + const files: string[] = [`${projectFolder}/lib/sub/util.js`, `${projectFolder}/tsconfig.tsbuildinfo`]; + const keepModificationTimes = (): void => { + for (const file of files) { + fs.utimesSync(file, KEPT_MODIFICATION_TIME_SECONDS, KEPT_MODIFICATION_TIME_SECONDS); + } + }; + keepModificationTimes(); + const { signature } = await readAsync(); + change(projectFolder); + keepModificationTimes(); + expect((await readAsync()).signature).not.toBe(signature); + } + ); + it.each([ [ 'a file is added to a subfolder', (folder: string) => fs.writeFileSync(`${folder}/lib/sub/new.js`, 'new') ], ['a file is deleted', (folder: string) => fs.rmSync(`${folder}/lib/sub/util.js`)], + [ + 'an output file that is not in a folder is rewritten in place', + (folder: string) => fs.appendFileSync(`${folder}/tsconfig.tsbuildinfo`, ' ') + ], [ 'a file in a subfolder is replaced by a rename', (folder: string) => { From 53f3b7e07af24f2742aceab297b39f65517626fa Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 23:23:48 +0000 Subject: [PATCH 044/265] [rush-daemon] A served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it (board 113) (and 5 more) Swarm integration step 29; original commit 3857bc9926 (merge of swarm/r05-stack-int at c35a77cc5c). Brings r05's stack (board 1931): - Task 55: a served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it. - Task 7 part 3 text: admission timeouts name the client's wait timeout and the time that isn't counted. - Task 96: a rushx script that arrives while a restart is pending waits for the restart instead of postponing it, and admission failures in legacy output and rushx-client print the daemon's reason. - Task 98: a request spends its wait timeout only while it waits for other requests. - The race fix (c35a77cc5c, restacked from 9370662d5f): a client that follows a daemon restart connects to the successor that the restarting daemon launches instead of racing it. ch01 on tree b500e451d4 (board 2429): build rc 0 with 0 warnings; rush-daemon 544/0, rush-lib 1231/0, rush-cli-client 374/0, rush-client-core 129/0 on its own (2 known flakes in the full run), protocol 166/0, transport 73/0, reporter 519/0, wire-e2e 4/0. The red checks fail as intended without each change. 36 of 43 mutants are killed; the 7 survivors are test gaps and change no claim. The end-to-end rows R55, R96 and R96A pass on the stack and fail on s14b's files. Gate: ch01 GATE OK board 2429 (tree b500e451d4 on 6eae8fadad; 87e0e3a10f together with task 138's merge, in either order) Commits folded into this step (6): - c8cd36f574 [rush-daemon] A served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it (board 113) - 4a5a13bddf [rush-daemon] Name the client's wait timeout and uncounted time in admission timeouts - cafa954eb4 [rush-daemon] A rushx script that arrives while a restart is pending waits for it instead of postponing it (task 96) - 7ba7021ed6 [rush-cli-client] Admission failures in legacy output and rushx-client print the daemon's reason (task 96) - 4ddb1fdbfe [rush-daemon] Spend a request's wait timeout only while it waits for other requests (task 98) - c35a77cc5c [rush-client-core] A client that follows a daemon restart connects to the successor the restarting daemon launches instead of racing it (task 96) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 30 +- .../src/ClientAdmissionControls.ts | 26 +- apps/rush-cli-client/src/launchClient.ts | 20 +- apps/rush-cli-client/src/resultDiagnostics.ts | 20 +- .../src/test/ClientAdmissionControls.test.ts | 40 +++ .../src/test/resultDiagnostics.test.ts | 32 +- ...ssion-failure-reason_2026-09-28-17-39.json | 10 + ...paration-wait-budget_2026-09-28-18-20.json | 10 + ...rt-planned-successor_2026-09-28-19-50.json | 11 + ...rt-planned-successor_2026-09-28-19-50.json | 11 + ...rt-planned-successor_2026-09-28-19-50.json | 11 + ...mission-timeout-text_2026-09-28-16-02.json | 10 + ...paration-wait-budget_2026-09-28-18-20.json | 10 + ...ushx-pending-restart_2026-09-28-16-45.json | 10 + ...-served-rushx-reload_2026-09-28-15-31.json | 10 + libraries/rush-client-core/README.md | 4 + .../src/connectOrStartDaemon.ts | 50 +++ .../src/executeWithDaemonRestart.ts | 5 +- .../src/test/connectOrStartDaemon.test.ts | 97 ++++++ .../src/test/fixtures/daemon.ts | 51 ++- .../src/DaemonCommandResult.ts | 2 + libraries/rush-daemon/README.md | 31 +- .../rush-daemon/src/GlobalCommandRequest.ts | 3 +- .../src/WorkspaceRequestAdmission.ts | 188 +++++++---- .../src/WorkspaceRequestLifecycle.ts | 85 +++-- .../src/WorkspaceRestartArbiter.ts | 108 +++++-- .../src/test/DaemonGraphTestFixture.ts | 8 +- .../test/RequestAdmissionIntegration.test.ts | 59 +++- .../src/test/RestartDrainAdmission.test.ts | 130 +++++++- .../WorkspacePreparationAdmission.test.ts | 130 ++++++++ .../test/WorkspaceRequestAdmission.test.ts | 155 ++++++++- .../src/test/WorkspaceRestartArbiter.test.ts | 198 +++++++++++- .../WorkspaceServedScriptAdmission.test.ts | 305 ++++++++++++++++++ 33 files changed, 1713 insertions(+), 157 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r05-admission-failure-reason_2026-09-28-17-39.json create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json create mode 100644 common/changes/@rushstack/rush-client-core/restart-planned-successor_2026-09-28-19-50.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/restart-planned-successor_2026-09-28-19-50.json create mode 100644 common/changes/@rushstack/rush-daemon/restart-planned-successor_2026-09-28-19-50.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-admission-timeout-text_2026-09-28-16-02.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-rushx-pending-restart_2026-09-28-16-45.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-served-rushx-reload_2026-09-28-15-31.json create mode 100644 libraries/rush-daemon/src/test/WorkspacePreparationAdmission.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 676cae2822..2aef793974 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -63,15 +63,16 @@ rounded down to milliseconds. These controls are mutually exclusive and are consumed before forwarding, never appended to a project script. Arguments after `--` remain literal script arguments. -The queue timeout is measured from when the daemon receives the request, and only -time spent waiting for other requests counts against it, whether it comes from -`--wait-timeout`, `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, `daemon.queueTimeoutSeconds` -in `rush.json`, or the built-in 30-second default. Waiting while another request +Only time that the request spends waiting for other requests counts against the +queue timeout, whether it comes from `--wait-timeout`, +`RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, `daemon.queueTimeoutSeconds` in `rush.json`, +or the built-in 30-second default. Waiting while another request loads or reloads the workspace graph does not count, so every build that arrives while the first build after startup loads the graph runs once the load finishes. That wait fails after 10 times the timeout (5 minutes with the default), so a load -that never finishes does not hold other requests forever. The request's own routing -and execution do not count either. A configured or per-invocation timeout also +that never finishes does not hold other requests forever. The request's own work, +such as checking its inputs, loading the graph, routing and execution, does not +count either. A configured or per-invocation timeout also limits waiting for a running build that the request could not join, and waiting for the requests that the daemon is serving to finish before it restarts for the request's environment. The built-in default does not: with it, a build that arrives @@ -80,11 +81,18 @@ instead of failing after 30 seconds, and a request that needs a restart waits fo the requests that were running when it arrived to finish and then runs on the restarted daemon. The default still limits a restart wait while the daemon runs a `rushx` script, such as a dev server, which may not exit until it is stopped, and -while it serves requests that arrived later. `--no-wait` fails wherever the request -would wait. On a timeout, the client exits with code 1 and suggests -`--wait-timeout`. It does not suggest exporting `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, -because Rush versions that do not recognize a `RUSH_` environment variable fail -every command while it is set. +while it serves requests that arrived later. A `rushx-client` script that arrives +while another request waits for the daemon to restart does not start on the old +daemon, where the restart would wait for it to exit: it waits for the restart and +then runs on the restarted daemon, and its timeout applies to that wait as it does +to the restart wait. `--no-wait` fails wherever the request would wait. On a +timeout, the client exits with code 1 and suggests `--wait-timeout`. It does not +suggest exporting `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, because Rush versions that do +not recognize a `RUSH_` environment variable fail every command while it is set. +In legacy output and in `rushx-client`, the admission failure line +(`rush-client: daemon admission failed (wait-timeout): …`, or `(no-wait)`) gives the +daemon's reason, as agent mode's summary line does, so it names what the request +waited for, such as a daemon restart. Admission controls also apply to experimental graph requests, but not `start|stop|restart|status|logs`. They affect daemon admission only; native fallback diff --git a/apps/rush-cli-client/src/ClientAdmissionControls.ts b/apps/rush-cli-client/src/ClientAdmissionControls.ts index bd3d04edeb..9be7c67b63 100644 --- a/apps/rush-cli-client/src/ClientAdmissionControls.ts +++ b/apps/rush-cli-client/src/ClientAdmissionControls.ts @@ -72,25 +72,39 @@ export function getConfiguredAdmission(options: IConfiguredAdmissionOptions): ID return options.explicit ? { waitTimeoutMs } : { waitTimeoutMs, waitTimeoutIsDefault: true }; } -/** Explains a daemon admission failure and how to wait longer. */ +// Only the per-invocation flag is offered: Rush versions that do not recognize the variable reject it. +const WAIT_LONGER_REMEDY: string = 'To wait longer, pass --wait-timeout .'; + +/** + * Explains a daemon admission failure and how to wait longer. + * + * @remarks + * The daemon's reason for the failure (`daemonMessage`), when present, replaces the generic explanation, + * because it names what the request waited for, such as a daemon restart. + */ export function formatAdmissionFailure( code: DaemonRequestAdmissionErrorCode, - admission: IDaemonRequestAdmissionOptions | undefined + admission: IDaemonRequestAdmissionOptions | undefined, + daemonMessage?: string ): string { const prefix: string = `rush-client: daemon admission failed (${code})`; if (code === 'no-wait') { - return `${prefix}: another daemon request is using this workspace and --no-wait was specified.\n`; + return daemonMessage + ? `${prefix}: ${daemonMessage}\n` + : `${prefix}: another daemon request is using this workspace and --no-wait was specified.\n`; } if (code === 'wait-timeout') { + if (daemonMessage) { + const remedy: string = daemonMessage.includes('--wait-timeout') ? '' : ` ${WAIT_LONGER_REMEDY}`; + return `${prefix}: ${daemonMessage}${remedy}\n`; + } const timeout: string = admission?.waitTimeoutMs === undefined ? '' : ` after its ${admission.waitTimeoutMs / 1000}s wait timeout`; return ( `${prefix}: timed out${timeout} waiting for another daemon request in this workspace to finish ` + - '(a command that needs exclusive access, or a running build). ' + - // Only the per-invocation flag is offered: Rush versions that do not recognize the variable reject it. - 'To wait longer, pass --wait-timeout .\n' + `(a command that needs exclusive access, or a running build). ${WAIT_LONGER_REMEDY}\n` ); } return `${prefix}.\n`; diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 7ff5cd2572..b2fbd46089 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -23,7 +23,7 @@ import type { DaemonVerbosity, IDaemonRequestEnvelope } from '@rushstack/rush-da import { ConsoleTerminalProvider } from '@rushstack/terminal'; import { executeDaemonCommandAsync } from './daemonCommands'; -import { formatAdmissionFailure, getConfiguredAdmission } from './ClientAdmissionControls'; +import { getConfiguredAdmission } from './ClientAdmissionControls'; import { ClientOperationRenderer } from './ClientOperationRenderer'; import type { AgentProgressRenderer } from './AgentProgressRenderer'; import { @@ -35,7 +35,7 @@ import { import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; -import { getResultDiagnostic } from './resultDiagnostics'; +import { getResultStderr } from './resultDiagnostics'; import { getTerminalColumns } from './terminalColumns'; import { writeStreamAsync } from './writeStreamAsync'; import { @@ -269,16 +269,12 @@ export async function launchClientAsync( // In agent mode the summary line may already carry the complete error message; do not repeat it. const reportedByAgent: boolean = agentRenderer?.finish(outcome.result) ?? false; process.exitCode = outcome.result.exitCode; - const diagnostic: string | undefined = getResultDiagnostic(outcome.result); - if (reportedByAgent) { - // The summary line already explains the failure. - } else if (diagnostic) { - await writeStreamAsync(process.stderr, Buffer.from(diagnostic)); - } else if (outcome.result.admissionErrorCode) { - await writeStreamAsync( - process.stderr, - Buffer.from(formatAdmissionFailure(outcome.result.admissionErrorCode, request.admission)) - ); + // When the agent summary line explains the failure, nothing more is printed. + const stderr: string | undefined = reportedByAgent + ? undefined + : getResultStderr(outcome.result, request.admission); + if (stderr) { + await writeStreamAsync(process.stderr, Buffer.from(stderr)); } } else if (outcome.kind === 'rejected') { const message: string = `Daemon rejected the request (${outcome.rejection.code}): ${outcome.rejection.message}`; diff --git a/apps/rush-cli-client/src/resultDiagnostics.ts b/apps/rush-cli-client/src/resultDiagnostics.ts index e65be24364..6657102ad8 100644 --- a/apps/rush-cli-client/src/resultDiagnostics.ts +++ b/apps/rush-cli-client/src/resultDiagnostics.ts @@ -1,7 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import type { IDaemonCommandResult } from '@rushstack/rush-daemon-protocol'; +import type { IDaemonCommandResult, IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; + +import { formatAdmissionFailure } from './ClientAdmissionControls'; /** * Returns the stderr line that explains a failed daemon result, if any. @@ -23,3 +25,19 @@ export function getResultDiagnostic( } return undefined; } + +/** + * Returns the stderr text that explains a daemon result when no agent summary line explains it, if any. + * + * @remarks + * An admission failure is explained with the daemon's reason when it sent one, so that the text names what + * the request waited for, such as a daemon restart. + */ +export function getResultStderr( + result: Pick, + admission: IDaemonRequestAdmissionOptions | undefined +): string | undefined { + const diagnostic: string | undefined = getResultDiagnostic(result); + if (diagnostic || !result.admissionErrorCode) return diagnostic; + return formatAdmissionFailure(result.admissionErrorCode, admission, result.errorMessage); +} diff --git a/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts b/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts index c835699a2f..317e29401f 100644 --- a/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts +++ b/apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts @@ -87,4 +87,44 @@ describe(formatAdmissionFailure.name, () => { it('explains a no-wait failure', () => { expect(formatAdmissionFailure('no-wait', { noWait: true })).toContain('--no-wait was specified'); }); + + it("prints the daemon's reason for a wait timeout instead of the generic explanation", () => { + const reason: string = + "The rushx script was not admitted before the daemon could restart for another request's environment. " + + 'Use --wait-timeout to wait longer.'; + expect(formatAdmissionFailure('wait-timeout', { waitTimeoutMs: 5000 }, reason)).toBe( + `rush-client: daemon admission failed (wait-timeout): ${reason}\n` + ); + }); + + it("adds the way to wait longer when the daemon's reason does not name it", () => { + expect( + formatAdmissionFailure( + 'wait-timeout', + { waitTimeoutMs: 5000 }, + 'The request was not admitted within 5000ms.' + ) + ).toBe( + 'rush-client: daemon admission failed (wait-timeout): The request was not admitted within 5000ms. ' + + 'To wait longer, pass --wait-timeout .\n' + ); + }); + + it("prints the daemon's reason for a no-wait failure", () => { + const reason: string = + 'Another request is waiting to restart the daemon for its environment; ' + + 'the rushx script did not wait for the restart.'; + expect(formatAdmissionFailure('no-wait', { noWait: true }, reason)).toBe( + `rush-client: daemon admission failed (no-wait): ${reason}\n` + ); + }); + + it('keeps the generic explanation when the daemon sent no reason', () => { + expect(formatAdmissionFailure('wait-timeout', { waitTimeoutMs: 5000 }, '')).toBe( + formatAdmissionFailure('wait-timeout', { waitTimeoutMs: 5000 }) + ); + expect(formatAdmissionFailure('no-wait', { noWait: true }, '')).toBe( + formatAdmissionFailure('no-wait', { noWait: true }) + ); + }); }); diff --git a/apps/rush-cli-client/src/test/resultDiagnostics.test.ts b/apps/rush-cli-client/src/test/resultDiagnostics.test.ts index 46c0ad5814..f64b6e2f72 100644 --- a/apps/rush-cli-client/src/test/resultDiagnostics.test.ts +++ b/apps/rush-cli-client/src/test/resultDiagnostics.test.ts @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { getResultDiagnostic } from '../resultDiagnostics'; +import { getResultDiagnostic, getResultStderr } from '../resultDiagnostics'; describe(getResultDiagnostic.name, () => { it('prints the error message of a failed result', () => { @@ -34,3 +34,33 @@ describe(getResultDiagnostic.name, () => { expect(getResultDiagnostic({ exitCode: 1 })).toBeUndefined(); }); }); + +describe(getResultStderr.name, () => { + it("explains an admission failure with the daemon's reason", () => { + const reason: string = + "The rushx script was not admitted before the daemon could restart for another request's environment. " + + 'Use --wait-timeout to wait longer.'; + expect( + getResultStderr( + { exitCode: 1, admissionErrorCode: 'wait-timeout', errorMessage: reason }, + { waitTimeoutMs: 5000 } + ) + ).toBe(`rush-client: daemon admission failed (wait-timeout): ${reason}\n`); + }); + + it('falls back to the generic admission explanation without a reason', () => { + expect( + getResultStderr({ exitCode: 1, admissionErrorCode: 'wait-timeout' }, { waitTimeoutMs: 5000 }) + ).toContain('timed out after its 5s wait timeout waiting for another daemon request'); + }); + + it('keeps the diagnostic of other failures and stays silent on success', () => { + expect( + getResultStderr( + { exitCode: 1, admissionErrorCode: 'aborted', errorMessage: 'daemon shut down' }, + undefined + ) + ).toBe('rush-client: daemon shut down\n'); + expect(getResultStderr({ exitCode: 0 }, undefined)).toBeUndefined(); + }); +}); diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r05-admission-failure-reason_2026-09-28-17-39.json b/common/changes/@rushstack/rush-cli-client/swarm-r05-admission-failure-reason_2026-09-28-17-39.json new file mode 100644 index 0000000000..89d1e864db --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r05-admission-failure-reason_2026-09-28-17-39.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In legacy output and in rushx-client, a daemon admission failure prints the daemon's reason, such as a pending daemon restart, instead of only a generic explanation.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json b/common/changes/@rushstack/rush-cli-client/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json new file mode 100644 index 0000000000..7e1855fc2c --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "The README says that the request's own work, such as checking its inputs and loading the workspace graph, does not count against the queue timeout.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client" +} diff --git a/common/changes/@rushstack/rush-client-core/restart-planned-successor_2026-09-28-19-50.json b/common/changes/@rushstack/rush-client-core/restart-planned-successor_2026-09-28-19-50.json new file mode 100644 index 0000000000..15e4ba7da5 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/restart-planned-successor_2026-09-28-19-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "A client that retries a request after the daemon answered it with retryAfterRestart no longer starts a daemon while the restarting daemon's process lives; it connects to the successor that daemon launches, and starts one only if that process exits without a ready successor. Previously such a client could win the startup race with its own environment.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/restart-planned-successor_2026-09-28-19-50.json b/common/changes/@rushstack/rush-daemon-protocol/restart-planned-successor_2026-09-28-19-50.json new file mode 100644 index 0000000000..1e024d49b5 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/restart-planned-successor_2026-09-28-19-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Document that a client retrying a retryAfterRestart result must not start a daemon while the predecessor process lives, because the predecessor launches the selected successor itself.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/restart-planned-successor_2026-09-28-19-50.json b/common/changes/@rushstack/rush-daemon/restart-planned-successor_2026-09-28-19-50.json new file mode 100644 index 0000000000..7ff52a9d3b --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/restart-planned-successor_2026-09-28-19-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Document that clients following a planned restart connect to the successor this process launches instead of starting their own.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-admission-timeout-text_2026-09-28-16-02.json b/common/changes/@rushstack/rush-daemon/swarm-r05-admission-timeout-text_2026-09-28-16-02.json new file mode 100644 index 0000000000..f221c20ed7 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-admission-timeout-text_2026-09-28-16-02.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "An admission timeout at a later routing boundary, such as the graph-execution gate or a global command's admission, now names the wait timeout that the client asked for instead of the remaining budget, and says how long the request waited behind another request's graph load without spending its timeout. A zero wait timeout behind a graph load now reports a plain timeout instead of claiming that 10 times its timeout was reached.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json b/common/changes/@rushstack/rush-daemon/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json new file mode 100644 index 0000000000..eb2cdb6b9d --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-preparation-wait-budget_2026-09-28-18-20.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A request's wait timeout is spent only while the request waits for other requests, so its own work, such as capturing its inputs or loading the workspace graph, no longer uses up an explicit timeout before the request starts to wait.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-rushx-pending-restart_2026-09-28-16-45.json b/common/changes/@rushstack/rush-daemon/swarm-r05-rushx-pending-restart_2026-09-28-16-45.json new file mode 100644 index 0000000000..54754e613e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-rushx-pending-restart_2026-09-28-16-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A rushx script that arrives while another request waits to restart the daemon for its environment now waits for that restart and then runs on the successor, instead of starting on the old daemon and keeping the restart waiting until the script exits.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-served-rushx-reload_2026-09-28-15-31.json b/common/changes/@rushstack/rush-daemon/swarm-r05-served-rushx-reload_2026-09-28-15-31.json new file mode 100644 index 0000000000..36f666d5d1 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-served-rushx-reload_2026-09-28-15-31.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A served rushx script releases the workspace generation lease once it starts, so a build that needs a graph reload no longer waits for a long-running script such as a dev server. A restart, a native install or update, and daemon shutdown still wait for running scripts to exit, and a planned restart still counts a script as running work until it exits.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index b611c372fc..1af7286b24 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -32,6 +32,10 @@ on success, cancellation, disconnect and failure. No resize messages are sent. retries an explicit `retryAfterRestart: true` result a bounded number of times. Before each hand-off it captures the endpoint's PID/start identity, requires protocol 0.10, waits for that ownership to be released, and reconnects through the same startup mutex. +The restarting daemon launches the successor it selected after that release, so while +its process lives the client only connects: starting a daemon itself could win the +startup mutex with the client's own environment instead of the one the restart was for. +It starts one only if that process exits without a ready successor. Retries after the first use jittered backoff, and the backoff, the successor hand-off and the resubmitted request all share the request's admission deadline. The original immutable request and unread input are preserved. Output, events, diff --git a/libraries/rush-client-core/src/connectOrStartDaemon.ts b/libraries/rush-client-core/src/connectOrStartDaemon.ts index b6f0b46e4b..369ec81128 100644 --- a/libraries/rush-client-core/src/connectOrStartDaemon.ts +++ b/libraries/rush-client-core/src/connectOrStartDaemon.ts @@ -101,6 +101,26 @@ export interface IConnectOrStartDaemonOptions extends Omit { + return await connectOrStartAsync(options, false); +} + +/** + * Like {@link connectOrStartDaemonAsync}, after `previousDaemon` answered a request with `retryAfterRestart`. + * @remarks That daemon releases its ownership and then launches the successor it selected, and its process exits + * only once that launch settles. A client that started a daemon meanwhile would race that launch, and if it won, + * the successor would have this client's environment rather than the one the restart was for. So while the + * previous process lives, this only connects; after it exits without a ready successor, this may start one. + */ +export async function connectToPlannedSuccessorAsync( + options: IConnectOrStartDaemonOptions & { readonly previousDaemon: DaemonOwnership } +): Promise { + return await connectOrStartAsync(options, true); +} + +async function connectOrStartAsync( + options: IConnectOrStartDaemonOptions, + previousStartsSuccessor: boolean ): Promise { const timeoutMs: number = options.startupTimeoutMs ?? 15000; if (!Number.isInteger(timeoutMs) || timeoutMs <= 0 || timeoutMs > 0x7fffffff) { @@ -112,6 +132,14 @@ export async function connectOrStartDaemonAsync( await waitForPreviousDaemonAsync(options.paths, options.previousDaemon, deadline, options.abortSignal); const initial: DaemonClient | undefined = await tryConnectAsync(options, deadline); if (initial) return initial; + if (previousStartsSuccessor && options.previousDaemon) { + const successor: DaemonClient | undefined = await waitForPlannedSuccessorAsync( + options, + options.previousDaemon, + deadline + ); + if (successor) return successor; + } const startCommand: IDaemonStartCommand | undefined = options.startCommand ?? (await options.resolveStartCommandAsync?.()); if (!startCommand) { @@ -454,6 +482,28 @@ async function tryConnectEndpointAsync( } } +/** Connects to a successor while the previous daemon process lives; undefined once it has exited without one. */ +async function waitForPlannedSuccessorAsync( + options: IConnectOrStartDaemonOptions, + previous: DaemonOwnership, + deadline: number +): Promise { + while (isOwnerProcessAlive(previous)) { + if (Date.now() >= deadline) { + throw startupError( + options, + `timed out waiting for the successor that the previous daemon (PID ${previous.pid}) is starting` + ); + } + await delayAsync(Math.min(100, Math.max(1, deadline - Date.now())), undefined, { + signal: options.abortSignal + }); + const successor: DaemonClient | undefined = await tryConnectAsync(options, deadline); + if (successor) return successor; + } + return undefined; +} + async function waitForHandoffAsync( options: IConnectOrStartDaemonOptions, deadline: number diff --git a/libraries/rush-client-core/src/executeWithDaemonRestart.ts b/libraries/rush-client-core/src/executeWithDaemonRestart.ts index 87f260e61f..ed6124beb7 100644 --- a/libraries/rush-client-core/src/executeWithDaemonRestart.ts +++ b/libraries/rush-client-core/src/executeWithDaemonRestart.ts @@ -9,7 +9,7 @@ import { } from '@rushstack/rush-daemon-protocol'; import { readDaemonLockfile, type IDaemonLockfile } from '@rushstack/rush-daemon-transport'; -import { connectOrStartDaemonAsync, type IConnectOrStartDaemonOptions } from './connectOrStartDaemon'; +import { connectToPlannedSuccessorAsync, type IConnectOrStartDaemonOptions } from './connectOrStartDaemon'; import { captureDaemonRequest } from './captureDaemonRequest'; import type { DaemonClient, DaemonClientOutcome, IDaemonClientExecuteOptions } from './DaemonClient'; import { DaemonClientError } from './DaemonClientError'; @@ -90,7 +90,8 @@ export async function executeWithDaemonRestartAsync( const startupTimeoutMs: number = connection.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS; // The successor handoff shares the request's admission deadline rather than starting a fresh one. boundedByAdmission = remainingMs !== undefined && remainingMs < startupTimeoutMs; - successor = await connectOrStartDaemonAsync({ + // The restarting daemon launches the successor itself; this only connects while that process lives. + successor = await connectToPlannedSuccessorAsync({ ...connection, startupTimeoutMs: boundedByAdmission ? Math.max(1, Math.ceil(remainingMs!)) : startupTimeoutMs, previousDaemon: { pid: owner.pid, startedAt: owner.startedAt }, diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index 324e7b01be..16f76b614f 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -25,6 +25,7 @@ import { getDaemonLogFilePath } from '../DaemonLogFile'; import { resetDaemonArtifactsAsync } from '../DaemonOwnership'; import { connectOrStartDaemonAsync, + connectToPlannedSuccessorAsync, requestDaemonShutdownAsync, resolveDaemonStartupReservationAsync, type IConnectOrStartDaemonOptions @@ -1064,6 +1065,102 @@ describe('detached daemon startup', () => { } }); + it('lets a restarting daemon start its planned successor while the clients that follow it only connect', async () => { + // The old daemon launches only after both clients could have started one, so a race would always be lost. + fs.writeFileSync(path.join(folder, 'planned-requests'), '2'); + fs.writeFileSync(path.join(folder, 'planned-launch-delay-ms'), '1500'); + const planned: IConnectOrStartDaemonOptions = { + ...options, + startCommand: { + ...options.startCommand!, + args: [...options.startCommand!.args, 'fixture', 'restart-planned'] + } + }; + const clients: DaemonClient[] = [ + await connectOrStartDaemonAsync(planned), + await connectOrStartDaemonAsync(planned) + ]; + const { pid: restartingPid } = await clients[0].status; + const outcomes = await Promise.all( + clients.map((client, index) => + executeWithDaemonRestartAsync(client, options, { + request: captureDaemonRequest({ + requestId: `follower-${index}`, + argv: ['test'], + commandName: 'test', + commandOrigin: 'custom', + cwd: folder, + environment: {}, + terminal: { isTTY: false, supportsColor: false } + }) + }) + ) + ); + await Promise.all(clients.map((client) => client.closeAsync())); + expect(outcomes).toMatchObject([ + { kind: 'result', result: { exitCode: 0 } }, + { kind: 'result', result: { exitCode: 0 } } + ]); + // The restarting daemon exits only after it attests the successor it launched. + await waitForTestProcessExitAsync(restartingPid!); + const identities: string[] = fs.readFileSync(path.join(folder, 'identities'), 'utf8').trim().split('\n'); + expect(identities.map((line) => line.split(' ')[1])).toEqual(['client', 'planned']); + expect(fs.readFileSync(path.join(folder, 'planned-successor'), 'utf8')).toBe(identities[1].split(' ')[0]); + expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n')).toHaveLength(4); + }, 15000); + + it('connects to the successor that another process starts while the restarting predecessor lives', async () => { + const waiting = connectToPlannedSuccessorAsync({ + ...options, + previousDaemon: { pid: process.pid, startedAt: new Date().toISOString() } + }); + await delayAsync(300); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + const starter = await connectOrStartDaemonAsync(options); + try { + const follower = await waiting; + expect((await follower.status).pid).toBe((await starter.status).pid); + await follower.closeAsync(); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + } finally { + await starter.closeAsync(); + } + }); + + it('never starts a daemon while the restarting predecessor lives, even when its deadline expires', async () => { + await expect( + connectToPlannedSuccessorAsync({ + ...options, + previousDaemon: { pid: process.pid, startedAt: new Date().toISOString() }, + startupTimeoutMs: 300 + }) + ).rejects.toThrow(`timed out waiting for the successor that the previous daemon (PID ${process.pid}) is starting`); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }); + + it('starts a daemon once the restarting predecessor exits without a successor', async () => { + // The predecessor exits only when its stdin ends, so it provably lives during the first check. + const predecessor: ChildProcess = spawn(process.execPath, ['-e', 'process.stdin.resume()'], { + stdio: ['pipe', 'ignore', 'ignore'] + }); + await once(predecessor, 'spawn'); + const exited: Promise = once(predecessor, 'exit'); + const pending = connectToPlannedSuccessorAsync({ + ...options, + previousDaemon: { pid: predecessor.pid!, startedAt: new Date().toISOString() } + }); + try { + await delayAsync(300); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + } finally { + predecessor.stdin!.end(); + } + await exited; + const client = await pending; + await client.closeAsync(); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + }); + it('does not stop a mismatched daemon with unverifiable ownership', async () => { const running = await connectOrStartDaemonAsync(options); await running.closeAsync(); diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index d4d05b7cab..ea78f0408b 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -16,6 +16,8 @@ import { type IDaemonPaths } from '@rushstack/rush-daemon-transport'; +import { connectOrStartDaemonAsync } from '../../connectOrStartDaemon'; + async function mainAsync(): Promise { const paths: IDaemonPaths = JSON.parse(process.argv[2]); const folder: string = path.dirname(paths.lockfilePath); @@ -25,7 +27,12 @@ async function mainAsync(): Promise { const connections: Set = new Set(); let closing: Promise | undefined; let heldRequest: { connection: DaemonFrameConnection; requestId: string } | undefined; + let plannedRestartAnswers: number = 0; fs.appendFileSync(path.join(folder, 'starts'), `${process.pid}\n`); + fs.appendFileSync( + path.join(folder, 'identities'), + `${process.pid} ${process.env.FIXTURE_IDENTITY ?? 'client'}\n` + ); fs.appendFileSync(path.join(folder, 'parents'), `${process.ppid}\n`); fs.writeFileSync(path.join(folder, 'runtime-base'), process.env.RUSHD_RUNTIME_DIR ?? '(unset)'); process.stdout.write('launcher stdout\n'); @@ -114,7 +121,15 @@ async function mainAsync(): Promise { } }) }); - if (restart && restartMode !== 'restart-held') { + if (restart && restartMode === 'restart-planned') { + // Like RushDaemonHost: answer every request, then release ownership and launch the successor itself. + if (++plannedRestartAnswers === readNumber(folder, 'planned-requests', 1)) { + void launchPlannedSuccessorAsync().catch((error: Error) => { + process.stderr.write(`${error.stack}\n`); + process.exitCode = 1; + }); + } + } else if (restart && restartMode !== 'restart-held') { fs.appendFileSync(path.join(folder, 'restarted'), 'r'); await stopAsync(); } @@ -168,6 +183,32 @@ async function mainAsync(): Promise { }); } + async function launchPlannedSuccessorAsync(): Promise { + await stopAsync(); + // The "planned-launch-delay-ms" file models a launch that begins well after ownership is released. + await new Promise((resolve) => setTimeout(resolve, readNumber(folder, 'planned-launch-delay-ms', 0))); + const environment: Record = {}; + for (const [name, value] of Object.entries(process.env)) { + if (value !== undefined) environment[name] = value; + } + const successor = await connectOrStartDaemonAsync({ + paths, + expectedDaemonVersion: daemonVersion, + startCommand: { + command: process.execPath, + args: [__filename, JSON.stringify(paths), daemonVersion], + cwd: folder, + environment: { ...environment, FIXTURE_IDENTITY: 'planned' } + } + }); + try { + const { pid } = await successor.status; + fs.writeFileSync(path.join(folder, 'planned-successor'), String(pid)); + } finally { + await successor.closeAsync(); + } + } + async function closeOnceAsync(): Promise { clearInterval(timer); const stopped: Promise = listener.stopAcceptingAsync(); @@ -181,8 +222,12 @@ async function mainAsync(): Promise { /** How long the daemon waits before it listens: 250 milliseconds, or the number in the "startup-delay-ms" file. */ function readStartupDelayMs(folder: string): number { - const delayPath: string = path.join(folder, 'startup-delay-ms'); - return fs.existsSync(delayPath) ? Number(fs.readFileSync(delayPath, 'utf8')) : 250; + return readNumber(folder, 'startup-delay-ms', 250); +} + +function readNumber(folder: string, name: string, defaultValue: number): number { + const filePath: string = path.join(folder, name); + return fs.existsSync(filePath) ? Number(fs.readFileSync(filePath, 'utf8')) : defaultValue; } mainAsync().catch((error: Error) => { diff --git a/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts b/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts index 293776da32..4a133c4e4b 100644 --- a/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts +++ b/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts @@ -20,6 +20,8 @@ export interface IDaemonCommandResult { * Protocol 0.10: no execution or request IO occurred, and a successor has been selected. * Retry only after attested predecessor ownership release, within the request's admission deadline and a * small client-defined retry bound; then fall back instead of retrying. Never infer this from an error. + * The predecessor launches the selected successor itself after that release, and its process exits once the + * launch settles: until then, a retrying client connects to the successor but must not start a daemon. */ readonly retryAfterRestart?: true; /** Whether cancellation or disconnect was observed, even if a cleanup failure determines the outcome. */ diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index dc809142de..d53bb7f280 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -105,7 +105,8 @@ taken or while it executes is not kept as up to date, whether or not cache write it and its consumers again, even if the files were changed back in between. Operations whose build cache is disabled, and workspaces without a build cache, don't get this check. -A generation lease spans resolution through final output. Reload also takes exclusive workspace admission and +A generation lease spans resolution through final output; a Rushx script releases it when the script starts (see +below). Reload also takes exclusive workspace admission and the native preparation lock, discards paused prepared work, and awaits old runner/plugin/watcher cleanup before publishing the replacement. The initiating request atomically downgrades its admission so another reload cannot dispose the newly selected graph before it runs. Watch requests are cancelled and drained before their generation @@ -174,7 +175,9 @@ The default entrypoint supports the `rush.json` version, not a separate preview- Successor startup reuses `connectOrStartDaemonAsync`: acknowledged old ownership must be released after all old resources finish, startup is serialized with ordinary clients, and hello/ping readiness attests a different PID. -`restartCompleted` reports completion or failure. +`restartCompleted` reports completion or failure. Clients that retry a `retryAfterRestart: true` result do not start a +daemon while this process lives, so the successor is the one it selected; only a client that did not follow the +restart can still take the startup mutex first. A request whose environment needs another process does not restart the daemon while it serves other requests. It first waits for the requests that this process is serving to finish (the restart drain), and its queue position is the @@ -187,6 +190,14 @@ drain, which could otherwise keep it waiting for as long as they keep arriving. `waitTimeoutMs` limits the whole drain, and only its remaining time carries over to the successor. When a drain times out, its message names the time that did not count. +A restart is pending from when a request begins its restart drain until the request has planned the restart, or has +failed or been cancelled. A rushx script that arrives while a restart is pending does not start, since the restart +would then wait for it to exit: the script waits for the pending restart instead, and the drain does not count it. If +the restart was planned, the script's result carries `retryAfterRestart: true` so that the client runs it on the +successor; otherwise it runs on this process. Its queue position is the number of requests that are served or waiting +to restart, and its wait timeout applies as it does to the drain, relative to the requests that were served when the +script began to wait. + Protocol 0.10 (`DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR`) provides bounded, typed retry authorization. Only a pre-execution command result may carry `retryAfterRestart: true`. During a planned restart, accepted queued requests drain those typed results before disconnect rather than being reduced to ambiguous connection @@ -223,8 +234,13 @@ this.workspaceLifecycle = wrapWorkspaceResolverLifecycle( ``` The helper returns `undefined` for a delegate without lifecycle support. Explicit `invocationKind: "rushx"` -requests retain a generation lease but go directly to the composite resolver, without native build/mutation/graph -interception or phased environment matching. They use exclusive global admission. The host disposes each old +requests go directly to the composite resolver, without native build/mutation/graph interception, phased +environment matching, or workspace admission. A script uses its generation only to resolve, so it holds its +generation lease only until it starts: a reload that another request needs never waits for a long-running script +such as a dev server. A restart, a native `install` or `update`, and lifecycle disposal would end a running script, +so they still wait for every running script to exit, and a planned restart counts a script as running work until it +exits. A script that arrives while a restart is pending waits for the restart instead of starting. The host disposes +each old resolver before replacing its session, and disposes the current resolver at shutdown; the composite must forward its normal disposer to its owned delegates. @@ -482,10 +498,13 @@ the graph does not spend its budget during that load, so every build that arrive loads the graph is admitted when the load finishes. That wait is limited separately, to 10 times `waitTimeoutMs`, so a load that never finishes does not hold the requests behind it indefinitely. The budget does run while that other request still waits for exclusive admission, so requests behind a reload that cannot start, for example behind a long -build, still time out. Routing and executing an admitted request do not spend the budget either. Routing boundaries +build, still time out. The request's own work does not spend the budget either, before or after admission: capturing +its inputs, loading or reloading the graph, routing and execution. Routing boundaries such as the graph-execution gate apply the remaining budget they receive, and a request that re-enters workspace admission to reload the graph after its inputs changed starts again from the budget it had when it was admitted; time -it spent at those boundaries is not charged again. The default and an explicit value differ only at the +it spent at those boundaries is not charged again. A timeout message at any boundary names the timeout that the +client asked for rather than the remaining budget, and says how long the request waited behind another request's +graph load without spending it. The default and an explicit value differ only at the graph-execution gate and at a restart drain (see "Process restart and isolated install/update"). When the client marks `waitTimeoutMs` as its default (`waitTimeoutIsDefault`), a `SHARED-BUILD` request that arrives after the current batch has closed waits at the graph-execution gate without a deadline, because it is queued only behind running compatible diff --git a/libraries/rush-daemon/src/GlobalCommandRequest.ts b/libraries/rush-daemon/src/GlobalCommandRequest.ts index 0b41a05f27..bb3b1f4360 100644 --- a/libraries/rush-daemon/src/GlobalCommandRequest.ts +++ b/libraries/rush-daemon/src/GlobalCommandRequest.ts @@ -13,6 +13,7 @@ import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; +import { freezeDaemonRequestAdmissionOptions } from './WorkspaceRequestAdmission'; import type { IWorkspaceSession } from './WorkspaceSession'; /** @@ -100,7 +101,7 @@ export function resolveGlobalCommandRequest( validateDaemonRequestAdmissionOptions(options.admission); const cwd: string = resolveGlobalCommandWorkingDirectory(options.cwd, workspaceSession); const request: IResolvedGlobalCommandRequest = Object.freeze({ - admission: options.admission ? Object.freeze({ ...options.admission }) : undefined, + admission: options.admission ? freezeDaemonRequestAdmissionOptions(options.admission) : undefined, commandName: options.commandName, commandOrigin: options.commandOrigin, cwd, diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index c5bf80e37e..7a82501c47 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -21,7 +21,11 @@ import { } from './RequestScheduler'; import type { IWorkspaceSession } from './WorkspaceSession'; import { assertWorkspaceRequestResourcesHealthy } from './WorkspaceRequestResources'; -import type { IWorkspaceRestartTicket, WorkspaceRestartArbiter } from './WorkspaceRestartArbiter'; +import type { + IWorkspaceRestartDrainOptions, + IWorkspaceRestartTicket, + WorkspaceRestartArbiter +} from './WorkspaceRestartArbiter'; export interface IRequestAdmissionClient { readonly abortSignal: AbortSignal; @@ -36,6 +40,21 @@ export interface IRequestAdmissionControllerOptions { } const REQUEST_SCHEDULER_BY_SESSION: WeakMap = new WeakMap(); + +/** What a remaining admission budget does not show about the request that it came from. */ +interface IAdmissionBudgetHistory { + /** The wait timeout that the client asked for. */ + readonly waitTimeoutMs: number; + /** Time spent behind another request's graph load or reload, which did not count against the wait timeout. */ + readonly pausedMs: number; +} + +/** + * The history of each remaining budget that a controller hands to another routing boundary, so that the boundary's + * timeout message names the client's timeout and the time that did not count, rather than the remainder alone. + */ +const HISTORY_BY_REMAINING_ADMISSION: WeakMap = + new WeakMap(); /** A request waits for another request's graph load or reload for up to this many times its wait timeout. */ const GRAPH_LOAD_WAIT_FACTOR: number = 10; // Only the per-invocation flag is offered: Rush versions that do not recognize the environment variable reject it. @@ -45,6 +64,22 @@ function formatSeconds(ms: number): string { return `${Math.round(ms / 100) / 10}s`; } +/** Describes time that did not count against a request's wait timeout, unless it rounds to nothing. */ +function formatUncountedTime(pausedMs: number, spentWhile: string): string { + const seconds: string = formatSeconds(pausedMs); + return seconds === '0s' ? '' : `; ${seconds} spent ${spentWhile} did not count`; +} + +/** Returns a frozen copy of admission options that keeps the history of a remaining budget. */ +export function freezeDaemonRequestAdmissionOptions( + admission: IDaemonRequestAdmissionOptions +): IDaemonRequestAdmissionOptions { + const copy: IDaemonRequestAdmissionOptions = Object.freeze({ ...admission }); + const history: IAdmissionBudgetHistory | undefined = HISTORY_BY_REMAINING_ADMISSION.get(admission); + if (history) HISTORY_BY_REMAINING_ADMISSION.set(copy, history); + return copy; +} + class WorkspaceRequestScheduler extends RequestScheduler { readonly #session: IWorkspaceSession; @@ -208,17 +243,27 @@ export class RequestAdmissionController { readonly #abortFromClient: () => void; readonly #admission: IDaemonRequestAdmissionOptions | undefined; readonly #client: IRequestAdmissionClient; - #deadlineMs: number | undefined; + /** The wait timeout that the client asked for; `#admission` holds only the remainder at a later boundary. */ + readonly #configuredWaitTimeoutMs: number | undefined; + /** Time spent behind another request's graph load or reload, which did not count against the wait timeout. */ + #pausedMs: number; + /** + * The unspent wait timeout, or undefined when waiting is not limited. Only this controller's waits for other + * requests spend it, so the request's own work, such as capturing its inputs, loading or reloading the workspace + * graph, routing and execution, does not. + */ + #remainingMs: number | undefined; readonly #writer: QueuePositionWriter | undefined; public constructor(options: IRequestAdmissionControllerOptions) { validateDaemonRequestAdmissionOptions(options.admission); this.#admission = options.admission; + const history: IAdmissionBudgetHistory | undefined = + options.admission && HISTORY_BY_REMAINING_ADMISSION.get(options.admission); + this.#configuredWaitTimeoutMs = history?.waitTimeoutMs ?? options.admission?.waitTimeoutMs; + this.#pausedMs = history?.pausedMs ?? 0; this.#client = options.client; - this.#deadlineMs = - options.admission?.waitTimeoutMs === undefined - ? undefined - : Date.now() + options.admission.waitTimeoutMs; + this.#remainingMs = options.admission?.waitTimeoutMs; this.#writer = options.client.supportsRequestAdmission === true ? new QueuePositionWriter(options.client, options.requestId, this.#abortController) @@ -231,17 +276,16 @@ export class RequestAdmissionController { } } - /** Waits for workspace admission within the request's remaining admission budget. */ + /** + * Waits for workspace admission within the request's remaining admission budget. A wait-timeout error says the + * request was waiting for `waitingFor`. + */ public async acquireAsync( scheduler: RequestScheduler, - exclusivityClass: RequestExclusivityClass + exclusivityClass: RequestExclusivityClass, + waitingFor: string = 'workspace admission' ): Promise { - return await this.#acquireAsync( - scheduler, - exclusivityClass, - this.#getRemainingWaitTimeoutMs(), - 'workspace admission' - ); + return await this.#acquireAsync(scheduler, exclusivityClass, this.#remainingMs, waitingFor); } /** @@ -259,7 +303,7 @@ export class RequestAdmissionController { const waitTimeoutMs: number | undefined = exclusivityClass === RequestExclusivityClass.SharedBuild && this.#admission?.waitTimeoutIsDefault ? undefined - : this.#getRemainingWaitTimeoutMs(); + : this.#remainingMs; return await this.#acquireAsync( scheduler, exclusivityClass, @@ -285,8 +329,8 @@ export class RequestAdmissionController { transition: AdmissionProgress ): Promise { const waitingFor: string = "another request's load or reload of the workspace graph"; - const remainingMs: number | undefined = this.#getRemainingWaitTimeoutMs(); - const waitTimeoutMs: number | undefined = this.#admission?.waitTimeoutMs; + const remainingMs: number | undefined = this.#remainingMs; + const waitTimeoutMs: number | undefined = this.#configuredWaitTimeoutMs; if (remainingMs === undefined || waitTimeoutMs === undefined) { return await this.#acquireAsync( scheduler, @@ -319,42 +363,22 @@ export class RequestAdmissionController { } catch (error) { budget.stop(); if (!exhausted.signal.aborted || this.#abortController.signal.aborted) throw error; - const message: string = pausedLimitReached - ? `The request was not admitted within ${GRAPH_LOAD_WAIT_FACTOR} times its ${waitTimeoutMs}ms wait ` + - `timeout because ${waitingFor} was still running after ${formatSeconds(budget.pausedMs)}.` - : `The request was not admitted within its ${waitTimeoutMs}ms wait timeout while waiting for ` + - `${waitingFor}` + - (budget.pausedMs > 0 - ? `; ${formatSeconds(budget.pausedMs)} spent while that request loaded the graph did not count.` - : '.'); + // A zero timeout has no paused allowance, so it fails at once without reaching a limit worth naming. + const message: string = + pausedLimitReached && waitTimeoutMs > 0 + ? `The request was not admitted within ${GRAPH_LOAD_WAIT_FACTOR} times its ${waitTimeoutMs}ms wait ` + + `timeout because ${waitingFor} was still running after ${formatSeconds(budget.pausedMs)}.` + : `The request was not admitted within its ${waitTimeoutMs}ms wait timeout while waiting for ` + + `${waitingFor}` + + `${formatUncountedTime(this.#pausedMs + budget.pausedMs, 'while that request loaded the graph')}.`; throw new RequestSchedulerError( RequestSchedulerErrorCode.WaitTimeout, `${message} ${WAIT_LONGER_HINT}` ); } finally { budget.stop(); - this.#deadlineMs = Date.now() + budget.remainingMs; - } - } - - /** - * Runs `action`, such as routing and executing an admitted request, without spending the request's wait timeout. - * - * @remarks - * Work after admission either runs or waits at a routing boundary that applies the timeout it received, such as the - * graph-execution gate. A request that re-enters workspace admission afterwards, for example to reload the graph - * after its inputs changed, therefore keeps the unspent timeout it had before `action`, whether that timeout is the - * client default or explicit. - */ - public async runOutsideWaitBudgetAsync(action: () => Promise): Promise { - const remainingMs: number | undefined = this.#getRemainingWaitTimeoutMs(); - if (remainingMs === undefined) { - return await action(); - } - try { - return await action(); - } finally { - this.#deadlineMs = Date.now() + remainingMs; + this.#pausedMs += budget.pausedMs; + this.#remainingMs = budget.remainingMs; } } @@ -366,6 +390,7 @@ export class RequestAdmissionController { abortSignal: AbortSignal = this.#abortController.signal ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; + const startMs: number = Date.now(); let lease: IRequestLease | undefined; try { lease = await scheduler.acquireAsync({ @@ -387,6 +412,9 @@ export class RequestAdmissionController { lease?.release(); await writer?.flushAsync(); throw this.#getReportedError(error, waitingFor); + } finally { + // A wait that the timeout does not limit does not spend it either; see `acquireGraphExecutionAsync`. + if (waitTimeoutMs !== undefined) this.#spend(Date.now() - startMs); } } @@ -400,26 +428,53 @@ export class RequestAdmissionController { * still being served and no rushx script is, and that time does not count against it. The default still limits * the wait while a rushx script is served, since a script may not exit until it is stopped, and waiting for * requests that arrived later, which could otherwise keep the request waiting for as long as they keep arriving. - * An explicit `noWait` or `waitTimeoutMs` applies to the whole wait, using the same absolute deadline as workspace - * admission. + * An explicit `noWait` or `waitTimeoutMs` applies to the whole wait, using the same budget as workspace admission. */ public async waitForRestartDrainAsync( arbiter: WorkspaceRestartArbiter, ticket: IWorkspaceRestartTicket + ): Promise { + await this.#waitForRestartArbiterAsync((options: IWorkspaceRestartDrainOptions) => + arbiter.waitForDrainAsync(ticket, options) + ); + } + + /** + * Waits, for a rushx script, until no other request needs to restart the daemon for its environment, so that the + * restart does not also wait for the script. The client is told how many requests are served or need a restart, as + * a queue position. + * + * @remarks + * The wait timeout applies as it does to the restart drain, relative to the requests served when this wait began: + * a client-default timeout is not spent while one of them is still being served and no rushx script is. + */ + public async waitForPendingRestartAsync( + arbiter: WorkspaceRestartArbiter, + ticket: IWorkspaceRestartTicket + ): Promise { + await this.#waitForRestartArbiterAsync((options: IWorkspaceRestartDrainOptions) => + arbiter.waitForPendingRestartAsync(ticket, options) + ); + } + + async #waitForRestartArbiterAsync( + waitAsync: (options: IWorkspaceRestartDrainOptions) => Promise ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; + const startMs: number = Date.now(); + let waivedMs: number = 0; try { // The arbiter reports its own admission errors, so this does not depend on the scheduler error mapping. - const waivedMs: number = await arbiter.waitForDrainAsync(ticket, { + waivedMs = await waitAsync({ abortSignal: this.#abortController.signal, noWait: this.#admission?.noWait, - waitTimeoutMs: this.#getRemainingWaitTimeoutMs(), + waitTimeoutMs: this.#remainingMs, waivesTimeoutForServedWork: this.#admission?.waitTimeoutIsDefault === true, - onServingCountChanged: writer ? (servingCount: number) => writer.enqueue(servingCount) : undefined + onServingCountChanged: writer ? (count: number) => writer.enqueue(count) : undefined }); - if (this.#deadlineMs !== undefined) this.#deadlineMs += waivedMs; } finally { await writer?.flushAsync(); + this.#spend(Date.now() - startMs - waivedMs); } } @@ -429,17 +484,29 @@ export class RequestAdmissionController { /** Passes the remaining admission budget to another existing routing boundary. */ public get remainingAdmission(): IDaemonRequestAdmissionOptions | undefined { - return this.#admission - ? { ...this.#admission, waitTimeoutMs: this.#getRemainingWaitTimeoutMs() } - : undefined; + if (!this.#admission) return undefined; + const remaining: IDaemonRequestAdmissionOptions = { + ...this.#admission, + waitTimeoutMs: this.#remainingMs + }; + if (this.#configuredWaitTimeoutMs !== undefined) { + HISTORY_BY_REMAINING_ADMISSION.set(remaining, { + waitTimeoutMs: this.#configuredWaitTimeoutMs, + pausedMs: this.#pausedMs + }); + } + return remaining; } - #getRemainingWaitTimeoutMs(): number | undefined { - return this.#deadlineMs === undefined ? undefined : Math.max(0, this.#deadlineMs - Date.now()); + /** Spends `elapsedMs` of the wait timeout, if one applies. */ + #spend(elapsedMs: number): void { + if (this.#remainingMs !== undefined) { + this.#remainingMs = Math.max(0, this.#remainingMs - Math.max(0, elapsedMs)); + } } #getReportedError(error: unknown, waitingFor: string): unknown { - const waitTimeoutMs: number | undefined = this.#admission?.waitTimeoutMs; + const waitTimeoutMs: number | undefined = this.#configuredWaitTimeoutMs; if ( waitTimeoutMs !== undefined && error instanceof RequestSchedulerError && @@ -447,7 +514,8 @@ export class RequestAdmissionController { ) { return new RequestSchedulerError( RequestSchedulerErrorCode.WaitTimeout, - `The request was not admitted within its ${waitTimeoutMs}ms wait timeout while waiting for ${waitingFor}. ` + + `The request was not admitted within its ${waitTimeoutMs}ms wait timeout while waiting for ${waitingFor}` + + `${formatUncountedTime(this.#pausedMs, 'earlier while another request loaded the workspace graph')}. ` + WAIT_LONGER_HINT ); } diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 030ec677f2..3b8a306224 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -114,6 +114,13 @@ class RestartPendingBeforeExecution extends Error { export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { readonly #options: IWorkspaceRequestLifecycleOptions; readonly #gate: RequestScheduler = new RequestScheduler(); + /** + * A served rushx script needs its generation only to resolve, so once it starts it releases `#gate` and holds a + * shared lease here until it exits: a reload no longer waits for a dev server or watch script (#113). Whatever + * would end the script with this process (a restart, a native mutation, disposal) waits for this lease after + * taking `#gate` exclusively, when no other script can start. + */ + readonly #scripts: RequestScheduler = new RequestScheduler(); readonly #restartArbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); readonly #abortController: AbortController = new AbortController(); readonly #observers: Set = new Set(); @@ -216,14 +223,16 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { client, requestId: envelope.requestId }); - // Long-lived observers are cancelled by a transition, so they never delay a restart. A rushx script does delay - // one until it exits, so a client-default timeout still limits waiting for it. + // Long-lived observers are cancelled by a transition, so they never delay a restart. A running rushx script does + // delay one until it exits, so a client-default timeout still limits waiting for it; a script that arrives while a + // restart is pending waits for that restart instead (#prepareAsync). const ticket: IWorkspaceRestartTicket | undefined = observer ? undefined : this.#restartArbiter.enter({ runsScript: isRushxInvocation(envelope) }); let generation: IPreparedGeneration | undefined; try { for (let attempt: number = 0; ; attempt++) { + let scriptLease: IRequestLease | undefined; try { const prepared: IPreparedGeneration = await this.#prepareAsync( envelope, @@ -233,27 +242,32 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { receivedTimeMs ); generation = prepared; + if (isRushxInvocation(envelope)) { + // Never waits: exclusive holders of this lease also hold `#gate` exclusively, and this request holds it. + scriptLease = await admission.acquireAsync(this.#scripts, RequestExclusivityClass.SharedBuild); + } + // Routing boundaries spend a copy of the remaining budget, so a retry after dispatch starts from this one. const requestEnvelope: IDaemonRequestEnvelope = { ...envelope, admission: admission.remainingAdmission }; - await admission.runOutsideWaitBudgetAsync(async () => { - if (isMutation(envelope)) { - await this.#executeMutationAsync(prepared, requestEnvelope, client, state, dispatchAsync); - } else { - await dispatchAsync({ - envelope: requestEnvelope, - client, - workspaceSession: prepared.session, - resolver: prepared.resolver, - receivedTimeMs, - onExecutionStarting: () => { - this.#assertGeneration(prepared); - state.began = true; - } - }); - } - }); + if (isMutation(envelope)) { + await this.#executeMutationAsync(prepared, requestEnvelope, client, state, dispatchAsync); + } else { + await dispatchAsync({ + envelope: requestEnvelope, + client, + workspaceSession: prepared.session, + resolver: prepared.resolver, + receivedTimeMs, + onExecutionStarting: () => { + this.#assertGeneration(prepared); + state.began = true; + // The script keeps its restart ticket and script lease; only reloads stop waiting for it. + if (scriptLease) prepared.lease.release(); + } + }); + } return; } catch (error) { if ( @@ -308,6 +322,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } finally { generation?.lease.release(); generation = undefined; + scriptLease?.release(); } } } finally { @@ -339,6 +354,13 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { let session: IWorkspaceSession = await this.#options.provider.getSessionAsync(); assertWorkspaceRequestResourcesHealthy(session); if (isRushxInvocation(envelope)) { + if (ticket && this.#restartArbiter.hasPendingRestart(ticket)) { + // The restart would wait for the script to exit, so the script waits for the restart instead. If the restart + // is planned, the next attempt tells the client to run the script on the successor. + lease.release(); + await admission.waitForPendingRestartAsync(this.#restartArbiter, ticket); + return await this.#prepareAsync(envelope, client, admission, ticket, receivedTimeMs); + } return { session, resolver: this.#resolver, @@ -368,6 +390,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } this.#cancelObservers(); lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); + await this.#waitForServedScriptsAsync(admission); session = await this.#options.provider.getSessionAsync(); await this.#quiesceWarmSetAsync(session); const workspaceLease: IRequestLease = await admission.acquireAsync( @@ -478,6 +501,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { fingerprint = await this.#captureAsync(session, envelope); tier = this.#classify(fingerprint, isMutation(envelope)); if (tier === WorkspaceInputChangeTier.Restart) { + await this.#waitForServedScriptsAsync(admission); await this.#quiesceWarmSetAsync(session); const workspaceLease: IRequestLease = await admission.acquireAsync( getWorkspaceRequestScheduler(session), @@ -502,6 +526,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { 'Native mutations require a successor launcher. No worker was started.' ); } + await this.#waitForServedScriptsAsync(admission); await this.#quiesceWarmSetAsync(session); return { session, @@ -652,6 +677,26 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } } + /** + * Waits, while holding `#gate` exclusively, until no served rushx script is running; none can start meanwhile. + * This is contention, not graph-load progress, so requests queued behind it spend their wait timeouts. + */ + async #waitForServedScriptsAsync(admission: RequestAdmissionController): Promise { + const loading: boolean = this.#transitionProgress.active; + this.#transitionProgress.setActive(false); + try { + ( + await admission.acquireAsync( + this.#scripts, + RequestExclusivityClass.Exclusive, + 'a rushx script that this daemon runs to exit' + ) + ).release(); + } finally { + this.#transitionProgress.setActive(loading); + } + } + async #quiesceWarmSetAsync(session: IWorkspaceSession): Promise { this.#forceReload = true; try { @@ -853,6 +898,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { }); const failures: unknown[] = []; try { + // Served scripts don't hold `#gate`; they were aborted above, so wait for them to stop. + (await this.#scripts.acquireAsync({ exclusivityClass: RequestExclusivityClass.Exclusive })).release(); for (const resolver of this.#ownedResolvers) { try { await resolver[Symbol.asyncDispose]?.(); diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts index ee76274a88..f431baf85c 100644 --- a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -29,11 +29,27 @@ export interface IWorkspaceRestartTicketOptions { } interface IMutableTicket { + /** The request waits for a drain or for a pending restart, so it is not counted as served. */ waitingForDrain: boolean; left: boolean; readonly runsScript: boolean; } +interface IWaitKind { + /** + * The request waits to restart the daemon, so {@link WorkspaceRestartArbiter.hasPendingRestart} reports its restart + * to other requests until it leaves. + */ + readonly restarts: boolean; + /** Whether the request must keep waiting. */ + readonly isBlocked: () => boolean; + /** How many other requests the request waits for, reported as its queue position. */ + readonly countWaitedFor: () => number; + readonly noWaitMessage: string; + /** Begins the timeout message, which goes on to name the requests that the restart waits for. */ + readonly timeoutPrefix: string; +} + /** Options for {@link WorkspaceRestartArbiter.waitForDrainAsync}, supplied by request admission. */ export interface IWorkspaceRestartDrainOptions { readonly abortSignal: AbortSignal; @@ -57,11 +73,15 @@ export interface IWorkspaceRestartDrainOptions { * Arbitrates process restarts between requests whose environments differ from the running daemon. * A request that needs a restart waits until every other request this process can serve has finished, * so a mismatched environment never preempts queued or in-flight work that matches the running process. + * A rushx script that arrives while a restart is pending waits for the restart instead, since it may not exit until + * it is stopped and the restart would otherwise wait for it. */ export class WorkspaceRestartArbiter { readonly #listeners: Set<() => void> = new Set(); - readonly #countListeners: Set<(servingCount: number) => void> = new Set(); + readonly #countListeners: Set<() => void> = new Set(); readonly #serving: Set = new Set(); + /** Requests that began a restart drain and have not left, so their restarts are still pending. */ + readonly #restartCandidates: Set = new Set(); /** The number of tracked requests that are not waiting for a restart. */ public get servingCount(): number { @@ -82,41 +102,92 @@ export class WorkspaceRestartArbiter { const state: IMutableTicket = ticket as IMutableTicket; if (state.left) return; state.left = true; + this.#restartCandidates.delete(state); if (!state.waitingForDrain) this.#stopServing(state); } + /** + * Whether another tracked request needs to restart the daemon for its environment: it waits for the drain, or it + * has drained and not yet left. A request that needs a restart leaves once the restart is planned, or once it + * fails or is cancelled. + */ + public hasPendingRestart(ticket: IWorkspaceRestartTicket): boolean { + return Array.from(this.#restartCandidates).some((candidate: IMutableTicket) => candidate !== ticket); + } + /** * Waits until no other tracked request is still being served by this process, then counts the ticket * as served again so concurrent restart candidates proceed one at a time. Returns how many milliseconds of the * wait did not spend `waitTimeoutMs` (see {@link IWorkspaceRestartDrainOptions.waivesTimeoutForServedWork}). + * From when the wait begins until the ticket leaves, {@link WorkspaceRestartArbiter.hasPendingRestart} reports + * the restart to other requests. */ public async waitForDrainAsync( ticket: IWorkspaceRestartTicket, options: IWorkspaceRestartDrainOptions + ): Promise { + return await this.#waitAsync(ticket, options, { + restarts: true, + isBlocked: () => this.#serving.size > 0, + countWaitedFor: () => this.#serving.size, + noWaitMessage: 'Another environment is still being served; the request did not wait for a restart.', + timeoutPrefix: + 'The request was not admitted before the daemon could restart for its environment, which waits for' + }); + } + + /** + * Waits, for a request that runs a rushx script, until no other tracked request needs to restart the daemon for + * its environment, then counts the ticket as served again. A script may not exit until it is stopped, so it waits + * for a pending restart instead of starting and delaying the restart until it exits. The options apply as for + * {@link WorkspaceRestartArbiter.waitForDrainAsync}, and the queue position counts the requests that are served or + * need a restart. + */ + public async waitForPendingRestartAsync( + ticket: IWorkspaceRestartTicket, + options: IWorkspaceRestartDrainOptions + ): Promise { + return await this.#waitAsync(ticket, options, { + restarts: false, + isBlocked: () => this.hasPendingRestart(ticket), + countWaitedFor: () => new Set([...this.#serving, ...this.#restartCandidates]).size, + noWaitMessage: + 'Another request is waiting to restart the daemon for its environment; the rushx script did not wait ' + + 'for the restart.', + timeoutPrefix: + "The rushx script was not admitted before the daemon could restart for another request's environment. A " + + 'script waits for a pending restart so that the restart does not wait for the script, and the restart ' + + 'waits for' + }); + } + + async #waitAsync( + ticket: IWorkspaceRestartTicket, + options: IWorkspaceRestartDrainOptions, + kind: IWaitKind ): Promise { const state: IMutableTicket = ticket as IMutableTicket; if (state.left || state.waitingForDrain) throw new Error('The restart ticket is not being served.'); + if (kind.restarts) this.#restartCandidates.add(state); state.waitingForDrain = true; this.#stopServing(state); const waivedFor: IMutableTicket[] = options.waivesTimeoutForServedWork ? Array.from(this.#serving) : []; let remainingMs: number | undefined = options.waitTimeoutMs; let waivedMs: number = 0; let reported: number | undefined; - const report = (servingCount: number): void => { - if (servingCount > 0 && servingCount !== reported) { - reported = servingCount; - options.onServingCountChanged?.(servingCount); + const report = (): void => { + const count: number = kind.countWaitedFor(); + if (count > 0 && count !== reported) { + reported = count; + options.onServingCountChanged?.(count); } }; try { - while (this.#serving.size > 0) { + while (kind.isBlocked()) { if (options.noWait) { - throw new RequestSchedulerError( - RequestSchedulerErrorCode.NoWait, - 'Another environment is still being served; the request did not wait for a restart.' - ); + throw new RequestSchedulerError(RequestSchedulerErrorCode.NoWait, kind.noWaitMessage); } - report(this.#serving.size); + report(); const waived: boolean = !this.#isServingScript() && waivedFor.some((served: IMutableTicket) => this.#serving.has(served)); const startedAt: number = Date.now(); @@ -125,7 +196,7 @@ export class WorkspaceRestartArbiter { await this.#waitForChangeAsync( options.abortSignal, waived || remainingMs === undefined ? undefined : startedAt + remainingMs, - waivedMs + () => this.#createTimeoutError(kind, waivedMs) ); } finally { this.#countListeners.delete(report); @@ -151,9 +222,9 @@ export class WorkspaceRestartArbiter { this.#notifyChange(); } - /** Reports the new count, and wakes every waiting candidate to re-check what it waits for. */ + /** Reports the new count, and wakes every waiting request to re-check what it waits for. */ #notifyChange(): void { - for (const listener of Array.from(this.#countListeners)) listener(this.#serving.size); + for (const listener of Array.from(this.#countListeners)) listener(); for (const listener of Array.from(this.#listeners)) listener(); } @@ -161,12 +232,11 @@ export class WorkspaceRestartArbiter { return Array.from(this.#serving).some((served: IMutableTicket) => served.runsScript); } - #createTimeoutError(waivedMs: number): RequestSchedulerError { + #createTimeoutError(kind: IWaitKind, waivedMs: number): RequestSchedulerError { const script: boolean = this.#isServingScript(); return new RequestSchedulerError( RequestSchedulerErrorCode.WaitTimeout, - 'The request was not admitted before the daemon could restart for its environment, which waits for the ' + - `requests that the daemon is serving to finish${script ? SCRIPT_TIMEOUT_CLAUSE : ''}` + + `${kind.timeoutPrefix} the requests that the daemon is serving to finish${script ? SCRIPT_TIMEOUT_CLAUSE : ''}` + `${formatWaivedTime(waivedMs)}. ${script ? SCRIPT_TIMEOUT_REMEDY : TIMEOUT_REMEDY}` ); } @@ -174,7 +244,7 @@ export class WorkspaceRestartArbiter { #waitForChangeAsync( abortSignal: AbortSignal, deadline: number | undefined, - waivedMs: number + createTimeoutError: () => RequestSchedulerError ): Promise { return new Promise((resolve, reject) => { let timer: ReturnType | undefined; @@ -201,7 +271,7 @@ export class WorkspaceRestartArbiter { abortSignal.addEventListener('abort', settleAborted, { once: true, signal: unsubscribe.signal }); if (deadline !== undefined) { timer = setTimeout( - () => settle(this.#createTimeoutError(waivedMs)), + () => settle(createTimeoutError()), Math.min(MAX_TIMER_DELAY_MS, Math.max(0, deadline - Date.now())) ); } diff --git a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts index 844b581ed1..84f3e26eff 100644 --- a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts +++ b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts @@ -19,6 +19,7 @@ import { import { ProductionDaemonRequestResolver } from '../ProductionDaemonRequestResolver'; import type { IDaemonRequestResolver } from '../DaemonRequestDispatcher'; import { RushDaemonHost } from '../RushDaemonHost'; +import { RushDaemonRequestResolver } from '../RushDaemonRequestResolver'; import { WorkspaceSession } from '../WorkspaceSession'; import { getWorkspaceGenerationToken } from '../WorkspaceGeneration'; import type { GetWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; @@ -36,6 +37,8 @@ export class DaemonGraphTestFixture implements AsyncDisposable { public getSuccessorLaunchAsync: GetWorkspaceSuccessorLaunchAsync | undefined; /** Awaited before each workspace session is created, including a request's graph load or reload. */ public beforeCreateSessionAsync: (() => Promise) | undefined; + /** Also serves rushx package scripts, like the production host. Set it in `createAsync`'s `configure`. */ + public servesRushx: boolean = false; public readonly folder: string = fs.realpathSync.native( fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-graph-')) ); @@ -141,7 +144,10 @@ export class DaemonGraphTestFixture implements AsyncDisposable { } private async _startAsync(): Promise { - const resolver: IDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const production: IDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const resolver: IDaemonRequestResolver = this.servesRushx + ? new RushDaemonRequestResolver(production) + : production; this.host = await RushDaemonHost.startAsync({ repoRoot: this.folder, rushVersion: Rush.version, diff --git a/libraries/rush-daemon/src/test/RequestAdmissionIntegration.test.ts b/libraries/rush-daemon/src/test/RequestAdmissionIntegration.test.ts index fb0f0e31fe..48206eda81 100644 --- a/libraries/rush-daemon/src/test/RequestAdmissionIntegration.test.ts +++ b/libraries/rush-daemon/src/test/RequestAdmissionIntegration.test.ts @@ -21,11 +21,14 @@ import { } from '../GlobalCommandRequestRouter'; import type { GlobalCommandExecutor, - IGlobalCommandExecutionResult + IGlobalCommandExecutionResult, + IGlobalCommandRequestResult } from '../GlobalCommandRequestRouter'; import type { IInteractiveRequestSession } from '../InteractiveRequestInputRouter'; import { InteractiveRequestInputRouter } from '../InteractiveRequestInputRouter'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import { type IRequestLease, RequestExclusivityClass, RequestScheduler } from '../RequestScheduler'; +import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; import { TEST_ENGINE_SHAPE, TestOperationRunner, @@ -262,6 +265,60 @@ describe('request admission integration', () => { jest.useRealTimers(); }); + it('names the configured wait timeout when a global command times out on a remaining budget', async () => { + jest.useFakeTimers(); + const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter( + new TestWorkspaceSession(TEST_REPO_ROOT) + ); + const release: { readonly promise: Promise; readonly resolve: () => void } = createDeferred(); + const activeStarted: { readonly promise: Promise; readonly resolve: () => void } = createDeferred(); + const active: Promise = router.executeAsync( + createRequest(router, 'active', 'custom-active'), + createBlockingExecutor(activeStarted.resolve, release.promise), + new AdmissionClient() + ); + await activeStarted.promise; + // The workspace lifecycle spends part of the client's budget waiting at its gate, then hands the remainder to the + // router. + const lifecycle: RequestAdmissionController = new RequestAdmissionController({ + admission: { waitTimeoutMs: 100 }, + client: { abortSignal: new AbortController().signal }, + requestId: 'timeout' + }); + try { + const gate: RequestScheduler = new RequestScheduler(); + const gateHolder: IRequestLease = await gate.acquireAsync({ + exclusivityClass: RequestExclusivityClass.Exclusive + }); + const gateLease: Promise = lifecycle.acquireAsync(gate, RequestExclusivityClass.SharedBuild); + await jest.advanceTimersByTimeAsync(40); + gateHolder.release(); + (await gateLease).release(); + const executor: jest.Mock, []> = jest.fn(async () => ({ + exitCode: 0 + })); + const timeoutPromise: Promise = router.executeAsync( + createRequest(router, 'timeout', 'list', lifecycle.remainingAdmission), + executor, + new AdmissionClient() + ); + await jest.advanceTimersByTimeAsync(60); + + await expect(timeoutPromise).resolves.toMatchObject({ + admissionErrorCode: 'wait-timeout', + errorMessage: + 'The request was not admitted within its 100ms wait timeout while waiting for workspace admission. ' + + 'Use --wait-timeout to wait longer.' + }); + expect(executor).not.toHaveBeenCalled(); + } finally { + lifecycle.dispose(); + release.resolve(); + await active; + jest.useRealTimers(); + } + }); + it('cancels queued work on a progress write failure without leaking admission', async () => { const router: GlobalCommandRequestRouter = new GlobalCommandRequestRouter( new TestWorkspaceSession(TEST_REPO_ROOT) diff --git a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts index a91e129d94..328356e3ac 100644 --- a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts +++ b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts @@ -3,7 +3,10 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; -import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; +import type { + IDaemonRequestAdmissionOptions, + IDaemonRequestQueuePositionMessage +} from '@rushstack/rush-daemon-protocol'; import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestScheduler'; import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; @@ -117,6 +120,23 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { expect(arbiter.servingCount).toBe(0); }); + it('spends an explicit timeout while it waits for the drain, but not on the work around that wait', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest({ waitTimeoutMs: 10_000 }); + // Such as capturing the request's inputs before the wait, and planning the restart after it. + await delayAsync(1_000); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); + await delayAsync(300); + arbiter.leave(serving); + await draining; + await delayAsync(1_000); + const remainingMs: number | undefined = admission.remainingAdmission?.waitTimeoutMs; + expect(remainingMs).toBeLessThanOrEqual(9_710); + expect(remainingMs).toBeGreaterThan(8_700); + arbiter.leave(ticket); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); + it('applies no-wait to the drain', async () => { const { admission, arbiter, serving, ticket } = createDrainTest({ noWait: true }); const error: unknown = await admission @@ -129,3 +149,111 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { expect(arbiter.servingCount).toBe(0); }); }); + +interface IPendingRestartTest { + readonly admission: RequestAdmissionController; + readonly arbiter: WorkspaceRestartArbiter; + readonly candidate: IWorkspaceRestartTicket; + readonly draining: Promise; + readonly positions: number[]; + readonly script: IWorkspaceRestartTicket; + readonly serving: IWorkspaceRestartTicket; +} + +/** A rushx script with the given admission options, which arrives while a restart candidate drains one request. */ +function createPendingRestartTest( + options: IDaemonRequestAdmissionOptions, + servingOptions?: IWorkspaceRestartTicketOptions +): IPendingRestartTest { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(servingOptions); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const draining: Promise = arbiter.waitForDrainAsync(candidate, { + abortSignal: new AbortController().signal, + noWait: undefined, + waitTimeoutMs: undefined + }); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const positions: number[] = []; + const admission: RequestAdmissionController = new RequestAdmissionController({ + admission: options, + client: { + abortSignal: new AbortController().signal, + supportsRequestAdmission: true, + writeQueuePositionAsync: (message: IDaemonRequestQueuePositionMessage) => { + positions.push(message.payload.position); + return Promise.resolve(); + } + }, + requestId: 'rushx-script' + }); + return { admission, arbiter, candidate, draining, positions, script, serving }; +} + +async function finishPendingRestartTestAsync(test: IPendingRestartTest): Promise { + test.arbiter.leave(test.serving); + test.arbiter.leave(test.script); + await test.draining; + test.arbiter.leave(test.candidate); + test.admission.dispose(); + expect(test.arbiter.servingCount).toBe(0); +} + +describe('RequestAdmissionController.waitForPendingRestartAsync', () => { + it('waits past a client-default timeout while the restart drains an earlier build, and reports a position', async () => { + const test: IPendingRestartTest = createPendingRestartTest({ + waitTimeoutMs: 50, + waitTimeoutIsDefault: true + }); + const waiting: Promise = test.admission.waitForPendingRestartAsync(test.arbiter, test.script); + await delayAsync(250); + expect(await isSettledAsync(waiting)).toBe(false); + test.arbiter.leave(test.serving); + await test.draining; + test.arbiter.leave(test.candidate); + await waiting; + // The build and the candidate, then the candidate alone. + expect(test.positions).toEqual([2, 1]); + // The script is then told to run on the successor, which gets its default again. + expect(test.admission.remainingAdmission?.waitTimeoutMs).toBeGreaterThan(0); + await finishPendingRestartTestAsync(test); + }); + + it('applies a client-default timeout while a rushx script is served, and says why', async () => { + const test: IPendingRestartTest = createPendingRestartTest( + { waitTimeoutMs: 50, waitTimeoutIsDefault: true }, + { runsScript: true } + ); + const error: unknown = await test.admission + .waitForPendingRestartAsync(test.arbiter, test.script) + .catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + expect((error as Error).message).toContain( + 'A script waits for a pending restart so that the restart does not wait for the script' + ); + expect((error as Error).message).toContain( + 'including a rushx script that may not exit until it is stopped' + ); + await finishPendingRestartTestAsync(test); + }); + + it('applies an explicit timeout while the restart drains an earlier build', async () => { + const test: IPendingRestartTest = createPendingRestartTest({ waitTimeoutMs: 50 }); + const error: unknown = await test.admission + .waitForPendingRestartAsync(test.arbiter, test.script) + .catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + expect((error as Error).message).not.toContain('including a rushx script'); + await finishPendingRestartTestAsync(test); + }); + + it('applies no-wait at once', async () => { + const test: IPendingRestartTest = createPendingRestartTest({ noWait: true }); + const error: unknown = await test.admission + .waitForPendingRestartAsync(test.arbiter, test.script) + .catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.NoWait); + expect(test.positions).toEqual([]); + await finishPendingRestartTestAsync(test); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspacePreparationAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspacePreparationAdmission.test.ts new file mode 100644 index 0000000000..a1d868b232 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspacePreparationAdmission.test.ts @@ -0,0 +1,130 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('@microsoft/rush-lib', () => { + const actual: typeof import('@microsoft/rush-lib') = jest.requireActual('@microsoft/rush-lib'); + return { + ...actual, + captureProjectConfigurationFingerprintAsync: jest.fn(actual.captureProjectConfigurationFingerprintAsync) + }; +}); + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import * as rushLib from '@microsoft/rush-lib'; + +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { createDeferred, type IDeferred, type ITerminalExchange } from './DaemonRequestWireTestUtilities'; + +jest.setTimeout(60_000); + +const BUILD_A: string[] = ['build', '--to', 'a', '--parallelism', '3']; +/** Longer than the wait timeout, so that a request whose capture counted would have none left. */ +const CAPTURE_MS: number = 2_500; +const WAIT_TIMEOUT_MS: number = 2_000; +const captureMock: jest.MockedFunction = + jest.mocked(rushLib.captureProjectConfigurationFingerprintAsync); + +function delayAsync(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +async function waitForAsync(predicate: () => boolean, description: string): Promise { + const deadline: number = Date.now() + 30_000; + while (!predicate()) { + if (Date.now() > deadline) throw new Error(`Timed out waiting for ${description}.`); + await delayAsync(20); + } +} + +async function createFixtureAsync(): Promise { + return await DaemonGraphTestFixture.createAsync((created) => { + created.write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\nruns.txt\nrelease-a\n'); + created.write( + 'a/build.cjs', + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','a\\n');" + + "const wait=()=>fs.existsSync('../release-a')?console.log('finished-a'):setTimeout(wait,20);wait();" + ); + }); +} + +/** Makes the next project configuration capture, which a warm build runs before it is routed, take `CAPTURE_MS`. */ +function slowDownNextCapture(): IDeferred { + const captured: IDeferred = createDeferred(); + const actual: typeof rushLib.captureProjectConfigurationFingerprintAsync = jest.requireActual< + typeof rushLib + >('@microsoft/rush-lib').captureProjectConfigurationFingerprintAsync; + captureMock.mockClear(); + captureMock.mockImplementationOnce(async (...args) => { + await delayAsync(CAPTURE_MS); + try { + return await actual(...args); + } finally { + captured.resolve(); + } + }); + return captured; +} + +describe('explicit wait timeouts and a request preparing to run', () => { + it('does not spend the timeout on capturing the request inputs, so the request can still wait', async () => { + const fixture: DaemonGraphTestFixture = await createFixtureAsync(); + const releaseFile: string = path.join(fixture.folder, 'release-a'); + try { + const long: Promise = fixture.runAsync(BUILD_A); + await waitForAsync(() => fixture.runs().includes('a'), 'the long build to start'); + + const captured: IDeferred = slowDownNextCapture(); + const behind: Promise = fixture.runAsync(BUILD_A, { + admission: { waitTimeoutMs: WAIT_TIMEOUT_MS } + }); + await captured.promise; + // The request now waits for the running build, which ends well within its timeout. + await delayAsync(200); + fs.writeFileSync(releaseFile, ''); + + expect((await long).terminal).toMatchObject({ payload: { exitCode: 0 } }); + expect((await behind).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(captureMock).toHaveBeenCalledTimes(1); + } finally { + // A failed expectation must not leave the long build running, which would keep the fixture from shutting down. + fs.writeFileSync(releaseFile, ''); + await fixture[Symbol.asyncDispose](); + } + }); + + it('still spends the timeout while the request waits after capturing its inputs', async () => { + const fixture: DaemonGraphTestFixture = await createFixtureAsync(); + const releaseFile: string = path.join(fixture.folder, 'release-a'); + try { + const long: Promise = fixture.runAsync(BUILD_A); + await waitForAsync(() => fixture.runs().includes('a'), 'the long build to start'); + + const captured: IDeferred = slowDownNextCapture(); + const startedAt: number = Date.now(); + const behind: Promise = fixture.runAsync(BUILD_A, { + admission: { waitTimeoutMs: WAIT_TIMEOUT_MS } + }); + await captured.promise; + const timedOut: ITerminalExchange = await behind; + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(CAPTURE_MS + WAIT_TIMEOUT_MS - 100); + expect(timedOut.terminal).toMatchObject({ + kind: 'requestResult', + payload: { + exitCode: 1, + admissionErrorCode: 'wait-timeout', + errorMessage: + `The request was not admitted within its ${WAIT_TIMEOUT_MS}ms wait timeout while waiting for the ` + + 'running build of the workspace operation graph. Use --wait-timeout to wait longer.' + } + }); + + fs.writeFileSync(releaseFile, ''); + expect((await long).terminal).toMatchObject({ payload: { exitCode: 0 } }); + } finally { + fs.writeFileSync(releaseFile, ''); + await fixture[Symbol.asyncDispose](); + } + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts index 23b261466b..d88b99799f 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRequestAdmission.test.ts @@ -12,7 +12,11 @@ import { RequestScheduler, RequestSchedulerErrorCode } from '../RequestScheduler'; -import { AdmissionProgress, RequestAdmissionController } from '../WorkspaceRequestAdmission'; +import { + AdmissionProgress, + freezeDaemonRequestAdmissionOptions, + RequestAdmissionController +} from '../WorkspaceRequestAdmission'; const DEFAULT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 100, waitTimeoutIsDefault: true }; const EXPLICIT_BUDGET: IDaemonRequestAdmissionOptions = { waitTimeoutMs: 100 }; @@ -49,6 +53,24 @@ function createController( return new RequestAdmissionController({ admission, client: { abortSignal }, requestId: 'request' }); } +/** Makes `controller` wait `waitMs` behind another request, on a scheduler of its own, and then be admitted. */ +async function waitBehindAnotherRequestAsync( + controller: RequestAdmissionController, + waitMs: number +): Promise { + const scheduler: RequestScheduler = new RequestScheduler(); + const other: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.Exclusive + }); + const waiting: IAcquisition = track(controller.acquireAsync(scheduler, RequestExclusivityClass.SharedBuild)); + await jest.advanceTimersByTimeAsync(waitMs); + expect(waiting.settled).toBe(false); + other.release(); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.lease).toBeDefined(); + waiting.lease?.release(); +} + describe(RequestAdmissionController.name, () => { let scheduler: RequestScheduler; let owner: IRequestLease; @@ -111,6 +133,25 @@ describe(RequestAdmissionController.name, () => { } ); + it.each(BUDGETS)( + 'carries what is left of $kind timeout after it waits behind a transition to its later waits', + async ({ budget }) => { + const controller: RequestAdmissionController = createController(budget); + const behind: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + // The transition itself waits for 30ms, which the request spends, and then loads the graph. + await jest.advanceTimersByTimeAsync(30); + transition.setActive(true); + await jest.advanceTimersByTimeAsync(500); + scheduler.downgradeExclusiveLease(owner, RequestExclusivityClass.SharedBuild); + transition.setActive(false); + await jest.advanceTimersByTimeAsync(0); + expect(behind.error).toBeUndefined(); + behind.lease?.release(); + expect(controller.remainingAdmission).toEqual({ ...budget, waitTimeoutMs: 70 }); + controller.dispose(); + } + ); + it('fails once the transition it waits behind has made progress for ten times its timeout', async () => { const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); transition.setActive(true); @@ -162,6 +203,57 @@ describe(RequestAdmissionController.name, () => { controller.dispose(); }); + it('reports a zero timeout behind a transition as a plain timeout, not as the paused limit', async () => { + const controller: RequestAdmissionController = createController({ waitTimeoutMs: 0 }); + transition.setActive(true); + const waiting: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(0); + expect(waiting.error).toMatchObject({ + code: RequestSchedulerErrorCode.WaitTimeout, + message: + "The request was not admitted within its 0ms wait timeout while waiting for another request's load or " + + 'reload of the workspace graph. Use --wait-timeout to wait longer.' + }); + expect(scheduler.queuedRequestCount).toBe(0); + controller.dispose(); + }); + + it('reports paused time that did not count when a later boundary times out', async () => { + const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); + transition.setActive(true); + const behind: IAcquisition = track(controller.acquireBehindTransitionAsync(scheduler, transition)); + await jest.advanceTimersByTimeAsync(500); + scheduler.downgradeExclusiveLease(owner, RequestExclusivityClass.SharedBuild); + transition.setActive(false); + await jest.advanceTimersByTimeAsync(0); + expect(behind.error).toBeUndefined(); + behind.lease?.release(); + // A routing boundary receives a frozen copy of the remaining budget. + const remaining: IDaemonRequestAdmissionOptions | undefined = controller.remainingAdmission; + const boundary: RequestAdmissionController = new RequestAdmissionController({ + admission: remaining && freezeDaemonRequestAdmissionOptions(remaining), + client: { abortSignal: new AbortController().signal }, + requestId: 'request' + }); + + const here: IAcquisition = track(controller.acquireAsync(scheduler, RequestExclusivityClass.Exclusive)); + const there: IAcquisition = track(boundary.acquireAsync(scheduler, RequestExclusivityClass.Exclusive)); + await jest.advanceTimersByTimeAsync(99); + expect(here.settled || there.settled).toBe(false); + await jest.advanceTimersByTimeAsync(1); + const error: { code: RequestSchedulerErrorCode; message: string } = { + code: RequestSchedulerErrorCode.WaitTimeout, + message: + 'The request was not admitted within its 100ms wait timeout while waiting for workspace admission; 0.5s ' + + 'spent earlier while another request loaded the workspace graph did not count. ' + + 'Use --wait-timeout to wait longer.' + }; + expect(here.error).toMatchObject(error); + expect(there.error).toMatchObject(error); + boundary.dispose(); + controller.dispose(); + }); + it('reports cancellation behind a transition as an abort', async () => { const client: AbortController = new AbortController(); const controller: RequestAdmissionController = createController(DEFAULT_BUDGET, client.signal); @@ -175,15 +267,15 @@ describe(RequestAdmissionController.name, () => { }); it.each(BUDGETS)( - 'keeps the unspent part of $kind timeout across admitted work for a later admission wait', + 'spends $kind timeout only while it waits, not on the work before and between its waits', async ({ budget }) => { const controller: RequestAdmissionController = createController(budget); - await jest.advanceTimersByTimeAsync(40); - const work: Promise = controller.runOutsideWaitBudgetAsync( - () => new Promise((resolve) => setTimeout(resolve, 10_000)) - ); + // Such as capturing the request's inputs before its first wait. + await jest.advanceTimersByTimeAsync(10_000); + expect(controller.remainingAdmission).toEqual({ ...budget, waitTimeoutMs: 100 }); + await waitBehindAnotherRequestAsync(controller, 40); + // Such as loading the graph, routing and execution. await jest.advanceTimersByTimeAsync(10_000); - await work; expect(controller.remainingAdmission).toEqual({ ...budget, waitTimeoutMs: 60 }); const waiting: IAcquisition = track( @@ -201,4 +293,53 @@ describe(RequestAdmissionController.name, () => { controller.dispose(); } ); + + it('does not spend a default timeout at the graph-execution gate, which it does not limit', async () => { + const controller: RequestAdmissionController = createController(DEFAULT_BUDGET); + const running: IAcquisition = track( + controller.acquireGraphExecutionAsync(scheduler, RequestExclusivityClass.SharedBuild) + ); + await jest.advanceTimersByTimeAsync(10_000); + expect(running.settled).toBe(false); + owner.release(); + await jest.advanceTimersByTimeAsync(0); + running.lease?.release(); + expect(controller.remainingAdmission).toEqual({ ...DEFAULT_BUDGET, waitTimeoutMs: 100 }); + controller.dispose(); + }); + + it('names the configured timeout, not the remainder, when a later routing boundary times out', async () => { + const controller: RequestAdmissionController = createController(EXPLICIT_BUDGET); + await waitBehindAnotherRequestAsync(controller, 23); + const remaining: IDaemonRequestAdmissionOptions | undefined = controller.remainingAdmission; + expect(remaining).toEqual({ ...EXPLICIT_BUDGET, waitTimeoutMs: 77 }); + const boundary: RequestAdmissionController = new RequestAdmissionController({ + admission: remaining, + client: { abortSignal: new AbortController().signal }, + requestId: 'request' + }); + await waitBehindAnotherRequestAsync(boundary, 7); + // A boundary that hands its own remainder on again still names the client's timeout. + const nested: RequestAdmissionController = new RequestAdmissionController({ + admission: boundary.remainingAdmission, + client: { abortSignal: new AbortController().signal }, + requestId: 'request' + }); + + const waiting: IAcquisition = track( + nested.acquireGraphExecutionAsync(scheduler, RequestExclusivityClass.SharedBuild) + ); + await jest.advanceTimersByTimeAsync(69); + expect(waiting.settled).toBe(false); + await jest.advanceTimersByTimeAsync(1); + expect(waiting.error).toMatchObject({ + code: RequestSchedulerErrorCode.WaitTimeout, + message: + 'The request was not admitted within its 100ms wait timeout while waiting for the running build of the ' + + 'workspace operation graph. Use --wait-timeout to wait longer.' + }); + nested.dispose(); + boundary.dispose(); + controller.dispose(); + }); }); diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts index 10a86e2314..3f91594741 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts @@ -26,6 +26,12 @@ async function isSettledAsync(promise: Promise): Promise { return settled; } +function expectWaitTimeout(error: unknown): RequestSchedulerError { + expect(error).toBeInstanceOf(RequestSchedulerError); + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + return error as RequestSchedulerError; +} + describe(WorkspaceRestartArbiter.name, () => { it('proceeds immediately when no other request is being served', async () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); @@ -143,12 +149,6 @@ describe(WorkspaceRestartArbiter.name, () => { waivesTimeoutForServedWork: true }; - function expectWaitTimeout(error: unknown): RequestSchedulerError { - expect(error).toBeInstanceOf(RequestSchedulerError); - expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); - return error as RequestSchedulerError; - } - it('does not spend the timeout on requests served when the wait began, and returns that time', async () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const earlier: IWorkspaceRestartTicket = arbiter.enter(); @@ -260,4 +260,190 @@ describe(WorkspaceRestartArbiter.name, () => { expect(arbiter.servingCount).toBe(0); }); }); + + describe('for a rushx script while a restart is pending', () => { + const WAIVED: IWorkspaceRestartDrainOptions = { + ...WAIT, + waitTimeoutMs: 50, + waivesTimeoutForServedWork: true + }; + + it('reports a restart as pending from the start of its drain until the candidate leaves', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + expect(arbiter.hasPendingRestart(running)).toBe(false); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + expect(arbiter.hasPendingRestart(running)).toBe(true); + expect(arbiter.hasPendingRestart(candidate)).toBe(false); + arbiter.leave(running); + await draining; + // After its drain, the candidate still waits for #gate and plans the restart, so a new script must not start. + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + expect(arbiter.hasPendingRestart(script)).toBe(true); + arbiter.leave(candidate); + expect(arbiter.hasPendingRestart(script)).toBe(false); + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + + it('proceeds immediately when no restart is pending', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const counts: number[] = []; + const waivedMs: number = await arbiter.waitForPendingRestartAsync(script, { + ...WAIT, + noWait: true, + onServingCountChanged: (count: number) => counts.push(count) + }); + expect(waivedMs).toBe(0); + expect(counts).toEqual([]); + expect(arbiter.servingCount).toBe(2); + for (const served of [script, serving]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('waits until the candidate leaves, and the drain does not wait for the waiting script', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const counts: number[] = []; + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { + ...WAIT, + onServingCountChanged: (count: number) => counts.push(count) + }); + // The script waits for the running script and for the candidate. + expect(counts).toEqual([2]); + arbiter.leave(running); + await draining; + expect(counts).toEqual([2, 1]); + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(candidate); + expect(await waiting).toBe(0); + expect(arbiter.servingCount).toBe(1); + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + + it('waits for every pending restart', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const first: IWorkspaceRestartTicket = arbiter.enter(); + const second: IWorkspaceRestartTicket = arbiter.enter(); + const firstDrain: Promise = arbiter.waitForDrainAsync(first, WAIT); + const secondDrain: Promise = arbiter.waitForDrainAsync(second, WAIT); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIT); + arbiter.leave(running); + await firstDrain; + arbiter.leave(first); + await secondDrain; + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(second); + await waiting; + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + + it.each([ + ['no-wait', RequestSchedulerErrorCode.NoWait, 'the rushx script did not wait for the restart.'], + [ + 'timeout', + RequestSchedulerErrorCode.WaitTimeout, + "The rushx script was not admitted before the daemon could restart for another request's environment. " + + 'A script waits for a pending restart so that the restart does not wait for the script, and the ' + + 'restart waits for the requests that the daemon is serving to finish, including a rushx script that ' + + 'may not exit until it is stopped. Stop the script, or use --wait-timeout to wait longer.' + ], + ['abort', RequestSchedulerErrorCode.Aborted, 'The request was aborted before execution.'] + ])( + 'reports %s as an admission failure and counts the script as served again', + async (mode, code, text) => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const abort: AbortController = new AbortController(); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { + abortSignal: abort.signal, + noWait: mode === 'no-wait' ? true : undefined, + waitTimeoutMs: mode === 'timeout' ? 10 : undefined + }); + if (mode === 'abort') abort.abort(); + const error: unknown = await waiting.catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(RequestSchedulerError); + expect((error as RequestSchedulerError).code).toBe(code); + expect((error as Error).message).toContain(text); + expect(arbiter.servingCount).toBe(2); + arbiter.leave(running); + expect(await isSettledAsync(draining)).toBe(false); + arbiter.leave(script); + await draining; + arbiter.leave(candidate); + expect(arbiter.servingCount).toBe(0); + } + ); + + it('with waivesTimeoutForServedWork, does not spend the timeout while an earlier build is served', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIVED); + await delayAsync(150); + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(earlier); + await draining; + arbiter.leave(candidate); + expect(await waiting).toBeGreaterThanOrEqual(140); + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + + it('with waivesTimeoutForServedWork, spends the timeout once the drain ends, and names the waived time', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const outcome: Promise = arbiter + .waitForPendingRestartAsync(script, WAIVED) + .catch((caught: unknown) => caught); + await delayAsync(150); + expect(await isSettledAsync(outcome)).toBe(false); + arbiter.leave(earlier); + await draining; + const error: RequestSchedulerError = expectWaitTimeout(await outcome); + expect(error.message).toMatch(/^The rushx script was not admitted before the daemon could restart/); + expect(error.message).toMatch( + /to finish; \d+(\.\d)?s spent waiting for requests that were already running did not count\. Use --wait-timeout to wait longer\.$/ + ); + for (const served of [candidate, script]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('with waivesTimeoutForServedWork, spends the timeout while a rushx script is served', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const earlier: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const outcome: Promise = arbiter + .waitForPendingRestartAsync(script, WAIVED) + .catch((caught: unknown) => caught); + await delayAsync(150); + expect(await isSettledAsync(outcome)).toBe(true); + expect(expectWaitTimeout(await outcome).message).toContain('including a rushx script'); + for (const served of [running, earlier, script]) arbiter.leave(served); + await draining; + arbiter.leave(candidate); + expect(arbiter.servingCount).toBe(0); + }); + }); }); diff --git a/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts new file mode 100644 index 0000000000..4bff3d0a71 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts @@ -0,0 +1,305 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; +import { + DaemonFrameType, + decodeDaemonControlMessage, + type DaemonControlMessage, + type IDaemonFrame, + type IDaemonRequestEnvelope +} from '@rushstack/rush-daemon-protocol'; + +import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import type { DaemonRequestWireClient, ITerminalExchange } from './DaemonRequestWireTestUtilities'; +import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; +import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; + +jest.setTimeout(60_000); + +const BUILD_A: string[] = ['build', '--to', 'a', '--parallelism', '3']; +const RELEASE_FILE: string = 'release-serve'; +const LATE_RELEASE_FILE: string = 'release-serve2'; + +function delayAsync(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +async function waitForAsync(predicate: () => boolean, description: string): Promise { + const deadline: number = Date.now() + 30_000; + while (!predicate()) { + if (Date.now() > deadline) throw new Error(`Timed out waiting for ${description}.`); + await delayAsync(20); + } +} + +function expectSuccess(exchange: ITerminalExchange): void { + expect(exchange.terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); +} + +/** + * Project a's `serve` and `serve2` scripts run, like dev servers, until the test creates their release files, + * `release-serve` and `release-serve2`. + */ +function createServingFixtureAsync( + configure?: (fixture: DaemonGraphTestFixture) => void +): Promise { + return DaemonGraphTestFixture.createAsync((created) => { + created.servesRushx = true; + created.write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\nruns.txt\nrelease-*\n'); + created.write( + 'a/package.json', + JSON.stringify({ + name: 'a', + version: '1.0.0', + dependencies: {}, + scripts: { + '_phase:compile': 'node build.cjs', + serve: 'node serve.cjs', + serve2: 'node serve.cjs serve2' + } + }) + ); + created.write( + 'a/serve.cjs', + "const fs=require('node:fs');const n=process.argv[2]||'serve';fs.appendFileSync('../runs.txt',n+'-start\\n');" + + "const t=setInterval(()=>{if(fs.existsSync('../release-'+n)){clearInterval(t);" + + "fs.appendFileSync('../runs.txt',n+'-end\\n');}},20);" + ); + configure?.(created); + }); +} + +interface IServedScript { + readonly exchange: Promise; + readonly settled: () => boolean; +} + +async function serveAsync(fixture: DaemonGraphTestFixture): Promise { + let settled: boolean = false; + const exchange: Promise = fixture + .runAsync(['serve'], { + commandOrigin: 'custom', + invocationKind: 'rushx', + cwd: path.join(fixture.folder, 'a') + }) + .finally(() => { + settled = true; + }); + await waitForAsync(() => fixture.runs().includes('serve-start') || settled, 'the served script to start'); + expect(settled).toBe(false); + return { exchange, settled: () => settled }; +} + +interface IStreamedRequest { + readonly exchange: Promise; + /** The queue positions that the daemon has reported so far. */ + readonly positions: ReadonlyArray; + readonly settled: () => boolean; +} + +/** Starts a request, recording its queue positions as they arrive. */ +async function startRequestAsync( + fixture: DaemonGraphTestFixture, + argv: string[], + overrides: Partial +): Promise { + const client: DaemonRequestWireClient = await fixture.connectAsync(); + const payload: IDaemonRequestEnvelope = fixture.envelope(argv, overrides); + await client.sendControlAsync({ kind: 'requestStart', payload }); + const positions: number[] = []; + let settled: boolean = false; + const readAsync = async (): Promise => { + const frames: IDaemonFrame[] = []; + try { + for (;;) { + const frame: IDaemonFrame = await client.readFrameAsync(); + frames.push(frame); + if (frame.kind !== DaemonFrameType.controlJson) continue; + const message: DaemonControlMessage = decodeDaemonControlMessage(frame.payload); + if (message.kind === 'queuePosition') positions.push(message.payload.position); + if (message.kind === 'requestResult' && message.payload.requestId === payload.requestId) { + return { frames, terminal: message }; + } + } + } finally { + settled = true; + await client.closeAsync(); + } + }; + const exchange: Promise = readAsync(); + // A failed expectation leaves the exchange unread until the fixture closes the connection. + exchange.catch(() => undefined); + return { exchange, positions, settled: () => settled }; +} + +function changeProjectConfiguration(fixture: DaemonGraphTestFixture): void { + const packageJsonPath: string = path.join(fixture.folder, 'c/package.json'); + const packageJson: Record = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8')); + fs.writeFileSync(packageJsonPath, JSON.stringify({ ...packageJson, description: 'changed' })); +} + +describe('workspace admission while a served rushx script runs', () => { + it('admits a reload-needing build at once and keeps the script running', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync(); + try { + expectSuccess(await fixture.runAsync(BUILD_A)); + const generation: number = fixture.host.workspaceGeneration; + const script: IServedScript = await serveAsync(fixture); + + changeProjectConfiguration(fixture); + // The reload used to wait for #gate until the script exited (#113). + expectSuccess(await fixture.runAsync(BUILD_A, { admission: { waitTimeoutMs: 3000 } })); + expect(fixture.host.workspaceGeneration).toBeGreaterThan(generation); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(WorkspaceInputChangeTier.Reload); + expect(script.settled()).toBe(false); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + expect(fixture.runs().slice(-1)).toEqual(['serve-end']); + } finally { + // A failed expectation must not leave the script running, which would keep the fixture from shutting down. + fixture.write(RELEASE_FILE, ''); + await fixture[Symbol.asyncDispose](); + } + }); + + it('keeps a native install waiting for the script, which would end with this process', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + // The install is never admitted, so no successor is launched. + created.getSuccessorLaunchAsync = () => Promise.reject(new Error('No successor was expected.')); + }); + try { + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + const install: ITerminalExchange = await fixture.runAsync(['install'], { + admission: { waitTimeoutMs: 500 } + }); + expect(install.terminal).toMatchObject({ + kind: 'requestResult', + payload: { + exitCode: 1, + admissionErrorCode: 'wait-timeout', + errorMessage: expect.stringContaining( + 'within its 500ms wait timeout while waiting for a rushx script that this daemon runs to exit.' + ) + } + }); + expect(script.settled()).toBe(false); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + expect(fixture.runs().slice(-1)).toEqual(['serve-end']); + // The refused install left the workspace serving builds. + expectSuccess(await fixture.runAsync(BUILD_A)); + } finally { + fixture.write(RELEASE_FILE, ''); + await fixture[Symbol.asyncDispose](); + } + }); + + it('restarts for a changed environment only after the script exits', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + let restartSettled: boolean = false; + const restart: Promise = fixture + .runAsync(BUILD_A, { environment: { ...fixture.environment, RUSHD_RELOAD_TIER_TEST: 'changed' } }) + .finally(() => { + restartSettled = true; + }); + await delayAsync(1000); + // The script keeps its restart ticket after it releases #gate, so the restart still drains it (#198). + expect(restartSettled).toBe(false); + expect(script.settled()).toBe(false); + expect(fixture.host.workspaceStatus.lastReloadTier).not.toBe(WorkspaceInputChangeTier.Restart); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + expect(fixture.runs().slice(-1)).toEqual(['serve-end']); + expect((await restart).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true } + }); + const restarted = await fixture.host.restartCompleted; + expect(restarted?.pid).not.toBe(before.pid); + } finally { + fixture.write(RELEASE_FILE, ''); + try { + await fixture.host.closeAsync(); + await fixture.host.restartCompleted; + } finally { + await stopSuccessorAsync(fixture.host.paths); + await fixture[Symbol.asyncDispose](); + } + } + }); + + it('has a script that arrives while a restart is pending run on the successor, without delaying the restart', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + const restart: IStreamedRequest = await startRequestAsync(fixture, BUILD_A, { + environment: { ...fixture.environment, RUSHD_RELOAD_TIER_TEST: 'changed' } + }); + await waitForAsync( + () => restart.positions.length > 0 || restart.settled(), + 'the restart to wait for the served script' + ); + const late: IStreamedRequest = await startRequestAsync(fixture, ['serve2'], { + commandOrigin: 'custom', + invocationKind: 'rushx', + cwd: path.join(fixture.folder, 'a') + }); + await waitForAsync( + () => late.positions.length > 0 || late.settled() || fixture.runs().includes('serve2-start'), + 'the later script to wait for the restart' + ); + // The later script used to start and keep the restart waiting until it exited (#96). + expect(fixture.runs()).not.toContain('serve2-start'); + // It waits for the served script and the restart. + expect(late.positions).toEqual([2]); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + // The restart does not wait for the later script, which is told to run on the successor. + for (const exchange of await Promise.all([restart.exchange, late.exchange])) { + expect(exchange.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true } + }); + } + expect(late.positions).toEqual([2, 1]); + expect(fixture.runs()).not.toContain('serve2-start'); + const restarted = await fixture.host.restartCompleted; + expect(restarted?.pid).not.toBe(before.pid); + } finally { + fixture.write(RELEASE_FILE, ''); + fixture.write(LATE_RELEASE_FILE, ''); + try { + await fixture.host.closeAsync(); + await fixture.host.restartCompleted; + } finally { + await stopSuccessorAsync(fixture.host.paths); + await fixture[Symbol.asyncDispose](); + } + } + }); +}); From 8f52b6880d0262e1a3fc432d39d9a5f1601d9272 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Mon, 28 Sep 2026 23:47:30 +0000 Subject: [PATCH 045/265] [rush-daemon] A warm-set pass takes the repository lock only when it has work Swarm integration step 30; original commit a4de651852 (merge of swarm/r07-t76 at 6b32aa340e). Scope: task 76. Brings r07's task 76 (board 1730; o05 CONFIRMED board 1904). A warm-set maintenance pass takes the native repository lease only when it has something to evict, a watcher change to apply, or a failed observation to retry. An idle daemon no longer takes common/temp/rush#.lock every 30 s, and a native command that holds the lock no longer puts the warm set in 'native-busy'. Two commits: 40a9f8f5eb (product) and 6b32aa340e (tests for the two lease exemptions a pass relies on). ch01 on tree fe0b6a452a (board 2511): rush-daemon 549/0 (544 + 5 new tests). The old WorkspaceWarmSet.ts fails exactly the 5 new tests. 15 of 16 WorkspaceWarmSet.ts mutants are killed; B6 (no protected check in #shouldEvict) survives and is a test NIT. E2E on ch01's own daemons: 0 lock holds in 150 s idle (5 before), 0 daemon lock attempts while native holds the lock 40 s (12 before, with 'native-busy'). Gate: ch01 GATE OK board 2511 (tree fe0b6a452a) Commits folded into this step (2): - 40a9f8f5eb [rush-daemon] Warm set: take the repository lock only for a maintenance pass that has work - 6b32aa340e [rush-daemon] Warm set: test the two lease exemptions a pass relies on Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...ase-only-when-needed_2026-09-28-18-45.json | 10 + libraries/rush-daemon/src/WorkspaceWarmSet.ts | 69 ++++-- .../src/test/WorkspaceWarmQuiescence.test.ts | 2 + .../src/test/WorkspaceWarmSet.test.ts | 20 ++ .../WorkspaceWarmSetMaintenanceCost.test.ts | 202 +++++++++++++++++- 5 files changed, 278 insertions(+), 25 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/warm-set-lease-only-when-needed_2026-09-28-18-45.json diff --git a/common/changes/@rushstack/rush-daemon/warm-set-lease-only-when-needed_2026-09-28-18-45.json b/common/changes/@rushstack/rush-daemon/warm-set-lease-only-when-needed_2026-09-28-18-45.json new file mode 100644 index 0000000000..984ea21e0e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/warm-set-lease-only-when-needed_2026-09-28-18-45.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Warm set: an idle maintenance pass takes the repository lock only when it releases warm resources or changes project observation, so native Rush commands no longer fail with \"Another Rush command is already running\" because of a pass that has nothing to do.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/WorkspaceWarmSet.ts b/libraries/rush-daemon/src/WorkspaceWarmSet.ts index 0be59f2249..6f1bfd9527 100644 --- a/libraries/rush-daemon/src/WorkspaceWarmSet.ts +++ b/libraries/rush-daemon/src/WorkspaceWarmSet.ts @@ -288,12 +288,17 @@ export class WorkspaceWarmSet implements AsyncDisposable { } else if (isGraphBusy(graph)) { this.#deferredReason = 'graph-busy'; } else { - nativeLease = await this.#options.acquireExecutionLeaseAsync(); - if (this.#disposed) this.#deferredReason = 'disposed'; - else if (isGraphBusy(graph)) this.#deferredReason = 'graph-busy'; - else { - await this.#evictIdleAsync(); - if (!this.#disposed) await this.#reconcileWatcherPolicyAsync(); + const status: IWorkspaceWarmSetStatus = this.getStatus(); + // Native Rush commands can't start while the repository lock is held, so a pass that would neither + // release anything nor change project observation doesn't take it. + if (this.#needsNativeLease(status)) { + nativeLease = await this.#options.acquireExecutionLeaseAsync(); + if (this.#disposed) this.#deferredReason = 'disposed'; + else if (isGraphBusy(graph)) this.#deferredReason = 'graph-busy'; + else { + await this.#evictIdleAsync(status); + if (!this.#disposed) await this.#reconcileWatcherPolicyAsync(); + } } } } @@ -353,30 +358,48 @@ export class WorkspaceWarmSet implements AsyncDisposable { this.#diagnose(this.#watcherPolicyFailure); } - async #evictIdleAsync(): Promise { + /** Whether a pass must own the repository to release resources or to apply the observation policy. */ + #needsNativeLease(status: IWorkspaceWarmSetStatus): boolean { + if (this.#watcherPolicyFailure) return true; + const projects: IWarmProject[] = this.#rankProjects(); + if (projects.some((project) => this.#shouldEvict(project, status))) return true; + const watched: ReadonlySet = this.#options.watcher.watchedProjectNames; + // The same projects that #reconcileWatcherPolicyAsync observes or stops observing. + return this.#configuration.watch + ? projects.some((project) => !this.#cleanupFailures.has(project.key) && !watched.has(project.key)) + : projects.some((project) => !project.protected && watched.has(project.key)); + } + + #shouldEvict(project: IWarmProject, status: IWorkspaceWarmSetStatus): boolean { + if (project.protected) return false; + const graph: IOperationGraph = this.#options.operationGraph; + const expired: boolean = + performance.now() - project.lastUsed >= this.#configuration.warmIdleTimeoutSeconds * 1000; + const unrequested: boolean = + project.frequency === 0 && + project.operations.every( + (operation) => !graph.resultByOperation.has(operation) && !operation.runner?.isActive + ); + // Idle expiry and every limit release runners and watchers, and finish an eviction that failed earlier. + // Otherwise the retained results of a resource-free project stay until the generation ends: they are + // revalidated on every request, they are what makes a warm no-op skip possible, and dropping them cannot + // bring daemon RSS below the budget. + return ( + unrequested || + (this.#mayRelease(project) && (expired || status.overMemoryBudget || status.overProjectLimit)) + ); + } + + async #evictIdleAsync(initialStatus: IWorkspaceWarmSetStatus): Promise { const { operationGraph: graph, watcher } = this.#options; // getStatus() re-ranks every project. Only an eviction attempt (which awaits) can change it during a pass, so // it is reused until then; recomputing it per project made a pass without evictions quadratic. - let currentStatus: IWorkspaceWarmSetStatus | undefined; + let currentStatus: IWorkspaceWarmSetStatus | undefined = initialStatus; // Retention and eviction use exactly the same ordering, reversed only to release the lowest value first. for (const project of this.#rankProjects().reverse()) { if (this.#disposed) break; if (project.protected) continue; - const status: IWorkspaceWarmSetStatus = (currentStatus ??= this.getStatus()); - const expired: boolean = - performance.now() - project.lastUsed >= this.#configuration.warmIdleTimeoutSeconds * 1000; - const unrequested: boolean = - project.frequency === 0 && - project.operations.every( - (operation) => !graph.resultByOperation.has(operation) && !operation.runner?.isActive - ); - // Idle expiry and every limit release runners and watchers, and finish an eviction that failed earlier. - // Otherwise the retained results of a resource-free project stay until the generation ends: they are - // revalidated on every request, they are what makes a warm no-op skip possible, and dropping them cannot - // bring daemon RSS below the budget. - const release: boolean = - this.#mayRelease(project) && (expired || status.overMemoryBudget || status.overProjectLimit); - if (!unrequested && !release) continue; + if (!this.#shouldEvict(project, (currentStatus ??= this.getStatus()))) continue; try { await graph.closeRunnersAsync(project.operations); if (project.operations.some((operation) => operation.runner?.isActive)) { diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmQuiescence.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmQuiescence.test.ts index ef688c54dd..a0e1a6f78e 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWarmQuiescence.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWarmQuiescence.test.ts @@ -269,6 +269,8 @@ describe('warm-set generation barriers', () => { }); await session.initializeEngineAsync(async () => engine); try { + // Observation of a project that was never requested is released, so this pass takes the native lease. + watcher.watchProjects(['a']); await warm.maintainAsync(); expect(warm.getStatus()).toMatchObject({ maintenanceState: 'failed', diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts index 8f3d81c702..ee7c972e8e 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWarmSet.test.ts @@ -393,6 +393,26 @@ describe('warm policies attached to native graphs and real filesystem watchers', expect(graph.resultByOperation.size).toBe(0); }); + it('leaves the real repository lock to a native CLI command in a pass that has nothing to release', async () => { + const { fixture, warm, graph, watcher } = await startAsync(); + const acquire = jest.spyOn(fixture.session, 'acquireExecutionLeaseAsync'); + const gate = await createNativeScriptGateAsync(fixture.folder, 'c'); + const native = runNativeCommandAsync(fixture.folder, ['build', '--only', 'c', '--parallelism', '3']); + try { + await gate.entered; + const status = await warm.maintainAsync(); + expect(status.deferredReason).toBeUndefined(); + expect(acquire).not.toHaveBeenCalled(); + expect(graph.resultByOperation.size).toBe(2); + expect([...watcher.watchedProjectNames].sort()).toEqual(['a', 'b']); + } finally { + await gate.releaseAsync(); + acquire.mockRestore(); + await native; + } + expect((await native).exitCode).toBe(0); + }); + it('does not touch paused prepared records or native ownership until a plan has been discarded', async () => { const { fixture, warm, graph } = await startAsync(); const admission = await getWorkspaceRequestScheduler(fixture.session).acquireAsync({ diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts index b9d42c040c..f8ee0fa972 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts @@ -12,6 +12,7 @@ import type { RequestScheduler } from '../RequestScheduler'; import type { WorkspaceSessionFileWatcher } from '../WorkspaceSessionFileWatcher'; import { WorkspaceWarmSet, + type IWorkspaceWarmSetOptions, type IWorkspaceWarmSetStatus, type WorkspaceWarmSetConfiguration } from '../WorkspaceWarmSet'; @@ -86,7 +87,8 @@ function createGraph(projectCount: number): ITestGraph { function attach( graph: IOperationGraph, - configuration: Partial = {} + configuration: Partial = {}, + options: Partial = {} ): WorkspaceWarmSet { return WorkspaceWarmSet.attach({ operationGraph: graph, @@ -97,10 +99,54 @@ function attach( watchedProjectNames: new Set(), watchProjects: () => undefined, unwatchProjectsAsync: async () => undefined - } as unknown as WorkspaceSessionFileWatcher + } as unknown as WorkspaceSessionFileWatcher, + ...options }); } +interface ITestWatcher { + readonly watcher: WorkspaceSessionFileWatcher; + readonly watched: Set; + readonly failNextWatch: () => void; +} + +function createWatcher(): ITestWatcher { + const watched: Set = new Set(); + let fail: boolean = false; + const watcher: WorkspaceSessionFileWatcher = { + get watchedProjectNames(): ReadonlySet { + return new Set(watched); + }, + watchProjects: (projectNames: Iterable): void => { + if (fail) { + fail = false; + throw new Error('watch failed'); + } + for (const name of projectNames) watched.add(name); + }, + unwatchProjectsAsync: async (projectNames: Iterable): Promise => { + for (const name of projectNames) watched.delete(name); + } + } as unknown as WorkspaceSessionFileWatcher; + return { + watcher, + watched, + failNextWatch: () => { + fail = true; + } + }; +} + +function createLease(): jest.Mock, []> { + return jest.fn(async () => ({ [Symbol.asyncDispose]: async () => undefined })); +} + +async function settleAsync(warm: WorkspaceWarmSet): Promise { + // Lets the passes scheduled by attachment and by a simulated request finish. + await new Promise((resolve) => setTimeout(resolve, 10)); + await warm.maintainAsync(); +} + function createResidentRunner(): IOperationRunner { let active: boolean = true; return { @@ -173,4 +219,156 @@ describe('warm-set maintenance cost', () => { expect(evicted.every((operation) => operation.runner?.isActive === false)).toBe(true); expect(graph.resultByOperation.size).toBe(38); }); + + it('takes the native repository lease only for a pass that releases resources or stops observing a project', async () => { + const { graph, operations, requestAll } = createGraph(10); + const { watcher, watched } = createWatcher(); + const acquire: jest.Mock, []> = createLease(); + const warm: WorkspaceWarmSet = attach( + graph, + { warmSetMaxProjects: 3 }, + { watcher, acquireExecutionLeaseAsync: acquire } + ); + disposables.push(warm); + requestAll(); + await settleAsync(warm); + acquire.mockClear(); + + // Retained results with nothing to release: native Rush commands can take the repository lock meanwhile. + const idle: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).not.toHaveBeenCalled(); + expect(idle.deferredReason).toBeUndefined(); + expect(idle.retainedProjectNames).toHaveLength(10); + + // With daemon.watch off, a remaining project watcher is closed under the lease. + watched.add('p3'); + await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(1); + expect(watched.size).toBe(0); + await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(1); + + // Resource holders over the project cap are released under the lease. + for (const operation of operations.slice(0, 5)) operation.runner = createResidentRunner(); + const evicted: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(2); + expect(evicted.overProjectLimit).toBe(false); + expect(operations.filter((operation) => operation.runner?.isActive)).toHaveLength(3); + await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(2); + }); + + it('takes the native repository lease to observe a requested project again and to clear an observation failure', async () => { + const { graph, requestAll } = createGraph(4); + const { watcher, watched, failNextWatch } = createWatcher(); + const acquire: jest.Mock, []> = createLease(); + const diagnostics: string[] = []; + const warm: WorkspaceWarmSet = attach( + graph, + { watch: true }, + { + watcher, + acquireExecutionLeaseAsync: acquire, + onDiagnostic: (error: Error) => diagnostics.push(error.message) + } + ); + disposables.push(warm); + requestAll(); + await settleAsync(warm); + expect(watched.size).toBe(4); + acquire.mockClear(); + await warm.maintainAsync(); + expect(acquire).not.toHaveBeenCalled(); + + watched.delete('p1'); + failNextWatch(); + const failed: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(1); + expect(failed.cleanupFailures).toEqual([expect.stringContaining('watch failed')]); + expect(watched.has('p1')).toBe(false); + + const recovered: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(2); + expect(recovered.cleanupFailures).toEqual([]); + expect(watched.has('p1')).toBe(true); + await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(2); + + // A request observes its projects again, but the reported failure is cleared only by a maintenance pass. + watched.delete('p2'); + failNextWatch(); + await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(3); + requestAll(); + expect(watched.size).toBe(4); + const cleared: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(4); + expect(cleared.cleanupFailures).toEqual([]); + await warm.maintainAsync(); + expect(acquire).toHaveBeenCalledTimes(4); + expect(diagnostics).toEqual([ + expect.stringContaining('watch failed'), + expect.stringContaining('watch failed') + ]); + }); + + it('takes no native repository lease for a project that daemon.watch leaves unobserved after its eviction failed', async () => { + const { graph, requestAll } = createGraph(4); + const { watcher, watched } = createWatcher(); + const acquire: jest.Mock, []> = createLease(); + const diagnostics: string[] = []; + // The eviction stops observing the project, then fails to drop its results. + graph.deleteResults = () => { + throw new Error('delete failed'); + }; + const warm: WorkspaceWarmSet = attach( + graph, + { watch: true, warmSetMaxProjects: 3 }, + { + watcher, + acquireExecutionLeaseAsync: acquire, + onDiagnostic: (error: Error) => diagnostics.push(error.message) + } + ); + disposables.push(warm); + requestAll(); + await settleAsync(warm); + expect(watched.size).toBe(3); + expect(diagnostics).toEqual([expect.stringContaining('Could not evict warm project')]); + acquire.mockClear(); + + // daemon.watch doesn't observe a project whose eviction failed again, so a pass has nothing to apply + const status: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).not.toHaveBeenCalled(); + expect(status.cleanupFailures).toEqual([expect.stringContaining('delete failed')]); + expect(status.retainedProjectNames).toHaveLength(4); + expect(watched.size).toBe(3); + expect(diagnostics).toHaveLength(1); + }); + + it('takes no native repository lease for a protected project that stays observed with daemon.watch off', async () => { + const { graph, operations, requestAll } = createGraph(4); + const { watcher, watched } = createWatcher(); + const acquire: jest.Mock, []> = createLease(); + const warm: WorkspaceWarmSet = attach( + graph, + {}, + { + watcher, + acquireExecutionLeaseAsync: acquire, + getProtectedOperations: () => new Set([operations[0]]) + } + ); + disposables.push(warm); + requestAll(); + await settleAsync(warm); + watched.add('p0'); + acquire.mockClear(); + + // daemon.watch off stops observing only the projects that aren't protected + const status: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(acquire).not.toHaveBeenCalled(); + expect(status.protectedProjectNames).toEqual(['p0']); + expect(watched.has('p0')).toBe(true); + }); }); From 3821da01653e66dd053e4e56ee23ed99ebde5718 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 00:09:37 +0000 Subject: [PATCH 046/265] [node-core-library] A leftover LockFile whose pid now belongs to an older process is deleted Swarm integration step 31; original commit 02087b3f11 (merge of o07/lockfile-tzrule at d5e74b1813). Scope: task 121. Brings o07's task 121 (board 1815, re-tipped in board 1949; t05 CONFIRMED board 1923 on faa7675fa6). A rush#.lock whose pid has been reused by a live process that started before the file was written no longer blocks every command until that process exits. The lock file keeps its holder only if the recorded start time matches the live process's start time in some real time zone, and on Linux that start time is read from /proc instead of spawning ps. Four commits: 595f8f63c7 (the time zone rule), faa7675fa6 (/proc start time for another time zone or locale), 315e1eae81 (ch04's test for a process that started after the file) and d5e74b1813 (t05's note that a clock change can make a start time fail the rule). ch01 on tree 73696a280a (board 2566): build rc 0 with 0 warnings, rush-daemon 549/0; on b10e8c53e0 node-core-library 329/0 (318 + 11 new tests) and rush-client-core 129/0. The new tests fail 7 times on the old LockFile.ts. 21 LockFile.ts mutants, 16 killed; the 5 survivors are 4 test NITs and one equivalent. E2E with 12 live holders in other time zones and locales: all kept, with 0 ps calls after the first attempt (1.2-7 ms per attempt, against 410-520 ms before); the stale files (pid 1 at +7 min, pid 1 with a 2024 start time, a predating process at +3 min) are now deleted. Gate: ch01 GATE OK board 2566 (tree 73696a280a) Commits folded into this step (4): - 595f8f63c7 [node-core-library] Keep a mismatched LockFile only if its start time is in some time zone - faa7675fa6 [node-core-library] Read the start time from /proc for a LockFile with another time zone or locale - 315e1eae81 [node-core-library] Test a LockFile whose PID started after the file, with a start time in another time zone - d5e74b1813 [node-core-library] Note that a clock change can make a LockFile's start time fail the time zone check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../lockfile-proc-start-time_2026-09-28.json | 2 +- ...lockfile-start-time-locale_2026-09-28.json | 2 +- libraries/node-core-library/src/LockFile.ts | 93 +++++++-- .../src/test/LockFile.test.ts | 183 +++++++++++++++++- 4 files changed, 259 insertions(+), 21 deletions(-) diff --git a/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json b/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json index e06b6e046b..f8abc83d11 100644 --- a/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json +++ b/common/changes/@rushstack/node-core-library/lockfile-proc-start-time_2026-09-28.json @@ -2,7 +2,7 @@ "changes": [ { "packageName": "@rushstack/node-core-library", - "comment": "Make `LockFile` on Linux much faster when other processes hold or wait for the same lock. It now confirms that another process's lockfile is valid by reading /proc, instead of running a `ps` command for each lockfile on each attempt.", + "comment": "Make `LockFile` on Linux much faster when other processes hold or wait for the same lock. It now confirms that another process's lockfile is valid by reading /proc, instead of running a `ps` command for each lockfile on each attempt. This also works when the other process has a different time zone or locale.", "type": "patch" } ], diff --git a/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json b/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json index 0c02c7c84f..5901dd4255 100644 --- a/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json +++ b/common/changes/@rushstack/node-core-library/lockfile-start-time-locale_2026-09-28.json @@ -2,7 +2,7 @@ "changes": [ { "packageName": "@rushstack/node-core-library", - "comment": "Fix `LockFile` on Linux and macOS deleting the lockfile of a running process, and granting the lock again, when that process has a different time zone or locale. A lockfile whose start time doesn't match is now kept if its process started before the lockfile was created.", + "comment": "Fix `LockFile` on Linux and macOS deleting the lockfile of a running process, and granting the lock again, when that process has a different time zone or locale. A lockfile whose start time doesn't match is now kept if its process started before the lockfile was created, and if the two start times differ by a time zone offset or the lockfile's start time is in the format of another locale.", "type": "patch" } ], diff --git a/libraries/node-core-library/src/LockFile.ts b/libraries/node-core-library/src/LockFile.ts index ea39a31b97..881287ff34 100644 --- a/libraries/node-core-library/src/LockFile.ts +++ b/libraries/node-core-library/src/LockFile.ts @@ -179,11 +179,17 @@ export function getProcessStartTimeMs(pid: number): number | undefined { env: { ...process.env, LC_ALL: 'C', TZ: 'UTC0' } }); - // For example: "Sun Sep 27 17:15:08 2026" + return _parseLstartAsUtcMs((psResult.stdout || '').split('\n')[1] || ''); +} + +/** + * Parses a start time that "ps -o lstart" printed with the C locale, for example "Sun Sep 27 17:15:08 2026", + * as if it were in UTC. Returns the time in milliseconds since the epoch, or undefined if the text has another + * format, such as the format of another locale. + */ +function _parseLstartAsUtcMs(lstart: string): number | undefined { const match: RegExpExecArray | null = - /^\s*[A-Za-z]{3} ([A-Za-z]{3}) +(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) (\d{4})\s*$/.exec( - (psResult.stdout || '').split('\n')[1] || '' - ); + /^\s*[A-Za-z]{3} ([A-Za-z]{3}) +(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) (\d{4})\s*$/.exec(lstart); if (!match) { return undefined; } @@ -233,6 +239,11 @@ export interface ILinuxProcessStartTime { * "ps" can't be run. */ ticks: string; + /** + * The start time in milliseconds since the epoch, rounded down to a whole second like lstart. This is what + * getProcessStartTimeMs() returns, and it doesn't depend on the time zone. + */ + startTimeMs: number; } /** @@ -267,16 +278,16 @@ export function getLinuxProcessStartTime( } // Like "ps", round down to a whole second and use "%a %b %e %H:%M:%S %Y" in the local time zone. - const date: Date = new Date( - (getBootTimeSeconds() + Math.floor(Number(ticks) / LINUX_CLOCK_TICKS_PER_SECOND)) * 1000 - ); + const startTimeMs: number = + (getBootTimeSeconds() + Math.floor(Number(ticks) / LINUX_CLOCK_TICKS_PER_SECOND)) * 1000; + const date: Date = new Date(startTimeMs); const twoDigits: (value: number) => string = (value: number) => (value < 10 ? `0${value}` : `${value}`); const lstart: string = `${LSTART_DAYS[date.getDay()]} ${LSTART_MONTHS[date.getMonth()]} ` + `${date.getDate() < 10 ? ' ' : ''}${date.getDate()} ` + `${twoDigits(date.getHours())}:${twoDigits(date.getMinutes())}:${twoDigits(date.getSeconds())} ` + `${date.getFullYear()}`; - return { lstart, ticks }; + return { lstart, ticks, startTimeMs }; } // A set of locks that currently exist in the current process, to be used when @@ -589,6 +600,36 @@ function _tryAcquireInner( // seconds, and the system clock can be adjusted while a process runs. const START_TIME_TOLERANCE_MS: number = 5000; +// Every time zone is ahead of or behind UTC by a whole number of 15-minute steps, from UTC-12 to UTC+14. +const TIME_ZONE_OFFSET_STEP_MS: number = 15 * 60 * 1000; +const MIN_TIME_ZONE_OFFSET_MS: number = -12 * 60 * 60 * 1000; +const MAX_TIME_ZONE_OFFSET_MS: number = 14 * 60 * 60 * 1000; + +/** + * Returns false if the start time in a lockfile can't be what "ps -o lstart" printed for a process that started + * at `startTimeMs`, in any time zone. Returns true if it can, or if the start time is in a format other than the + * C locale's, which can't be checked. + * + * On Linux, the start time that "ps" reports for a process moves with the system clock. So if the clock is + * changed by more than START_TIME_TOLERANCE_MS while a process holds a lock, this can return false for the + * lockfile of that process, which is then treated as stale. + */ +function _isStartTimeInSomeTimeZone(lockFileStartTime: string, startTimeMs: number): boolean { + const lockFileStartTimeMs: number | undefined = _parseLstartAsUtcMs(lockFileStartTime); + if (lockFileStartTimeMs === undefined) { + return true; + } + const offsetMs: number = lockFileStartTimeMs - startTimeMs; + if ( + offsetMs < MIN_TIME_ZONE_OFFSET_MS - START_TIME_TOLERANCE_MS || + offsetMs > MAX_TIME_ZONE_OFFSET_MS + START_TIME_TOLERANCE_MS + ) { + return false; + } + const stepOffsetMs: number = Math.round(offsetMs / TIME_ZONE_OFFSET_STEP_MS) * TIME_ZONE_OFFSET_STEP_MS; + return Math.abs(offsetMs - stepOffsetMs) <= START_TIME_TOLERANCE_MS; +} + /** * Called when the start time in the lockfile of another running process differs from the start time that * we got for its PID. Returns true if the lockfile still belongs to that process. @@ -601,13 +642,19 @@ function _isLockFileOfRunningProcess( // "ps -o lstart" formats the start time using the time zone and locale of the process that runs it, // so a process whose TZ, LANG, LC_TIME or LC_ALL differs from ours wrote its start time differently. // If the start time differs because the lockfile's process exited and the OS gave its PID to a new - // process, then the new process started after the lockfile was created. An empty lockfile is still - // treated as stale here, as before. + // process, then the new process usually started after the lockfile was created. It can have started + // before, if the lockfile was copied or restored, or if a process in another PID namespace (such as a + // container) wrote it. So the lockfile must also hold the process's start time in some time zone. + // An empty lockfile is still treated as stale here, as before. if (!lockFileStartTime || lockFileBirthtimeMs === undefined) { return false; } const startTimeMs: number | undefined = getProcessStartTimeMs(parseInt(pid, 10)); - return startTimeMs !== undefined && startTimeMs <= lockFileBirthtimeMs + START_TIME_TOLERANCE_MS; + return ( + startTimeMs !== undefined && + startTimeMs <= lockFileBirthtimeMs + START_TIME_TOLERANCE_MS && + _isStartTimeInSomeTimeZone(lockFileStartTime, startTimeMs) + ); } /** @@ -619,6 +666,7 @@ function _isLockFileOfRunningProcess( function _isLockFileOfLinuxProcess( pid: string, lockFileStartTime: string | undefined, + lockFileBirthtimeMs: number | undefined, getBootTimeSeconds: () => number ): boolean { if (process.platform !== 'linux' || !lockFileStartTime) { @@ -631,11 +679,19 @@ function _isLockFileOfLinuxProcess( // For example, /proc isn't mounted, or it doesn't let us read the files of this process. return false; } - // These are the formats that getProcessStartTime() returns. If the other process wrote its start time with a - // locale other than C, the caller runs "ps". + if (startTime === undefined) { + return false; + } + // These are the formats that getProcessStartTime() returns. + if (lockFileStartTime === startTime.lstart || lockFileStartTime === startTime.ticks) { + return true; + } + // The other process may have written its start time with another time zone or locale. This is the check + // that _isLockFileOfRunningProcess() makes after running "ps" twice, with the start time from /proc. return ( - startTime !== undefined && - (lockFileStartTime === startTime.lstart || lockFileStartTime === startTime.ticks) + lockFileBirthtimeMs !== undefined && + startTime.startTimeMs <= lockFileBirthtimeMs + START_TIME_TOLERANCE_MS && + _isStartTimeInSomeTimeZone(lockFileStartTime, startTime.startTimeMs) ); } @@ -799,12 +855,13 @@ function _tryAcquireMacOrLinuxOnce( // console.log(`Other pid ${otherPid} lockfile has start time: "${otherPidOldStartTime}"`); // Actual start time of the other PID. On Linux, /proc usually shows that the file belongs to the - // process with that PID, and then we don't need to run "ps", which is slow when there are many - // processes. When many processes wait for the same lock, each of their attempts checks the file - // of every other process. + // process with that PID, even if that process has another time zone or locale, and then we don't need + // to run "ps", which is slow when there are many processes. When many processes wait for the same + // lock, each of their attempts checks the file of every other process. const otherPidCurrentStartTime: string | undefined = _isLockFileOfLinuxProcess( otherPid, otherPidOldStartTime, + otherBirthtimeMs, getLinuxBootTime ) ? otherPidOldStartTime diff --git a/libraries/node-core-library/src/test/LockFile.test.ts b/libraries/node-core-library/src/test/LockFile.test.ts index f9ea13aee2..bec5b86e03 100644 --- a/libraries/node-core-library/src/test/LockFile.test.ts +++ b/libraries/node-core-library/src/test/LockFile.test.ts @@ -687,6 +687,136 @@ describe(LockFile.name, () => { expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); }); + // Formats a time as "ps -o lstart" prints it with the C locale and TZ=UTC0, for example + // "Sun Sep 27 17:15:08 2026". + function formatLstartInUtc(timeMs: number): string { + const date: Date = new Date(timeMs); + const days: string[] = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']; + const months: string[] = [ + 'Jan', + 'Feb', + 'Mar', + 'Apr', + 'May', + 'Jun', + 'Jul', + 'Aug', + 'Sep', + 'Oct', + 'Nov', + 'Dec' + ]; + const twoDigits: (value: number) => string = (value: number) => `${value}`.padStart(2, '0'); + return ( + `${days[date.getUTCDay()]} ${months[date.getUTCMonth()]} ${`${date.getUTCDate()}`.padStart(2, ' ')} ` + + `${twoDigits(date.getUTCHours())}:${twoDigits(date.getUTCMinutes())}:${twoDigits(date.getUTCSeconds())} ` + + `${date.getUTCFullYear()}` + ); + } + + // What the other process wrote if its time zone is offsetMs ahead of UTC + function getOtherPidStartTimeWithOffset(offsetMs: number): string { + const otherPidStartTimeMs: number | undefined = getProcessStartTimeMs(otherPid); + expect(otherPidStartTimeMs).toBeDefined(); + return formatLstartInUtc(otherPidStartTimeMs! + offsetMs); + } + + test.each<[string, string, number]>([ + ['5 hours and 45 minutes ahead of UTC', '20', (5 * 60 + 45) * 60 * 1000], + ['12 hours behind UTC', '21', -12 * 60 * 60 * 1000], + ['7 hours behind UTC, printed 2 seconds off', '22', -7 * 60 * 60 * 1000 + 2000] + ])( + 'cannot acquire a lock if the other process wrote its start time in a time zone %s', + (description: string, folderName: string, offsetMs: number) => { + const testFolder: string = path.join(libTestFolder, folderName); + const otherPidLockFileName: string = createOtherLockFile( + testFolder, + getOtherPidStartTimeWithOffset(offsetMs) + ); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeUndefined(); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + } + ); + + test.each<[string, string, number]>([ + ['7 minutes after', '23', 7 * 60 * 1000], + ['25 hours after', '24', 25 * 60 * 60 * 1000], + ['13 hours before', '25', -13 * 60 * 60 * 1000] + ])( + 'deletes the lockfile of a process that started before the lockfile if its start time is %s the start time of the process', + (description: string, folderName: string, offsetMs: number) => { + const testFolder: string = path.join(libTestFolder, folderName); + // No time zone is this far from UTC, so the lockfile was written by another process that had the same + // PID, for example in a container or before the lockfile was copied. + const otherPidLockFileName: string = createOtherLockFile( + testFolder, + getOtherPidStartTimeWithOffset(offsetMs) + ); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(true); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(false); + lock!.release(); + } + ); + + test('deletes the lockfile of a process that started before the lockfile if its start time is from another year', () => { + const testFolder: string = path.join(libTestFolder, '26'); + const otherPidLockFileName: string = createOtherLockFile(testFolder, 'Mon Jan 1 00:00:00 2024'); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(true); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(false); + lock!.release(); + }); + + test('cannot acquire a lock if the other process wrote its start time in the format of another locale', () => { + const testFolder: string = path.join(libTestFolder, '27'); + // Only the C locale's format can be compared with the start time of the process. + const otherPidLockFileName: string = createOtherLockFile(testFolder, 'Mo 28 Sep 2026 18:15:31'); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeUndefined(); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(true); + }); + + test('deletes the lockfile of a process that started after the lockfile, even if its start time is in another time zone', () => { + const testFolder: string = path.join(libTestFolder, '30'); + // UTC+14, or UTC-12 if that is the local time zone + const otherTimeZone: string = new Date().getTimezoneOffset() === -840 ? 'XYZ+12' : 'XYZ-14'; + const otherPidStartTime: string = child_process + .spawnSync('ps', ['-p', `${otherPid}`, '-o', 'lstart'], { + encoding: 'utf8', + env: { ...process.env, LC_ALL: 'C', TZ: otherTimeZone } + }) + .stdout.split('\n')[1] + .trim(); + const otherPidLockFileName: string = createOtherLockFile(testFolder, otherPidStartTime); + // The lockfile was created 1 minute before the process with its PID started. + const otherBirthtimeMs: number = getProcessStartTimeMs(otherPid)! - 60000; + const originalGetStatistics: typeof FileSystem.getStatistics = FileSystem.getStatistics; + jest.spyOn(FileSystem, 'getStatistics').mockImplementation((filePath: string) => { + return path.resolve(filePath) === otherPidLockFileName + ? ({ birthtime: new Date(otherBirthtimeMs) } as FileSystemStats) + : originalGetStatistics(filePath); + }); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(true); + expect(FileSystem.exists(otherPidLockFileName)).toEqual(false); + lock!.release(); + }); + test('deletes the lockfile if its PID now belongs to a process that started after the lockfile was created', () => { const testFolder: string = path.join(libTestFolder, '12'); const otherPidLockFileName: string = createOtherLockFile(testFolder, 'Thu Jan 1 00:00:10 1970'); @@ -835,7 +965,8 @@ describe(LockFile.name, () => { expect(getLinuxProcessStartTime(12345, () => 1000000000)).toEqual({ lstart: expectedLstart, - ticks: '250' + ticks: '250', + startTimeMs: 1000000002000 }); expect(expectedLstart).toMatch(/^[A-Z][a-z]{2} Sep {2}[89] \d\d:\d\d:\d\d 2001$/); }); @@ -966,6 +1097,56 @@ describe(LockFile.name, () => { expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [otherPids[0]]]); }); + + test('does not run "ps" for them if they wrote their start times with other time zones or locales', () => { + const testFolder: string = path.join(libTestFolder, '28'); + const otherPids: number[] = startProcesses(3); + const otherPidLockFileNames: string[] = createNewerLockFiles( + testFolder, + otherPids, + (otherPid: number) => { + switch (otherPids.indexOf(otherPid)) { + case 0: + return getLstartWithCLocale(otherPid, 'XYZ-05:45'); + case 1: + return getLstartWithCLocale(otherPid, 'XYZ+12'); + default: + // The format of another locale + return '28.09.2026 18:15:31'; + } + } + ); + const nativeChildProcess: typeof child_process = jest.requireActual('node:child_process'); + const spawnSyncSpy: jest.SpyInstance = jest.spyOn(nativeChildProcess, 'spawnSync'); + + expectToAcquireAndKeep(testFolder, otherPidLockFileNames); + + // "ps" ran only for the start time of this process. + expect(spawnSyncSpy).toHaveBeenCalledTimes(1); + expect(spawnSyncSpy.mock.calls[0][1]).toEqual(['-p', `${process.pid}`, '-o', 'lstart']); + }); + + test('runs "ps" for a lockfile whose start time is not the start time of its process in any time zone', () => { + const testFolder: string = path.join(libTestFolder, '29'); + const otherPids: number[] = startProcesses(1); + // Another process that had the same PID wrote this lockfile. It started 7 minutes after this process. + const otherPidLockFileNames: string[] = createNewerLockFiles( + testFolder, + otherPids, + (otherPid: number) => getLstartWithCLocale(otherPid, 'XYZ-00:07') + ); + const getStartTimeSpy: jest.Mock = jest.fn(getProcessStartTime); + setLockFileGetProcessStartTime(getStartTimeSpy); + + const lock: LockFile | undefined = LockFile.tryAcquire(testFolder, resourceName); + + // /proc never makes a lockfile stale, so "ps" decided it. + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [otherPids[0]]]); + expect(lock).toBeDefined(); + expect(lock!.dirtyWhenAcquired).toEqual(true); + expect(FileSystem.exists(otherPidLockFileNames[0])).toEqual(false); + lock!.release(); + }); }); describe('when this process acquires locks again', () => { From 1d1a2176b6889fe36478a32517fcbfa9793c324b Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 00:09:42 +0000 Subject: [PATCH 047/265] [credential-cache] A credentials.json that isn't valid JSON no longer prints the stored SAS Swarm integration step 32; original commit 0bef326909 (merge of swarm/r03-t143 at 92cbe2336e). Scope: task 143. Brings r03's task 143 (board 2316; t07 CONFIRMED board 2443 and board 2445). CredentialCache no longer rethrows JsonFile's parse error, which quoted jju's excerpt of the failing line, and in credentials.json that line holds the credential. The error now gives only the line and column and says the contents aren't shown because the file stores credentials. A missing file still gives undefined, other read errors are still wrapped as before, and schema errors are unchanged. The branch carries 08e26894e7 (task 101 part 1: compile the credentials file schema once) through r03's merge eae2fc9ad3, so tasks 101 and 143 merge in either order. ch01 on 2c425fc4f5 (board 2576): build rc 0 with 0 warnings, credential-cache 34/0, azure plugin 10/0, http plugin 14/0, amazon-s3 plugin 52/0. The new tests fail exactly the three torn-file tests on eae2fc9ad3. 12 CredentialCache.ts mutants, 9 killed; the survivors are one equivalent and one test NIT (a read error other than ENOENT). E2E with a marker in the SAS: on the old engine a torn file printed the marker in the daemon, --no-daemon and update-cloud-credentials paths; now it is printed 0 times, and 11 of 20 outputs are byte-identical. Task 142 must land with or after this merge. Gate: ch01 GATE OK board 2576 (tree 692db7d548 after task 121) Commits folded into this step (2): - 08e26894e7 [credential-cache] Compile the credentials file schema once per process - 92cbe2336e [credential-cache] Don't quote the credentials file in its parse error (task 143) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...e-redact-parse-error_2026-09-28-22-44.json | 11 +++ ...al-cache-schema-once_2026-09-28-21-50.json | 11 +++ .../credential-cache/src/CredentialCache.ts | 47 ++++++++-- .../src/test/CredentialCache.test.ts | 91 ++++++++++++++++++- 4 files changed, 149 insertions(+), 11 deletions(-) create mode 100644 common/changes/@rushstack/credential-cache/credential-cache-redact-parse-error_2026-09-28-22-44.json create mode 100644 common/changes/@rushstack/credential-cache/credential-cache-schema-once_2026-09-28-21-50.json diff --git a/common/changes/@rushstack/credential-cache/credential-cache-redact-parse-error_2026-09-28-22-44.json b/common/changes/@rushstack/credential-cache/credential-cache-redact-parse-error_2026-09-28-22-44.json new file mode 100644 index 0000000000..8a9a25f96c --- /dev/null +++ b/common/changes/@rushstack/credential-cache/credential-cache-redact-parse-error_2026-09-28-22-44.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/credential-cache", + "comment": "When the credentials file is not valid JSON, report the line and column without quoting the file, because the quoted text can include a credential.", + "type": "patch" + } + ], + "packageName": "@rushstack/credential-cache", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/credential-cache/credential-cache-schema-once_2026-09-28-21-50.json b/common/changes/@rushstack/credential-cache/credential-cache-schema-once_2026-09-28-21-50.json new file mode 100644 index 0000000000..71788df859 --- /dev/null +++ b/common/changes/@rushstack/credential-cache/credential-cache-schema-once_2026-09-28-21-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/credential-cache", + "comment": "Compile the JSON schema of the credentials file once per process, instead of each time the file is loaded.", + "type": "patch" + } + ], + "packageName": "@rushstack/credential-cache", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/credential-cache/src/CredentialCache.ts b/libraries/credential-cache/src/CredentialCache.ts index c7535b8187..7dc23b781d 100644 --- a/libraries/credential-cache/src/CredentialCache.ts +++ b/libraries/credential-cache/src/CredentialCache.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as os from 'node:os'; import * as path from 'node:path'; import { FileSystem, JsonFile, JsonSchema, LockFile, User, Objects } from '@rushstack/node-core-library'; @@ -15,6 +16,8 @@ export const RUSH_USER_FOLDER_NAME: '.rush-user' = '.rush-user'; const DEFAULT_CACHE_FILENAME: 'credentials.json' = 'credentials.json'; const LATEST_CREDENTIALS_JSON_VERSION: string = '0.1.0'; +// Shared so that the schema is compiled once, rather than on every load (about 5 ms each) +const CREDENTIALS_JSON_SCHEMA: JsonSchema = JsonSchema.fromLoadedObject(schemaJson); interface ICredentialCacheJson { version: string; @@ -29,6 +32,39 @@ interface ICacheEntryJson { credentialMetadata?: object; } +/** + * Loads and validates the credentials file. Returns undefined if the file doesn't exist. + */ +async function loadCredentialsJsonAsync(cacheFilePath: string): Promise { + let contents: string; + try { + contents = await FileSystem.readFileAsync(cacheFilePath); + } catch (e) { + if (FileSystem.isNotExistError(e as Error)) { + return undefined; + } + + throw new Error(`Error reading "${cacheFilePath}":${os.EOL} ${(e as Error).message}`); + } + + let credentialsJson: ICredentialCacheJson; + try { + credentialsJson = JsonFile.parseString(contents); + } catch (e) { + // Don't use the parser's message: it quotes the text around the problem, which can be a credential + const { row, column } = e as { row?: unknown; column?: unknown }; + const position: string = + typeof row === 'number' && typeof column === 'number' ? ` (line ${row}, column ${column})` : ''; + throw new Error( + `Error reading "${cacheFilePath}": the file is not valid JSON${position}. ` + + 'Its contents are not shown because it stores credentials. Correct the file or delete it.' + ); + } + + CREDENTIALS_JSON_SCHEMA.validateObject(credentialsJson, cacheFilePath); + return credentialsJson; +} + /** * @public */ @@ -84,16 +120,7 @@ export class CredentialCache implements Disposable { } const cacheFilePath: string = `${cacheDirectory}/${cacheFileName}`; - const jsonSchema: JsonSchema = JsonSchema.fromLoadedObject(schemaJson); - - let loadedJson: ICredentialCacheJson | undefined; - try { - loadedJson = await JsonFile.loadAndValidateAsync(cacheFilePath, jsonSchema); - } catch (e) { - if (!FileSystem.isErrnoException(e as Error)) { - throw e; - } - } + const loadedJson: ICredentialCacheJson | undefined = await loadCredentialsJsonAsync(cacheFilePath); let lockfile: LockFile | undefined; if (options.supportEditing) { diff --git a/libraries/credential-cache/src/test/CredentialCache.test.ts b/libraries/credential-cache/src/test/CredentialCache.test.ts index 5bdb13c49e..f8bbc6a764 100644 --- a/libraries/credential-cache/src/test/CredentialCache.test.ts +++ b/libraries/credential-cache/src/test/CredentialCache.test.ts @@ -2,11 +2,12 @@ // See LICENSE in the project root for license information. import { mockGetHomeFolder } from './CredentialCache.mock'; -import { LockFile, Async, FileSystem } from '@rushstack/node-core-library'; +import { LockFile, Async, FileSystem, JsonFile, JsonSchema } from '@rushstack/node-core-library'; import { CredentialCache, type ICredentialCacheOptions, RUSH_USER_FOLDER_NAME } from '../CredentialCache'; const FAKE_HOME_FOLDER: string = 'temp'; const FAKE_RUSH_USER_FOLDER: string = `${FAKE_HOME_FOLDER}/${RUSH_USER_FOLDER_NAME}`; +const FAKE_SAS: string = 'sv=2025-01-05&sp=rwl&sig=FAKESIGNATURE0123'; interface IPathsTestCase extends Required> { testCaseName: string; @@ -392,4 +393,92 @@ describe(CredentialCache.name, () => { `"This instance of CredentialCache does not support editing."` ); }); + + it('validates every load with the same schema object, so the schema is compiled once', async () => { + fakeFilesystem[`${FAKE_RUSH_USER_FOLDER}/credentials.json`] = JSON.stringify({ + version: '0.1.0', + cacheEntries: {} + }); + const validateObjectSpy: jest.SpyInstance = jest.spyOn(JsonSchema.prototype, 'validateObject'); + + for (let i: number = 0; i < 2; i++) { + const credentialCache: CredentialCache = await CredentialCache.initializeAsync({ + supportEditing: false + }); + credentialCache.dispose(); + } + + expect(validateObjectSpy).toHaveBeenCalledTimes(2); + expect(validateObjectSpy.mock.instances[1]).toBe(validateObjectSpy.mock.instances[0]); + validateObjectSpy.mockRestore(); + }); + + describe('a credentials file that is not valid JSON', () => { + const cacheFilePath: string = `${FAKE_RUSH_USER_FOLDER}/credentials.json`; + const validJson: string = JSON.stringify( + { + version: '0.1.0', + cacheEntries: { + 'test-credential': { + expires: 0, + credential: FAKE_SAS + } + } + }, + undefined, + 2 + ); + const endOfSas: number = validJson.indexOf(FAKE_SAS) + FAKE_SAS.length; + + it.each([ + { + name: 'cut off after the credential', + contents: validJson.slice(0, endOfSas + 1) + }, + { + name: 'cut off inside the credential', + contents: validJson.slice(0, endOfSas - 4) + }, + { + name: 'with an unexpected character after the credential', + contents: `${validJson.slice(0, endOfSas + 1)} x${validJson.slice(endOfSas + 1)}` + } + ])('reports the position without the contents when it is $name', async ({ contents }) => { + fakeFilesystem[cacheFilePath] = contents; + let parserError: (Error & { row?: number; column?: number }) | undefined; + try { + JsonFile.parseString(contents); + } catch (e) { + parserError = e as Error; + } + // The parser's own message quotes the credential + expect(parserError?.message).toContain('sig=FAKESIGNATURE'); + expect(parserError?.row).toEqual(6); + + const expectedError: Error = new Error( + `Error reading "${cacheFilePath}": the file is not valid JSON ` + + `(line ${parserError?.row}, column ${parserError?.column}). ` + + 'Its contents are not shown because it stores credentials. Correct the file or delete it.' + ); + for (const supportEditing of [false, true]) { + await expect(CredentialCache.initializeAsync({ supportEditing })).rejects.toThrow(expectedError); + } + }); + }); + + it('reports a credentials file that does not match its schema without the values', async () => { + fakeFilesystem[`${FAKE_RUSH_USER_FOLDER}/credentials.json`] = JSON.stringify({ + version: '0.1.0', + cacheEntries: { + 'test-credential': { + expires: 'never', + credential: FAKE_SAS + } + } + }); + + await expect(CredentialCache.initializeAsync({ supportEditing: false })).rejects.toThrow( + /^JSON validation failed:\s+temp\/\.rush-user\/credentials\.json\s+Error: #\/cacheEntries\/test-credential\/expires\s+must be number$/ + ); + }); }); From 1840241beb407f30fbfea663212e2988989b67e8 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 00:41:18 +0000 Subject: [PATCH 048/265] [rush-daemon] A failed agent-output build returns as soon as nothing unfinished can change its result MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swarm integration step 33; original commit 1b51e7af08 (merge of swarm/r04-t108-int at c76de10302). Scope: task 108 a. Brings r04's task 108 item 2, "108(a)": 129e04f8bb on the nits 4bad455570 and item 2 2e066a7758, re-merged onto 3857bc9926 as c76de10302 (board 2133, board 2533). With agent output, a failed `build` returns as soon as nothing unfinished can change its result. Its summary line ends `· independent operations continue in rushd`, and those operations finish in the daemon. The next build waits for them (`queued behind another request`) and doesn't run them again; the daemon's own install and rebuild stop the leftover, and a request that falls back in-process stops it first. Until the leftover ends, a native `rush install` or `rush-client --no-daemon` fails with "Another Rush command is already running in this repository." (o01 board 2598, board 2612), and `daemon status` doesn't count it yet (r04's output follow-up, board 2133, board 2652). Legacy output keeps the old behavior. o02 CONFIRMED item 2 with one row refuted (board 2055), which 129e04f8bb addresses; CONFIRMED by t07 (board 2183) and t05 (board 2193). ch01 on 140b42ad50 (board 2678): build rc 0 with 0 warnings, rush-daemon 573/0, rush-cli-client 383/0, rush-client-core 129/0, rush-daemon-protocol 172/0. 31 new and changed tests fail on the old product. 27 mutants, 22 killed; F4 and Y4 are equivalent, and three are test NITs (the stop ignores the client leaving, the offer ignores a record that is already aborted, and silent operations count as continuing). E2E: a failed build returns in 0.3 s where the base takes 15.3 s, and its independent operation finishes 15.0 s later. Gate: ch01 GATE OK board 2678 (tree 140b42ad50) Commits folded into this step (3): - 2e066a7758 Return a failed build's result early in agent output (task 108 item 2) - 4bad455570 Name a failure without output sooner, and pluralize failures, in agent output (task 108 nits) - 129e04f8bb Stop continuing work before an in-process fallback (task 108, coord board 1981 a) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 19 +- .../src/AgentProgressRenderer.ts | 49 ++- .../src/OperationOutputExcerpt.ts | 7 +- apps/rush-cli-client/src/launchClient.ts | 2 + .../src/test/AgentProgressRenderer.test.ts | 158 ++++++- .../src/test/OperationOutputExcerpt.test.ts | 19 + .../src/test/returnEarlyOnFailure.test.ts | 99 +++++ .../swarm-r04-t108-nits_2026-09-28-21-10.json | 11 + ...04-t108-return-early_2026-09-28-20-25.json | 11 + ...04-t108-return-early_2026-09-28-20-25.json | 11 + ...rm-r04-t108-fallback_2026-09-28-21-40.json | 11 + ...04-t108-return-early_2026-09-28-20-25.json | 11 + .../reviews/api/rush-daemon-protocol.api.md | 2 + common/reviews/api/rush-daemon.api.md | 2 + .../src/DaemonPhasedRequest.ts | 2 + .../src/DaemonRequestEnvelope.ts | 8 + .../src/RequestControlValidation.ts | 5 +- .../src/RequestEnvelopeValidation.ts | 13 +- .../test/RequestReturnEarlyOnFailure.test.ts | 37 ++ .../rush-daemon/src/DaemonControlSession.ts | 16 +- .../rush-daemon/src/PhasedRequestEventSink.ts | 92 +++- .../rush-daemon/src/PhasedRequestRouter.ts | 202 ++++++++- .../src/ProductionDaemonRequestResolver.ts | 1 + libraries/rush-daemon/src/RequestScheduler.ts | 82 +++- .../src/WorkspaceRequestLifecycle.ts | 101 ++++- .../DaemonRequestWireEarlyFailure.test.ts | 166 +++++++ .../test/PhasedRequestEarlyFailure.test.ts | 415 ++++++++++++++++++ .../src/test/RequestScheduler.test.ts | 145 ++++++ .../test/WorkspaceEarlyFailureResult.test.ts | 253 +++++++++++ 29 files changed, 1890 insertions(+), 60 deletions(-) create mode 100644 apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t108-nits_2026-09-28-21-10.json create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t108-return-early_2026-09-28-20-25.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/swarm-r04-t108-return-early_2026-09-28-20-25.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t108-fallback_2026-09-28-21-40.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t108-return-early_2026-09-28-20-25.json create mode 100644 libraries/rush-daemon-protocol/src/test/RequestReturnEarlyOnFailure.test.ts create mode 100644 libraries/rush-daemon/src/test/DaemonRequestWireEarlyFailure.test.ts create mode 100644 libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 2aef793974..ccb39a9801 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -156,9 +156,22 @@ error count that the shown errors account for, and, when the first error shown n location, the lines before it. Up to three operations are reported. Two kinds are reported just before the summary line instead: a failed operation that wrote no output, with the error from the daemon's result, and, when no operation failed, the operations whose warnings failed the request -(`warnings: …`). The summary line names up to five failed (or warning) operations. Every -operation's full output is in its project's `rush-logs/` folder, whether or not it was printed. -When a request falls back to in-process Rush, agent mode stops and native output follows. +(`warnings: …`). On a pipe, a status line names a failed operation that wrote no output 1 s after +it failed, unless the result came first. The summary line names up to five failed (or warning) +operations. Every operation's full output is in its project's `rush-logs/` folder, whether or not +it was printed. When a request falls back to in-process Rush, agent mode stops and native output +follows. + +In agent mode a failed `rush build` doesn't wait for all of its work. Its result comes once an +operation failed and none of the selected projects that no other selected project depends on (for +example, the projects named by `--to`) is still waiting or running. The daemon keeps running the +operations that the failure didn't block, so that the next build finds them done, and the summary +line counts them (`· 2 independent operations continue in rushd`). A later `rush build` waits for +them. `rush rebuild`, `rush install` and `rush update`, a restart of the daemon for another +environment, and `rush-client daemon stop` stop them instead. Rushx scripts, and commands that the +daemon doesn't run (such as `rush list` or a custom command, which run in-process), run alongside +them, like two Rush commands at once in one checkout. +Older daemons report the failure when all of the work has ended. Positively identified built-in `install` and `update` follow the same opt-in routing precedence as workspace builds and require protocol **0.10** diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index 2f3dc95820..5313181609 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -26,6 +26,11 @@ const PIPE_CONNECTING_LINE_DELAY_MS: number = 10_000; * request from a hung one. Agent shells return partial output after 30 s. A request that ends sooner writes none. */ const PIPE_STATUS_INTERVAL_MS: number = 25_000; +/** + * On a pipe, a failed operation that wrote no output is reported only with the daemon's result, which carries its + * error. Unless that result comes first, the next status line, which names it, is written this long after it failed. + */ +const PIPE_UNREPORTED_FAILURE_DELAY_MS: number = 1_000; const SENT_PHASE: string = 'sent to rushd; preparing the workspace graph'; const STARTING_PHASE: string = 'rushd is still starting; waiting for it'; const FAILURE_STATUS: string = 'FAILURE'; @@ -54,9 +59,17 @@ const STATUS_LABELS: ReadonlyMap = new Map([ ['SKIPPED', 'up to date'], ['NO OP', 'up to date'] ]); +/** The plural of a summary label, where it differs. */ +const PLURAL_LABELS: ReadonlyMap = new Map([['failure', 'failures']]); type Verdict = 'SUCCESS' | 'FAILURE' | 'CANCELLED'; +/** + * Statuses that a failed result reports for operations that are still unfinished: a daemon that returns a failure + * early lets the operations that the failure did not block run on. + */ +const UNFINISHED_STATUSES: ReadonlySet = new Set(['WAITING', 'READY', 'QUEUED', 'EXECUTING']); + /** A queue position that the daemon reported, and when. */ interface IQueuePosition { readonly position: number; @@ -83,6 +96,15 @@ export interface IAgentFinalResult { readonly admissionErrorCode?: DaemonRequestAdmissionErrorCode; } +/** Says how many operations the daemon still runs after it reported the failure, if any. */ +function formatUnfinishedOperations(results: ReadonlyArray | undefined): string { + const count: number = results?.filter(({ status }) => UNFINISHED_STATUSES.has(status)).length ?? 0; + if (count === 0) { + return ''; + } + return ` · ${count} independent ${count === 1 ? 'operation continues' : 'operations continue'} in rushd`; +} + function formatNames(names: ReadonlyArray, maxNames: number): string { const shown: string = names.slice(0, maxNames).join(', '); return names.length > maxNames ? `${shown} +${names.length - maxNames} more` : shown; @@ -112,7 +134,8 @@ function getErrorDetail(lines: ReadonlyArray): string[] { * takes longer than 10 s gets one. A wait for a daemon that is still starting also gets a line, once. Only the * first three failed operations are reported. Whether warnings fail the request is only known at its end, so * operations with warnings are reported before the summary line; so is a failed operation that wrote no output, - * whose error only the daemon's result carries. + * whose error only the daemon's result carries. On a pipe, the next status line, which names that operation, is + * then due 1 s after it failed, so that a result that the daemon returns early can come first and make it moot. */ export class AgentProgressRenderer { readonly #options: IAgentProgressRendererOptions; @@ -136,6 +159,8 @@ export class AgentProgressRenderer { /** The first queue position, for the summary line. */ #firstQueued: IQueuePosition | undefined; #stopped: boolean = false; + /** On a pipe: an operation that wrote no output failed, and no line has named it yet. */ + #unnamedFailure: boolean = false; /** The error message that the summary line contains in full, once written. */ #reportedErrorMessage: string | undefined; @@ -285,6 +310,9 @@ export class AgentProgressRenderer { const emptySelection: boolean = verdict === 'SUCCESS' && result?.operationResults?.length === 0 && !this.#tracker.hasOperations; let summary: string = this.#getSummaryLine(verdict, emptySelection); + if (verdict === 'FAILURE') { + summary += formatUnfinishedOperations(result?.operationResults); + } // An admission failure says that the request waited, and why it stopped waiting. if (this.#firstQueued && !result?.admissionErrorCode) { const { position, elapsed } = this.#firstQueued; @@ -335,7 +363,8 @@ export class AgentProgressRenderer { /** * Writes a failed operation's log file and output excerpt as soon as it fails, while the rest of the request * runs on. The operation's output all arrived before its status. An operation that wrote nothing is left to - * the failure report, which has the error from the daemon's result. + * the failure report, which has the error from the daemon's result; on a pipe, the next status line names it + * sooner. */ #reportFailure(operationId: string): void { if (this.#stopped || this.#reported.has(operationId) || this.#reported.size >= MAX_REPORTED_OPERATIONS) { @@ -343,6 +372,10 @@ export class AgentProgressRenderer { } const problem: IAgentProblemOperation = this.#tracker.getProblemOperation(operationId); if (!problem.excerpt?.lineCount) { + if (this.#statusTimer && !this.#unnamedFailure) { + this.#unnamedFailure = true; + this.#scheduleStatusLine(); + } return; } const lines: string[] = this.#getProblemLines('failed', problem); @@ -364,7 +397,9 @@ export class AgentProgressRenderer { const label: string = STATUS_LABELS.get(status) ?? status.toLowerCase(); countsByLabel.set(label, (countsByLabel.get(label) ?? 0) + count); } - const counts: string[] = [...countsByLabel].map(([label, count]) => `${count} ${label}`); + const counts: string[] = [...countsByLabel].map( + ([label, count]) => `${count} ${(count !== 1 && PLURAL_LABELS.get(label)) || label}` + ); const matchedNothing: boolean = total === 0 && done === 0 && emptySelection && !tracker.hasGlobalOutput; let scope: string = ''; if (total > 0 || done > 0) { @@ -488,7 +523,13 @@ export class AgentProgressRenderer { #scheduleStatusLine(): void { clearTimeout(this.#statusTimer); - this.#statusTimer = setTimeout(() => this.#writePipeLine(this.#getStatusLine()), PIPE_STATUS_INTERVAL_MS); + this.#statusTimer = setTimeout( + () => { + this.#unnamedFailure = false; + this.#writePipeLine(this.#getStatusLine()); + }, + this.#unnamedFailure ? PIPE_UNREPORTED_FAILURE_DELAY_MS : PIPE_STATUS_INTERVAL_MS + ); this.#statusTimer.unref?.(); } diff --git a/apps/rush-cli-client/src/OperationOutputExcerpt.ts b/apps/rush-cli-client/src/OperationOutputExcerpt.ts index cb5b66d54a..b924c6a8d1 100644 --- a/apps/rush-cli-client/src/OperationOutputExcerpt.ts +++ b/apps/rush-cli-client/src/OperationOutputExcerpt.ts @@ -40,8 +40,11 @@ const SEVERITY_PREFIX_PATTERN: RegExp = /^(?:error|warning)\s*:\s*/i; const WHITESPACE_PATTERN: RegExp = /\s+/g; /** A tool's error count, such as Heft's `Encountered 2 errors` or tsc's `Found 1 error.` */ const ERROR_COUNT_PATTERN: RegExp = /^(?:encountered|found) (\d+) errors?\b/i; -/** A source location such as `src/x.ts:3:7` or `src/x.ts(3,7)`. */ -const SOURCE_LOCATION_PATTERN: RegExp = /[\w-]\.[A-Za-z]\w{0,5}(?::\d+|\(\d+,\d+\))/; +/** + * A source location such as `src/x.ts:3:7` or `src/x.ts(3,7)`, with a line and a column, so that a host and port + * such as `registry.example.com:443` is not taken for one. + */ +const SOURCE_LOCATION_PATTERN: RegExp = /[\w-]\.[A-Za-z]\w{0,5}(?::\d+:\d+|\(\d+,\d+\))/; /** Lines longer than this keep their start and end, joined by an ellipsis. */ const MAX_LINE_LENGTH: number = 300; diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index b2fbd46089..03fb79ca0c 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -108,6 +108,8 @@ export async function launchClientAsync( commandOrigin: !rushx && ['build', 'rebuild', 'install', 'update'].includes(route.commandName) ? 'built-in' : 'custom', invocationKind: rushx ? 'rushx' : 'rush', + // An agent acts on a failure as soon as it is known; the daemon finishes the independent work without it. + ...(agentRenderer ? { returnEarlyOnFailure: true } : {}), cwd, environment, terminal: { diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts index 9d6921e648..d6f64f22ca 100644 --- a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -156,6 +156,38 @@ describe(AgentProgressRenderer.name, () => { ]); }); + it('says how many operations continue in rushd after a failure that the daemon reported early', () => { + const operationIds: ReadonlyArray = ['a (build)', 'b (build)', 'c (build)', 'd (build)']; + const finishWith = (unfinished: ReadonlyArray<[string, string]>, exitCode: number = 1): string[] => { + const { renderer, lines } = createRenderer(false); + for (const operationId of operationIds) renderer.onEvent(registered(operationId)); + fail(renderer, 'b (build)', ['error']); + renderer.onEvent(status('a (build)', 'BLOCKED')); + renderer.finish({ + exitCode, + operationResults: [ + { operationId: 'a (build)', status: 'BLOCKED' }, + { operationId: 'b (build)', status: 'FAILURE' }, + ...unfinished.map(([operationId, value]) => ({ operationId, status: value })) + ] + }); + return lines(); + }; + + expect(finishWith([['c (build)', 'EXECUTING']]).pop()).toBe( + 'rush build: FAILURE 2/4 operations (1 failure, 1 blocked) in 0.0s · failed: b (build)' + + ' · 1 independent operation continues in rushd' + ); + expect( + finishWith([ + ['c (build)', 'EXECUTING'], + ['d (build)', 'QUEUED'] + ]).pop() + ).toMatch(/ · failed: b \(build\) · 2 independent operations continue in rushd$/); + expect(finishWith([['c (build)', 'SUCCESS']]).pop()).toMatch(/ · failed: b \(build\)$/); + expect(finishWith([['c (build)', 'EXECUTING']], 0).pop()).not.toContain('continue'); + }); + it('keeps failure diagnostics when successful operations wrote stderr first', () => { const { renderer, output } = createRenderer(false); renderer.onEvent(status('noisy (build)', 'EXECUTING')); @@ -585,7 +617,7 @@ describe(AgentProgressRenderer.name, () => { expect(report.filter((line) => line.startsWith(' f2 '))).toHaveLength(3); expect(report.slice(-2)).toEqual([ "+5 more failed operations; their logs are in each project's rush-logs folder", - 'rush build: FAILURE 8/8 operations (8 failure) in 0.0s · failed: f0 (build), f1 (build), f2 (build), ' + + 'rush build: FAILURE 8/8 operations (8 failures) in 0.0s · failed: f0 (build), f1 (build), f2 (build), ' + 'f3 (build), f4 (build) +3 more' ]); }); @@ -611,7 +643,7 @@ describe(AgentProgressRenderer.name, () => { ' src/b.ts:1:1 - error TS2322: b', 'failed: quiet (build)', ' spawn heft ENOENT', - 'rush build: FAILURE 3/3 operations (3 failure) in 0.0s · failed: a (build), quiet (build), b (build)' + 'rush build: FAILURE 3/3 operations (3 failures) in 0.0s · failed: a (build), quiet (build), b (build)' ]); }); @@ -631,7 +663,7 @@ describe(AgentProgressRenderer.name, () => { 'failed: q1 (build)', ' (no output)', "+2 more failed operations; their logs are in each project's rush-logs folder", - 'rush build: FAILURE 5/5 operations (5 failure) in 0.0s · ' + + 'rush build: FAILURE 5/5 operations (5 failures) in 0.0s · ' + 'failed: a (build), q1 (build), q2 (build), q3 (build), b (build)' ]); }); @@ -768,7 +800,10 @@ describe(AgentProgressRenderer.name, () => { renderer.onEvent(status('d (build)', 'EXECUTING')); renderer.onEvent(status('e (build)', 'EXECUTING')); fail(renderer, 'a (build)', ['error TS2322']); - advance(clock, 25_000); + // The failure report postpones the status line that was due at 50 s. + advance(clock, 24_999); + expect(lines()).toHaveLength(4); + advance(clock, 1); renderer.onEvent(status('b (build)', 'SUCCESS')); advance(clock, 25_000); clock.ms += 1000; @@ -819,6 +854,104 @@ describe(AgentProgressRenderer.name, () => { advance(clock, 100_000); expect(output).toHaveLength(written); }); + + it('names a failed operation that wrote no output in a status line 1 s after it failed (#1792)', () => { + const { renderer, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + for (const name of ['a', 'b', 'q']) { + renderer.onEvent(registered(`${name} (build)`)); + renderer.onEvent(status(`${name} (build)`, 'EXECUTING')); + } + advance(clock, 2000); + renderer.onEvent(status('q (build)', 'FAILURE', '/repo/q/rush-logs/x.log')); + advance(clock, 999); + expect(lines()).toHaveLength(1); + advance(clock, 1); + expect(lines()).toHaveLength(2); + // The next status line is due 25 s after that one, as usual. + advance(clock, 24_999); + expect(lines()).toHaveLength(2); + advance(clock, 1); + renderer.finish({ + exitCode: 1, + operationResults: [ + { operationId: 'a (build)', status: 'SUCCESS' }, + { operationId: 'b (build)', status: 'SUCCESS' }, + { operationId: 'q (build)', status: 'FAILURE', errorMessage: 'Returned error code: 1' } + ] + }); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush build 1/3 · 3.0s · running: a (build), b (build) · failed: q (build)', + 'rush build 1/3 · 28.0s · running: a (build), b (build) · failed: q (build)', + 'failed: q (build) · full log: /repo/q/rush-logs/x.log', + ' Returned error code: 1', + 'rush build: FAILURE 3/3 operations (1 failure, 2 success) in 28.0s · failed: q (build)' + ]); + }); + + it('writes no status line for a failure without output when the result comes within 1 s', () => { + const { renderer, output, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + renderer.onEvent(registered('q (build)')); + renderer.onEvent(registered('top (build)')); + renderer.onEvent(status('q (build)', 'EXECUTING')); + renderer.onEvent(status('q (build)', 'FAILURE')); + renderer.onEvent(status('top (build)', 'BLOCKED')); + advance(clock, 999); + renderer.finish({ + exitCode: 1, + operationResults: [ + { operationId: 'q (build)', status: 'FAILURE', errorMessage: 'Returned error code: 1' }, + { operationId: 'top (build)', status: 'BLOCKED' } + ] + }); + const written: number = output.length; + advance(clock, 100_000); + expect(output).toHaveLength(written); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'failed: q (build)', + ' Returned error code: 1', + 'rush build: FAILURE 2/2 operations (1 failure, 1 blocked) in 1.0s · failed: q (build)' + ]); + }); + + it('names failures without output that no line named yet in one status line, however they interleave', () => { + const { renderer, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + for (const name of ['a', 'b', 'q1', 'q2', 'q3']) { + renderer.onEvent(registered(`${name} (build)`)); + renderer.onEvent(status(`${name} (build)`, 'EXECUTING')); + } + renderer.onEvent(status('q1 (build)', 'FAILURE')); + advance(clock, 600); + // A second failure without output does not postpone the line. + renderer.onEvent(status('q2 (build)', 'FAILURE')); + advance(clock, 399); + expect(lines()).toHaveLength(1); + advance(clock, 1); + expect(lines()[1]).toBe( + 'rush build 2/5 · 1.0s · running: a (build), b (build), q3 (build) · failed: q1 (build), q2 (build)' + ); + // A failure report does not name an earlier failure without output, so a status line follows 1 s after it. + advance(clock, 5000); + renderer.onEvent(status('q3 (build)', 'FAILURE')); + advance(clock, 500); + fail(renderer, 'a (build)', ['src/a.ts:1:1 - error TS2322: a']); + advance(clock, 999); + expect(lines()).toHaveLength(4); + advance(clock, 1); + renderer.dispose(); + expect(lines().slice(2)).toEqual([ + 'failed: a (build) · full log: /repo/a/rush-logs/x.log', + ' src/a.ts:1:1 - error TS2322: a', + 'rush build 4/5 · 7.5s · running: b (build) · failed: q1 (build), q2 (build), q3 (build) +1 more' + ]); + }); }); it('renders at most three live rows on a TTY and clears them before the summary', () => { @@ -873,4 +1006,21 @@ describe(AgentProgressRenderer.name, () => { jest.useRealTimers(); } }); + + it('writes no status line on a TTY for a failed operation that wrote no output', () => { + jest.useFakeTimers(); + try { + const { renderer, output } = createRenderer(true); + renderer.start(); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onEvent(status('a (build)', 'FAILURE')); + jest.advanceTimersByTime(30_000); + expect(output.length).toBeGreaterThan(1); + // Every write after the first paint repaints the live rows. + expect(output.slice(1).filter((text) => !text.startsWith('\x1b[3A\x1b[0J'))).toEqual([]); + renderer.dispose(); + } finally { + jest.useRealTimers(); + } + }); }); diff --git a/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts b/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts index bc139a88fd..fb4046c2ff 100644 --- a/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts +++ b/apps/rush-cli-client/src/test/OperationOutputExcerpt.test.ts @@ -242,6 +242,25 @@ describe(OperationOutputExcerpt.name, () => { expect(unlocated.getExcerpt(8)).toEqual(['src/a.ts(1,1): something unexpected', 'Build failed']); }); + it('does not take a host and port for a source location (ch01 #1980)', () => { + for (const error of [ + 'npm ERR! connect ECONNREFUSED registry.example.com:443', + 'Error: request to https://pkgs.dev.azure.com:443/x failed' + ]) { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append('Installing @x/a from the registry\n', 'stderr'); + excerpt.append(`${error}\n`, 'stderr'); + expect(excerpt.getExcerpt(8)).toEqual(['Installing @x/a from the registry', error]); + } + }); + + it('shows a message that is repeated in another case once', () => { + const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); + excerpt.append("Error: Cannot find module 'X'\n", 'stdout'); + excerpt.append("error: cannot find module 'x'\n", 'stderr'); + expect(excerpt.getExcerpt(8)).toEqual(["Error: Cannot find module 'X'"]); + }); + it('always keeps the last line, which usually is the tool summary', () => { const excerpt: OperationOutputExcerpt = new OperationOutputExcerpt(); for (let i: number = 0; i < 20; i++) { diff --git a/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts b/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts new file mode 100644 index 0000000000..13d0a016b4 --- /dev/null +++ b/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts @@ -0,0 +1,99 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('@microsoft/rush/lib/start', () => ({})); +jest.mock('@rushstack/rush-client-core', () => ({ + ...jest.requireActual('@rushstack/rush-client-core'), + connectOrAwaitDaemonStartupAsync: jest.fn(), + executeWithDaemonRestartAsync: jest.fn() +})); + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + connectOrAwaitDaemonStartupAsync, + executeWithDaemonRestartAsync, + type DaemonClient +} from '@rushstack/rush-client-core'; +import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; + +import { AgentProgressRenderer } from '../AgentProgressRenderer'; +import * as connectionOptions from '../daemonConnectionOptions'; +import { launchClientAsync } from '../launchClient'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; + +describe('returnEarlyOnFailure', () => { + let folder: string; + let originalArgv: string[]; + let originalEnvironment: NodeJS.ProcessEnv; + let originalExitCode: typeof process.exitCode; + let requests: IDaemonRequestEnvelope[]; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-client-early-failure-')); + originalArgv = process.argv; + originalEnvironment = process.env; + originalExitCode = process.exitCode; + requests = []; + fs.writeFileSync( + path.join(folder, 'rush.json'), + JSON.stringify({ rushVersion: '5.178.1', pnpmVersion: '10.27.0', projects: [] }) + ); + jest.spyOn(connectionOptions, 'getDaemonConnectionOptionsAsync').mockResolvedValue({ + paths: { + runtimeDir: folder, + socketPath: path.join(folder, 'd.sock'), + lockfilePath: path.join(folder, 'daemon.pid.json') + } + }); + jest.spyOn(process.stderr, 'write').mockReturnValue(true); + jest.spyOn(process.stdout, 'write').mockReturnValue(true); + jest.spyOn(process, 'cwd').mockReturnValue(folder); + jest + .mocked(connectOrAwaitDaemonStartupAsync) + .mockResolvedValue({ closeAsync: async () => undefined } as unknown as DaemonClient); + jest.mocked(executeWithDaemonRestartAsync).mockImplementation(async (client, connection, options) => { + requests.push(options.request); + return { + kind: 'result', + result: { requestId: options.request.requestId, exitCode: 0, outcome: 'success', aborted: false } + }; + }); + process.argv = [process.execPath, 'rush-client', 'build', '--to', 'project']; + const environment: NodeJS.ProcessEnv = getTestProcessEnvironment(originalEnvironment); + for (const name of Object.keys(environment)) { + if (name.startsWith('RUSH_')) delete environment[name]; + } + process.env = { ...environment, RUSH_DAEMON: '1' }; + }); + + afterEach(() => { + process.argv = originalArgv; + process.env = originalEnvironment; + process.exitCode = originalExitCode; + jest.restoreAllMocks(); + jest.mocked(connectOrAwaitDaemonStartupAsync).mockReset(); + jest.mocked(executeWithDaemonRestartAsync).mockReset(); + fs.rmSync(folder, { recursive: true }); + }); + + it('is requested by agent output, which reports the failure and leaves the independent work to rushd', async () => { + const renderer: AgentProgressRenderer = new AgentProgressRenderer({ + commandName: 'build', + isTTY: false, + columns: 80, + write: () => undefined + }); + await launchClientAsync(false, renderer); + expect(requests).toHaveLength(1); + expect(requests[0].returnEarlyOnFailure).toBe(true); + }); + + it('is not requested by the default output, whose collated logs cover the whole request', async () => { + await launchClientAsync(false); + expect(requests).toHaveLength(1); + expect(requests[0]).not.toHaveProperty('returnEarlyOnFailure'); + }); +}); diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-nits_2026-09-28-21-10.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-nits_2026-09-28-21-10.json new file mode 100644 index 0000000000..28c7bdb478 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-nits_2026-09-28-21-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output on a pipe, name a failed operation that wrote no output in a status line 1 s after it failed, and say \"failures\" for more than one failure in the summary line. A failure excerpt no longer takes a host and port, such as `registry.example.com:443`, for a source location, so it keeps the lines before such an error.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-return-early_2026-09-28-20-25.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-return-early_2026-09-28-20-25.json new file mode 100644 index 0000000000..5ee20dfb56 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t108-return-early_2026-09-28-20-25.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, ask the daemon to return a failed build's result early, and say in the summary line how many independent operations continue in rushd.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/swarm-r04-t108-return-early_2026-09-28-20-25.json b/common/changes/@rushstack/rush-daemon-protocol/swarm-r04-t108-return-early_2026-09-28-20-25.json new file mode 100644 index 0000000000..74be47df86 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/swarm-r04-t108-return-early_2026-09-28-20-25.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add an optional `returnEarlyOnFailure` field to request envelopes and phased requests. A daemon that honors it writes a failed shared build's result once no unfinished target operation can change it. Older daemons ignore the field.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t108-fallback_2026-09-28-21-40.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t108-fallback_2026-09-28-21-40.json new file mode 100644 index 0000000000..62a195147a --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t108-fallback_2026-09-28-21-40.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Before rejecting a request that the client then runs in-process, stop the operations that a failed build continues after returning its result early, and wait until they have stopped and released the repository lock. A `list`, `check` or `scan` command and a rushx script still run alongside them. Adds `RequestScheduler.preemptLeasesAsync()`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t108-return-early_2026-09-28-20-25.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t108-return-early_2026-09-28-20-25.json new file mode 100644 index 0000000000..98d102f837 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t108-return-early_2026-09-28-20-25.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Honor `returnEarlyOnFailure` for shared builds: write a failed build's result once no unfinished target operation can change it, and keep running the operations that the failure didn't block. A later build waits for them; a request that can't run alongside a build, a restart for another environment, or a daemon shutdown stops them. Adds `RequestScheduler.markLeasePreemptible()`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index c7ec3d5863..da5aff785b 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -450,6 +450,7 @@ export interface IDaemonPhasedRequest { readonly environment: Readonly>; readonly operationSelection: ReadonlyArray; readonly requestId: string; + readonly returnEarlyOnFailure?: boolean; readonly terminalRequirement?: DaemonTerminalRequirement; } @@ -533,6 +534,7 @@ export interface IDaemonRequestEnvelope { readonly expectedWorkspaceGeneration?: string; readonly invocationKind?: DaemonInvocationKind; readonly requestId: string; + readonly returnEarlyOnFailure?: boolean; readonly terminal: IDaemonRequestTerminal; } diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index 0d30e1b2dc..1bc9b6fdba 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -724,6 +724,8 @@ export class RequestScheduler { acquireAsync(options: IRequestSchedulerAcquireOptions): Promise; get activeRequestCount(): number; downgradeExclusiveLease(lease: IRequestLease, target: RequestExclusivityClass.SharedBuild | RequestExclusivityClass.SharedRead): void; + markLeasePreemptible(lease: IRequestLease, onPreempted: () => void): void; + preemptLeasesAsync(): Promise; get queuedRequestCount(): number; } diff --git a/libraries/rush-daemon-protocol/src/DaemonPhasedRequest.ts b/libraries/rush-daemon-protocol/src/DaemonPhasedRequest.ts index 7dbc06d8ce..55b8eaed4c 100644 --- a/libraries/rush-daemon-protocol/src/DaemonPhasedRequest.ts +++ b/libraries/rush-daemon-protocol/src/DaemonPhasedRequest.ts @@ -63,6 +63,8 @@ export interface IDaemonPhasedRequest { readonly operationSelection: ReadonlyArray; /** A client-generated identifier unique within the connection. */ readonly requestId: string; + /** Copied from {@link IDaemonRequestEnvelope.returnEarlyOnFailure}; only shared builds honor it. */ + readonly returnEarlyOnFailure?: boolean; /** Terminal capability needed by the resolved command. */ readonly terminalRequirement?: DaemonTerminalRequirement; } diff --git a/libraries/rush-daemon-protocol/src/DaemonRequestEnvelope.ts b/libraries/rush-daemon-protocol/src/DaemonRequestEnvelope.ts index 9db2a24707..5c74f3bb39 100644 --- a/libraries/rush-daemon-protocol/src/DaemonRequestEnvelope.ts +++ b/libraries/rush-daemon-protocol/src/DaemonRequestEnvelope.ts @@ -52,6 +52,14 @@ export interface IDaemonRequestEnvelope { readonly invocationKind?: DaemonInvocationKind; /** A client-generated identifier unique within this connection. */ readonly requestId: string; + /** + * Whether a shared build may report its failure before the rest of its work ends. Once an operation failed and + * no operation of the selected projects that no other selected project consumes is unfinished, the daemon + * writes the result, in which unfinished operations report their current status. Those keep running in the + * daemon, and the request stays active until they end, unless a request that cannot run alongside them, or a + * restart, stops them first. Older daemons ignore this field. + */ + readonly returnEarlyOnFailure?: boolean; /** Request-local terminal capabilities. */ readonly terminal: IDaemonRequestTerminal; } diff --git a/libraries/rush-daemon-protocol/src/RequestControlValidation.ts b/libraries/rush-daemon-protocol/src/RequestControlValidation.ts index 5f628a2cda..5686cac294 100644 --- a/libraries/rush-daemon-protocol/src/RequestControlValidation.ts +++ b/libraries/rush-daemon-protocol/src/RequestControlValidation.ts @@ -4,7 +4,7 @@ import { isDaemonControlRecord } from './ControlRecord'; import { validateDaemonInvocationKind, validateExpectedWorkspaceGeneration } from './DaemonInvocationKind'; import { DaemonProtocolError } from './DaemonProtocolError'; -import { validateRequestAdmission, validateRequestTerminal } from './RequestEnvelopeValidation'; +import { validateRequestHandling } from './RequestEnvelopeValidation'; import { validateRequestId } from './RequestIdentifierValidation'; import { validateRequestResultFields } from './RequestResultValidation'; @@ -33,8 +33,7 @@ export function validateRequestStartControl(payload: Record): v requireString(payload.cwd, 'requestStart payload.cwd'); requireStringArray(payload.argv, 'requestStart payload.argv'); requireStringRecord(payload.environment, 'requestStart payload.environment'); - validateRequestTerminal(payload.terminal); - validateRequestAdmission(payload.admission); + validateRequestHandling(payload); } /** Validates a request-cancel payload. @internal */ diff --git a/libraries/rush-daemon-protocol/src/RequestEnvelopeValidation.ts b/libraries/rush-daemon-protocol/src/RequestEnvelopeValidation.ts index 1bf3a7f6be..0a60e01634 100644 --- a/libraries/rush-daemon-protocol/src/RequestEnvelopeValidation.ts +++ b/libraries/rush-daemon-protocol/src/RequestEnvelopeValidation.ts @@ -8,8 +8,14 @@ import { MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS } from './DaemonRequestAdmission'; const FIRST_TERMINAL_COLUMN: number = 1; const MINIMUM_WAIT_TIMEOUT_MS: number = 0; -/** Validates request terminal fields. @internal */ -export function validateRequestTerminal(value: unknown): void { +/** Validates the request's terminal, its admission and whether it may return early on failure. @internal */ +export function validateRequestHandling(payload: Record): void { + validateRequestTerminal(payload.terminal); + validateRequestAdmission(payload.admission); + validateOptionalBoolean(payload.returnEarlyOnFailure, 'requestStart payload.returnEarlyOnFailure'); +} + +function validateRequestTerminal(value: unknown): void { const terminal: Record = requireRecord(value, 'requestStart payload.terminal'); requireBoolean(terminal.isTTY, 'Request terminal isTTY'); requireBoolean(terminal.supportsColor, 'Request terminal supportsColor'); @@ -18,8 +24,7 @@ export function validateRequestTerminal(value: unknown): void { validateTerminalRequirement(terminal.terminalRequirement); } -/** Validates request admission fields. @internal */ -export function validateRequestAdmission(value: unknown): void { +function validateRequestAdmission(value: unknown): void { if (value === undefined) return; const admission: Record = requireRecord(value, 'requestStart payload.admission'); validateOptionalBoolean(admission.noWait, 'Request admission noWait'); diff --git a/libraries/rush-daemon-protocol/src/test/RequestReturnEarlyOnFailure.test.ts b/libraries/rush-daemon-protocol/src/test/RequestReturnEarlyOnFailure.test.ts new file mode 100644 index 0000000000..449ece70f2 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/test/RequestReturnEarlyOnFailure.test.ts @@ -0,0 +1,37 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { decodeDaemonControlMessage, encodeDaemonControlMessage } from '../ControlFrameCodec'; +import type { IDaemonRequestStartMessage } from '../DaemonRequestControl'; + +const NUMBER_FLAG: number = 1; + +function createRequestStart(returnEarlyOnFailure: unknown): IDaemonRequestStartMessage { + return { + kind: 'requestStart', + payload: { + argv: ['build', '--to', 'project-a'], + commandName: 'build', + commandOrigin: 'built-in', + cwd: '/repo', + environment: {}, + requestId: 'early-failure-request', + returnEarlyOnFailure: returnEarlyOnFailure as boolean | undefined, + terminal: { isTTY: false, supportsColor: false } + } + }; +} + +function roundTrip(message: IDaemonRequestStartMessage): unknown { + return decodeDaemonControlMessage(encodeDaemonControlMessage(message)); +} + +it.each([true, false, undefined])('round-trips returnEarlyOnFailure %p', (value: boolean | undefined) => { + expect(roundTrip(createRequestStart(value))).toEqual(createRequestStart(value)); +}); + +it.each(['true', NUMBER_FLAG, null])('rejects a non-boolean returnEarlyOnFailure %p', (value: unknown) => { + expect(() => roundTrip(createRequestStart(value))).toThrow( + 'requestStart payload.returnEarlyOnFailure must be a boolean.' + ); +}); diff --git a/libraries/rush-daemon/src/DaemonControlSession.ts b/libraries/rush-daemon/src/DaemonControlSession.ts index 51f9bff38a..4e043b4f33 100644 --- a/libraries/rush-daemon/src/DaemonControlSession.ts +++ b/libraries/rush-daemon/src/DaemonControlSession.ts @@ -107,6 +107,13 @@ export class DaemonControlSession { } public closeAsync(drainRequests: boolean = false, reason?: DaemonShutdownError): Promise { + // Unlike a disconnect, closing the session also stops the work that a request still runs after it sent its + // result (a failed build that returned early). No client waits for a restart result from such a request. + for (const state of this.#requestById.values()) { + if (state.client.terminalOutcomeSent) { + state.abortController.abort(reason ?? new Error('The daemon control session is closing.')); + } + } this.#closePromise ??= this.#closeOnceAsync(drainRequests, reason); return this.#closePromise; } @@ -448,12 +455,14 @@ export class DaemonControlSession { return this.#closePromise; } - #markClosing(reason: Error): void { + #markClosing(reason: Error, keepFinishedRequests: boolean = false): void { if (this.#isClosing) return; this.#isClosing = true; this.#interactiveConnection.close(reason); for (const state of this.#requestById.values()) { - state.abortController.abort(reason); + if (!keepFinishedRequests || !state.client.terminalOutcomeSent) { + state.abortController.abort(reason); + } } } @@ -490,7 +499,8 @@ export class DaemonControlSession { async #handleConnectionClosedAsync(error: Error | undefined): Promise { if (this.#connectionClosed) return; this.#connectionClosed = true; - this.#markClosing(error ?? new Error('The daemon client connection closed.')); + // A client that disconnects after its result does not stop the work that its request still runs. + this.#markClosing(error ?? new Error('The daemon client connection closed.'), true); const settlements: PromiseSettledResult[] = await Promise.allSettled( Array.from(this.#requestById.values(), (state: IRequestState) => state.completion) ); diff --git a/libraries/rush-daemon/src/PhasedRequestEventSink.ts b/libraries/rush-daemon/src/PhasedRequestEventSink.ts index e726b07ccb..58988b2d59 100644 --- a/libraries/rush-daemon/src/PhasedRequestEventSink.ts +++ b/libraries/rush-daemon/src/PhasedRequestEventSink.ts @@ -48,6 +48,18 @@ interface IEventOptions { readonly scope?: IDaemonEventScope; } +/** How a sink reports that a request which returns early on failure can have its result; see the constructor. */ +export interface IEarlyFailureOptions { + /** The operations whose results decide the request's outcome. */ + readonly targetOperationIds: ReadonlySet; + /** Receives the number of the client's operations, not counting silent ones, that are still unfinished. */ + readonly onSettled: (unfinishedOperations: number) => void; +} + +function isFailedStatus(status: OperationStatus): boolean { + return status === OperationStatus.Failure || status === OperationStatus.Blocked; +} + class OrderedClientWriter { readonly #client: IPhasedRequestClient; readonly #onFailure: (error: Error) => void; @@ -101,8 +113,13 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { readonly #rushVersion: string; readonly #writer: OrderedClientWriter; readonly #onActiveOperationsSettled: (() => void) | undefined; + readonly #earlyFailure: IEarlyFailureOptions | undefined; readonly #pendingOperationIds: Set = new Set(); + /** The current iteration's records of this client's operations; their statuses change as the iteration runs. */ + readonly #scheduledResults: Map = new Map(); #completedOperations: number = 0; + #failed: boolean = false; + #earlyFailureOffered: boolean = false; #settled: boolean = false; #totalOperations: number = 0; @@ -118,11 +135,18 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { * enqueued on this sink's writer before the callback runs. */ onActiveOperationsSettled?: () => void; + /** + * For a request that returns early on failure: `onSettled` is called at most once per iteration, after any + * operation's completion event, when one of this client's operations failed or was blocked, none was aborted, + * none of the targets is unfinished, and `onActiveOperationsSettled` was not called. + */ + earlyFailure?: IEarlyFailureOptions; }) { this.#activeOperationIds = options.activeOperationIds; this.#client = options.client; this.#getNextSequence = options.getNextSequence; this.#onActiveOperationsSettled = options.onActiveOperationsSettled; + this.#earlyFailure = options.earlyFailure; this.#rushVersion = options.rushVersion; this.#writer = new OrderedClientWriter(options.client, options.onWriteFailure); } @@ -131,6 +155,11 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { return this.#observedResults.get(operation); } + /** The current iteration's record of one of this client's operations, with its current status. */ + public getScheduledResult(operation: Operation): IOperationExecutionResult | undefined { + return this.#scheduledResults.get(operation); + } + public flushAsync(): Promise { return this.#writer.flushAsync(); } @@ -145,12 +174,17 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { this.#completedOperations = 0; this.#totalOperations = 0; this.#pendingOperationIds.clear(); + this.#scheduledResults.clear(); + this.#failed = false; + this.#earlyFailureOffered = false; this.#settled = false; for (const record of records) { const operationId: string = record.operation.name; if (!this.#activeOperationIds.has(operationId)) { continue; } + this.#scheduledResults.set(record.operation, record); + this.#failed ||= isFailedStatus(record.status); if (!record.silent) { this.#totalOperations++; } @@ -161,18 +195,9 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { } public onOperationCompleted(result: IOperationExecutionResult): void { - if (!this.#pendingOperationIds.delete(result.operation.name) || this.#settled) { - return; - } - if (result.status === OperationStatus.Aborted) { - // The iteration is being aborted or failed to start; leave this client's result to the batch. - this.#settled = true; - return; - } - if (this.#pendingOperationIds.size === 0) { - this.#settled = true; - this.#onActiveOperationsSettled?.(); - } + this.#settleActiveOperation(result); + // A failure elsewhere can block this client's operations, so any operation's completion can decide its result. + this.#offerEarlyFailure(); } public onOperationStatusChanged( @@ -187,6 +212,7 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { executionResult: result, status: result.status }); + this.#failed ||= isFailedStatus(result.status); // Summarizing clients (agent output) point at the full log of the operations that explain a failure. const logFilePath: string | undefined = result.status === OperationStatus.Failure || result.status === OperationStatus.SuccessWithWarning @@ -257,6 +283,48 @@ export class PhasedRequestEventSink implements _IOperationGraphEventSink { }); } + #settleActiveOperation(result: IOperationExecutionResult): void { + if (!this.#pendingOperationIds.delete(result.operation.name) || this.#settled) { + return; + } + if (result.status === OperationStatus.Aborted) { + // The iteration is being aborted or failed to start; leave this client's result to the batch. + this.#settled = true; + return; + } + if (this.#pendingOperationIds.size === 0) { + this.#settled = true; + this.#onActiveOperationsSettled?.(); + } + } + + /** + * Offers a failed request's result once nothing that is unfinished can change it. Blocked operations emit their + * completion events only when the iteration ends, so this reads the records' current statuses instead. + */ + #offerEarlyFailure(): void { + if (!this.#earlyFailure || !this.#failed || this.#settled || this.#earlyFailureOffered) { + return; + } + const { targetOperationIds, onSettled } = this.#earlyFailure; + let unfinishedOperations: number = 0; + for (const record of this.#scheduledResults.values()) { + if (record.status === OperationStatus.Aborted) { + return; + } + if (!TERMINAL_OPERATION_STATUSES.has(record.status)) { + if (targetOperationIds.has(record.operation.name)) { + return; + } + if (!record.silent) { + unfinishedOperations++; + } + } + } + this.#earlyFailureOffered = true; + onSettled(unfinishedOperations); + } + #emitEvent(type: DaemonEventType, payload: unknown, options?: IEventOptions): void { this.#writer.writeEvent(() => ({ eventId: randomUUID(), diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index 1d2dfd9fd7..f8e59bc411 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -68,6 +68,8 @@ interface IPreparedPhasedRequest { readonly client: IPhasedRequestClient; readonly exclusivityClass: RequestExclusivityClass; readonly interactiveSession: IInteractiveRequestSession | undefined; + /** Lets requests that cannot run alongside this one preempt its workspace admission; see `#settleEntry`. */ + readonly markAdmissionPreemptible: (onPreempted: () => void) => void; readonly request: IDaemonPhasedRequest; /** Only requests with the same settings share one graph iteration. */ readonly requestSettings: IPhasedCommandEngineRequestSettings | undefined; @@ -84,6 +86,12 @@ interface IBatchEntry extends IPreparedPhasedRequest { abortListener: (() => void) | undefined; abortRequested: boolean; completed: boolean; + /** + * Set when a failed result is published while operations of this request that the failure did not block are + * still unfinished. They keep running, and the request stays active until the iteration ends; see + * `#finishFailedEntry`. + */ + continuesAfterResult: boolean; executionStarted: boolean; /** * Set when this entry's result starts being produced, so it is produced exactly once. An entry can finish while @@ -95,9 +103,18 @@ interface IBatchEntry extends IPreparedPhasedRequest { reject: (error: unknown) => void; requestSink: PhasedRequestEventSink | undefined; resolve: (result: IDaemonPhasedRequestResult) => void; + /** Settles the request of an entry that continues after its result, once its iteration ended. */ + settleAfterIteration: (() => void) | undefined; unsubscribe: (() => void) | undefined; } +/** + * How a result reports the client's operations: after the iteration ended (`'final'`), or while it still runs, + * with unfinished operations reported as aborted because the client stopped waiting for them (`'abandoned'`), or + * with their current status because they run on after an early failure result (`'running'`). + */ +type OperationReport = 'final' | 'abandoned' | 'running'; + const ROUTING_STATE_BY_GRAPH: WeakMap = new WeakMap(); const OBSERVED_STATUS_OVERRIDES_RETAINED: ReadonlySet = new Set([ OperationStatus.Aborted, @@ -164,6 +181,7 @@ export class PhasedRequestRouter { commandName: request.commandName, commandOrigin: request.commandOrigin }); + const workspaceScheduler: RequestScheduler = getWorkspaceRequestScheduler(this.#workspaceSession); let admissionController: RequestAdmissionController | undefined; let admissionLease: IRequestLease; try { @@ -172,10 +190,7 @@ export class PhasedRequestRouter { client, requestId: request.requestId }); - admissionLease = await admissionController.acquireAsync( - getWorkspaceRequestScheduler(this.#workspaceSession), - exclusivityClass - ); + admissionLease = await admissionController.acquireAsync(workspaceScheduler, exclusivityClass); } catch (error) { admissionController?.dispose(); return await finishAfterAdmissionErrorAsync(request, client, interactiveSession, error); @@ -214,11 +229,14 @@ export class PhasedRequestRouter { if (client.abortSignal.aborted) { return await writeAbortedResultAsync(request.requestId, client, interactiveSession); } + const lease: IRequestLease = admissionLease; return await routingState.coordinator.enqueueAsync( { client, exclusivityClass, interactiveSession, + markAdmissionPreemptible: (onPreempted: () => void) => + workspaceScheduler.markLeasePreemptible(lease, onPreempted), receivedTimeMs: receivedTimeMs ?? startTimeMs, request, requestSettings, @@ -297,6 +315,7 @@ class PhasedRequestBatchCoordinator { abortListener: undefined, abortRequested: false, completed: false, + continuesAfterResult: false, executionStarted: false, finishPromise: undefined, outputError: undefined, @@ -304,6 +323,7 @@ class PhasedRequestBatchCoordinator { reject, requestSink: undefined, resolve, + settleAfterIteration: undefined, unsubscribe: undefined }; entry.abortListener = () => this.#deactivateEntry(entry, true); @@ -471,6 +491,15 @@ class PhasedRequestBatchCoordinator { getNextSequence: () => entry.client.getNextEventSequence(), onWriteFailure: (error: Error) => this.#deactivateEntry(entry, false, error), onActiveOperationsSettled: () => this.#finishSettledEntry(entry), + earlyFailure: + entry.request.returnEarlyOnFailure === true && + entry.exclusivityClass === RequestExclusivityClass.SharedBuild + ? { + targetOperationIds: getTargetOperationIds(entry.selection.activeOperations), + onSettled: (unfinishedOperations: number) => + this.#finishFailedEntry(entry, unfinishedOperations) + } + : undefined, rushVersion: this.#workspaceSession.metadata.rushVersion }); entry.unsubscribe = this.#multiplexer.subscribe(entry.requestSink); @@ -544,6 +573,9 @@ class PhasedRequestBatchCoordinator { await releaseExecutionLeaseAsync(); } finally { graphLease.release(); + for (const entry of batch) { + this.#settleContinuingEntry(entry); + } } } } @@ -602,7 +634,7 @@ class PhasedRequestBatchCoordinator { undefined, [], undefined, - true + 'abandoned' ).catch((error: unknown) => { if (!entry.completed) { this.#completeEntry(entry); @@ -656,9 +688,12 @@ class PhasedRequestBatchCoordinator { ); } - /** Whether a live participant still waits for the running iteration to produce its result. */ + /** + * Whether a live participant still waits for the running iteration to produce its result, or has its result but + * continues until the iteration ends. + */ #needsIteration(entry: IBatchEntry): boolean { - return entry.finishPromise === undefined && this.#isEntryLive(entry); + return (entry.finishPromise === undefined || entry.continuesAfterResult) && this.#isEntryLive(entry); } /** @@ -674,31 +709,99 @@ class PhasedRequestBatchCoordinator { */ #finishSettledEntry(entry: IBatchEntry): void { if ( - !this.#needsIteration(entry) || + entry.finishPromise !== undefined || + !this.#isEntryLive(entry) || !entry.participated || - !this.#currentBatch?.some( - (candidate: IBatchEntry) => candidate !== entry && this.#needsIteration(candidate) - ) + !this.#hasOtherBatchParticipant(entry) + ) { + return; + } + this.#startEarlyResult(entry, 'abandoned'); + } + + /** + * Publishes a failed result as soon as nothing that is unfinished can change it, for a request that asked for + * this (agent output): one of its operations failed or was blocked, and none of its targets is unfinished. + * + * @remarks + * Its operations that the failure did not block keep running, so that later requests find them done. The + * request stays active until the iteration ends, and `#settleContinuingEntry` then settles it. Meanwhile it keeps + * its admission, so exclusive requests still wait for that work, and it holds the iteration like any live + * participant, so another participant's departure does not abort that work. When none of its operations is + * unfinished and no other participant needs the iteration, the iteration is ending anyway, and the ordinary + * contract applies. + */ + #finishFailedEntry(entry: IBatchEntry, unfinishedOperations: number): void { + if ( + entry.finishPromise !== undefined || + !this.#isEntryLive(entry) || + !entry.participated || + (unfinishedOperations === 0 && !this.#hasOtherBatchParticipant(entry)) ) { return; } + entry.continuesAfterResult = unfinishedOperations > 0; + this.#startEarlyResult(entry, 'running'); + } + + #hasOtherBatchParticipant(entry: IBatchEntry): boolean { + return !!this.#currentBatch?.some( + (candidate: IBatchEntry) => candidate !== entry && this.#needsIteration(candidate) + ); + } + + #startEarlyResult(entry: IBatchEntry, report: OperationReport): void { entry.unsubscribe?.(); entry.unsubscribe = undefined; - entry.finishPromise = this.#produceEarlyResultAsync(entry).catch((error: unknown) => { + entry.finishPromise = this.#produceEarlyResultAsync(entry, report).catch((error: unknown) => { // Unlike a batch-wide failure, an early result's failure concerns only this client. if (!entry.completed) { + this.#abandonContinuingEntry(entry); this.#completeEntry(entry); - entry.reject(error); + // The abandoned operations may still be stopping, so the request keeps its admission until they are. + this.#settleEntry(entry, () => entry.reject(error)); } }); } - async #produceEarlyResultAsync(entry: IBatchEntry): Promise { + async #produceEarlyResultAsync(entry: IBatchEntry, report: OperationReport): Promise { // The sink is notified from the record's `finalizeOperation()`, which synchronously precedes the close of // the record's StdioSummarizer and ProblemCollector. The summary reads the failure tail from the closed // summarizer, so yield once to let the notifying record finish closing before the summary is written. await Promise.resolve(); - await this.#produceResultAsync(entry, true, undefined, [], undefined, true); + await this.#produceResultAsync(entry, true, undefined, [], undefined, report); + } + + /** + * Stops the work that only an entry which continues after its result still holds, for example when the daemon + * shuts down or when that result could not be produced. + */ + #abandonContinuingEntry(entry: IBatchEntry): void { + if (!entry.continuesAfterResult || entry.abortRequested) { + return; + } + entry.abortRequested = true; + if (!this.#currentBatch?.includes(entry)) { + return; + } + if (this.#hasLiveBatchParticipant()) { + this.#restrictBatchDemand(); + } else if (this.#graph.hasScheduledIteration || this.#graph.status === OperationStatus.Executing) { + this.#requestIterationAbort(); + } + } + + #settleContinuingEntry(entry: IBatchEntry): void { + const settle: (() => void) | undefined = entry.settleAfterIteration; + if (!settle) { + return; + } + entry.settleAfterIteration = undefined; + if (entry.abortListener) { + entry.client.abortSignal.removeEventListener('abort', entry.abortListener); + entry.abortListener = undefined; + } + settle(); } #requestIterationAbort(): void { @@ -724,7 +827,7 @@ class PhasedRequestBatchCoordinator { executionError, batchCleanupErrors, beforeResultAsync, - false + 'final' ); return entry.finishPromise; } @@ -735,7 +838,7 @@ class PhasedRequestBatchCoordinator { executionError: unknown, batchCleanupErrors: ReadonlyArray, beforeResultAsync: (() => Promise) | undefined, - iterationInProgress: boolean + report: OperationReport ): Promise { if (entry.completed) { return; @@ -772,7 +875,7 @@ class PhasedRequestBatchCoordinator { this.#graph, entry.requestSink, aborted && entry.participated, - iterationInProgress + report ) : []; const result: IDaemonPhasedRequestResult = createPhasedCommandResult({ @@ -790,11 +893,25 @@ class PhasedRequestBatchCoordinator { try { await entry.client.writeResultAsync(result); this.#completeEntry(entry); - entry.resolve(result); + this.#settleEntry(entry, () => entry.resolve(result)); } catch (error) { this.#completeEntry(entry); - entry.reject(error); + this.#settleEntry(entry, () => entry.reject(error)); + } + } + + #settleEntry(entry: IBatchEntry, settle: () => void): void { + if (!entry.continuesAfterResult) { + settle(); + return; } + entry.settleAfterIteration = settle; + // The client has its result, so from now on its departure no longer matters, but the request's own abort + // (the daemon shutting down) still stops the work it holds. + entry.abortListener = () => this.#abandonContinuingEntry(entry); + entry.client.abortSignal.addEventListener('abort', entry.abortListener, { once: true }); + // Nobody waits for that work, so it must not delay requests that cannot run alongside it, such as a rebuild. + entry.markAdmissionPreemptible(() => this.#abandonContinuingEntry(entry)); } async #rejectEntryAsync(entry: IBatchEntry, error: unknown): Promise { @@ -806,6 +923,8 @@ class PhasedRequestBatchCoordinator { } } if (entry.completed) { + // The batch failed after this entry's early result: its iteration has ended. + this.#settleContinuingEntry(entry); return; } entry.unsubscribe?.(); @@ -919,6 +1038,9 @@ function validateRequestIdentity(request: IDaemonPhasedRequest): void { if (request.acceptsStdin !== undefined && typeof request.acceptsStdin !== 'boolean') { throw new Error('Phased request acceptsStdin must be a boolean value.'); } + if (request.returnEarlyOnFailure !== undefined && typeof request.returnEarlyOnFailure !== 'boolean') { + throw new Error('Phased request returnEarlyOnFailure must be a boolean value.'); + } if ( request.terminalRequirement !== undefined && request.terminalRequirement !== 'none' && @@ -1062,6 +1184,33 @@ function collectSelectionClosure( return Array.from(activeOperations); } +/** + * The operations whose results decide a request's outcome: those of the selected projects that no other selected + * project consumes, such as the projects named by `--to`. The other selected operations only feed them. + * + * @remarks + * Projects rather than operations are compared, because an operation that nothing consumes, such as the last + * phase of a dependency, still only serves a consuming project's request. + */ +function getTargetOperationIds(activeOperations: ReadonlyArray): ReadonlySet { + const active: ReadonlySet = new Set(activeOperations); + const consumedProjects: Set = new Set(); + for (const consumer of activeOperations) { + for (const dependency of consumer.dependencies) { + if (active.has(dependency) && dependency.associatedProject !== consumer.associatedProject) { + consumedProjects.add(dependency.associatedProject); + } + } + } + const targetOperationIds: Set = new Set(); + for (const operation of activeOperations) { + if (!consumedProjects.has(operation.associatedProject)) { + targetOperationIds.add(operation.name); + } + } + return targetOperationIds; +} + /** Presentation/scheduling settings are request-scoped, so they are applied per iteration, not per graph. */ function applyRequestSettings( graph: IOperationGraph, @@ -1113,16 +1262,25 @@ function collectOperationOutcomes( graph: IOperationGraph, requestSink: PhasedRequestEventSink, fillMissingAsAborted: boolean = false, - iterationInProgress: boolean = false + report: OperationReport = 'final' ): ReadonlyArray { const outcomes: IPhasedOperationOutcome[] = []; for (const operation of [...activeOperations].sort(compareOperations)) { const observed: ReturnType = requestSink.getObservedResult(operation); const retained: IOperationExecutionResult | undefined = graph.resultByOperation.get(operation); + const current: IOperationExecutionResult | undefined = + report === 'running' ? requestSink.getScheduledResult(operation) : undefined; let status: string | undefined; let errorMessage: string | undefined; - if (iterationInProgress && observed !== undefined) { + if (current !== undefined) { + if (current.silent && IN_PROGRESS_STATUSES.has(current.status)) { + // An operation that runs nothing, for example a phase that the project does not define. + continue; + } + status = current.status; + errorMessage = current.error?.message; + } else if (report !== 'final' && observed !== undefined) { // While the iteration still runs, retained results may predate this iteration, and work this client // stopped observing before it finished (a detached cancellation) was abandoned. status = IN_PROGRESS_STATUSES.has(observed.status) ? OperationStatus.Aborted : observed.status; diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 2edb24d3b7..4ac806c7d1 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -139,6 +139,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { }, operationSelection, requestId: envelope.requestId, + returnEarlyOnFailure: envelope.returnEarlyOnFailure, terminalRequirement: envelope.terminal.terminalRequirement } }; diff --git a/libraries/rush-daemon/src/RequestScheduler.ts b/libraries/rush-daemon/src/RequestScheduler.ts index b5fdab3020..eb84e15d00 100644 --- a/libraries/rush-daemon/src/RequestScheduler.ts +++ b/libraries/rush-daemon/src/RequestScheduler.ts @@ -93,7 +93,11 @@ interface IQueuedRequest { interface ILeaseState { exclusivityClass: RequestExclusivityClass; + onPreempted: (() => void) | undefined; + /** Whether `onPreempted` was called, so that the lease's owner is stopping its work. */ + preempted: boolean; released: boolean; + readonly onReleased: (() => void)[]; } /** @@ -109,6 +113,7 @@ export class RequestScheduler { readonly #queue: IQueuedRequest[] = []; #activeClass: RequestExclusivityClass | undefined; #activeRequestCount: number = 0; + readonly #activeLeaseStates: Set = new Set(); readonly #leaseStates: WeakMap = new WeakMap(); /** @@ -128,6 +133,42 @@ export class RequestScheduler { this.#drainQueue(); } + /** + * Lets requests that cannot be admitted alongside an active lease preempt it. `onPreempted` is called once, as + * soon as such a request waits for admission; the lease's owner should then stop its work and release the lease. + * + * @remarks + * For a request that no client waits for any more, such as a failed build that returned its result while its + * independent operations continue, so that it never delays another request. + */ + public markLeasePreemptible(lease: IRequestLease, onPreempted: () => void): void { + const state: ILeaseState | undefined = this.#leaseStates.get(lease); + if (!state || state.released) { + throw new Error('Only an active lease from this scheduler can be marked preemptible.'); + } + state.onPreempted = onPreempted; + this.#preemptIfContended(); + } + + /** + * Preempts every active lease that was marked preemptible, as a request that cannot be admitted alongside it + * would, and resolves once all of them are released. Other leases and queued requests are not affected. + * + * @remarks + * For a request that the daemon does not serve, so that the command which its client then runs in-process does + * not run alongside work that nobody waits for. + */ + public preemptLeasesAsync(): Promise { + const releases: Promise[] = []; + for (const state of Array.from(this.#activeLeaseStates)) { + if (state.onPreempted || state.preempted) { + releases.push(new Promise((resolve) => state.onReleased.push(resolve))); + this.#preempt(state); + } + } + return Promise.all(releases).then(() => undefined); + } + /** * The number of requests currently waiting for admission. */ @@ -236,7 +277,13 @@ export class RequestScheduler { this.#activeClass = exclusivityClass; this.#activeRequestCount++; - const state: ILeaseState = { exclusivityClass, released: false }; + const state: ILeaseState = { + exclusivityClass, + onPreempted: undefined, + preempted: false, + released: false, + onReleased: [] + }; const lease: IRequestLease = { get exclusivityClass(): RequestExclusivityClass { return state.exclusivityClass; @@ -247,14 +294,20 @@ export class RequestScheduler { } state.released = true; + state.onPreempted = undefined; + this.#activeLeaseStates.delete(state); this.#activeRequestCount--; if (this.#activeRequestCount === 0) { this.#activeClass = undefined; } this.#drainQueue(); + for (const onReleased of state.onReleased.splice(0)) { + onReleased(); + } } }; this.#leaseStates.set(lease, state); + this.#activeLeaseStates.add(state); return lease; } @@ -275,6 +328,33 @@ export class RequestScheduler { if (admittedRequest) { this.#notifyQueuePositions(); } + this.#preemptIfContended(); + } + + #preemptIfContended(): void { + const head: IQueuedRequest | undefined = this.#queue[0]; + if (!head || this.#canAdmit(head.options.exclusivityClass)) { + return; + } + for (const state of Array.from(this.#activeLeaseStates)) { + this.#preempt(state); + } + } + + #preempt(state: ILeaseState): void { + const onPreempted: (() => void) | undefined = state.onPreempted; + if (!onPreempted) { + return; + } + state.onPreempted = undefined; + state.preempted = true; + try { + onPreempted(); + } catch (error) { + process.emitWarning(error instanceof Error ? error : String(error), { + code: 'RUSH_DAEMON_LEASE_PREEMPTION_CALLBACK_ERROR' + }); + } } #rejectQueuedRequest(request: IQueuedRequest, error: Error): void { diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 3b8a306224..4f0cc8d9a0 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -56,6 +56,7 @@ import type { IWorkspaceSuccessorLaunch } from './WorkspaceProcessRestart'; import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from './WorkspaceRestartArbiter'; +import { classifyRushCommand } from './RushCommandRequestPolicy'; import { FreshCaptureCoalescer } from './FreshCaptureCoalescer'; interface IExecutionState { @@ -212,12 +213,18 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { const state: IExecutionState = { began: false, terminalAttempted: false, resultDrained: false }; const observer: AbortController | undefined = isGraphWatch(envelope) ? new AbortController() : undefined; if (observer) this.#observers.add(observer); + const preemption: AbortController = new AbortController(); const signal: AbortSignal = AbortSignal.any([ destination.abortSignal, this.#abortController.signal, + preemption.signal, ...(observer ? [observer.signal] : []) ]); - const client: IDaemonRequestDispatchClient = createLifecycleClient(destination, signal, state); + // Set while a request whose work may outlast its result (see `#yieldAfterResult`) is dispatched. + let onResultDrained: (() => void) | undefined; + const client: IDaemonRequestDispatchClient = createLifecycleClient(destination, signal, state, () => + onResultDrained?.() + ); const admission: RequestAdmissionController = new RequestAdmissionController({ admission: envelope.admission, client, @@ -242,6 +249,9 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { receivedTimeMs ); generation = prepared; + if (mayContinueAfterResult(envelope)) { + onResultDrained = () => this.#yieldAfterResult(prepared.lease, ticket, preemption); + } if (isRushxInvocation(envelope)) { // Never waits: exclusive holders of this lease also hold `#gate` exclusively, and this request holds it. scriptLease = await admission.acquireAsync(this.#scripts, RequestExclusivityClass.SharedBuild); @@ -318,8 +328,20 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { }); return; } + if ( + isFallbackRejection(error) && + !state.began && + !state.terminalAttempted && + !mayFallBackAlongsideContinuingWork(envelope) + ) { + // The client runs this command in-process instead. Work that finished requests continue would run + // alongside it, like two Rush commands in one checkout. + generation?.lease.release(); + await this.#stopContinuingWorkAsync(client.abortSignal); + } throw error; } finally { + onResultDrained = undefined; generation?.lease.release(); generation = undefined; scriptLease?.release(); @@ -332,6 +354,42 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } } + /** + * A request that may return its result before its work ends (a failed build whose independent operations + * continue) must not delay other requests once its client has that result, since nobody waits for that work: + * a restart no longer waits for it, and a request that needs this generation exclusively stops it, as does a + * request that the client runs in-process instead (see `#stopContinuingWorkAsync`). + */ + #yieldAfterResult( + lease: IRequestLease, + ticket: IWorkspaceRestartTicket | undefined, + preemption: AbortController + ): void { + if (ticket) this.#restartArbiter.leave(ticket); + this.#gate.markLeasePreemptible(lease, () => + preemption.abort( + new Error('A request that cannot run alongside this finished request stopped its remaining work.') + ) + ); + } + + /** Stops the work that finished requests continue (see `#yieldAfterResult`) and waits until it has stopped. */ + async #stopContinuingWorkAsync(abortSignal: AbortSignal): Promise { + const stopped: Promise = this.#gate.preemptLeasesAsync(); + if (abortSignal.aborted) return; + // A client that leaves no longer runs the command, so it need not wait any more. + let onAbort: () => void = () => undefined; + const aborted: Promise = new Promise((resolve) => { + onAbort = resolve; + abortSignal.addEventListener('abort', onAbort, { once: true }); + }); + try { + await Promise.race([stopped, aborted]); + } finally { + abortSignal.removeEventListener('abort', onAbort); + } + } + async #prepareAsync( envelope: IDaemonRequestEnvelope, client: IDaemonRequestDispatchClient, @@ -958,6 +1016,36 @@ function isGraphWatch(envelope: IDaemonRequestEnvelope): boolean { return isGraphRequest(envelope) && envelope.argv[1] === 'graph' && envelope.argv[2] === 'watch'; } +/** Only shared builds honor `returnEarlyOnFailure`, as `PhasedRequestRouter` does. */ +function mayContinueAfterResult(envelope: IDaemonRequestEnvelope): boolean { + return ( + envelope.returnEarlyOnFailure === true && + classifyRushCommand({ commandName: envelope.commandName, commandOrigin: envelope.commandOrigin }) === + RequestExclusivityClass.SharedBuild + ); +} + +/** A rejection after which the client runs the command in-process. */ +function isFallbackRejection(error: unknown): error is DaemonRequestDispatchError { + return error instanceof DaemonRequestDispatchError && error.code === 'unsupported'; +} + +/** + * Whether a command that the client runs in-process may run alongside the work that finished requests continue: + * a rushx script, or a built-in command that only reads the workspace, such as `rush list`. + * + * @remarks + * The client cannot tell a built-in command from a custom one, so it sends every command that the daemon does not + * serve as a custom command. command-line.json cannot reuse a built-in command's name, so the name identifies one. + */ +function mayFallBackAlongsideContinuingWork(envelope: IDaemonRequestEnvelope): boolean { + return ( + isRushxInvocation(envelope) || + classifyRushCommand({ commandName: envelope.commandName, commandOrigin: 'built-in' }) === + RequestExclusivityClass.SharedRead + ); +} + function preExecutionFailure(requestId: string, error: Error): IDaemonCommandResult { return { requestId, exitCode: 1, outcome: 'failure', aborted: false, errorMessage: error.message }; } @@ -965,7 +1053,8 @@ function preExecutionFailure(requestId: string, error: Error): IDaemonCommandRes function createLifecycleClient( client: IDaemonRequestDispatchClient, abortSignal: AbortSignal, - state: IExecutionState + state: IExecutionState, + onResultDrained: () => void ): IDaemonRequestDispatchClient { return { abortSignal, @@ -994,6 +1083,14 @@ function createLifecycleClient( state.terminalAttempted = true; await client.writeResultAsync(result); state.resultDrained = true; + try { + onResultDrained(); + } catch (error) { + // The result was delivered, so this must not turn into a failure to write it. + process.emitWarning(error instanceof Error ? error : String(error), { + code: 'RUSH_DAEMON_RESULT_DRAINED_CALLBACK_ERROR' + }); + } } }; } diff --git a/libraries/rush-daemon/src/test/DaemonRequestWireEarlyFailure.test.ts b/libraries/rush-daemon/src/test/DaemonRequestWireEarlyFailure.test.ts new file mode 100644 index 0000000000..3c862e4771 --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonRequestWireEarlyFailure.test.ts @@ -0,0 +1,166 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { OperationStatus } from '@microsoft/rush-lib'; +import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; + +import { RushDaemonHost } from '../RushDaemonHost'; +import { + TEST_ENGINE_SHAPE, + TestOperationRunner, + createRoutingFixture +} from './PhasedRequestRouterTestUtilities'; +import type { ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; +import { + CallbackDaemonRequestResolver, + DaemonRequestWireClient, + createDeferred, + createWireEnvelope +} from './DaemonRequestWireTestUtilities'; +import type { IDeferred, ITerminalExchange } from './DaemonRequestWireTestUtilities'; + +const OPERATION_A: string = 'project-a (_phase:test)'; +const OPERATION_B: string = 'project-b (_phase:test)'; +const OPERATION_C: string = 'project-c (_phase:test)'; +const testRepoRoots: Set = new Set(); + +afterEach(() => { + for (const repoRoot of testRepoRoots) fs.rmSync(repoRoot, { force: true, recursive: true }); + testRepoRoots.clear(); +}); + +interface IEarlyFailureHost { + readonly abortSpy: jest.SpyInstance; + readonly fixture: ITestRoutingFixture; + readonly host: RushDaemonHost; + readonly releaseC: IDeferred; + readonly repoRoot: string; +} + +/** A consumes B and C. B fails once C runs, and C runs until `releaseC` resolves. */ +async function startEarlyFailureHostAsync(): Promise { + const repoRoot: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-wire-early-failure-')); + testRepoRoots.add(repoRoot); + const startedC: IDeferred = createDeferred(); + const releaseC: IDeferred = createDeferred(); + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A)], + [OPERATION_B, new TestOperationRunner(OPERATION_B, OperationStatus.Failure, () => startedC.promise)], + [ + OPERATION_C, + new TestOperationRunner(OPERATION_C, OperationStatus.Success, async (): Promise => { + startedC.resolve(); + await releaseC.promise; + }) + ] + ]), + [ + [OPERATION_A, OPERATION_B], + [OPERATION_A, OPERATION_C] + ], + { parallelism: 2 } + ); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + const host: RushDaemonHost = await RushDaemonHost.startAsync({ + createWorkspaceSessionAsync: () => Promise.resolve(fixture.session), + daemonVersion: 'wire-test', + repoRoot, + requestResolver: new CallbackDaemonRequestResolver(async ({ envelope }) => ({ + kind: 'phased', + request: { + commandName: envelope.commandName, + commandOrigin: envelope.commandOrigin, + engineShape: TEST_ENGINE_SHAPE, + environment: envelope.environment, + operationSelection: envelope.argv + .slice(1) + .map((operationId: string) => ({ enabledState: true, operationId })), + requestId: envelope.requestId, + returnEarlyOnFailure: envelope.returnEarlyOnFailure + } + })), + rushVersion: '5.178.1' + }); + return { abortSpy, fixture, host, releaseC, repoRoot }; +} + +/** Counts the aborts that the router requested; every iteration's start also calls the spy without options. */ +function countTerminatingAborts(abortSpy: jest.SpyInstance): number { + return abortSpy.mock.calls.filter( + ([options]: ReadonlyArray<{ terminateRunning?: boolean } | undefined>) => + options?.terminateRunning === true + ).length; +} + +async function settleAsync(): Promise { + for (let turn: number = 0; turn < 20; turn++) { + await new Promise((resolve) => setImmediate(resolve)); + } +} + +async function buildAndDisconnectAsync({ host, repoRoot }: IEarlyFailureHost): Promise { + const client: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync(host.paths.socketPath); + await client.handshakeAsync(); + const envelope: IDaemonRequestEnvelope = createWireEnvelope('agent', 'build', repoRoot, { + argv: ['build', OPERATION_A], + commandOrigin: 'built-in', + returnEarlyOnFailure: true + }); + await client.sendControlAsync({ kind: 'requestStart', payload: envelope }); + const exchange: ITerminalExchange = await client.readTerminalAsync('agent'); + await client.closeAsync(); + await client.closed; + return exchange; +} + +describe('daemon requests that return early on failure', () => { + it('keeps the work that continues after the result when the client disconnects, until the daemon closes', async () => { + const setup: IEarlyFailureHost = await startEarlyFailureHostAsync(); + try { + const exchange: ITerminalExchange = await buildAndDisconnectAsync(setup); + expect(exchange.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, outcome: 'failure', requestId: 'agent' } + }); + await settleAsync(); + expect(setup.fixture.graph.status).toBe(OperationStatus.Executing); + expect(countTerminatingAborts(setup.abortSpy)).toBe(0); + + const closed: Promise = setup.host.closeAsync(); + await settleAsync(); + expect(countTerminatingAborts(setup.abortSpy)).toBe(1); + setup.releaseC.resolve(); + await closed; + } finally { + setup.releaseC.resolve(); + await setup.host.closeAsync(); + } + }); + + it('lets the work that continues finish after the client disconnects', async () => { + const setup: IEarlyFailureHost = await startEarlyFailureHostAsync(); + try { + await buildAndDisconnectAsync(setup); + setup.releaseC.resolve(); + while (setup.fixture.graph.status === OperationStatus.Executing) { + await settleAsync(); + } + await settleAsync(); + await setup.host.closeAsync(); + + expect(countTerminatingAborts(setup.abortSpy)).toBe(0); + const retainedC: OperationStatus | undefined = [...setup.fixture.graph.resultByOperation.values()].find( + ({ operation }) => operation.name === OPERATION_C + )?.status; + expect(retainedC).toBe(OperationStatus.Success); + } finally { + setup.releaseC.resolve(); + await setup.host.closeAsync(); + } + }); +}); diff --git a/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts b/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts new file mode 100644 index 0000000000..f24c64d808 --- /dev/null +++ b/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts @@ -0,0 +1,415 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonPhasedRequest, IDaemonPhasedRequestResult } from '@rushstack/rush-daemon-protocol'; +import { OperationStatus } from '@microsoft/rush-lib'; + +import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import { + TEST_ENGINE_SHAPE, + TestOperationRunner, + TestPhasedRequestClient, + createRoutingFixture +} from './PhasedRequestRouterTestUtilities'; +import type { ITestClientWrite, ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; + +const OPERATION_A: string = 'project-a (_phase:test)'; +const OPERATION_B: string = 'project-b (_phase:test)'; +const OPERATION_C: string = 'project-c (_phase:test)'; +/** A consumes B and C, so A is the only target of a request that selects A. */ +const A_CONSUMES_B_AND_C: ReadonlyArray = [ + [OPERATION_A, OPERATION_B], + [OPERATION_A, OPERATION_C] +]; + +interface IDeferred { + readonly promise: Promise; + readonly resolve: () => void; +} + +function createDeferred(): IDeferred { + let resolvePromise: (() => void) | undefined; + const promise: Promise = new Promise((resolve) => { + resolvePromise = resolve; + }); + return { promise, resolve: () => resolvePromise?.() }; +} + +function createRequest( + requestId: string, + returnEarlyOnFailure: boolean, + ...selectedOperationIds: ReadonlyArray +): IDaemonPhasedRequest { + return { + commandName: 'build', + commandOrigin: 'built-in', + engineShape: TEST_ENGINE_SHAPE, + environment: {}, + operationSelection: selectedOperationIds.map((operationId: string) => ({ + enabledState: true, + operationId + })), + requestId, + ...(returnEarlyOnFailure ? { returnEarlyOnFailure } : {}) + }; +} + +async function settleAsync(): Promise { + for (let turn: number = 0; turn < 20; turn++) { + await new Promise((resolve) => setImmediate(resolve)); + } +} + +interface IEarlyFailureFixture { + readonly events: string[]; + readonly fixture: ITestRoutingFixture; + /** Lets C, which is slow, finish. */ + readonly releaseC: () => void; + readonly router: PhasedRequestRouter; + readonly startedC: Promise; +} + +/** + * B fails and C is slow. Unless `failBeforeCStarts` is set, B fails only once C runs, so C is executing when B's + * failure decides a request's result. + */ +function createEarlyFailureFixture( + dependencies: ReadonlyArray = A_CONSUMES_B_AND_C, + failBeforeCStarts: boolean = false +): IEarlyFailureFixture { + const startedC: IDeferred = createDeferred(); + const releaseC: IDeferred = createDeferred(); + const events: string[] = []; + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A)], + [ + OPERATION_B, + new TestOperationRunner(OPERATION_B, OperationStatus.Failure, async (): Promise => { + if (!failBeforeCStarts) { + await startedC.promise; + } + }) + ], + [ + OPERATION_C, + new TestOperationRunner(OPERATION_C, OperationStatus.Success, async (): Promise => { + startedC.resolve(); + await releaseC.promise; + }) + ] + ]), + dependencies, + { parallelism: 2 } + ); + fixture.session.acquireExecutionLeaseAsync = async (): Promise => { + events.push('acquired'); + return { + [Symbol.asyncDispose]: async (): Promise => { + events.push('released'); + } + }; + }; + return { + events, + fixture, + releaseC: releaseC.resolve, + router: new PhasedRequestRouter(fixture.session), + startedC: startedC.promise + }; +} + +interface ITrackedClient { + readonly client: TestPhasedRequestClient; + /** Resolves when the client's result was written, with the graph status at that moment. */ + readonly written: Promise; +} + +function trackClient(label: string, { events, fixture }: IEarlyFailureFixture): ITrackedClient { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(label); + let onWritten: (status: OperationStatus) => void = () => undefined; + const written: Promise = new Promise((resolve) => { + onWritten = resolve; + }); + client.onWriteAsync = async (write: ITestClientWrite): Promise => { + if (write.result) { + events.push(`wrote:${label}`); + onWritten(fixture.graph.status); + } + }; + return { client, written }; +} + +function trackResult( + resultPromise: Promise, + label: string, + events: string[] +): Promise { + return resultPromise.then((result: IDaemonPhasedRequestResult) => { + events.push(`result:${label}`); + return result; + }); +} + +/** Counts the aborts that the router requested; every iteration's start also calls the spy without options. */ +function countTerminatingAborts(abortSpy: jest.SpyInstance): number { + return abortSpy.mock.calls.filter( + ([options]: ReadonlyArray<{ terminateRunning?: boolean } | undefined>) => + options?.terminateRunning === true + ).length; +} + +function getRetainedStatus({ graph }: ITestRoutingFixture, operationId: string): OperationStatus | undefined { + return [...graph.resultByOperation.values()].find(({ operation }) => operation.name === operationId) + ?.status; +} + +function getWrittenResults(client: TestPhasedRequestClient): ReadonlyArray { + return client.writes.flatMap(({ result }: ITestClientWrite) => (result ? [result] : [])); +} + +interface IOrdinaryCase { + readonly commandName: string; + readonly dependencies: ReadonlyArray; + readonly failBeforeCStarts: boolean; + readonly name: string; + readonly selection: ReadonlyArray; +} + +/** Requests that ask to return early on failure, but whose failed result can only be written at the end. */ +const ORDINARY_CASES: ReadonlyArray = [ + { + commandName: 'build', + dependencies: [[OPERATION_A, OPERATION_B]], + failBeforeCStarts: false, + name: 'a target is still running', + selection: [OPERATION_A, OPERATION_C] + }, + { + commandName: 'build', + dependencies: [[OPERATION_A, OPERATION_B]], + failBeforeCStarts: true, + name: 'nothing else is unfinished', + selection: [OPERATION_A] + }, + { + commandName: 'rebuild', + dependencies: A_CONSUMES_B_AND_C, + failBeforeCStarts: false, + name: 'the request is not a shared build', + selection: [OPERATION_A] + } +]; + +describe('phased requests that return early on failure', () => { + it('writes a failed result once no unfinished operation can change it, and keeps independent work running', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + const { events, fixture, router } = setup; + const agent: ITrackedClient = trackClient('agent', setup); + + const resultPromise: Promise = trackResult( + router.executeAsync(createRequest('agent', true, OPERATION_A), agent.client), + 'agent', + events + ); + + expect(await agent.written).toBe(OperationStatus.Executing); + await settleAsync(); + // The request stays active, holding its admission and the execution lease, until C finishes. + expect(events).toEqual(['acquired', 'wrote:agent']); + const [early] = getWrittenResults(agent.client); + expect(early).toMatchObject({ aborted: false, exitCode: 1, outcome: 'failure', scheduled: true }); + expect(early.operationResults).toEqual([ + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Blocked }), + expect.objectContaining({ operationId: OPERATION_B, status: OperationStatus.Failure }), + expect.objectContaining({ operationId: OPERATION_C, status: OperationStatus.Executing }) + ]); + + setup.releaseC(); + expect(await resultPromise).toBe(early); + expect(events).toEqual(['acquired', 'wrote:agent', 'released', 'result:agent']); + expect(getWrittenResults(agent.client)).toHaveLength(1); + expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); + // A later build can use it, e.g. from the build cache. + expect(getRetainedStatus(fixture, OPERATION_C)).toBe(OperationStatus.Success); + }); + + it('keeps the ordinary contract for a request that did not ask to return early', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + const tracked: ITrackedClient = trackClient('human', setup); + const resultPromise: Promise = trackResult( + setup.router.executeAsync(createRequest('human', false, OPERATION_A), tracked.client), + 'human', + setup.events + ); + + await setup.startedC; + await settleAsync(); + expect(setup.events).toEqual(['acquired']); + setup.releaseC(); + const result: IDaemonPhasedRequestResult = await resultPromise; + + expect(setup.events).toEqual(['acquired', 'released', 'wrote:human', 'result:human']); + expect(result.operationResults).toEqual([ + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Blocked }), + expect.objectContaining({ operationId: OPERATION_B, status: OperationStatus.Failure }), + expect.objectContaining({ operationId: OPERATION_C, status: OperationStatus.Success }) + ]); + }); + + it.each(ORDINARY_CASES)( + 'writes the result after the iteration when $name', + async ({ commandName, dependencies, failBeforeCStarts, selection }: IOrdinaryCase) => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(dependencies, failBeforeCStarts); + const tracked: ITrackedClient = trackClient('agent', setup); + const resultPromise: Promise = trackResult( + setup.router.executeAsync( + { ...createRequest('agent', true, ...selection), commandName }, + tracked.client + ), + 'agent', + setup.events + ); + + if (!failBeforeCStarts) { + await setup.startedC; + await settleAsync(); + expect(setup.events).toEqual(['acquired']); + } + setup.releaseC(); + const result: IDaemonPhasedRequestResult = await resultPromise; + + expect(setup.events).toEqual(['acquired', 'released', 'wrote:agent', 'result:agent']); + expect(result).toMatchObject({ exitCode: 1, outcome: 'failure' }); + } + ); + + it('keeps work that it shares with a participant that cancels after the early result', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + const { events, fixture, router } = setup; + const agent: ITrackedClient = trackClient('agent', setup); + const departing: TestPhasedRequestClient = new TestPhasedRequestClient('departing'); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + + const agentPromise: Promise = trackResult( + router.executeAsync(createRequest('agent', true, OPERATION_A), agent.client), + 'agent', + events + ); + const departingPromise: Promise = router.executeAsync( + createRequest('departing', false, OPERATION_C), + departing + ); + await agent.written; + departing.abortController.abort(); + + expect(await departingPromise).toMatchObject({ aborted: true, outcome: 'aborted' }); + setup.releaseC(); + await agentPromise; + + expect(countTerminatingAborts(abortSpy)).toBe(0); + expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); + expect(getRetainedStatus(fixture, OPERATION_C)).toBe(OperationStatus.Success); + }); + + it('lets a later build wait for the work that continues instead of stopping it', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + const { events, fixture, router } = setup; + const agent: ITrackedClient = trackClient('agent', setup); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + + const agentPromise: Promise = trackResult( + router.executeAsync(createRequest('agent', true, OPERATION_A), agent.client), + 'agent', + events + ); + await agent.written; + await settleAsync(); + const laterPromise: Promise = trackResult( + router.executeAsync(createRequest('later', false, OPERATION_C), new TestPhasedRequestClient('later')), + 'later', + events + ); + await settleAsync(); + + expect(countTerminatingAborts(abortSpy)).toBe(0); + expect(events).toEqual(['acquired', 'wrote:agent']); + expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(1); + setup.releaseC(); + expect(await laterPromise).toMatchObject({ exitCode: 0, outcome: 'success' }); + await agentPromise; + + expect(countTerminatingAborts(abortSpy)).toBe(0); + expect(getRetainedStatus(fixture, OPERATION_C)).toBe(OperationStatus.Success); + + // The later build ran its own iteration only after the one that continued had ended. + expect(events.slice(0, 3)).toEqual(['acquired', 'wrote:agent', 'released']); + expect(events.indexOf('result:agent')).toBeLessThan(events.indexOf('result:later')); + expect(events.filter((event: string) => event === 'acquired')).toHaveLength(2); + }); + + it.each(['rebuild', 'list'])( + 'lets a %s request that cannot run alongside the work that continues stop it', + async (commandName: string) => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + const { events, fixture, router } = setup; + const agent: ITrackedClient = trackClient('agent', setup); + const abortSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'abortCurrentIterationAsync'); + + const agentPromise: Promise = trackResult( + router.executeAsync(createRequest('agent', true, OPERATION_A), agent.client), + 'agent', + events + ); + await agent.written; + await settleAsync(); + const otherPromise: Promise = trackResult( + router.executeAsync( + { ...createRequest(commandName, false, OPERATION_C), commandName }, + new TestPhasedRequestClient(commandName) + ), + commandName, + events + ); + await settleAsync(); + + expect(countTerminatingAborts(abortSpy)).toBe(1); + setup.releaseC(); + await Promise.all([agentPromise, otherPromise]); + + expect(events.indexOf('result:agent')).toBeLessThan(events.indexOf(`result:${commandName}`)); + expect(getWrittenResults(agent.client)).toHaveLength(1); + } + ); + + it('stops the work that continues when the request is aborted after its result', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + const agent: ITrackedClient = trackClient('agent', setup); + const abortSpy: jest.SpyInstance = jest.spyOn(setup.fixture.graph, 'abortCurrentIterationAsync'); + + const resultPromise: Promise = setup.router.executeAsync( + createRequest('agent', true, OPERATION_A), + agent.client + ); + await agent.written; + await settleAsync(); + expect(countTerminatingAborts(abortSpy)).toBe(0); + agent.client.abortController.abort(); + + expect(countTerminatingAborts(abortSpy)).toBe(1); + setup.releaseC(); + const result: IDaemonPhasedRequestResult = await resultPromise; + expect(result).toBe(getWrittenResults(agent.client)[0]); + expect(getWrittenResults(agent.client)).toHaveLength(1); + }); + + it('rejects a flag that is not a boolean', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(); + await expect( + setup.router.executeAsync( + { ...createRequest('agent', false, OPERATION_A), returnEarlyOnFailure: 'yes' as unknown as boolean }, + new TestPhasedRequestClient('agent') + ) + ).rejects.toThrow('Phased request returnEarlyOnFailure must be a boolean value.'); + }); +}); diff --git a/libraries/rush-daemon/src/test/RequestScheduler.test.ts b/libraries/rush-daemon/src/test/RequestScheduler.test.ts index e927b1f9a3..8c37d84cd4 100644 --- a/libraries/rush-daemon/src/test/RequestScheduler.test.ts +++ b/libraries/rush-daemon/src/test/RequestScheduler.test.ts @@ -257,3 +257,148 @@ describe(RequestScheduler.name, () => { expect(scheduler.activeRequestCount).toBe(0); }); }); + +describe('preemptible leases', () => { + it('preempts a marked lease once a request that it blocks waits, and only once', async () => { + const scheduler: RequestScheduler = new RequestScheduler(); + const leftover: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const running: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const onPreempted: jest.Mock = jest.fn(); + scheduler.markLeasePreemptible(leftover, onPreempted); + + // A request of the same shared class is admitted alongside it. + const build: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + expect(onPreempted).not.toHaveBeenCalled(); + + const exclusive: Promise = scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.Exclusive + }); + expect(onPreempted).toHaveBeenCalledTimes(1); + const read: Promise = scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedRead + }); + leftover.release(); + build.release(); + expect(scheduler.queuedRequestCount).toBe(2); + running.release(); + (await exclusive).release(); + (await read).release(); + expect(onPreempted).toHaveBeenCalledTimes(1); + expect(scheduler.activeRequestCount).toBe(0); + }); + + it('preempts at once when a blocked request already waits', async () => { + const scheduler: RequestScheduler = new RequestScheduler(); + const leftover: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const read: Promise = scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedRead + }); + const onPreempted: jest.Mock = jest.fn(() => leftover.release()); + + scheduler.markLeasePreemptible(leftover, onPreempted); + + expect(onPreempted).toHaveBeenCalledTimes(1); + (await read).release(); + expect(scheduler.activeRequestCount).toBe(0); + }); + + it('forgets the mark when the lease is released and rejects marking an inactive lease', async () => { + const scheduler: RequestScheduler = new RequestScheduler(); + const lease: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const onPreempted: jest.Mock = jest.fn(); + scheduler.markLeasePreemptible(lease, onPreempted); + lease.release(); + + const other: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const exclusive: Promise = scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.Exclusive + }); + other.release(); + (await exclusive).release(); + + expect(onPreempted).not.toHaveBeenCalled(); + expect(() => scheduler.markLeasePreemptible(lease, onPreempted)).toThrow( + 'Only an active lease from this scheduler can be marked preemptible.' + ); + expect(() => new RequestScheduler().markLeasePreemptible(other, onPreempted)).toThrow(); + }); + + it('reports a failing preemption callback as a warning and keeps scheduling', async () => { + const scheduler: RequestScheduler = new RequestScheduler(); + const warningSpy: jest.SpyInstance = jest.spyOn(process, 'emitWarning').mockImplementation(() => {}); + try { + const lease: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + scheduler.markLeasePreemptible(lease, () => { + throw new Error('preemption failed'); + }); + const exclusive: Promise = scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.Exclusive + }); + + expect(warningSpy).toHaveBeenCalledWith(expect.objectContaining({ message: 'preemption failed' }), { + code: 'RUSH_DAEMON_LEASE_PREEMPTION_CALLBACK_ERROR' + }); + lease.release(); + (await exclusive).release(); + expect(scheduler.activeRequestCount).toBe(0); + } finally { + warningSpy.mockRestore(); + } + }); + + it('preempts marked leases on request and resolves once they are released', async () => { + const scheduler: RequestScheduler = new RequestScheduler(); + const leftover: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const running: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + const onPreempted: jest.Mock = jest.fn(); + scheduler.markLeasePreemptible(leftover, onPreempted); + const released: string[] = []; + + void scheduler.preemptLeasesAsync().then(() => released.push('first')); + expect(onPreempted).toHaveBeenCalledTimes(1); + // A later caller also waits for the lease that is stopping, without preempting it again. + void scheduler.preemptLeasesAsync().then(() => released.push('second')); + expect(onPreempted).toHaveBeenCalledTimes(1); + expect(scheduler.queuedRequestCount).toBe(0); + await Promise.resolve(); + await Promise.resolve(); + expect(released).toEqual([]); + + leftover.release(); + await new Promise((resolve) => setImmediate(resolve)); + expect(released).toEqual(['first', 'second']); + // The lease that is not preemptible stays active, and with no preemptible lease the call resolves at once. + expect(scheduler.activeRequestCount).toBe(1); + await expect(scheduler.preemptLeasesAsync()).resolves.toBeUndefined(); + running.release(); + }); + + it('waits for a lease whose preemption callback releases it at once', async () => { + const scheduler: RequestScheduler = new RequestScheduler(); + const leftover: IRequestLease = await scheduler.acquireAsync({ + exclusivityClass: RequestExclusivityClass.SharedBuild + }); + scheduler.markLeasePreemptible(leftover, () => leftover.release()); + + await scheduler.preemptLeasesAsync(); + expect(scheduler.activeRequestCount).toBe(0); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts b/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts new file mode 100644 index 0000000000..2cda8f5ef8 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts @@ -0,0 +1,253 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { OperationStatus } from '@microsoft/rush-lib'; +import { LockFile } from '@rushstack/node-core-library'; +import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; + +import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import type { ITerminalExchange } from './DaemonRequestWireTestUtilities'; +import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; +import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; + +jest.setTimeout(60_000); + +const BUILD_B: string[] = ['build', '--to', 'b', '--parallelism', '3']; + +/** + * b consumes a and c. a fails, and c holds its build open until the test removes the `hold` marker, so a build of b + * that returns early on failure leaves c running. + */ +function createEarlyFailureFixtureAsync(restartable: boolean = false): Promise { + return DaemonGraphTestFixture.createAsync((created: DaemonGraphTestFixture) => { + if (restartable) { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + } + created.write('hold', ''); + created.write( + 'b/package.json', + JSON.stringify({ + name: 'b', + version: '1.0.0', + dependencies: { a: '1.0.0', c: '1.0.0' }, + scripts: { '_phase:compile': 'node build.cjs' } + }) + ); + created.write( + 'a/build.cjs', + "require('node:fs').appendFileSync('../runs.txt','a\\n');console.error('failed-a');process.exitCode=1;" + ); + created.write( + 'c/build.cjs', + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','c\\n');" + + "const t=setInterval(()=>{if(!fs.existsSync('../hold')){clearInterval(t);console.log('finished-c');}},20);" + ); + }); +} + +function countRuns(fixture: DaemonGraphTestFixture, name: string): number { + return fixture.runs().filter((run: string) => run === name).length; +} + +async function waitForRunsAsync(fixture: DaemonGraphTestFixture, name: string, count: number): Promise { + const deadline: number = Date.now() + 30_000; + while (countRuns(fixture, name) < count && Date.now() < deadline) await delayAsync(20); +} + +/** Whether an in-process Rush command could take the Rush lock that the daemon holds while it executes. */ +function isNativeLockFree(fixture: DaemonGraphTestFixture): boolean { + const probe: LockFile | undefined = LockFile.tryAcquire( + fixture.session.rushConfiguration.commonTempFolder, + 'rush' + ); + probe?.release(); + return probe !== undefined; +} + +async function isSettledAsync(promise: Promise): Promise { + let settled: boolean = false; + promise.then( + () => (settled = true), + () => (settled = true) + ); + await delayAsync(0); + return settled; +} + +async function returnEarlyAsync(fixture: DaemonGraphTestFixture): Promise { + const early: ITerminalExchange = await fixture.runAsync(BUILD_B, { returnEarlyOnFailure: true }); + expect(early.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, outcome: 'failure' } + }); + // The client disconnected with its result, while c still runs. + await waitForRunsAsync(fixture, 'c', 1); + expect(countRuns(fixture, 'c')).toBe(1); + expect(countRuns(fixture, 'b')).toBe(0); +} + +describe('a failed build that returns early', () => { + it('lets a rebuild stop the work that continues instead of waiting for it', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); + const hold: string = path.join(fixture.folder, 'hold'); + try { + await returnEarlyAsync(fixture); + + const rebuild: Promise = fixture.runAsync([ + 'rebuild', + '--to', + 'c', + '--parallelism', + '3' + ]); + // The rebuild runs c again while the first c is still held, so it did not wait for that c to finish. + await waitForRunsAsync(fixture, 'c', 2); + expect(countRuns(fixture, 'c')).toBe(2); + expect(fs.existsSync(hold)).toBe(true); + + fs.rmSync(hold); + expect((await rebuild).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + } finally { + fs.rmSync(hold, { force: true }); + await fixture[Symbol.asyncDispose](); + } + }); + + it('lets a later build wait for the work that continues and find it done', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); + const hold: string = path.join(fixture.folder, 'hold'); + try { + await returnEarlyAsync(fixture); + + const later: Promise = fixture.runAsync([ + 'build', + '--to', + 'c', + '--parallelism', + '3' + ]); + await delayAsync(1000); + expect(await isSettledAsync(later)).toBe(false); + expect(countRuns(fixture, 'c')).toBe(1); + + fs.rmSync(hold); + expect((await later).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(countRuns(fixture, 'c')).toBe(1); + } finally { + fs.rmSync(hold, { force: true }); + await fixture[Symbol.asyncDispose](); + } + }); + + it('stops the work that continues before it rejects a command that the client then runs in-process', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); + const hold: string = path.join(fixture.folder, 'hold'); + try { + await returnEarlyAsync(fixture); + + const custom: ITerminalExchange = await fixture.runAsync(['test', '--to', 'c'], { + commandOrigin: 'custom' + }); + expect(custom.terminal).toMatchObject({ kind: 'requestRejected', payload: { code: 'unsupported' } }); + // The held c was stopped before the rejection was sent, and the in-process command can take the Rush lock. + expect(fs.existsSync(hold)).toBe(true); + expect(fixture.session.operationGraph?.status).not.toBe(OperationStatus.Executing); + expect(isNativeLockFree(fixture)).toBe(true); + + // So a later build runs c again at once, instead of waiting for the held c. + const later: Promise = fixture.runAsync([ + 'build', + '--to', + 'c', + '--parallelism', + '3' + ]); + await waitForRunsAsync(fixture, 'c', 2); + expect(countRuns(fixture, 'c')).toBe(2); + fs.rmSync(hold); + expect((await later).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + } finally { + fs.rmSync(hold, { force: true }); + await fixture[Symbol.asyncDispose](); + } + }); + + it('leaves the work that continues running for a read-only command or a rushx script that it rejects', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); + const hold: string = path.join(fixture.folder, 'hold'); + try { + await returnEarlyAsync(fixture); + + // As rush-client sends them: it marks every command that the daemon does not serve as a custom command. + const requests: [string[], Partial][] = [ + [['list'], { commandOrigin: 'custom' }], + [['start'], { commandOrigin: 'custom', invocationKind: 'rushx' }] + ]; + for (const [argv, overrides] of requests) { + expect((await fixture.runAsync(argv, overrides)).terminal).toMatchObject({ + kind: 'requestRejected', + payload: { code: 'unsupported' } + }); + } + expect(fixture.session.operationGraph?.status).toBe(OperationStatus.Executing); + expect(isNativeLockFree(fixture)).toBe(false); + expect(countRuns(fixture, 'c')).toBe(1); + + const later: Promise = fixture.runAsync([ + 'build', + '--to', + 'c', + '--parallelism', + '3' + ]); + fs.rmSync(hold); + expect((await later).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + expect(countRuns(fixture, 'c')).toBe(1); + } finally { + fs.rmSync(hold, { force: true }); + await fixture[Symbol.asyncDispose](); + } + }); + + it('lets a restart for another environment proceed without waiting for the work that continues', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(true); + const hold: string = path.join(fixture.folder, 'hold'); + try { + const before = await pongAsync(fixture); + await returnEarlyAsync(fixture); + + const mismatched: Promise = fixture.runAsync( + ['build', '--to', 'c', '--parallelism', '3'], + { + environment: { ...fixture.environment, RUSHD_RELOAD_TIER_TEST: 'changed' } + } + ); + const restart: ITerminalExchange | undefined = await Promise.race([ + mismatched, + delayAsync(20_000).then(() => undefined) + ]); + expect(fs.existsSync(hold)).toBe(true); + expect(restart?.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true } + }); + const restarted = await fixture.host.restartCompleted; + expect(restarted?.pid).not.toBe(before.pid); + } finally { + fs.rmSync(hold, { force: true }); + try { + await fixture.host.closeAsync(); + await fixture.host.restartCompleted; + } finally { + await stopSuccessorAsync(fixture.host.paths); + await fixture[Symbol.asyncDispose](); + } + } + }); +}); From 7945a0e6657105888e327ee0fd885988d0421591 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:01:59 +0000 Subject: [PATCH 049/265] [rush-daemon] A request that meets a removed or changed installation waits for the old daemon, then restarts it and runs Swarm integration step 34; original commit c0e17d82ab (merge of swarm/r03-t91f at 503c9e6589). Scope: tasks 125, 91 and 116. Brings r03's task 125 (E1 and E2 of t05 board 1726) with task 91 (a note restarts the 25 s status timer) and task 116 (long socket paths, board 1476), re-tipped onto integration 1b51e7af08 with ch01's resolution of the 108(a) conflicts in 2 files and one test mock line (board 2697, board 2732). When the daemon's installation is removed or changes, a request waits for the running requests to drain, the daemon restarts and the request runs, and the wait line names the installation as the reason; `--wait-timeout` and `--no-wait` fail with that reason. CONFIRMED by t05 on r03-t91d (board 2154, board 2197: the daemon side of cold E2 is 19 of 19 rc 0 where s13 fails 3 of 3). r05 reviewed the drain part of the re-tip (board 2680). ch01 on d9f459d151 (board 2756): build rc 0 with 0 warnings, rush-daemon 590/0, rush-cli-client 398/0, rush-client-core 136/0, rush-daemon-protocol 200/0, rush-daemon-transport 81/0. 33 new tests fail on the old product and 4 suites can't load. 14 mutants, 8 killed; the 6 survivors are test NITs. E2E: with the installation renamed away mid-build, the next build is rc 0 in 21.4 s where the base fails at once with `Cannot find module '@microsoft/rush-lib'`. r03's own run (board 2760): rush-cli-client 398/0, rush-daemon 590/0. Gate: ch01 GATE OK board 2756 (tree d9f459d151) Commits folded into this step (4): - 29b5678225 [rush-daemon] Refuse a socket path too long to reach, and keep older daemons in their folder (task 116, board 1476) - 08ed06de32 [rush-daemon] Restart a daemon whose installation was removed or replaced (task 91) - 1393c18030 [rush-daemon] Answer requests with the installation restart once the running requests finish (task 125) - c2218d0792 [rush-daemon] Wait for an installation restart in the restart drain, like a restart for an environment (task 125, aligned with task 83) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 40 +- .../src/AgentProgressRenderer.ts | 17 + apps/rush-cli-client/src/daemonCommands.ts | 39 +- apps/rush-cli-client/src/daemonGraph.ts | 7 +- .../src/daemonRestartNotice.ts | 120 +++++ apps/rush-cli-client/src/launchClient.ts | 17 +- .../src/test/AgentProgressRenderer.test.ts | 41 ++ .../test/daemonConnectionSelection.test.ts | 44 +- .../src/test/daemonRestartNotice.test.ts | 187 ++++++++ .../src/test/launchClient.test.ts | 22 + .../src/test/returnEarlyOnFailure.test.ts | 7 +- ...3-socket-path-length_2026-09-28-19-19.json | 11 + ...installation-changed_2026-09-28-18-45.json | 11 + ...llation-restart-wait_2026-09-28-20-30.json | 11 + ...3-socket-path-length_2026-09-28-19-19.json | 11 + ...installation-changed_2026-09-28-18-45.json | 11 + ...llation-restart-wait_2026-09-28-20-30.json | 11 + ...3-socket-path-length_2026-09-28-19-19.json | 11 + ...installation-changed_2026-09-28-18-45.json | 11 + ...llation-restart-wait_2026-09-28-20-30.json | 11 + ...3-socket-path-length_2026-09-28-19-19.json | 11 + ...installation-changed_2026-09-28-18-45.json | 11 + ...llation-restart-wait_2026-09-28-20-30.json | 11 + common/reviews/api/rush-client-core.api.md | 18 +- .../reviews/api/rush-daemon-protocol.api.md | 20 + .../reviews/api/rush-daemon-transport.api.md | 1 + common/reviews/api/rush-daemon.api.md | 11 +- libraries/rush-client-core/README.md | 17 +- .../rush-client-core/src/DaemonClient.ts | 15 +- .../src/DaemonRequestEnvironment.ts | 33 ++ .../src/DaemonRuntimeFolder.ts | 16 +- .../src/executeWithDaemonRestart.ts | 28 +- libraries/rush-client-core/src/index.ts | 6 +- .../src/test/DaemonClient.test.ts | 79 +++ .../src/test/connectOrStartDaemon.test.ts | 71 ++- .../src/test/fixtures/daemon.ts | 14 +- libraries/rush-daemon-protocol/README.md | 4 +- .../src/DaemonCommandResult.ts | 6 +- .../src/DaemonInstallationChange.ts | 40 ++ .../src/DaemonPongMessage.ts | 6 + .../src/DaemonPongValidation.ts | 2 + .../src/DaemonRequestAdmission.ts | 11 +- .../src/InstallationChangeValidation.ts | 51 ++ .../src/RequestAdmissionControlValidation.ts | 2 + .../src/RequestResultValidation.ts | 2 + libraries/rush-daemon-protocol/src/index.ts | 11 +- .../src/test/InstallationChange.test.ts | 72 +++ .../src/test/QueuedRestartReason.test.ts | 51 ++ libraries/rush-daemon-transport/README.md | 5 +- .../src/DaemonReclaim.ts | 5 +- .../src/DaemonRuntimeDir.ts | 9 +- .../src/DaemonSocketPathLength.ts | 61 +++ .../src/DaemonTransportError.ts | 4 +- .../src/test/ListenerSuccession.test.ts | 23 + .../src/test/SocketPathLength.test.ts | 80 ++++ .../src/test/SocketPathRefusal.test.ts | 54 +++ libraries/rush-daemon/README.md | 19 + .../rush-daemon/src/DaemonControlSession.ts | 17 + .../src/DaemonInstallationMonitor.ts | 102 ++++ .../rush-daemon/src/RushDaemonCommandLine.ts | 1 + libraries/rush-daemon/src/RushDaemonHost.ts | 31 +- .../src/SelectedDaemonBootstrap.ts | 3 + .../src/WorkspaceProcessRestart.ts | 12 +- .../src/WorkspaceRequestAdmission.ts | 58 ++- .../src/WorkspaceRequestLifecycle.ts | 178 ++++++- .../src/WorkspaceRestartArbiter.ts | 17 +- libraries/rush-daemon/src/index.ts | 1 + libraries/rush-daemon/src/serveRushDaemon.ts | 8 + .../src/test/DaemonGraphTestFixture.ts | 6 + .../src/test/DaemonInstallationChange.test.ts | 453 ++++++++++++++++++ .../src/test/RestartDrainAdmission.test.ts | 133 ++++- .../utilities/test/RushLibPathHandoff.test.ts | 40 +- 72 files changed, 2447 insertions(+), 133 deletions(-) create mode 100644 apps/rush-cli-client/src/daemonRestartNotice.ts create mode 100644 apps/rush-cli-client/src/test/daemonRestartNotice.test.ts create mode 100644 common/changes/@microsoft/rush/r03-socket-path-length_2026-09-28-19-19.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-installation-changed_2026-09-28-18-45.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-installation-restart-wait_2026-09-28-20-30.json create mode 100644 common/changes/@rushstack/rush-cli-client/r03-socket-path-length_2026-09-28-19-19.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-installation-changed_2026-09-28-18-45.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-installation-restart-wait_2026-09-28-20-30.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-socket-path-length_2026-09-28-19-19.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/r03-installation-changed_2026-09-28-18-45.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/r03-installation-restart-wait_2026-09-28-20-30.json create mode 100644 common/changes/@rushstack/rush-daemon-transport/r03-socket-path-length_2026-09-28-19-19.json create mode 100644 common/changes/@rushstack/rush-daemon/r03-installation-changed_2026-09-28-18-45.json create mode 100644 common/changes/@rushstack/rush-daemon/r03-installation-restart-wait_2026-09-28-20-30.json create mode 100644 libraries/rush-client-core/src/DaemonRequestEnvironment.ts create mode 100644 libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts create mode 100644 libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts create mode 100644 libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts create mode 100644 libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts create mode 100644 libraries/rush-daemon-transport/src/DaemonSocketPathLength.ts create mode 100644 libraries/rush-daemon-transport/src/test/SocketPathLength.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/SocketPathRefusal.test.ts create mode 100644 libraries/rush-daemon/src/DaemonInstallationMonitor.ts create mode 100644 libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index ccb39a9801..e83a16ca03 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -259,6 +259,23 @@ only an unstarted request can receive the typed retry authorization. Accepted queued requests drain their typed restart results before the old connection closes. +When the daemon's own installation was removed or replaced (for example a deleted +snapshot folder or a reinstalled Rush release), the daemon lets its running requests +finish, answers each other request with that typed restart once they have, and then +exits. While a command waits, the agent progress status (or stderr: on a terminal at +each position, on a pipe once) says why: +`rush-client: waiting for the running requests to finish (position 1); the daemon +(PID ) then restarts, because its installation at was removed.` The +timeout rules of a restart for the request's environment apply (see above): the +built-in default does not limit waiting for the requests that were running when the +command arrived, but still limits it while the daemon runs a `rushx` script, and +`--no-wait` and an explicit `--wait-timeout` limit the whole wait. A command that +times out exits with code 1, names the changed folder and, if a script runs, suggests +stopping it. Otherwise the client starts a daemon from its own launcher once the wait +ends, resubmits the request, and prints one line on stderr (or above the agent +progress rows): `rush-client: The daemon's installation at was removed; +restarted the daemon (PID ).` + When the connection is lost before a command's result, the command fails with exit code 1 and is not retried. The diagnostic keeps "Daemon disconnected before delivering a result; the command was not retried." and says what happened to rushd. If its process exited (a crash, an @@ -352,18 +369,28 @@ macOS, `/tmp/rushd-/`, whatever `TMPDIR` or `XDG_RUNTIME_DIR` a shell, job, sandbox sets. It holds the socket (`.sock`), the ownership record (`.pid.json`) and the launcher log. To move it, set `RUSHD_RUNTIME_DIR` to an absolute path for every client of that checkout; the folder becomes `$RUSHD_RUNTIME_DIR/rushd-/`, -and its file system must support hard links. A client passes the folder to the daemon it starts. +and its file system must support hard links. A relative `RUSHD_RUNTIME_DIR` is ignored. Clients +that disagree about `RUSHD_RUNTIME_DIR` use different folders, so each folder gets its own daemon +for the checkout. A client passes the folder to the daemon it starts. +The socket path must fit in a socket address: at most 108 bytes on Linux and 104 on macOS. +`RUSHD_RUNTIME_DIR` can therefore be at most 57 bytes on Linux and 53 on macOS, minus the +number of digits in your uid (50 bytes on Linux for uid 1234567). Windows uses the named pipe `\\.\pipe\rushd-` and is unchanged. The client refuses a runtime folder that is a symbolic link, is not a directory or belongs to -another user: commands run in-process with that reason, and `daemon` commands exit 1. Remove -the folder or set `RUSHD_RUNTIME_DIR`. A folder that others can open is made owner-only (`0700`). -`TMPDIR`, `TMP`, `TEMP`, `XDG_RUNTIME_DIR` and `RUSHD_RUNTIME_DIR` never select a different -daemon; each operation receives the requesting client's values. +another user, and a `RUSHD_RUNTIME_DIR` too long for the socket path: commands run in-process +with that reason, and `daemon` commands exit 1. Remove the folder or change `RUSHD_RUNTIME_DIR`. +A folder that others can open is made owner-only (`0700`). +Within one runtime folder, `TMPDIR`, `TMP`, `TEMP`, `XDG_RUNTIME_DIR` and `RUSHD_RUNTIME_DIR` +never select a different daemon; each operation receives the requesting client's values. Clients and daemons before protocol 0.12 used `$XDG_RUNTIME_DIR/rushd-/` or the temporary folder instead. A daemon started there stays there, where current clients do not look, until it idles out or is stopped with that older client (`rush-client daemon stop`). An older client that starts a current engine while `XDG_RUNTIME_DIR` or `TMPDIR` is set does not find it and runs in-process, so upgrade `rush-cli-client` with the engine. +When a daemon before protocol 0.12 serves a current client, for example one listening in +`/tmp/rushd-/`, the client leaves `XDG_RUNTIME_DIR`, `TMPDIR`, `TMP` and `TEMP` out of +its requests on Linux and macOS, because that daemon would restart into the folder they name. +Its operations see the daemon's own values; stop it (`rush-client daemon stop`) to use yours. `rush-client daemon status` only connects and checks hello/pong. It never starts a process, reclaims files, or treats a PID file as evidence of readiness. Both @@ -375,6 +402,9 @@ arguments, or startup failure returns exit code 1 with a diagnostic. When the en refuses connections and its ownership record (`.pid.json`) names a PID that no longer exists, the diagnostic adds that rushd exited without shutting down (an orderly shutdown removes the record) and that `daemon logs` may show why. +A daemon whose installation was removed or replaced still answers, but it restarts on +the next command: status then prints `state: "installationChanged"` with the pong's +`installationChange` (`change` and `folder`), a hint on stderr, and exits with code 1. A startup reservation (`.pid.json.starting`) refuses another daemon launch until the daemon it reserved becomes ready. Status reports one that remains as diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index 5313181609..faa7918fca 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -224,6 +224,23 @@ export class AgentProgressRenderer { } } + /** + * Writes one line. On a TTY, the live rows are redrawn below it; on a pipe, the next status line is then due 25 s + * later. + */ + public note(line: string): void { + if (this.#stopped) { + return; + } + if (this.#options.isTTY) { + this.#clear(); + this.#options.write(`${line}\n`); + this.#paint(); + } else { + this.#writePipeLine(line); + } + } + /** The request waits for admission. The status lines and the summary line say so. */ public onQueuePosition(position: number): void { this.#queued = { position, elapsed: this.#elapsed() }; diff --git a/apps/rush-cli-client/src/daemonCommands.ts b/apps/rush-cli-client/src/daemonCommands.ts index 6568b5e298..adea0f631c 100644 --- a/apps/rush-cli-client/src/daemonCommands.ts +++ b/apps/rush-cli-client/src/daemonCommands.ts @@ -21,7 +21,7 @@ import { type IDaemonLockfile, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; -import type { IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; +import type { IDaemonPongMessage, IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; import { printDaemonLogAsync } from './daemonLogs'; @@ -102,12 +102,7 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): // Nothing to shut down: restart behaves like start. const started: DaemonClient = await connectOrStartDaemonAsync(connectionOptions); try { - await writeStatusAsync({ - state: 'ready', - socketPath: connectionOptions.paths.socketPath, - ...(await started.status), - ...getStartupReservationStatus(connectionOptions.paths) - }); + await writeReadyStatusAsync(connectionOptions.paths, await started.status); } finally { await started.closeAsync(); } @@ -168,12 +163,7 @@ export async function executeDaemonCommandAsync(options: IDaemonCommandOptions): const readyClient: DaemonClient = command === 'restart' ? await restartDaemonAsync(client, connectionOptions) : client; try { - await writeStatusAsync({ - state: 'ready', - socketPath: connectionOptions.paths.socketPath, - ...(await readyClient.status), - ...getStartupReservationStatus(connectionOptions.paths) - }); + await writeReadyStatusAsync(connectionOptions.paths, await readyClient.status); } finally { if (readyClient !== client) await readyClient.closeAsync(); } @@ -270,3 +260,26 @@ async function restartDaemonAsync( function writeStatusAsync(status: object): Promise { return writeStreamAsync(process.stdout, Buffer.from(`${JSON.stringify(status)}\n`)); } + +/** A daemon whose installation was removed or replaced still answers, but restarts on its next request. */ +async function writeReadyStatusAsync( + paths: IDaemonPaths, + status: IDaemonPongMessage['payload'] +): Promise { + const change: IDaemonPongMessage['payload']['installationChange'] = status.installationChange; + await writeStatusAsync({ + state: change ? 'installationChanged' : 'ready', + socketPath: paths.socketPath, + ...status, + ...getStartupReservationStatus(paths) + }); + if (!change) return; + process.exitCode = 1; + await writeStreamAsync( + process.stderr, + Buffer.from( + `rush-client: The daemon's installation at ${change.folder} was ${change.change}. ` + + 'The next command restarts the daemon, or run "rush-client daemon restart".\n' + ) + ); +} diff --git a/apps/rush-cli-client/src/daemonGraph.ts b/apps/rush-cli-client/src/daemonGraph.ts index 4ff1947272..cbdd9bafac 100644 --- a/apps/rush-cli-client/src/daemonGraph.ts +++ b/apps/rush-cli-client/src/daemonGraph.ts @@ -116,8 +116,11 @@ async function executeGraphRequestAsync( sawSnapshot = true; await writeJsonAsync(event); }, - onQueuePositionAsync: (position) => - writeJsonAsync({ kind: 'queuePosition', payload: { requestId: request.requestId, position } }) + onQueuePositionAsync: (position, restartReason) => + writeJsonAsync({ + kind: 'queuePosition', + payload: { requestId: request.requestId, position, ...(restartReason && { restartReason }) } + }) }); if (outcome.kind === 'fallback') { throw new Error( diff --git a/apps/rush-cli-client/src/daemonRestartNotice.ts b/apps/rush-cli-client/src/daemonRestartNotice.ts new file mode 100644 index 0000000000..e5d3eafe93 --- /dev/null +++ b/apps/rush-cli-client/src/daemonRestartNotice.ts @@ -0,0 +1,120 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonRestartNotice } from '@rushstack/rush-client-core'; +import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; + +/** + * Returns the line that tells the user why the daemon restarted during a command, or `undefined` for a restart + * that needs no explanation. + */ +export function formatDaemonRestartNotice(notice: IDaemonRestartNotice, rushx: boolean): string | undefined { + const { reason, successorPid } = notice; + if (reason?.kind !== 'installationChanged') return undefined; + const pid: string = successorPid === undefined ? '' : ` (PID ${successorPid})`; + return ( + `${rushx ? 'rushx-client' : 'rush-client'}: The daemon's installation at ${reason.folder} was ` + + `${reason.change}; restarted the daemon${pid}.` + ); +} + +/** + * Where a restart line goes: the agent renderer when one is active, and otherwise stderr. + */ +export interface IDaemonRestartNoticeTarget { + readonly rushx: boolean; + readonly agentRenderer: { note(line: string): void } | undefined; + readonly writeStderrAsync: (text: string) => Promise; +} + +/** + * Creates the `onRestartAsync` callback that prints {@link formatDaemonRestartNotice}'s line. + */ +export function createDaemonRestartNoticeHandler( + target: IDaemonRestartNoticeTarget +): (notice: IDaemonRestartNotice) => Promise { + return async (notice: IDaemonRestartNotice): Promise => { + const line: string | undefined = formatDaemonRestartNotice(notice, target.rushx); + if (!line) return; + if (target.agentRenderer) target.agentRenderer.note(line); + else await target.writeStderrAsync(`${line}\n`); + }; +} + +/** The agent phase after a request that waited for a restart follows it to the new daemon. */ +export const RESUBMITTED_PHASE: string = + 'request resubmitted to the new daemon; preparing the workspace graph'; + +/** + * Returns what a queued request waits for when the daemon answers it with a restart once the requests ahead of it + * finish, or `undefined` for a queue position that needs no explanation. + */ +export function formatDaemonRestartWait( + position: number, + reason: DaemonRestartReason | undefined, + daemonPid: number | undefined +): string | undefined { + if (reason?.kind !== 'installationChanged') return undefined; + const pid: string = daemonPid === undefined ? '' : ` (PID ${daemonPid})`; + return ( + `waiting for the running requests to finish (position ${position}); the daemon${pid} then restarts, ` + + `because its installation at ${reason.folder} was ${reason.change}` + ); +} + +/** + * Where a request's queue positions and restart lines go. + */ +export interface IDaemonRequestNoticeTarget extends IDaemonRestartNoticeTarget { + readonly agentRenderer: + | { note(line: string): void; setPhase(phase: string): void; onQueuePosition(position: number): void } + | undefined; + /** On a pipe, only the first {@link formatDaemonRestartWait} line is written, and plain positions are not. */ + readonly stderrIsTTY: boolean; + /** The process ID of the daemon that serves the request first. */ + readonly daemonPid: number | undefined; +} + +/** + * The `onRestartAsync` and `onQueuePositionAsync` callbacks of one request. + */ +export interface IDaemonRequestNoticeHandlers { + readonly onRestartAsync: (notice: IDaemonRestartNotice) => Promise; + readonly onQueuePositionAsync: (position: number, restartReason?: DaemonRestartReason) => Promise; +} + +/** + * Creates the callbacks that tell the user why a request waits or restarted. Queue positions name the daemon that + * serves the request, which changes when the request follows a restart. + */ +export function createDaemonRequestNoticeHandlers( + target: IDaemonRequestNoticeTarget +): IDaemonRequestNoticeHandlers { + const { agentRenderer, stderrIsTTY } = target; + const writeRestartNoticeAsync: (notice: IDaemonRestartNotice) => Promise = + createDaemonRestartNoticeHandler(target); + let daemonPid: number | undefined = target.daemonPid; + let wroteRestartWait: boolean = false; + return { + onRestartAsync: async (notice: IDaemonRestartNotice): Promise => { + daemonPid = notice.successorPid; + await writeRestartNoticeAsync(notice); + // The phase still says that the request waits for the previous daemon. + if (agentRenderer && wroteRestartWait) agentRenderer.setPhase(RESUBMITTED_PHASE); + wroteRestartWait = false; + }, + onQueuePositionAsync: async (position: number, restartReason?: DaemonRestartReason): Promise => { + const wait: string | undefined = formatDaemonRestartWait(position, restartReason, daemonPid); + if (agentRenderer) { + wroteRestartWait ||= wait !== undefined; + if (wait) agentRenderer.setPhase(wait); + else agentRenderer.onQueuePosition(position); + } else if (wait && (stderrIsTTY || !wroteRestartWait)) { + wroteRestartWait = true; + await target.writeStderrAsync(`${target.rushx ? 'rushx-client' : 'rush-client'}: ${wait}.\n`); + } else if (!wait && stderrIsTTY) { + await target.writeStderrAsync(`rush-client: waiting for daemon admission (position ${position}).\n`); + } + } + }; +} diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 03fb79ca0c..9bd42f0ae0 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -37,6 +37,7 @@ import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; import { getResultStderr } from './resultDiagnostics'; import { getTerminalColumns } from './terminalColumns'; +import { createDaemonRequestNoticeHandlers } from './daemonRestartNotice'; import { writeStreamAsync } from './writeStreamAsync'; import { getBundledRushVersion, @@ -224,15 +225,13 @@ export async function launchClientAsync( }, onEventAsync: async (event) => agentRenderer ? agentRenderer.onEvent(event) : renderer.writeEventAsync(event), - onQueuePositionAsync: agentRenderer - ? async (position) => agentRenderer.onQueuePosition(position) - : process.stderr.isTTY - ? (position) => - writeStreamAsync( - process.stderr, - Buffer.from(`rush-client: waiting for daemon admission (position ${position}).\n`) - ) - : undefined, + ...createDaemonRequestNoticeHandlers({ + rushx, + agentRenderer, + stderrIsTTY: !!process.stderr.isTTY, + daemonPid: (await client.status).pid, + writeStderrAsync: (text) => writeStreamAsync(process.stderr, Buffer.from(text)) + }), stdin: process.stdin, requiresStdinEnd: !process.stdin.isTTY, cancelOnCtrlC: !!process.stdin.isTTY, diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts index d6f64f22ca..b8cfc5f1b9 100644 --- a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -952,6 +952,47 @@ describe(AgentProgressRenderer.name, () => { 'rush build 4/5 · 7.5s · running: b (build) · failed: q1 (build), q2 (build), q3 (build) +1 more' ]); }); + + it('writes a note like any other line, so the next status line is due 25 s after it', () => { + const { renderer, clock, lines } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + advance(clock, 20_000); + renderer.note('rush-client: restarted the daemon.'); + advance(clock, 24_999); + expect(lines()).toHaveLength(2); + advance(clock, 1); + renderer.dispose(); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + 'rush-client: restarted the daemon.', + 'rush build · 45.0s · sent to rushd; preparing the workspace graph' + ]); + }); + }); + + it('writes a note between progress lines on a pipe', () => { + const { renderer, output } = createRenderer(false); + renderer.start(); + renderer.onRequestSent(); + renderer.note('rush-client: restarted the daemon.'); + renderer.finish({ exitCode: 0 }); + renderer.note('after the summary'); + expect(output).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)\n', + 'rush-client: restarted the daemon.\n', + 'rush build: SUCCESS up to date (no operations needed) in 0.0s\n' + ]); + }); + + it('writes a note above the live rows on a TTY and redraws them below it', () => { + const { renderer, output } = createRenderer(true); + renderer.start(); + renderer.note('rush-client: restarted the daemon.'); + expect(output.slice(1, 3)).toEqual(['\x1b[3A\x1b[0J\x1b[?25h', 'rush-client: restarted the daemon.\n']); + expect(output[3].startsWith('\x1b[?25l')).toBe(true); + expect(output[3].replace(ANSI_ESCAPE, '')).toMatch(/^. rush build · 0\.0s · connecting/); + renderer.dispose(); }); it('renders at most three live rows on a TTY and clears them before the summary', () => { diff --git a/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts b/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts index 52e503a603..987c7bff2a 100644 --- a/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts +++ b/apps/rush-cli-client/src/test/daemonConnectionSelection.test.ts @@ -11,6 +11,19 @@ import { computeDaemonWorkspaceKey, resolveDaemonPathsFromProcess } from '@rushs import { getDaemonConnectionOptions, getDaemonConnectionOptionsAsync } from '../daemonConnectionOptions'; +function captureErrorWithRuntimeDir(base: string, action: () => void): unknown { + const previous: string | undefined = process.env.RUSHD_RUNTIME_DIR; + process.env.RUSHD_RUNTIME_DIR = base; + try { + action(); + } catch (error) { + return error; + } finally { + if (previous === undefined) delete process.env.RUSHD_RUNTIME_DIR; + else process.env.RUSHD_RUNTIME_DIR = previous; + } +} + describe('version-selected daemon connection options', () => { let repoRoot: string; @@ -55,23 +68,32 @@ describe('version-selected daemon connection options', () => { const base: string = path.join(repoRoot, 'runtime'); fs.mkdirSync(base); fs.symlinkSync(repoRoot, path.join(base, `rushd-${process.getuid?.()}`)); - const previous: string | undefined = process.env.RUSHD_RUNTIME_DIR; - process.env.RUSHD_RUNTIME_DIR = base; - let error: unknown; - try { - getDaemonConnectionOptions(repoRoot, Rush.version, process.env, false); - } catch (thrown) { - error = thrown; - } finally { - if (previous === undefined) delete process.env.RUSHD_RUNTIME_DIR; - else process.env.RUSHD_RUNTIME_DIR = previous; - } + const error: unknown = captureErrorWithRuntimeDir(base, () => + getDaemonConnectionOptions(repoRoot, Rush.version, process.env, false) + ); expect(error).toBeInstanceOf(DaemonClientError); expect(error).toMatchObject({ code: 'startupFailed' }); expect((error as Error).message).toContain('is unsafe: it is a symbolic link'); } ); + (process.platform === 'win32' ? it.skip : it)( + 'refuses a RUSHD_RUNTIME_DIR too long for a socket path before any daemon command uses it', + () => { + // Too long on every platform: the base alone is longer than a socket address allows. + const base: string = path.join(repoRoot, 'r'.repeat(80)); + const error: unknown = captureErrorWithRuntimeDir(base, () => + getDaemonConnectionOptions(repoRoot, Rush.version, process.env, true) + ); + expect(error).toBeInstanceOf(DaemonClientError); + expect(error).toMatchObject({ code: 'startupFailed' }); + expect((error as Error).message).toMatch( + /^The daemon socket path .*\.sock is \d+ bytes long, but this platform allows at most 10[48]\. Set RUSHD_RUNTIME_DIR to an absolute path of at most \d+ bytes, or unset it\.$/ + ); + expect(fs.existsSync(base)).toBe(false); + } + ); + it('does not select or install a launcher for a connect-only invocation', async () => { const options = await getDaemonConnectionOptionsAsync(repoRoot, '5.178.1', {}, false); expect(options.startCommand).toBeUndefined(); diff --git a/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts b/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts new file mode 100644 index 0000000000..1bed6ad88f --- /dev/null +++ b/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts @@ -0,0 +1,187 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; + +import { + RESUBMITTED_PHASE, + createDaemonRequestNoticeHandlers, + createDaemonRestartNoticeHandler, + formatDaemonRestartNotice, + formatDaemonRestartWait, + type IDaemonRequestNoticeHandlers +} from '../daemonRestartNotice'; + +// A kind that only a newer daemon knows. +const NEWER_REASON: DaemonRestartReason = { kind: 'environmentChanged' } as unknown as DaemonRestartReason; + +describe(formatDaemonRestartNotice.name, () => { + it('names the changed installation and the new daemon', () => { + expect( + formatDaemonRestartNotice( + { + restart: 1, + reason: { kind: 'installationChanged', change: 'replaced', folder: '/snapshots/s9' }, + successorPid: 42 + }, + false + ) + ).toBe( + "rush-client: The daemon's installation at /snapshots/s9 was replaced; restarted the daemon (PID 42)." + ); + expect( + formatDaemonRestartNotice( + { + restart: 1, + reason: { kind: 'installationChanged', change: 'removed', folder: '/snapshots/s9' }, + successorPid: undefined + }, + true + ) + ).toBe("rushx-client: The daemon's installation at /snapshots/s9 was removed; restarted the daemon."); + }); + + it('says nothing about restarts that need no explanation', () => { + expect( + formatDaemonRestartNotice({ restart: 1, reason: undefined, successorPid: 42 }, false) + ).toBeUndefined(); + expect( + formatDaemonRestartNotice({ restart: 1, reason: NEWER_REASON, successorPid: 42 }, false) + ).toBeUndefined(); + }); +}); + +describe(createDaemonRestartNoticeHandler.name, () => { + const notice = { + restart: 1, + reason: { kind: 'installationChanged', change: 'removed', folder: '/snapshots/s9' }, + successorPid: 42 + } as const; + const line: string = + "rush-client: The daemon's installation at /snapshots/s9 was removed; restarted the daemon (PID 42)."; + + it('gives the line to the agent renderer when one is active', async () => { + const notes: string[] = []; + const written: string[] = []; + const handler = createDaemonRestartNoticeHandler({ + rushx: false, + agentRenderer: { note: (text: string) => notes.push(text) }, + writeStderrAsync: async (text: string) => { + written.push(text); + } + }); + await handler(notice); + expect(notes).toEqual([line]); + expect(written).toEqual([]); + }); + + it('writes the line to stderr without an agent renderer, and stays quiet for other restarts', async () => { + const written: string[] = []; + const handler = createDaemonRestartNoticeHandler({ + rushx: false, + agentRenderer: undefined, + writeStderrAsync: async (text: string) => { + written.push(text); + } + }); + await handler(notice); + await handler({ restart: 2, reason: undefined, successorPid: 43 }); + expect(written).toEqual([`${line}\n`]); + }); +}); + +const REMOVED: DaemonRestartReason = { + kind: 'installationChanged', + change: 'removed', + folder: '/snapshots/s9' +}; + +describe(formatDaemonRestartWait.name, () => { + it('says what the request waits for, and why the daemon then restarts', () => { + expect(formatDaemonRestartWait(2, REMOVED, 41)).toBe( + 'waiting for the running requests to finish (position 2); the daemon (PID 41) then restarts, because ' + + 'its installation at /snapshots/s9 was removed' + ); + expect(formatDaemonRestartWait(1, { ...REMOVED, change: 'replaced' }, undefined)).toBe( + 'waiting for the running requests to finish (position 1); the daemon then restarts, because its ' + + 'installation at /snapshots/s9 was replaced' + ); + }); + + it('says nothing about plain queue positions or reasons that it does not know', () => { + expect(formatDaemonRestartWait(1, undefined, 41)).toBeUndefined(); + expect(formatDaemonRestartWait(1, NEWER_REASON, 41)).toBeUndefined(); + }); +}); + +describe(createDaemonRequestNoticeHandlers.name, () => { + function createHandlers(options: { agent: boolean; stderrIsTTY: boolean; rushx?: boolean }): { + calls: string[]; + handlers: IDaemonRequestNoticeHandlers; + } { + const calls: string[] = []; + const handlers: IDaemonRequestNoticeHandlers = createDaemonRequestNoticeHandlers({ + rushx: !!options.rushx, + stderrIsTTY: options.stderrIsTTY, + daemonPid: 41, + agentRenderer: options.agent + ? { + note: (line: string) => calls.push(`note: ${line}`), + setPhase: (phase: string) => calls.push(`phase: ${phase}`), + onQueuePosition: (position: number) => calls.push(`position: ${position}`) + } + : undefined, + writeStderrAsync: async (text: string) => { + calls.push(`stderr: ${text}`); + } + }); + return { calls, handlers }; + } + + it('shows a restart wait as the agent phase, and names the new daemon after the restart', async () => { + const { calls, handlers } = createHandlers({ agent: true, stderrIsTTY: false }); + await handlers.onQueuePositionAsync(2); + await handlers.onQueuePositionAsync(1, REMOVED); + await handlers.onRestartAsync({ restart: 1, reason: REMOVED, successorPid: 42 }); + await handlers.onQueuePositionAsync(1, REMOVED); + expect(calls).toEqual([ + 'position: 2', + `phase: ${formatDaemonRestartWait(1, REMOVED, 41)}`, + "note: rush-client: The daemon's installation at /snapshots/s9 was removed; restarted the daemon (PID 42).", + `phase: ${RESUBMITTED_PHASE}`, + `phase: ${formatDaemonRestartWait(1, REMOVED, 42)}` + ]); + }); + + it('keeps the agent phase after a restart that the request did not wait for', async () => { + const { calls, handlers } = createHandlers({ agent: true, stderrIsTTY: false }); + await handlers.onQueuePositionAsync(1); + await handlers.onRestartAsync({ restart: 1, reason: undefined, successorPid: 42 }); + expect(calls).toEqual(['position: 1']); + }); + + it('writes every queue position to a terminal', async () => { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: true }); + await handlers.onQueuePositionAsync(2); + await handlers.onQueuePositionAsync(2, REMOVED); + await handlers.onQueuePositionAsync(1, REMOVED); + expect(calls).toEqual([ + 'stderr: rush-client: waiting for daemon admission (position 2).\n', + `stderr: rush-client: ${formatDaemonRestartWait(2, REMOVED, 41)}.\n`, + `stderr: rush-client: ${formatDaemonRestartWait(1, REMOVED, 41)}.\n` + ]); + }); + + it('writes only the first restart wait for each daemon to a pipe', async () => { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: false, rushx: true }); + await handlers.onQueuePositionAsync(3); + await handlers.onQueuePositionAsync(2, REMOVED); + await handlers.onQueuePositionAsync(1, REMOVED); + await handlers.onRestartAsync({ restart: 1, reason: undefined, successorPid: 42 }); + await handlers.onQueuePositionAsync(1, REMOVED); + expect(calls).toEqual([ + `stderr: rushx-client: ${formatDaemonRestartWait(2, REMOVED, 41)}.\n`, + `stderr: rushx-client: ${formatDaemonRestartWait(1, REMOVED, 42)}.\n` + ]); + }); +}); diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index eadfc26bdc..0e52e5a8bc 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -186,6 +186,28 @@ describe('standalone rushx fallback', () => { } }, 15000); + it('status reports a daemon whose installation was removed, with rc 1', async () => { + const daemonPackage: { version: string } = require('@rushstack/rush-daemon/package.json'); + host = await RushDaemonHost.startAsync({ + repoRoot: folder, + rushVersion: Rush.version, + daemonVersion: daemonPackage.version, + checkInstallation: () => ({ change: 'removed', folder: '/snapshots/gone' }) + }); + const result: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'status']); + expect(result.code).toBe(1); + expect(JSON.parse(result.stdout)).toMatchObject({ + state: 'installationChanged', + socketPath: host.paths.socketPath, + pid: process.pid, + installationChange: { change: 'removed', folder: '/snapshots/gone' } + }); + expect(result.stderr).toBe( + "rush-client: The daemon's installation at /snapshots/gone was removed. " + + 'The next command restarts the daemon, or run "rush-client daemon restart".\n' + ); + }, 15000); + it('status never starts an absent daemon or falls back to command execution', async () => { const result: IInvocationResult = await invokeAsync(true, true, false, ['daemon', 'status']); expect(result.code).toBe(1); diff --git a/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts b/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts index 13d0a016b4..3cb0891805 100644 --- a/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts +++ b/apps/rush-cli-client/src/test/returnEarlyOnFailure.test.ts @@ -51,9 +51,10 @@ describe('returnEarlyOnFailure', () => { jest.spyOn(process.stderr, 'write').mockReturnValue(true); jest.spyOn(process.stdout, 'write').mockReturnValue(true); jest.spyOn(process, 'cwd').mockReturnValue(folder); - jest - .mocked(connectOrAwaitDaemonStartupAsync) - .mockResolvedValue({ closeAsync: async () => undefined } as unknown as DaemonClient); + jest.mocked(connectOrAwaitDaemonStartupAsync).mockResolvedValue({ + closeAsync: async () => undefined, + status: Promise.resolve({ pid: process.pid }) + } as unknown as DaemonClient); jest.mocked(executeWithDaemonRestartAsync).mockImplementation(async (client, connection, options) => { requests.push(options.request); return { diff --git a/common/changes/@microsoft/rush/r03-socket-path-length_2026-09-28-19-19.json b/common/changes/@microsoft/rush/r03-socket-path-length_2026-09-28-19-19.json new file mode 100644 index 0000000000..bba1834b76 --- /dev/null +++ b/common/changes/@microsoft/rush/r03-socket-path-length_2026-09-28-19-19.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-installation-changed_2026-09-28-18-45.json b/common/changes/@rushstack/rush-cli-client/r03-installation-changed_2026-09-28-18-45.json new file mode 100644 index 0000000000..48b6b0446e --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-installation-changed_2026-09-28-18-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Print one line when a command restarted the daemon because its installation was removed or replaced, and make `rush-client daemon status` report such a daemon as `installationChanged` with exit code 1.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-installation-restart-wait_2026-09-28-20-30.json b/common/changes/@rushstack/rush-cli-client/r03-installation-restart-wait_2026-09-28-20-30.json new file mode 100644 index 0000000000..1ff943b440 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-installation-restart-wait_2026-09-28-20-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "While a command waits for the running requests to finish before a daemon whose installation changed restarts, say so in the agent progress status or on stderr, and name the daemon's PID and the changed folder.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/r03-socket-path-length_2026-09-28-19-19.json b/common/changes/@rushstack/rush-cli-client/r03-socket-path-length_2026-09-28-19-19.json new file mode 100644 index 0000000000..7d1936bca0 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-socket-path-length_2026-09-28-19-19.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Refuse a `RUSHD_RUNTIME_DIR` too long for the daemon socket path: commands run in-process with the reason, and `daemon` commands exit 1.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-installation-changed_2026-09-28-18-45.json b/common/changes/@rushstack/rush-client-core/r03-installation-changed_2026-09-28-18-45.json new file mode 100644 index 0000000000..cdcd77cad1 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-installation-changed_2026-09-28-18-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Add an `onRestartAsync` callback to `executeWithDaemonRestartAsync` that reports each hand-off to a successor daemon, with the restart reason that the old daemon gave.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-installation-restart-wait_2026-09-28-20-30.json b/common/changes/@rushstack/rush-client-core/r03-installation-restart-wait_2026-09-28-20-30.json new file mode 100644 index 0000000000..869fcc8abd --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-installation-restart-wait_2026-09-28-20-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Pass a queue position's `restartReason` to `onQueuePositionAsync` as its second argument.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-socket-path-length_2026-09-28-19-19.json b/common/changes/@rushstack/rush-client-core/r03-socket-path-length_2026-09-28-19-19.json new file mode 100644 index 0000000000..69f5de0395 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-socket-path-length_2026-09-28-19-19.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Report a socket path too long for a socket address as `startupFailed`, before a daemon is started. On POSIX, leave `XDG_RUNTIME_DIR`, `TMPDIR`, `TMP` and `TEMP` out of requests to a daemon before protocol 0.12, which would otherwise restart into a runtime folder that current clients never look in.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/r03-installation-changed_2026-09-28-18-45.json b/common/changes/@rushstack/rush-daemon-protocol/r03-installation-changed_2026-09-28-18-45.json new file mode 100644 index 0000000000..02efe6110d --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/r03-installation-changed_2026-09-28-18-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add an optional `restartReason` to restart results and an optional `installationChange` to `pong`, so that a daemon can say that its installation was removed or replaced. Older peers ignore both.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/r03-installation-restart-wait_2026-09-28-20-30.json b/common/changes/@rushstack/rush-daemon-protocol/r03-installation-restart-wait_2026-09-28-20-30.json new file mode 100644 index 0000000000..551b74d262 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/r03-installation-restart-wait_2026-09-28-20-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add the optional `restartReason` to `queuePosition` control messages, for a request that the daemon answers with a restart once the requests ahead of it finish.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-transport/r03-socket-path-length_2026-09-28-19-19.json b/common/changes/@rushstack/rush-daemon-transport/r03-socket-path-length_2026-09-28-19-19.json new file mode 100644 index 0000000000..e6b41a1cbd --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-transport/r03-socket-path-length_2026-09-28-19-19.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-transport", + "comment": "Refuse a POSIX socket path longer than a socket address allows (108 bytes on Linux, 104 elsewhere) with the new `socketPathTooLong` error code, before the runtime folder is checked or created. Node.js would truncate it, so no client could reach the daemon.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-transport", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r03-installation-changed_2026-09-28-18-45.json b/common/changes/@rushstack/rush-daemon/r03-installation-changed_2026-09-28-18-45.json new file mode 100644 index 0000000000..334e482033 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r03-installation-changed_2026-09-28-18-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Restart instead of failing every request with `Cannot find module '@microsoft/rush-lib'` after the daemon's installation folder was removed or replaced: new and queued requests get a typed restart result that names the folder, running requests finish, and the daemon exits so that the client starts a new one. `pong` reports the change, the daemon resolves `_RUSH_LIB_PATH` once at startup, and the daemon log records every rejected request.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r03-installation-restart-wait_2026-09-28-20-30.json b/common/changes/@rushstack/rush-daemon/r03-installation-restart-wait_2026-09-28-20-30.json new file mode 100644 index 0000000000..5a524ab6a7 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r03-installation-restart-wait_2026-09-28-20-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "After its installation was removed or replaced, the daemon answers each other request with the typed restart result only once the requests that it serves finish (the restart drain), instead of at once, so that clients do not give up waiting for it to exit while a long build still runs. As for a restart for an environment, a client-default wait timeout does not limit waiting for the requests that were already being served unless a rushx script runs, queue positions carry the restart reason, and timeouts name the changed folder. A request that fails before it begins while the installation is changed gets the restart result instead of `routingFailed`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index c28959d376..6955908922 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -4,6 +4,7 @@ ```ts +import { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; import { IDaemonClientCaps } from '@rushstack/rush-daemon-protocol'; import { IDaemonCommandResult } from '@rushstack/rush-daemon-protocol'; import { IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; @@ -72,7 +73,7 @@ export class DaemonStartupPendingError extends Error { } // @beta -export function executeWithDaemonRestartAsync(client: DaemonClient, connection: IConnectOrStartDaemonOptions, execution: IDaemonClientExecuteOptions): Promise; +export function executeWithDaemonRestartAsync(client: DaemonClient, connection: IConnectOrStartDaemonOptions, options: IExecuteWithDaemonRestartOptions): Promise; // @beta export function getDaemonLogFilePath(paths: IDaemonPaths): string; @@ -132,8 +133,7 @@ export interface IDaemonClientExecuteOptions { readonly initialRawMode?: boolean; // (undocumented) readonly onEventAsync?: (event: IDaemonEventEnvelope) => Promise; - // (undocumented) - readonly onQueuePositionAsync?: (position: number) => Promise; + readonly onQueuePositionAsync?: (position: number, restartReason?: DaemonRestartReason) => Promise; // (undocumented) readonly onStderrAsync?: (bytes: Uint8Array, operationId: string) => Promise; // (undocumented) @@ -146,6 +146,13 @@ export interface IDaemonClientExecuteOptions { readonly stdin?: Readable; } +// @beta +export interface IDaemonRestartNotice { + readonly reason: DaemonRestartReason | undefined; + readonly restart: number; + readonly successorPid: number | undefined; +} + // @beta export interface IDaemonStartCommand { // (undocumented) @@ -165,6 +172,11 @@ export interface IDaemonStartupReservationInfo { readonly path: string; } +// @beta +export interface IExecuteWithDaemonRestartOptions extends IDaemonClientExecuteOptions { + readonly onRestartAsync?: (notice: IDaemonRestartNotice) => Promise; +} + // @beta export function inspectDaemonStartupReservation(paths: IDaemonPaths): IDaemonStartupReservationInfo | undefined; diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index da5aff785b..3e1b962294 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -137,6 +137,9 @@ export type DaemonHandshakeOutcome = { readonly error: ProtocolVersionMismatchError; }; +// @beta +export type DaemonInstallationChangeKind = 'removed' | 'replaced'; + // @beta export type DaemonInvocationKind = 'rush' | 'rushx'; @@ -166,6 +169,9 @@ export type DaemonRequestAdmissionErrorCode = 'aborted' | 'no-wait' | 'wait-time // @beta export type DaemonRequestRejectionCode = 'invalidRequest' | 'routingFailed' | 'unsupported' | 'workspaceRecreationRequired'; +// @beta +export type DaemonRestartReason = IDaemonInstallationChangedRestartReason; + // @beta export type DaemonRushCommandOrigin = 'built-in' | 'custom'; @@ -244,6 +250,7 @@ export interface IDaemonCommandResult { readonly exitCode: number; readonly outcome: DaemonCommandOutcome; readonly requestId: string; + readonly restartReason?: DaemonRestartReason; readonly retryAfterRestart?: true; } @@ -389,6 +396,17 @@ export interface IDaemonInitializedGraphSnapshot { readonly workspaceGeneration?: string; } +// @beta +export interface IDaemonInstallationChange { + readonly change: DaemonInstallationChangeKind; + readonly folder: string; +} + +// @beta +export interface IDaemonInstallationChangedRestartReason extends IDaemonInstallationChange { + readonly kind: 'installationChanged'; +} + // @beta export interface IDaemonLogChunk { readonly chunk: Uint8Array; @@ -480,6 +498,7 @@ export interface IDaemonPongMessage { readonly pid?: number; readonly residentMemoryBytes?: number; readonly workspace?: IDaemonWorkspaceStatus; + readonly installationChange?: IDaemonInstallationChange; readonly uptimeMs: number; }; } @@ -546,6 +565,7 @@ export interface IDaemonRequestQueuePositionMessage { readonly payload: { readonly position: number; readonly requestId: string; + readonly restartReason?: DaemonRestartReason; }; } diff --git a/common/reviews/api/rush-daemon-transport.api.md b/common/reviews/api/rush-daemon-transport.api.md index df4aaef215..fa87bdd96d 100644 --- a/common/reviews/api/rush-daemon-transport.api.md +++ b/common/reviews/api/rush-daemon-transport.api.md @@ -59,6 +59,7 @@ export enum DaemonTransportErrorCode { connectionRefused = "connectionRefused", connectionTimeout = "connectionTimeout", daemonAlreadyRunning = "daemonAlreadyRunning", + socketPathTooLong = "socketPathTooLong", transportClosed = "transportClosed", unsafeRuntimeDirectory = "unsafeRuntimeDirectory" } diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index 1bc9b6fdba..3dd40483e3 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -15,6 +15,7 @@ import type { GetInputsSnapshotAsyncFn } from '@microsoft/rush-lib'; import type { IDaemonCommandResult } from '@rushstack/rush-daemon-protocol'; import { IDaemonConfigurationJson } from '@microsoft/rush-lib'; import type { IDaemonEventEnvelope } from '@rushstack/rush-daemon-protocol'; +import type { IDaemonInstallationChange } from '@rushstack/rush-daemon-protocol'; import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; import type { IDaemonPhasedRequest } from '@rushstack/rush-daemon-protocol'; import type { IDaemonPhasedRequestResult } from '@rushstack/rush-daemon-protocol'; @@ -36,6 +37,12 @@ import { RushConfiguration } from '@microsoft/rush-lib'; import type { RushConfigurationProject } from '@microsoft/rush-lib'; import type { RushSession } from '@microsoft/rush-lib'; +// @beta +export function captureDaemonInstallation(folders: ReadonlyArray): CheckDaemonInstallation; + +// @beta +export type CheckDaemonInstallation = () => IDaemonInstallationChange | undefined; + // @beta export type CreateWorkspaceEngineComponentsAsync = (options: ICreateWorkspaceEngineComponentsOptions) => Promise; @@ -465,12 +472,14 @@ export interface IResolveGlobalCommandRequestOptions { // @beta export interface IRushDaemonHostOptions { + readonly checkInstallation?: CheckDaemonInstallation; readonly createWorkspaceSessionAsync?: WorkspaceSessionFactory; readonly daemonVersion: string; readonly getSuccessorLaunchAsync?: GetWorkspaceSuccessorLaunchAsync; readonly idleTimeoutSeconds?: number; readonly onError?: (error: Error) => void; readonly onInteractiveConnection?: (connection: IDaemonInteractiveConnection) => void; + readonly onLog?: (message: string) => void; readonly repoRoot: string; readonly requestResolver?: IDaemonRequestResolver; readonly rushVersion: string; @@ -556,7 +565,7 @@ export interface IWorkspaceProcessRestartContext { // (undocumented) readonly environment: Readonly>; // (undocumented) - readonly reason: 'hard-input-change' | 'native-mutation'; + readonly reason: 'hard-input-change' | 'native-mutation' | 'installation-changed'; // (undocumented) readonly repoRoot: string; // (undocumented) diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 1af7286b24..41581c27bd 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -12,7 +12,9 @@ This package has no capabilities and admission settings before startup. `DaemonClient.connectAsync()` negotiates hello, subscribes capabilities, and awaits a matching pong. `executeAsync()` uses one fresh connection per invocation and closes it after the -authoritative result. Async stdout/stderr/event callbacks are awaited in wire order, +authoritative result. On POSIX, a request to a peer before 0.12 leaves out `XDG_RUNTIME_DIR`, +`TMPDIR`, `TMP` and `TEMP`, because such a daemon restarts into the runtime folder they name, +where current clients never look; its operations see the daemon's own values. Async stdout/stderr/event callbacks are awaited in wire order, so slow destinations backpressure the transport. Log callbacks receive raw bytes and the protocol's operation ID. The calling client owns terminal presentation. @@ -44,6 +46,14 @@ error messages. When the retry bound or the admission deadline is exhausted, it returns a `restartRetriesExhausted` fallback outcome so the caller can run in-process. Cancellation stops waiting without killing a daemon. Disabling auto-start still permits waiting for a host-started successor, but never lets the client spawn one. +A result may say why the daemon restarts (`restartReason`); for `installationChanged`, +the daemon's installation was removed or replaced, so it exits without a successor and +the client's own `startCommand` starts one. Such a daemon answers only once the requests +ahead of the request finish; meanwhile `onQueuePositionAsync` gets the reason as its second +argument. After each hand-off to a ready successor, +the optional `onRestartAsync` callback gets the restart number, the reason (`undefined` +when the daemon gave none, as older daemons do) and the successor's PID, before the +request is resubmitted. A connection lost before the result stays a `disconnected` `DaemonClientError`. Its message starts with "Daemon disconnected before delivering a result; the command was not retried." @@ -60,8 +70,9 @@ signal fires, the error is unchanged, so the caller reports the cancellation. arguments, environment and cwd. It does not discover or install a Rush version. It adds `RUSHD_RUNTIME_DIR`, set to the base of `paths.runtimeDir`, to that environment, so the daemon resolves the same paths as its clients whatever environment it inherits. -A runtime folder that is a symbolic link, is not a directory or belongs to another user is -refused as `startupFailed` before anything in it is trusted +A runtime folder that is a symbolic link, is not a directory or belongs to another user, or +whose socket path is too long for a socket address (from a long `RUSHD_RUNTIME_DIR`), is +refused as `startupFailed` before anything in it is trusted or created (`assertDaemonRuntimeFolderIsPrivate()`); one that others can open is made owner-only. It reuses transport paths/reclaim checks and node-core-library's process-identity aware `LockFile` for the first-start mutex, including kernel-enforced exclusive diff --git a/libraries/rush-client-core/src/DaemonClient.ts b/libraries/rush-client-core/src/DaemonClient.ts index 2e7b2831c4..2bf2a57f1b 100644 --- a/libraries/rush-client-core/src/DaemonClient.ts +++ b/libraries/rush-client-core/src/DaemonClient.ts @@ -18,6 +18,7 @@ import { encodeDaemonControlMessage, encodeDaemonStdinChunk, type DaemonControlMessage, + type DaemonRestartReason, type IDaemonClientCaps, type IDaemonCommandResult, type IDaemonEventEnvelope, @@ -31,6 +32,7 @@ import { import { connectDaemonAsync, type DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from './DaemonClientError'; +import { adaptDaemonRequestToPeer } from './DaemonRequestEnvironment'; const MAX_STDIN_CHUNK_BYTES: number = 64 * 1024; @@ -49,7 +51,11 @@ export interface IDaemonClientExecuteOptions { readonly onStdoutAsync?: (bytes: Uint8Array, operationId: string) => Promise; readonly onStderrAsync?: (bytes: Uint8Array, operationId: string) => Promise; readonly onEventAsync?: (event: IDaemonEventEnvelope) => Promise; - readonly onQueuePositionAsync?: (position: number) => Promise; + /** + * Called with the request's one-based queue position whenever it changes. `restartReason` is set when the daemon + * answers the request with a restart result for that reason once the requests ahead of it finish. + */ + readonly onQueuePositionAsync?: (position: number, restartReason?: DaemonRestartReason) => Promise; readonly abortSignal?: AbortSignal; /** Protocol 0.7 input waits for stdinReady credits; older peers use the legacy raw-mode/terminal policy. */ readonly stdin?: Readable; @@ -273,7 +279,10 @@ export class DaemonClient { } this.#result = deferred(); await Promise.all([ - this.#sendControlAsync({ kind: 'requestStart', payload: options.request }), + this.#sendControlAsync({ + kind: 'requestStart', + payload: adaptDaemonRequestToPeer(options.request, this.protocolVersion, process.platform) + }), this.#result.promise ]); return await this.#result.promise; @@ -439,7 +448,7 @@ export class DaemonClient { } return; case 'queuePosition': - await execution.onQueuePositionAsync?.(message.payload.position); + await execution.onQueuePositionAsync?.(message.payload.position, message.payload.restartReason); return; default: throw new DaemonProtocolError( diff --git a/libraries/rush-client-core/src/DaemonRequestEnvironment.ts b/libraries/rush-client-core/src/DaemonRequestEnvironment.ts new file mode 100644 index 0000000000..dee83b52bc --- /dev/null +++ b/libraries/rush-client-core/src/DaemonRequestEnvironment.ts @@ -0,0 +1,33 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { + DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR, + type IDaemonProtocolVersion, + type IDaemonRequestEnvelope +} from '@rushstack/rush-daemon-protocol'; + +// Before protocol 0.12, a daemon listened in XDG_RUNTIME_DIR or else in os.tmpdir(), which reads TMPDIR, then +// TMP, then TEMP. It restarted when a request's XDG_RUNTIME_DIR or TMPDIR differed from its own, and its +// successor listened in the folder that the request named. +const RUNTIME_FOLDER_VARIABLES: readonly string[] = ['XDG_RUNTIME_DIR', 'TMPDIR', 'TMP', 'TEMP']; + +/** + * Returns the request to send to a daemon that negotiated `peer`. + * + * @remarks + * A daemon older than protocol 0.12 would restart into a folder that current clients never look in when a + * request names another runtime folder, and every later client of the checkout would then wait for a daemon it + * cannot find. On POSIX, its requests leave those variables out, so that it serves them with its own values. + * Windows is unchanged: its pipe name does not depend on them, and its builds need `TMP` and `TEMP`. + */ +export function adaptDaemonRequestToPeer( + request: IDaemonRequestEnvelope, + peer: IDaemonProtocolVersion, + platform: NodeJS.Platform +): IDaemonRequestEnvelope { + if (platform === 'win32' || peer.minor >= DAEMON_RUNTIME_FOLDER_PROTOCOL_MINOR) return request; + const environment: Record = { ...request.environment }; + for (const name of RUNTIME_FOLDER_VARIABLES) delete environment[name]; + return { ...request, environment }; +} diff --git a/libraries/rush-client-core/src/DaemonRuntimeFolder.ts b/libraries/rush-client-core/src/DaemonRuntimeFolder.ts index 3d5694fc55..d7583e70c3 100644 --- a/libraries/rush-client-core/src/DaemonRuntimeFolder.ts +++ b/libraries/rush-client-core/src/DaemonRuntimeFolder.ts @@ -14,19 +14,25 @@ import { import { DaemonClientError } from './DaemonClientError'; +const RUNTIME_FOLDER_ERROR_CODES: ReadonlySet = new Set([ + DaemonTransportErrorCode.unsafeRuntimeDirectory, + DaemonTransportErrorCode.socketPathTooLong +]); + function toClientError(error: unknown): unknown { - return error instanceof DaemonTransportError && - error.code === DaemonTransportErrorCode.unsafeRuntimeDirectory + return error instanceof DaemonTransportError && RUNTIME_FOLDER_ERROR_CODES.has(error.code) ? new DaemonClientError('startupFailed', error.message, { cause: error }) : error; } /** * Checks the daemon runtime folder, when it exists, before a client trusts the socket, lockfile or log inside - * it (see `assertDaemonRuntimeDirIsPrivate` in `@rushstack/rush-daemon-transport`). + * it (see `assertDaemonRuntimeDirIsPrivate` in `@rushstack/rush-daemon-transport`). This includes a socket + * path too long to connect to, which only a long `RUSHD_RUNTIME_DIR` produces. * - * @throws {@link DaemonClientError} with code `startupFailed` when the folder is unsafe, so that a caller that - * falls back to in-process Rush for startup failures also does so here. + * @throws {@link DaemonClientError} with code `startupFailed` when the folder is unsafe or the socket path is + * too long, so that a caller that falls back to in-process Rush for startup failures also does so here, before + * it starts a daemon that no client could reach. * * @beta */ diff --git a/libraries/rush-client-core/src/executeWithDaemonRestart.ts b/libraries/rush-client-core/src/executeWithDaemonRestart.ts index ed6124beb7..9d5fc8d4a2 100644 --- a/libraries/rush-client-core/src/executeWithDaemonRestart.ts +++ b/libraries/rush-client-core/src/executeWithDaemonRestart.ts @@ -5,6 +5,7 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; import { DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR, + type DaemonRestartReason, type IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; import { readDaemonLockfile, type IDaemonLockfile } from '@rushstack/rush-daemon-transport'; @@ -29,6 +30,28 @@ const RETRY_JITTER_MAX_MS: number = 1000; /** Matches the default of {@link IConnectOrStartDaemonOptions.startupTimeoutMs}. */ const DEFAULT_STARTUP_TIMEOUT_MS: number = 15000; +/** + * A daemon restart that a request followed. + * @beta + */ +export interface IDaemonRestartNotice { + /** 1 for the first restart that the request followed. */ + readonly restart: number; + /** Why the previous daemon asked for the restart; `undefined` when it did not say. */ + readonly reason: DaemonRestartReason | undefined; + /** The process ID of the daemon that the request is sent to next. */ + readonly successorPid: number | undefined; +} + +/** + * Options for {@link executeWithDaemonRestartAsync}. + * @beta + */ +export interface IExecuteWithDaemonRestartOptions extends IDaemonClientExecuteOptions { + /** Called after a successor daemon is ready, before the request is sent to it. */ + readonly onRestartAsync?: (notice: IDaemonRestartNotice) => Promise; +} + /** * Executes on a ready client, retrying only for a typed pre-execution restart. * Preserves the original request, callbacks and unread input; never retries connection loss. @@ -43,8 +66,9 @@ const DEFAULT_STARTUP_TIMEOUT_MS: number = 15000; export async function executeWithDaemonRestartAsync( client: DaemonClient, connection: IConnectOrStartDaemonOptions, - execution: IDaemonClientExecuteOptions + options: IExecuteWithDaemonRestartOptions ): Promise { + const { onRestartAsync, ...execution } = options; const startedAt: number = Date.now(); const abortSignal: AbortSignal | undefined = execution.abortSignal && connection.abortSignal @@ -65,6 +89,7 @@ export async function executeWithDaemonRestartAsync( let previous: DaemonClient | undefined; try { for (let retry: number = 1; outcome.kind === 'result' && outcome.result.retryAfterRestart; retry++) { + const reason: DaemonRestartReason | undefined = outcome.result.restartReason; if (!owner) { throw new DaemonClientError( 'startupFailed', @@ -118,6 +143,7 @@ export async function executeWithDaemonRestartAsync( const remainingMs: number | undefined = getRemainingMs(); if (isExpired(remainingMs)) return restartExhaustedOutcome(retry); owner = await attestOwnerAsync(successor, connection); + await onRestartAsync?.({ restart: retry, reason, successorPid: (await successor.status).pid }); outcome = await executeOnDaemonAsync(successor, connection, { ...execution, abortSignal, diff --git a/libraries/rush-client-core/src/index.ts b/libraries/rush-client-core/src/index.ts index c1ae540535..32bc6691f2 100644 --- a/libraries/rush-client-core/src/index.ts +++ b/libraries/rush-client-core/src/index.ts @@ -26,7 +26,11 @@ export { type DaemonStartupHelperState, type IDaemonStartupReservationInfo } from './DaemonStartupReservation'; -export { executeWithDaemonRestartAsync } from './executeWithDaemonRestart'; +export { + executeWithDaemonRestartAsync, + type IDaemonRestartNotice, + type IExecuteWithDaemonRestartOptions +} from './executeWithDaemonRestart'; export { connectOrAwaitDaemonStartupAsync, DaemonStartupPendingError, diff --git a/libraries/rush-client-core/src/test/DaemonClient.test.ts b/libraries/rush-client-core/src/test/DaemonClient.test.ts index 9a2ccc6af8..5498a3d4ea 100644 --- a/libraries/rush-client-core/src/test/DaemonClient.test.ts +++ b/libraries/rush-client-core/src/test/DaemonClient.test.ts @@ -18,6 +18,7 @@ import { import { DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; import { DaemonClient } from '../DaemonClient'; +import { adaptDaemonRequestToPeer } from '../DaemonRequestEnvironment'; import { captureDaemonRequest } from '../captureDaemonRequest'; describe('DaemonClient', () => { @@ -187,6 +188,46 @@ describe('DaemonClient', () => { expect(stdin.read().toString()).toBe('untouched'); }); + const RUNTIME_FOLDER_ENVIRONMENT: Readonly> = { + TEST: 'one', + XDG_RUNTIME_DIR: '/run/user/1000', + TMPDIR: '/scratch/tmp', + TMP: '/scratch/tmp', + TEMP: '/scratch/tmp' + }; + + (process.platform === 'win32' ? it.skip : it).each([ + [11, ['TEST']], + [12, ['TEMP', 'TEST', 'TMP', 'TMPDIR', 'XDG_RUNTIME_DIR']] + ])('sends a protocol 0.%s daemon only the runtime folder variables it can use', async (minor, names) => { + peerVersion = { major: 0, minor: Number(minor) }; + onRequest = async (message) => { + if (message.kind !== 'requestStart') return; + await sendAsync({ + kind: 'requestResult', + payload: { requestId: message.payload.requestId, exitCode: 0, outcome: 'success', aborted: false } + }); + }; + const client = await DaemonClient.connectAsync({ socketPath: address }); + const envelope = captureDaemonRequest({ ...request(), environment: RUNTIME_FOLDER_ENVIRONMENT }); + expect(await client.executeAsync({ request: envelope })).toMatchObject({ kind: 'result' }); + const start: DaemonControlMessage | undefined = controls.find( + (message) => message.kind === 'requestStart' + ); + expect(Object.keys(start?.kind === 'requestStart' ? start.payload.environment : {}).sort()).toEqual( + names + ); + expect(envelope.environment).toEqual(RUNTIME_FOLDER_ENVIRONMENT); + }); + + it('keeps the runtime folder variables for an older daemon on Windows', () => { + const envelope = captureDaemonRequest({ ...request(), environment: RUNTIME_FOLDER_ENVIRONMENT }); + expect(adaptDaemonRequestToPeer(envelope, { major: 0, minor: 11 }, 'win32')).toBe(envelope); + expect(adaptDaemonRequestToPeer(envelope, { major: 0, minor: 11 }, 'darwin').environment).toEqual({ + TEST: 'one' + }); + }); + it.each(['output', 'event', 'stdin', 'raw-mode', 'old-peer'])( 'does not authorize restart replay after %s', async (mode) => { @@ -288,6 +329,44 @@ describe('DaemonClient', () => { expect(controls.filter((message) => message.kind === 'requestCancel')).toHaveLength(1); }); + it('reports why a queued request waits when the daemon restarts after the requests ahead of it', async () => { + const restartReason = { + kind: 'installationChanged', + change: 'removed', + folder: '/snapshots/s9' + } as const; + onRequest = async (message) => { + if (message.kind !== 'requestStart') return; + const { requestId } = message.payload; + await sendAsync({ kind: 'queuePosition', payload: { position: 2, requestId } }); + await sendAsync({ kind: 'queuePosition', payload: { position: 1, requestId, restartReason } }); + await sendAsync({ + kind: 'requestResult', + payload: { + requestId, + exitCode: 1, + outcome: 'failure', + aborted: false, + retryAfterRestart: true, + restartReason + } + }); + }; + const positions: unknown[] = []; + const client = await DaemonClient.connectAsync({ socketPath: address }); + const outcome = await client.executeAsync({ + request: request(), + onQueuePositionAsync: async (position, reason) => { + positions.push([position, reason]); + } + }); + expect(positions).toEqual([ + [2, undefined], + [1, restartReason] + ]); + expect(outcome).toMatchObject({ kind: 'result', result: { retryAfterRestart: true, restartReason } }); + }); + it('forwards raw stdin only after acknowledgement and restores raw mode', async () => { const stdin = new PassThrough(); const rawModes: boolean[] = []; diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index 16f76b614f..dca59836f5 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -30,7 +30,7 @@ import { resolveDaemonStartupReservationAsync, type IConnectOrStartDaemonOptions } from '../connectOrStartDaemon'; -import { executeWithDaemonRestartAsync } from '../executeWithDaemonRestart'; +import { executeWithDaemonRestartAsync, type IDaemonRestartNotice } from '../executeWithDaemonRestart'; import { getDaemonStartupFilePath, releaseDaemonStartup, reserveDaemonStartup } from '../DaemonStartup'; import { inspectDaemonStartupReservation } from '../DaemonStartupReservation'; import { tryAcquireStartupLockAsync, type IStartupLock } from '../StartupLock'; @@ -777,6 +777,50 @@ describe('detached daemon startup', () => { 15000 ); + it.each([ + ['restart-once', false], + ['restart-installation', true] + ])('tells the caller about the restart that it followed for %s', async (mode, reported) => { + const connection: IConnectOrStartDaemonOptions = { + ...options, + startCommand: { ...options.startCommand!, args: [...options.startCommand!.args, 'fixture', mode] } + }; + const client = await connectOrStartDaemonAsync(connection); + const request = captureDaemonRequest({ + argv: ['test'], + commandName: 'test', + commandOrigin: 'custom', + cwd: folder, + environment: {}, + terminal: { isTTY: false, supportsColor: false } + }); + const notices: IDaemonRestartNotice[] = []; + const outcome = await executeWithDaemonRestartAsync(client, connection, { + request, + onRestartAsync: async (notice) => { + // The successor has not received the request yet. + expect(fs.readFileSync(path.join(folder, 'requests'), 'utf8').trim().split('\n')).toHaveLength(1); + notices.push(notice); + } + }); + expect(outcome).toMatchObject({ kind: 'result', result: { exitCode: 0 } }); + const starts: number[] = fs + .readFileSync(path.join(folder, 'starts'), 'utf8') + .trim() + .split('\n') + .map(Number); + expect(starts).toHaveLength(2); + expect(notices).toEqual([ + { + restart: 1, + reason: reported + ? { kind: 'installationChanged', change: 'removed', folder: path.join(folder, 'gone') } + : undefined, + successorPid: starts[1] + } + ]); + }); + it.each(['execution', 'connection'])( 'cancels successor waiting using the %s signal without replay', async (source) => { @@ -1246,6 +1290,31 @@ describe('detached daemon startup', () => { } ); + (process.platform === 'win32' ? it.skip : it)( + 'reports a socket path too long to connect to as a startup failure without starting a daemon', + async () => { + // Node cuts a socket path longer than sun_path (108 bytes on Linux, 104 on macOS) short, so no client + // could reach a daemon published there. + const base: string = path.join(folder, 'r'.repeat(100)); + const runtimeDir: string = path.join(base, `rushd-${process.getuid?.()}`); + const failure: Promise = connectOrStartDaemonAsync({ + ...options, + paths: { + runtimeDir, + socketPath: path.join(runtimeDir, 'd.sock'), + lockfilePath: path.join(runtimeDir, 'daemon.pid.json') + } + }); + await expect(failure).rejects.toBeInstanceOf(DaemonClientError); + await expect(failure).rejects.toMatchObject({ code: 'startupFailed' }); + await expect(failure).rejects.toThrow( + /^The daemon socket path .*\/d\.sock is \d+ bytes long, but this platform allows at most 10[48]\. Set RUSHD_RUNTIME_DIR to an absolute path of at most \d+ bytes, or unset it\.$/ + ); + expect(fs.existsSync(base)).toBe(false); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + } + ); + (process.platform === 'win32' ? it.skip : it)( 'refuses linked log destinations without changing their target', async () => { diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index ea78f0408b..faeac1e486 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -101,7 +101,8 @@ async function mainAsync(): Promise { : 0; const restart: boolean = restartMode !== undefined && - (restartMode !== 'restart-once' || restartCount < 1) && + ((restartMode !== 'restart-once' && restartMode !== 'restart-installation') || + restartCount < 1) && (restartMode !== 'restart-twice' || restartCount < 2); const drainMsPath: string = path.join(folder, 'drain-ms'); if (restart && fs.existsSync(drainMsPath)) { @@ -117,7 +118,16 @@ async function mainAsync(): Promise { exitCode: restart ? 1 : 0, outcome: restart ? 'failure' : 'success', aborted: false, - ...(restart ? { retryAfterRestart: true as const } : {}) + ...(restart ? { retryAfterRestart: true as const } : {}), + ...(restart && restartMode === 'restart-installation' + ? { + restartReason: { + kind: 'installationChanged' as const, + change: 'removed' as const, + folder: path.join(folder, 'gone') + } + } + : {}) } }) }); diff --git a/libraries/rush-daemon-protocol/README.md b/libraries/rush-daemon-protocol/README.md index f88dbe9636..d95633c5db 100644 --- a/libraries/rush-daemon-protocol/README.md +++ b/libraries/rush-daemon-protocol/README.md @@ -25,7 +25,9 @@ The engine-agnostic **wire layer** spoken by every client of the Rush daemon (`r - **Interactive request contracts** — request-tagged stdin frames preserve arbitrary bytes, while acknowledged raw-mode controls and typed terminal-policy results remain scoped to one request. - **Request admission contracts** — resolved no-wait and bounded-timeout options, typed admission - failure codes, and capability-gated one-based queue-position control messages. + failure codes, and capability-gated one-based queue-position control messages. A queue position + may carry the `restartReason` with which the daemon answers the request once the requests ahead + of it finish; older daemons omit it, and clients ignore reason kinds they do not know. - **Request lifecycle contracts** — a validated presentation-free command envelope, cancellation, typed routing rejection/fallback, and one authoritative terminal result control. Command parsing and Rush action construction remain outside the protocol. diff --git a/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts b/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts index 4a133c4e4b..b45ce88ca0 100644 --- a/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts +++ b/libraries/rush-daemon-protocol/src/DaemonCommandResult.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { DaemonRestartReason } from './DaemonInstallationChange'; import type { DaemonRequestAdmissionErrorCode } from './DaemonRequestAdmission'; /** @@ -17,13 +18,16 @@ export type DaemonCommandOutcome = 'success' | 'success-with-warning' | 'failure */ export interface IDaemonCommandResult { /** - * Protocol 0.10: no execution or request IO occurred, and a successor has been selected. + * Protocol 0.10: no execution or request IO occurred, and a successor has been selected. With a + * `restartReason` of `installationChanged`, the daemon exits without one instead, and the client starts it. * Retry only after attested predecessor ownership release, within the request's admission deadline and a * small client-defined retry bound; then fall back instead of retrying. Never infer this from an error. * The predecessor launches the selected successor itself after that release, and its process exits once the * launch settles: until then, a retrying client connects to the successor but must not start a daemon. */ readonly retryAfterRestart?: true; + /** Why the daemon restarts, when it says; only together with `retryAfterRestart`. Older daemons omit it. */ + readonly restartReason?: DaemonRestartReason; /** Whether cancellation or disconnect was observed, even if a cleanup failure determines the outcome. */ readonly aborted: boolean; /** The typed admission failure, when execution never started. */ diff --git a/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts b/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts new file mode 100644 index 0000000000..0c9112e204 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts @@ -0,0 +1,40 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * How a folder of a running daemon's installation changed after the daemon started. + * + * @beta + */ +export type DaemonInstallationChangeKind = 'removed' | 'replaced'; + +/** + * A folder of the daemon's own installation, or of the Rush engine that it loaded, that was removed or replaced + * after the daemon started. Such a daemon cannot load the rest of its code, so it restarts instead of serving. + * + * @beta + */ +export interface IDaemonInstallationChange { + /** `removed` when the folder no longer exists; `replaced` when a different folder now has its path. */ + readonly change: DaemonInstallationChangeKind; + /** The absolute path of the outermost folder that was removed or replaced. */ + readonly folder: string; +} + +/** + * The daemon's installation was removed or replaced. The daemon exits without selecting a successor, and the + * client starts one with its own current launcher. + * + * @beta + */ +export interface IDaemonInstallationChangedRestartReason extends IDaemonInstallationChange { + /** Identifies this reason. */ + readonly kind: 'installationChanged'; +} + +/** + * Why a daemon asked the client to retry after a restart. Clients ignore kinds that they do not know. + * + * @beta + */ +export type DaemonRestartReason = IDaemonInstallationChangedRestartReason; diff --git a/libraries/rush-daemon-protocol/src/DaemonPongMessage.ts b/libraries/rush-daemon-protocol/src/DaemonPongMessage.ts index e964c89de5..ddeda56bfc 100644 --- a/libraries/rush-daemon-protocol/src/DaemonPongMessage.ts +++ b/libraries/rush-daemon-protocol/src/DaemonPongMessage.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { IDaemonInstallationChange } from './DaemonInstallationChange'; import type { IDaemonProtocolVersion } from './DaemonProtocolVersion'; import type { IDaemonWorkspaceStatus } from './DaemonWorkspaceStatus'; @@ -18,6 +19,11 @@ export interface IDaemonPongMessage { readonly residentMemoryBytes?: number; /** Optional generation and warm accounting; older peers may omit this snapshot. */ readonly workspace?: IDaemonWorkspaceStatus; + /** + * Present when the daemon found its own installation removed or replaced after it started. Its next request + * restarts it. Older daemons omit it. + */ + readonly installationChange?: IDaemonInstallationChange; readonly uptimeMs: number; }; } diff --git a/libraries/rush-daemon-protocol/src/DaemonPongValidation.ts b/libraries/rush-daemon-protocol/src/DaemonPongValidation.ts index 54fc61970e..7862c87c80 100644 --- a/libraries/rush-daemon-protocol/src/DaemonPongValidation.ts +++ b/libraries/rush-daemon-protocol/src/DaemonPongValidation.ts @@ -3,6 +3,7 @@ import { isDaemonControlRecord } from './ControlRecord'; import { DaemonProtocolError } from './DaemonProtocolError'; +import { validateInstallationChange } from './InstallationChangeValidation'; import { validateWorkspaceStatus } from './WorkspaceStatusValidation'; const ZERO: number = 0; @@ -18,6 +19,7 @@ export function validateDaemonPong(payload: Record): void { validatePid(payload.pid); validateResidentMemory(payload.residentMemoryBytes); validateWorkspaceStatus(payload.workspace); + validateInstallationChange(payload.installationChange, 'pong field "installationChange"'); } function validateDaemonVersion(value: unknown): void { diff --git a/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts b/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts index 2320840043..5ec2f64646 100644 --- a/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts +++ b/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { DaemonRestartReason } from './DaemonInstallationChange'; + /** The largest wait timeout accepted by Node.js timers. @beta */ export const MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS: number = 0x7fffffff; const MINIMUM_WAIT_TIMEOUT_MS: number = 0; @@ -15,8 +17,8 @@ export interface IDaemonRequestAdmissionOptions { /** * True when `waitTimeoutMs` is a client default rather than an explicit user choice. A default timeout applies * to each daemon's workspace admission only: not to waiting behind running compatible shared builds, nor, when - * the daemon restarts for the request's environment, to waiting for the requests that it was already serving - * while it serves no rushx script. + * the daemon restarts for the request's environment or because its installation changed, to waiting for the + * requests that it was already serving while it serves no rushx script. */ readonly waitTimeoutIsDefault?: boolean; /** Maximum queue wait in milliseconds. Omission means no timeout. */ @@ -29,6 +31,11 @@ export interface IDaemonRequestQueuePositionMessage { readonly payload: { readonly position: number; readonly requestId: string; + /** + * Set while the daemon holds the request until the requests ahead of it finish, and then answers it with a + * restart result for this reason instead of running it. Older daemons omit it; clients ignore unknown kinds. + */ + readonly restartReason?: DaemonRestartReason; }; } diff --git a/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts b/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts new file mode 100644 index 0000000000..77884d9583 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts @@ -0,0 +1,51 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { isDaemonControlRecord } from './ControlRecord'; +import { DaemonProtocolError } from './DaemonProtocolError'; + +const INSTALLATION_CHANGE_KINDS: ReadonlySet = new Set(['removed', 'replaced']); +const INSTALLATION_CHANGED: string = 'installationChanged'; +const EMPTY_LENGTH: number = 0; + +/** Validates an optional installation change, as reported by pong or by a restart reason. @internal */ +export function validateInstallationChange(value: unknown, field: string): void { + if (value === undefined) return; + requireRecord(value, field); + if (!INSTALLATION_CHANGE_KINDS.has(value.change)) fail(`${field}.change`); + requireFolder(value.folder, `${field}.folder`); +} + +/** Validates an optional restart reason. Unknown kinds from newer daemons are accepted and ignored. @internal */ +export function validateRestartReason(payload: Record): void { + if (payload.restartReason === undefined) return; + if (payload.retryAfterRestart !== true) fail('restartReason without retryAfterRestart'); + validateReason(payload.restartReason); +} + +/** Validates the optional restart reason of a queue position, which needs no retry flag. @internal */ +export function validateQueuedRestartReason(payload: Record): void { + if (payload.restartReason !== undefined) validateReason(payload.restartReason); +} + +function validateReason(reason: unknown): void { + requireRecord(reason, 'restartReason'); + requireKind(reason.kind); + if (reason.kind === INSTALLATION_CHANGED) validateInstallationChange(reason, 'restartReason'); +} + +function requireRecord(value: unknown, field: string): asserts value is Record { + if (!isDaemonControlRecord(value) || Array.isArray(value)) fail(field); +} + +function requireKind(value: unknown): void { + if (typeof value !== 'string' || value.length === EMPTY_LENGTH) fail('restartReason.kind'); +} + +function requireFolder(value: unknown, field: string): void { + if (typeof value !== 'string' || value.length === EMPTY_LENGTH) fail(field); +} + +function fail(field: string): never { + throw new DaemonProtocolError('malformedControlMessage', `Invalid ${field}.`); +} diff --git a/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts b/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts index 6b0419bc07..ead34a05a6 100644 --- a/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts +++ b/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import { DaemonProtocolError } from './DaemonProtocolError'; +import { validateQueuedRestartReason } from './InstallationChangeValidation'; const EMPTY_STRING_LENGTH: number = 0; const FIRST_QUEUE_POSITION: number = 1; @@ -20,6 +21,7 @@ export function validateRequestAdmissionCapability(payload: Record): void { validateRequestId(payload.requestId); validateQueuePosition(payload.position); + validateQueuedRestartReason(payload); } function validateRequestId(value: unknown): void { diff --git a/libraries/rush-daemon-protocol/src/RequestResultValidation.ts b/libraries/rush-daemon-protocol/src/RequestResultValidation.ts index ff6c8e6fb7..0ed3fe24b1 100644 --- a/libraries/rush-daemon-protocol/src/RequestResultValidation.ts +++ b/libraries/rush-daemon-protocol/src/RequestResultValidation.ts @@ -3,6 +3,7 @@ import { isDaemonControlRecord } from './ControlRecord'; import { DaemonProtocolError } from './DaemonProtocolError'; +import { validateRestartReason } from './InstallationChangeValidation'; import { validateRestartResult } from './RestartResultValidation'; const ADMISSION_ERROR_CODES: ReadonlySet = new Set(['aborted', 'no-wait', 'wait-timeout']); @@ -14,6 +15,7 @@ export function validateRequestResultFields(payload: Record): v validateAdmissionErrorCode(payload.admissionErrorCode); validatePhasedResultShape(payload); validateRestartResult(payload); + validateRestartReason(payload); } function validateAdmissionErrorCode(value: unknown): void { diff --git a/libraries/rush-daemon-protocol/src/index.ts b/libraries/rush-daemon-protocol/src/index.ts index 575a87e8e3..4e984fe8ef 100644 --- a/libraries/rush-daemon-protocol/src/index.ts +++ b/libraries/rush-daemon-protocol/src/index.ts @@ -45,12 +45,8 @@ export type { IDaemonRawModeChangedMessage, IDaemonSetRawModeMessage } from './D export type { IDaemonStdinEndMessage, IDaemonStdinReadyMessage } from './DaemonInteractiveControl'; export type { IDaemonTerminalPolicyMessage } from './DaemonInteractiveControl'; export type { IDaemonPongMessage } from './DaemonPongMessage'; -export type { - IDaemonWarmProjectRank, - IDaemonWarmSetConfiguration, - IDaemonWarmSetStatus -} from './DaemonWorkspaceStatus'; -export type { IDaemonWorkspaceStatus } from './DaemonWorkspaceStatus'; +export type { IDaemonWarmProjectRank, IDaemonWarmSetConfiguration } from './DaemonWorkspaceStatus'; +export type { IDaemonWarmSetStatus, IDaemonWorkspaceStatus } from './DaemonWorkspaceStatus'; export type { IDaemonShutdownAckMessage, IDaemonShutdownMessage } from './DaemonLifecycleControl'; export { isDaemonControlRecord } from './ControlRecord'; export { validateDaemonControlMessage } from './ControlMessageValidation'; @@ -60,6 +56,9 @@ export { createDaemonHello, createDaemonHelloAck, negotiateDaemonHello } from '. export type { DaemonHandshakeOutcome } from './DaemonHandshake'; export type { DaemonJsonNull, DaemonJsonValue } from './DaemonJsonValue'; export type { DaemonCommandOutcome, IDaemonCommandResult } from './DaemonCommandResult'; +export type { DaemonInstallationChangeKind, DaemonRestartReason } from './DaemonInstallationChange'; +export type { IDaemonInstallationChange } from './DaemonInstallationChange'; +export type { IDaemonInstallationChangedRestartReason } from './DaemonInstallationChange'; export { MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS } from './DaemonRequestAdmission'; export { validateDaemonRequestAdmissionOptions } from './DaemonRequestAdmission'; export type { DaemonRequestRejectionCode, IDaemonRequestCancelMessage } from './DaemonRequestControl'; diff --git a/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts b/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts new file mode 100644 index 0000000000..c9335c9c01 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts @@ -0,0 +1,72 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { decodeDaemonControlMessage, encodeDaemonControlMessage } from '../ControlFrameCodec'; +import type { IDaemonCommandResult } from '../DaemonCommandResult'; +import type { DaemonControlMessage } from '../DaemonControlMessage'; +import type { IDaemonInstallationChange } from '../DaemonInstallationChange'; +import { WIRE_TEXT_ENCODER } from '../DaemonWireText'; + +const FAILURE_EXIT_CODE: number = 1; +const ZERO: number = 0; +const CHANGE: IDaemonInstallationChange = { change: 'removed', folder: '/snapshots/s9' }; +const RESULT: IDaemonCommandResult = { + aborted: false, + exitCode: FAILURE_EXIT_CODE, + outcome: 'failure', + requestId: 'installation', + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', ...CHANGE } +}; + +function resultFrame(payload: object): Uint8Array { + return WIRE_TEXT_ENCODER.encode(JSON.stringify({ kind: 'requestResult', payload })); +} + +function pongFrame(installationChange: unknown): Uint8Array { + return WIRE_TEXT_ENCODER.encode( + JSON.stringify({ kind: 'pong', payload: { uptimeMs: ZERO, installationChange } }) + ); +} + +it('round-trips a restart result that names the changed installation', () => { + const message: DaemonControlMessage = { kind: 'requestResult', payload: RESULT }; + expect(decodeDaemonControlMessage(encodeDaemonControlMessage(message))).toEqual(message); +}); + +it('accepts a restart reason kind from a newer daemon', () => { + const payload: object = { + ...RESULT, + restartReason: { kind: 'environmentChanged', names: ['NODE_OPTIONS'] } + }; + expect(decodeDaemonControlMessage(resultFrame(payload))).toMatchObject({ payload }); +}); + +it.each([ + { restartReason: null }, + { restartReason: 'installationChanged' }, + { restartReason: [] }, + { restartReason: { kind: '' } }, + { restartReason: { ...CHANGE } }, + { restartReason: { kind: 'installationChanged', change: 'moved', folder: CHANGE.folder } }, + { restartReason: { kind: 'installationChanged', change: 'removed', folder: '' } }, + { restartReason: { kind: 'installationChanged', change: 'removed' } }, + { retryAfterRestart: undefined, exitCode: ZERO, outcome: 'success' } +])('rejects a malformed restart reason %j', (override: object) => { + expect(() => decodeDaemonControlMessage(resultFrame({ ...RESULT, ...override }))).toThrow(); +}); + +it.each([undefined, CHANGE, { change: 'replaced', folder: '/snapshots/s9/libraries' }])( + 'round-trips an optional pong installation change %j', + (installationChange: unknown) => { + const frame: Uint8Array = pongFrame(installationChange); + expect(encodeDaemonControlMessage(decodeDaemonControlMessage(frame))).toEqual(frame); + } +); + +it.each([null, [], {}, { change: 'moved', folder: '/x' }, { change: 'removed', folder: ZERO }])( + 'rejects a malformed pong installation change %j', + (installationChange: unknown) => { + expect(() => decodeDaemonControlMessage(pongFrame(installationChange))).toThrow(/installationChange/); + } +); diff --git a/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts b/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts new file mode 100644 index 0000000000..05b4a02860 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts @@ -0,0 +1,51 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { decodeDaemonControlMessage, encodeDaemonControlMessage } from '../ControlFrameCodec'; +import type { DaemonControlMessage } from '../DaemonControlMessage'; +import type { DaemonRestartReason } from '../DaemonInstallationChange'; +import { WIRE_TEXT_ENCODER } from '../DaemonWireText'; + +const FIRST_POSITION: number = 1; +const REQUEST_ID: string = 'queued-before-restart'; +const REASON: DaemonRestartReason = { + kind: 'installationChanged', + change: 'removed', + folder: '/snapshots/s9' +}; + +function queuePositionFrame(restartReason: unknown): Uint8Array { + return WIRE_TEXT_ENCODER.encode( + JSON.stringify({ + kind: 'queuePosition', + payload: { position: FIRST_POSITION, requestId: REQUEST_ID, restartReason } + }) + ); +} + +it('round-trips a queue position that says the daemon restarts once the requests ahead finish', () => { + const message: DaemonControlMessage = { + kind: 'queuePosition', + payload: { position: FIRST_POSITION, requestId: REQUEST_ID, restartReason: REASON } + }; + expect(decodeDaemonControlMessage(encodeDaemonControlMessage(message))).toEqual(message); +}); + +it('accepts a queued restart reason kind from a newer daemon', () => { + const restartReason: object = { kind: 'environmentChanged', names: ['NODE_OPTIONS'] }; + expect(decodeDaemonControlMessage(queuePositionFrame(restartReason))).toMatchObject({ + payload: { restartReason } + }); +}); + +it.each([ + null, + 'installationChanged', + [], + { kind: '' }, + { change: 'removed', folder: '/x' }, + { kind: 'installationChanged', change: 'moved', folder: '/x' }, + { kind: 'installationChanged', change: 'removed', folder: '' } +])('rejects a malformed queued restart reason %j', (restartReason: unknown) => { + expect(() => decodeDaemonControlMessage(queuePositionFrame(restartReason))).toThrow(/restartReason/); +}); diff --git a/libraries/rush-daemon-transport/README.md b/libraries/rush-daemon-transport/README.md index f0322bf8cf..97b51ffaba 100644 --- a/libraries/rush-daemon-transport/README.md +++ b/libraries/rush-daemon-transport/README.md @@ -13,7 +13,10 @@ The workspace-keyed socket/pipe **transport** for the Rush daemon (`rushd`): consulted, because they differ between the shells, jobs and services of one user. A daemon resolves its paths with the same rule, and a client that starts one passes the folder it chose as `RUSHD_RUNTIME_DIR`. The folder must be a directory (not a symbolic link) that the - user owns; one that others can open is made owner-only (`0700`). + user owns; one that others can open is made owner-only (`0700`). The socket path must fit in + a socket address, at most 108 bytes on Linux and 104 on other POSIX platforms, because Node.js + silently truncates a longer one; a longer path is refused with `socketPathTooLong` before the + folder is checked or created. - **`net` listener and connector** — framed with [`@rushstack/rush-daemon-protocol`](https://www.npmjs.com/package/@rushstack/rush-daemon-protocol), with backpressure-aware writes and serialized async frame handlers for inbound flow control. diff --git a/libraries/rush-daemon-transport/src/DaemonReclaim.ts b/libraries/rush-daemon-transport/src/DaemonReclaim.ts index 779b343008..b330f4c808 100644 --- a/libraries/rush-daemon-transport/src/DaemonReclaim.ts +++ b/libraries/rush-daemon-transport/src/DaemonReclaim.ts @@ -29,8 +29,9 @@ import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTranspor * * @throws {@link DaemonTransportError} with code `daemonAlreadyRunning` when a * live (or plausibly live) daemon owns the path, or when another starter holds - * the reclaim lock, and with code `unsafeRuntimeDirectory` for an unsafe - * runtime directory. + * the reclaim lock, with code `unsafeRuntimeDirectory` for an unsafe + * runtime directory, and with code `socketPathTooLong` for a socket path that + * no client could connect to. * * @beta */ diff --git a/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts b/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts index 914b5240e4..700fbd8d6c 100644 --- a/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts +++ b/libraries/rush-daemon-transport/src/DaemonRuntimeDir.ts @@ -5,6 +5,7 @@ import * as fs from 'node:fs'; import type { IDaemonPaths } from './DaemonPaths'; import { verifyRuntimeFolder } from './DaemonRuntimeFolderCheck'; +import { assertDaemonSocketPathFits } from './DaemonSocketPathLength'; const DIR_MODE: number = 0o700; @@ -29,22 +30,28 @@ function tryCreateFolder(folder: string): unknown { * open to others is changed to mode `0700`. Otherwise another user could have created it first, for example * in `/tmp`, and could listen at the socket path or plant the records that reclaim acts on. * - * @throws {@link DaemonTransportError} with code `unsafeRuntimeDirectory`. + * The socket path must also fit in a socket address (108 bytes on Linux, 104 on macOS), which only a long + * `RUSHD_RUNTIME_DIR` exceeds. Node would cut a longer path short, so no client could reach the daemon. + * + * @throws {@link DaemonTransportError} with code `socketPathTooLong` or `unsafeRuntimeDirectory`. * * @beta */ export function assertDaemonRuntimeDirIsPrivate(paths: IDaemonPaths): void { + assertDaemonSocketPathFits(paths, process.platform); if (paths.runtimeDir !== undefined) verifyRuntimeFolder(paths.runtimeDir, getCurrentUid()); } /** * Creates the per-user runtime directory (mode `0700`) when the platform has one, and checks it as * {@link assertDaemonRuntimeDirIsPrivate} does. Must be called before binding a POSIX socket inside it. + * A socket path that is too long is refused before anything is created. * * @beta */ export function ensureDaemonRuntimeDir(paths: IDaemonPaths): void { if (paths.runtimeDir === undefined) return; + assertDaemonSocketPathFits(paths, process.platform); const failure: unknown = tryCreateFolder(paths.runtimeDir); // An entry in the way (such as a file or a dangling link) is explained by the check, not by mkdir. verifyRuntimeFolder(paths.runtimeDir, getCurrentUid()); diff --git a/libraries/rush-daemon-transport/src/DaemonSocketPathLength.ts b/libraries/rush-daemon-transport/src/DaemonSocketPathLength.ts new file mode 100644 index 0000000000..29cf3f1998 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonSocketPathLength.ts @@ -0,0 +1,61 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as path from 'node:path'; + +import { DAEMON_RUNTIME_DIR_ENV_VAR } from './DaemonPaths'; +import type { IDaemonPaths } from './DaemonPaths'; +import { DaemonTransportError, DaemonTransportErrorCode } from './DaemonTransportError'; + +// The size of sockaddr_un.sun_path. Node truncates a longer path to this many bytes when it binds or connects, +// but a daemon publishes its socket with link(2), which does not, so clients would look for a name nobody bound. +const LINUX_MAX_SOCKET_PATH_BYTES: number = 108; +// macOS and the BSDs. +const OTHER_MAX_SOCKET_PATH_BYTES: number = 104; +const LINUX_PLATFORM: NodeJS.Platform = 'linux'; +// The shortest absolute base, `/`. +const MIN_BASE_BYTES: number = 1; + +interface ISocketPathLength { + readonly socketPath: string; + readonly runtimeDir: string; + readonly bytes: number; + readonly limit: number; +} + +/** The longest socket path, in bytes, that a POSIX `platform` binds and connects to without truncating it. */ +export function getMaxDaemonSocketPathBytes(platform: NodeJS.Platform): number { + return platform === LINUX_PLATFORM ? LINUX_MAX_SOCKET_PATH_BYTES : OTHER_MAX_SOCKET_PATH_BYTES; +} + +// Paths from resolveDaemonPaths always leave room for a base; other layouts may not. +function describeRemedy(maxBaseBytes: number): string { + return maxBaseBytes < MIN_BASE_BYTES + ? `Use a shorter runtime folder, or unset ${DAEMON_RUNTIME_DIR_ENV_VAR}.` + : `Set ${DAEMON_RUNTIME_DIR_ENV_VAR} to an absolute path of at most ${maxBaseBytes} bytes, or unset it.`; +} + +function createTooLongError(length: ISocketPathLength): DaemonTransportError { + const { socketPath, runtimeDir, bytes, limit } = length; + // The runtime folder is `/rushd-`, so everything after its parent is fixed. + const suffixBytes: number = bytes - Buffer.byteLength(path.posix.dirname(runtimeDir)); + return new DaemonTransportError( + DaemonTransportErrorCode.socketPathTooLong, + `The daemon socket path ${socketPath} is ${bytes} bytes long, but this platform allows at most ${limit}. ` + + describeRemedy(limit - suffixBytes) + ); +} + +/** + * Throws unless the POSIX socket path of `paths` fits in a socket address on `platform`. Windows named pipes + * (no runtime directory) are not checked. + * + * @throws {@link DaemonTransportError} with code `socketPathTooLong`. + */ +export function assertDaemonSocketPathFits(paths: IDaemonPaths, platform: NodeJS.Platform): void { + const { runtimeDir, socketPath } = paths; + if (runtimeDir === undefined) return; + const bytes: number = Buffer.byteLength(socketPath); + const limit: number = getMaxDaemonSocketPathBytes(platform); + if (bytes > limit) throw createTooLongError({ socketPath, runtimeDir, bytes, limit }); +} diff --git a/libraries/rush-daemon-transport/src/DaemonTransportError.ts b/libraries/rush-daemon-transport/src/DaemonTransportError.ts index 48e9dfb121..4257851d55 100644 --- a/libraries/rush-daemon-transport/src/DaemonTransportError.ts +++ b/libraries/rush-daemon-transport/src/DaemonTransportError.ts @@ -16,7 +16,9 @@ export enum DaemonTransportErrorCode { /** The transport was closed while an operation was in flight. */ transportClosed = 'transportClosed', /** The per-user runtime directory is a symbolic link, is not a directory, or another user owns it. */ - unsafeRuntimeDirectory = 'unsafeRuntimeDirectory' + unsafeRuntimeDirectory = 'unsafeRuntimeDirectory', + /** The POSIX socket path is longer than a socket address allows, so no client could connect to it. */ + socketPathTooLong = 'socketPathTooLong' } /** diff --git a/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts b/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts index abd43150e2..0290879702 100644 --- a/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts +++ b/libraries/rush-daemon-transport/src/test/ListenerSuccession.test.ts @@ -10,6 +10,7 @@ import type { DaemonFrameConnection } from '../DaemonFrameConnection'; import { DaemonFrameListener } from '../DaemonListener'; import { readDaemonLockfile } from '../DaemonLockfile'; import type { IDaemonPaths } from '../DaemonPaths'; +import { DaemonTransportErrorCode } from '../DaemonTransportError'; import { createDeferred, createIsolatedTestDaemonPaths, removeIsolatedBase } from './TestDaemonFixture'; import type { IDeferred } from './TestDaemonFixture'; @@ -17,6 +18,8 @@ import type { IDeferred } from './TestDaemonFixture'; // A named pipe can't be deleted while its server runs, so only POSIX has a successor in this sense. const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; const SUCCESSOR: string = 'successor'; +// The name that a listener binds before it publishes its socket. +const PRIVATE_NAME_PREFIX: string = '.bind-'; function listenAsync(paths: IDaemonPaths, reached?: IDeferred): Promise { return DaemonFrameListener.listenAsync(paths, { @@ -62,6 +65,26 @@ posixIt('keeps a successor reachable after a predecessor whose runtime folder wa }) ); +posixIt('does not replace the socket of a live listener whose lockfile was deleted', async () => { + const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); + const reached: IDeferred = createDeferred(); + const live: DaemonFrameListener = await listenAsync(paths, reached); + fs.rmSync(paths.lockfilePath); + try { + const refused: DaemonTransportErrorCode = DaemonTransportErrorCode.daemonAlreadyRunning; + await expect(listenAsync(paths)).rejects.toMatchObject({ code: refused }); + const names: string[] = fs.readdirSync(paths.runtimeDir ?? paths.socketPath); + expect(names.filter((name: string) => name.startsWith(PRIVATE_NAME_PREFIX))).toEqual([]); + // The refused listener must not have replaced the name, so the connection reaches the live one. + const client: DaemonFrameConnection = await connectDaemonAsync(paths.socketPath); + await expect(reached.promise).resolves.toBe(SUCCESSOR); + await client.closeAsync(); + } finally { + await live.closeAsync(); + removeIsolatedBase(paths); + } +}); + posixIt('removes its own socket and lockfile when it closes', async () => { const paths: IDaemonPaths = createIsolatedTestDaemonPaths(); const listener: DaemonFrameListener = await listenAsync(paths); diff --git a/libraries/rush-daemon-transport/src/test/SocketPathLength.test.ts b/libraries/rush-daemon-transport/src/test/SocketPathLength.test.ts new file mode 100644 index 0000000000..bdbc1612ce --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/SocketPathLength.test.ts @@ -0,0 +1,80 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonPaths } from '../DaemonPaths'; +import { DAEMON_RUNTIME_DIR_ENV_VAR, resolveDaemonPaths } from '../DaemonPaths'; +import { assertDaemonSocketPathFits, getMaxDaemonSocketPathBytes } from '../DaemonSocketPathLength'; +import { DaemonTransportErrorCode } from '../DaemonTransportError'; +import { WORKSPACE_KEY_LENGTH } from '../WorkspaceKey'; + +const UID: number = 9127721; +const KEY: string = `rushd-${'0'.repeat(WORKSPACE_KEY_LENGTH)}`; +// What a base gains on its way to the socket path: `/rushd-/rushd-.sock`. +const SUFFIX_BYTES: number = Buffer.byteLength(`/rushd-${UID}/${KEY}.sock`); +const ONE: number = 1; +// Longer than the limit of every platform. +const LONG_NAME_BYTES: number = 160; +// 25 two-byte characters make a 51-byte base of only 26 characters, too long for Linux by one byte. +const TWO_BYTE_CHARACTER_COUNT: number = 25; +const TOO_LONG: object = { code: DaemonTransportErrorCode.socketPathTooLong }; +const LINUX_LIMIT: number = 108; +const BSD_LIMIT: number = 104; +const LIMITS: readonly [NodeJS.Platform, number][] = [ + ['linux', LINUX_LIMIT], + ['darwin', BSD_LIMIT], + ['freebsd', BSD_LIMIT] +]; + +function resolvePaths(platform: NodeJS.Platform, base: string): IDaemonPaths { + const env: Record = { [DAEMON_RUNTIME_DIR_ENV_VAR]: base }; + return resolveDaemonPaths({ platform, env, tmpdir: '/tmp', uid: UID }, KEY); +} + +function createBase(bytes: number): string { + return `/${'a'.repeat(bytes - ONE)}`; +} + +function captureMessage(paths: IDaemonPaths, platform: NodeJS.Platform): string { + try { + assertDaemonSocketPathFits(paths, platform); + } catch (error) { + expect(error).toMatchObject(TOO_LONG); + return (error as Error).message; + } + return ''; +} + +it.each(LIMITS)( + 'allows a socket path of at most sun_path bytes on %s', + (platform: NodeJS.Platform, limit: number) => { + const maxBaseBytes: number = limit - SUFFIX_BYTES; + const fits: IDaemonPaths = resolvePaths(platform, createBase(maxBaseBytes)); + expect(getMaxDaemonSocketPathBytes(platform)).toBe(limit); + expect(Buffer.byteLength(fits.socketPath)).toBe(limit); + expect(captureMessage(fits, platform)).toBe(''); + const tooLong: IDaemonPaths = resolvePaths(platform, createBase(maxBaseBytes + ONE)); + expect(captureMessage(tooLong, platform)).toBe( + `The daemon socket path ${tooLong.socketPath} is ${limit + ONE} bytes long, but this platform allows at ` + + `most ${limit}. Set ${DAEMON_RUNTIME_DIR_ENV_VAR} to an absolute path of at most ${maxBaseBytes} ` + + 'bytes, or unset it.' + ); + } +); + +it('counts bytes, not characters', () => { + const paths: IDaemonPaths = resolvePaths('linux', `/${'é'.repeat(TWO_BYTE_CHARACTER_COUNT)}`); + expect(captureMessage(paths, 'linux')).toContain('is 109 bytes long'); +}); + +it('names no length to aim for when a runtime folder of another layout leaves no room', () => { + const runtimeDir: string = createBase(LONG_NAME_BYTES); + const paths: IDaemonPaths = { runtimeDir, socketPath: `${runtimeDir}/d.sock`, lockfilePath: '' }; + expect(captureMessage(paths, 'linux')).toMatch( + /at most 108\. Use a shorter runtime folder, or unset RUSHD_/ + ); +}); + +it('does not check a Windows named pipe', () => { + const socketPath: string = `\\\\.\\pipe\\${'p'.repeat(LONG_NAME_BYTES)}`; + expect(captureMessage({ runtimeDir: undefined, socketPath, lockfilePath: '' }, 'win32')).toBe(''); +}); diff --git a/libraries/rush-daemon-transport/src/test/SocketPathRefusal.test.ts b/libraries/rush-daemon-transport/src/test/SocketPathRefusal.test.ts new file mode 100644 index 0000000000..c2ca10a40f --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/SocketPathRefusal.test.ts @@ -0,0 +1,54 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import { DaemonFrameListener } from '../DaemonListener'; +import type { IDaemonPaths } from '../DaemonPaths'; +import { DAEMON_RUNTIME_DIR_ENV_VAR, resolveDaemonPaths } from '../DaemonPaths'; +import { reclaimStaleDaemonAsync } from '../DaemonReclaim'; +import { assertDaemonRuntimeDirIsPrivate, ensureDaemonRuntimeDir } from '../DaemonRuntimeDir'; +import { DaemonTransportErrorCode } from '../DaemonTransportError'; + +import { createIsolatedTestDaemonPaths, removeIsolatedBase } from './TestDaemonFixture'; + +// Windows has no runtime folder: its named pipes live in the kernel's pipe namespace. +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +// Longer than the limit of every platform, before the runtime folder and socket name are added. +const LONG_NAME_BYTES: number = 110; +const TOO_LONG: object = { + code: DaemonTransportErrorCode.socketPathTooLong, + message: expect.stringContaining(DAEMON_RUNTIME_DIR_ENV_VAR) +}; + +function captureError(action: () => void): unknown { + try { + action(); + } catch (error) { + return error; + } +} + +posixIt('refuses a long socket path before it creates, checks, reclaims or binds anything', async () => { + const isolated: IDaemonPaths = createIsolatedTestDaemonPaths(); + const base: string = path.join(path.dirname(isolated.runtimeDir ?? ''), 'b'.repeat(LONG_NAME_BYTES)); + const env: Record = { [DAEMON_RUNTIME_DIR_ENV_VAR]: base }; + const paths: IDaemonPaths = resolveDaemonPaths( + { platform: process.platform, env, tmpdir: os.tmpdir(), uid: process.getuid?.() }, + 'rushd-long' + ); + expect(captureError(() => assertDaemonRuntimeDirIsPrivate(paths))).toMatchObject(TOO_LONG); + expect(captureError(() => ensureDaemonRuntimeDir(paths))).toMatchObject(TOO_LONG); + await expect(reclaimStaleDaemonAsync(paths)).rejects.toMatchObject(TOO_LONG); + const listening: Promise = DaemonFrameListener.listenAsync(paths, { + protocolVersion: DAEMON_PROTOCOL_VERSION, + onConnection: () => undefined + }); + await expect(listening).rejects.toMatchObject(TOO_LONG); + expect(fs.existsSync(base)).toBe(false); + removeIsolatedBase(isolated); +}); diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index d53bb7f280..24f1f68c86 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -206,6 +206,25 @@ then retries an eligible request **at most once**. Command input/output or cance even with the typed flag. Error text, a changed PID, or connection loss never authorizes replay. Ordinary shutdown and disconnect retain cancellation semantics. +A daemon also restarts when its own installation changes. `serveRushDaemonAsync` records the identity (device, +inode and birth time) of the folder it loaded the daemon from, of the Rush engine folder, and of their parents. +Before each request it checks them. When one was removed, or another folder now has its path (a deleted snapshot, +or a reinstalled `~/.rush` release), the daemon cannot load the rest of its code, so from then on it admits no +more requests, and `pong` reports `installationChange`. Requests that it had already admitted finish. Each new or +queued request, and each request that fails before it begins while the installation is changed (for example on a +module that the daemon can no longer load), waits for them in the restart drain (see above), with queue positions +that carry the `restartReason`. It then gets the typed `retryAfterRestart: true` result with +`restartReason: { kind: 'installationChanged', change, folder }` instead of an early answer, so that its client +does not wait for the old daemon to exit while a long build still runs. The drain's timeout rules are the same as +for an environment: a client-default `waitTimeoutMs` does not limit waiting for the requests that were already +being served when the wait began, as long as no rushx script is being served, and a timeout names the changed +folder. A request that times out there, or that sets `noWait`, gets its admission error code and requests no +restart. The first request that gets the result makes the daemon exit without selecting a successor, and each +client starts one with its own launcher. Embedded hosts opt in with `checkInstallation` +(`captureDaemonInstallation`). +The daemon log (`onLog`) gets one line for the change and one for each rejected request, with its code, its +message and, for an unexpected `routingFailed`, the stack. + Positively identified built-in `install` and `update` requests execute in `NativeMutationWorker`, a single-shot native Rush parser process owned by `GlobalCommandExecutionContext`. This is not the phased warm engine. Native arguments, policies, hooks, stdin/EOF, output and numeric exit status are preserved. Even a failed mutation diff --git a/libraries/rush-daemon/src/DaemonControlSession.ts b/libraries/rush-daemon/src/DaemonControlSession.ts index 4e043b4f33..d38cb8ffe3 100644 --- a/libraries/rush-daemon/src/DaemonControlSession.ts +++ b/libraries/rush-daemon/src/DaemonControlSession.ts @@ -21,6 +21,7 @@ import type { DaemonRequestRejectionCode, IDaemonErrorMessage, IDaemonFrame, + IDaemonInstallationChange, IDaemonPongMessage, IDaemonRequestEnvelope, IDaemonWorkspaceStatus @@ -53,6 +54,10 @@ export interface IDaemonControlSessionOptions { /** Counts requests running on every connection, reported in the shutdown acknowledgement. */ readonly getActiveRequestCount?: () => number; readonly getWorkspaceStatus?: () => IDaemonWorkspaceStatus; + /** Reports a removed or replaced installation in `pong`. */ + readonly checkInstallation?: () => IDaemonInstallationChange | undefined; + /** Receives a message for the daemon log for each rejected request. */ + readonly onLog?: (message: string) => void; } interface IRequestState { @@ -361,6 +366,12 @@ export class DaemonControlSession { } if (dispatchError !== undefined && !state.client.terminalOutcomeSent && !this.#connectionClosed) { const rejection: IClassifiedRejection = classifyRejection(dispatchError); + // The client prints only the message; keep the rest where `rush-client daemon logs` finds it. + this.#options.onLog?.( + `rushd: rejected request ${envelope.requestId} (${rejection.code}): ${ + rejection.code === 'routingFailed' ? describeError(dispatchError) : rejection.message + }` + ); await state.client.writeRejectionAsync(rejection.code, rejection.message); } } @@ -398,6 +409,7 @@ export class DaemonControlSession { pid: process.pid, residentMemoryBytes: process.memoryUsage().rss, workspace: this.#options.getWorkspaceStatus?.(), + installationChange: this.#options.checkInstallation?.(), uptimeMs: Date.now() - this.#options.startedAtMs } }; @@ -548,6 +560,11 @@ function classifyRejection(error: unknown): IClassifiedRejection { return { code: 'routingFailed', message: normalizeError(error).message }; } +function describeError(error: unknown): string { + const normalized: Error = normalizeError(error); + return normalized.stack ?? normalized.message; +} + function combineCloseErrors( error: Error | undefined, cleanupErrors: ReadonlyArray diff --git a/libraries/rush-daemon/src/DaemonInstallationMonitor.ts b/libraries/rush-daemon/src/DaemonInstallationMonitor.ts new file mode 100644 index 0000000000..d30f98af13 --- /dev/null +++ b/libraries/rush-daemon/src/DaemonInstallationMonitor.ts @@ -0,0 +1,102 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import type { + DaemonInstallationChangeKind, + IDaemonInstallationChange +} from '@rushstack/rush-daemon-protocol'; + +/** + * Reports a folder of the running daemon's installation that was removed or replaced after startup, or + * `undefined` while the installation is intact. + * + * @beta + */ +export type CheckDaemonInstallation = () => IDaemonInstallationChange | undefined; + +interface IFolderIdentity { + readonly folder: string; + readonly dev: bigint; + readonly ino: bigint; + /** Zero where the file system does not record it; it tells a recreated folder that reused an inode. */ + readonly birthtimeNs: bigint; +} + +/** + * Remembers which folder each path and each of its ancestors named at startup. + * + * @remarks + * A check stats only the given folders. A folder keeps its identity while files inside it change, so only a + * removed, renamed or recreated folder is reported: the outermost ancestor that no longer exists, or that is now + * a different folder. Errors other than a missing path are ignored rather than reported, because restarting a + * daemon whose code is still in place would only lose its warm state. + * + * @beta + */ +export function captureDaemonInstallation(folders: ReadonlyArray): CheckDaemonInstallation { + const chains: IFolderIdentity[][] = []; + for (const folder of folders) { + const chain: IFolderIdentity[] | undefined = captureChain(path.resolve(folder)); + if (chain) chains.push(chain); + } + return () => { + for (const chain of chains) { + if (compareIdentity(chain[chain.length - 1]) === undefined) continue; + for (const identity of chain) { + const change: DaemonInstallationChangeKind | undefined = compareIdentity(identity); + if (change) return { change, folder: identity.folder }; + } + } + return undefined; + }; +} + +/** The folders this process loaded its code from: its own package's output and the Rush engine. */ +export function getDaemonInstallationFolders(): string[] { + return [__dirname, path.dirname(require.resolve('@microsoft/rush-lib'))]; +} + +function captureChain(folder: string): IFolderIdentity[] | undefined { + const ancestors: string[] = []; + for (let current: string = folder; ; current = path.dirname(current)) { + ancestors.unshift(current); + if (path.dirname(current) === current) break; + } + const chain: IFolderIdentity[] = []; + for (const ancestor of ancestors) { + const stats: fs.BigIntStats | undefined = tryStat(ancestor); + if (stats) + chain.push({ folder: ancestor, dev: stats.dev, ino: stats.ino, birthtimeNs: stats.birthtimeNs }); + } + return chain[chain.length - 1]?.folder === folder ? chain : undefined; +} + +function compareIdentity(identity: IFolderIdentity): DaemonInstallationChangeKind | undefined { + let stats: fs.BigIntStats | undefined; + try { + stats = fs.statSync(identity.folder, { bigint: true }); + } catch (error) { + return isMissing(error) ? 'removed' : undefined; + } + return stats.dev === identity.dev && + stats.ino === identity.ino && + stats.birthtimeNs === identity.birthtimeNs + ? undefined + : 'replaced'; +} + +function tryStat(folder: string): fs.BigIntStats | undefined { + try { + return fs.statSync(folder, { bigint: true }); + } catch { + return undefined; + } +} + +function isMissing(error: unknown): boolean { + const code: unknown = (error as NodeJS.ErrnoException | undefined)?.code; + return code === 'ENOENT' || code === 'ENOTDIR'; +} diff --git a/libraries/rush-daemon/src/RushDaemonCommandLine.ts b/libraries/rush-daemon/src/RushDaemonCommandLine.ts index 3d4a93892e..e7715cee11 100644 --- a/libraries/rush-daemon/src/RushDaemonCommandLine.ts +++ b/libraries/rush-daemon/src/RushDaemonCommandLine.ts @@ -50,6 +50,7 @@ export async function launchRushDaemonAsync(startingFolder: string = process.cwd requestResolver: new RushDaemonRequestResolver(new ProductionDaemonRequestResolver()), idleTimeoutSeconds: configuration.idleTimeoutSeconds, onError: (error: Error) => process.stderr.write(`${error.stack ?? error.message}\n`), + onLog: (message: string) => process.stderr.write(`${new Date().toISOString()} ${message}\n`), onReady: (host) => { process.stdout.write(`rushd ready at ${host.paths.socketPath}\n`); } diff --git a/libraries/rush-daemon/src/RushDaemonHost.ts b/libraries/rush-daemon/src/RushDaemonHost.ts index 17ff4a7bed..52364c7886 100644 --- a/libraries/rush-daemon/src/RushDaemonHost.ts +++ b/libraries/rush-daemon/src/RushDaemonHost.ts @@ -14,6 +14,7 @@ import { import type { DaemonFrameConnection, IDaemonPaths } from '@rushstack/rush-daemon-transport'; import { DaemonControlSession } from './DaemonControlSession'; +import type { CheckDaemonInstallation } from './DaemonInstallationMonitor'; import { DaemonIdleTimer } from './DaemonIdleTimer'; import type { IDaemonInteractiveConnection } from './DaemonInteractiveConnection'; import { DaemonRequestDispatcher } from './DaemonRequestDispatcher'; @@ -46,6 +47,18 @@ export interface IRushDaemonHostOptions { readonly idleTimeoutSeconds?: number; /** Reports connection-level failures. */ readonly onError?: (error: Error) => void; + /** + * Receives messages for the daemon log: one for each rejected request, with the stack when the failure was + * unexpected, and one for each restart that the clients must finish. + */ + readonly onLog?: (message: string) => void; + /** + * Reports a folder of this daemon's installation that was removed or replaced after startup. Checked before + * each request: after a change, new and queued requests get a typed restart result, and once running requests + * finish, the host closes without a successor so that the clients start one. `pong` reports the change. + * Not checked when omitted. + */ + readonly checkInstallation?: CheckDaemonInstallation; /** Resolves validated wire envelopes into existing typed phased or global requests. */ readonly requestResolver?: IDaemonRequestResolver; /** Receives the request-scoped interactive broker owned by each accepted connection. */ @@ -76,10 +89,13 @@ export class RushDaemonHost { #notifyClosed: (() => void) | undefined; readonly #options: IRushDaemonHostOptions; readonly #startedAt: string; - #restartPromise: Promise | undefined; + #restartPromise: Promise | undefined; #resolveRestart: ((result: IWorkspaceProcessRestartResult | undefined) => void) | undefined; #rejectRestart: ((error: Error) => void) | undefined; - /** Settles after an accepted restart reaches a new ready process, or normal shutdown finishes without restarting. */ + /** + * Settles after an accepted restart reaches a new ready process, or normal shutdown finishes without restarting. + * Resolves `undefined` when the installation changed, because the clients start the next process. + */ public readonly restartCompleted: Promise; /** Resolves after shutdown cleanup finishes. Use closeAsync() to observe cleanup failures. */ @@ -147,7 +163,9 @@ export class RushDaemonHost { resolver: options.requestResolver, rushVersion: options.rushVersion, getSuccessorLaunchAsync: options.getSuccessorLaunchAsync, - onRestartRequested: requestRestart + onRestartRequested: requestRestart, + checkInstallation: options.checkInstallation, + onLog: options.onLog }); } } catch (error) { @@ -172,6 +190,8 @@ export class RushDaemonHost { dispatcher: requestDispatcher, startedAtMs, getWorkspaceStatus: readWorkspaceStatus, + checkInstallation: options.checkInstallation, + onLog: options.onLog, onInteractiveConnection: options.onInteractiveConnection, onClosed: (closedSession: DaemonControlSession, error: Error | undefined) => { sessions.delete(closedSession); @@ -281,8 +301,11 @@ export class RushDaemonHost { ); } - async #restartOnceAsync(plan: IWorkspaceProcessRestartPlan): Promise { + async #restartOnceAsync( + plan: IWorkspaceProcessRestartPlan + ): Promise { await this.closeAsync(new DaemonShutdownError({ initiator: 'restart' })); + if (plan.reason === 'installation-changed') return undefined; if (plan.failure) throw plan.failure; if (!plan.launch) throw new Error('A successor was not selected.'); const paths: IDaemonPaths = resolveDaemonPathsFromProcess( diff --git a/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts b/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts index f39cbd2717..be0748b6df 100644 --- a/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts +++ b/libraries/rush-daemon/src/SelectedDaemonBootstrap.ts @@ -111,6 +111,9 @@ async function mainAsync(): Promise { onError: (error) => { process.stderr.write(`${error.stack ?? error.message}\n`); }, + onLog: (message) => { + process.stderr.write(`${new Date().toISOString()} ${message}\n`); + }, onReady: (host) => { process.stdout.write(`rushd ready at ${host.paths.socketPath} (Rush ${installation.rushVersion})\n`); } diff --git a/libraries/rush-daemon/src/WorkspaceProcessRestart.ts b/libraries/rush-daemon/src/WorkspaceProcessRestart.ts index 26897d8602..cffa348353 100644 --- a/libraries/rush-daemon/src/WorkspaceProcessRestart.ts +++ b/libraries/rush-daemon/src/WorkspaceProcessRestart.ts @@ -5,12 +5,20 @@ import type { IDaemonStartCommand } from '@rushstack/rush-client-core'; import { selectDaemonLauncherAsync } from './VersionSelectedDaemonLauncher'; -/** Inputs for selecting a successor, before the old host gives up ownership. @beta */ +/** + * Inputs for selecting a successor, before the old host gives up ownership. + * + * @remarks + * For `installation-changed`, the host selects no successor: its installation was removed or replaced, so it + * exits once running requests finish, and each client starts a daemon with its own launcher. + * + * @beta + */ export interface IWorkspaceProcessRestartContext { readonly repoRoot: string; readonly rushVersion: string; readonly environment: Readonly>; - readonly reason: 'hard-input-change' | 'native-mutation'; + readonly reason: 'hard-input-change' | 'native-mutation' | 'installation-changed'; } /** An explicitly selected launch command consumed by the existing core startup contract. @beta */ diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index 7a82501c47..a8cef149f0 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -7,6 +7,7 @@ import { } from '@rushstack/rush-daemon-protocol'; import type { DaemonRequestAdmissionErrorCode, + DaemonRestartReason, IDaemonRequestAdmissionOptions, IDaemonRequestQueuePositionMessage } from '@rushstack/rush-daemon-protocol'; @@ -80,6 +81,11 @@ export function freezeDaemonRequestAdmissionOptions( return copy; } +/** Says why the daemon restarts, completing "the daemon could restart ". */ +function formatRestartCause(restartReason: DaemonRestartReason): string { + return `because its installation at ${restartReason.folder} was ${restartReason.change}`; +} + class WorkspaceRequestScheduler extends RequestScheduler { readonly #session: IWorkspaceSession; @@ -120,12 +126,12 @@ class QueuePositionWriter { writeQueuePositionAsync.call(client, message); } - public enqueue(position: number): void { + public enqueue(position: number, restartReason?: DaemonRestartReason): void { this.#tail = this.#tail .then(() => this.#writeQueuePositionAsync({ kind: 'queuePosition', - payload: { position, requestId: this.#requestId } + payload: { position, requestId: this.#requestId, ...(restartReason && { restartReason }) } }) ) .catch((error: unknown) => { @@ -312,6 +318,30 @@ export class RequestAdmissionController { ); } + /** + * After the restart drain ({@link RequestAdmissionController.waitForRestartDrainAsync}), waits until no other + * request holds `scheduler`, so that this request can be answered with a restart result for `restartReason` + * without interrupting them. + * + * @remarks + * Once the drain is over, only requests that the drain does not track can still hold `scheduler`, such as + * observers that are winding down after their cancellation, so this wait uses the request's remaining admission + * budget. Queue positions carry `restartReason`, and a timeout names it, so that the client can say why it waits. + */ + public async acquireBeforeRestartAsync( + scheduler: RequestScheduler, + restartReason: DaemonRestartReason + ): Promise { + return await this.#acquireAsync( + scheduler, + RequestExclusivityClass.Exclusive, + this.#remainingMs, + `the running requests to finish before the daemon restarts ${formatRestartCause(restartReason)}`, + this.#abortController.signal, + restartReason + ); + } + /** * Waits for shared-build workspace admission while another request loads or reloads the workspace graph. * @@ -387,7 +417,8 @@ export class RequestAdmissionController { exclusivityClass: RequestExclusivityClass, waitTimeoutMs: number | undefined, waitingFor: string, - abortSignal: AbortSignal = this.#abortController.signal + abortSignal: AbortSignal = this.#abortController.signal, + restartReason?: DaemonRestartReason ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; const startMs: number = Date.now(); @@ -397,7 +428,9 @@ export class RequestAdmissionController { abortSignal, exclusivityClass, noWait: this.#admission?.noWait, - onQueuePositionChanged: writer ? (position: number) => writer.enqueue(position) : undefined, + onQueuePositionChanged: writer + ? (position: number) => writer.enqueue(position, restartReason) + : undefined, waitTimeoutMs }); await writer?.flushAsync(); @@ -429,13 +462,18 @@ export class RequestAdmissionController { * the wait while a rushx script is served, since a script may not exit until it is stopped, and waiting for * requests that arrived later, which could otherwise keep the request waiting for as long as they keep arriving. * An explicit `noWait` or `waitTimeoutMs` applies to the whole wait, using the same budget as workspace admission. + * + * A `restartReason` says that the daemon restarts for that reason rather than for the request's environment. Queue + * positions then carry it, and admission errors name it. */ public async waitForRestartDrainAsync( arbiter: WorkspaceRestartArbiter, - ticket: IWorkspaceRestartTicket + ticket: IWorkspaceRestartTicket, + restartReason?: DaemonRestartReason ): Promise { - await this.#waitForRestartArbiterAsync((options: IWorkspaceRestartDrainOptions) => - arbiter.waitForDrainAsync(ticket, options) + await this.#waitForRestartArbiterAsync( + (options: IWorkspaceRestartDrainOptions) => arbiter.waitForDrainAsync(ticket, options), + restartReason ); } @@ -458,7 +496,8 @@ export class RequestAdmissionController { } async #waitForRestartArbiterAsync( - waitAsync: (options: IWorkspaceRestartDrainOptions) => Promise + waitAsync: (options: IWorkspaceRestartDrainOptions) => Promise, + restartReason?: DaemonRestartReason ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; const startMs: number = Date.now(); @@ -470,7 +509,8 @@ export class RequestAdmissionController { noWait: this.#admission?.noWait, waitTimeoutMs: this.#remainingMs, waivesTimeoutForServedWork: this.#admission?.waitTimeoutIsDefault === true, - onServingCountChanged: writer ? (count: number) => writer.enqueue(count) : undefined + restartCause: restartReason && formatRestartCause(restartReason), + onServingCountChanged: writer ? (count: number) => writer.enqueue(count, restartReason) : undefined }); } finally { await writer?.flushAsync(); diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 4f0cc8d9a0..24f3a40d64 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -19,7 +19,12 @@ import { } from '@microsoft/rush-lib'; import { LockFile } from '@rushstack/node-core-library'; import { NoOpTerminalProvider, Terminal } from '@rushstack/terminal'; -import type { IDaemonCommandResult, IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; +import type { + DaemonRestartReason, + IDaemonCommandResult, + IDaemonInstallationChange, + IDaemonRequestEnvelope +} from '@rushstack/rush-daemon-protocol'; import { DaemonRequestDispatchError, @@ -58,6 +63,7 @@ import type { import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from './WorkspaceRestartArbiter'; import { classifyRushCommand } from './RushCommandRequestPolicy'; import { FreshCaptureCoalescer } from './FreshCaptureCoalescer'; +import type { CheckDaemonInstallation } from './DaemonInstallationMonitor'; interface IExecutionState { began: boolean; @@ -79,6 +85,10 @@ export interface IWorkspaceRequestLifecycleOptions { readonly rushVersion: string; readonly getSuccessorLaunchAsync: GetWorkspaceSuccessorLaunchAsync | undefined; readonly onRestartRequested: (plan: IWorkspaceProcessRestartPlan) => void; + /** Checked before each request; a removed or replaced installation restarts the process without a successor. */ + readonly checkInstallation?: CheckDaemonInstallation; + /** Receives messages for the daemon log. */ + readonly onLog?: (message: string) => void; } class RestartBeforeExecution extends Error { @@ -108,6 +118,14 @@ class RestartPendingBeforeExecution extends Error { } } +class InstallationChangedBeforeExecution extends Error { + public constructor(change: IDaemonInstallationChange) { + super( + `The daemon's installation at ${change.folder} was ${change.change}. No operation was scheduled or executed. Reconnect and submit a new request after restart.` + ); + } +} + /** * Generation admission composes the existing request schedulers, native locks and session provider. * It never releases a resolved request onto a different session, and never replays scheduled work. @@ -127,6 +145,12 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { readonly #observers: Set = new Set(); readonly #terminal: Terminal = new Terminal(new NoOpTerminalProvider()); readonly #runtimePaths: ReadonlyArray = [__dirname, path.resolve(__dirname, '../package.json')]; + // Resolved once: after the installation is removed, resolving it again would fail before the restart check. + readonly #rushLibPath: string = getRushLibPathHandoff( + require.resolve('@microsoft/rush-lib'), + process.env[EnvironmentVariableNames._RUSH_LIB_PATH] + ); + readonly #repoRoot: string; readonly #startupFingerprint: IWorkspaceInputFingerprint; readonly #runtimeCache: WorkspaceRuntimeFingerprintCache; // Concurrent requests share captures; each capture still starts after the requests it serves arrived. @@ -145,6 +169,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { #closing: boolean = false; #restartPending: boolean = false; #lastReloadTier: WorkspaceInputChangeTier = WorkspaceInputChangeTier.Reuse; + #installationChange: IDaemonInstallationChange | undefined; #transitioning: boolean = false; /** Active while the transition owner holds the exclusive gate and loads or reloads the workspace graph. */ readonly #transitionProgress: AdmissionProgress = new AdmissionProgress(); @@ -153,10 +178,12 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { private constructor( options: IWorkspaceRequestLifecycleOptions, + repoRoot: string, fingerprint: IWorkspaceInputFingerprint, runtimeCache: WorkspaceRuntimeFingerprintCache ) { this.#options = options; + this.#repoRoot = repoRoot; this.#startupFingerprint = this.#fingerprint = fingerprint; this.#resolver = options.resolver; this.#ownedResolvers.add(options.resolver); @@ -174,7 +201,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { runtimePaths: [__dirname, path.resolve(__dirname, '../package.json')], runtimeCache }); - return new WorkspaceRequestLifecycle(options, fingerprint, runtimeCache); + return new WorkspaceRequestLifecycle(options, session.metadata.repoRoot, fingerprint, runtimeCache); } /** The last applied input decision; reading status never changes or reloads the workspace. */ @@ -193,23 +220,21 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { ...request, environment: { ...request.environment, - [EnvironmentVariableNames._RUSH_LIB_PATH]: getRushLibPathHandoff( - require.resolve('@microsoft/rush-lib'), - process.env[EnvironmentVariableNames._RUSH_LIB_PATH] - ) + [EnvironmentVariableNames._RUSH_LIB_PATH]: this.#rushLibPath } }; + this.#detectInstallationChange(); if (this.#restartPending) { await destination.interactiveSession.finishAsync(); - await destination.writeResultAsync({ - ...preExecutionFailure(envelope.requestId, new RestartPendingBeforeExecution()), - retryAfterRestart: true - }); + await destination.writeResultAsync( + this.#restartPendingResult(envelope.requestId, new RestartPendingBeforeExecution()) + ); return; } if (this.#closing) throw new Error('The workspace lifecycle is closing. No operation was scheduled or executed.'); - if (this.#cleanupFailure !== undefined) throw this.#cleanupFailure; + // A restart for a changed installation also replaces resources that could not be cleaned up. + if (this.#cleanupFailure !== undefined && !this.#installationChange) throw this.#cleanupFailure; const state: IExecutionState = { began: false, terminalAttempted: false, resultDrained: false }; const observer: AbortController | undefined = isGraphWatch(envelope) ? new AbortController() : undefined; if (observer) this.#observers.add(observer); @@ -233,12 +258,24 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { // Long-lived observers are cancelled by a transition, so they never delay a restart. A running rushx script does // delay one until it exits, so a client-default timeout still limits waiting for it; a script that arrives while a // restart is pending waits for that restart instead (#prepareAsync). - const ticket: IWorkspaceRestartTicket | undefined = observer + let ticket: IWorkspaceRestartTicket | undefined = observer ? undefined : this.#restartArbiter.enter({ runsScript: isRushxInvocation(envelope) }); let generation: IPreparedGeneration | undefined; try { for (let attempt: number = 0; ; attempt++) { + if (this.#installationChange) { + // An observer waits for the restart like any other request, so from here on the drain tracks it too. + ticket ??= this.#restartArbiter.enter(); + await this.#restartForInstallationAsync( + envelope, + client, + admission, + ticket, + this.#installationChange + ); + return; + } let scriptLease: IRequestLease | undefined; try { const prepared: IPreparedGeneration = await this.#prepareAsync( @@ -310,24 +347,16 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } if (error instanceof RestartPendingBeforeExecution && !state.began && !state.terminalAttempted) { await client.interactiveSession.finishAsync(); - await client.writeResultAsync({ - ...preExecutionFailure(envelope.requestId, error), - retryAfterRestart: true - }); + await client.writeResultAsync(this.#restartPendingResult(envelope.requestId, error)); return; } if (error instanceof RequestSchedulerError && !state.began && !state.terminalAttempted) { - await client.interactiveSession.finishAsync(); - await client.writeResultAsync({ - ...preExecutionFailure( - envelope.requestId, - getDaemonShutdownReason(client.abortSignal) ?? error - ), - aborted: client.abortSignal.aborted, - admissionErrorCode: getRequestAdmissionErrorCode(error) - }); + await writeAdmissionFailureAsync(envelope, client, error); return; } + // A request that failed before it began because the installation changed under it, for example on a + // module that the daemon could no longer load, waits for the restart like any other request. + if (!state.began && !state.terminalAttempted && this.#detectInstallationChange()) continue; if ( isFallbackRejection(error) && !state.began && @@ -406,6 +435,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { let ownsTransition: boolean = false; try { if (this.#restartPending) throw new RestartPendingBeforeExecution(); + this.#throwIfInstallationChanged(); if (this.#closing) throw new Error('The workspace is restarting. No operation was scheduled or executed.'); if (this.#cleanupFailure !== undefined) throw this.#cleanupFailure; @@ -448,6 +478,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } this.#cancelObservers(); lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); + this.#throwIfInstallationChanged(); await this.#waitForServedScriptsAsync(admission); session = await this.#options.provider.getSessionAsync(); await this.#quiesceWarmSetAsync(session); @@ -553,6 +584,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); this.#transitionProgress.setActive(true); if (this.#restartPending) throw new RestartPendingBeforeExecution(); + this.#throwIfInstallationChanged(); if (this.#closing) throw new Error('The workspace is restarting. No operation was scheduled or executed.'); session = await this.#options.provider.getSessionAsync(); @@ -932,6 +964,89 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } } + /** + * Returns the change when the daemon's installation was removed or replaced. Such a daemon cannot load the rest of + * its code, so from the first detection on it admits no more requests; each request waits in the restart drain for + * the requests that the daemon is serving and then gets a restart result instead. + */ + #detectInstallationChange(): IDaemonInstallationChange | undefined { + if (this.#installationChange || this.#closing) return this.#installationChange; + const change: IDaemonInstallationChange | undefined = this.#options.checkInstallation?.(); + if (!change) return undefined; + this.#installationChange = change; + this.#cancelObservers(); + this.#options.onLog?.( + `rushd: the installation at ${change.folder} was ${change.change}; exiting once running requests finish, ` + + 'so that the next client starts a new daemon' + ); + return change; + } + + #throwIfInstallationChanged(): void { + const change: IDaemonInstallationChange | undefined = this.#detectInstallationChange(); + if (change) throw new InstallationChangedBeforeExecution(change); + } + + /** + * Answers a request with a restart result once the requests that the daemon is serving finish (the restart drain), + * so that its client does not wait for this daemon to exit while a long build still runs. The first request to get + * there asks the host to exit without selecting a successor; each client then starts one with its own launcher. + */ + async #restartForInstallationAsync( + envelope: IDaemonRequestEnvelope, + client: IDaemonRequestDispatchClient, + admission: RequestAdmissionController, + ticket: IWorkspaceRestartTicket, + change: IDaemonInstallationChange + ): Promise { + const restartReason: DaemonRestartReason = { + kind: 'installationChanged', + change: change.change, + folder: change.folder + }; + let lease: IRequestLease; + try { + await admission.waitForRestartDrainAsync(this.#restartArbiter, ticket, restartReason); + lease = await admission.acquireBeforeRestartAsync(this.#gate, restartReason); + } catch (error) { + if (!(error instanceof RequestSchedulerError)) throw error; + await writeAdmissionFailureAsync(envelope, client, error); + return; + } + const restarting: boolean = !this.#restartPending; + this.#lastReloadTier = WorkspaceInputChangeTier.Restart; + this.#restartPending = true; + this.#closing = true; + try { + await client.interactiveSession.finishAsync(); + await client.writeResultAsync( + this.#restartPendingResult(envelope.requestId, new RestartPendingBeforeExecution()) + ); + } finally { + if (restarting) { + this.#options.onRestartRequested({ + repoRoot: this.#repoRoot, + rushVersion: this.#options.rushVersion, + environment: Object.freeze({ ...envelope.environment }), + reason: 'installation-changed', + launch: undefined, + failure: undefined + }); + } + lease.release(); + } + } + + #restartPendingResult(requestId: string, pending: RestartPendingBeforeExecution): IDaemonCommandResult { + const change: IDaemonInstallationChange | undefined = this.#installationChange; + if (!change) return { ...preExecutionFailure(requestId, pending), retryAfterRestart: true }; + return { + ...preExecutionFailure(requestId, new InstallationChangedBeforeExecution(change)), + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: change.change, folder: change.folder } + }; + } + #assertGeneration(generation: IPreparedGeneration): void { assertWorkspaceRequestResourcesHealthy(generation.session); generation.session.assertActive?.(); @@ -1046,6 +1161,19 @@ function mayFallBackAlongsideContinuingWork(envelope: IDaemonRequestEnvelope): b ); } +async function writeAdmissionFailureAsync( + envelope: IDaemonRequestEnvelope, + client: IDaemonRequestDispatchClient, + error: RequestSchedulerError +): Promise { + await client.interactiveSession.finishAsync(); + await client.writeResultAsync({ + ...preExecutionFailure(envelope.requestId, getDaemonShutdownReason(client.abortSignal) ?? error), + aborted: client.abortSignal.aborted, + admissionErrorCode: getRequestAdmissionErrorCode(error) + }); +} + function preExecutionFailure(requestId: string, error: Error): IDaemonCommandResult { return { requestId, exitCode: 1, outcome: 'failure', aborted: false, errorMessage: error.message }; } diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts index f431baf85c..b23c4a364c 100644 --- a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -4,6 +4,7 @@ import { RequestSchedulerError, RequestSchedulerErrorCode } from './RequestScheduler'; const MAX_TIMER_DELAY_MS: number = 0x7fffffff; +const ENVIRONMENT_RESTART_CAUSE: string = 'for its environment'; const SCRIPT_TIMEOUT_CLAUSE: string = ', including a rushx script that may not exit until it is stopped'; // Waiting longer may not help behind a script, such as a dev server, that runs until it is stopped. const SCRIPT_TIMEOUT_REMEDY: string = 'Stop the script, or use --wait-timeout to wait longer.'; @@ -62,6 +63,12 @@ export interface IWorkspaceRestartDrainOptions { * request waiting for as long as they keep arriving. */ readonly waivesTimeoutForServedWork?: boolean; + /** + * Why the daemon restarts, as the admission errors of {@link WorkspaceRestartArbiter.waitForDrainAsync} say it: + * "the daemon could restart ", for example `because its installation at /x was removed`. The default is + * `for its environment`. + */ + readonly restartCause?: string; /** * Called while the request waits with the number of other requests that it waits for, when the wait begins and * whenever that number changes, so that the client can report the wait as a queue position. @@ -126,13 +133,19 @@ export class WorkspaceRestartArbiter { ticket: IWorkspaceRestartTicket, options: IWorkspaceRestartDrainOptions ): Promise { + const { restartCause } = options; return await this.#waitAsync(ticket, options, { restarts: true, isBlocked: () => this.#serving.size > 0, countWaitedFor: () => this.#serving.size, - noWaitMessage: 'Another environment is still being served; the request did not wait for a restart.', + noWaitMessage: + restartCause === undefined + ? 'Another environment is still being served; the request did not wait for a restart.' + : `The daemon is still serving other requests, which finish before it restarts ${restartCause}; ` + + 'the request did not wait for a restart.', timeoutPrefix: - 'The request was not admitted before the daemon could restart for its environment, which waits for' + `The request was not admitted before the daemon could restart ${restartCause ?? ENVIRONMENT_RESTART_CAUSE}, ` + + 'which waits for' }); } diff --git a/libraries/rush-daemon/src/index.ts b/libraries/rush-daemon/src/index.ts index b895226872..a075f6aba5 100644 --- a/libraries/rush-daemon/src/index.ts +++ b/libraries/rush-daemon/src/index.ts @@ -57,6 +57,7 @@ export { type IGlobalCommandRequestResult } from './GlobalCommandRequestRouter'; export { RushDaemonHost, type IRushDaemonHostOptions } from './RushDaemonHost'; +export { captureDaemonInstallation, type CheckDaemonInstallation } from './DaemonInstallationMonitor'; export { DaemonShutdownError, type DaemonShutdownInitiator, diff --git a/libraries/rush-daemon/src/serveRushDaemon.ts b/libraries/rush-daemon/src/serveRushDaemon.ts index 1baec925df..5568767526 100644 --- a/libraries/rush-daemon/src/serveRushDaemon.ts +++ b/libraries/rush-daemon/src/serveRushDaemon.ts @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import { captureDaemonInstallation, getDaemonInstallationFolders } from './DaemonInstallationMonitor'; import { DaemonShutdownError } from './DaemonShutdownError'; import { RushDaemonHost } from './RushDaemonHost'; import type { IRushDaemonHostOptions } from './RushDaemonHost'; @@ -21,6 +22,11 @@ export interface IRushDaemonServeOptions extends IRushDaemonHostOptions { /** * Starts a daemon host, signals readiness, and serves until shutdown is requested. * + * @remarks + * Unless `checkInstallation` is given, the host checks the folders that this process loaded the daemon and the + * Rush engine from before each request. After one of them was removed or replaced, it starts no more requests + * and exits once running requests finish. + * * @beta */ export async function serveRushDaemonAsync(options: IRushDaemonServeOptions): Promise { @@ -31,6 +37,8 @@ export async function serveRushDaemonAsync(options: IRushDaemonServeOptions): Pr try { host = await RushDaemonHost.startAsync({ ...options, + checkInstallation: + options.checkInstallation ?? captureDaemonInstallation(getDaemonInstallationFolders()), getSuccessorLaunchAsync: options.getSuccessorLaunchAsync ?? (async (context) => { diff --git a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts index 84f3e26eff..d1c90fb8e2 100644 --- a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts +++ b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts @@ -23,6 +23,7 @@ import { RushDaemonRequestResolver } from '../RushDaemonRequestResolver'; import { WorkspaceSession } from '../WorkspaceSession'; import { getWorkspaceGenerationToken } from '../WorkspaceGeneration'; import type { GetWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import type { CheckDaemonInstallation } from '../DaemonInstallationMonitor'; import { createWireEnvelope, DaemonRequestWireClient, @@ -35,6 +36,9 @@ export class DaemonGraphTestFixture implements AsyncDisposable { public session!: WorkspaceSession; public host!: RushDaemonHost; public getSuccessorLaunchAsync: GetWorkspaceSuccessorLaunchAsync | undefined; + public checkInstallation: CheckDaemonInstallation | undefined; + /** Every message the host wrote to its daemon log. */ + public readonly logs: string[] = []; /** Awaited before each workspace session is created, including a request's graph load or reload. */ public beforeCreateSessionAsync: (() => Promise) | undefined; /** Also serves rushx package scripts, like the production host. Set it in `createAsync`'s `configure`. */ @@ -153,6 +157,8 @@ export class DaemonGraphTestFixture implements AsyncDisposable { rushVersion: Rush.version, daemonVersion: 'graph-test', getSuccessorLaunchAsync: this.getSuccessorLaunchAsync, + checkInstallation: this.checkInstallation, + onLog: (message: string) => this.logs.push(message), requestResolver: this._lifecycle ? resolver : { diff --git a/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts b/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts new file mode 100644 index 0000000000..febcf65b25 --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts @@ -0,0 +1,453 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { + DaemonFrameType, + decodeDaemonControlMessage, + type DaemonControlMessage, + type IDaemonCommandResult, + type IDaemonFrame, + type IDaemonInstallationChange, + type IDaemonRequestEnvelope, + type IDaemonRequestQueuePositionMessage +} from '@rushstack/rush-daemon-protocol'; + +import { captureDaemonInstallation, type CheckDaemonInstallation } from '../DaemonInstallationMonitor'; +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { + createDeferred, + type DaemonRequestWireClient, + type IDeferred, + type ITerminalExchange +} from './DaemonRequestWireTestUtilities'; +import { assertSuccessfulNativeBuild } from './NativeBuildTestResult'; +import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; + +jest.setTimeout(30_000); + +const BUILD_B: string[] = ['build', '--to', 'b', '--parallelism', '3']; +const LONG_BUILD_A: string = + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','a\\n');" + + "const wait=()=>fs.existsSync('../common/temp/release.flag')?console.log('finished-a'):setTimeout(wait,20);" + + 'wait();'; +// A script such as a dev server, which runs until it is stopped. +const SERVE_A: string = + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','serve\\n');" + + "const wait=()=>fs.existsSync('../common/temp/release.flag')?console.log('stopped'):setTimeout(wait,20);" + + 'wait();'; + +function queuePositions( + frames: ReadonlyArray +): IDaemonRequestQueuePositionMessage['payload'][] { + return frames + .filter((frame: IDaemonFrame) => frame.kind === DaemonFrameType.controlJson) + .map((frame: IDaemonFrame) => decodeDaemonControlMessage(frame.payload)) + .filter((message: DaemonControlMessage) => message.kind === 'queuePosition') + .map((message: DaemonControlMessage) => (message as IDaemonRequestQueuePositionMessage).payload); +} + +async function readUntilQueuedAsync(client: DaemonRequestWireClient): Promise { + const frames: IDaemonFrame[] = []; + while (queuePositions(frames).length === 0) frames.push(await client.readFrameAsync()); + return frames; +} + +async function waitForAsync(predicate: () => boolean): Promise { + while (!predicate()) await delayAsync(20); +} + +interface IInstallation { + readonly root: string; + readonly folder: string; + readonly lib: string; +} + +function createInstallation(): IInstallation { + const root: string = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-installation-'))); + const folder: string = path.join(root, 'daemon'); + const lib: string = path.join(folder, 'lib-commonjs'); + fs.mkdirSync(lib, { recursive: true }); + return { root, folder, lib }; +} + +describe(captureDaemonInstallation.name, () => { + let installation: IInstallation; + beforeEach(() => { + installation = createInstallation(); + }); + afterEach(() => { + fs.rmSync(installation.root, { recursive: true, force: true }); + }); + + it('ignores files that change inside the folders', () => { + const check: CheckDaemonInstallation = captureDaemonInstallation([installation.lib]); + fs.writeFileSync(path.join(installation.lib, 'index.js'), ''); + fs.mkdirSync(path.join(installation.lib, 'nested')); + fs.rmSync(path.join(installation.lib, 'index.js')); + expect(check()).toBeUndefined(); + }); + + it('reports the outermost folder that was renamed away, and nothing once it is back', () => { + const check: CheckDaemonInstallation = captureDaemonInstallation([installation.lib]); + fs.renameSync(installation.folder, `${installation.folder}.moved`); + expect(check()).toEqual({ change: 'removed', folder: installation.folder }); + fs.renameSync(`${installation.folder}.moved`, installation.folder); + expect(check()).toBeUndefined(); + }); + + it('reports a folder that another folder took the place of', () => { + const check: CheckDaemonInstallation = captureDaemonInstallation([installation.lib]); + // The old folder stays on disk, so the new one cannot reuse its inode. + fs.renameSync(installation.folder, `${installation.folder}.old`); + fs.mkdirSync(installation.lib, { recursive: true }); + expect(check()).toEqual({ change: 'replaced', folder: installation.folder }); + }); + + it('reports only the innermost folder when its parents are intact', () => { + const check: CheckDaemonInstallation = captureDaemonInstallation([installation.folder, installation.lib]); + fs.renameSync(installation.lib, `${installation.lib}.old`); + fs.mkdirSync(installation.lib); + expect(check()).toEqual({ change: 'replaced', folder: installation.lib }); + fs.rmSync(installation.lib, { recursive: true }); + expect(check()).toEqual({ change: 'removed', folder: installation.lib }); + }); + + it('does not watch a folder that was missing at startup', () => { + const missing: string = path.join(installation.root, 'missing'); + const check: CheckDaemonInstallation = captureDaemonInstallation([missing]); + fs.mkdirSync(missing); + expect(check()).toBeUndefined(); + }); +}); + +describe('a daemon whose installation changed', () => { + let installation: IInstallation; + let fixture: DaemonGraphTestFixture | undefined; + beforeEach(() => { + installation = createInstallation(); + }); + afterEach(async () => { + const created: DaemonGraphTestFixture | undefined = fixture; + fixture = undefined; + await created?.[Symbol.asyncDispose](); + fs.rmSync(installation.root, { recursive: true, force: true }); + }); + + it('asks the client to start a new daemon, exits without a successor, and says why', async () => { + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = captureDaemonInstallation([installation.lib]); + }); + await fixture.buildSuccessfullyAsync(); + expect(fixture.runs()).toEqual(['a', 'b']); + expect((await pongAsync(fixture)).installationChange).toBeUndefined(); + + fs.renameSync(installation.folder, `${installation.folder}.moved`); + expect((await pongAsync(fixture)).installationChange).toEqual({ + change: 'removed', + folder: installation.folder + }); + const { terminal } = await fixture.buildAsync(); + expect(terminal.kind).toBe('requestResult'); + const result: IDaemonCommandResult = terminal.payload as IDaemonCommandResult; + expect(result).toMatchObject({ + exitCode: 1, + outcome: 'failure', + aborted: false, + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'removed', folder: installation.folder } + }); + expect(result.errorMessage).toContain(`installation at ${installation.folder} was removed`); + + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + await fixture.host.closed; + expect(fixture.runs()).toEqual(['a', 'b']); + expect(fixture.logs).toEqual([ + `rushd: the installation at ${installation.folder} was removed; exiting once running requests finish, ` + + 'so that the next client starts a new daemon' + ]); + }); + + it('answers a request that arrives during a running build once that build finishes', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + created.write('a/build.cjs', LONG_BUILD_A); + }); + const running: Promise = fixture.buildAsync(); + await waitForAsync(() => fixture!.runs().includes('a')); + current.change = { change: 'replaced', folder: installation.folder }; + + // The build runs for longer than a client-default wait timeout, which must not fail the request. + let answered: boolean = false; + const queued: Promise = fixture + .runAsync(BUILD_B, { admission: { waitTimeoutMs: 100, waitTimeoutIsDefault: true } }) + .finally(() => { + answered = true; + }); + let restarted: boolean = false; + void fixture.host.restartCompleted.then(() => { + restarted = true; + }); + expect((await pongAsync(fixture)).installationChange).toEqual(current.change); + await delayAsync(300); + expect(answered).toBe(false); + expect(restarted).toBe(false); + + fixture.write('common/temp/release.flag', ''); + assertSuccessfulNativeBuild(await running, fixture.session.operationGraph); + const { frames, terminal } = await queued; + const restartReason: object = { + kind: 'installationChanged', + change: 'replaced', + folder: installation.folder + }; + expect(terminal.payload).toMatchObject({ retryAfterRestart: true, restartReason }); + expect(queuePositions(frames)).toEqual([expect.objectContaining({ position: 1, restartReason })]); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + await fixture.host.closed; + expect(fixture.runs()).toEqual(['a', 'b']); + expect(fixture.logs).toHaveLength(1); + }); + + it('applies an explicit wait timeout to that wait, and restarts for a later request', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + created.write('a/build.cjs', LONG_BUILD_A); + }); + const running: Promise = fixture.buildAsync(); + await waitForAsync(() => fixture!.runs().includes('a')); + current.change = { change: 'removed', folder: installation.folder }; + + const timedOut: ITerminalExchange = await fixture.runAsync(BUILD_B, { + admission: { waitTimeoutMs: 100 } + }); + expect(timedOut.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, admissionErrorCode: 'wait-timeout' } + }); + expect((timedOut.terminal.payload as IDaemonCommandResult).retryAfterRestart).toBeUndefined(); + const notWaiting: ITerminalExchange = await fixture.runAsync(BUILD_B, { admission: { noWait: true } }); + expect(notWaiting.terminal).toMatchObject({ payload: { exitCode: 1, admissionErrorCode: 'no-wait' } }); + + fixture.write('common/temp/release.flag', ''); + assertSuccessfulNativeBuild(await running, fixture.session.operationGraph); + let restarted: boolean = false; + void fixture.host.restartCompleted.then(() => { + restarted = true; + }); + await delayAsync(100); + expect(restarted).toBe(false); + + const { terminal } = await fixture.buildAsync(); + expect(terminal.payload).toMatchObject({ + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'removed', folder: installation.folder } + }); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + await fixture.host.closed; + expect(fixture.runs()).toEqual(['a', 'b']); + }); + + it('still applies a client-default timeout to that wait while a rushx script runs, and says why', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + created.servesRushx = true; + created.write( + 'a/package.json', + JSON.stringify({ + name: 'a', + version: '1.0.0', + scripts: { '_phase:compile': 'node build.cjs', serve: 'node serve.cjs' } + }) + ); + created.write('a/serve.cjs', SERVE_A); + }); + const serving: Promise = fixture.runAsync(['serve'], { + invocationKind: 'rushx', + commandOrigin: 'custom', + cwd: path.join(fixture.folder, 'a') + }); + await Promise.race([ + waitForAsync(() => fixture!.runs().includes('serve')), + serving.then((exchange: ITerminalExchange) => { + throw new Error(`The script ended before it ran: ${JSON.stringify(exchange.terminal)}`); + }) + ]); + current.change = { change: 'replaced', folder: installation.folder }; + + const timedOut: ITerminalExchange = await fixture.runAsync(BUILD_B, { + admission: { waitTimeoutMs: 100, waitTimeoutIsDefault: true } + }); + expect(timedOut.terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, admissionErrorCode: 'wait-timeout' } + }); + const result: IDaemonCommandResult = timedOut.terminal.payload as IDaemonCommandResult; + expect(result.retryAfterRestart).toBeUndefined(); + expect(result.errorMessage).toContain( + `could restart because its installation at ${installation.folder} was replaced, which waits for the ` + + 'requests that the daemon is serving to finish, including a rushx script that may not exit until it is ' + + 'stopped. Stop the script, or use --wait-timeout to wait longer.' + ); + + fixture.write('common/temp/release.flag', ''); + expect((await serving).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + const { terminal } = await fixture.buildAsync(); + expect(terminal.payload).toMatchObject({ + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'replaced', folder: installation.folder } + }); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + await fixture.host.closed; + expect(fixture.runs()).toEqual(['serve']); + }); + + it('answers a request that waited behind a graph load with the restart instead of running it', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + }); + const loadStarted: IDeferred = createDeferred(); + const releaseLoad: IDeferred = createDeferred(); + // Only the first build's graph load waits; the one that shuts the daemon down does not. + let loaded: boolean = false; + fixture.beforeCreateSessionAsync = async () => { + if (loaded) return; + loaded = true; + loadStarted.resolve(); + await releaseLoad.promise; + }; + const loading: Promise = fixture.buildAsync(); + await loadStarted.promise; + const client: DaemonRequestWireClient = await fixture.connectAsync(); + try { + const envelope: IDaemonRequestEnvelope = fixture.envelope(BUILD_B); + await client.sendControlAsync({ kind: 'requestStart', payload: envelope }); + const frames: IDaemonFrame[] = await readUntilQueuedAsync(client); + current.change = { change: 'removed', folder: installation.folder }; + releaseLoad.resolve(); + + // The load was admitted before the change, so its build runs on the code that is already loaded. + assertSuccessfulNativeBuild(await loading, fixture.session.operationGraph); + const exchange: ITerminalExchange = await client.readTerminalAsync(envelope.requestId); + expect(exchange.terminal).toMatchObject({ + kind: 'requestResult', + payload: { + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'removed', folder: installation.folder } + } + }); + expect(queuePositions([...frames, ...exchange.frames])[0]).not.toHaveProperty('restartReason'); + } finally { + releaseLoad.resolve(); + await client.closeAsync(); + } + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + expect(fixture.runs()).toEqual(['a', 'b']); + }); + + it('answers a request that waited to reload the graph with the restart instead of reloading', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + created.write('a/build.cjs', LONG_BUILD_A); + }); + const running: Promise = fixture.buildAsync(); + await waitForAsync(() => fixture!.runs().includes('a')); + const packageJsonPath: string = path.join(fixture.folder, 'c/package.json'); + const packageJson: Record = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8')); + fs.writeFileSync(packageJsonPath, JSON.stringify({ ...packageJson, description: 'changed' })); + const client: DaemonRequestWireClient = await fixture.connectAsync(); + try { + // Needs a reload, so it waits for exclusive admission until the running build ends. + const envelope: IDaemonRequestEnvelope = fixture.envelope(BUILD_B); + await client.sendControlAsync({ kind: 'requestStart', payload: envelope }); + await readUntilQueuedAsync(client); + current.change = { change: 'replaced', folder: installation.folder }; + fixture.write('common/temp/release.flag', ''); + + assertSuccessfulNativeBuild(await running, fixture.session.operationGraph); + expect((await client.readTerminalAsync(envelope.requestId)).terminal).toMatchObject({ + kind: 'requestResult', + payload: { + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'replaced', folder: installation.folder } + } + }); + } finally { + await client.closeAsync(); + } + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + expect(fixture.runs()).toEqual(['a', 'b']); + }); + + it('answers a request that the change broke before it began with the restart', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + }); + // Only the first build's graph load fails; the one that shuts the daemon down succeeds. + fixture.beforeCreateSessionAsync = async () => { + if (current.change) return; + current.change = { change: 'removed', folder: installation.folder }; + throw new Error("Cannot find module 'validate-npm-package-name'"); + }; + const { terminal } = await fixture.buildAsync(); + expect(terminal).toMatchObject({ + kind: 'requestResult', + payload: { + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'removed', folder: installation.folder } + } + }); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + await fixture.host.closed; + expect(fixture.runs()).toEqual([]); + expect(fixture.logs).toEqual([ + `rushd: the installation at ${installation.folder} was removed; exiting once running requests finish, ` + + 'so that the next client starts a new daemon' + ]); + }); + + it('keeps serving while its installation is intact', async () => { + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = captureDaemonInstallation([installation.lib]); + }); + await fixture.buildSuccessfullyAsync(); + fs.writeFileSync(path.join(installation.lib, 'late.js'), ''); + await fixture.buildSuccessfullyAsync(); + expect(fixture.logs).toEqual([]); + }); +}); + +it('logs each rejected request, with the stack of an unexpected failure', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); + try { + const { terminal } = await fixture.graphAsync('pause'); + expect(terminal).toMatchObject({ kind: 'requestRejected', payload: { code: 'routingFailed' } }); + const { requestId, message } = terminal.payload as { requestId: string; message: string }; + expect(fixture.logs).toHaveLength(1); + expect(fixture.logs[0]).toMatch( + new RegExp(`^rushd: rejected request ${requestId} \\(routingFailed\\): `) + ); + expect(fixture.logs[0]).toContain(message); + expect(fixture.logs[0]).toMatch(/\n\s+at /); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); diff --git a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts index 328356e3ac..ccdf2c637a 100644 --- a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts +++ b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts @@ -4,11 +4,18 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; import type { + DaemonRestartReason, IDaemonRequestAdmissionOptions, IDaemonRequestQueuePositionMessage } from '@rushstack/rush-daemon-protocol'; -import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestScheduler'; +import { + type IRequestLease, + RequestExclusivityClass, + RequestScheduler, + RequestSchedulerError, + RequestSchedulerErrorCode +} from '../RequestScheduler'; import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; import { WorkspaceRestartArbiter, @@ -16,6 +23,12 @@ import { type IWorkspaceRestartTicketOptions } from '../WorkspaceRestartArbiter'; +const INSTALLATION_REMOVED: DaemonRestartReason = { + kind: 'installationChanged', + change: 'removed', + folder: '/old/daemon' +}; + interface IDrainTest { readonly admission: RequestAdmissionController; readonly arbiter: WorkspaceRestartArbiter; @@ -23,6 +36,17 @@ interface IDrainTest { readonly ticket: IWorkspaceRestartTicket; } +function createCandidate( + options: IDaemonRequestAdmissionOptions, + abortSignal: AbortSignal = new AbortController().signal +): RequestAdmissionController { + return new RequestAdmissionController({ + admission: options, + client: { abortSignal }, + requestId: 'restart-candidate' + }); +} + /** A restart candidate with the given admission options, and one other request that the daemon is serving. */ function createDrainTest( options: IDaemonRequestAdmissionOptions, @@ -31,12 +55,7 @@ function createDrainTest( const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const serving: IWorkspaceRestartTicket = arbiter.enter(servingOptions); const ticket: IWorkspaceRestartTicket = arbiter.enter(); - const admission: RequestAdmissionController = new RequestAdmissionController({ - admission: options, - client: { abortSignal: new AbortController().signal }, - requestId: 'restart-candidate' - }); - return { admission, arbiter, serving, ticket }; + return { admission: createCandidate(options), arbiter, serving, ticket }; } async function isSettledAsync(promise: Promise): Promise { @@ -148,6 +167,106 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { admission.dispose(); expect(arbiter.servingCount).toBe(0); }); + + it('names a restart reason in its queue positions and its admission errors', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const ticket: IWorkspaceRestartTicket = arbiter.enter(); + const restartReason: DaemonRestartReason = INSTALLATION_REMOVED; + const positions: IDaemonRequestQueuePositionMessage['payload'][] = []; + const createAdmission = (options: IDaemonRequestAdmissionOptions): RequestAdmissionController => + new RequestAdmissionController({ + admission: options, + client: { + abortSignal: new AbortController().signal, + supportsRequestAdmission: true, + writeQueuePositionAsync: async (message: IDaemonRequestQueuePositionMessage) => { + positions.push(message.payload); + } + }, + requestId: 'restart-candidate' + }); + + const notWaiting: RequestAdmissionController = createAdmission({ noWait: true }); + const noWaitError: unknown = await notWaiting + .waitForRestartDrainAsync(arbiter, ticket, restartReason) + .catch((caught: unknown) => caught); + expect((noWaitError as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.NoWait); + expect((noWaitError as Error).message).toBe( + 'The daemon is still serving other requests, which finish before it restarts because its installation at ' + + '/old/daemon was removed; the request did not wait for a restart.' + ); + notWaiting.dispose(); + + const waiting: RequestAdmissionController = createAdmission({ waitTimeoutMs: 50 }); + const timeoutError: unknown = await waiting + .waitForRestartDrainAsync(arbiter, ticket, restartReason) + .catch((caught: unknown) => caught); + expect((timeoutError as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + expect((timeoutError as Error).message).toBe( + 'The request was not admitted before the daemon could restart because its installation at /old/daemon was ' + + 'removed, which waits for the requests that the daemon is serving to finish. Use --wait-timeout ' + + ' to wait longer.' + ); + expect(positions).toEqual([{ position: 1, requestId: 'restart-candidate', restartReason }]); + waiting.dispose(); + arbiter.leave(ticket); + arbiter.leave(serving); + expect(arbiter.servingCount).toBe(0); + }); +}); + +describe('RequestAdmissionController.acquireBeforeRestartAsync', () => { + let scheduler: RequestScheduler; + // After the drain, only requests that it does not track, such as observers that are winding down, can still + // hold the workspace. + let untracked: IRequestLease; + + beforeEach(async () => { + scheduler = new RequestScheduler(); + untracked = await scheduler.acquireAsync({ exclusivityClass: RequestExclusivityClass.SharedBuild }); + }); + + it('waits behind requests that the drain does not track, then admits the request exclusively', async () => { + const admission: RequestAdmissionController = createCandidate({ + waitTimeoutMs: 5000, + waitTimeoutIsDefault: true + }); + const acquiring: Promise = admission.acquireBeforeRestartAsync( + scheduler, + INSTALLATION_REMOVED + ); + await delayAsync(50); + expect(await isSettledAsync(acquiring)).toBe(false); + untracked.release(); + const lease: IRequestLease = await acquiring; + expect(lease.exclusivityClass).toBe(RequestExclusivityClass.Exclusive); + lease.release(); + admission.dispose(); + }); + + it('still applies a client-default timeout to that wait, and names the restart reason', async () => { + const abortController: AbortController = new AbortController(); + const admission: RequestAdmissionController = createCandidate( + { waitTimeoutMs: 50, waitTimeoutIsDefault: true }, + abortController.signal + ); + const acquiring: Promise = admission + .acquireBeforeRestartAsync(scheduler, INSTALLATION_REMOVED) + .catch((caught: unknown) => caught); + // A wait without a deadline fails here rather than at the test timeout. + const error: unknown = await Promise.race([acquiring, delayAsync(1000, 'still waiting')]); + abortController.abort(); + await acquiring; + expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); + expect((error as Error).message).toBe( + 'The request was not admitted within its 50ms wait timeout while waiting for the running requests to ' + + 'finish before the daemon restarts because its installation at /old/daemon was removed. Use ' + + '--wait-timeout to wait longer.' + ); + untracked.release(); + admission.dispose(); + }); }); interface IPendingRestartTest { diff --git a/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts b/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts index 0b3437cb10..82bc08b203 100644 --- a/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts +++ b/libraries/rush-lib/src/utilities/test/RushLibPathHandoff.test.ts @@ -53,7 +53,10 @@ describe(getRushLibPathHandoff.name, () => { // A "rush deploy" layout: rush-lib is a local project folder, and each host links it. localRushLib = path.join(folder, 'deploy', 'libraries', 'rush-lib'); writePackage(localRushLib, '@microsoft/rush-lib'); - writePackage(path.join(folder, 'deploy', 'libraries', 'node-core-library'), '@rushstack/node-core-library'); + writePackage( + path.join(folder, 'deploy', 'libraries', 'node-core-library'), + '@rushstack/node-core-library' + ); link( path.join(folder, 'deploy', 'libraries', 'node-core-library'), path.join(localRushLib, 'node_modules', '@rushstack', 'node-core-library') @@ -84,6 +87,26 @@ describe(getRushLibPathHandoff.name, () => { ).toEqual({ entryPoint, packageFolder: installedRushLib }); }); + it('keeps the real path of an installed rush-lib that the host script also links', () => { + const installedRushLib: string = path.join(folder, 'install', 'node_modules', '@microsoft', 'rush-lib'); + writePackage(installedRushLib, '@microsoft/rush-lib'); + const entryPoint: string = path.join(installedRushLib, ENTRY_POINT_SUBPATH); + // A pnpm-style host, whose own node_modules links the same installed package. + const linkingHostScript: string = path.join(folder, 'linking-host', 'bin', 'host'); + fs.mkdirSync(path.dirname(linkingHostScript), { recursive: true }); + fs.writeFileSync(linkingHostScript, ''); + link(installedRushLib, path.join(folder, 'linking-host', 'node_modules', '@microsoft', 'rush-lib')); + + expect( + getRushLibPathHandoff({ + packageFolder: installedRushLib, + entryPoint, + hostScriptPaths: [linkingHostScript], + inheritedEntryPoint: undefined + }) + ).toEqual({ entryPoint, packageFolder: installedRushLib }); + }); + it('spells a local rush-lib through the node_modules link of the host script', () => { const handoff: IRushLibPathHandoff = getRushLibPathHandoff({ packageFolder: localRushLib, @@ -92,13 +115,18 @@ describe(getRushLibPathHandoff.name, () => { inheritedEntryPoint: undefined }); - expect(handoff).toEqual({ entryPoint: path.join(hostLink, ENTRY_POINT_SUBPATH), packageFolder: hostLink }); + expect(handoff).toEqual({ + entryPoint: path.join(hostLink, ENTRY_POINT_SUBPATH), + packageFolder: hostLink + }); // This is how plugins find rush-lib and its dependencies from _RUSH_LIB_PATH. - expect(fs.realpathSync.native(resolveByName('@microsoft/rush-lib/package.json', handoff.entryPoint))).toBe( - path.join(localRushLib, 'package.json') - ); expect( - fs.realpathSync.native(resolveByName('@rushstack/node-core-library/package.json', handoff.packageFolder)) + fs.realpathSync.native(resolveByName('@microsoft/rush-lib/package.json', handoff.entryPoint)) + ).toBe(path.join(localRushLib, 'package.json')); + expect( + fs.realpathSync.native( + resolveByName('@rushstack/node-core-library/package.json', handoff.packageFolder) + ) ).toBe(path.join(folder, 'deploy', 'libraries', 'node-core-library', 'package.json')); }); From 1f2c904bbef4ae392dba6d584afbbcaffe5b1bf2 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:15:34 +0000 Subject: [PATCH 050/265] [rush-daemon] A daemon-served build logs one native telemetry entry per request, and the daemon keeps serving when its own PATH repeats an entry Swarm integration step 35; original commit d87702c813 (merge of swarm/r02-int at c9d60eb5a4). Scope: task 49 and the PATH fix. Brings r02's task 49 (daemon-served builds emitted no Rush command telemetry, board 61): a long-lived engine logs one native telemetry entry per served build request, with the request id, and an iteration's requests are reported before the next iteration starts. Engine disposal waits at most 2 seconds for telemetry uploads (8eccf3a6f1, CONFIRMED by ch05 board 1987). The daemon keeps serving after a plugin adds names to process.env (80704de4e3), and keeps serving when its own startup PATH repeats an entry, where it used to turn away every build (202412e57b, board 2657, CONFIRMED by o05 board 2671; the blobs carry over byte for byte, board 2776). Re-tipped onto integration c0e17d82ab with ch01's order for the 125 conflict in WorkspaceRequestLifecycle.ts (board 2756, board 2774); 108(a)'s 'running' early result counts as early in the telemetry entry. ch01 on b8b1dc6e2f (board 2795): build rc 0 with 0 warnings, rush-daemon 607/0, rush-lib 1249/0, rush-cli-client 398/0, rush-client-core 136/0, rush-daemon-protocol 200/0, rush-daemon-transport 81/0. Reverting the PATH fix or the 2 s flush wait each fails its test; 3 of 5 mutants killed, and the 2 survivors are test NITs for s16. E2E: the early row logs earlyResult true, and a PATH that repeats an entry keeps the same daemon where the old product falls back. r02's own run on c9d60eb5a4 (board 2777) gives the same counts. ch01's finding that ODSP_TELEMETRY_TAG restarts the daemon is pre-existing and is task 168. Gate: ch01 GATE OK board 2795 (tree b8b1dc6e2f) Commits folded into this step (10): - 490b3d94f9 [rush-lib] Extract the phased telemetry entry builder from OperationGraph - 251d987b06 [rush-lib] Let a long-lived engine log one native telemetry entry per request - 6f62ff03f2 [rush-daemon] Log one native telemetry entry per served build request - eb0fd9090e Add change files for per-request daemon telemetry - 80704de4e3 [rush-daemon] Keep serving after a plugin adds names to process.env - 6a17f4bbdc [rush-daemon] Test that an iteration's requests are reported before the next iteration starts - e7b848d6a7 [rush-daemon] Add the request id to each served request's telemetry entry - 8eccf3a6f1 [rush-lib] Wait at most 2 seconds for telemetry uploads when an engine is disposed - ddff7a24ff [rush-daemon] Move the native engine test fixture into its own module - 202412e57b [rush-daemon] Keep serving when the daemon's own PATH repeats an entry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...on-request-telemetry_2026-09-28-13-25.json | 10 + ...telemetry-flush-wait_2026-09-28-19-20.json | 10 + ...n-plugin-environment_2026-09-28-14-10.json | 10 + ...-repeated-path-entry_2026-09-29-00-00.json | 10 + ...on-request-telemetry_2026-09-28-13-25.json | 10 + ...rushd-telemetry-request-id_2026-09-28.json | 11 + common/reviews/api/rush-daemon.api.md | 44 +- common/reviews/api/rush-lib.api.md | 28 ++ .../src/DaemonRequestDispatcher.ts | 29 +- .../rush-daemon/src/DaemonRequestTelemetry.ts | 222 +++++++++ .../rush-daemon/src/PhasedRequestRouter.ts | 121 +++++ .../rush-daemon/src/PhasedRequestTelemetry.ts | 145 ++++++ .../src/ProductionDaemonRequestResolver.ts | 96 +++- .../src/WorkspaceRequestLifecycle.ts | 20 +- libraries/rush-daemon/src/index.ts | 6 + .../src/test/NativeEngineTestFixture.ts | 346 +++++++++++++ .../src/test/PhasedRequestBatching.test.ts | 1 + .../src/test/PhasedRequestTelemetry.test.ts | 454 +++++++++++++++++ ...roductionDaemonEnvironmentIdentity.test.ts | 100 +++- .../ProductionDaemonRequestResolver.test.ts | 465 ++++++------------ .../test/WorkspacePluginEnvironment.test.ts | 40 ++ .../rush-lib/src/api/PhasedCommandEngine.ts | 150 +++++- .../test/PhasedCommandEngineTelemetry.test.ts | 256 ++++++++++ .../cli/scriptActions/PhasedScriptAction.ts | 32 +- libraries/rush-lib/src/index.ts | 3 + libraries/rush-lib/src/logic/Telemetry.ts | 17 +- .../src/logic/operations/OperationGraph.ts | 139 +----- .../operations/PhasedCommandTelemetry.ts | 168 +++++++ .../rush-lib/src/logic/test/Telemetry.test.ts | 47 +- 29 files changed, 2501 insertions(+), 489 deletions(-) create mode 100644 common/changes/@microsoft/rush/daemon-request-telemetry_2026-09-28-13-25.json create mode 100644 common/changes/@microsoft/rush/daemon-telemetry-flush-wait_2026-09-28-19-20.json create mode 100644 common/changes/@rushstack/rush-daemon/daemon-plugin-environment_2026-09-28-14-10.json create mode 100644 common/changes/@rushstack/rush-daemon/daemon-repeated-path-entry_2026-09-29-00-00.json create mode 100644 common/changes/@rushstack/rush-daemon/daemon-request-telemetry_2026-09-28-13-25.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-telemetry-request-id_2026-09-28.json create mode 100644 libraries/rush-daemon/src/DaemonRequestTelemetry.ts create mode 100644 libraries/rush-daemon/src/PhasedRequestTelemetry.ts create mode 100644 libraries/rush-daemon/src/test/NativeEngineTestFixture.ts create mode 100644 libraries/rush-daemon/src/test/PhasedRequestTelemetry.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspacePluginEnvironment.test.ts create mode 100644 libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts create mode 100644 libraries/rush-lib/src/logic/operations/PhasedCommandTelemetry.ts diff --git a/common/changes/@microsoft/rush/daemon-request-telemetry_2026-09-28-13-25.json b/common/changes/@microsoft/rush/daemon-request-telemetry_2026-09-28-13-25.json new file mode 100644 index 0000000000..c5d1dc96ca --- /dev/null +++ b/common/changes/@microsoft/rush/daemon-request-telemetry_2026-09-28-13-25.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Let a long-lived phased command engine log one native telemetry entry per request through `IPhasedCommandEngine.logTelemetry()` and `createTelemetryData()`, using the request's own parameters and a host-supplied time origin. Give every flushed telemetry file a distinct name, so entries flushed within the same millisecond no longer overwrite each other.", + "type": "minor" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@microsoft/rush/daemon-telemetry-flush-wait_2026-09-28-19-20.json b/common/changes/@microsoft/rush/daemon-telemetry-flush-wait_2026-09-28-19-20.json new file mode 100644 index 0000000000..54062a88f0 --- /dev/null +++ b/common/changes/@microsoft/rush/daemon-telemetry-flush-wait_2026-09-28-19-20.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "When a long-lived phased command engine is disposed, wait at most 2 seconds for `flushTelemetry` taps that are still running, such as an upload over a stalled network. A slower tap keeps running in the background instead of holding the engine's host.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-daemon/daemon-plugin-environment_2026-09-28-14-10.json b/common/changes/@rushstack/rush-daemon/daemon-plugin-environment_2026-09-28-14-10.json new file mode 100644 index 0000000000..e578312bc5 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/daemon-plugin-environment_2026-09-28-14-10.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Keep serving requests after a Rush plugin adds its own variables to the daemon's `process.env`, as native Rush keeps them for the rest of a command. A change to, or removal of, a variable the daemon started with is still rejected, now with a message that names it.", + "type": "minor" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/daemon-repeated-path-entry_2026-09-29-00-00.json b/common/changes/@rushstack/rush-daemon/daemon-repeated-path-entry_2026-09-29-00-00.json new file mode 100644 index 0000000000..ff5155a9e7 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/daemon-repeated-path-entry_2026-09-29-00-00.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Keep serving requests when the daemon's own `PATH` repeats an entry. The check for a change to the daemon's startup environment compared the live `PATH` with its value without repeated entries, so it rejected every build and rebuild.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/daemon-request-telemetry_2026-09-28-13-25.json b/common/changes/@rushstack/rush-daemon/daemon-request-telemetry_2026-09-28-13-25.json new file mode 100644 index 0000000000..850f294935 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/daemon-request-telemetry_2026-09-28-13-25.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Log one native Rush telemetry entry for each served build request, including coalesced, early and no-op requests, with the request's own selection and daemon fields such as generation, reload tier, batch size, queue wait and request duration.", + "type": "minor" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-telemetry-request-id_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-telemetry-request-id_2026-09-28.json new file mode 100644 index 0000000000..a624761e19 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-telemetry-request-id_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Add the request's `requestId` to the native telemetry entry that the daemon logs for each served request, so that it can be joined with the per-operation data that plugins attribute through `getOperationRequestId`.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} \ No newline at end of file diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index 3dd40483e3..e9fa05e2cf 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -30,12 +30,14 @@ import type { IDaemonWorkspaceStatus } from '@rushstack/rush-daemon-protocol'; import type { IInputsSnapshot } from '@microsoft/rush-lib'; import { IOperationGraph } from '@microsoft/rush-lib'; import type { IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; +import type { IPhasedCommandEngineTelemetryRecord } from '@microsoft/rush-lib'; import type { ITerminal } from '@rushstack/terminal'; import type { LockFile } from '@rushstack/node-core-library'; import { Operation } from '@microsoft/rush-lib'; import { RushConfiguration } from '@microsoft/rush-lib'; import type { RushConfigurationProject } from '@microsoft/rush-lib'; import type { RushSession } from '@microsoft/rush-lib'; +import type { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; // @beta export function captureDaemonInstallation(folders: ReadonlyArray): CheckDaemonInstallation; @@ -188,6 +190,13 @@ export interface IDaemonRequestLifecycle extends AsyncDisposable { dispatchAsync(envelope: IDaemonRequestEnvelope, client: IDaemonRequestDispatchClient, dispatchAsync: DispatchWorkspaceRequestAsync): Promise; } +// @beta +export interface IDaemonRequestLifecycleInfo { + readonly preparedTimeMs: number; + readonly receivedTimeMs: number; + readonly reloadTier: WorkspaceInputChangeTier; +} + // @beta export interface IDaemonRequestResolver { // (undocumented) @@ -211,9 +220,9 @@ export interface IDispatchWorkspaceRequestOptions { readonly client: IDaemonRequestDispatchClient; // (undocumented) readonly envelope: IDaemonRequestEnvelope; + readonly lifecycleInfo?: IDaemonRequestLifecycleInfo; // (undocumented) readonly onExecutionStarting?: () => void; - readonly receivedTimeMs?: number; // (undocumented) readonly resolver: IDaemonRequestResolver | undefined; // (undocumented) @@ -387,6 +396,34 @@ export interface IPhasedRequestClient { writeTerminalPolicyAsync(result: IDaemonTerminalPolicyResult): Promise; } +// @beta +export interface IPhasedRequestTelemetryMeasure { + readonly endTimeMs: number; + readonly name: string; + readonly startTimeMs: number; +} + +// @beta +export interface IPhasedRequestTelemetryReport { + readonly batchSize: number; + readonly countRetained: number; + readonly earlyResult: boolean; + readonly executionStartTimeMs: number; + readonly iterationStartTimeMs: number | undefined; + readonly measures: ReadonlyArray; + readonly receivedTimeMs: number; + readonly records: ReadonlyMap; + readonly request: IDaemonPhasedRequest; + readonly result: IDaemonPhasedRequestResult; + readonly resultTimeMs: number; + readonly scheduled: boolean; +} + +// @beta +export interface IPhasedRequestTelemetrySink { + logRequest(report: IPhasedRequestTelemetryReport): void; +} + // @public export interface IRequestLease { // (undocumented) @@ -409,6 +446,7 @@ export interface IResolveDaemonRequestOptions { readonly abortSignal: AbortSignal; // (undocumented) readonly envelope: IDaemonRequestEnvelope; + readonly lifecycleInfo?: IDaemonRequestLifecycleInfo; // (undocumented) readonly workspaceSession: IWorkspaceSession; } @@ -429,6 +467,7 @@ export interface IResolvedDaemonPhasedRequest { // (undocumented) readonly request: IDaemonPhasedRequest; readonly requestSettings?: IPhasedCommandEngineRequestSettings; + readonly telemetry?: IPhasedRequestTelemetrySink; } // @beta @@ -701,7 +740,7 @@ export type MapWorkspaceInvalidationsToOperationsAsync = (options: IMapWorkspace // @beta export class PhasedRequestRouter { constructor(workspaceSession: IWorkspaceSession); - executeAsync(request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection?: boolean, onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings, receivedTimeMs?: number): Promise; + executeAsync(request: IDaemonPhasedRequest, client: IPhasedRequestClient, exactSelection?: boolean, onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings, telemetry?: IPhasedRequestTelemetrySink, receivedTimeMs?: number): Promise; } // @beta @@ -709,6 +748,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { constructor(options?: { readonly preparationLock?: LockFile; readonly validateGraphInputsAsync?: () => Promise; + readonly startupEnvironment?: Readonly>; }); createForSession(preparationLock?: LockFile, validateGraphInputsAsync?: () => Promise): ProductionDaemonRequestResolver; getCommandParameterIdentityAsync(options: IResolveDaemonRequestOptions): Promise; diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 55724ef1d0..045cf759e9 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -959,6 +959,7 @@ export interface IPhasedCommandEngine extends AsyncDisposable { readonly inputsSnapshot: IInputsSnapshot; // (undocumented) readonly isIncremental: boolean; + readonly logTelemetry?: (data: ITelemetryData, options?: IPhasedCommandEngineLogTelemetryOptions) => void; // (undocumented) readonly operationGraph: IOperationGraph; // (undocumented) @@ -969,6 +970,11 @@ export interface IPhasedCommandEngine extends AsyncDisposable { readonly rushSession: RushSession; } +// @alpha +export interface IPhasedCommandEngineLogTelemetryOptions { + readonly servedByIteration?: boolean; +} + // @alpha export interface IPhasedCommandEngineRequestSettings { // (undocumented) @@ -977,6 +983,27 @@ export interface IPhasedCommandEngineRequestSettings { readonly quietMode: boolean; } +// @alpha +export interface IPhasedCommandEngineTelemetryOptions { + readonly durationInSeconds: number; + readonly extraData?: Readonly>; + readonly performanceEntries?: ReadonlyArray; + readonly records: ReadonlyMap; + readonly succeeded: boolean; + readonly timeOriginMs: number; +} + +// @alpha +export interface IPhasedCommandEngineTelemetryRecord { + readonly nonCachedDurationMs: number | undefined; + readonly silent: boolean; + readonly status: OperationStatus; + readonly stopwatch: { + readonly startTime: number | undefined; + readonly endTime: number | undefined; + }; +} + // @alpha export interface IPhasedCommandPlugin { apply(hooks: PhasedCommandHooks): void; @@ -1531,6 +1558,7 @@ export class PhasedCommandEngine { // (undocumented) readonly commandName: string; createEngineAsync(preparationLock?: LockFile): Promise; + createTelemetryData(options: IPhasedCommandEngineTelemetryOptions): ITelemetryData; // (undocumented) readonly parameterIdentity: string; // (undocumented) diff --git a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts index 8a72ae6f9c..60aaa0c4d4 100644 --- a/libraries/rush-daemon/src/DaemonRequestDispatcher.ts +++ b/libraries/rush-daemon/src/DaemonRequestDispatcher.ts @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import type { IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; +import type { IPhasedCommandEngineRequestSettings, WorkspaceInputChangeTier } from '@microsoft/rush-lib'; import type { IDaemonCommandResult, IDaemonEventEnvelope, @@ -19,6 +19,7 @@ import type { IInteractiveRequestSession } from './InteractiveRequestInputRouter import type { IPhasedRequestClient } from './PhasedRequestClient'; import { PhasedRequestRouter } from './PhasedRequestRouter'; import type { IGlobalCommandRequestClient } from './GlobalCommandRequestClient'; +import type { IPhasedRequestTelemetrySink } from './PhasedRequestTelemetry'; import type { IWorkspaceSession } from './WorkspaceSession'; import { DaemonGraphRequestRouter } from './DaemonGraphRequestRouter'; import { getDaemonGraphObserver } from './DaemonGraphObserver'; @@ -35,6 +36,8 @@ export interface IResolvedDaemonPhasedRequest { readonly exactSelection?: boolean; /** Verbosity and parallelism for this request; applied to the shared graph before its iteration. */ readonly requestSettings?: IPhasedCommandEngineRequestSettings; + /** Receives the request's telemetry report once it has taken part in a graph iteration or no-op check. */ + readonly telemetry?: IPhasedRequestTelemetrySink; } /** A resolver outcome that uses the existing isolated global executor contract. @beta */ @@ -43,11 +46,23 @@ export interface IResolvedDaemonGlobalRequest { readonly kind: 'global'; } +/** How the host lifecycle admitted one request, for request-scoped telemetry. @beta */ +export interface IDaemonRequestLifecycleInfo { + /** The `performance.now()` timestamp at which the lifecycle received the request. */ + readonly receivedTimeMs: number; + /** The `performance.now()` timestamp at which the lifecycle had prepared the request's workspace generation. */ + readonly preparedTimeMs: number; + /** How the lifecycle reconciled the workspace inputs for this request. */ + readonly reloadTier: WorkspaceInputChangeTier; +} + /** Context supplied to an integration-owned request resolver. @beta */ export interface IResolveDaemonRequestOptions { /** Aborts when the request is cancelled, disconnected, or the host shuts down. */ readonly abortSignal: AbortSignal; readonly envelope: IDaemonRequestEnvelope; + /** Present when a host lifecycle admitted the request. */ + readonly lifecycleInfo?: IDaemonRequestLifecycleInfo; readonly workspaceSession: IWorkspaceSession; } @@ -95,10 +110,10 @@ export interface IDispatchWorkspaceRequestOptions { readonly resolver: IDaemonRequestResolver | undefined; readonly onExecutionStarting?: () => void; /** - * The `performance.now()` timestamp at which the daemon received the request. A phased request can join a batch - * whose input reconcile started after this time; see {@link PhasedRequestRouter.executeAsync}. + * Present when a host lifecycle admitted the request. Its `receivedTimeMs` also decides whether a phased request + * can join a batch whose input reconcile has started; see {@link PhasedRequestRouter.executeAsync}. */ - readonly receivedTimeMs?: number; + readonly lifecycleInfo?: IDaemonRequestLifecycleInfo; } /** Executes an already admitted workspace generation without resolving against another session. @beta */ @@ -163,7 +178,7 @@ export class DaemonRequestDispatcher implements AsyncDisposable { async function dispatchWorkspaceRequestAsync( options: IDispatchWorkspaceRequestOptions ): Promise { - const { envelope, client, workspaceSession, resolver, onExecutionStarting, receivedTimeMs } = options; + const { envelope, client, workspaceSession, resolver, onExecutionStarting, lifecycleInfo } = options; workspaceSession.assertActive?.(); if ( !isRushxInvocation(envelope) && @@ -183,6 +198,7 @@ async function dispatchWorkspaceRequestAsync( const resolved: ResolvedDaemonRequest = await resolver.resolveRequestAsync({ abortSignal: client.abortSignal, envelope, + lifecycleInfo, workspaceSession }); workspaceSession.assertActive?.(); @@ -197,7 +213,8 @@ async function dispatchWorkspaceRequestAsync( resolved.exactSelection, onExecutionStarting, resolved.requestSettings, - receivedTimeMs + resolved.telemetry, + lifecycleInfo?.receivedTimeMs ); } const globalRouter: GlobalCommandRequestRouter = new GlobalCommandRequestRouter(workspaceSession); diff --git a/libraries/rush-daemon/src/DaemonRequestTelemetry.ts b/libraries/rush-daemon/src/DaemonRequestTelemetry.ts new file mode 100644 index 0000000000..728d1a6a88 --- /dev/null +++ b/libraries/rush-daemon/src/DaemonRequestTelemetry.ts @@ -0,0 +1,222 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { PerformanceEntry } from 'node:perf_hooks'; + +import { PackageJsonLookup } from '@rushstack/node-core-library'; +import type { + IPhasedCommandEngineLogTelemetryOptions, + ITelemetryData, + PhasedCommandEngine +} from '@microsoft/rush-lib'; + +import type { IDaemonRequestLifecycleInfo } from './DaemonRequestDispatcher'; +import type { + IPhasedRequestTelemetryMeasure, + IPhasedRequestTelemetryReport, + IPhasedRequestTelemetrySink +} from './PhasedRequestTelemetry'; +import { getWorkspaceGenerationToken } from './WorkspaceGeneration'; +import type { IWorkspaceSession } from './WorkspaceSession'; + +const DAEMON_PACKAGE_VERSION: string = PackageJsonLookup.loadOwnPackageJson(__dirname).version; +const MILLISECONDS_PER_SECOND: number = 1000; +const GENERATION_TOKEN_PREFIX_LENGTH: number = 8; +const NATIVE_ITERATION_MEASURE_PREFIX: string = 'rush:executionManager:'; +const DAEMON_MEASURE_PREFIX: string = 'rush:daemon:'; +const RUSH_MEASURE_PREFIX: string = 'rush:'; + +/** + * Request environment variables that attribute an entry to its caller. They are read from the request, never from + * the daemon's own environment, and are not part of the daemon's environment identity. Native Rush rejects unknown + * `RUSH_` variables, so the tag variable does not use that prefix. + */ +const ATTRIBUTION_VARIABLES: Readonly> = { + agentSessionId: 'COPILOT_AGENT_SESSION_ID', + telemetryTag: 'ODSP_TELEMETRY_TAG' +}; + +/** When the warm engine was created, as `performance.now()` values. */ +export interface IDaemonEngineCreationTiming { + readonly startTimeMs: number; + readonly endTimeMs: number; +} + +/** What the production resolver knows about one resolved request. */ +export interface IDaemonRequestTelemetryContext { + /** The request's own parsed command. */ + readonly command: Pick; + readonly logTelemetry: (data: ITelemetryData, options?: IPhasedCommandEngineLogTelemetryOptions) => void; + readonly workspaceSession: IWorkspaceSession; + readonly lifecycleInfo: IDaemonRequestLifecycleInfo | undefined; + readonly resolveStartTimeMs: number; + readonly resolveEndTimeMs: number; + /** Set only for the request whose handling created the warm engine. */ + readonly engineCreation: IDaemonEngineCreationTiming | undefined; + /** Returns the 1-based position of the entry among those the warm engine logged. */ + readonly getRequestIndex: () => number; +} + +/** + * Creates the sink that logs one native telemetry entry for a request served by the warm engine. + */ +export function createDaemonRequestTelemetrySink( + context: IDaemonRequestTelemetryContext +): IPhasedRequestTelemetrySink { + return { + logRequest: (report: IPhasedRequestTelemetryReport) => { + try { + context.logTelemetry(createDaemonRequestTelemetryData(context, report), { + servedByIteration: report.scheduled + }); + } catch (error) { + process.stderr.write( + `Unable to log telemetry for request ${report.request.requestId}: ${(error as Error).message}\n` + ); + } + } + }; +} + +/** + * Builds the native telemetry entry for one request. + * + * @remarks + * Times are relative to the moment the daemon received the request, as a native command's are relative to its + * process start. `durationInSeconds` has the native meaning: it starts when the graph iteration was scheduled, or, + * for a request that needed no iteration, when its batch began handling it. `bootDurationSeconds` and + * `totalDurationSeconds` also start at the daemon's receipt of the request, so they exclude the client's own + * startup and connection time. + */ +export function createDaemonRequestTelemetryData( + context: IDaemonRequestTelemetryContext, + report: IPhasedRequestTelemetryReport +): ITelemetryData { + const timeOriginMs: number = context.lifecycleInfo?.receivedTimeMs ?? context.resolveStartTimeMs; + const durationStartTimeMs: number = report.iterationStartTimeMs ?? report.executionStartTimeMs; + return context.command.createTelemetryData({ + records: report.records, + succeeded: report.result.exitCode === 0, + durationInSeconds: toSeconds(report.resultTimeMs - durationStartTimeMs), + timeOriginMs, + extraData: { + ...getDaemonExtraData(context, report), + durationBasis: report.iterationStartTimeMs === undefined ? 'batch' : 'iteration', + bootDurationSeconds: toSeconds(durationStartTimeMs - timeOriginMs), + totalDurationSeconds: toSeconds(report.resultTimeMs - timeOriginMs), + ...getRequestAttribution(report.request.environment) + }, + performanceEntries: [ + ...getLifecycleEntries(context), + ...report.measures.map(createMeasureEntry), + ...getNativeEntries(context, report, durationStartTimeMs) + ] + }); +} + +function getDaemonExtraData( + context: IDaemonRequestTelemetryContext, + report: IPhasedRequestTelemetryReport +): Record { + const { lifecycleInfo, workspaceSession } = context; + const queueWait: IPhasedRequestTelemetryMeasure | undefined = report.measures.find( + (measure: IPhasedRequestTelemetryMeasure) => measure.name === `${DAEMON_MEASURE_PREFIX}queueWait` + ); + const generation: number | undefined = workspaceSession.metadata.generation; + return { + daemon: true, + daemonVersion: DAEMON_PACKAGE_VERSION, + daemonPid: process.pid, + requestId: report.request.requestId, + ...(generation === undefined ? {} : { generation }), + generationToken: getWorkspaceGenerationToken(workspaceSession).slice(0, GENERATION_TOKEN_PREFIX_LENGTH), + ...(lifecycleInfo === undefined ? {} : { reloadTier: lifecycleInfo.reloadTier }), + requestIndex: context.getRequestIndex(), + batchSize: report.batchSize, + queueWaitSeconds: queueWait ? toSeconds(queueWait.endTimeMs - queueWait.startTimeMs) : 0, + graphWasInitialized: context.engineCreation === undefined, + persistentIpcRunners: workspaceSession.rushConfiguration.daemon.usePersistentIpcRunners, + scheduled: report.scheduled, + earlyResult: report.earlyResult, + countRetained: report.countRetained, + exitCode: report.result.exitCode, + outcome: report.result.outcome + }; +} + +function getRequestAttribution( + environment: Readonly> +): Record { + const attribution: Record = {}; + for (const [field, variable] of Object.entries(ATTRIBUTION_VARIABLES)) { + const value: string | undefined = environment[variable]; + if (value) attribution[field] = value; + } + return attribution; +} + +function getLifecycleEntries(context: IDaemonRequestTelemetryContext): PerformanceEntry[] { + const { engineCreation, lifecycleInfo } = context; + const measures: IPhasedRequestTelemetryMeasure[] = []; + if (lifecycleInfo) { + measures.push({ + name: `${DAEMON_MEASURE_PREFIX}prepareWorkspace`, + startTimeMs: lifecycleInfo.receivedTimeMs, + endTimeMs: lifecycleInfo.preparedTimeMs + }); + } + if (engineCreation) { + measures.push({ name: `${DAEMON_MEASURE_PREFIX}createEngine`, ...engineCreation }); + } + measures.push({ + name: `${DAEMON_MEASURE_PREFIX}resolve`, + startTimeMs: context.resolveStartTimeMs, + endTimeMs: context.resolveEndTimeMs + }); + return measures.map(createMeasureEntry); +} + +/** + * The native measures recorded while serving this request: engine creation for the request that created the + * engine, and the graph iteration's own measures under their native names. One daemon serves one workspace and + * runs one iteration at a time, so the time windows attribute the measures to this request. + */ +function getNativeEntries( + context: IDaemonRequestTelemetryContext, + report: IPhasedRequestTelemetryReport, + durationStartTimeMs: number +): PerformanceEntry[] { + const { engineCreation } = context; + return performance.getEntriesByType('measure').filter((entry: PerformanceEntry) => { + const endTime: number = entry.startTime + entry.duration; + if (entry.name.startsWith(DAEMON_MEASURE_PREFIX)) { + return false; + } + if (entry.name.startsWith(NATIVE_ITERATION_MEASURE_PREFIX)) { + return report.scheduled && entry.startTime >= durationStartTimeMs && endTime <= report.resultTimeMs; + } + return ( + engineCreation !== undefined && + entry.name.startsWith(RUSH_MEASURE_PREFIX) && + entry.startTime >= engineCreation.startTimeMs && + endTime <= engineCreation.endTimeMs + ); + }); +} + +function createMeasureEntry(measure: IPhasedRequestTelemetryMeasure): PerformanceEntry { + const { name, startTimeMs: startTime } = measure; + const duration: number = measure.endTimeMs - startTime; + return { + name, + entryType: 'measure', + startTime, + duration, + detail: undefined, + toJSON: () => ({ name, entryType: 'measure', startTime, duration }) + }; +} + +function toSeconds(milliseconds: number): number { + return milliseconds / MILLISECONDS_PER_SECOND; +} diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index f8e59bc411..cf783fec90 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -46,6 +46,12 @@ import { type IPhasedOperationOutcome, parseWarningsAllowedByEnvironment } from './CommandResultPolicy'; +import { + collectPhasedRequestTelemetryRecords, + type IPhasedRequestTelemetryMeasure, + type IPhasedRequestTelemetryRecords, + type IPhasedRequestTelemetrySink +} from './PhasedRequestTelemetry'; interface IDualEmitOperationGraph extends IOperationGraph { eventSink: _IOperationGraphEventSink | undefined; @@ -65,6 +71,8 @@ interface IGraphRoutingState { interface IPreparedPhasedRequest { readonly onExecutionStarting: (() => void) | undefined; + /** The `performance.now()` timestamp at which the request was admitted. */ + readonly admittedTimeMs: number; readonly client: IPhasedRequestClient; readonly exclusivityClass: RequestExclusivityClass; readonly interactiveSession: IInteractiveRequestSession | undefined; @@ -79,12 +87,28 @@ interface IPreparedPhasedRequest { readonly receivedTimeMs: number; /** The `performance.now()` timestamp at which the router received the request. */ readonly startTimeMs: number; + readonly telemetry: IPhasedRequestTelemetrySink | undefined; readonly warningsAllowedByEnvironment: boolean; } +/** `performance.now()` timestamps of one batch's handling, shared by its participants' telemetry. */ +interface IBatchTimings { + readonly startTimeMs: number; + batchSize: number; + leasesAcquiredTimeMs: number | undefined; + reconciledTimeMs: number | undefined; + selectionsAppliedTimeMs: number | undefined; + /** The start of `scheduleIterationAsync`, which is where a native iteration's duration starts. */ + scheduleStartTimeMs: number | undefined; + scheduledTimeMs: number | undefined; + executionStartTimeMs: number | undefined; + iterationEndTimeMs: number | undefined; +} + interface IBatchEntry extends IPreparedPhasedRequest { abortListener: (() => void) | undefined; abortRequested: boolean; + batchTimings: IBatchTimings | undefined; completed: boolean; /** * Set when a failed result is published while operations of this request that the failure did not block are @@ -98,6 +122,8 @@ interface IBatchEntry extends IPreparedPhasedRequest { * its batch's iteration is still running for other participants; see `#finishSettledEntry`. */ finishPromise: Promise | undefined; + /** The `performance.now()` timestamp at which the entry was taken into a batch. */ + joinedTimeMs: number | undefined; outputError: unknown; participated: boolean; reject: (error: unknown) => void; @@ -149,6 +175,9 @@ export class PhasedRequestRouter { * Validates and executes one resolved phased request against the warm graph. * * @remarks + * If `telemetry` is provided, it receives this request's report once the request has taken part in a graph + * iteration or a no-op check, before the result is written to the client. + * * `receivedTimeMs` is the `performance.now()` timestamp at which the daemon received the request. The request can * join a batch whose input reconcile started after this time. It defaults to the time of this call. */ @@ -158,6 +187,7 @@ export class PhasedRequestRouter { exactSelection: boolean = false, onExecutionStarting?: () => void, requestSettings?: IPhasedCommandEngineRequestSettings, + telemetry?: IPhasedRequestTelemetrySink, receivedTimeMs?: number ): Promise { const startTimeMs: number = performance.now(); @@ -195,6 +225,7 @@ export class PhasedRequestRouter { admissionController?.dispose(); return await finishAfterAdmissionErrorAsync(request, client, interactiveSession, error); } + const admittedTimeMs: number = performance.now(); try { let inputAttachment: Disposable | undefined; @@ -232,6 +263,7 @@ export class PhasedRequestRouter { const lease: IRequestLease = admissionLease; return await routingState.coordinator.enqueueAsync( { + admittedTimeMs, client, exclusivityClass, interactiveSession, @@ -243,6 +275,7 @@ export class PhasedRequestRouter { requestSettingsKey: JSON.stringify(requestSettings ?? null), selection, startTimeMs, + telemetry, warningsAllowedByEnvironment, onExecutionStarting }, @@ -314,10 +347,12 @@ class PhasedRequestBatchCoordinator { ...request, abortListener: undefined, abortRequested: false, + batchTimings: undefined, completed: false, continuesAfterResult: false, executionStarted: false, finishPromise: undefined, + joinedTimeMs: undefined, outputError: undefined, participated: false, reject, @@ -368,8 +403,10 @@ class PhasedRequestBatchCoordinator { this.#currentBatch = batch; this.#acceptingCurrentBatch = first.exclusivityClass === RequestExclusivityClass.SharedBuild; this.#reconcileStartTimeMs = undefined; + const joinedTimeMs: number = performance.now(); for (const entry of batch) { entry.executionStarted = true; + entry.joinedTimeMs ??= joinedTimeMs; } try { await this.#executeBatchAsync(batch); @@ -415,6 +452,7 @@ class PhasedRequestBatchCoordinator { ) { this.#pending.splice(index, 1); entry.executionStarted = true; + entry.joinedTimeMs = performance.now(); batch.push(entry); } else { index++; @@ -423,6 +461,17 @@ class PhasedRequestBatchCoordinator { } async #executeBatchAsync(batch: IBatchEntry[]): Promise { + const timings: IBatchTimings = { + startTimeMs: performance.now(), + batchSize: 0, + leasesAcquiredTimeMs: undefined, + reconciledTimeMs: undefined, + selectionsAppliedTimeMs: undefined, + scheduleStartTimeMs: undefined, + scheduledTimeMs: undefined, + executionStartTimeMs: undefined, + iterationEndTimeMs: undefined + }; const graphLeasePromise: Promise = this.#nextGraphLeasePromise ?? this.#graphExecutionScheduler.acquireAsync({ @@ -443,10 +492,12 @@ class PhasedRequestBatchCoordinator { throw new Error('The warm workspace operation graph is not idle.'); } executionLease = await this.#workspaceSession.acquireExecutionLeaseAsync?.(); + timings.leasesAcquiredTimeMs = performance.now(); // Requests received before this point made their changes before the reconcile reads the inputs, so they can // still join while it runs. Requests received later wait for the next batch, which reconciles again. this.#reconcileStartTimeMs = performance.now(); await this.#workspaceSession.reconcileInvalidationsAsync(); + timings.reconciledTimeMs = performance.now(); if (batch[0].exclusivityClass === RequestExclusivityClass.SharedBuild) { this.#takeCompatiblePending(batch); @@ -471,6 +522,8 @@ class PhasedRequestBatchCoordinator { this.#graph, participants.map((entry: IBatchEntry) => entry.selection) ); + timings.selectionsAppliedTimeMs = performance.now(); + timings.batchSize = participants.length; for (const entry of batch) { if (!participants.includes(entry)) { // Clients that cancelled before execution must not wait for the participants' work. @@ -482,6 +535,7 @@ class PhasedRequestBatchCoordinator { const unsubscribeDemand: () => void = this.#multiplexer.subscribe(demand); for (const entry of participants) { entry.participated = true; + entry.batchTimings = timings; const activeOperationIds: ReadonlySet = new Set( entry.selection.activeOperations.map((operation: Operation) => operation.name) ); @@ -512,10 +566,12 @@ class PhasedRequestBatchCoordinator { const iterationCleanupErrors: unknown[] = []; try { for (const entry of participants) entry.onExecutionStarting?.(); + timings.scheduleStartTimeMs = performance.now(); scheduled = await this.#graph.scheduleIterationAsync({ inputsSnapshot: this.#workspaceSession.inputsSnapshot, getOperationEnvironment: createOperationEnvironmentLookup(participants) }); + timings.scheduledTimeMs = performance.now(); if (scheduled) { await Promise.all( participants.map(async (entry: IBatchEntry) => { @@ -526,6 +582,7 @@ class PhasedRequestBatchCoordinator { } }) ); + timings.executionStartTimeMs = performance.now(); const executionPromise: Promise = this.#graph.executeScheduledIterationAsync(); if (!participants.some((entry: IBatchEntry) => this.#needsIteration(entry)) || demand.abandoned) { // Let executeScheduledIterationAsync promote the scheduled iteration before aborting it. @@ -534,6 +591,7 @@ class PhasedRequestBatchCoordinator { await this.#abortTail; } await executionPromise; + timings.iterationEndTimeMs = performance.now(); } } catch (error) { executionError = error; @@ -890,6 +948,9 @@ class PhasedRequestBatchCoordinator { scheduled: entry.participated && batchScheduled, warningsAllowedByEnvironment: entry.warningsAllowedByEnvironment }); + if (entry.participated) { + this.#logTelemetry(entry, result, batchScheduled, report !== 'final'); + } try { await entry.client.writeResultAsync(result); this.#completeEntry(entry); @@ -900,6 +961,44 @@ class PhasedRequestBatchCoordinator { } } + #logTelemetry( + entry: IBatchEntry, + result: IDaemonPhasedRequestResult, + batchScheduled: boolean, + iterationInProgress: boolean + ): void { + const { batchTimings: timings, requestSink, telemetry } = entry; + if (!telemetry || !requestSink || !timings) { + return; + } + try { + const resultTimeMs: number = performance.now(); + const executionStartTimeMs: number = Math.max(timings.startTimeMs, entry.joinedTimeMs ?? 0); + const { records, countRetained }: IPhasedRequestTelemetryRecords = collectPhasedRequestTelemetryRecords({ + activeOperations: entry.selection.activeOperations, + graph: this.#graph, + observations: requestSink, + upToDateTimeMs: executionStartTimeMs + }); + telemetry.logRequest({ + request: entry.request, + result, + records, + countRetained, + batchSize: timings.batchSize, + scheduled: batchScheduled, + earlyResult: iterationInProgress, + receivedTimeMs: entry.startTimeMs, + executionStartTimeMs, + iterationStartTimeMs: batchScheduled ? timings.scheduleStartTimeMs : undefined, + resultTimeMs, + measures: createTelemetryMeasures(entry, timings, executionStartTimeMs, resultTimeMs) + }); + } catch { + // Telemetry never changes a request's result. + } + } + #settleEntry(entry: IBatchEntry, settle: () => void): void { if (!entry.continuesAfterResult) { settle(); @@ -950,6 +1049,28 @@ class PhasedRequestBatchCoordinator { } } +function createTelemetryMeasures( + entry: IBatchEntry, + timings: IBatchTimings, + executionStartTimeMs: number, + resultTimeMs: number +): IPhasedRequestTelemetryMeasure[] { + const measures: IPhasedRequestTelemetryMeasure[] = []; + function addMeasure(name: string, startTimeMs: number | undefined, endTimeMs: number | undefined): void { + if (startTimeMs !== undefined && endTimeMs !== undefined) { + measures.push({ name: `rush:daemon:${name}`, startTimeMs, endTimeMs }); + } + } + addMeasure('admission', entry.startTimeMs, entry.admittedTimeMs); + addMeasure('queueWait', entry.admittedTimeMs, executionStartTimeMs); + addMeasure('acquireExecutionLease', timings.startTimeMs, timings.leasesAcquiredTimeMs); + addMeasure('reconcileInvalidations', timings.leasesAcquiredTimeMs, timings.reconciledTimeMs); + addMeasure('applySelections', timings.reconciledTimeMs, timings.selectionsAppliedTimeMs); + addMeasure('scheduleIteration', timings.scheduleStartTimeMs, timings.scheduledTimeMs); + addMeasure('executeIteration', timings.executionStartTimeMs, timings.iterationEndTimeMs ?? resultTimeMs); + return measures; +} + function createBatchReleaseBarrier( batch: ReadonlyArray, releaseAsync: () => Promise diff --git a/libraries/rush-daemon/src/PhasedRequestTelemetry.ts b/libraries/rush-daemon/src/PhasedRequestTelemetry.ts new file mode 100644 index 0000000000..aa1240dc21 --- /dev/null +++ b/libraries/rush-daemon/src/PhasedRequestTelemetry.ts @@ -0,0 +1,145 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { + IOperationExecutionResult, + IOperationGraph, + IPhasedCommandEngineTelemetryRecord, + Operation +} from '@microsoft/rush-lib'; +import { OperationStatus } from '@microsoft/rush-lib'; +import type { IDaemonPhasedRequest, IDaemonPhasedRequestResult } from '@rushstack/rush-daemon-protocol'; + +/** One phase of a phased request's handling, with `performance.now()` times. @beta */ +export interface IPhasedRequestTelemetryMeasure { + /** The measure name, such as `rush:daemon:queueWait`. */ + readonly name: string; + /** A `performance.now()` value. */ + readonly startTimeMs: number; + /** A `performance.now()` value. */ + readonly endTimeMs: number; +} + +/** + * The outcome and timing of one phased request that took part in a graph iteration or no-op check. + * + * @remarks + * All times are `performance.now()` values of the daemon process. + * + * @beta + */ +export interface IPhasedRequestTelemetryReport { + /** The request as the router executed it. */ + readonly request: IDaemonPhasedRequest; + /** The result sent to the client. */ + readonly result: IDaemonPhasedRequestResult; + /** + * The request's non-silent selected operations. Operations that this request did not need to run, because + * the warm graph had them up to date, are reported as `Skipped` with a zero-length stopwatch. + */ + readonly records: ReadonlyMap; + /** How many of `records` were already up to date. */ + readonly countRetained: number; + /** How many requests took part in the same graph iteration or no-op check. */ + readonly batchSize: number; + /** Whether the graph scheduled an iteration for the batch. */ + readonly scheduled: boolean; + /** Whether the result was produced while the shared iteration was still running for other requests. */ + readonly earlyResult: boolean; + /** When the router received the request. */ + readonly receivedTimeMs: number; + /** When this request's batch began handling it. */ + readonly executionStartTimeMs: number; + /** When the graph iteration began executing operations, if one was scheduled. */ + readonly iterationStartTimeMs: number | undefined; + /** When the request's result was produced. */ + readonly resultTimeMs: number; + /** The router's handling phases for this request, in order. */ + readonly measures: ReadonlyArray; +} + +/** + * Receives one report for each phased request that took part in a graph iteration or no-op check. + * + * @remarks + * The router invokes the sink before it writes the request's result. The sink must not throw; the router ignores + * its errors so that telemetry never changes a result. + * + * @beta + */ +export interface IPhasedRequestTelemetrySink { + /** Called once, before the result is written to the client. Errors are ignored. */ + logRequest(report: IPhasedRequestTelemetryReport): void; +} + +/** The subset of a request event sink used to collect a request's telemetry records. */ +export interface IPhasedRequestTelemetryObservations { + getObservedResult(operation: Operation): { readonly executionResult: IOperationExecutionResult } | undefined; +} + +export interface ICollectPhasedRequestTelemetryRecordsOptions { + readonly activeOperations: ReadonlyArray; + readonly graph: IOperationGraph; + readonly observations: IPhasedRequestTelemetryObservations; + /** The timestamp given to operations that the request did not need to run. */ + readonly upToDateTimeMs: number; +} + +export interface IPhasedRequestTelemetryRecords { + readonly records: ReadonlyMap; + readonly countRetained: number; +} + +const UNFINISHED_STATUSES: ReadonlySet = new Set([ + OperationStatus.Waiting, + OperationStatus.Ready, + OperationStatus.Queued, + OperationStatus.Executing +]); + +/** + * Collects the telemetry records of one request's selection, matching the request's own end-of-run summary. + */ +export function collectPhasedRequestTelemetryRecords( + options: ICollectPhasedRequestTelemetryRecordsOptions +): IPhasedRequestTelemetryRecords { + const { activeOperations, graph, observations, upToDateTimeMs } = options; + const active: ReadonlySet = new Set(activeOperations); + const records: Map = new Map(); + let countRetained: number = 0; + // Iterate the graph so that entries list operations in the same order as native entries. + for (const operation of graph.operations) { + if (!active.has(operation) || operation.runner?.silent !== false) { + continue; + } + const observed: IOperationExecutionResult | undefined = + observations.getObservedResult(operation)?.executionResult; + if (observed && !observed.silent) { + records.set( + operation, + UNFINISHED_STATUSES.has(observed.status) ? createAbortedRecord(observed) : observed + ); + } else if (observed ?? graph.resultByOperation.get(operation)) { + // A silent observed record belongs to an operation the graph disabled because it was already up to date. + // Its retained stopwatch describes an earlier request, so it is not reported again. + records.set(operation, { + status: OperationStatus.Skipped, + silent: false, + stopwatch: { startTime: upToDateTimeMs, endTime: upToDateTimeMs }, + nonCachedDurationMs: undefined + }); + countRetained++; + } + } + return { records, countRetained }; +} + +function createAbortedRecord(observed: IOperationExecutionResult): IPhasedCommandEngineTelemetryRecord { + // The client stopped observing before this operation finished. + return { + status: OperationStatus.Aborted, + silent: false, + stopwatch: observed.stopwatch, + nonCachedDurationMs: undefined + }; +} diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 4ac806c7d1..27f5e135e5 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -13,8 +13,10 @@ import { PhasedCommandEngineConfigurationChangedError, PhasedCommandEngineProjectConfigurationError, type IPhasedCommandEngine, + type IPhasedCommandEngineLogTelemetryOptions, type IInputsSnapshot, type IOperationGraph, + type ITelemetryData, type Operation, type OperationEnabledState } from '@microsoft/rush-lib'; @@ -38,6 +40,7 @@ import { EngineTerminalProvider } from './EngineTerminalProvider'; import { OperationOutputFingerprints } from './OperationOutputFingerprints'; import { getDaemonShutdownReason } from './DaemonShutdownError'; import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; +import { createDaemonRequestTelemetrySink, type IDaemonEngineCreationTiming } from './DaemonRequestTelemetry'; /** * Binds the standalone host to a real native build/rebuild graph on its first request. @@ -54,30 +57,47 @@ import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; */ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { #binding: Promise | undefined; + /** The request whose handling created the warm engine, and when it did. */ + #engineCreation: (IDaemonEngineCreationTiming & { readonly requestId: string }) | undefined; + #logTelemetry: EngineLogTelemetry | undefined; + #loggedRequestCount: number = 0; #parameterIdentity: string | undefined; #workspaceSession: IWorkspaceSession | undefined; - readonly #environmentIdentity: string = environmentIdentity(process.env); + readonly #environmentIdentity: string; readonly #preparationLock: LockFile | undefined; + /** The daemon's environment when it started. Replacement sessions keep it; see `createForSession`. */ + readonly #startupEnvironment: Readonly>; readonly #validateGraphInputsAsync: (() => Promise) | undefined; public constructor(options?: { readonly preparationLock?: LockFile; readonly validateGraphInputsAsync?: () => Promise; + /** The environment that requests must match. Defaults to a copy of `process.env`. */ + readonly startupEnvironment?: Readonly>; }) { this.#preparationLock = options?.preparationLock; this.#validateGraphInputsAsync = options?.validateGraphInputsAsync; + this.#startupEnvironment = options?.startupEnvironment ?? { ...process.env }; + this.#environmentIdentity = environmentIdentity(this.#startupEnvironment); } public get workspaceLifecycle(): IWorkspaceResolverLifecycle { return this; } - /** Creates an unbound resolver for a replacement session without carrying old runner definitions. */ + /** + * Creates an unbound resolver for a replacement session without carrying old runner definitions. + * It keeps the startup environment, because a plugin may have added names to `process.env` since then. + */ public createForSession( preparationLock?: LockFile, validateGraphInputsAsync?: () => Promise ): ProductionDaemonRequestResolver { - return new ProductionDaemonRequestResolver({ preparationLock, validateGraphInputsAsync }); + return new ProductionDaemonRequestResolver({ + preparationLock, + validateGraphInputsAsync, + startupEnvironment: this.#startupEnvironment + }); } /** Inspects the native command shape without constructing or executing an operation graph. */ @@ -86,9 +106,11 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { } public async resolveRequestAsync(options: IResolveDaemonRequestOptions): Promise { + const resolveStartTimeMs: number = performance.now(); const { envelope, workspaceSession } = options; const terminal: EngineTerminalProvider = new EngineTerminalProvider(); const command: PhasedCommandEngine = await this.#parseCommandAsync(options, terminal); + let bindingStartTimeMs: number | undefined; if (this.#binding) { if ( this.#parameterIdentity !== command.parameterIdentity || @@ -99,6 +121,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { } else { this.#parameterIdentity = command.parameterIdentity; this.#workspaceSession = workspaceSession; + bindingStartTimeMs = performance.now(); const binding: Promise = this.#bindAsync(command, terminal, workspaceSession); this.#binding = binding; void binding.catch((error: unknown) => { @@ -110,6 +133,13 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { }); } await this.#binding; + if (bindingStartTimeMs !== undefined) { + this.#engineCreation = { + requestId: envelope.requestId, + startTimeMs: bindingStartTimeMs, + endTimeMs: performance.now() + }; + } const graph: IOperationGraph | undefined = workspaceSession.operationGraph; const shape: IWorkspaceEngineShape | undefined = workspaceSession.engineShape; if (!graph || !shape) throw new Error('Native engine initialization did not bind a workspace graph.'); @@ -123,10 +153,26 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { for (const [operation, enabledState] of selection) { if (enabledState !== false) operationSelection.push({ operationId: operation.name, enabledState }); } + const logTelemetry: EngineLogTelemetry | undefined = this.#logTelemetry; + // A workspace lifecycle may bind the engine with this request before it dispatches the request. + const engineCreation: IDaemonEngineCreationTiming | undefined = + this.#engineCreation?.requestId === envelope.requestId ? this.#engineCreation : undefined; return { kind: 'phased', exactSelection: true, requestSettings: command.requestSettings, + telemetry: logTelemetry + ? createDaemonRequestTelemetrySink({ + command, + logTelemetry, + workspaceSession, + lifecycleInfo: options.lifecycleInfo, + resolveStartTimeMs, + resolveEndTimeMs: performance.now(), + engineCreation, + getRequestIndex: () => ++this.#loggedRequestCount + }) + : undefined, request: { admission: envelope.admission, commandName: envelope.commandName, @@ -156,15 +202,20 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { 'The production daemon requires an explicitly identified native build/rebuild request. Ambiguous custom/rushx requests require --no-daemon.' ); } - if ( - environmentIdentity(envelope.environment) !== this.#environmentIdentity || - environmentIdentity(process.env) !== this.#environmentIdentity - ) { + if (environmentIdentity(envelope.environment) !== this.#environmentIdentity) { throw new DaemonRequestDispatchError( 'unsupported', 'The request environment differs from the daemon startup environment. Restart the daemon from this environment or use --no-daemon.' ); } + const changedNames: string[] = this.#getChangedStartupNames(); + if (changedNames.length > 0) { + throw new DaemonRequestDispatchError( + 'unsupported', + `The daemon's own environment changed after it started (${changedNames.join(', ')}); a Rush plugin ` + + 'may have changed process.env. Restart the daemon or use --no-daemon.' + ); + } let command: PhasedCommandEngine; try { command = await PhasedCommandEngine.parseAsync({ @@ -192,6 +243,26 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { return command; } + /** + * The names of the startup environment whose value in `process.env` changed or was removed since startup. + * + * @remarks + * Engine code runs in this process, and a plugin may add its own names to `process.env`, for example to pass + * a session ID to its operations. Native Rush keeps such names for the rest of the command, so they are not + * a difference from the client's environment, and are not reported here. Otherwise every request after the + * plugin's first write would fall back to in-process Rush. A request whose own environment sets an added name + * still differs from the startup environment. + * + * Both values are compared as the workspace fingerprint records them. The startup entries drop repeated PATH + * entries, so a raw live PATH that repeats an entry would otherwise count as changed on every request. + */ + #getChangedStartupNames(): string[] { + const liveEnvironment: NodeJS.ProcessEnv = process.env; + return getWorkspaceFingerprintEnvironmentEntries(this.#startupEnvironment) + .filter(([name, value]) => getFingerprintValue(name, liveEnvironment[name]) !== value) + .map(([name]) => name); + } + async #bindAsync( command: PhasedCommandEngine, terminal: EngineTerminalProvider, @@ -217,6 +288,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { throw new Error(terminal.describeError(error), { cause: error }); } try { + this.#logTelemetry = engine.logTelemetry; terminal.attach(engine.operationGraph); const outputFingerprints: OperationOutputFingerprints = new OperationOutputFingerprints( engine.operationGraph @@ -261,6 +333,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { } }; } catch (error) { + this.#logTelemetry = undefined; try { await engine[Symbol.asyncDispose](); } catch (cleanupError) { @@ -301,10 +374,19 @@ function createProjectConfigurationFallback( ); } +type EngineLogTelemetry = (data: ITelemetryData, options?: IPhasedCommandEngineLogTelemetryOptions) => void; + function environmentIdentity(environment: Readonly>): string { return JSON.stringify(getWorkspaceFingerprintEnvironmentEntries(environment)); } +/** The value of one variable as {@link getWorkspaceFingerprintEnvironmentEntries} records it, if it is set. */ +function getFingerprintValue(name: string, value: string | undefined): string | undefined { + return value === undefined + ? undefined + : getWorkspaceFingerprintEnvironmentEntries({ [name]: value })[0]?.[1]; +} + function getChangedOperations(options: IMapWorkspaceInvalidationsOptions): Iterable { const { currentInputsSnapshot: current, nextInputsSnapshot: next, operationGraph } = options; return Array.from(operationGraph.operations).filter( diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 24f3a40d64..dc25a53b33 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -151,6 +151,10 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { process.env[EnvironmentVariableNames._RUSH_LIB_PATH] ); readonly #repoRoot: string; + /** The daemon's environment before any engine ran. A plugin may add names to `process.env` later. */ + readonly #startupEnvironment: Record = Object.fromEntries( + Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) + ); readonly #startupFingerprint: IWorkspaceInputFingerprint; readonly #runtimeCache: WorkspaceRuntimeFingerprintCache; // Concurrent requests share captures; each capture still starts after the requests it serves arrived. @@ -212,9 +216,18 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { public async dispatchAsync( request: IDaemonRequestEnvelope, destination: IDaemonRequestDispatchClient, - dispatchAsync: DispatchWorkspaceRequestAsync + dispatchWorkspaceRequestAsync: DispatchWorkspaceRequestAsync ): Promise { const receivedTimeMs: number = performance.now(); + const dispatchAsync: DispatchWorkspaceRequestAsync = (options) => + dispatchWorkspaceRequestAsync({ + ...options, + lifecycleInfo: { + receivedTimeMs, + preparedTimeMs: performance.now(), + reloadTier: this.#lastReloadTier + } + }); // Native Rush owns its SDK handoff; a foreign client's bundled engine must not override this one. const envelope: IDaemonRequestEnvelope = { ...request, @@ -306,7 +319,6 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { client, workspaceSession: prepared.session, resolver: prepared.resolver, - receivedTimeMs, onExecutionStarting: () => { this.#assertGeneration(prepared); state.began = true; @@ -463,9 +475,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { // Graph control environment flags are not a request to change the retained engine environment. const controlEnvelope: IDaemonRequestEnvelope = { ...envelope, - environment: Object.fromEntries( - Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) - ) + environment: { ...this.#startupEnvironment } }; const current: IWorkspaceInputFingerprint = await this.#captureAsync(session, controlEnvelope); const currentTier: WorkspaceInputChangeTier = this.#classify(current, false); diff --git a/libraries/rush-daemon/src/index.ts b/libraries/rush-daemon/src/index.ts index a075f6aba5..282d9a6f75 100644 --- a/libraries/rush-daemon/src/index.ts +++ b/libraries/rush-daemon/src/index.ts @@ -10,6 +10,7 @@ export { type IDaemonRequestDispatchClient, type IDaemonRequestResolver, type IDaemonRequestLifecycle, + type IDaemonRequestLifecycleInfo, type IDispatchWorkspaceRequestOptions, type DispatchWorkspaceRequestAsync, type IResolvedDaemonGlobalRequest, @@ -95,6 +96,11 @@ export { } from './WorkspaceInvalidationTracker'; export { type IPhasedRequestClient } from './PhasedRequestClient'; export { PhasedRequestRouter } from './PhasedRequestRouter'; +export type { + IPhasedRequestTelemetryMeasure, + IPhasedRequestTelemetryReport, + IPhasedRequestTelemetrySink +} from './PhasedRequestTelemetry'; export { ProductionDaemonRequestResolver } from './ProductionDaemonRequestResolver'; export { getWorkspaceGenerationToken } from './WorkspaceGeneration'; export { RushDaemonRequestResolver } from './RushDaemonRequestResolver'; diff --git a/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts b/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts new file mode 100644 index 0000000000..30eb93eb2a --- /dev/null +++ b/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts @@ -0,0 +1,346 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { execFileSync } from 'node:child_process'; + +import { Rush, RushUserConfiguration } from '@microsoft/rush-lib'; +import { + DaemonFrameType, + decodeDaemonLogChunk, + type IDaemonRequestEnvelope +} from '@rushstack/rush-daemon-protocol'; + +import { ProductionDaemonRequestResolver } from '../ProductionDaemonRequestResolver'; +import { RushDaemonHost } from '../RushDaemonHost'; +import { WorkspaceSession } from '../WorkspaceSession'; +import { removeTestFolderAsync } from './TestProcessExit'; +import type { GetWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import type { + IDaemonRequestResolver, + IResolveDaemonRequestOptions, + ResolvedDaemonRequest +} from '../DaemonRequestDispatcher'; +import { + isRushxInvocation, + wrapWorkspaceResolverLifecycle, + type IWorkspaceResolverLifecycle +} from '../WorkspaceResolverLifecycle'; +import { + DaemonRequestWireClient, + createWireEnvelope, + type ITerminalExchange +} from './DaemonRequestWireTestUtilities'; + +const RUSH_VERSION: string = Rush.version; + +export interface IFixture extends AsyncDisposable { + readonly repoRoot: string; + readonly host: RushDaemonHost; + readonly session: WorkspaceSession; + readonly client: DaemonRequestWireClient; +} + +export interface IFixtureOptions { + readonly getSuccessorLaunchAsync?: GetWorkspaceSuccessorLaunchAsync; + readonly onSessionCreated?: (session: WorkspaceSession) => void; + readonly resolver?: IDaemonRequestResolver; + /** Adds the `_phase:compile:incremental` script, which passes `--incremental` to build.cjs. */ + readonly incrementalScript?: boolean; + /** Sets `daemon.incrementalBuilds` in rush.json. */ + readonly incrementalBuilds?: boolean; + /** Uses PNPM, which installs a dependency file (shrinkwrap-deps.json) that change detection hashes per project. */ + readonly pnpm?: boolean; + readonly telemetryEnabled?: boolean; +} + +export class DecoratedTestResolver implements IDaemonRequestResolver { + public readonly workspaceLifecycle: IWorkspaceResolverLifecycle | undefined; + private readonly _inner: IDaemonRequestResolver; + private readonly _events: string[]; + private readonly _id: number; + private _disposed: boolean = false; + + public constructor(inner: IDaemonRequestResolver, events: string[]) { + this._inner = inner; + this._events = events; + this._id = events.filter((event) => event.startsWith('created')).length; + events.push(`created:${this._id}`); + this.workspaceLifecycle = wrapWorkspaceResolverLifecycle( + inner, + (replacement) => new DecoratedTestResolver(replacement, events) + ); + } + + public async resolveRequestAsync(options: IResolveDaemonRequestOptions): Promise { + if (this._disposed) throw new Error('A disposed resolver was invoked.'); + if (isRushxInvocation(options.envelope)) { + this._events.push(`isolated:${this._id}`); + throw new Error('Explicit isolated invocation reached the decorated resolver.'); + } + return await this._inner.resolveRequestAsync(options); + } + + public async [Symbol.asyncDispose](): Promise { + if (this._disposed) throw new Error('A resolver was disposed twice.'); + this._disposed = true; + this._events.push(`disposed:${this._id}`); + await this._inner[Symbol.asyncDispose]?.(); + } +} + +/** Creates a Git repository with the projects a, b and c (b depends on a), and a daemon host that serves it. */ +export async function createFixtureAsync( + cache: boolean = false, + configurationKind: 'direct' | 'rig' | 'inherited' = 'direct', + options: IFixtureOptions = {} +): Promise { + const repoRoot: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-native-engine-')); + const cacheNamespace: string = path.basename(repoRoot); + const userConfiguration: RushUserConfiguration = await RushUserConfiguration.initializeAsync(); + const cacheFolder: string = path.join( + userConfiguration.buildCacheFolder ?? path.join(repoRoot, 'common/temp/build-cache'), + cacheNamespace + ); + const write: (name: string, text: string) => void = (name, text) => { + const filename: string = path.join(repoRoot, name); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, text); + }; + write( + 'rush.json', + JSON.stringify({ + rushVersion: RUSH_VERSION, + ...(options.pnpm ? { pnpmVersion: '9.15.9' } : { npmVersion: '10.0.0' }), + // Retention assertions must not depend on the surrounding Jest worker's accumulated RSS. + daemon: { + warmMemoryBudgetMB: 100_000, + ...(options.incrementalBuilds === undefined ? {} : { incrementalBuilds: options.incrementalBuilds }) + }, + ...(options.telemetryEnabled ? { telemetryEnabled: true } : {}), + projectFolderMinDepth: 2, + projectFolderMaxDepth: 2, + projects: ['a', 'b', 'c'].map((name) => ({ + packageName: name, + projectFolder: `projects/${name}` + })) + }) + ); + write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\n**/lib/\n**/node_modules/\nruns.txt\n'); + write('common/temp/last-link.flag', '{}'); + if (options.pnpm) { + write('common/config/rush/pnpm-lock.yaml', "lockfileVersion: '9.0'\n"); + } else { + write('common/config/rush/npm-shrinkwrap.json', '{"lockfileVersion":3,"packages":{}}'); + } + write( + 'common/config/rush/command-line.json', + JSON.stringify({ + phases: [{ name: '_phase:compile', dependencies: { upstream: ['_phase:compile'] } }], + commands: [ + { + commandKind: 'phased', + name: 'build', + phases: ['_phase:compile'], + incremental: true, + enableParallelism: true + } + ], + parameters: [ + { + parameterKind: 'flag', + longName: '--production', + description: 'Production build', + associatedCommands: ['build'], + associatedPhases: ['_phase:compile'] + } + ] + }) + ); + if (cache) { + write( + 'common/config/rush/build-cache.json', + JSON.stringify({ + buildCacheEnabled: true, + cacheProvider: 'local-only', + cacheEntryNamePattern: `${cacheNamespace}/[hash]` + }) + ); + } + for (const name of ['a', 'b', 'c']) { + write( + `projects/${name}/package.json`, + JSON.stringify({ + name, + version: '1.0.0', + scripts: { + '_phase:compile': 'node build.cjs', + ...(options.incrementalScript + ? { '_phase:compile:incremental': 'node build.cjs --incremental' } + : {}) + }, + dependencies: name === 'b' ? { a: '1.0.0' } : {} + }) + ); + write( + `projects/${name}/config/rush-project.json`, + JSON.stringify({ + operationSettings: [{ operationName: '_phase:compile', outputFolderNames: ['lib'] }] + }) + ); + write(`projects/${name}/input.txt`, 'one'); + if (options.pnpm) write(`projects/${name}/.rush/temp/shrinkwrap-deps.json`, '{}'); + write( + `projects/${name}/build.cjs`, + ` +const fs = require('node:fs'); +const path = require('node:path'); +const name = require('./package.json').name; +const input = fs.readFileSync(fs.existsSync('src/input.txt') ? 'src/input.txt' : 'input.txt', 'utf8'); +(async () => { +const gateFile = path.resolve('../../common/temp/gate-' + name + '.json'); +if (fs.existsSync(gateFile)) { + const { port } = JSON.parse(fs.readFileSync(gateFile, 'utf8')); + await new Promise((resolve, reject) => { + const socket = require('node:net').connect(port, '127.0.0.1'); + socket.once('error', reject); + socket.once('data', () => { socket.end(); resolve(); }); + }); +} +fs.appendFileSync('../../runs.txt', name + ':' + input + ':' + process.argv.slice(2).join(' ') + '\\n'); +const environmentFile = path.resolve('../../common/temp/operation-environment.txt'); +if (fs.existsSync(environmentFile)) { + const { COPILOT_AGENT_SESSION_ID = null, RUSH_INVOKED_FOLDER = null } = process.env; + fs.appendFileSync(environmentFile, JSON.stringify([name, COPILOT_AGENT_SESSION_ID, RUSH_INVOKED_FOLDER]) + '\\n'); +} +fs.mkdirSync('lib', { recursive: true }); +fs.writeFileSync('lib/output.txt', input); +console.log('built-' + name + '-' + input); +if (input === 'warning') console.error('warning-' + name); +if (input === 'failure') process.exitCode = 7; +})().catch((error) => { console.error(error); process.exitCode = 1; }); +` + ); + } + if (configurationKind === 'rig') { + fs.rmSync(path.join(repoRoot, 'projects/a/config/rush-project.json')); + write('projects/a/config/rig.json', '{"rigPackageName":"fixture-rig"}'); + write('projects/a/node_modules/fixture-rig/package.json', '{"name":"fixture-rig","version":"1.0.0"}'); + write( + 'projects/a/node_modules/fixture-rig/profiles/default/config/rush-project.json', + JSON.stringify({ + operationSettings: [{ operationName: '_phase:compile', outputFolderNames: ['lib'] }] + }) + ); + } else if (configurationKind === 'inherited') { + write( + 'common/temp/inherited-rush-project.json', + JSON.stringify({ + operationSettings: [{ operationName: '_phase:compile', outputFolderNames: ['lib'] }] + }) + ); + write( + 'projects/a/config/rush-project.json', + JSON.stringify({ + extends: '../../../common/temp/inherited-rush-project.json', + incrementalBuildIgnoredGlobs: ['ignored.txt'] + }) + ); + } + execFileSync('git', ['init', '--quiet'], { cwd: repoRoot }); + execFileSync('git', ['config', '--local', 'core.autocrlf', 'false'], { cwd: repoRoot }); + execFileSync('git', ['add', '.'], { cwd: repoRoot }); + execFileSync( + 'git', + [ + '-c', + 'user.name=Engine Test', + '-c', + 'user.email=engine@example.invalid', + 'commit', + '--quiet', + '-m', + 'fixture' + ], + { cwd: repoRoot } + ); + let session: WorkspaceSession | undefined; + let host: RushDaemonHost | undefined; + try { + host = await RushDaemonHost.startAsync({ + repoRoot, + rushVersion: RUSH_VERSION, + daemonVersion: 'native-engine-test', + requestResolver: options.resolver ?? new ProductionDaemonRequestResolver(), + getSuccessorLaunchAsync: options.getSuccessorLaunchAsync, + createWorkspaceSessionAsync: async (sessionOptions) => { + session = await WorkspaceSession.createAsync(sessionOptions); + options.onSessionCreated?.(session); + return session; + } + }); + const client: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync(host.paths.socketPath); + await client.handshakeAsync(); + const runningHost: RushDaemonHost = host; + return { + repoRoot, + host, + get session(): WorkspaceSession { + return session!; + }, + client, + [Symbol.asyncDispose]: async () => { + await client.closeAsync().finally(() => runningHost.closeAsync()); + if (cache) await removeTestFolderAsync(cacheFolder, true); + await removeTestFolderAsync(repoRoot, true); + } + }; + } catch (error) { + await host?.closeAsync(); + if (cache) await removeTestFolderAsync(cacheFolder, true); + await removeTestFolderAsync(repoRoot, true); + throw error; + } +} + +export async function runAsync( + fixture: IFixture, + requestId: string, + argv: string[], + overrides: Partial = {} +): Promise { + const environment: Record = {}; + for (const [name, value] of Object.entries(process.env)) { + if (value !== undefined) environment[name] = value; + } + await fixture.client.sendControlAsync({ + kind: 'requestStart', + payload: createWireEnvelope(requestId, argv[0], fixture.repoRoot, { + argv, + environment, + commandOrigin: 'built-in', + ...overrides + }) + }); + return await fixture.client.readTerminalAsync(requestId); +} + +export function runs(fixture: IFixture): string[] { + const filename: string = path.join(fixture.repoRoot, 'runs.txt'); + return fs.existsSync(filename) ? fs.readFileSync(filename, 'utf8').trim().split('\n') : []; +} + +export function logText(exchange: ITerminalExchange): string { + return exchange.frames + .filter((frame) => frame.kind === DaemonFrameType.logStdout || frame.kind === DaemonFrameType.logStderr) + .map((frame) => Buffer.from(decodeDaemonLogChunk(frame.payload).chunk).toString()) + .join(''); +} + +export function requestEnvironment(): Record { + return Object.fromEntries( + Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) + ); +} diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index 6333cea8d6..dea724566b 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -794,6 +794,7 @@ describe('shared phased request batching', () => { false, undefined, undefined, + undefined, secondReceivedTimeMs ); await settleAsync(); diff --git a/libraries/rush-daemon/src/test/PhasedRequestTelemetry.test.ts b/libraries/rush-daemon/src/test/PhasedRequestTelemetry.test.ts new file mode 100644 index 0000000000..ead18ae7b1 --- /dev/null +++ b/libraries/rush-daemon/src/test/PhasedRequestTelemetry.test.ts @@ -0,0 +1,454 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IDaemonPhasedRequest } from '@rushstack/rush-daemon-protocol'; +import { + OperationStatus, + type IPhasedCommandEngineTelemetryOptions, + type IPhasedCommandEngineTelemetryRecord, + type ITelemetryData, + type Operation +} from '@microsoft/rush-lib'; + +import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import type { IPhasedRequestTelemetryReport, IPhasedRequestTelemetrySink } from '../PhasedRequestTelemetry'; +import { + createDaemonRequestTelemetryData, + createDaemonRequestTelemetrySink, + type IDaemonRequestTelemetryContext +} from '../DaemonRequestTelemetry'; +import { + TEST_ENGINE_SHAPE, + TestOperationRunner, + TestPhasedRequestClient, + createRoutingFixture +} from './PhasedRequestRouterTestUtilities'; +import type { ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; + +const OPERATION_A: string = 'project-a (_phase:test)'; +const OPERATION_B: string = 'project-b (_phase:test)'; + +const DAEMON_MEASURES: ReadonlyArray = [ + 'rush:daemon:admission', + 'rush:daemon:queueWait', + 'rush:daemon:acquireExecutionLease', + 'rush:daemon:reconcileInvalidations', + 'rush:daemon:applySelections', + 'rush:daemon:scheduleIteration', + 'rush:daemon:executeIteration' +]; + +function createRequest(requestId: string, ...operationIds: ReadonlyArray): IDaemonPhasedRequest { + return { + commandName: 'build', + commandOrigin: 'built-in', + engineShape: TEST_ENGINE_SHAPE, + environment: {}, + operationSelection: operationIds.map((operationId: string) => ({ enabledState: true, operationId })), + requestId + }; +} + +function createFixture(statusesA: ReadonlyArray = []): ITestRoutingFixture { + const pendingStatusesA: OperationStatus[] = [...statusesA]; + return createRoutingFixture( + new Map([ + [ + OPERATION_A, + new TestOperationRunner(OPERATION_A, OperationStatus.Success, async () => pendingStatusesA.shift()) + ], + [OPERATION_B, new TestOperationRunner(OPERATION_B)] + ]), + [[OPERATION_B, OPERATION_A]] + ); +} + +class RecordingTelemetrySink implements IPhasedRequestTelemetrySink { + public readonly reports: IPhasedRequestTelemetryReport[] = []; + public readonly resultsWrittenBeforeReport: boolean[] = []; + readonly #client: TestPhasedRequestClient; + + public constructor(client: TestPhasedRequestClient) { + this.#client = client; + } + + public logRequest(report: IPhasedRequestTelemetryReport): void { + this.resultsWrittenBeforeReport.push(this.#client.writes.some(({ result }) => result !== undefined)); + this.reports.push(report); + } +} + +function getStatuses(report: IPhasedRequestTelemetryReport): Record { + const statuses: Record = {}; + for (const [operation, record] of report.records) { + statuses[operation.name] = record.status; + } + return statuses; +} + +async function executeAsync( + router: PhasedRequestRouter, + request: IDaemonPhasedRequest +): Promise { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(request.requestId); + const sink: RecordingTelemetrySink = new RecordingTelemetrySink(client); + await router.executeAsync(request, client, false, undefined, undefined, sink); + return sink; +} + +describe('phased request telemetry', () => { + it('reports each coalesced request once, with its own selection, before its result is written', async () => { + const fixture: ITestRoutingFixture = createFixture(); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + try { + const [dependency, consumer] = await Promise.all([ + executeAsync(router, createRequest('dependency', OPERATION_A)), + executeAsync(router, createRequest('consumer', OPERATION_B)) + ]); + + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + for (const sink of [dependency, consumer]) { + expect(sink.reports).toHaveLength(1); + expect(sink.resultsWrittenBeforeReport).toEqual([false]); + const [report] = sink.reports; + expect(report).toMatchObject({ batchSize: 2, scheduled: true, countRetained: 0 }); + expect(report.result).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(report.measures.map(({ name }) => name)).toEqual(DAEMON_MEASURES); + for (const { startTimeMs, endTimeMs } of report.measures) { + expect(endTimeMs).toBeGreaterThanOrEqual(startTimeMs); + } + expect(report.receivedTimeMs).toBeLessThanOrEqual(report.executionStartTimeMs); + expect(report.executionStartTimeMs).toBeLessThanOrEqual(report.iterationStartTimeMs!); + expect(report.iterationStartTimeMs!).toBeLessThanOrEqual(report.resultTimeMs); + } + expect(dependency.reports[0].request.requestId).toBe('dependency'); + expect(getStatuses(dependency.reports[0])).toEqual({ [OPERATION_A]: OperationStatus.Success }); + expect(getStatuses(consumer.reports[0])).toEqual({ + [OPERATION_A]: OperationStatus.Success, + [OPERATION_B]: OperationStatus.Success + }); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('reports every request of an iteration before the next queued iteration starts', async () => { + const OPERATION_C: string = 'project-c (_phase:test)'; + let resolveStarted: () => void = () => undefined; + let resolveRelease: () => void = () => undefined; + const consumerStarted: Promise = new Promise((resolve) => (resolveStarted = resolve)); + const releaseConsumer: Promise = new Promise((resolve) => (resolveRelease = resolve)); + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A)], + [ + OPERATION_B, + new TestOperationRunner(OPERATION_B, OperationStatus.Success, async () => { + resolveStarted(); + await releaseConsumer; + }) + ], + [OPERATION_C, new TestOperationRunner(OPERATION_C)] + ]), + [[OPERATION_B, OPERATION_A]] + ); + // Like a telemetry plugin that starts a new session for each iteration and reads it in beforeLog. + let iteration: number = 0; + const events: string[] = []; + fixture.graph.hooks.beforeExecuteIterationAsync.tap('iteration session', () => { + events.push(`iteration ${++iteration}`); + }); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + async function executeRecordedAsync(requestId: string, operationId: string): Promise { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(requestId); + // A slow reader spreads each request's final flush, which precedes its report, over several event loop turns. + client.onWriteAsync = async (): Promise => { + await new Promise((resolve) => setImmediate(resolve)); + }; + await router.executeAsync(createRequest(requestId, operationId), client, false, undefined, undefined, { + logRequest: ({ batchSize, earlyResult }: IPhasedRequestTelemetryReport) => { + events.push(`log ${requestId} in iteration ${iteration} (batch ${batchSize}, early ${earlyResult})`); + } + }); + } + try { + const firstBatch: Promise = Promise.all([ + executeRecordedAsync('dependency', OPERATION_A), + executeRecordedAsync('consumer', OPERATION_B) + ]); + await consumerStarted; + const queued: Promise = executeRecordedAsync('queued', OPERATION_C); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(fixture.runners.get(OPERATION_C)?.runCount).toBe(0); + resolveRelease(); + await Promise.all([firstBatch, queued]); + + expect(events).toEqual([ + 'iteration 1', + 'log dependency in iteration 1 (batch 2, early true)', + 'log consumer in iteration 1 (batch 2, early false)', + 'iteration 2', + 'log queued in iteration 2 (batch 1, early false)' + ]); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('reports the operations of a request with no work as retained and skipped', async () => { + const fixture: ITestRoutingFixture = createFixture(); + let iteration: number = 0; + fixture.graph.hooks.configureIteration.tap('warm no-op', (records, previousResults) => { + if (iteration++ === 0) { + return; + } + for (const record of records.values()) { + if (previousResults.has(record.operation)) { + record.enabled = false; + } + } + }); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + try { + await executeAsync(router, createRequest('initial', OPERATION_B)); + const sink: RecordingTelemetrySink = await executeAsync(router, createRequest('repeat', OPERATION_B)); + + expect(fixture.runners.get(OPERATION_B)?.runCount).toBe(1); + expect(sink.reports).toHaveLength(1); + const [report] = sink.reports; + expect(report).toMatchObject({ batchSize: 1, scheduled: false, countRetained: 2 }); + expect(report.result).toMatchObject({ exitCode: 0, scheduled: false }); + expect(report.iterationStartTimeMs).toBeUndefined(); + expect(getStatuses(report)).toEqual({ + [OPERATION_A]: OperationStatus.Skipped, + [OPERATION_B]: OperationStatus.Skipped + }); + for (const record of report.records.values()) { + expect(record.stopwatch).toEqual({ + startTime: report.executionStartTimeMs, + endTime: report.executionStartTimeMs + }); + } + expect(report.measures.map(({ name }) => name)).not.toContain('rush:daemon:executeIteration'); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('reports a failed request and a later passing request separately', async () => { + const fixture: ITestRoutingFixture = createFixture([OperationStatus.Failure]); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + try { + const failed: RecordingTelemetrySink = await executeAsync(router, createRequest('failed', OPERATION_B)); + const passed: RecordingTelemetrySink = await executeAsync(router, createRequest('passed', OPERATION_B)); + + expect(failed.reports[0].result).toMatchObject({ exitCode: 1, outcome: 'failure' }); + expect(getStatuses(failed.reports[0])).toEqual({ + [OPERATION_A]: OperationStatus.Failure, + [OPERATION_B]: OperationStatus.Blocked + }); + expect(passed.reports[0].result).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(getStatuses(passed.reports[0])).toEqual({ + [OPERATION_A]: OperationStatus.Success, + [OPERATION_B]: OperationStatus.Success + }); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); + + it('does not report a request rejected before execution, and ignores sink errors', async () => { + const fixture: ITestRoutingFixture = createFixture(); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + try { + const rejectedSink: jest.Mocked = { logRequest: jest.fn() }; + await expect( + router.executeAsync( + createRequest('rejected', 'missing (_phase:test)'), + new TestPhasedRequestClient(), + false, + undefined, + undefined, + rejectedSink + ) + ).rejects.toThrow(); + expect(rejectedSink.logRequest).not.toHaveBeenCalled(); + + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + await expect( + router.executeAsync(createRequest('accepted', OPERATION_A), client, false, undefined, undefined, { + logRequest: () => { + throw new Error('telemetry failure'); + } + }) + ).resolves.toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(client.writes.some(({ result }) => result?.exitCode === 0)).toBe(true); + } finally { + await fixture.session[Symbol.asyncDispose](); + } + }); +}); + +describe(createDaemonRequestTelemetryData.name, () => { + // Other suites' graph iterations record native measures in the same process-wide timeline. + beforeEach(() => performance.clearMeasures()); + afterEach(() => performance.clearMeasures()); + + function createContext( + overrides: Partial = {} + ): IDaemonRequestTelemetryContext & { readonly calls: IPhasedCommandEngineTelemetryOptions[] } { + const fixture: ITestRoutingFixture = createFixture(); + const calls: IPhasedCommandEngineTelemetryOptions[] = []; + return { + calls, + command: { + createTelemetryData: (options: IPhasedCommandEngineTelemetryOptions): ITelemetryData => { + calls.push(options); + return { name: 'build', durationInSeconds: options.durationInSeconds, result: 'Succeeded' }; + } + }, + logTelemetry: jest.fn(), + workspaceSession: fixture.session, + lifecycleInfo: { receivedTimeMs: 100, preparedTimeMs: 150, reloadTier: 1 }, + resolveStartTimeMs: 150, + resolveEndTimeMs: 200, + engineCreation: undefined, + getRequestIndex: () => 4, + ...overrides + }; + } + + function createReport(overrides: Partial = {}): IPhasedRequestTelemetryReport { + const records: Map = new Map(); + return { + request: { + ...createRequest('request', OPERATION_A), + environment: { COPILOT_AGENT_SESSION_ID: 'agent-1', ODSP_TELEMETRY_TAG: 'tag-1' } + }, + result: { + aborted: false, + exitCode: 0, + operationResults: [], + outcome: 'success', + scheduled: true + } as unknown as IPhasedRequestTelemetryReport['result'], + records, + countRetained: 0, + batchSize: 2, + scheduled: true, + earlyResult: false, + receivedTimeMs: 210, + executionStartTimeMs: 250, + iterationStartTimeMs: 300, + resultTimeMs: 1300, + measures: [{ name: 'rush:daemon:queueWait', startTimeMs: 220, endTimeMs: 250 }], + ...overrides + }; + } + + it('measures the iteration as a native command does and adds the daemon fields', () => { + const context: ReturnType = createContext(); + createDaemonRequestTelemetryData(context, createReport()); + + const [options] = context.calls; + expect(options).toMatchObject({ succeeded: true, durationInSeconds: 1, timeOriginMs: 100 }); + expect(options.extraData).toMatchObject({ + daemon: true, + daemonPid: process.pid, + requestId: 'request', + reloadTier: 1, + requestIndex: 4, + batchSize: 2, + queueWaitSeconds: 0.03, + graphWasInitialized: true, + scheduled: true, + earlyResult: false, + countRetained: 0, + exitCode: 0, + outcome: 'success', + durationBasis: 'iteration', + bootDurationSeconds: 0.2, + totalDurationSeconds: 1.2, + agentSessionId: 'agent-1', + telemetryTag: 'tag-1' + }); + expect(typeof options.extraData?.daemonVersion).toBe('string'); + expect(options.extraData?.generationToken).toHaveLength(8); + expect(options.performanceEntries?.map(({ name, startTime, duration }) => [name, startTime, duration])).toEqual([ + ['rush:daemon:prepareWorkspace', 100, 50], + ['rush:daemon:resolve', 150, 50], + ['rush:daemon:queueWait', 220, 30] + ]); + }); + + it('attributes engine creation and its native measures to the request that created the engine', () => { + performance.measure('rush:test:beforeEngine', { start: 110, end: 115 }); + performance.measure('rush:test:createEngine', { start: 160, end: 170 }); + performance.measure('rush:executionManager:test', { start: 400, end: 900 }); + performance.measure('rush:executionManager:afterResult', { start: 400, end: 1400 }); + const context: ReturnType = createContext({ + engineCreation: { startTimeMs: 155, endTimeMs: 180 } + }); + createDaemonRequestTelemetryData(context, createReport()); + + const [options] = context.calls; + expect(options.extraData).toMatchObject({ graphWasInitialized: false }); + expect(options.performanceEntries?.map(({ name }) => name)).toEqual([ + 'rush:daemon:prepareWorkspace', + 'rush:daemon:createEngine', + 'rush:daemon:resolve', + 'rush:daemon:queueWait', + 'rush:test:createEngine', + 'rush:executionManager:test' + ]); + }); + + it('measures a request that needed no iteration from when its batch began handling it', () => { + const context: ReturnType = createContext({ lifecycleInfo: undefined }); + createDaemonRequestTelemetryData( + context, + createReport({ + scheduled: false, + iterationStartTimeMs: undefined, + resultTimeMs: 350, + result: { exitCode: 1, outcome: 'failure' } as unknown as IPhasedRequestTelemetryReport['result'] + }) + ); + + const [options] = context.calls; + expect(options).toMatchObject({ succeeded: false, durationInSeconds: 0.1, timeOriginMs: 150 }); + expect(options.extraData).toMatchObject({ durationBasis: 'batch', exitCode: 1, outcome: 'failure' }); + expect(options.extraData).not.toHaveProperty('reloadTier'); + expect(options.performanceEntries?.map(({ name }) => name)).toEqual([ + 'rush:daemon:resolve', + 'rush:daemon:queueWait' + ]); + }); + + it('runs beforeLog taps only for entries that an iteration served', () => { + const logTelemetry: jest.Mock = jest.fn(); + const sink: IPhasedRequestTelemetrySink = createDaemonRequestTelemetrySink(createContext({ logTelemetry })); + sink.logRequest(createReport()); + sink.logRequest(createReport({ scheduled: false, iterationStartTimeMs: undefined })); + + expect(logTelemetry.mock.calls.map(([, options]) => options)).toEqual([ + { servedByIteration: true }, + { servedByIteration: false } + ]); + }); + + it('logs through the engine and reports a logging failure without throwing', () => { + const context: ReturnType = createContext({ + logTelemetry: () => { + throw new Error('disk full'); + } + }); + const stderrSpy: jest.SpyInstance = jest.spyOn(process.stderr, 'write').mockImplementation(() => true); + try { + expect(() => createDaemonRequestTelemetrySink(context).logRequest(createReport())).not.toThrow(); + expect(stderrSpy).toHaveBeenCalledWith(expect.stringContaining('disk full')); + } finally { + stderrSpy.mockRestore(); + } + }); +}); diff --git a/libraries/rush-daemon/src/test/ProductionDaemonEnvironmentIdentity.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonEnvironmentIdentity.test.ts index 3425fd4dc9..645709e596 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonEnvironmentIdentity.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonEnvironmentIdentity.test.ts @@ -1,20 +1,111 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as path from 'node:path'; + import { PhasedCommandEngine } from '@microsoft/rush-lib'; +import type { IResolveDaemonRequestOptions } from '../DaemonRequestDispatcher'; import { ProductionDaemonRequestResolver } from '../ProductionDaemonRequestResolver'; import type { IWorkspaceSession } from '../WorkspaceSession'; import { createWireEnvelope } from './DaemonRequestWireTestUtilities'; describe('ProductionDaemonRequestResolver environment identity', () => { - it('ignores insertion order for distinct environment names with identical locale collation', async () => { - const names: string[] = ['RUSHD_TEST_\u00e9', 'RUSHD_TEST_e\u0301']; - const original: (string | undefined)[] = names.map((name) => process.env[name]); - const parse = jest.spyOn(PhasedCommandEngine, 'parseAsync').mockResolvedValue({ + const ADDED_NAME: string = 'RUSHD_TEST_PLUGIN_ADDED'; + const STARTUP_NAME: string = 'RUSHD_TEST_STARTUP'; + let parse: jest.SpyInstance; + beforeEach(() => { + parse = jest.spyOn(PhasedCommandEngine, 'parseAsync').mockResolvedValue({ commandName: 'build', parameterIdentity: 'parameters' } as PhasedCommandEngine); + }); + afterEach(() => { + parse.mockRestore(); + delete process.env[ADDED_NAME]; + delete process.env[STARTUP_NAME]; + }); + + function getEnvironment(): Record { + return Object.fromEntries( + Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) + ); + } + function createOptions(environment: Record): IResolveDaemonRequestOptions { + return { + abortSignal: new AbortController().signal, + envelope: createWireEnvelope('identity', 'build', process.cwd(), { + commandOrigin: 'built-in', + environment + }), + workspaceSession: { rushConfiguration: {} } as IWorkspaceSession + }; + } + + it('ignores names that the daemon process added after startup, as a plugin does', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const startupEnvironment: Record = getEnvironment(); + // A plugin that records its iteration id in process.env, in every iteration. + process.env[ADDED_NAME] = 'iteration-1'; + await expect( + resolver.getCommandParameterIdentityAsync(createOptions(startupEnvironment)) + ).resolves.toBe('parameters'); + process.env[ADDED_NAME] = 'iteration-2'; + await expect( + resolver.getCommandParameterIdentityAsync(createOptions(startupEnvironment)) + ).resolves.toBe('parameters'); + // A client that sets the added name is still a different environment. + await expect( + resolver.getCommandParameterIdentityAsync(createOptions({ ...startupEnvironment, [ADDED_NAME]: 'x' })) + ).rejects.toThrow('differs from the daemon startup environment'); + }); + + it('still rejects a live change to, or removal of, a startup name, and names it', async () => { + process.env[STARTUP_NAME] = 'startup'; + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(getEnvironment()); + const message: string = `The daemon's own environment changed after it started (${STARTUP_NAME})`; + process.env[STARTUP_NAME] = 'changed'; + await expect(resolver.getCommandParameterIdentityAsync(options)).rejects.toThrow(message); + delete process.env[STARTUP_NAME]; + await expect(resolver.getCommandParameterIdentityAsync(options)).rejects.toThrow(message); + process.env[STARTUP_NAME] = 'startup'; + await expect(resolver.getCommandParameterIdentityAsync(options)).resolves.toBe('parameters'); + }); + + it('keeps serving with a startup PATH that repeats an entry, and still names a changed PATH', async () => { + const originalPath: string | undefined = process.env.PATH; + const repeatedEntry: string = path.resolve('/rushd-test-repeated-path-entry'); + const otherEntries: string[] = originalPath === undefined ? [] : [originalPath]; + try { + process.env.PATH = [repeatedEntry, repeatedEntry, ...otherEntries].join(path.delimiter); + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(getEnvironment()); + await expect(resolver.getCommandParameterIdentityAsync(options)).resolves.toBe('parameters'); + // Only the repeated entry differs, so the fingerprint is the same. + process.env.PATH = [repeatedEntry, ...otherEntries].join(path.delimiter); + await expect(resolver.getCommandParameterIdentityAsync(options)).resolves.toBe('parameters'); + process.env.PATH = otherEntries.join(path.delimiter); + await expect(resolver.getCommandParameterIdentityAsync(options)).rejects.toThrow( + "The daemon's own environment changed after it started (PATH)" + ); + } finally { + if (originalPath === undefined) delete process.env.PATH; + else process.env.PATH = originalPath; + } + }); + + it('keeps the startup environment for a replacement session', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(getEnvironment()); + process.env[ADDED_NAME] = 'iteration-1'; + const replacement: ProductionDaemonRequestResolver = resolver.createForSession(); + await expect(replacement.getCommandParameterIdentityAsync(options)).resolves.toBe('parameters'); + }); + + it('ignores insertion order for distinct environment names with identical locale collation', async () => { + const names: string[] = ['RUSHD_TEST_\u00e9', 'RUSHD_TEST_e\u0301']; + const original: (string | undefined)[] = names.map((name) => process.env[name]); try { process.env[names[0]] = 'composed'; process.env[names[1]] = 'decomposed'; @@ -38,7 +129,6 @@ describe('ProductionDaemonRequestResolver environment identity', () => { 'differs from the daemon startup environment' ); } finally { - parse.mockRestore(); names.forEach((name, index) => { if (original[index] === undefined) delete process.env[name]; else process.env[name] = original[index]; diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index f307a3a75d..3a8426c049 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -7,44 +7,33 @@ import * as path from 'node:path'; import { execFileSync } from 'node:child_process'; import { - Rush, + PhasedCommandEngine, RushProjectConfiguration, - RushUserConfiguration, type IOperationGraph, + type IPhasedCommandEngine, + type ITelemetryData, type Operation, type RushConfigurationProject } from '@microsoft/rush-lib'; +import type { LockFile } from '@rushstack/node-core-library'; import { DaemonFrameType, decodeDaemonEventFrame, - decodeDaemonLogChunk, - type IDaemonPhasedRequestResult, - type IDaemonRequestEnvelope + type IDaemonPhasedRequestResult } from '@rushstack/rush-daemon-protocol'; import { NoOpTerminalProvider, Terminal, TerminalProviderSeverity } from '@rushstack/terminal'; import { StandardScriptUpdater } from '@microsoft/rush-lib/lib/logic/StandardScriptUpdater'; import { RushConfiguration as InternalRushConfiguration } from '@microsoft/rush-lib/lib/api/RushConfiguration'; import { ProductionDaemonRequestResolver } from '../ProductionDaemonRequestResolver'; -import { RushDaemonHost } from '../RushDaemonHost'; -import { WorkspaceSession } from '../WorkspaceSession'; +import type { WorkspaceSession } from '../WorkspaceSession'; import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; -import { removeTestFolderAsync } from './TestProcessExit'; import { readDaemonLockfile } from '@rushstack/rush-daemon-transport'; import { EngineTerminalProvider } from '../EngineTerminalProvider'; import { DaemonShutdownError } from '../DaemonShutdownError'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; -import type { - GetWorkspaceSuccessorLaunchAsync, - IWorkspaceProcessRestartResult -} from '../WorkspaceProcessRestart'; +import type { IWorkspaceProcessRestartResult } from '../WorkspaceProcessRestart'; import type { IResolveDaemonRequestOptions, ResolvedDaemonRequest } from '../DaemonRequestDispatcher'; -import type { IDaemonRequestResolver } from '../DaemonRequestDispatcher'; -import { - isRushxInvocation, - wrapWorkspaceResolverLifecycle, - type IWorkspaceResolverLifecycle -} from '../WorkspaceResolverLifecycle'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; import type { IRequestLease } from '../RequestScheduler'; import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; @@ -62,314 +51,24 @@ import { type IDeferred, type ITerminalExchange } from './DaemonRequestWireTestUtilities'; +import { + createFixtureAsync, + DecoratedTestResolver, + logText, + requestEnvironment, + runAsync, + runs, + type IFixture +} from './NativeEngineTestFixture'; -const RUSH_VERSION: string = Rush.version; jest.setTimeout(30_000); -interface IFixture extends AsyncDisposable { - readonly repoRoot: string; - readonly host: RushDaemonHost; - readonly session: WorkspaceSession; - readonly client: DaemonRequestWireClient; -} - -interface IFixtureOptions { - readonly getSuccessorLaunchAsync?: GetWorkspaceSuccessorLaunchAsync; - readonly onSessionCreated?: (session: WorkspaceSession) => void; - readonly resolver?: IDaemonRequestResolver; - /** Adds the `_phase:compile:incremental` script, which passes `--incremental` to build.cjs. */ - readonly incrementalScript?: boolean; - /** Sets `daemon.incrementalBuilds` in rush.json. */ - readonly incrementalBuilds?: boolean; - /** Uses PNPM, which installs a dependency file (shrinkwrap-deps.json) that change detection hashes per project. */ - readonly pnpm?: boolean; -} - -class DecoratedTestResolver implements IDaemonRequestResolver { - public readonly workspaceLifecycle: IWorkspaceResolverLifecycle | undefined; - private readonly _inner: IDaemonRequestResolver; - private readonly _events: string[]; - private readonly _id: number; - private _disposed: boolean = false; - - public constructor(inner: IDaemonRequestResolver, events: string[]) { - this._inner = inner; - this._events = events; - this._id = events.filter((event) => event.startsWith('created')).length; - events.push(`created:${this._id}`); - this.workspaceLifecycle = wrapWorkspaceResolverLifecycle( - inner, - (replacement) => new DecoratedTestResolver(replacement, events) - ); - } - - public async resolveRequestAsync(options: IResolveDaemonRequestOptions): Promise { - if (this._disposed) throw new Error('A disposed resolver was invoked.'); - if (isRushxInvocation(options.envelope)) { - this._events.push(`isolated:${this._id}`); - throw new Error('Explicit isolated invocation reached the decorated resolver.'); - } - return await this._inner.resolveRequestAsync(options); - } - - public async [Symbol.asyncDispose](): Promise { - if (this._disposed) throw new Error('A resolver was disposed twice.'); - this._disposed = true; - this._events.push(`disposed:${this._id}`); - await this._inner[Symbol.asyncDispose]?.(); - } -} - -async function createFixtureAsync( - cache: boolean = false, - configurationKind: 'direct' | 'rig' | 'inherited' = 'direct', - options: IFixtureOptions = {} -): Promise { - const repoRoot: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-native-engine-')); - const cacheNamespace: string = path.basename(repoRoot); - const userConfiguration: RushUserConfiguration = await RushUserConfiguration.initializeAsync(); - const cacheFolder: string = path.join( - userConfiguration.buildCacheFolder ?? path.join(repoRoot, 'common/temp/build-cache'), - cacheNamespace - ); - const write: (name: string, text: string) => void = (name, text) => { - const filename: string = path.join(repoRoot, name); - fs.mkdirSync(path.dirname(filename), { recursive: true }); - fs.writeFileSync(filename, text); - }; - write( - 'rush.json', - JSON.stringify({ - rushVersion: RUSH_VERSION, - ...(options.pnpm ? { pnpmVersion: '9.15.9' } : { npmVersion: '10.0.0' }), - // Retention assertions must not depend on the surrounding Jest worker's accumulated RSS. - daemon: { - warmMemoryBudgetMB: 100_000, - ...(options.incrementalBuilds === undefined ? {} : { incrementalBuilds: options.incrementalBuilds }) - }, - projectFolderMinDepth: 2, - projectFolderMaxDepth: 2, - projects: ['a', 'b', 'c'].map((name) => ({ - packageName: name, - projectFolder: `projects/${name}` - })) - }) - ); - write('.gitignore', 'common/temp/\n**/.rush/\n**/rush-logs/\n**/lib/\n**/node_modules/\nruns.txt\n'); - write('common/temp/last-link.flag', '{}'); - if (options.pnpm) { - write('common/config/rush/pnpm-lock.yaml', "lockfileVersion: '9.0'\n"); - } else { - write('common/config/rush/npm-shrinkwrap.json', '{"lockfileVersion":3,"packages":{}}'); - } - write( - 'common/config/rush/command-line.json', - JSON.stringify({ - phases: [{ name: '_phase:compile', dependencies: { upstream: ['_phase:compile'] } }], - commands: [ - { - commandKind: 'phased', - name: 'build', - phases: ['_phase:compile'], - incremental: true, - enableParallelism: true - } - ], - parameters: [ - { - parameterKind: 'flag', - longName: '--production', - description: 'Production build', - associatedCommands: ['build'], - associatedPhases: ['_phase:compile'] - } - ] - }) - ); - if (cache) { - write( - 'common/config/rush/build-cache.json', - JSON.stringify({ - buildCacheEnabled: true, - cacheProvider: 'local-only', - cacheEntryNamePattern: `${cacheNamespace}/[hash]` - }) - ); - } - for (const name of ['a', 'b', 'c']) { - write( - `projects/${name}/package.json`, - JSON.stringify({ - name, - version: '1.0.0', - scripts: { - '_phase:compile': 'node build.cjs', - ...(options.incrementalScript - ? { '_phase:compile:incremental': 'node build.cjs --incremental' } - : {}) - }, - dependencies: name === 'b' ? { a: '1.0.0' } : {} - }) - ); - write( - `projects/${name}/config/rush-project.json`, - JSON.stringify({ - operationSettings: [{ operationName: '_phase:compile', outputFolderNames: ['lib'] }] - }) - ); - write(`projects/${name}/input.txt`, 'one'); - if (options.pnpm) write(`projects/${name}/.rush/temp/shrinkwrap-deps.json`, '{}'); - write( - `projects/${name}/build.cjs`, - ` -const fs = require('node:fs'); -const path = require('node:path'); -const name = require('./package.json').name; -const input = fs.readFileSync(fs.existsSync('src/input.txt') ? 'src/input.txt' : 'input.txt', 'utf8'); -(async () => { -const gateFile = path.resolve('../../common/temp/gate-' + name + '.json'); -if (fs.existsSync(gateFile)) { - const { port } = JSON.parse(fs.readFileSync(gateFile, 'utf8')); - await new Promise((resolve, reject) => { - const socket = require('node:net').connect(port, '127.0.0.1'); - socket.once('error', reject); - socket.once('data', () => { socket.end(); resolve(); }); - }); -} -fs.appendFileSync('../../runs.txt', name + ':' + input + ':' + process.argv.slice(2).join(' ') + '\\n'); -const environmentFile = path.resolve('../../common/temp/operation-environment.txt'); -if (fs.existsSync(environmentFile)) { - const { COPILOT_AGENT_SESSION_ID = null, RUSH_INVOKED_FOLDER = null } = process.env; - fs.appendFileSync(environmentFile, JSON.stringify([name, COPILOT_AGENT_SESSION_ID, RUSH_INVOKED_FOLDER]) + '\\n'); -} -fs.mkdirSync('lib', { recursive: true }); -fs.writeFileSync('lib/output.txt', input); -console.log('built-' + name + '-' + input); -if (input === 'warning') console.error('warning-' + name); -if (input === 'failure') process.exitCode = 7; -})().catch((error) => { console.error(error); process.exitCode = 1; }); -` - ); - } - if (configurationKind === 'rig') { - fs.rmSync(path.join(repoRoot, 'projects/a/config/rush-project.json')); - write('projects/a/config/rig.json', '{"rigPackageName":"fixture-rig"}'); - write('projects/a/node_modules/fixture-rig/package.json', '{"name":"fixture-rig","version":"1.0.0"}'); - write( - 'projects/a/node_modules/fixture-rig/profiles/default/config/rush-project.json', - JSON.stringify({ - operationSettings: [{ operationName: '_phase:compile', outputFolderNames: ['lib'] }] - }) - ); - } else if (configurationKind === 'inherited') { - write( - 'common/temp/inherited-rush-project.json', - JSON.stringify({ - operationSettings: [{ operationName: '_phase:compile', outputFolderNames: ['lib'] }] - }) - ); - write( - 'projects/a/config/rush-project.json', - JSON.stringify({ - extends: '../../../common/temp/inherited-rush-project.json', - incrementalBuildIgnoredGlobs: ['ignored.txt'] - }) - ); - } - execFileSync('git', ['init', '--quiet'], { cwd: repoRoot }); - execFileSync('git', ['config', '--local', 'core.autocrlf', 'false'], { cwd: repoRoot }); - execFileSync('git', ['add', '.'], { cwd: repoRoot }); - execFileSync( - 'git', - [ - '-c', - 'user.name=Engine Test', - '-c', - 'user.email=engine@example.invalid', - 'commit', - '--quiet', - '-m', - 'fixture' - ], - { cwd: repoRoot } - ); - let session: WorkspaceSession | undefined; - let host: RushDaemonHost | undefined; - try { - host = await RushDaemonHost.startAsync({ - repoRoot, - rushVersion: RUSH_VERSION, - daemonVersion: 'native-engine-test', - requestResolver: options.resolver ?? new ProductionDaemonRequestResolver(), - getSuccessorLaunchAsync: options.getSuccessorLaunchAsync, - createWorkspaceSessionAsync: async (sessionOptions) => { - session = await WorkspaceSession.createAsync(sessionOptions); - options.onSessionCreated?.(session); - return session; - } - }); - const client: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync(host.paths.socketPath); - await client.handshakeAsync(); - const runningHost: RushDaemonHost = host; - return { - repoRoot, - host, - get session(): WorkspaceSession { - return session!; - }, - client, - [Symbol.asyncDispose]: async () => { - await client.closeAsync().finally(() => runningHost.closeAsync()); - if (cache) await removeTestFolderAsync(cacheFolder, true); - await removeTestFolderAsync(repoRoot, true); - } - }; - } catch (error) { - await host?.closeAsync(); - if (cache) await removeTestFolderAsync(cacheFolder, true); - await removeTestFolderAsync(repoRoot, true); - throw error; - } -} - -async function runAsync( - fixture: IFixture, - requestId: string, - argv: string[], - overrides: Partial = {} -): Promise { - const environment: Record = {}; - for (const [name, value] of Object.entries(process.env)) { - if (value !== undefined) environment[name] = value; - } - await fixture.client.sendControlAsync({ - kind: 'requestStart', - payload: createWireEnvelope(requestId, argv[0], fixture.repoRoot, { - argv, - environment, - commandOrigin: 'built-in', - ...overrides - }) - }); - return await fixture.client.readTerminalAsync(requestId); -} - -function runs(fixture: IFixture): string[] { - const filename: string = path.join(fixture.repoRoot, 'runs.txt'); - return fs.existsSync(filename) ? fs.readFileSync(filename, 'utf8').trim().split('\n') : []; -} - -function logText(exchange: ITerminalExchange): string { - return exchange.frames - .filter((frame) => frame.kind === DaemonFrameType.logStdout || frame.kind === DaemonFrameType.logStderr) - .map((frame) => Buffer.from(decodeDaemonLogChunk(frame.payload).chunk).toString()) - .join(''); -} - -function requestEnvironment(): Record { - return Object.fromEntries( - Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) - ); +function readTelemetryEntries(repoRoot: string): ITelemetryData[] { + const folder: string = path.join(repoRoot, 'common/temp/telemetry'); + return fs + .readdirSync(folder) + .sort() + .flatMap((name: string) => JSON.parse(fs.readFileSync(path.join(folder, name), 'utf8'))); } describe('native production daemon engine', () => { @@ -1970,4 +1669,124 @@ process.exit(23); await fixture[Symbol.asyncDispose](); } }); + + it('logs one native telemetry entry for each build request served by the warm engine', async () => { + const fixture: IFixture = await createFixtureAsync(false, 'direct', { telemetryEnabled: true }); + try { + const beforeLogIndexes: unknown[] = []; + for (const [requestId, argv] of [ + ['initial', ['build', '--only', 'a']], + ['repeat', ['build', '--only', 'a']], + ['consumer', ['build', '--to', 'b']] + ] as const) { + expect((await runAsync(fixture, requestId, [...argv])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0 } + }); + if (requestId === 'initial') { + fixture.session.operationGraph!.hooks.beforeLog.tap('test', (data: ITelemetryData) => { + beforeLogIndexes.push(data.extraData?.requestIndex); + }); + } + } + const entries: ITelemetryData[] = readTelemetryEntries(fixture.repoRoot); + + expect(entries).toHaveLength(3); + expect(entries.map(({ name, result }) => [name, result])).toEqual([ + ['build', 'Succeeded'], + ['build', 'Succeeded'], + ['build', 'Succeeded'] + ]); + expect(entries[0].extraData).toMatchObject({ + daemon: true, + requestIndex: 1, + graphWasInitialized: false, + scheduled: true, + durationBasis: 'iteration', + isInitial: true, + isWatch: false, + command_only: 'true', + '--only': 'a', + countAll: 1, + countSuccess: 1 + }); + expect(entries[1].extraData).toMatchObject({ + requestIndex: 2, + graphWasInitialized: true, + scheduled: false, + durationBasis: 'batch', + countAll: 1, + countSkipped: 1, + countRetained: 1 + }); + expect(entries[2].extraData).toMatchObject({ + requestIndex: 3, + scheduled: true, + command_only: 'false', + command_to: 'true', + '--to': 'b', + countAll: 2, + countSuccess: 1, + countSkipped: 1, + countRetained: 1 + }); + for (const entry of entries) { + const totalSeconds: number = entry.extraData!.totalDurationSeconds as number; + const requestMs: number = totalSeconds * 1000; + expect(entry.durationInSeconds).toBeLessThanOrEqual(totalSeconds); + expect(entry.extraData!.bootDurationSeconds).toBeLessThanOrEqual(totalSeconds); + for (const { startTimestampMs, endTimestampMs } of Object.values(entry.operationResults!)) { + expect(startTimestampMs).toBeGreaterThanOrEqual(0); + expect(endTimestampMs).toBeLessThanOrEqual(requestMs); + } + expect(entry.performanceEntries?.map(({ name }) => name)).toContain('rush:daemon:resolve'); + } + // The repeated request needed no iteration, so iteration-scoped beforeLog taps do not see its entry. + expect(beforeLogIndexes).toEqual([3]); + expect(runs(fixture)).toEqual(['a:one:', 'b:one:']); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + + it('releases the lockfile within seconds when a flushTelemetry tap never settles', async () => { + const stalledUploads: string[] = []; + const createEngineAsync = PhasedCommandEngine.prototype.createEngineAsync; + const createEngineSpy: jest.SpyInstance = jest + .spyOn(PhasedCommandEngine.prototype, 'createEngineAsync') + .mockImplementation(async function ( + this: PhasedCommandEngine, + lock?: LockFile + ): Promise { + const engine: IPhasedCommandEngine = await createEngineAsync.call(this, lock); + // Like an upload to a server that accepts the connection and never answers. + engine.rushSession.hooks.flushTelemetry.tapPromise('StalledUpload', (data) => { + stalledUploads.push(...data.map(({ name }) => name)); + return new Promise(() => undefined); + }); + return engine; + }); + const fixture: IFixture = await createFixtureAsync(false, 'direct', { telemetryEnabled: true }); + try { + expect((await runAsync(fixture, 'initial', ['build', '--only', 'a'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0 } + }); + expect(stalledUploads).toEqual(['build']); + + const startMs: number = Date.now(); + await fixture.host.closeAsync(); + const elapsedMs: number = Date.now() - startMs; + + // The engine gives the upload 2 seconds, then the host releases its lockfile and socket, well before a + // waiting client gives up on the handoff and falls back to in-process Rush. + expect(elapsedMs).toBeGreaterThanOrEqual(1900); + expect(elapsedMs).toBeLessThan(8000); + expect(readDaemonLockfile(fixture.host.paths.lockfilePath)).toBeUndefined(); + expect(fs.existsSync(fixture.host.paths.socketPath)).toBe(false); + } finally { + createEngineSpy.mockRestore(); + await fixture[Symbol.asyncDispose](); + } + }); }); diff --git a/libraries/rush-daemon/src/test/WorkspacePluginEnvironment.test.ts b/libraries/rush-daemon/src/test/WorkspacePluginEnvironment.test.ts new file mode 100644 index 0000000000..6f6cb6ff1c --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspacePluginEnvironment.test.ts @@ -0,0 +1,40 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; + +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { setDaemonPolicy } from './WarmGenerationTestUtilities'; + +jest.setTimeout(30_000); + +// Engine code runs in the daemon process, and a Rush plugin may add its own names to process.env there. +const ADDED_NAME: string = 'RUSHD_TEST_PLUGIN_ADDED'; + +afterEach(() => { + delete process.env[ADDED_NAME]; +}); + +it('keeps serving the warm generation after a plugin adds a name to process.env', async () => { + const fixture = await DaemonGraphTestFixture.createAsync((created) => setDaemonPolicy(created, {})); + try { + await fixture.buildSuccessfullyAsync(); + await fixture.buildSuccessfullyAsync(); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(WorkspaceInputChangeTier.Reuse); + const generation: number = fixture.host.workspaceGeneration; + const graph = fixture.session.operationGraph; + process.env[ADDED_NAME] = 'iteration-1'; + + expect((await fixture.graphAsync('invalidate', '--project', 'a')).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0 } + }); + process.env[ADDED_NAME] = 'iteration-2'; + await fixture.buildSuccessfullyAsync(); + expect(fixture.host.workspaceStatus.lastReloadTier).toBe(WorkspaceInputChangeTier.Reuse); + expect(fixture.host.workspaceGeneration).toBe(generation); + expect(fixture.session.operationGraph).toBe(graph); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); diff --git a/libraries/rush-lib/src/api/PhasedCommandEngine.ts b/libraries/rush-lib/src/api/PhasedCommandEngine.ts index 56940a44ca..8172ff36c0 100644 --- a/libraries/rush-lib/src/api/PhasedCommandEngine.ts +++ b/libraries/rush-lib/src/api/PhasedCommandEngine.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import * as path from 'node:path'; +import type { PerformanceEntry } from 'node:perf_hooks'; import { FileSystem, LockFile } from '@rushstack/node-core-library'; import { Terminal, type ITerminalProvider } from '@rushstack/terminal'; @@ -12,14 +13,20 @@ import { PhasedScriptAction } from '../cli/scriptActions/PhasedScriptAction'; import type { GetInputsSnapshotAsyncFn, IInputsSnapshot } from '../logic/incremental/InputsSnapshot'; import type { IOperationGraph } from '../logic/operations/IOperationGraph'; import type { Operation, OperationEnabledState } from '../logic/operations/Operation'; +import type { OperationStatus } from '../logic/operations/OperationStatus'; import type { Parallelism } from '../logic/operations/ParseParallelism'; import { PhasedCommandEngineExecution } from '../logic/operations/PhasedCommandEngineExecution'; +import { createPhasedTelemetryData } from '../logic/operations/PhasedCommandTelemetry'; +import { type ITelemetryData, Telemetry } from '../logic/Telemetry'; import type { RushSession } from '../pluginFramework/RushSession'; import type { RushConfiguration } from './RushConfiguration'; import { RushUserConfiguration } from './RushUserConfiguration'; import { PhasedCommandEngineBusyError } from './PhasedCommandEngineBusyError'; import { resolvePhasedCommandCwdAsync } from '../utilities/resolvePhasedCommandCwd'; +/** How long disposing an engine waits for `flushTelemetry` taps that are still running. */ +const TELEMETRY_FLUSH_WAIT_MS: number = 2000; + /** * A native phased command graph prepared without executing an iteration. * @alpha @@ -35,6 +42,65 @@ export interface IPhasedCommandEngine extends AsyncDisposable { readonly phaseNames: ReadonlyArray; readonly pluginNames: ReadonlyArray; readonly isIncremental: boolean; + /** + * Logs one request's telemetry entry the way a native iteration logs its own: `beforeLog` taps run first, then + * the entry is saved under `common/temp/telemetry` and passed to `flushTelemetry` taps. Disposing the engine waits + * up to 2 seconds for taps that are still running; a tap that takes longer, such as an upload over a stalled + * network, keeps running in the background, so that it cannot hold the host. Saving does nothing when telemetry + * is disabled for the repository. + * + * @remarks + * `beforeLog` taps describe the latest iteration, so a host logs an iteration's entries before it starts the + * next iteration. + */ + readonly logTelemetry?: (data: ITelemetryData, options?: IPhasedCommandEngineLogTelemetryOptions) => void; +} + +/** + * Options for `IPhasedCommandEngine.logTelemetry`. + * @alpha + */ +export interface IPhasedCommandEngineLogTelemetryOptions { + /** + * Whether a graph iteration served the request. If `false`, as for a request that the warm graph answered + * without an iteration, `beforeLog` taps are skipped, because they would describe an earlier iteration. + * Defaults to `true`. + */ + readonly servedByIteration?: boolean; +} + +/** + * One operation's result in a request-scoped telemetry entry. + * @alpha + */ +export interface IPhasedCommandEngineTelemetryRecord { + /** The operation's status in this request. */ + readonly status: OperationStatus; + /** Whether the operation's runner is silent. Silent operations are omitted from the entry. */ + readonly silent: boolean; + /** `performance.now()` values for when the operation started and ended in this request. */ + readonly stopwatch: { readonly startTime: number | undefined; readonly endTime: number | undefined }; + /** How long the operation would have taken without the build cache. */ + readonly nonCachedDurationMs: number | undefined; +} + +/** + * The results of one request served by a long-lived engine. + * @alpha + */ +export interface IPhasedCommandEngineTelemetryOptions { + /** The results of the request's selected operations. Silent operations are omitted from the entry. */ + readonly records: ReadonlyMap; + /** Whether the request succeeded. As for a native command, only a `Success` status counts as success. */ + readonly succeeded: boolean; + /** How long the request's graph iteration took, in seconds. */ + readonly durationInSeconds: number; + /** A `performance.now()` value. Operation and performance entry times are reported relative to it. */ + readonly timeOriginMs: number; + /** Host-specific fields, added after the native fields. */ + readonly extraData?: Readonly>; + /** The request's performance entries, with `performance.now()` start times. */ + readonly performanceEntries?: ReadonlyArray; } /** Options for parsing a command for a long-lived engine host. @alpha */ @@ -187,10 +253,25 @@ export class PhasedCommandEngine { engine, this._parser.rushConfiguration.commonTempFolder ); + const { operationGraph } = engine; + const telemetry: Telemetry = new Telemetry(this._parser.rushConfiguration, this._parser.rushSession); return { ...engine, acquireExecutionLeaseAsync: () => execution.acquireExecutionLeaseAsync(), - [Symbol.asyncDispose]: () => execution[Symbol.asyncDispose]() + logTelemetry: (data: ITelemetryData, options?: IPhasedCommandEngineLogTelemetryOptions) => { + if (options?.servedByIteration !== false) { + operationGraph.hooks.beforeLog.call(data); + } + telemetry.log(data); + telemetry.flush(); + }, + [Symbol.asyncDispose]: async () => { + try { + await execution[Symbol.asyncDispose](); + } finally { + await waitForTelemetryFlushAsync(telemetry.ensureFlushedAsync(), TELEMETRY_FLUSH_WAIT_MS); + } + } }; } catch (error) { const cleanupErrors: unknown[] = []; @@ -229,4 +310,71 @@ export class PhasedCommandEngine { public get requestSettings(): IPhasedCommandEngineRequestSettings { return this._action.getEngineRequestSettings(); } + + /** + * Builds the native telemetry entry for one request that this command made of a long-lived engine. + * + * @remarks + * The entry carries this command's own parameters, not those of the command that created the engine, and reports + * one initial, non-watch execution, as the native command's entry would. + */ + public createTelemetryData(options: IPhasedCommandEngineTelemetryOptions): ITelemetryData { + const { timeOriginMs } = options; + const data: ITelemetryData = createPhasedTelemetryData({ + ...this._action.getTelemetryFields(), + isWatch: false, + isInitial: true, + durationInSeconds: options.durationInSeconds, + succeeded: options.succeeded, + records: options.records, + timeOriginMs + }); + return { + ...data, + extraData: { ...data.extraData, ...options.extraData }, + performanceEntries: (options.performanceEntries ?? []).map((entry: PerformanceEntry) => + rebasePerformanceEntry(entry, timeOriginMs) + ) + }; + } +} + +function rebasePerformanceEntry(entry: PerformanceEntry, timeOriginMs: number): PerformanceEntry { + const { name, entryType, duration, detail } = entry; + const startTime: number = entry.startTime - timeOriginMs; + return { + name, + entryType, + startTime, + duration, + detail, + toJSON: () => ({ name, entryType, startTime, duration, detail }) + }; +} + +/** + * Waits for pending `flushTelemetry` taps, but no longer than `timeoutMs`. As in the native CLI, a failed tap does + * not fail the command. + * + * @returns `true` if the taps settled in time, or `false` if they are still running. + */ +export async function waitForTelemetryFlushAsync( + flushPromise: Promise, + timeoutMs: number +): Promise { + let timeout: NodeJS.Timeout | undefined; + const expiredPromise: Promise = new Promise((resolve) => { + timeout = setTimeout(() => resolve(false), timeoutMs); + }); + try { + return await Promise.race([ + flushPromise.then( + () => true, + () => true + ), + expiredPromise + ]); + } finally { + clearTimeout(timeout); + } } diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts new file mode 100644 index 0000000000..3e9931220d --- /dev/null +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts @@ -0,0 +1,256 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import type { PerformanceEntry } from 'node:perf_hooks'; + +import { NoOpTerminalProvider } from '@rushstack/terminal'; + +import { + PhasedCommandEngine, + waitForTelemetryFlushAsync, + type IPhasedCommandEngineTelemetryOptions, + type IPhasedCommandEngineTelemetryRecord +} from '../PhasedCommandEngine'; +import { RushConfiguration } from '../RushConfiguration'; +import { Rush } from '../Rush'; +import type { IPhase } from '../CommandLineConfiguration'; +import type { ITelemetryData } from '../../logic/Telemetry'; +import { Operation } from '../../logic/operations/Operation'; +import { OperationStatus } from '../../logic/operations/OperationStatus'; +import { MockOperationRunner } from '../../logic/operations/test/MockOperationRunner'; + +describe(`${PhasedCommandEngine.name} telemetry`, () => { + let folder: string; + let rushConfiguration: RushConfiguration; + let operationA: Operation; + let operationB: Operation; + + function write(name: string, value: unknown): void { + const filename: string = path.join(folder, name); + fs.mkdirSync(path.dirname(filename), { recursive: true }); + fs.writeFileSync(filename, JSON.stringify(value)); + } + + async function parseAsync(...argv: string[]): Promise { + return await PhasedCommandEngine.parseAsync({ + argv, + cwd: folder, + rushConfiguration, + terminalProvider: new NoOpTerminalProvider() + }); + } + + function createRecord( + status: OperationStatus, + startTime: number | undefined, + endTime: number | undefined + ): IPhasedCommandEngineTelemetryRecord { + return { status, silent: false, stopwatch: { startTime, endTime }, nonCachedDurationMs: undefined }; + } + + function createOptions( + overrides: Partial = {} + ): IPhasedCommandEngineTelemetryOptions { + return { + records: new Map([ + [operationB, createRecord(OperationStatus.Skipped, 1500, 1500)], + [operationA, createRecord(OperationStatus.Success, 1600, 1900)] + ]), + succeeded: true, + durationInSeconds: 0.4, + timeOriginMs: 1000, + ...overrides + }; + } + + beforeAll(() => { + folder = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'rush-engine-telemetry-'))); + write('rush.json', { + rushVersion: Rush.version, + npmVersion: '10.0.0', + projectFolderMinDepth: 1, + projects: [ + { packageName: 'a', projectFolder: 'a' }, + { packageName: 'b', projectFolder: 'b' } + ] + }); + write('a/package.json', { + name: 'a', + version: '1.0.0', + dependencies: { b: 'workspace:*' }, + scripts: { '_phase:compile': 'node -v' } + }); + write('b/package.json', { name: 'b', version: '1.0.0', scripts: { '_phase:compile': 'node -v' } }); + write('common/config/rush/command-line.json', { + phases: [{ name: '_phase:compile', dependencies: { upstream: ['_phase:compile'] } }], + commands: [ + { + commandKind: 'phased', + name: 'build', + summary: 'Build', + phases: ['_phase:compile'], + incremental: true, + enableParallelism: true + } + ], + parameters: [ + { + parameterKind: 'flag', + longName: '--production', + description: 'A graph-affecting custom parameter', + associatedCommands: ['build'], + associatedPhases: ['_phase:compile'] + } + ] + }); + rushConfiguration = RushConfiguration.loadFromConfigurationFile(path.join(folder, 'rush.json')); + const phase: IPhase = { name: '_phase:compile' } as IPhase; + operationB = new Operation({ + phase, + project: rushConfiguration.getProjectByName('b')!, + runner: new MockOperationRunner('b (compile)'), + logFilenameIdentifier: 'b_compile' + }); + operationA = new Operation({ + phase, + project: rushConfiguration.getProjectByName('a')!, + runner: new MockOperationRunner('a (compile)'), + logFilenameIdentifier: 'a_compile' + }); + operationA.addDependency(operationB); + }); + + afterAll(() => { + fs.rmSync(folder, { recursive: true, force: true }); + }); + + it("reports the request's own parameters as one initial, non-watch execution", async () => { + const first: PhasedCommandEngine = await parseAsync('build', '--to', 'a', '--production'); + const second: PhasedCommandEngine = await parseAsync('build', '--only', 'b'); + const firstData: ITelemetryData = first.createTelemetryData(createOptions()); + const secondData: ITelemetryData = second.createTelemetryData(createOptions()); + + expect(firstData).toMatchObject({ + name: 'build', + result: 'Succeeded', + durationInSeconds: 0.4, + extraData: { + isWatch: false, + isInitial: true, + command_to: 'true', + command_only: 'false', + '--to': 'a', + '--production': 'true', + '--changed-projects-only': false, + countAll: 2, + countSuccess: 1, + countSkipped: 1 + } + }); + expect(secondData.extraData).toMatchObject({ + command_to: 'false', + command_only: 'true', + '--only': 'b', + '--production': 'false' + }); + }); + + it('reports operation and performance entry times relative to the time origin', async () => { + const engine: PhasedCommandEngine = await parseAsync('build'); + const data: ITelemetryData = engine.createTelemetryData( + createOptions({ + records: new Map([ + [operationB, createRecord(OperationStatus.Aborted, 1500, undefined)], + [operationA, createRecord(OperationStatus.Failure, 1600, 1900)] + ]), + succeeded: false, + performanceEntries: [ + createMeasure('rush:daemon:queueWait', 1010, 5), + createMeasure('rush:executionManager:executeOperations', 1600, 300) + ] + }) + ); + + expect(data.result).toBe('Failed'); + expect(data.operationResults).toEqual({ + 'b (compile)': { + startTimestampMs: 500, + endTimestampMs: undefined, + nonCachedDurationMs: undefined, + wasExecutedOnThisMachine: true, + result: OperationStatus.Aborted, + dependencies: [] + }, + 'a (compile)': { + startTimestampMs: 600, + endTimestampMs: 900, + nonCachedDurationMs: undefined, + wasExecutedOnThisMachine: true, + result: OperationStatus.Failure, + dependencies: ['b (compile)'] + } + }); + expect(JSON.parse(JSON.stringify(data.performanceEntries))).toEqual([ + { name: 'rush:daemon:queueWait', entryType: 'measure', startTime: 10, duration: 5, detail: null }, + { + name: 'rush:executionManager:executeOperations', + entryType: 'measure', + startTime: 600, + duration: 300, + detail: null + } + ]); + }); + + it('adds host fields after the native fields and never falls back to process-wide entries', async () => { + const engine: PhasedCommandEngine = await parseAsync('build'); + const data: ITelemetryData = engine.createTelemetryData( + createOptions({ extraData: { daemon: true, requestIndex: 3, countAll: 99 } }) + ); + + expect(data.extraData).toMatchObject({ daemon: true, requestIndex: 3, countAll: 99, countSuccess: 1 }); + expect(data.performanceEntries).toEqual([]); + }); +}); + +describe(waitForTelemetryFlushAsync.name, () => { + it('waits for taps that settle before the timeout', async () => { + let settled: boolean = false; + const flushPromise: Promise = new Promise((resolve) => { + setTimeout(() => { + settled = true; + resolve(); + }, 20); + }); + + await expect(waitForTelemetryFlushAsync(flushPromise, 60_000)).resolves.toBe(true); + expect(settled).toBe(true); + }); + + it('treats a failed tap as settled', async () => { + await expect(waitForTelemetryFlushAsync(Promise.reject(new Error('upload failed')), 60_000)).resolves.toBe( + true + ); + }); + + it('stops waiting for taps that never settle, such as an upload over a stalled network', async () => { + const startMs: number = Date.now(); + + await expect(waitForTelemetryFlushAsync(new Promise(() => undefined), 50)).resolves.toBe(false); + expect(Date.now() - startMs).toBeGreaterThanOrEqual(45); + }); +}); + +function createMeasure(name: string, startTime: number, duration: number): PerformanceEntry { + return { + name, + entryType: 'measure', + startTime, + duration, + detail: null, + toJSON: () => ({ name, entryType: 'measure', startTime, duration, detail: null }) + } as PerformanceEntry; +} diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index da402638e6..3d084b3973 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -43,6 +43,7 @@ import { Stopwatch } from '../../utilities/Stopwatch'; import { BaseScriptAction, type IBaseScriptActionOptions } from './BaseScriptAction'; import type { IOperationGraphOptions, IOperationGraphTelemetry } from '../../logic/operations/OperationGraph'; import { OperationGraph } from '../../logic/operations/OperationGraph'; +import type { IPhasedCommandTelemetryFields } from '../../logic/operations/PhasedCommandTelemetry'; import { RushConstants } from '../../logic/RushConstants'; import { EnvironmentVariableNames } from '../../api/EnvironmentConfiguration'; import type { RushConfigurationProject } from '../../api/RushConfigurationProject'; @@ -481,6 +482,23 @@ export class PhasedScriptAction extends BaseScriptAction i return selected; } + /** The command-scoped fields of this command's phased telemetry entries. */ + public getTelemetryFields(): IPhasedCommandTelemetryFields { + const changedProjectsOnlyParameter: CommandLineFlagParameter | undefined = + this.#changedProjectsOnlyParameter; + return { + changedProjectsOnlyKey: + changedProjectsOnlyParameter?.scopedLongName ?? changedProjectsOnlyParameter?.longName, + changedProjectsOnly: !!changedProjectsOnlyParameter?.value, + initialExtraData: { + // Fields preserved across the command invocation + ...this.#selectionParameters.getTelemetry(), + ...this.getParameterStringMap() + }, + nameForLog: this.actionName + }; + } + public async createEngineAsync(): Promise { this.validateEngineCommand(); await this.initializePluginsAsync(); @@ -813,17 +831,11 @@ export class PhasedScriptAction extends BaseScriptAction i let executionTelemetryHandler: IOperationGraphTelemetry | undefined; const { telemetry: parserTelemetry } = this.parser; if (parserTelemetry) { - const changedProjectsOnlyParameter: CommandLineFlagParameter | undefined = - this.#changedProjectsOnlyParameter; + const { changedProjectsOnlyKey, initialExtraData, nameForLog } = this.getTelemetryFields(); executionTelemetryHandler = { - changedProjectsOnlyKey: - changedProjectsOnlyParameter?.scopedLongName ?? changedProjectsOnlyParameter?.longName, - initialExtraData: { - // Fields preserved across the command invocation - ...this.#selectionParameters.getTelemetry(), - ...this.getParameterStringMap() - }, - nameForLog: this.actionName, + changedProjectsOnlyKey, + initialExtraData, + nameForLog, log: (logEntry: ITelemetryData) => { parserTelemetry.log(logEntry); parserTelemetry.flush(); diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index b45248e94f..7abdab3c2b 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -179,6 +179,9 @@ export { PhasedCommandEngine, type IPhasedCommandEngine, type IPhasedCommandEngineRequestSettings, + type IPhasedCommandEngineLogTelemetryOptions, + type IPhasedCommandEngineTelemetryOptions, + type IPhasedCommandEngineTelemetryRecord, type IParsePhasedCommandOptions } from './api/PhasedCommandEngine'; export { PhasedCommandEngineConfigurationChangedError } from './api/PhasedCommandEngineConfigurationChangedError'; diff --git a/libraries/rush-lib/src/logic/Telemetry.ts b/libraries/rush-lib/src/logic/Telemetry.ts index 8c6ac3fab7..c8adc46bdc 100644 --- a/libraries/rush-lib/src/logic/Telemetry.ts +++ b/libraries/rush-lib/src/logic/Telemetry.ts @@ -163,6 +163,7 @@ export class Telemetry { #rushSession: RushSession; readonly #flushAsyncTasks: Set> = new Set(); #telemetryStartTime: number = 0; + #lastFileTimeMs: number = 0; public constructor(rushConfiguration: RushConfiguration, rushSession: RushSession) { this.#rushConfiguration = rushConfiguration; @@ -296,8 +297,18 @@ export class Telemetry { } #getFilePath(): string { - let fileName: string = `telemetry_${new Date().toISOString()}`; - fileName = fileName.replace(/[\-\:\.]/g, '_') + '.json'; - return path.join(this.#dataFolder, fileName); + // A long-lived host can flush several entries within one millisecond, so keep every file name distinct. + let timeMs: number = Math.max(Date.now(), this.#lastFileTimeMs + 1); + let fullPath: string = path.join(this.#dataFolder, getFileName(timeMs)); + while (FileSystem.exists(fullPath)) { + timeMs++; + fullPath = path.join(this.#dataFolder, getFileName(timeMs)); + } + this.#lastFileTimeMs = timeMs; + return fullPath; } } + +function getFileName(timeMs: number): string { + return `telemetry_${new Date(timeMs).toISOString()}`.replace(/[\-\:\.]/g, '_') + '.json'; +} diff --git a/libraries/rush-lib/src/logic/operations/OperationGraph.ts b/libraries/rush-lib/src/logic/operations/OperationGraph.ts index b0408511bb..c78fcf862a 100644 --- a/libraries/rush-lib/src/logic/operations/OperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/OperationGraph.ts @@ -29,7 +29,8 @@ import type { IOperationGraph, IOperationGraphIterationOptions } from './IOperat import { OperationGraphHooks } from '../../pluginFramework/OperationGraphHooks'; import { type Parallelism, coerceParallelism, getNumberOfCores } from './ParseParallelism'; import { measureAsyncFn, measureFn } from '../../utilities/performance'; -import type { ITelemetryData, ITelemetryOperationResult } from '../Telemetry'; +import type { ITelemetryData } from '../Telemetry'; +import { createPhasedTelemetryData } from './PhasedCommandTelemetry'; export interface IOperationGraphTelemetry { initialExtraData: Record; @@ -117,25 +118,6 @@ class OperationRunnerCloseError extends Error { } } -/** - * Telemetry data for a phased execution - */ -interface IPhasedExecutionTelemetry { - [key: string]: string | number | boolean; - isInitial: boolean; - isWatch: boolean; - - countAll: number; - countSuccess: number; - countSuccessWithWarnings: number; - countFailure: number; - countBlocked: number; - countFromCache: number; - countSkipped: number; - countNoOp: number; - countAborted: number; -} - const PERF_PREFIX: 'rush:executionManager' = 'rush:executionManager'; /** @@ -1115,28 +1097,6 @@ export class OperationGraph implements IOperationGraph { const telemetry: IOperationGraphTelemetry | undefined = this.#telemetry; if (telemetry) { const logEntry: ITelemetryData = measureFn(`${PERF_PREFIX}:prepareTelemetry`, () => { - const isWatch: boolean = this.#isWatch; - const jsonOperationResults: Record = {}; - - const durationInSeconds: number = (performance.now() - (iterationContext.startTime ?? 0)) / 1000; - - const extraData: IPhasedExecutionTelemetry = { - ...telemetry.initialExtraData, - isWatch, - // Fields specific to the current operation set - isInitial, - - countAll: 0, - countSuccess: 0, - countSuccessWithWarnings: 0, - countFailure: 0, - countBlocked: 0, - countFromCache: 0, - countSkipped: 0, - countNoOp: 0, - countAborted: 0 - }; - let changedProjectsOnly: boolean = false; for (const operation of executionRecords.keys()) { if (operation.enabled === 'ignore-dependency-changes') { @@ -1145,90 +1105,17 @@ export class OperationGraph implements IOperationGraph { } } - if (telemetry.changedProjectsOnlyKey) { - // Overwrite this value since we allow changing it at runtime. - extraData[telemetry.changedProjectsOnlyKey] = changedProjectsOnly; - } - - const nonSilentDependenciesByOperation: Map> = new Map(); - function getNonSilentDependencies(operation: Operation): ReadonlySet { - let realDependencies: Set | undefined = nonSilentDependenciesByOperation.get(operation); - if (!realDependencies) { - realDependencies = new Set(); - nonSilentDependenciesByOperation.set(operation, realDependencies); - for (const dependency of operation.dependencies) { - const dependencyRecord: OperationExecutionRecord | undefined = executionRecords.get(dependency); - if (dependencyRecord?.silent) { - for (const deepDependency of getNonSilentDependencies(dependency)) { - realDependencies.add(deepDependency); - } - } else { - realDependencies.add(dependency.name!); - } - } - } - return realDependencies; - } - - for (const [operation, operationResult] of executionRecords) { - if (operationResult.silent) { - // Architectural operation. Ignore. - continue; - } - - const { _operationMetadataManager: operationMetadataManager } = operationResult; - - const { startTime, endTime } = operationResult.stopwatch; - jsonOperationResults[operation.name!] = { - startTimestampMs: startTime, - endTimestampMs: endTime, - nonCachedDurationMs: operationResult.nonCachedDurationMs, - wasExecutedOnThisMachine: operationMetadataManager?.wasCobuilt !== true, - result: operationResult.status, - dependencies: Array.from(getNonSilentDependencies(operation)).sort() - }; - - extraData.countAll++; - switch (operationResult.status) { - case OperationStatus.Success: - extraData.countSuccess++; - break; - case OperationStatus.SuccessWithWarning: - extraData.countSuccessWithWarnings++; - break; - case OperationStatus.Failure: - extraData.countFailure++; - break; - case OperationStatus.Blocked: - extraData.countBlocked++; - break; - case OperationStatus.FromCache: - extraData.countFromCache++; - break; - case OperationStatus.Skipped: - extraData.countSkipped++; - break; - case OperationStatus.NoOp: - extraData.countNoOp++; - break; - case OperationStatus.Aborted: - extraData.countAborted++; - break; - default: - // Do nothing. - break; - } - } - - const innerLogEntry: ITelemetryData = { - name: telemetry.nameForLog, - durationInSeconds, - result: status === OperationStatus.Success ? 'Succeeded' : 'Failed', - extraData, - operationResults: jsonOperationResults - }; - - return innerLogEntry; + return createPhasedTelemetryData({ + nameForLog: telemetry.nameForLog, + initialExtraData: telemetry.initialExtraData, + changedProjectsOnlyKey: telemetry.changedProjectsOnlyKey, + changedProjectsOnly, + isWatch: this.#isWatch, + isInitial, + durationInSeconds: (performance.now() - (iterationContext.startTime ?? 0)) / 1000, + succeeded: status === OperationStatus.Success, + records: executionRecords + }); }); measureFn(`${PERF_PREFIX}:beforeLog`, () => this.hooks.beforeLog.call(logEntry)); diff --git a/libraries/rush-lib/src/logic/operations/PhasedCommandTelemetry.ts b/libraries/rush-lib/src/logic/operations/PhasedCommandTelemetry.ts new file mode 100644 index 0000000000..d3f0e5d24d --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/PhasedCommandTelemetry.ts @@ -0,0 +1,168 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { ITelemetryData, ITelemetryOperationResult } from '../Telemetry'; +import type { IStopwatchResult } from '../../utilities/Stopwatch'; +import type { Operation } from './Operation'; +import { OperationStatus } from './OperationStatus'; + +/** + * Telemetry data for a phased execution + */ +export interface IPhasedExecutionTelemetry { + [key: string]: string | number | boolean; + isInitial: boolean; + isWatch: boolean; + + countAll: number; + countSuccess: number; + countSuccessWithWarnings: number; + countFailure: number; + countBlocked: number; + countFromCache: number; + countSkipped: number; + countNoOp: number; + countAborted: number; +} + +/** + * The fields of an operation's execution record that phased command telemetry reports. + */ +export interface IPhasedTelemetryOperationRecord { + readonly status: OperationStatus; + readonly silent: boolean; + readonly stopwatch: Pick; + readonly nonCachedDurationMs: number | undefined; + readonly _operationMetadataManager?: { readonly wasCobuilt: boolean } | undefined; +} + +/** + * The command-scoped fields of a phased command's telemetry entries. + */ +export interface IPhasedCommandTelemetryFields { + readonly nameForLog: string; + readonly initialExtraData: Record; + readonly changedProjectsOnlyKey: string | undefined; + readonly changedProjectsOnly: boolean; +} + +export interface ICreatePhasedTelemetryDataOptions extends IPhasedCommandTelemetryFields { + readonly isWatch: boolean; + readonly isInitial: boolean; + readonly durationInSeconds: number; + readonly succeeded: boolean; + readonly records: ReadonlyMap; + /** + * A `performance.now()` value subtracted from every operation timestamp. Long-lived hosts use it to report + * timestamps relative to the request, as a native process reports them relative to its own start. + */ + readonly timeOriginMs?: number; +} + +/** + * Builds the telemetry entry that a phased command logs for one set of operation results. + */ +export function createPhasedTelemetryData(options: ICreatePhasedTelemetryDataOptions): ITelemetryData { + const { records, timeOriginMs = 0 } = options; + const jsonOperationResults: Record = {}; + + const extraData: IPhasedExecutionTelemetry = { + ...options.initialExtraData, + isWatch: options.isWatch, + // Fields specific to the current operation set + isInitial: options.isInitial, + + countAll: 0, + countSuccess: 0, + countSuccessWithWarnings: 0, + countFailure: 0, + countBlocked: 0, + countFromCache: 0, + countSkipped: 0, + countNoOp: 0, + countAborted: 0 + }; + + if (options.changedProjectsOnlyKey) { + // Overwrite this value since we allow changing it at runtime. + extraData[options.changedProjectsOnlyKey] = options.changedProjectsOnly; + } + + const nonSilentDependenciesByOperation: Map> = new Map(); + function getNonSilentDependencies(operation: Operation): ReadonlySet { + let realDependencies: Set | undefined = nonSilentDependenciesByOperation.get(operation); + if (!realDependencies) { + realDependencies = new Set(); + nonSilentDependenciesByOperation.set(operation, realDependencies); + for (const dependency of operation.dependencies) { + const dependencyRecord: IPhasedTelemetryOperationRecord | undefined = records.get(dependency); + if (dependencyRecord?.silent) { + for (const deepDependency of getNonSilentDependencies(dependency)) { + realDependencies.add(deepDependency); + } + } else { + realDependencies.add(dependency.name!); + } + } + } + return realDependencies; + } + + for (const [operation, operationResult] of records) { + if (operationResult.silent) { + // Architectural operation. Ignore. + continue; + } + + const { _operationMetadataManager: operationMetadataManager } = operationResult; + + const { startTime, endTime } = operationResult.stopwatch; + jsonOperationResults[operation.name!] = { + startTimestampMs: startTime === undefined ? undefined : startTime - timeOriginMs, + endTimestampMs: endTime === undefined ? undefined : endTime - timeOriginMs, + nonCachedDurationMs: operationResult.nonCachedDurationMs, + wasExecutedOnThisMachine: operationMetadataManager?.wasCobuilt !== true, + result: operationResult.status, + dependencies: Array.from(getNonSilentDependencies(operation)).sort() + }; + + extraData.countAll++; + switch (operationResult.status) { + case OperationStatus.Success: + extraData.countSuccess++; + break; + case OperationStatus.SuccessWithWarning: + extraData.countSuccessWithWarnings++; + break; + case OperationStatus.Failure: + extraData.countFailure++; + break; + case OperationStatus.Blocked: + extraData.countBlocked++; + break; + case OperationStatus.FromCache: + extraData.countFromCache++; + break; + case OperationStatus.Skipped: + extraData.countSkipped++; + break; + case OperationStatus.NoOp: + extraData.countNoOp++; + break; + case OperationStatus.Aborted: + extraData.countAborted++; + break; + default: + // Do nothing. + break; + } + } + + return { + name: options.nameForLog, + durationInSeconds: options.durationInSeconds, + result: options.succeeded ? 'Succeeded' : 'Failed', + extraData, + operationResults: jsonOperationResults + }; +} diff --git a/libraries/rush-lib/src/logic/test/Telemetry.test.ts b/libraries/rush-lib/src/logic/test/Telemetry.test.ts index 977d3b9d3c..92d53f927b 100644 --- a/libraries/rush-lib/src/logic/test/Telemetry.test.ts +++ b/libraries/rush-lib/src/logic/test/Telemetry.test.ts @@ -1,7 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { JsonFile } from '@rushstack/node-core-library'; +import * as path from 'node:path'; + +import { FileSystem, JsonFile } from '@rushstack/node-core-library'; import type { IReporterEmitEventInput, IReporterEventSink } from '@rushstack/rush-reporter'; import { ConsoleTerminalProvider } from '@rushstack/terminal'; @@ -131,6 +133,49 @@ describe(Telemetry.name, () => { expect(telemetry.store).toEqual([]); }); + it('gives entries flushed within one millisecond distinct file names', () => { + const filename: string = `${__dirname}/telemetry/telemetryEnabled.json`; + const rushConfig: RushConfiguration = RushConfiguration.loadFromConfigurationFile(filename); + const rushSession: RushSession = new RushSession({ + terminalProvider: new ConsoleTerminalProvider(), + getIsDebugMode: () => false + }); + const telemetry: Telemetry = new Telemetry(rushConfig, rushSession); + const nowMs: number = Date.UTC(2026, 8, 28, 13, 0, 0, 5); + const dateNowSpy: jest.SpyInstance = jest.spyOn(Date, 'now').mockReturnValue(nowMs); + const existingPaths: Set = new Set(); + const existsSpy: jest.SpyInstance = jest + .spyOn(FileSystem, 'exists') + .mockImplementation((filePath: string) => existingPaths.has(path.basename(filePath))); + existingPaths.add('telemetry_2026_09_28T13_00_00_007Z.json'); + const logData: ITelemetryData = { + name: 'testData1', + durationInSeconds: 1, + result: 'Succeeded', + timestampMs: nowMs, + platform: process.platform, + rushVersion: Rush.version, + machineInfo: {} as ITelemetryMachineInfo, + performanceEntries: [] + }; + + try { + for (let i: number = 0; i < 3; i++) { + telemetry.log(logData); + telemetry.flush(); + } + } finally { + dateNowSpy.mockRestore(); + existsSpy.mockRestore(); + } + + expect(mockedJsonFileSave.mock.calls.map((call) => path.basename(call[1]))).toEqual([ + 'telemetry_2026_09_28T13_00_00_005Z.json', + 'telemetry_2026_09_28T13_00_00_006Z.json', + 'telemetry_2026_09_28T13_00_00_008Z.json' + ]); + }); + it('populates default fields', () => { const filename: string = `${__dirname}/telemetry/telemetryEnabled.json`; const rushConfig: RushConfiguration = RushConfiguration.loadFromConfigurationFile(filename); From c97c702aa8492652d7f7615361a20cf043631237 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:51:44 +0000 Subject: [PATCH 051/265] [rush-daemon] A stuck request no longer keeps a stopped daemon alive Swarm integration step 36; original commit 372742afe7 (merge of swarm/r01-t147-int at a40a901822). Scope: tasks 115 and 147. Brings r01's task 115 (`daemon stop` and the first SIGTERM never ended a daemon whose request was stuck in an await that ignores cancellation) and task 147 (after a clean stop, a ref'd timer or handle kept the process alive after it released its socket and pid.json; t05 board 2199, o04 board 2269), re-tipped onto integration (board 2779, board 2793). The daemon now shuts down within a 10 s deadline, returns a typed result to the running request and releases its socket, lockfile and repository lock. CONFIRMED: 115 by ch05 board 2247 and o04 board 2266; 147 by t05 board 2608 and o04 board 2632. s16 batch A, item 1 of 5. ch01 on the batch (board 2867): build rc 0 with 0 warnings; rush-daemon 628/0, rush-lib 1252/0, rush-cli-client 398/0 on items 1-4; 45 of 48 mutants killed; in the e2e the daemon is gone 10.46 s after `daemon stop`, where s15's tree lingers 39.15 s. Gate: ch01 GATE OK board 2867 (tree 24fc1e6446) Commits folded into this step (2): - fe430fe236 rushd: a shutdown stuck on an await that ignores cancellation no longer hangs the daemon process (task 115) - 444107370d rushd: a daemon process that something else keeps running exits 2 s after the daemon stops (task 147) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...01-shutdown-deadline_2026-09-28-21-50.json | 11 + .../r01-exit-after-stop_2026-09-28-23-45.json | 11 + ...01-shutdown-deadline_2026-09-28-21-50.json | 11 + .../reviews/api/rush-daemon-transport.api.md | 1 + common/reviews/api/rush-daemon.api.md | 23 ++ .../src/DaemonExitRelease.ts | 28 +++ .../src/DaemonListener.ts | 6 + .../src/DaemonListenerLifetime.ts | 29 ++- .../src/test/ExitReleaseGroup.test.ts | 43 ++++ .../src/test/ListenerExitRelease.test.ts | 90 ++++++++ .../rush-daemon/src/DaemonControlSession.ts | 75 ++++++- .../rush-daemon/src/DaemonShutdownDeadline.ts | 76 +++++++ .../src/DaemonShutdownDeadlineError.ts | 90 ++++++++ .../rush-daemon/src/DaemonShutdownSignals.ts | 58 +++++ libraries/rush-daemon/src/RushDaemonHost.ts | 81 ++++++- libraries/rush-daemon/src/index.ts | 5 + libraries/rush-daemon/src/serveRushDaemon.ts | 152 +++++++++++-- .../src/test/DaemonProcessExit.test.ts | 178 +++++++++++++++ .../src/test/DaemonShutdownDeadline.test.ts | 204 ++++++++++++++++++ .../src/test/DaemonShutdownProcess.test.ts | 186 ++++++++++++++++ .../src/test/DaemonShutdownSignals.test.ts | 64 ++++++ .../src/test/TemporaryRepoWorkspaceSession.ts | 31 +++ .../src/test/fixtures/LingeringDaemon.ts | 70 ++++++ .../src/test/fixtures/StuckShutdownDaemon.ts | 74 +++++++ 24 files changed, 1565 insertions(+), 32 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon-transport/r01-shutdown-deadline_2026-09-28-21-50.json create mode 100644 common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json create mode 100644 common/changes/@rushstack/rush-daemon/r01-shutdown-deadline_2026-09-28-21-50.json create mode 100644 libraries/rush-daemon-transport/src/DaemonExitRelease.ts create mode 100644 libraries/rush-daemon-transport/src/test/ExitReleaseGroup.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/ListenerExitRelease.test.ts create mode 100644 libraries/rush-daemon/src/DaemonShutdownDeadline.ts create mode 100644 libraries/rush-daemon/src/DaemonShutdownDeadlineError.ts create mode 100644 libraries/rush-daemon/src/DaemonShutdownSignals.ts create mode 100644 libraries/rush-daemon/src/test/DaemonProcessExit.test.ts create mode 100644 libraries/rush-daemon/src/test/DaemonShutdownDeadline.test.ts create mode 100644 libraries/rush-daemon/src/test/DaemonShutdownProcess.test.ts create mode 100644 libraries/rush-daemon/src/test/DaemonShutdownSignals.test.ts create mode 100644 libraries/rush-daemon/src/test/TemporaryRepoWorkspaceSession.ts create mode 100644 libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts create mode 100644 libraries/rush-daemon/src/test/fixtures/StuckShutdownDaemon.ts diff --git a/common/changes/@rushstack/rush-daemon-transport/r01-shutdown-deadline_2026-09-28-21-50.json b/common/changes/@rushstack/rush-daemon-transport/r01-shutdown-deadline_2026-09-28-21-50.json new file mode 100644 index 0000000000..10c2417fcd --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-transport/r01-shutdown-deadline_2026-09-28-21-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-transport", + "comment": "Add `DaemonFrameListener.releaseForExit()`, which removes the listener's own socket and lockfile for a daemon process that exits before its shutdown finished, unless the daemon still has child processes that a successor must reap.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-transport", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json b/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json new file mode 100644 index 0000000000..9596f945ac --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A daemon that owns its process now exits 2 seconds after it stops if something else, such as a timer that a plugin left behind, keeps the process running after the daemon released its socket and lockfile. It reports the active resources that Node.js lists and keeps `process.exitCode`. Reports that come right before the daemon exits are now written synchronously when no `onError` callback is given.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r01-shutdown-deadline_2026-09-28-21-50.json b/common/changes/@rushstack/rush-daemon/r01-shutdown-deadline_2026-09-28-21-50.json new file mode 100644 index 0000000000..cdd4ec6047 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r01-shutdown-deadline_2026-09-28-21-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A daemon that owns its process no longer hangs in `daemon stop` or on SIGTERM when a request ignores cancellation, for example while it waits for a lock that another process holds. Each request that is still running after the 5 second drain gets a typed result that names the shutdown's reason. After 10 seconds (`shutdownDeadlineMs`), or on a second SIGINT or SIGTERM, the daemon reports a `DaemonShutdownDeadlineError` naming the stage and the unfinished requests, releases its socket, lockfile and repository lock when no child process is left to reap, and exits with code 1.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-daemon-transport.api.md b/common/reviews/api/rush-daemon-transport.api.md index fa87bdd96d..ad9b1e2f86 100644 --- a/common/reviews/api/rush-daemon-transport.api.md +++ b/common/reviews/api/rush-daemon-transport.api.md @@ -37,6 +37,7 @@ export class DaemonFrameConnection { export class DaemonFrameListener { closeAsync(): Promise; static listenAsync(paths: IDaemonPaths, options: IDaemonListenerOptions): Promise; + releaseForExit(): boolean; stopAcceptingAsync(): Promise; } diff --git a/common/reviews/api/rush-daemon.api.md b/common/reviews/api/rush-daemon.api.md index e9fa05e2cf..f9003bbbdf 100644 --- a/common/reviews/api/rush-daemon.api.md +++ b/common/reviews/api/rush-daemon.api.md @@ -77,6 +77,15 @@ export class DaemonRequiresInProcessError extends Error { readonly policy: IDaemonTerminalPolicyResult; } +// @beta +export class DaemonShutdownDeadlineError extends Error { + constructor(options: IDaemonShutdownDeadlineErrorOptions); + readonly elapsedMs: number; + readonly forcedBy: string | undefined; + readonly stage: DaemonShutdownStage; + readonly unfinishedRequests: ReadonlyArray; +} + // @beta export class DaemonShutdownError extends Error { constructor(options: IDaemonShutdownErrorOptions); @@ -89,6 +98,9 @@ export class DaemonShutdownError extends Error { // @beta export type DaemonShutdownInitiator = 'controlClient' | 'signal' | 'idleTimeout' | 'restart' | 'host'; +// @beta +export type DaemonShutdownStage = 'requests' | 'workspaceMaintenance' | 'requestDispatcher' | 'workspaceSession' | 'listener'; + // @beta export type DispatchWorkspaceRequestAsync = (options: IDispatchWorkspaceRequestOptions) => Promise; @@ -207,6 +219,14 @@ export interface IDaemonRequestResolver { readonly workspaceLifecycle?: IWorkspaceResolverLifecycle; } +// @beta +export interface IDaemonShutdownDeadlineErrorOptions { + readonly elapsedMs: number; + readonly forcedBy?: string; + readonly stage: DaemonShutdownStage; + readonly unfinishedRequests: ReadonlyArray; +} + // @beta export interface IDaemonShutdownErrorOptions { // (undocumented) @@ -522,6 +542,7 @@ export interface IRushDaemonHostOptions { readonly repoRoot: string; readonly requestResolver?: IDaemonRequestResolver; readonly rushVersion: string; + readonly shutdownDeadlineMs?: number; readonly startupOptions?: Readonly>; } @@ -802,9 +823,11 @@ export type ResolvedDaemonRequest = IResolvedDaemonPhasedRequest | IResolvedDaem export class RushDaemonHost { closeAsync(reason?: DaemonShutdownError): Promise; readonly closed: Promise; + expireShutdownDeadline(forcedBy: string): void; getWorkspaceSessionAsync(): Promise; // (undocumented) readonly paths: IDaemonPaths; + releaseForExit(): boolean; readonly restartCompleted: Promise; static startAsync(options: IRushDaemonHostOptions): Promise; get workspaceGeneration(): number; diff --git a/libraries/rush-daemon-transport/src/DaemonExitRelease.ts b/libraries/rush-daemon-transport/src/DaemonExitRelease.ts new file mode 100644 index 0000000000..5250fea764 --- /dev/null +++ b/libraries/rush-daemon-transport/src/DaemonExitRelease.ts @@ -0,0 +1,28 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { getOperationGroupsFolder, readOperationGroupRecords } from './DaemonOperationGroups'; +import type { IDaemonPaths } from './DaemonPaths'; +import { listLiveGroupMembers } from './DaemonProcessStat'; +import type { IProcessStat } from './DaemonProcessStat'; + +const NO_RECORDS: number = 0; + +function isOtherProcess(stat: IProcessStat): boolean { + return stat.pid !== process.pid; +} + +/** + * Whether this daemon process has children that a successor must reap if it exits without joining them: a + * recorded operation group that still runs, or another live member of the daemon's own process group. + * + * @remarks + * Linux only, like the records it reads. Elsewhere it finds none. + */ +export function hasProcessesToReap(paths: IDaemonPaths): boolean { + const folder: string = getOperationGroupsFolder(paths.lockfilePath, process.pid); + return ( + readOperationGroupRecords(folder).length > NO_RECORDS || + listLiveGroupMembers(process.pid).some(isOtherProcess) + ); +} diff --git a/libraries/rush-daemon-transport/src/DaemonListener.ts b/libraries/rush-daemon-transport/src/DaemonListener.ts index 6a8e18b76c..f131fafae8 100644 --- a/libraries/rush-daemon-transport/src/DaemonListener.ts +++ b/libraries/rush-daemon-transport/src/DaemonListener.ts @@ -73,6 +73,12 @@ export class DaemonFrameListener { public stopAcceptingAsync(): Promise { return this.#lifetime.stopAcceptingAsync(); } + /** For a daemon process that exits before its shutdown finished: releases the socket/pipe and lockfile, + * unless the daemon still has children that a successor must reap. Then both stay, as after a crash. + * @returns Whether they were released. */ + public releaseForExit(): boolean { + return this.#lifetime.releaseForExit(); + } } function writeListenerLockfile(paths: IDaemonPaths, options: IDaemonListenerOptions): IDaemonFileIdentity { diff --git a/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts b/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts index a7a81b2493..f3604f6f04 100644 --- a/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts +++ b/libraries/rush-daemon-transport/src/DaemonListenerLifetime.ts @@ -3,6 +3,7 @@ import type * as net from 'node:net'; +import { hasProcessesToReap } from './DaemonExitRelease'; import { removeOwnFile } from './DaemonFileIdentity'; import type { IDaemonFileIdentity } from './DaemonFileIdentity'; import type { StopOperationGroupRecording } from './DaemonOperationGroupRecorder'; @@ -25,6 +26,9 @@ export class DaemonListenerLifetime { readonly #stopRecording: StopOperationGroupRecording; #closePromise: Promise | undefined; #stopPromise: Promise | undefined; + // `removeOwnFile` releases the pinned identity, so each file is released once. + #socketReleased: boolean = false; + #lockfileReleased: boolean = false; public constructor( server: net.Server, @@ -48,15 +52,38 @@ export class DaemonListenerLifetime { return this.#closePromise; } + /** + * Releases the socket and the lockfile for a process that exits without closing the listener, unless it has + * children that a successor must reap first. Then both stay, as after a crash. Returns whether it released them. + */ + public releaseForExit(): boolean { + if (hasProcessesToReap(this.#paths)) return false; + this.#releaseSocket(); + this.#releaseLockfile(); + return true; + } + #stopOnceAsync(): Promise { // Unlink before closing, as libuv does for the path it bound, but only this listener's own socket: the // name may belong to a successor by now. - removeOwnFile(this.#paths.socketPath, this.#files.socket); + this.#releaseSocket(); return new Promise((resolve: () => void) => this.#server.close(() => resolve())); } async #closeOnceAsync(): Promise { await this.stopAcceptingAsync(); + this.#releaseLockfile(); + } + + #releaseSocket(): void { + if (this.#socketReleased) return; + this.#socketReleased = true; + removeOwnFile(this.#paths.socketPath, this.#files.socket); + } + + #releaseLockfile(): void { + if (this.#lockfileReleased) return; + this.#lockfileReleased = true; this.#stopRecording(); removeOwnFile(this.#paths.lockfilePath, this.#files.lockfile); } diff --git a/libraries/rush-daemon-transport/src/test/ExitReleaseGroup.test.ts b/libraries/rush-daemon-transport/src/test/ExitReleaseGroup.test.ts new file mode 100644 index 0000000000..a7061d1861 --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/ExitReleaseGroup.test.ts @@ -0,0 +1,43 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn } from 'node:child_process'; +import type { ChildProcess, SpawnOptions } from 'node:child_process'; +import { once } from 'node:events'; +import * as path from 'node:path'; +import type { Readable } from 'node:stream'; + +import { identifyStarted, killStillRunning } from './ProcessWaitFixture'; +import type { IStartedProcess } from './ProcessWaitFixture'; +import { createTestDaemonPaths } from './TestDaemonFixture'; + +const linuxIt: jest.It = process.platform === 'linux' ? it : it.skip; +// A detached process leads its own group, as a daemon that rush-client started does. It checks for children to +// reap before, while and after a child that it spawned into its own group runs. +const GROUP_LEADER_OPTIONS: SpawnOptions = { detached: true, stdio: ['ignore', 'pipe', 'ignore'] }; +const GROUP_LEADER_SCRIPT: string = ` +const { hasProcessesToReap } = require(process.argv[1]); +const paths = { lockfilePath: process.argv[2] }; +const before = hasProcessesToReap(paths); +const child = require('node:child_process').spawn('sleep', ['30'], { stdio: 'ignore' }); +child.once('spawn', () => { + const during = hasProcessesToReap(paths); + child.once('exit', () => process.stdout.write(JSON.stringify([before, during, hasProcessesToReap(paths)]))); + child.kill('SIGKILL'); +});`; + +let started: IStartedProcess[] = []; +afterEach(() => { + killStillRunning(started); + started = []; +}); + +linuxIt('finds a child that runs in the daemon process group', async () => { + const modulePath: string = path.join(__dirname, '..', 'DaemonExitRelease.js'); + const args: string[] = ['-e', GROUP_LEADER_SCRIPT, modulePath, createTestDaemonPaths().lockfilePath]; + const leader: ChildProcess = spawn(process.execPath, args, GROUP_LEADER_OPTIONS); + await once(leader, 'spawn'); + started = identifyStarted([Number(leader.pid)]); + const [chunk] = (await once(leader.stdout as Readable, 'data')) as [Buffer]; + expect(JSON.parse(chunk.toString())).toEqual([false, true, false]); +}); diff --git a/libraries/rush-daemon-transport/src/test/ListenerExitRelease.test.ts b/libraries/rush-daemon-transport/src/test/ListenerExitRelease.test.ts new file mode 100644 index 0000000000..7a2a5f3f7c --- /dev/null +++ b/libraries/rush-daemon-transport/src/test/ListenerExitRelease.test.ts @@ -0,0 +1,90 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn } from 'node:child_process'; +import type { ChildProcess, SpawnOptions } from 'node:child_process'; +import { once } from 'node:events'; +import * as fs from 'node:fs'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; + +import { DaemonFrameListener } from '../DaemonListener'; +import { readDaemonLockfile } from '../DaemonLockfile'; +import { getOperationGroupsFolder, readOperationGroupRecords } from '../DaemonOperationGroups'; +import type { IDaemonPaths } from '../DaemonPaths'; + +import { identifyStarted, killStillRunning, waitUntilAsync } from './ProcessWaitFixture'; +import type { IStartedProcess } from './ProcessWaitFixture'; +import { createTestDaemonPaths } from './TestDaemonFixture'; + +const posixIt: jest.It = process.platform === 'win32' ? it.skip : it; +const linuxIt: jest.It = process.platform === 'linux' ? it : it.skip; +const SLEEP_ARGS: string[] = ['-e', 'setTimeout(() => {}, 30000)']; +// SubprocessTerminator.RECOMMENDED_OPTIONS on POSIX, which gives the operation a group of its own. +const DETACHED: SpawnOptions = { detached: true, stdio: 'ignore' }; +const NO_RECORDS: number = 0; + +let started: IStartedProcess[] = []; +afterEach(() => { + killStillRunning(started); + started = []; +}); + +function listenAsync(paths: IDaemonPaths): Promise { + return DaemonFrameListener.listenAsync(paths, { + onConnection: () => undefined, + protocolVersion: DAEMON_PROTOCOL_VERSION + }); +} + +function countRecords(paths: IDaemonPaths): number { + return readOperationGroupRecords(getOperationGroupsFolder(paths.lockfilePath, process.pid)).length; +} + +it('releases the lockfile for exit when the daemon has no children', async () => { + const paths: IDaemonPaths = createTestDaemonPaths(); + const listener: DaemonFrameListener = await listenAsync(paths); + try { + expect(listener.releaseForExit()).toBe(true); + expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); + } finally { + await listener.closeAsync(); + } +}); + +posixIt('releases the socket for exit, and a later close leaves a successor alone', async () => { + const paths: IDaemonPaths = createTestDaemonPaths(); + const released: DaemonFrameListener = await listenAsync(paths); + expect(released.releaseForExit()).toBe(true); + expect(fs.existsSync(paths.socketPath)).toBe(false); + const successor: DaemonFrameListener = await listenAsync(paths); + try { + await released.closeAsync(); + expect(readDaemonLockfile(paths.lockfilePath)?.pid).toBe(process.pid); + expect(fs.existsSync(paths.socketPath)).toBe(true); + } finally { + await successor.closeAsync(); + } + expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); + expect(fs.existsSync(paths.socketPath)).toBe(false); +}); + +linuxIt('keeps the socket and the lockfile for a successor while a recorded operation runs', async () => { + const paths: IDaemonPaths = createTestDaemonPaths(); + const listener: DaemonFrameListener = await listenAsync(paths); + try { + const operation: ChildProcess = spawn(process.execPath, SLEEP_ARGS, DETACHED); + await once(operation, 'spawn'); + started = identifyStarted([Number(operation.pid)]); + expect(await waitUntilAsync(() => countRecords(paths) > NO_RECORDS)).toBe(true); + expect(listener.releaseForExit()).toBe(false); + expect(readDaemonLockfile(paths.lockfilePath)?.pid).toBe(process.pid); + expect(fs.existsSync(paths.socketPath)).toBe(true); + operation.kill('SIGKILL'); + expect(await waitUntilAsync(() => countRecords(paths) === NO_RECORDS)).toBe(true); + expect(listener.releaseForExit()).toBe(true); + expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); + } finally { + await listener.closeAsync(); + } +}); diff --git a/libraries/rush-daemon/src/DaemonControlSession.ts b/libraries/rush-daemon/src/DaemonControlSession.ts index d38cb8ffe3..a6995237c1 100644 --- a/libraries/rush-daemon/src/DaemonControlSession.ts +++ b/libraries/rush-daemon/src/DaemonControlSession.ts @@ -28,12 +28,13 @@ import type { } from '@rushstack/rush-daemon-protocol'; import type { DaemonFrameConnection } from '@rushstack/rush-daemon-transport'; +import { createGlobalCommandResult } from './CommandResultPolicy'; import { DaemonInteractiveConnection } from './DaemonInteractiveConnection'; import type { IDaemonInteractiveConnection } from './DaemonInteractiveConnection'; import { MAX_REQUESTS_PER_CONNECTION } from './DaemonConnectionLimits'; import { DaemonRequestDispatchError } from './DaemonRequestDispatcher'; import type { DaemonRequestDispatcher } from './DaemonRequestDispatcher'; -import type { DaemonShutdownError } from './DaemonShutdownError'; +import { DaemonShutdownError } from './DaemonShutdownError'; import { DaemonWireRequestClient } from './DaemonWireRequestClient'; import { InteractiveInputRoutingError, @@ -64,6 +65,9 @@ interface IRequestState { readonly abortController: AbortController; readonly client: DaemonWireRequestClient; completion: Promise; + /** The quoted command line, for reports about requests that did not finish. */ + readonly description: string; + readonly startedAtMs: number; } interface IClassifiedRejection { @@ -72,6 +76,11 @@ interface IClassifiedRejection { } const CLOSE_DRAIN_TIMEOUT_MS: number = 5000; +// How long the typed results for requests that did not stop get to reach their clients before the connection is +// aborted. +const SHUTDOWN_RESULT_SEND_TIMEOUT_MS: number = 1000; +const MAX_REQUEST_DESCRIPTION_LENGTH: number = 100; +const MS_PER_SECOND: number = 1000; export class DaemonControlSession { readonly #connection: DaemonFrameConnection; @@ -127,6 +136,15 @@ export class DaemonControlSession { return this.#requestById.size; } + /** Describes each request that has not finished, such as `"build -t a" (running for 12.3 s)`. */ + public describeActiveRequests(): string[] { + const nowMs: number = Date.now(); + return Array.from(this.#requestById.values(), (state: IRequestState) => { + const runningSeconds: string = ((nowMs - state.startedAtMs) / MS_PER_SECOND).toFixed(1); + return `${state.description} (running for ${runningSeconds} s)`; + }); + } + async #handleFrameSafelyAsync(frame: IDaemonFrame): Promise { try { await this.#onFrameAsync(frame); @@ -316,7 +334,13 @@ export class DaemonControlSession { sessionId, supportsRequestAdmission: this.#peerSupportsRequestAdmission }); - const state: IRequestState = { abortController, client, completion: Promise.resolve() }; + const state: IRequestState = { + abortController, + client, + completion: Promise.resolve(), + description: describeRequest(envelope), + startedAtMs: Date.now() + }; this.#requestById.set(requestId, state); const releaseActivity: (() => void) | undefined = this.#options.onRequestStarted?.(); state.completion = Promise.resolve() @@ -491,7 +515,7 @@ export class DaemonControlSession { CLOSE_DRAIN_TIMEOUT_MS )) ) { - this.#connection.abort(closeReason); + await this.#abortConnectionAsync(closeReason); } await pending; } @@ -501,13 +525,34 @@ export class DaemonControlSession { this.#sendQueue ]).then(() => undefined); if (!(await settlesWithinAsync(drainPromise, CLOSE_DRAIN_TIMEOUT_MS))) { - this.#connection.abort(closeReason); + await this.#abortConnectionAsync(closeReason); } await drainPromise; if (!this.#connectionClosed) await this.#connection.closeAsync(); await this.#closedPromise; } + /** + * Aborts a connection whose requests did not finish in time. When the daemon is shutting down, each request that + * has no terminal outcome yet first gets a typed result that carries the shutdown's reason, so that its client + * reports that instead of a lost connection. The request's own late result is then refused. + */ + async #abortConnectionAsync(reason: Error): Promise { + if (reason instanceof DaemonShutdownError && !this.#connectionClosed) { + const writes: Promise[] = []; + for (const [requestId, state] of this.#requestById) { + if (!state.client.terminalOutcomeSent) { + writes.push(writeShutdownResultAsync(requestId, state, reason)); + } + } + await settlesWithinAsync( + Promise.allSettled(writes).then(() => undefined), + SHUTDOWN_RESULT_SEND_TIMEOUT_MS + ); + } + this.#connection.abort(reason); + } + async #handleConnectionClosedAsync(error: Error | undefined): Promise { if (this.#connectionClosed) return; this.#connectionClosed = true; @@ -534,6 +579,28 @@ function createDeferred(): { promise: Promise; resolve: () => void } { return { promise, resolve: resolvePromise }; } +function describeRequest(envelope: IDaemonRequestEnvelope): string { + const command: string = envelope.argv.length > 0 ? envelope.argv.join(' ') : envelope.commandName; + return command.length > MAX_REQUEST_DESCRIPTION_LENGTH + ? `"${command.slice(0, MAX_REQUEST_DESCRIPTION_LENGTH - 1)}…"` + : `"${command}"`; +} + +function writeShutdownResultAsync( + requestId: string, + state: IRequestState, + reason: DaemonShutdownError +): Promise { + try { + // The result that a router writes for a request that the shutdown aborted. + return state.client.writeResultAsync( + createGlobalCommandResult({ aborted: true, error: reason, exitCode: undefined, requestId }) + ); + } catch (error) { + return Promise.reject(error); + } +} + function normalizeProtocolError(error: unknown): DaemonProtocolError { if (error instanceof DaemonProtocolError) return error; return new DaemonProtocolError('malformedControlMessage', normalizeError(error).message, { diff --git a/libraries/rush-daemon/src/DaemonShutdownDeadline.ts b/libraries/rush-daemon/src/DaemonShutdownDeadline.ts new file mode 100644 index 0000000000..ad36f71690 --- /dev/null +++ b/libraries/rush-daemon/src/DaemonShutdownDeadline.ts @@ -0,0 +1,76 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { DaemonShutdownDeadlineError, type DaemonShutdownStage } from './DaemonShutdownDeadlineError'; + +/** Where a shutdown is when it is cut short. */ +export interface IDaemonShutdownProgress { + readonly stage: DaemonShutdownStage; + readonly unfinishedRequests: ReadonlyArray; +} + +/** Options for {@link DaemonShutdownDeadline}. */ +export interface IDaemonShutdownDeadlineOptions { + /** How long a shutdown may take. Without it, only {@link DaemonShutdownDeadline.expire} cuts one short. */ + readonly timeoutMs: number | undefined; + readonly getProgress: () => IDaemonShutdownProgress; + /** Reports a failure of cleanup that finished after the shutdown was cut short. */ + readonly onLateFailure: (error: Error) => void; +} + +/** + * Races a shutdown's cleanup against its deadline, so that an await that ignores cancellation cannot keep the + * daemon from exiting. + */ +export class DaemonShutdownDeadline { + readonly #options: IDaemonShutdownDeadlineOptions; + #forcedBy: string | undefined; + #cutShort: ((forcedBy: string | undefined) => void) | undefined; + + public constructor(options: IDaemonShutdownDeadlineOptions) { + this.#options = options; + } + + /** + * Settles as the cleanup does, unless the deadline passes or {@link DaemonShutdownDeadline.expire} is called + * first. Then it rejects with {@link DaemonShutdownDeadlineError}, and the cleanup goes on in the background. + */ + public async raceAsync(cleanup: Promise): Promise { + const startedAtMs: number = Date.now(); + let timer: NodeJS.Timeout | undefined; + const cutShort: Promise = new Promise((resolve, reject) => { + this.#cutShort = (forcedBy: string | undefined) => + reject( + new DaemonShutdownDeadlineError({ + ...this.#options.getProgress(), + elapsedMs: Date.now() - startedAtMs, + forcedBy + }) + ); + }); + if (this.#forcedBy !== undefined) { + this.#cutShort?.(this.#forcedBy); + } else if (this.#options.timeoutMs !== undefined) { + timer = setTimeout(() => this.#cutShort?.(undefined), this.#options.timeoutMs); + } + try { + await Promise.race([cleanup, cutShort]); + } catch (error) { + if (error instanceof DaemonShutdownDeadlineError) { + cleanup.catch((lateError: unknown) => + this.#options.onLateFailure(lateError instanceof Error ? lateError : new Error(String(lateError))) + ); + } + throw error; + } finally { + clearTimeout(timer); + this.#cutShort = undefined; + } + } + + /** Cuts the running shutdown short now. A shutdown that has not started yet is cut short when it starts. */ + public expire(forcedBy: string): void { + this.#forcedBy ??= forcedBy; + this.#cutShort?.(forcedBy); + } +} diff --git a/libraries/rush-daemon/src/DaemonShutdownDeadlineError.ts b/libraries/rush-daemon/src/DaemonShutdownDeadlineError.ts new file mode 100644 index 0000000000..61c8cc111f --- /dev/null +++ b/libraries/rush-daemon/src/DaemonShutdownDeadlineError.ts @@ -0,0 +1,90 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * The step of a daemon shutdown that was still running when it was cut short. + * + * @beta + */ +export type DaemonShutdownStage = + | 'requests' + | 'workspaceMaintenance' + | 'requestDispatcher' + | 'workspaceSession' + | 'listener'; + +/** + * Options for {@link DaemonShutdownDeadlineError}. + * + * @beta + */ +export interface IDaemonShutdownDeadlineErrorOptions { + /** How long the shutdown had been running. */ + readonly elapsedMs: number; + /** What cut the shutdown short before its deadline, such as "a second SIGTERM". Omitted when the deadline did. */ + readonly forcedBy?: string; + /** The step that was still running. */ + readonly stage: DaemonShutdownStage; + /** A description of each request that had not finished. */ + readonly unfinishedRequests: ReadonlyArray; +} + +/** + * Reports a daemon shutdown that was cut short, by its deadline or by a second signal, before its cleanup + * finished. The cleanup may still be running. + * + * @remarks + * A daemon process that gets it releases what it safely can and exits, instead of waiting forever for an await + * that ignores cancellation, such as a lock that another process holds. + * + * @beta + */ +export class DaemonShutdownDeadlineError extends Error { + /** How long the shutdown had been running when it was cut short. */ + public readonly elapsedMs: number; + /** What cut the shutdown short before its deadline, such as "a second SIGTERM". Undefined when the deadline did. */ + public readonly forcedBy: string | undefined; + /** The step that was still running. */ + public readonly stage: DaemonShutdownStage; + /** A description of each request that had not finished, such as `"build -t a" (running for 12.3 s)`. */ + public readonly unfinishedRequests: ReadonlyArray; + + public constructor(options: IDaemonShutdownDeadlineErrorOptions) { + super(formatMessage(options)); + this.name = 'DaemonShutdownDeadlineError'; + this.elapsedMs = options.elapsedMs; + this.forcedBy = options.forcedBy; + this.stage = options.stage; + this.unfinishedRequests = options.unfinishedRequests; + } +} + +const MS_PER_SECOND: number = 1000; + +function formatMessage(options: IDaemonShutdownDeadlineErrorOptions): string { + const elapsed: string = `${(options.elapsedMs / MS_PER_SECOND).toFixed(1)} s`; + const ending: string = + options.forcedBy === undefined + ? `did not finish within ${elapsed}` + : `was cut short by ${options.forcedBy} after ${elapsed}`; + const requests: string = + options.unfinishedRequests.length > 0 + ? ` Unfinished requests: ${options.unfinishedRequests.join('; ')}.` + : ''; + return `The Rush daemon's shutdown ${ending}, while ${describeStage(options.stage)}.${requests}`; +} + +function describeStage(stage: DaemonShutdownStage): string { + switch (stage) { + case 'requests': + return 'waiting for requests to finish'; + case 'workspaceMaintenance': + return 'stopping workspace maintenance'; + case 'requestDispatcher': + return 'disposing the request dispatcher'; + case 'workspaceSession': + return 'disposing the workspace session'; + case 'listener': + return 'closing the listener'; + } +} diff --git a/libraries/rush-daemon/src/DaemonShutdownSignals.ts b/libraries/rush-daemon/src/DaemonShutdownSignals.ts new file mode 100644 index 0000000000..e26b6bad2f --- /dev/null +++ b/libraries/rush-daemon/src/DaemonShutdownSignals.ts @@ -0,0 +1,58 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { DaemonShutdownError } from './DaemonShutdownError'; + +/** The part of `process` that {@link listenForShutdownSignals} uses. */ +export interface IShutdownSignalEmitter { + on(event: NodeJS.Signals, listener: (signal: NodeJS.Signals) => void): unknown; + off(event: NodeJS.Signals, listener: (signal: NodeJS.Signals) => void): unknown; +} + +/** Options for {@link listenForShutdownSignals}. */ +export interface IShutdownSignalsOptions { + readonly emitter: IShutdownSignalEmitter; + /** Called for each termination signal after the first one, such as a second Ctrl+C. */ + readonly onForce: (signal: NodeJS.Signals) => void; + readonly getNowMs?: () => number; +} + +/** The shutdown signal of a daemon that owns its process. */ +export interface IShutdownSignals { + /** Aborted by the first SIGINT or SIGTERM, with a {@link DaemonShutdownError}. */ + readonly signal: AbortSignal; + readonly dispose: () => void; +} + +const SHUTDOWN_SIGNALS: ReadonlyArray = ['SIGINT', 'SIGTERM']; +// SubprocessTerminator's listener kills the tracked child processes, removes itself and sends the first signal to +// this process again, which arrives within a moment. That copy does not ask to force the shutdown. +const RELAY_WINDOW_MS: number = 1000; + +/** + * Listens for SIGINT and SIGTERM until disposed. The first one requests a clean shutdown. Each later one calls + * `onForce`, so that the daemon can exit even when its shutdown does not finish, except for one copy of the first + * signal that arrives within a second, which SubprocessTerminator sends. + */ +export function listenForShutdownSignals(options: IShutdownSignalsOptions): IShutdownSignals { + const { emitter, onForce, getNowMs = Date.now } = options; + const controller: AbortController = new AbortController(); + let relay: { readonly signal: NodeJS.Signals; readonly deadlineMs: number } | undefined; + const onSignal: (signal: NodeJS.Signals) => void = (signal: NodeJS.Signals) => { + if (!controller.signal.aborted) { + relay = { signal, deadlineMs: getNowMs() + RELAY_WINDOW_MS }; + controller.abort(new DaemonShutdownError({ initiator: 'signal', signal })); + return; + } + const isRelay: boolean = relay?.signal === signal && getNowMs() <= relay.deadlineMs; + relay = undefined; + if (!isRelay) onForce(signal); + }; + for (const name of SHUTDOWN_SIGNALS) emitter.on(name, onSignal); + return { + signal: controller.signal, + dispose: () => { + for (const name of SHUTDOWN_SIGNALS) emitter.off(name, onSignal); + } + }; +} diff --git a/libraries/rush-daemon/src/RushDaemonHost.ts b/libraries/rush-daemon/src/RushDaemonHost.ts index 52364c7886..d6f3f8a002 100644 --- a/libraries/rush-daemon/src/RushDaemonHost.ts +++ b/libraries/rush-daemon/src/RushDaemonHost.ts @@ -1,8 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as fs from 'node:fs'; import { realpath } from 'node:fs/promises'; +import { LockFile } from '@rushstack/node-core-library'; import { connectOrStartDaemonAsync, type DaemonClient } from '@rushstack/rush-client-core'; import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; import type { IDaemonWorkspaceStatus } from '@rushstack/rush-daemon-protocol'; @@ -19,6 +21,8 @@ import { DaemonIdleTimer } from './DaemonIdleTimer'; import type { IDaemonInteractiveConnection } from './DaemonInteractiveConnection'; import { DaemonRequestDispatcher } from './DaemonRequestDispatcher'; import type { IDaemonRequestResolver } from './DaemonRequestDispatcher'; +import { DaemonShutdownDeadline } from './DaemonShutdownDeadline'; +import { DaemonShutdownDeadlineError, type DaemonShutdownStage } from './DaemonShutdownDeadlineError'; import { DaemonShutdownError, type DaemonShutdownInitiator } from './DaemonShutdownError'; import { WorkspaceSession } from './WorkspaceSession'; import type { IWorkspaceSession, WorkspaceSessionFactory } from './WorkspaceSession'; @@ -45,6 +49,13 @@ export interface IRushDaemonHostOptions { readonly daemonVersion: string; /** Shuts down after this many seconds without pending requests. Disabled when omitted. */ readonly idleTimeoutSeconds?: number; + /** + * How long {@link RushDaemonHost.closeAsync} waits for shutdown cleanup before it rejects with + * {@link DaemonShutdownDeadlineError}, so that an await that ignores cancellation cannot keep the daemon from + * exiting. The cleanup goes on in the background. No deadline when omitted, except that + * {@link serveRushDaemonAsync} defaults it for a daemon that owns its process. + */ + readonly shutdownDeadlineMs?: number; /** Reports connection-level failures. */ readonly onError?: (error: Error) => void; /** @@ -86,7 +97,9 @@ export class RushDaemonHost { readonly #requestDispatcher: DaemonRequestDispatcher; public readonly paths: IDaemonPaths; #closePromise: Promise | undefined; + #closeStage: DaemonShutdownStage = 'requests'; #notifyClosed: (() => void) | undefined; + readonly #shutdownDeadline: DaemonShutdownDeadline; readonly #options: IRushDaemonHostOptions; readonly #startedAt: string; #restartPromise: Promise | undefined; @@ -98,7 +111,10 @@ export class RushDaemonHost { */ public readonly restartCompleted: Promise; - /** Resolves after shutdown cleanup finishes. Use closeAsync() to observe cleanup failures. */ + /** + * Resolves after shutdown cleanup finishes, fails, or is cut short by its deadline. Use closeAsync() to observe + * cleanup failures. + */ public readonly closed: Promise; private constructor( @@ -126,6 +142,16 @@ export class RushDaemonHost { this.#idleTimer = idleTimer; this.#options = options; this.#startedAt = startedAt; + this.#shutdownDeadline = new DaemonShutdownDeadline({ + timeoutMs: options.shutdownDeadlineMs, + getProgress: () => ({ + stage: this.#closeStage, + unfinishedRequests: Array.from(this.#sessions, (session: DaemonControlSession) => + session.describeActiveRequests() + ).flat() + }), + onLateFailure: (error: Error) => this.#reportError(error) + }); this.restartCompleted = new Promise((resolve, reject) => { this.#resolveRestart = resolve; this.#rejectRestart = reject; @@ -251,8 +277,8 @@ export class RushDaemonHost { } function requestShutdown(initiator: DaemonShutdownInitiator): void { void host.closeAsync(new DaemonShutdownError({ initiator })).catch((error: Error) => { - if (options.onError) options.onError(error); - else process.emitWarning(error); + // The host's owner sees a shutdown cut short through closeAsync(), and decides whether to exit. + if (!(error instanceof DaemonShutdownDeadlineError)) host.#reportError(error); }); } idleTimer.start(() => requestShutdown('idleTimeout')); @@ -277,16 +303,55 @@ export class RushDaemonHost { /** * Closes active connections, stops listening, and removes transport artifacts. * + * @remarks + * Rejects with {@link DaemonShutdownDeadlineError} when the cleanup does not finish within + * {@link IRushDaemonHostOptions.shutdownDeadlineMs}, or when {@link RushDaemonHost.expireShutdownDeadline} is + * called first. The listener then keeps its socket and lockfile; see {@link RushDaemonHost.releaseForExit}. + * * @param reason - Delivered to requests that are still running; only the first close call's reason is used. */ public closeAsync(reason?: DaemonShutdownError): Promise { - this.#closePromise ??= this.#closeOnceAsync(reason).finally(() => { + this.#closePromise ??= this.#shutdownDeadline.raceAsync(this.#closeOnceAsync(reason)).finally(() => { this.#notifyClosed?.(); if (!this.#restartPromise) this.#resolveRestart?.(undefined); }); return this.#closePromise; } + /** + * Cuts the shutdown short, for example on a second termination signal: a running or later + * {@link RushDaemonHost.closeAsync} rejects with {@link DaemonShutdownDeadlineError} at once. + * + * @param forcedBy - What cut the shutdown short, such as "a second SIGTERM", for the error message. + */ + public expireShutdownDeadline(forcedBy: string): void { + this.#shutdownDeadline.expire(forcedBy); + } + + /** + * For a daemon process that exits after its shutdown was cut short: removes the socket, the lockfile and this + * process's repository lock (`common/temp/rush#.lock`), unless the daemon still has child processes. Then + * they all stay, as after a crash, so that the next daemon reaps those processes when it reclaims the socket. + * + * @returns Whether they were removed. + */ + public releaseForExit(): boolean { + if (!this.#listener.releaseForExit()) return false; + const session: IWorkspaceSession | undefined = this.#workspaceSessionProvider.currentSession; + // On Windows the lock is an open handle that the exit releases. + if (session && process.platform !== 'win32') { + fs.rmSync(LockFile.getLockFilePath(session.rushConfiguration.commonTempFolder, 'rush'), { + force: true + }); + } + return true; + } + + #reportError(error: Error): void { + if (this.#options.onError) this.#options.onError(error); + else process.emitWarning(error); + } + #requestRestart(plan: IWorkspaceProcessRestartPlan): void { if (this.#restartPromise || this.#closePromise) return; this.#restartPromise = Promise.resolve().then(() => this.#restartOnceAsync(plan)); @@ -295,8 +360,7 @@ export class RushDaemonHost { (error: unknown) => { const failure: Error = error instanceof Error ? error : new Error(String(error)); this.#rejectRestart?.(failure); - if (this.#options.onError) this.#options.onError(failure); - else process.emitWarning(failure); + this.#reportError(failure); } ); } @@ -336,6 +400,7 @@ export class RushDaemonHost { const errors: unknown[] = []; // Refuse new sessions but keep the listener's live ownership until every resource join succeeds. // A failed standalone host must not exit naturally and become reclaimable over unjoined children. + this.#closeStage = 'requests'; const sessionSettlements: PromiseSettledResult[] = await Promise.allSettled( Array.from(this.#sessions, (session: DaemonControlSession) => session.closeAsync(!!this.#restartPromise, reason ?? new DaemonShutdownError({ initiator: 'host' })) @@ -346,17 +411,20 @@ export class RushDaemonHost { errors.push(settlement.reason); } } + this.#closeStage = 'workspaceMaintenance'; try { const workspace: IWorkspaceSession = await this.#workspaceSessionProvider.getSessionAsync(); await workspace.quiesceWarmSetAsync?.(); } catch (error) { throw new AggregateError([...errors, error], 'Could not quiesce workspace maintenance for shutdown.'); } + this.#closeStage = 'requestDispatcher'; try { await this.#requestDispatcher[Symbol.asyncDispose](); } catch (error) { errors.push(error); } + this.#closeStage = 'workspaceSession'; try { await this.#workspaceSessionProvider[Symbol.asyncDispose](); } catch (error) { @@ -364,6 +432,7 @@ export class RushDaemonHost { } if (errors.length === 0) { + this.#closeStage = 'listener'; try { await this.#listener.closeAsync(); } catch (error) { diff --git a/libraries/rush-daemon/src/index.ts b/libraries/rush-daemon/src/index.ts index 282d9a6f75..319866c7c6 100644 --- a/libraries/rush-daemon/src/index.ts +++ b/libraries/rush-daemon/src/index.ts @@ -64,6 +64,11 @@ export { type DaemonShutdownInitiator, type IDaemonShutdownErrorOptions } from './DaemonShutdownError'; +export { + DaemonShutdownDeadlineError, + type DaemonShutdownStage, + type IDaemonShutdownDeadlineErrorOptions +} from './DaemonShutdownDeadlineError'; export { serveRushDaemonAsync, type IRushDaemonServeOptions } from './serveRushDaemon'; export { WorkspaceEngineComponentFactory, diff --git a/libraries/rush-daemon/src/serveRushDaemon.ts b/libraries/rush-daemon/src/serveRushDaemon.ts index 5568767526..f47ff7c194 100644 --- a/libraries/rush-daemon/src/serveRushDaemon.ts +++ b/libraries/rush-daemon/src/serveRushDaemon.ts @@ -1,8 +1,12 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as fs from 'node:fs'; + import { captureDaemonInstallation, getDaemonInstallationFolders } from './DaemonInstallationMonitor'; +import { DaemonShutdownDeadlineError } from './DaemonShutdownDeadlineError'; import { DaemonShutdownError } from './DaemonShutdownError'; +import { listenForShutdownSignals } from './DaemonShutdownSignals'; import { RushDaemonHost } from './RushDaemonHost'; import type { IRushDaemonHostOptions } from './RushDaemonHost'; import { getInstalledWorkspaceSuccessorLaunchAsync } from './WorkspaceProcessRestart'; @@ -15,10 +19,29 @@ import { getInstalledWorkspaceSuccessorLaunchAsync } from './WorkspaceProcessRes export interface IRushDaemonServeOptions extends IRushDaemonHostOptions { /** Called after the listener is bound and the lockfile is available. */ readonly onReady?: (host: RushDaemonHost) => void | Promise; - /** Requests a clean shutdown. Process signals are used when omitted. */ + /** + * Requests a clean shutdown. When omitted, the daemon owns its process: the first SIGINT or SIGTERM requests a + * clean shutdown, and {@link IRushDaemonHostOptions.shutdownDeadlineMs} defaults to 10 seconds. If the shutdown + * does not finish by then, or another signal arrives first, the daemon reports why, releases what it safely can + * and exits the process with code 1. Once it stops, if something else keeps the process running for 2 seconds, + * it reports the active resources that Node.js lists and exits the process, keeping `process.exitCode`. + */ readonly shutdownSignal?: AbortSignal; } +/** + * How long a daemon that owns its process waits for its shutdown to finish before it exits anyway. It covers the + * 5 seconds for which a closing connection waits for its requests. + */ +export const DEFAULT_SHUTDOWN_DEADLINE_MS: number = 10000; + +/** + * How long a daemon that owns its process waits, after it stops serving, for the process to end by itself before it + * exits anyway. By then it has released its socket and lockfile, so `daemon status` and `daemon stop` can no longer + * see it; a timer or handle that something else left behind (a plugin, a tool, an SDK) must not keep it running. + */ +const EXIT_AFTER_STOP_MS: number = 2000; + /** * Starts a daemon host, signals readiness, and serves until shutdown is requested. * @@ -30,9 +53,38 @@ export interface IRushDaemonServeOptions extends IRushDaemonHostOptions { * @beta */ export async function serveRushDaemonAsync(options: IRushDaemonServeOptions): Promise { - const signalRegistration: IShutdownSignalRegistration = options.shutdownSignal - ? { signal: options.shutdownSignal, dispose: () => undefined } - : createProcessShutdownSignal(); + if (options.shutdownSignal) { + await serveUntilClosedAsync(options, { signal: options.shutdownSignal, dispose: () => undefined }); + return; + } + let host: RushDaemonHost | undefined; + const signalRegistration: IShutdownSignalRegistration = listenForShutdownSignals({ + emitter: process, + onForce: (signal: NodeJS.Signals) => { + // Nothing to release before the host has started. + if (host) host.expireShutdownDeadline(`a second ${signal}`); + else process.exit(1); + } + }); + try { + await serveUntilClosedAsync( + { ...options, shutdownDeadlineMs: options.shutdownDeadlineMs ?? DEFAULT_SHUTDOWN_DEADLINE_MS }, + signalRegistration, + (startedHost: RushDaemonHost) => (host = startedHost) + ); + } catch (error) { + if (host && error instanceof DaemonShutdownDeadlineError) exitAfterShutdownDeadline(host, error, options); + throw error; + } finally { + exitIfStillRunning(options); + } +} + +async function serveUntilClosedAsync( + options: IRushDaemonServeOptions, + signalRegistration: IShutdownSignalRegistration, + onStarted?: (host: RushDaemonHost) => void +): Promise { let host: RushDaemonHost | undefined; try { host = await RushDaemonHost.startAsync({ @@ -48,13 +100,86 @@ export async function serveRushDaemonAsync(options: IRushDaemonServeOptions): Pr return await getInstalledWorkspaceSuccessorLaunchAsync(context); }) }); + onStarted?.(host); await options.onReady?.(host); await waitForShutdownAsync(host, signalRegistration.signal); await host.closeAsync(getShutdownReason(signalRegistration.signal)); await host.restartCompleted; } finally { - signalRegistration.dispose(); - await host?.closeAsync(); + try { + await host?.closeAsync(); + } finally { + signalRegistration.dispose(); + } + } +} + +/** + * Exits a daemon process whose shutdown was cut short. The requests that did not finish already have their typed + * results. The 'exit' hook of SubprocessTerminator kills the child processes that it tracks. + */ +function exitAfterShutdownDeadline( + host: RushDaemonHost, + error: DaemonShutdownDeadlineError, + options: IRushDaemonServeOptions +): never { + let outcome: string; + try { + outcome = host.releaseForExit() + ? 'The daemon released its socket, lockfile and repository lock, and exits.' + : 'The daemon exits and leaves its socket and lockfile to the next daemon, which reaps its child processes.'; + } catch (releaseError) { + outcome = `The daemon exits; it could not release its socket and lockfile: ${String(releaseError)}`; + } + const report: Error = new Error(`${error.message} ${outcome}`, { cause: error }); + reportBeforeExit(report, options); + process.exit(1); +} + +/** + * Exits the process {@link EXIT_AFTER_STOP_MS} after the daemon stopped serving, if it is still running then, and + * reports what kept it running. The timer does not keep the process running itself. `process.exit()` keeps an exit + * code that the caller set, and the 'exit' hook of SubprocessTerminator kills the child processes that it still + * tracks. A successor daemon is not one of them: it is started detached and untracked. + */ +function exitIfStillRunning(options: IRushDaemonServeOptions): void { + setTimeout(() => { + reportBeforeExit( + new Error( + `The Rush daemon stopped, but something kept its process running for ${EXIT_AFTER_STOP_MS / 1000} s, ` + + `so it exits now. Active resources that Node.js reports: ${describeActiveResources()}.` + ), + options + ); + process.exit(); + }, EXIT_AFTER_STOP_MS).unref(); +} + +function describeActiveResources(): string { + const counts: Map = new Map(); + for (const resource of process.getActiveResourcesInfo()) { + counts.set(resource, (counts.get(resource) ?? 0) + 1); + } + const resources: string[] = []; + for (const [resource, count] of counts) { + resources.push(count > 1 ? `${resource} (${count})` : resource); + } + return resources.length > 0 ? resources.join(', ') : 'none'; +} + +/** + * Reports an error right before `process.exit()`. The report must be written synchronously: `process.emitWarning` + * prints on the next tick, which never comes. + */ +function reportBeforeExit(report: Error, options: IRushDaemonServeOptions): void { + if (options.onError) { + options.onError(report); + return; + } + try { + fs.writeSync(process.stderr.fd, `${report.stack ?? report.message}\n`); + } catch { + // The process exits either way. } } @@ -70,21 +195,6 @@ interface IShutdownSignalRegistration { readonly dispose: () => void; } -function createProcessShutdownSignal(): IShutdownSignalRegistration { - const controller: AbortController = new AbortController(); - const onSignal: (signal: NodeJS.Signals) => void = (signal: NodeJS.Signals) => - controller.abort(new DaemonShutdownError({ initiator: 'signal', signal })); - process.once('SIGINT', onSignal); - process.once('SIGTERM', onSignal); - return { - signal: controller.signal, - dispose: () => { - process.off('SIGINT', onSignal); - process.off('SIGTERM', onSignal); - } - }; -} - function waitForShutdownAsync(host: RushDaemonHost, signal: AbortSignal): Promise { if (signal.aborted) { return Promise.resolve(); diff --git a/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts b/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts new file mode 100644 index 0000000000..f05a509f7f --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts @@ -0,0 +1,178 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { readDaemonLockfile, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { DaemonRequestWireClient } from './DaemonRequestWireTestUtilities'; +import type { LeftBehind, Reporter } from './fixtures/LingeringDaemon'; +import { createTemporaryRepo } from './TemporaryRepoWorkspaceSession'; + +const FIXTURE_PATH: string = path.join(__dirname, 'fixtures', 'LingeringDaemon.js'); +const POLL_INTERVAL_MS: number = 20; +const START_TIMEOUT_MS: number = 15000; +// Far beyond the 2 s after which the daemon exits anyway, and far below the test's timeout. +const EXIT_TIMEOUT_MS: number = 8000; +const LINGER_REPORT: RegExp = + /The Rush daemon stopped, but something kept its process running for 2 s, so it exits now\. Active resources that Node\.js reports: [^\n]*\bTimeout\b/; + +interface IProcessExit { + readonly code: number | undefined; + readonly signal: NodeJS.Signals | undefined; +} + +interface IFixtureDaemon { + readonly process: ChildProcess; + readonly exited: Promise; + readonly getStderr: () => string; +} + +type StopRoute = 'daemon stop' | 'SIGTERM'; + +(process.platform === 'win32' ? describe.skip : describe)('a daemon process after the daemon stops', () => { + let folder: string; + let repoRoot: string; + let controlFolder: string; + let daemon: IFixtureDaemon | undefined; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-process-exit-')); + repoRoot = path.join(folder, 'repo'); + controlFolder = path.join(folder, 'control'); + createTemporaryRepo(repoRoot); + fs.mkdirSync(controlFolder); + fs.mkdirSync(path.join(folder, 'runtime'), { mode: 0o700 }); + }); + + afterEach(async () => { + const startedDaemon: IFixtureDaemon | undefined = daemon; + daemon = undefined; + // Only the fixture process that this test started, if a failed test left it running. + if ( + startedDaemon && + startedDaemon.process.exitCode === null && + startedDaemon.process.signalCode === null + ) { + startedDaemon.process.kill('SIGKILL'); + await startedDaemon.exited; + } + fs.rmSync(folder, { force: true, recursive: true }); + }); + + function startDaemon(leftBehind: LeftBehind, reporter: Reporter): IFixtureDaemon { + const child: ChildProcess = spawn( + process.execPath, + [FIXTURE_PATH, repoRoot, controlFolder, leftBehind, reporter], + { + // Like a daemon that rush-client launches: the leader of its own process group. + detached: true, + env: { ...process.env, RUSHD_RUNTIME_DIR: path.join(folder, 'runtime'), RUSH_TEMP_FOLDER: undefined }, + stdio: ['ignore', 'ignore', 'pipe'] + } + ); + let stderr: string = ''; + child.stderr?.setEncoding('utf8').on('data', (chunk: string) => (stderr += chunk)); + const exited: Promise = new Promise((resolve) => { + child.once('exit', (code, signal) => resolve({ code: code ?? undefined, signal: signal ?? undefined })); + }); + daemon = { process: child, exited, getStderr: () => stderr }; + return daemon; + } + + async function waitForJsonAsync(fixture: IFixtureDaemon, name: string): Promise { + const filename: string = path.join(controlFolder, name); + const deadlineMs: number = Date.now() + START_TIMEOUT_MS; + while (!fs.existsSync(filename)) { + if (fixture.process.exitCode !== null || Date.now() > deadlineMs) { + throw new Error(`The fixture did not write ${name}. Its stderr:\n${fixture.getStderr()}`); + } + await delayAsync(POLL_INTERVAL_MS); + } + return JSON.parse(fs.readFileSync(filename, 'utf8')); + } + + async function stopAsync(fixture: IFixtureDaemon, paths: IDaemonPaths, route: StopRoute): Promise { + if (route === 'SIGTERM') { + fixture.process.kill('SIGTERM'); + return; + } + const client: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync(paths.socketPath); + try { + await client.handshakeAsync(); + await client.sendControlAsync({ kind: 'shutdown', payload: {} }); + expect(await client.readControlAsync()).toEqual({ + kind: 'shutdownAck', + payload: { activeRequests: 0 } + }); + await client.closed; + } finally { + await client.closeAsync(); + } + } + + /** Resolves with the exit and how long after the daemon stopped it came. */ + async function waitForExitAsync(fixture: IFixtureDaemon): Promise { + const { stoppedAtMs } = await waitForJsonAsync<{ stoppedAtMs: number }>(fixture, 'stopped.json'); + const timeout: AbortController = new AbortController(); + const exit: IProcessExit | undefined = await Promise.race([ + fixture.exited, + delayAsync(EXIT_TIMEOUT_MS, undefined, { signal: timeout.signal }).catch(() => undefined) + ]); + timeout.abort(); + if (!exit) { + throw new Error( + `The fixture process was still running ${EXIT_TIMEOUT_MS} ms after the daemon stopped. ` + + `Its stderr:\n${fixture.getStderr()}` + ); + } + return { ...exit, afterStopMs: Date.now() - stoppedAtMs }; + } + + it.each<[StopRoute, Reporter]>([ + ['daemon stop', 'onError'], + ['SIGTERM', 'default'] + ])( + 'exits 2 s after %s if a timer that it did not start keeps it running, and reports that through %s', + async (route: StopRoute, reporter: Reporter) => { + const fixture: IFixtureDaemon = startDaemon('timer', reporter); + const { paths } = await waitForJsonAsync<{ paths: IDaemonPaths }>(fixture, 'ready.json'); + await stopAsync(fixture, paths, route); + + const { code, signal, afterStopMs } = await waitForExitAsync(fixture); + expect({ code, signal }).toEqual({ code: 0, signal: undefined }); + expect(afterStopMs).toBeGreaterThanOrEqual(1900); + expect(afterStopMs).toBeLessThan(3000); + expect(fixture.getStderr()).toMatch(LINGER_REPORT); + expect(fs.existsSync(paths.socketPath)).toBe(false); + expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); + }, + 30000 + ); + + it('keeps the exit code that its caller set when it failed', async () => { + const fixture: IFixtureDaemon = startDaemon('failing', 'onError'); + + const { code, signal, afterStopMs } = await waitForExitAsync(fixture); + expect({ code, signal }).toEqual({ code: 1, signal: undefined }); + expect(afterStopMs).toBeGreaterThanOrEqual(1900); + expect(fixture.getStderr()).toMatch(/The fixture failed after it started\./); + expect(fixture.getStderr()).toMatch(LINGER_REPORT); + }, 30000); + + it('exits as soon as it stops if nothing keeps it running', async () => { + const fixture: IFixtureDaemon = startDaemon('nothing', 'onError'); + const { paths } = await waitForJsonAsync<{ paths: IDaemonPaths }>(fixture, 'ready.json'); + await stopAsync(fixture, paths, 'daemon stop'); + + const { code, signal } = await waitForExitAsync(fixture); + expect({ code, signal }).toEqual({ code: 0, signal: undefined }); + expect(fixture.getStderr()).not.toMatch(/kept its process running/); + expect(fs.existsSync(paths.socketPath)).toBe(false); + expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); + }, 30000); +}); diff --git a/libraries/rush-daemon/src/test/DaemonShutdownDeadline.test.ts b/libraries/rush-daemon/src/test/DaemonShutdownDeadline.test.ts new file mode 100644 index 0000000000..b870a8b76e --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonShutdownDeadline.test.ts @@ -0,0 +1,204 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { LockFile } from '@rushstack/node-core-library'; +import { readDaemonLockfile } from '@rushstack/rush-daemon-transport'; + +import { DaemonShutdownDeadline } from '../DaemonShutdownDeadline'; +import { DaemonShutdownDeadlineError } from '../DaemonShutdownDeadlineError'; +import { DaemonShutdownError } from '../DaemonShutdownError'; +import { RushDaemonHost } from '../RushDaemonHost'; +import type { IRushDaemonHostOptions } from '../RushDaemonHost'; +import { + CallbackDaemonRequestResolver, + createDeferred, + createWireEnvelope, + DaemonRequestWireClient +} from './DaemonRequestWireTestUtilities'; +import { captureTestDaemonListenerAsync } from './TestDaemonListener'; +import { createTemporaryRepo, TemporaryRepoWorkspaceSession } from './TemporaryRepoWorkspaceSession'; + +const REQUEST_ID: string = 'waits-for-lock'; + +describe('daemon shutdown deadline', () => { + let repoRoot: string; + let commonTempFolder: string; + + beforeEach(() => { + repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-deadline-')); + commonTempFolder = createTemporaryRepo(repoRoot); + }); + afterEach(() => fs.rmSync(repoRoot, { force: true, recursive: true })); + + function createOptions(overrides: Partial = {}): IRushDaemonHostOptions { + return { + createWorkspaceSessionAsync: () => Promise.resolve(new TemporaryRepoWorkspaceSession(repoRoot)), + daemonVersion: 'deadline-test', + repoRoot, + rushVersion: '5.178.1', + // Like a request that waits for a lock that another process holds, it ignores its abort signal. + requestResolver: new CallbackDaemonRequestResolver(() => new Promise(() => undefined)), + ...overrides + }; + } + + async function startRequestAsync(host: RushDaemonHost): Promise { + const client: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync(host.paths.socketPath); + await client.handshakeAsync(); + await client.sendControlAsync({ + kind: 'requestStart', + payload: createWireEnvelope(REQUEST_ID, 'build', repoRoot, { argv: ['build', '-t', 'mini-a'] }) + }); + // The daemon handles a connection's frames in order, so the request has started once the pong arrives. + await client.sendControlAsync({ kind: 'ping', payload: {} }); + while ((await client.readControlAsync()).kind !== 'pong') { + // Skip the request's own messages. + } + return client; + } + + async function stopAsync(host: RushDaemonHost): Promise { + const client: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync(host.paths.socketPath); + await client.handshakeAsync(); + await client.sendControlAsync({ kind: 'shutdown', payload: {} }); + expect(await client.readControlAsync()).toEqual({ kind: 'shutdownAck', payload: { activeRequests: 1 } }); + } + + it('cuts a stopped daemon short, gives the request a typed result and releases on exit', async () => { + const onError: jest.Mock = jest.fn(); + const { value: host, listener } = await captureTestDaemonListenerAsync(() => + RushDaemonHost.startAsync(createOptions({ onError, shutdownDeadlineMs: 300 })) + ); + const client: DaemonRequestWireClient = await startRequestAsync(host); + // The lock that the stuck request would hold. + const repoLock: LockFile | undefined = LockFile.tryAcquire(commonTempFolder, 'rush'); + try { + expect(repoLock).toBeDefined(); + await stopAsync(host); + await host.closed; + const error: unknown = await host.closeAsync().catch((closeError: unknown) => closeError); + expect(error).toBeInstanceOf(DaemonShutdownDeadlineError); + expect(error).toMatchObject({ forcedBy: undefined, stage: 'requests' }); + expect((error as DaemonShutdownDeadlineError).unfinishedRequests).toEqual([ + expect.stringMatching(/^"build -t mini-a" \(running for \d+\.\d s\)$/) + ]); + expect((error as Error).message).toMatch( + /^The Rush daemon's shutdown did not finish within \d+\.\d s, while waiting for requests to finish\. Unfinished requests: "build -t mini-a" \(running for \d+\.\d s\)\.$/ + ); + expect(onError).not.toHaveBeenCalled(); + // The shutdown was cut short, so the daemon still owns its endpoint and the repository lock. + expect(readDaemonLockfile(host.paths.lockfilePath)?.pid).toBe(process.pid); + if (process.platform !== 'win32') expect(fs.existsSync(host.paths.socketPath)).toBe(true); + + // The request never finishes, so its connection's drain times out and the client gets a typed result. + expect((await client.readTerminalAsync(REQUEST_ID)).terminal).toEqual({ + kind: 'requestResult', + payload: { + aborted: true, + errorMessage: + 'The Rush daemon was shut down (requested by "rush-client daemon stop" or "daemon restart") ' + + 'while this request was running; re-run the command.', + exitCode: 1, + outcome: 'failure', + requestId: REQUEST_ID + } + }); + await client.closed; + + expect(host.releaseForExit()).toBe(true); + expect(readDaemonLockfile(host.paths.lockfilePath)).toBeUndefined(); + if (process.platform !== 'win32') { + expect(fs.existsSync(host.paths.socketPath)).toBe(false); + expect(fs.existsSync(LockFile.getLockFilePath(commonTempFolder, 'rush'))).toBe(false); + } + } finally { + repoLock?.release(); + await client.closeAsync(); + await listener.closeAsync(); + } + }, 20000); + + it('cuts a running shutdown short when it is expired, as a second signal does', async () => { + const { value: host, listener } = await captureTestDaemonListenerAsync(() => + RushDaemonHost.startAsync(createOptions()) + ); + const client: DaemonRequestWireClient = await startRequestAsync(host); + try { + const closing: Promise = host.closeAsync( + new DaemonShutdownError({ initiator: 'signal', signal: 'SIGTERM' }) + ); + host.expireShutdownDeadline('a second SIGTERM'); + await expect(closing).rejects.toThrow( + /^The Rush daemon's shutdown was cut short by a second SIGTERM after \d+\.\d s, while waiting for requests to finish\. Unfinished requests: "build -t mini-a"/ + ); + await expect(closing).rejects.toMatchObject({ forcedBy: 'a second SIGTERM', stage: 'requests' }); + } finally { + await client.closeAsync(); + await listener.closeAsync(); + } + }); + + it('does not change a shutdown that finishes in time', async () => { + const host: RushDaemonHost = await RushDaemonHost.startAsync( + createOptions({ shutdownDeadlineMs: 10000 }) + ); + await host.closeAsync(); + await host.closed; + expect(readDaemonLockfile(host.paths.lockfilePath)).toBeUndefined(); + }); +}); + +describe(DaemonShutdownDeadline.name, () => { + function createDeadline( + timeoutMs: number | undefined, + onLateFailure: jest.Mock = jest.fn() + ): DaemonShutdownDeadline { + return new DaemonShutdownDeadline({ + timeoutMs, + getProgress: () => ({ stage: 'workspaceSession', unfinishedRequests: [] }), + onLateFailure + }); + } + + it('settles as the cleanup does when it finishes in time', async () => { + await expect(createDeadline(10000).raceAsync(Promise.resolve())).resolves.toBeUndefined(); + const failure: Error = new Error('cleanup failed'); + await expect(createDeadline(10000).raceAsync(Promise.reject(failure))).rejects.toBe(failure); + }); + + it('rejects at the deadline and reports a cleanup that fails later', async () => { + const onLateFailure: jest.Mock = jest.fn(); + const cleanup = createDeferred(); + const failing: Promise = cleanup.promise.then(() => { + throw new Error('late failure'); + }); + await expect(createDeadline(10, onLateFailure).raceAsync(failing)).rejects.toMatchObject({ + forcedBy: undefined, + stage: 'workspaceSession', + message: expect.stringMatching( + /^The Rush daemon's shutdown did not finish within \d+\.\d s, while disposing the workspace session\.$/ + ) + }); + cleanup.resolve(); + await failing.catch(() => undefined); + expect(onLateFailure).toHaveBeenCalledWith(new Error('late failure')); + }); + + it('is cut short by expire(), also when that comes before the shutdown starts', async () => { + const running: DaemonShutdownDeadline = createDeadline(undefined); + const racing: Promise = running.raceAsync(new Promise(() => undefined)); + running.expire('a second SIGINT'); + await expect(racing).rejects.toMatchObject({ forcedBy: 'a second SIGINT' }); + + const early: DaemonShutdownDeadline = createDeadline(undefined); + early.expire('a second SIGTERM'); + early.expire('a third SIGTERM'); + await expect(early.raceAsync(new Promise(() => undefined))).rejects.toMatchObject({ + forcedBy: 'a second SIGTERM' + }); + }); +}); diff --git a/libraries/rush-daemon/src/test/DaemonShutdownProcess.test.ts b/libraries/rush-daemon/src/test/DaemonShutdownProcess.test.ts new file mode 100644 index 0000000000..f388017848 --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonShutdownProcess.test.ts @@ -0,0 +1,186 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { LockFile } from '@rushstack/node-core-library'; +import { readDaemonLockfile, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { createWireEnvelope, DaemonRequestWireClient } from './DaemonRequestWireTestUtilities'; +import { createTemporaryRepo } from './TemporaryRepoWorkspaceSession'; +import { + captureTestProcessIdentity, + isTestProcessRunning, + type ITestProcessIdentity +} from './TestProcessExit'; + +const FIXTURE_PATH: string = path.join(__dirname, 'fixtures', 'StuckShutdownDaemon.js'); +const REQUEST_ID: string = 'stuck-request'; +const POLL_INTERVAL_MS: number = 20; +const START_TIMEOUT_MS: number = 15000; + +interface IProcessExit { + readonly code: number | undefined; + readonly signal: NodeJS.Signals | undefined; +} + +interface IStuckDaemon { + readonly process: ChildProcess; + readonly pid: number; + readonly exited: Promise; + readonly getStderr: () => string; +} + +interface IStuckRequest { + readonly client: DaemonRequestWireClient; + readonly paths: IDaemonPaths; + readonly operation: ITestProcessIdentity; + readonly repoLockPath: string; +} + +// A daemon process whose request ignores its abort signal, as one that waits for another process's lock does. +(process.platform === 'win32' ? describe.skip : describe)('a daemon process whose shutdown is stuck', () => { + let folder: string; + let repoRoot: string; + let commonTempFolder: string; + let controlFolder: string; + let daemon: IStuckDaemon | undefined; + let client: DaemonRequestWireClient | undefined; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rushd-stuck-shutdown-')); + repoRoot = path.join(folder, 'repo'); + controlFolder = path.join(folder, 'control'); + commonTempFolder = createTemporaryRepo(repoRoot); + fs.mkdirSync(controlFolder); + fs.mkdirSync(path.join(folder, 'runtime'), { mode: 0o700 }); + }); + + afterEach(async () => { + const openClient: DaemonRequestWireClient | undefined = client; + const startedDaemon: IStuckDaemon | undefined = daemon; + client = undefined; + daemon = undefined; + await openClient?.closeAsync(); + // Only the fixture process that this test started, if a failed test left it running. + if ( + startedDaemon && + startedDaemon.process.exitCode === null && + startedDaemon.process.signalCode === null + ) { + startedDaemon.process.kill('SIGKILL'); + await startedDaemon.exited; + } + fs.rmSync(folder, { force: true, recursive: true }); + }); + + async function waitForJsonAsync(stuckDaemon: IStuckDaemon, name: string): Promise { + const filename: string = path.join(controlFolder, name); + const deadlineMs: number = Date.now() + START_TIMEOUT_MS; + while (!fs.existsSync(filename)) { + if (stuckDaemon.process.exitCode !== null || Date.now() > deadlineMs) { + throw new Error(`The fixture did not write ${name}. Its stderr:\n${stuckDaemon.getStderr()}`); + } + await delayAsync(POLL_INTERVAL_MS); + } + return JSON.parse(fs.readFileSync(filename, 'utf8')); + } + + function startDaemon(shutdownDeadlineMs: number): IStuckDaemon { + const child: ChildProcess = spawn( + process.execPath, + [FIXTURE_PATH, repoRoot, controlFolder, String(shutdownDeadlineMs)], + { + // Like a daemon that rush-client launches: the leader of its own process group. + detached: true, + env: { ...process.env, RUSHD_RUNTIME_DIR: path.join(folder, 'runtime'), RUSH_TEMP_FOLDER: undefined }, + stdio: ['ignore', 'ignore', 'pipe'] + } + ); + let stderr: string = ''; + child.stderr?.setEncoding('utf8').on('data', (chunk: string) => (stderr += chunk)); + const exited: Promise = new Promise((resolve) => { + child.once('exit', (code, signal) => resolve({ code: code ?? undefined, signal: signal ?? undefined })); + }); + if (child.pid === undefined) throw new Error('The fixture did not start.'); + daemon = { process: child, pid: child.pid, exited, getStderr: () => stderr }; + return daemon; + } + + async function startStuckRequestAsync(stuckDaemon: IStuckDaemon): Promise { + const { paths } = await waitForJsonAsync<{ paths: IDaemonPaths }>(stuckDaemon, 'ready.json'); + const requestClient: DaemonRequestWireClient = await DaemonRequestWireClient.connectAsync( + paths.socketPath + ); + client = requestClient; + await requestClient.handshakeAsync(); + await requestClient.sendControlAsync({ + kind: 'requestStart', + payload: createWireEnvelope(REQUEST_ID, 'build', repoRoot, { argv: ['build', '-t', 'mini-a'] }) + }); + const { operationPid } = await waitForJsonAsync<{ operationPid: number }>(stuckDaemon, 'request.json'); + const repoLockPath: string = LockFile.getLockFilePath(commonTempFolder, 'rush', stuckDaemon.pid); + expect(fs.existsSync(repoLockPath)).toBe(true); + return { + client: requestClient, + paths, + operation: captureTestProcessIdentity(operationPid), + repoLockPath + }; + } + + function expectReleased(request: IStuckRequest): void { + // SubprocessTerminator killed the operation on the first signal, so there was nothing left to reap. + expect(isTestProcessRunning(request.operation)).toBe(false); + expect(fs.existsSync(request.paths.socketPath)).toBe(false); + expect(readDaemonLockfile(request.paths.lockfilePath)).toBeUndefined(); + expect(fs.existsSync(request.repoLockPath)).toBe(false); + } + + it('exits at its deadline, after its request gets a typed result, and releases what it owned', async () => { + const stuckDaemon: IStuckDaemon = startDaemon(6000); + const request: IStuckRequest = await startStuckRequestAsync(stuckDaemon); + const signaledAtMs: number = Date.now(); + // SubprocessTerminator sends this signal to the daemon again; that copy must not force the exit. + stuckDaemon.process.kill('SIGTERM'); + + expect((await request.client.readTerminalAsync(REQUEST_ID)).terminal).toEqual({ + kind: 'requestResult', + payload: { + aborted: true, + errorMessage: + 'The Rush daemon was shut down (the daemon process received SIGTERM) while this request was ' + + 'running; re-run the command.', + exitCode: 1, + outcome: 'failure', + requestId: REQUEST_ID + } + }); + expect(await stuckDaemon.exited).toEqual({ code: 1, signal: undefined }); + expect(Date.now() - signaledAtMs).toBeGreaterThanOrEqual(5900); + expect(stuckDaemon.getStderr()).toMatch( + /The Rush daemon's shutdown did not finish within 6\.\d s, while waiting for requests to finish\. Unfinished requests: "build -t mini-a" \(running for \d+\.\d s\)\. The daemon released its socket, lockfile and repository lock, and exits\./ + ); + expectReleased(request); + }, 30000); + + it('exits at once on a second signal', async () => { + // Far beyond the test's timeout, so only the second signal can end the shutdown in time. + const stuckDaemon: IStuckDaemon = startDaemon(60000); + const request: IStuckRequest = await startStuckRequestAsync(stuckDaemon); + stuckDaemon.process.kill('SIGTERM'); + await delayAsync(1500); + stuckDaemon.process.kill('SIGTERM'); + + expect(await stuckDaemon.exited).toEqual({ code: 1, signal: undefined }); + expect(stuckDaemon.getStderr()).toMatch( + /The Rush daemon's shutdown was cut short by a second SIGTERM after \d+\.\d s, while waiting for requests to finish\. Unfinished requests: "build -t mini-a" \(running for \d+\.\d s\)\. The daemon released its socket, lockfile and repository lock, and exits\./ + ); + await request.client.closed; + expectReleased(request); + }, 30000); +}); diff --git a/libraries/rush-daemon/src/test/DaemonShutdownSignals.test.ts b/libraries/rush-daemon/src/test/DaemonShutdownSignals.test.ts new file mode 100644 index 0000000000..4d8bcc7ec2 --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonShutdownSignals.test.ts @@ -0,0 +1,64 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { EventEmitter } from 'node:events'; + +import { DaemonShutdownError } from '../DaemonShutdownError'; +import { listenForShutdownSignals, type IShutdownSignals } from '../DaemonShutdownSignals'; + +describe(listenForShutdownSignals.name, () => { + let emitter: EventEmitter; + let nowMs: number; + let onForce: jest.Mock; + let signals: IShutdownSignals; + + beforeEach(() => { + emitter = new EventEmitter(); + nowMs = 1000; + onForce = jest.fn(); + signals = listenForShutdownSignals({ emitter, onForce, getNowMs: () => nowMs }); + }); + afterEach(() => signals.dispose()); + + function send(signal: NodeJS.Signals, afterMs: number = 0): void { + nowMs += afterMs; + emitter.emit(signal, signal); + } + + it('requests a clean shutdown on the first signal', () => { + expect(signals.signal.aborted).toBe(false); + send('SIGTERM'); + expect(signals.signal.reason).toBeInstanceOf(DaemonShutdownError); + expect(signals.signal.reason).toMatchObject({ initiator: 'signal', signal: 'SIGTERM' }); + expect(onForce).not.toHaveBeenCalled(); + }); + + it("ignores SubprocessTerminator's relay of the first signal, and forces on the next one", () => { + send('SIGTERM'); + send('SIGTERM', 5); + expect(onForce).not.toHaveBeenCalled(); + send('SIGTERM', 5); + expect(onForce).toHaveBeenCalledWith('SIGTERM'); + }); + + it('forces on a different signal, or on the same one after a moment', () => { + send('SIGTERM'); + send('SIGINT', 5); + expect(onForce).toHaveBeenLastCalledWith('SIGINT'); + + signals.dispose(); + onForce = jest.fn(); + signals = listenForShutdownSignals({ emitter, onForce, getNowMs: () => nowMs }); + send('SIGINT'); + send('SIGINT', 1500); + expect(onForce).toHaveBeenCalledWith('SIGINT'); + }); + + it('stops listening when disposed', () => { + expect(emitter.listenerCount('SIGINT')).toBe(1); + expect(emitter.listenerCount('SIGTERM')).toBe(1); + signals.dispose(); + expect(emitter.listenerCount('SIGINT')).toBe(0); + expect(emitter.listenerCount('SIGTERM')).toBe(0); + }); +}); diff --git a/libraries/rush-daemon/src/test/TemporaryRepoWorkspaceSession.ts b/libraries/rush-daemon/src/test/TemporaryRepoWorkspaceSession.ts new file mode 100644 index 0000000000..e7d917b761 --- /dev/null +++ b/libraries/rush-daemon/src/test/TemporaryRepoWorkspaceSession.ts @@ -0,0 +1,31 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { RushConfiguration } from '@microsoft/rush-lib'; + +import { TestWorkspaceSession } from './TestWorkspaceSession'; + +/** Creates an empty Rush repository in the folder, and returns its common/temp folder. */ +export function createTemporaryRepo(repoRoot: string): string { + const commonTempFolder: string = path.join(repoRoot, 'common', 'temp'); + fs.mkdirSync(path.join(repoRoot, 'common', 'config', 'rush'), { recursive: true }); + fs.mkdirSync(commonTempFolder, { recursive: true }); + fs.writeFileSync( + path.join(repoRoot, 'rush.json'), + JSON.stringify({ rushVersion: '5.178.1', pnpmVersion: '8.14.0', projects: [] }) + ); + return commonTempFolder; +} + +/** A workspace in a repository of its own, from {@link createTemporaryRepo}, whose lock a test can hold. */ +export class TemporaryRepoWorkspaceSession extends TestWorkspaceSession { + public override readonly rushConfiguration: RushConfiguration; + + public constructor(repoRoot: string) { + super(repoRoot); + this.rushConfiguration = RushConfiguration.loadFromConfigurationFile(path.join(repoRoot, 'rush.json')); + } +} diff --git a/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts b/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts new file mode 100644 index 0000000000..5da51032d0 --- /dev/null +++ b/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts @@ -0,0 +1,70 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { serveRushDaemonAsync, type IRushDaemonServeOptions } from '../../serveRushDaemon'; +import { TemporaryRepoWorkspaceSession } from '../TemporaryRepoWorkspaceSession'; + +/** + * - `timer`: a timer that the daemon did not start keeps the process running after the daemon stops. + * - `failing`: the same, and the daemon fails right after it starts, so its caller sets exit code 1. + * - `nothing`: nothing keeps the process running. + */ +export type LeftBehind = 'timer' | 'failing' | 'nothing'; + +/** + * - `onError`: reports through `onError`, as the Rush daemon's own entry points do. + * - `default`: no `onError`. + */ +export type Reporter = 'onError' | 'default'; + +function writeJson(filename: string, value: unknown): void { + fs.writeFileSync(`${filename}.tmp`, JSON.stringify(value)); + fs.renameSync(`${filename}.tmp`, filename); +} + +function onReady(leftBehind: LeftBehind, controlFolder: string): IRushDaemonServeOptions['onReady'] { + return (host) => { + if (leftBehind !== 'nothing') { + // Like a plugin or an SDK client that polls and is never stopped. Its first poll would come long after + // the test ends, so it only keeps the process running. + setInterval(() => undefined, 600000); + } + if (leftBehind === 'failing') throw new Error('The fixture failed after it started.'); + writeJson(path.join(controlFolder, 'ready.json'), { paths: host.paths }); + }; +} + +async function runAsync(): Promise { + const [repoRoot, controlFolder, leftBehind, reporter] = process.argv.slice(2); + if (!repoRoot || !controlFolder || !leftBehind || !reporter) { + throw new Error( + 'The fixture needs a repository folder, a control folder, what it leaves behind and a reporter.' + ); + } + try { + // Process mode: no shutdownSignal, so the daemon handles SIGINT and SIGTERM itself. + await serveRushDaemonAsync({ + repoRoot, + rushVersion: '5.178.1', + daemonVersion: 'lingering-fixture', + createWorkspaceSessionAsync: () => Promise.resolve(new TemporaryRepoWorkspaceSession(repoRoot)), + onReady: onReady(leftBehind as LeftBehind, controlFolder), + onError: + (reporter as Reporter) === 'onError' + ? (error: Error) => process.stderr.write(`${error.stack ?? error.message}\n`) + : undefined + }); + } finally { + writeJson(path.join(controlFolder, 'stopped.json'), { stoppedAtMs: Date.now() }); + } +} + +if (require.main === module) { + void runAsync().catch((error: unknown) => { + process.stderr.write(`${error instanceof Error ? (error.stack ?? error.message) : String(error)}\n`); + process.exitCode = 1; + }); +} diff --git a/libraries/rush-daemon/src/test/fixtures/StuckShutdownDaemon.ts b/libraries/rush-daemon/src/test/fixtures/StuckShutdownDaemon.ts new file mode 100644 index 0000000000..50e197440b --- /dev/null +++ b/libraries/rush-daemon/src/test/fixtures/StuckShutdownDaemon.ts @@ -0,0 +1,74 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { LockFile, SubprocessTerminator } from '@rushstack/node-core-library'; + +import type { IResolveDaemonRequestOptions } from '../../DaemonRequestDispatcher'; +import { serveRushDaemonAsync } from '../../serveRushDaemon'; +import { CallbackDaemonRequestResolver } from '../DaemonRequestWireTestUtilities'; +import { TemporaryRepoWorkspaceSession } from '../TemporaryRepoWorkspaceSession'; + +// Outlives the test, but ends by itself if a failed test leaves it behind. +const OPERATION_SCRIPT: string = 'setTimeout(() => undefined, 60000);'; + +function writeJson(filename: string, value: unknown): void { + fs.writeFileSync(`${filename}.tmp`, JSON.stringify(value)); + fs.renameSync(`${filename}.tmp`, filename); +} + +/** + * Like a request that holds the repository lock and runs an operation, and then waits for a lock that another + * process holds: it ignores its abort signal and never settles. + */ +async function runStuckRequestAsync( + options: IResolveDaemonRequestOptions, + controlFolder: string +): Promise { + const repoLock: LockFile | undefined = LockFile.tryAcquire( + options.workspaceSession.rushConfiguration.commonTempFolder, + 'rush' + ); + if (!repoLock) throw new Error('The repository lock is taken.'); + const operation: ChildProcess = spawn(process.execPath, ['-e', OPERATION_SCRIPT], { + ...SubprocessTerminator.RECOMMENDED_OPTIONS, + stdio: 'ignore' + }); + SubprocessTerminator.killProcessTreeOnExit(operation, SubprocessTerminator.RECOMMENDED_OPTIONS); + await new Promise((resolve, reject) => { + operation.once('spawn', resolve); + operation.once('error', reject); + }); + writeJson(path.join(controlFolder, 'request.json'), { operationPid: operation.pid }); + return await new Promise(() => undefined); +} + +async function runAsync(): Promise { + const [repoRoot, controlFolder, shutdownDeadlineMs] = process.argv.slice(2); + if (!repoRoot || !controlFolder || !shutdownDeadlineMs) { + throw new Error('The fixture needs a repository folder, a control folder and a shutdown deadline.'); + } + // Process mode: no shutdownSignal, so the daemon handles SIGINT and SIGTERM itself. + await serveRushDaemonAsync({ + repoRoot, + rushVersion: '5.178.1', + daemonVersion: 'stuck-shutdown-fixture', + shutdownDeadlineMs: Number(shutdownDeadlineMs), + createWorkspaceSessionAsync: () => Promise.resolve(new TemporaryRepoWorkspaceSession(repoRoot)), + requestResolver: new CallbackDaemonRequestResolver((options: IResolveDaemonRequestOptions) => + runStuckRequestAsync(options, controlFolder) + ), + onReady: (host) => writeJson(path.join(controlFolder, 'ready.json'), { paths: host.paths }), + onError: (error: Error) => process.stderr.write(`${error.stack ?? error.message}\n`) + }); +} + +if (require.main === module) { + void runAsync().catch((error: unknown) => { + process.stderr.write(`${error instanceof Error ? (error.stack ?? error.message) : String(error)}\n`); + process.exitCode = 1; + }); +} From ce12b8c91765ff2c76ae709f27e8e19c982a15ba Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:51:44 +0000 Subject: [PATCH 052/265] [rush-daemon] A hot request reuses the latest input capture that started after it was received Swarm integration step 37; original commit 7ab68d689d (merge of r05 at a5e20e7c20). Scope: task 137. Brings r05's task 137: in a hot burst each request ran its own full-workspace project-config capture (about 0.09 s each), so n=32 ran 32 of them back to back and held the event loop for 3.1-3.3 s (board 2101). A request now reuses the latest capture that started after the request was received. CONFIRMED by t01 board 2303. s16 batch A, item 2 of 5 (ch01 board 2867; the suites and mutants are in item 1's message). Gate: ch01 GATE OK board 2867 (tree 3b55869396) Commits folded into this step (1): - a5e20e7c20 [rush-daemon] Reuse the latest input capture that started after a request was received Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...in-succeeded-capture_2026-09-28-22-05.json | 10 ++ .../rush-daemon/src/FreshCaptureCoalescer.ts | 38 ++++- .../src/test/FreshCaptureCoalescer.test.ts | 147 +++++++++++++++++- .../test/WorkspaceInputCaptureReceipt.test.ts | 44 ++++++ 4 files changed, 226 insertions(+), 13 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-retain-succeeded-capture_2026-09-28-22-05.json diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-retain-succeeded-capture_2026-09-28-22-05.json b/common/changes/@rushstack/rush-daemon/swarm-r05-retain-succeeded-capture_2026-09-28-22-05.json new file mode 100644 index 0000000000..18b21e640e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-retain-succeeded-capture_2026-09-28-22-05.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Concurrent hot build requests share one project configuration capture again: a request now also reuses the latest capture that succeeded if it started after the daemon received the request.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/FreshCaptureCoalescer.ts b/libraries/rush-daemon/src/FreshCaptureCoalescer.ts index a9f4865264..efef5bfdad 100644 --- a/libraries/rush-daemon/src/FreshCaptureCoalescer.ts +++ b/libraries/rush-daemon/src/FreshCaptureCoalescer.ts @@ -7,6 +7,11 @@ interface ICapture { next: Promise | undefined; } +interface ISucceededCapture { + readonly startTimeMs: number; + readonly result: Promise; +} + /** * Options for {@link FreshCaptureCoalescer}. */ @@ -28,16 +33,21 @@ function ignore(): void {} * before a later caller arrived. A caller that finds a capture running therefore waits for the next capture, * which starts when the running one settles and is shared by every caller that arrived in the meantime. * Concurrent callers cost at most two captures instead of one each, and every caller receives a result that is - * at least as fresh as a capture it started itself. Nothing is retained once a capture settles. + * at least as fresh as a capture it started itself. * * A caller that knows an earlier time after which every change it depends on was made, such as the time at which * the daemon received its request, can pass that time as `notBeforeMs`. It then shares a running capture that - * started at or after that time instead of waiting for the next one. + * started at or after that time instead of waiting for the next one. It also receives the latest capture that + * succeeded, if that capture started at or after that time: a capture can finish before such a caller asks for + * it, for example when it awaits no I/O and so holds the event loop until it returns. The coalescer retains that + * one result for each scope and key until a capture that started later succeeds. A caller that does not pass + * `notBeforeMs` never receives a capture that has settled. * * Callers that pass the same scope and key must request the same capture. */ export class FreshCaptureCoalescer { readonly #captures: WeakMap>> = new WeakMap(); + readonly #succeeded: WeakMap>> = new WeakMap(); readonly #now: () => number; public constructor(options: IFreshCaptureCoalescerOptions = {}) { @@ -50,8 +60,9 @@ export class FreshCaptureCoalescer { * @param scope - Captures are shared only within a scope. * @param key - Identifies the capture within the scope. * @param captureAsync - Starts a capture when the caller cannot share one. - * @param notBeforeMs - A time on this coalescer's clock. A running capture that started at or after this time is - * shared with the caller. If it is not specified, the caller shares only a capture that starts after this call. + * @param notBeforeMs - A time on this coalescer's clock. A running capture, or the latest capture that succeeded, + * is shared with the caller if it started at or after this time. If it is not specified, the caller shares only a + * capture that starts after this call. */ public captureAsync( scope: TScope, @@ -59,6 +70,10 @@ export class FreshCaptureCoalescer { captureAsync: () => Promise, notBeforeMs?: number ): Promise { + if (notBeforeMs !== undefined) { + const succeeded: ISucceededCapture | undefined = this.#succeeded.get(scope)?.get(key); + if (succeeded && succeeded.startTimeMs >= notBeforeMs) return succeeded.result; + } const capture: ICapture | undefined = this.#captures.get(scope)?.get(key); if (!capture) return this.#start(scope, key, captureAsync); if (notBeforeMs !== undefined && capture.startTimeMs >= notBeforeMs) return capture.running; @@ -79,7 +94,20 @@ export class FreshCaptureCoalescer { const forget: () => void = () => { if (captures.get(key) === capture) captures.delete(key); }; - running.then(forget, forget); + const retain: () => void = () => { + let succeeded: Map> | undefined = this.#succeeded.get(scope); + if (!succeeded) { + succeeded = new Map(); + this.#succeeded.set(scope, succeeded); + } + // Captures of one key can overlap, so one that started earlier can succeed later. + const latest: ISucceededCapture | undefined = succeeded.get(key); + if (!latest || latest.startTimeMs <= startTimeMs) succeeded.set(key, { startTimeMs, result: running }); + }; + running.then(() => { + retain(); + forget(); + }, forget); return running; } } diff --git a/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts b/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts index fe2dbb66f6..3f58530c38 100644 --- a/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts +++ b/libraries/rush-daemon/src/test/FreshCaptureCoalescer.test.ts @@ -5,33 +5,40 @@ import { FreshCaptureCoalescer } from '../FreshCaptureCoalescer'; interface IStartedCapture { readonly index: number; + readonly startTimeMs: number; readonly resolve: (value: string) => void; readonly reject: (error: Error) => void; } +/** A clock that moves only when the test says so. */ +class ManualClock { + public nowMs: number = 0; + public readonly now = (): number => this.nowMs; +} + /** Captures that start in order and settle only when the test says so. */ class ControlledCaptures { public readonly started: IStartedCapture[] = []; + readonly #clock: ManualClock | undefined; + + public constructor(clock?: ManualClock) { + this.#clock = clock; + } public readonly captureAsync = (): Promise => { return new Promise((resolve, reject) => { - this.started.push({ index: this.started.length, resolve, reject }); + const startTimeMs: number = this.#clock?.nowMs ?? Number.NaN; + this.started.push({ index: this.started.length, startTimeMs, resolve, reject }); }); }; } -/** A clock that moves only when the test says so. */ -class ManualClock { - public nowMs: number = 0; - public readonly now = (): number => this.nowMs; -} - async function flushAsync(): Promise { await new Promise((resolve) => setImmediate(resolve)); } describe(FreshCaptureCoalescer.name, () => { - it('starts a capture for a caller when none is running and retains nothing once it settles', async () => { + it('starts a new capture for a caller without notBeforeMs once the previous capture settled', async () => { const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); const scope: object = {}; const captures: ControlledCaptures = new ControlledCaptures(); @@ -252,6 +259,121 @@ describe(FreshCaptureCoalescer.name, () => { expect(captures.started).toHaveLength(1); }); + it('gives a caller the latest capture that succeeded if it started at or after the caller\'s notBeforeMs', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 10); + captures.started[0].resolve('started at 10'); + await expect(first).resolves.toBe('started at 10'); + + // Callers received before the capture started that ask only after it settled, as they do when it held the + // event loop from start to end. + clock.nowMs = 20; + await expect(coalescer.captureAsync(scope, 'key', captures.captureAsync, 10)).resolves.toBe('started at 10'); + await expect(coalescer.captureAsync(scope, 'key', captures.captureAsync, 5)).resolves.toBe('started at 10'); + expect(captures.started).toHaveLength(1); + + const receivedAfterItStarted: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + 11 + ); + expect(captures.started).toHaveLength(2); + captures.started[1].resolve('started at 20'); + await expect(receivedAfterItStarted).resolves.toBe('started at 20'); + + // A capture that started later replaces the retained one. + await expect(coalescer.captureAsync(scope, 'key', captures.captureAsync, 11)).resolves.toBe('started at 20'); + await expect(coalescer.captureAsync(scope, 'key', captures.captureAsync, 5)).resolves.toBe('started at 20'); + expect(captures.started).toHaveLength(2); + }); + + it('never gives a caller without notBeforeMs a capture that has settled', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 10); + captures.started[0].resolve('started at 10'); + await expect(first).resolves.toBe('started at 10'); + + clock.nowMs = 20; + const strict: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + expect(captures.started).toHaveLength(2); + captures.started[1].resolve('started at 20'); + await expect(strict).resolves.toBe('started at 20'); + }); + + it('gives a caller the retained capture while a later one runs, and never retains a capture that failed', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ now: clock.now }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(); + + clock.nowMs = 10; + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync, 10); + captures.started[0].resolve('started at 10'); + await expect(first).resolves.toBe('started at 10'); + + clock.nowMs = 20; + const failed: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + // A caller received before the retained capture started doesn't wait for the running one. + await expect(coalescer.captureAsync(scope, 'key', captures.captureAsync, 5)).resolves.toBe('started at 10'); + captures.started[1].reject(new Error('a configuration file is being rewritten')); + await expect(failed).rejects.toThrow('a configuration file is being rewritten'); + + // The failed capture started at 20, but a caller received at 15 must not receive its failure. + clock.nowMs = 30; + const receivedAfterTheRetainedOneStarted: Promise = coalescer.captureAsync( + scope, + 'key', + captures.captureAsync, + 15 + ); + expect(captures.started).toHaveLength(3); + captures.started[2].resolve('started at 30'); + await expect(receivedAfterTheRetainedOneStarted).resolves.toBe('started at 30'); + }); + + it('retains the capture that started last when overlapping captures succeed in the opposite order', async () => { + const clock: ManualClock = new ManualClock(); + const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer({ + now: () => (clock.nowMs += 10) + }); + const scope: object = {}; + const captures: ControlledCaptures = new ControlledCaptures(clock); + + const first: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + const queued: Promise = coalescer.captureAsync(scope, 'key', captures.captureAsync); + // A caller that asks as soon as the first capture settles starts a capture before the queued one starts. + const asksOnSettle: Promise = first.then(() => + coalescer.captureAsync(scope, 'key', captures.captureAsync) + ); + captures.started[0].resolve('first'); + await flushAsync(); + expect(captures.started).toHaveLength(3); + + const [earlier, later] = captures.started.slice(1).sort((a, b) => a.startTimeMs - b.startTimeMs); + expect(earlier.startTimeMs).toBeLessThan(later.startTimeMs); + later.resolve(`started at ${later.startTimeMs}`); + await flushAsync(); + earlier.resolve(`started at ${earlier.startTimeMs}`); + const results: string[] = await Promise.all([queued, asksOnSettle]); + expect(results.sort()).toEqual([`started at ${earlier.startTimeMs}`, `started at ${later.startTimeMs}`]); + + await expect( + coalescer.captureAsync(scope, 'key', captures.captureAsync, later.startTimeMs) + ).resolves.toBe(`started at ${later.startTimeMs}`); + expect(captures.started).toHaveLength(3); + }); + it('compares notBeforeMs with performance.now() by default', async () => { const coalescer: FreshCaptureCoalescer = new FreshCaptureCoalescer(); const scope: object = {}; @@ -290,5 +412,14 @@ describe(FreshCaptureCoalescer.name, () => { expect(captures.started).toHaveLength(3); for (const capture of captures.started) capture.resolve(String(capture.index)); await expect(Promise.all(results)).resolves.toEqual(['0', '1', '2']); + + // Retained captures are not shared across scopes or keys either. + const retained: Promise[] = [ + coalescer.captureAsync(firstScope, 'c', captures.captureAsync, 0), + coalescer.captureAsync(secondScope, 'b', captures.captureAsync, 0) + ]; + expect(captures.started).toHaveLength(5); + for (const capture of captures.started.slice(3)) capture.resolve(String(capture.index)); + await expect(Promise.all(retained)).resolves.toEqual(['3', '4']); }); }); diff --git a/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts b/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts index 93132421a0..3b6fc4f616 100644 --- a/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceInputCaptureReceipt.test.ts @@ -80,6 +80,33 @@ function holdWorkspaceRequests(requests: jest.SpyInstance, count: number): void }); } +/** + * Makes the first `count` project configuration requests wait until all of them have arrived, and then makes them + * one at a time, each after the previous request's capture settled. A capture that awaits no I/O holds the event + * loop from start to end, so requests received together reach the coalescer in this order. + */ +function serializeProjectRequests(requests: jest.SpyInstance, count: number): void { + const waiting: (() => Promise)[] = []; + requests.mockImplementation(function ( + this: FreshCaptureCoalescer, + ...request: CaptureRequest + ): Promise { + if (waiting.length === count || !isProjectRequest(request)) return originalCaptureAsync.apply(this, request); + return new Promise((resolve, reject) => { + waiting.push(() => { + const result: Promise = originalCaptureAsync.apply(this, request); + result.then(resolve, reject); + return result; + }); + if (waiting.length === count) { + void (async () => { + for (const ask of waiting) await ask().catch(() => undefined); + })(); + } + }); + }); +} + function gateNextCapture( mock: jest.MockedFunction<(...args: TArgs) => Promise>, actual: (...args: TArgs) => Promise @@ -172,6 +199,23 @@ describe('workspace input captures shared from the time a request was received', expect(warm.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); }); + it('gives warm builds the project configuration capture that started after they were received, even once it finished', async () => { + fixture = await createWarmFixtureAsync(); + const warm: DaemonGraphTestFixture = fixture; + const generation: number = warm.host.workspaceGeneration; + const spy: jest.SpyInstance = spyOnRequests(); + serializeProjectRequests(spy, 3); + projectCaptureMock.mockClear(); + + const builds: BuildExchange[] = [warm.buildAsync(), warm.buildAsync(), warm.buildAsync()]; + await expectSuccessfulBuildsAsync(builds); + expect(countRequests(spy, 'project')).toBe(3); + // Each build asked after the previous capture had finished; all of them were received before it started. + expect(projectCaptureMock).toHaveBeenCalledTimes(1); + expect(warm.host.workspaceGeneration).toBe(generation); + expect(warm.host.workspaceStatus.lastReloadTier).toBe(rushLib.WorkspaceInputChangeTier.Reuse); + }); + it('never shares a capture that started before a build was received', async () => { fixture = await createWarmFixtureAsync(); const warm: DaemonGraphTestFixture = fixture; From d5b3f82acd68bfed9eb8f7b9141004e6007fbe5a Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:51:44 +0000 Subject: [PATCH 053/265] [rush-azure-storage-build-cache-plugin] The Azure build cache picks up a credential written after the daemon's first cloud read Swarm integration step 38; original commit 0f5e259316 (merge of swarm/r03-t101 at f4a425ada3). Scope: task 101. Brings r03's task 101: the Azure storage build cache provider kept its first container client for the daemon's lifetime, so a credential written later was never used (ch05 board 1202, ch02 board 1229). It now uses the credential cached after the first cloud request. CONFIRMED by ch05 board 2290. s16 batch A, item 3 of 5 (ch01 board 2867). Gate: ch01 GATE OK board 2867 (tree 38dd18ec39) Commits folded into this step (1): - f4a425ada3 [rush-azure-storage-build-cache-plugin] Use a credential that is cached after the first cloud request (task 101) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...e-credential-refresh_2026-09-28-21-50.json | 11 + .../src/AzureStorageBuildCacheProvider.ts | 107 +++++-- .../AzureStorageBuildCacheProvider.test.ts | 265 +++++++++++++++++- 3 files changed, 353 insertions(+), 30 deletions(-) create mode 100644 common/changes/@microsoft/rush/azure-cache-credential-refresh_2026-09-28-21-50.json diff --git a/common/changes/@microsoft/rush/azure-cache-credential-refresh_2026-09-28-21-50.json b/common/changes/@microsoft/rush/azure-cache-credential-refresh_2026-09-28-21-50.json new file mode 100644 index 0000000000..7da2e2585a --- /dev/null +++ b/common/changes/@microsoft/rush/azure-cache-credential-refresh_2026-09-28-21-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Fix an issue where the Azure Storage build cache plugin kept the client from its first cloud cache request for the life of the process, so that a long-lived process such as the Rush daemon did not use a credential that was cached later. An anonymous client is now used only until a credential is cached, and a client whose credential Azure Storage rejects with status 401 or 403 is replaced on the next request.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageBuildCacheProvider.ts b/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageBuildCacheProvider.ts index c31205778e..18186b9e73 100644 --- a/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageBuildCacheProvider.ts +++ b/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageBuildCacheProvider.ts @@ -42,6 +42,11 @@ interface IBlobError extends Error { }; } +interface IKeptContainerClient { + client: ContainerClient; + isAnonymous: boolean; +} + export class AzureStorageBuildCacheProvider extends AzureStorageAuthentication implements ICloudBuildCacheProvider @@ -54,7 +59,10 @@ export class AzureStorageBuildCacheProvider return EnvironmentConfiguration.buildCacheWriteAllowed ?? this._isCacheWriteAllowedByConfiguration; } - #containerClient: ContainerClient | undefined; + // A long-lived process, such as the Rush daemon, keeps this provider across builds. So a client made + // from a credential is kept only until Azure Storage rejects it, and an anonymous client only until + // a credential is cached. + #keptContainerClient: IKeptContainerClient | undefined; public constructor(options: IAzureStorageBuildCacheProviderOptions) { super({ @@ -131,7 +139,8 @@ export class AzureStorageBuildCacheProvider cacheId: string, getBlobDataAsync: (blobClient: BlobClient) => Promise ): Promise { - const blobClient: BlobClient = await this.#getBlobClientForCacheIdAsync(cacheId, terminal); + const containerClient: ContainerClient = await this.#getContainerClientAsync(terminal); + const blobClient: BlobClient = this.#getBlobClient(containerClient, cacheId); try { const blobExists: boolean = await blobClient.exists(); @@ -142,6 +151,7 @@ export class AzureStorageBuildCacheProvider } } catch (err) { this.#logBlobError(terminal, err, 'Error getting cache entry from Azure Storage: '); + this.#forgetRejectedClient(containerClient, err); return undefined; } } @@ -163,7 +173,8 @@ export class AzureStorageBuildCacheProvider return false; } - const blobClient: BlobClient = await this.#getBlobClientForCacheIdAsync(cacheId, terminal); + const containerClient: ContainerClient = await this.#getContainerClientAsync(terminal); + const blobClient: BlobClient = this.#getBlobClient(containerClient, cacheId); const blockBlobClient: BlockBlobClient = blobClient.getBlockBlobClient(); let blobAlreadyExists: boolean = false; @@ -183,6 +194,7 @@ export class AzureStorageBuildCacheProvider .join(' '); terminal.writeWarningLine(errorMessage); + this.#forgetRejectedClient(containerClient, err); } if (blobAlreadyExists) { @@ -204,16 +216,32 @@ export class AzureStorageBuildCacheProvider return true; } else { terminal.writeWarningLine(`Error uploading cache entry to Azure Storage: ${e}`); + this.#forgetRejectedClient(containerClient, e); return false; } } } } - async #getBlobClientForCacheIdAsync(cacheId: string, terminal: ITerminal): Promise { - const client: ContainerClient = await this.#getContainerClientAsync(terminal); + #getBlobClient(containerClient: ContainerClient, cacheId: string): BlobClient { const blobName: string = this.#blobPrefix ? `${this.#blobPrefix}/${cacheId}` : cacheId; - return client.getBlobClient(blobName); + return containerClient.getBlobClient(blobName); + } + + /** + * Azure Storage answers 401 or 403 when it rejects a credential, for example once it has expired. + * Forget the client that was made from it, so that the next request reads the cached credential again. + */ + #forgetRejectedClient(containerClient: ContainerClient, error: unknown): void { + const statusCode: number | undefined = (error as IBlobError | undefined)?.statusCode; + const keptClient: IKeptContainerClient | undefined = this.#keptContainerClient; + if ( + (statusCode === 401 || statusCode === 403) && + keptClient?.client === containerClient && + !keptClient.isAnonymous + ) { + this.#keptContainerClient = undefined; + } } #logBlobError(terminal: ITerminal, err: unknown, prefix: string): void { @@ -253,37 +281,58 @@ export class AzureStorageBuildCacheProvider } async #getContainerClientAsync(terminal: ITerminal): Promise { - if (!this.#containerClient) { - let sasString: string | undefined = this.#environmentCredential; - if (!sasString) { - const credentialEntry: ICredentialCacheEntry | undefined = await this.tryGetCachedCredentialAsync({ + const keptClient: IKeptContainerClient | undefined = this.#keptContainerClient; + if (keptClient && !keptClient.isAnonymous) { + return keptClient.client; + } + + let sasString: string | undefined = this.#environmentCredential; + if (!sasString) { + let credentialEntry: ICredentialCacheEntry | undefined; + if (keptClient) { + // While the kept client is anonymous, look for a credential on each request, and don't repeat + // the warning about an expired one. If the file can't be read, for example while it is being + // written, keep using the anonymous client rather than failing the request. + try { + credentialEntry = await this.tryGetCachedCredentialAsync({ expiredCredentialBehavior: 'ignore' }); + } catch { + terminal.writeVerboseLine( + "Couldn't read the cached Azure Storage credentials. Using the build cache without them." + ); + } + } else { + credentialEntry = await this.tryGetCachedCredentialAsync({ expiredCredentialBehavior: 'logWarning', terminal }); - - sasString = credentialEntry?.credential; } - let blobServiceClient: BlobServiceClient; - if (sasString) { - const connectionString: string = this.#getConnectionString(sasString); - blobServiceClient = BlobServiceClient.fromConnectionString(connectionString); - } else if (!this.#readRequiresAuthentication && !this._isCacheWriteAllowedByConfiguration) { - // If we don't have a credential and read doesn't require authentication, we can still read from the cache. - blobServiceClient = new BlobServiceClient(this._storageAccountUrl); - } else { - throw new Error( - "An Azure Storage SAS credential hasn't been provided, or has expired. " + - `Update the credentials by running "rush ${RushConstants.updateCloudCredentialsCommandName}", ` + - `or provide a SAS in the ` + - `${EnvironmentVariableNames.RUSH_BUILD_CACHE_CREDENTIAL} environment variable` - ); - } + sasString = credentialEntry?.credential; + } + + if (keptClient && !sasString) { + return keptClient.client; + } - this.#containerClient = blobServiceClient.getContainerClient(this._storageContainerName); + let blobServiceClient: BlobServiceClient; + if (sasString) { + const connectionString: string = this.#getConnectionString(sasString); + blobServiceClient = BlobServiceClient.fromConnectionString(connectionString); + } else if (!this.#readRequiresAuthentication && !this._isCacheWriteAllowedByConfiguration) { + // If we don't have a credential and read doesn't require authentication, we can still read from the cache. + blobServiceClient = new BlobServiceClient(this._storageAccountUrl); + } else { + throw new Error( + "An Azure Storage SAS credential hasn't been provided, or has expired. " + + `Update the credentials by running "rush ${RushConstants.updateCloudCredentialsCommandName}", ` + + `or provide a SAS in the ` + + `${EnvironmentVariableNames.RUSH_BUILD_CACHE_CREDENTIAL} environment variable` + ); } - return this.#containerClient; + const containerClient: ContainerClient = blobServiceClient.getContainerClient(this._storageContainerName); + this.#keptContainerClient = { client: containerClient, isAnonymous: !sasString }; + return containerClient; } #getConnectionString(sasString: string | undefined): string { diff --git a/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts b/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts index 25cb7d50bc..3b5e5e0b28 100644 --- a/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts +++ b/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts @@ -1,7 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { CredentialCache } from '@rushstack/credential-cache'; +import { BlobServiceClient, type BlockBlobClient, type ContainerClient } from '@azure/storage-blob'; + +import { CredentialCache, type ICredentialCacheEntry } from '@rushstack/credential-cache'; import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import { EnvironmentConfiguration, RushUserConfiguration } from '@rushstack/rush-sdk'; @@ -133,4 +135,265 @@ describe(AzureStorageBuildCacheProvider.name, () => { it('Has an expected cached credential name (write allowed)', async () => { await testCredentialCache(true); }); + + describe('a credential that changes while the provider is in use', () => { + const FIRST_SAS: string = 'sv=2025-01-05&sp=rcw&sig=first-secret-signature'; + const SECOND_SAS: string = 'sv=2025-01-05&sp=rcw&sig=second-secret-signature'; + const SECRETS: string[] = ['first-secret-signature', 'second-secret-signature']; + + type BlobOperation = 'exists' | 'download' | 'upload'; + // 'hit' and 'miss' answer an existence check; a number is the status code of an error + type FakeResponse = 'hit' | 'miss' | 'ok' | number; + + let cachedCredential: ICredentialCacheEntry | undefined; + let credentialReads: number; + let requests: string[]; + let respond: (credentialName: string, operation: BlobOperation) => FakeResponse | Promise; + let terminalProvider: StringBufferTerminalProvider; + let terminal: Terminal; + + function getCredentialName(sasString: string | undefined): string { + return sasString === FIRST_SAS ? 'first' : sasString === SECOND_SAS ? 'second' : 'anonymous'; + } + + function createBlobError(statusCode: number): Error { + const errorCode: string = `Status${statusCode}ErrorCode`; + return Object.assign(new Error(`The request failed with status ${statusCode}.`), { + name: 'RestError', + statusCode, + code: errorCode, + response: { status: statusCode, parsedHeaders: { errorCode } } + }); + } + + function createFakeContainerClient(sasString: string | undefined): ContainerClient { + const credentialName: string = getCredentialName(sasString); + async function requestAsync(operation: BlobOperation): Promise { + requests.push(`${operation} ${credentialName}`); + const response: FakeResponse = await respond(credentialName, operation); + if (typeof response === 'number') { + throw createBlobError(response); + } + return response; + } + + const blobClient: Partial = { + exists: () => requestAsync('exists').then((response: FakeResponse) => response === 'hit'), + downloadToBuffer: (async () => { + await requestAsync('download'); + return Buffer.from(`read with ${credentialName}`); + }) as BlockBlobClient['downloadToBuffer'], + upload: (async () => { + await requestAsync('upload'); + }) as unknown as BlockBlobClient['upload'], + getBlockBlobClient: () => blobClient as BlockBlobClient + }; + return { getBlobClient: () => blobClient } as unknown as ContainerClient; + } + + function createSubject(isCacheWriteAllowed: boolean = false): AzureStorageBuildCacheProvider { + return new AzureStorageBuildCacheProvider({ + storageAccountName: 'storage-account', + storageContainerName: 'container-name', + isCacheWriteAllowed + }); + } + + beforeEach(() => { + cachedCredential = undefined; + credentialReads = 0; + requests = []; + terminalProvider = new StringBufferTerminalProvider(); + terminal = new Terminal(terminalProvider); + + jest.spyOn(CredentialCache, 'usingAsync').mockImplementation(async (options, doActionAsync) => { + credentialReads++; + await doActionAsync({ tryGetCacheEntry: () => cachedCredential } as unknown as CredentialCache); + }); + jest.spyOn(BlobServiceClient, 'fromConnectionString').mockImplementation( + (connectionString: string) => + ({ + getContainerClient: () => + createFakeContainerClient(connectionString.split('SharedAccessSignature=')[1]) + }) as unknown as BlobServiceClient + ); + jest + .spyOn(BlobServiceClient.prototype, 'getContainerClient') + .mockImplementation(() => createFakeContainerClient(undefined)); + }); + + afterEach(() => { + const allOutput: string = JSON.stringify(terminalProvider.getAllOutput()); + for (const secret of SECRETS) { + expect(allOutput).not.toContain(secret); + } + + jest.restoreAllMocks(); + }); + + it.each([ + { anonymousResult: 'a miss', anonymousResponse: 'miss' as const }, + { anonymousResult: '401', anonymousResponse: 401 } + ])( + 'uses a credential that was cached after an anonymous read got $anonymousResult', + async ({ anonymousResponse }) => { + const subject: AzureStorageBuildCacheProvider = createSubject(); + respond = (credentialName: string) => (credentialName === 'anonymous' ? anonymousResponse : 'hit'); + + expect(await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id')).toBeUndefined(); + cachedCredential = { credential: FIRST_SAS }; + const entry: Buffer | undefined = await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + + expect(entry?.toString()).toBe('read with first'); + expect(requests).toEqual(['exists anonymous', 'exists first', 'download first']); + expect(credentialReads).toBe(2); + } + ); + + it.each([ + { anonymousResult: 'a miss', anonymousResponse: 'miss' as const }, + { anonymousResult: '401', anonymousResponse: 401 } + ])( + 'reuses the anonymous client and warns once about an expired credential when reads get $anonymousResult', + async ({ anonymousResponse }) => { + const subject: AzureStorageBuildCacheProvider = createSubject(); + respond = () => anonymousResponse; + cachedCredential = { credential: FIRST_SAS, expires: new Date(Date.now() - 60_000) }; + + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + + expect(requests).toEqual(['exists anonymous', 'exists anonymous']); + expect(credentialReads).toBe(2); + expect(BlobServiceClient.prototype.getContainerClient).toHaveBeenCalledTimes(1); + expect(terminalProvider.getWarningOutput().match(/have expired/g)).toHaveLength(1); + } + ); + + it("keeps using the anonymous client when the cached credentials can't be read", async () => { + const subject: AzureStorageBuildCacheProvider = createSubject(); + respond = () => 'miss'; + + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + jest.mocked(CredentialCache.usingAsync).mockRejectedValueOnce(new Error('Unexpected end of input')); + expect(await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id')).toBeUndefined(); + cachedCredential = { credential: FIRST_SAS }; + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + + expect(requests).toEqual(['exists anonymous', 'exists anonymous', 'exists first']); + expect(BlobServiceClient.prototype.getContainerClient).toHaveBeenCalledTimes(1); + expect(terminalProvider.getWarningOutput()).toBe(''); + expect(terminalProvider.getErrorOutput()).toBe(''); + }); + + it('keeps a newer client when an older one is rejected late', async () => { + const subject: AzureStorageBuildCacheProvider = createSubject(); + let onLateRequest!: () => void; + const lateRequestSent: Promise = new Promise((resolve) => { + onLateRequest = resolve; + }); + let answerLateRequest!: () => void; + const lateResponse: Promise = new Promise((resolve) => { + answerLateRequest = () => resolve(403); + }); + let firstRequests: number = 0; + respond = (credentialName: string) => { + if (credentialName !== 'first') { + return 'hit'; + } else if (firstRequests++ === 0) { + onLateRequest(); + return lateResponse; + } else { + return 403; + } + }; + cachedCredential = { credential: FIRST_SAS }; + + const lateRead: Promise = subject.tryGetCacheEntryBufferByIdAsync( + terminal, + 'cache-id' + ); + await lateRequestSent; + expect(await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id')).toBeUndefined(); + cachedCredential = { credential: SECOND_SAS }; + expect((await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'))?.toString()).toBe( + 'read with second' + ); + answerLateRequest(); + expect(await lateRead).toBeUndefined(); + expect((await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'))?.toString()).toBe( + 'read with second' + ); + + expect(credentialReads).toBe(2); + }); + + it('keeps a client made from a credential while Azure Storage accepts it, or fails with another status', async () => { + const subject: AzureStorageBuildCacheProvider = createSubject(); + let failNextRequest: boolean = false; + respond = () => (failNextRequest ? 500 : 'hit'); + cachedCredential = { credential: FIRST_SAS }; + + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + failNextRequest = true; + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + failNextRequest = false; + cachedCredential = { credential: SECOND_SAS }; + await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + + expect(requests).toEqual([ + 'exists first', + 'download first', + 'exists first', + 'exists first', + 'download first' + ]); + expect(credentialReads).toBe(1); + }); + + it.each([401, 403])( + 'reads the cached credential again after a read was rejected with %i', + async (statusCode: number) => { + const subject: AzureStorageBuildCacheProvider = createSubject(); + respond = (credentialName: string) => (credentialName === 'first' ? statusCode : 'hit'); + cachedCredential = { credential: FIRST_SAS }; + + expect(await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id')).toBeUndefined(); + cachedCredential = { credential: SECOND_SAS }; + const entry: Buffer | undefined = await subject.tryGetCacheEntryBufferByIdAsync(terminal, 'cache-id'); + + expect(entry?.toString()).toBe('read with second'); + expect(requests).toEqual(['exists first', 'exists second', 'download second']); + expect(credentialReads).toBe(2); + } + ); + + it.each([ + { rejectedOperation: 'upload' as const, expectedResult: false }, + { rejectedOperation: 'exists' as const, expectedResult: true } + ])( + 'reads the cached credential again after the $rejectedOperation request of a write was rejected', + async ({ rejectedOperation, expectedResult }) => { + const subject: AzureStorageBuildCacheProvider = createSubject(true); + respond = (credentialName: string, operation: BlobOperation) => + credentialName === 'first' && operation === rejectedOperation + ? 403 + : operation === 'exists' + ? 'miss' + : 'ok'; + cachedCredential = { credential: FIRST_SAS }; + + expect(await subject.trySetCacheEntryBufferAsync(terminal, 'cache-id', Buffer.from('entry'))).toBe( + expectedResult + ); + cachedCredential = { credential: SECOND_SAS }; + expect(await subject.trySetCacheEntryBufferAsync(terminal, 'cache-id', Buffer.from('entry'))).toBe( + true + ); + + expect(requests).toEqual(['exists first', 'upload first', 'exists second', 'upload second']); + expect(credentialReads).toBe(2); + } + ); + }); }); From 02e9125d1497319a0ce980df3d0c3e9a7df74a20 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:51:44 +0000 Subject: [PATCH 054/265] [rush-lib] build-cache.json accepts azureBlobStorageConfiguration.storageEndpoint Swarm integration step 39; original commit ac2e51d2d2 (merge of swarm/r03-t144 at 04c7af3fbb). Scope: task 144. Brings r03's task 144: rush-lib's build-cache.schema.json rejected azureBlobStorageConfiguration.storageEndpoint, which the Azure plugin has read since microsoft/rushstack PR 5664. CONFIRMED by ch05 board 2396. s16 batch A, item 4 of 5 (ch01 board 2867). Gate: ch01 GATE OK board 2867 (tree 0fbdb27c44) Commits folded into this step (1): - 04c7af3fbb [rush-lib] Allow storageEndpoint in build-cache.json's azureBlobStorageConfiguration (task 144) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...rage-endpoint-schema_2026-09-28-22-28.json | 11 ++ .../common/config/rush/build-cache.json | 7 ++ .../api/test/BuildCacheConfiguration.test.ts | 102 ++++++++++++++++++ .../src/schemas/build-cache.schema.json | 5 + 4 files changed, 125 insertions(+) create mode 100644 common/changes/@microsoft/rush/azure-storage-endpoint-schema_2026-09-28-22-28.json create mode 100644 libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts diff --git a/common/changes/@microsoft/rush/azure-storage-endpoint-schema_2026-09-28-22-28.json b/common/changes/@microsoft/rush/azure-storage-endpoint-schema_2026-09-28-22-28.json new file mode 100644 index 0000000000..14abd8d0f7 --- /dev/null +++ b/common/changes/@microsoft/rush/azure-storage-endpoint-schema_2026-09-28-22-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Allow `storageEndpoint` in the `azureBlobStorageConfiguration` section of build-cache.json. The Azure Storage build cache plugin reads this setting, but the build-cache.json schema rejected it.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json b/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json index 072e9f7d49..b9bed9937c 100644 --- a/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json +++ b/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json @@ -60,6 +60,13 @@ */ // "azureEnvironment": "AzurePublicCloud", + /** + * An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, + * private endpoints, or storage emulators. When specified, this overrides the default endpoint derived + * from "storageAccountName". + */ + // "storageEndpoint": "http://127.0.0.1:10000/devstoreaccount1", + /** * An optional prefix for cache item blob names. */ diff --git a/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts b/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts new file mode 100644 index 0000000000..189a1747d8 --- /dev/null +++ b/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts @@ -0,0 +1,102 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; + +import { BuildCacheConfiguration } from '../BuildCacheConfiguration'; +import { EnvironmentConfiguration, EnvironmentVariableNames } from '../EnvironmentConfiguration'; +import { RushConfiguration } from '../RushConfiguration'; +import { RushSession, type CloudBuildCacheProviderFactory } from '../../pluginFramework/RushSession'; +import type { ICloudBuildCacheProvider } from '../../logic/buildCache/ICloudBuildCacheProvider'; + +type FactoryMock = jest.Mock< + ReturnType, + Parameters +>; + +const OVERRIDE_JSON_VARIABLE: string = EnvironmentVariableNames.RUSH_BUILD_CACHE_OVERRIDE_JSON; +const AZURE_CACHE_PROVIDER: string = 'azure-blob-storage'; +const STORAGE_ENDPOINT: string = 'http://127.0.0.1:10000/devstoreaccount1'; + +describe(BuildCacheConfiguration.name, () => { + const originalOverrideJson: string | undefined = process.env[OVERRIDE_JSON_VARIABLE]; + let rushConfiguration: RushConfiguration; + let factory: FactoryMock; + + beforeAll(() => { + rushConfiguration = RushConfiguration.loadFromConfigurationFile(`${__dirname}/repo/rush-npm.json`); + }); + + beforeEach(() => { + const cloudCacheProvider: ICloudBuildCacheProvider = {} as ICloudBuildCacheProvider; + factory = jest.fn, Parameters>( + () => cloudCacheProvider + ); + }); + + afterEach(() => { + if (originalOverrideJson === undefined) { + delete process.env[OVERRIDE_JSON_VARIABLE]; + } else { + process.env[OVERRIDE_JSON_VARIABLE] = originalOverrideJson; + } + EnvironmentConfiguration.reset(); + }); + + async function tryLoadAzureConfigurationAsync( + azureBlobStorageConfiguration: Record + ): Promise { + process.env[OVERRIDE_JSON_VARIABLE] = JSON.stringify({ + buildCacheEnabled: true, + cacheProvider: AZURE_CACHE_PROVIDER, + azureBlobStorageConfiguration + }); + EnvironmentConfiguration.reset(); + + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const rushSession: RushSession = new RushSession({ terminalProvider, getIsDebugMode: () => false }); + rushSession.registerCloudBuildCacheProviderFactory(AZURE_CACHE_PROVIDER, factory); + + return await BuildCacheConfiguration.tryLoadAsync( + new Terminal(terminalProvider), + rushConfiguration, + rushSession + ); + } + + it('passes azureBlobStorageConfiguration.storageEndpoint to the cloud cache provider', async () => { + const configuration: BuildCacheConfiguration | undefined = await tryLoadAzureConfigurationAsync({ + storageAccountName: 'example', + storageContainerName: 'build-cache', + storageEndpoint: STORAGE_ENDPOINT + }); + + expect(configuration?.buildCacheEnabled).toBe(true); + expect(factory).toHaveBeenCalledTimes(1); + expect(factory.mock.calls[0][0]).toMatchObject({ + azureBlobStorageConfiguration: { storageEndpoint: STORAGE_ENDPOINT } + }); + }); + + it('rejects a property that azureBlobStorageConfiguration does not define', async () => { + await expect( + tryLoadAzureConfigurationAsync({ + storageAccountName: 'example', + storageContainerName: 'build-cache', + storageEndpointUrl: STORAGE_ENDPOINT + }) + ).rejects.toThrow(/must NOT have additional properties: storageEndpointUrl/); + expect(factory).not.toHaveBeenCalled(); + }); + + it('rejects a storageEndpoint that is not a URI', async () => { + await expect( + tryLoadAzureConfigurationAsync({ + storageAccountName: 'example', + storageContainerName: 'build-cache', + storageEndpoint: '127.0.0.1 port 10000' + }) + ).rejects.toThrow(/#\/azureBlobStorageConfiguration\/storageEndpoint\s+must match format "uri"/); + expect(factory).not.toHaveBeenCalled(); + }); +}); diff --git a/libraries/rush-lib/src/schemas/build-cache.schema.json b/libraries/rush-lib/src/schemas/build-cache.schema.json index 2c7e8fd696..8029a685a6 100644 --- a/libraries/rush-lib/src/schemas/build-cache.schema.json +++ b/libraries/rush-lib/src/schemas/build-cache.schema.json @@ -71,6 +71,11 @@ "description": "The Azure environment the storage account exists in. Defaults to AzurePublicCloud.", "enum": ["AzurePublicCloud", "AzureChina", "AzureGermany", "AzureGovernment"] }, + "storageEndpoint": { + "type": "string", + "description": "An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, private endpoints, or storage emulators. When specified, this overrides the default endpoint derived from storageAccountName. Example: \"http://127.0.0.1:10000/devstoreaccount1\" or \"https://my-proxy.example.com/devstoreaccount1\"", + "format": "uri" + }, "loginFlow": { "$ref": "#/definitions/entraLoginFlow" }, From 3c0d3d993651f97d1330fa6ca69ac1d91565032d Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 01:51:44 +0000 Subject: [PATCH 055/265] [rush-daemon] A lingering daemon logs its PID and stop time, without a stack Swarm integration step 40; original commit 86fc1ff37e (merge of swarm/r01-t147-nit at 7a76edd6ef). Scope: task 147 NITs. Brings r01's fix for the NITs in t05 board 2608 and o04 board 2632 (board 2814): the report is a timestamped log line through onLog, not an error with frames. t05 read the delta (board 2832), and r01's run through the real entry point is board 2851. ch01 on the final tree: build rc 0 with 0 warnings, rush-daemon 629/0, 7 of 7 mutants killed. s16 batch A, item 5 of 5 (ch01 board 2867). Gate: ch01 GATE OK board 2867 (tree d3a1eb9dbc) Commits folded into this step (1): - 7a76edd6ef rushd: a lingering daemon logs its PID and stop time, without a stack (task 147 NITs) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../r01-exit-after-stop_2026-09-28-23-45.json | 2 +- libraries/rush-daemon/src/serveRushDaemon.ts | 36 ++++++--- .../src/test/DaemonProcessExit.test.ts | 77 ++++++++++++++----- .../src/test/fixtures/LingeringDaemon.ts | 35 ++++++--- 4 files changed, 109 insertions(+), 41 deletions(-) diff --git a/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json b/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json index 9596f945ac..3439a48aaf 100644 --- a/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json +++ b/common/changes/@rushstack/rush-daemon/r01-exit-after-stop_2026-09-28-23-45.json @@ -2,7 +2,7 @@ "changes": [ { "packageName": "@rushstack/rush-daemon", - "comment": "A daemon that owns its process now exits 2 seconds after it stops if something else, such as a timer that a plugin left behind, keeps the process running after the daemon released its socket and lockfile. It reports the active resources that Node.js lists and keeps `process.exitCode`. Reports that come right before the daemon exits are now written synchronously when no `onError` callback is given.", + "comment": "A daemon that owns its process now exits 2 seconds after it stops if something else, such as a timer that a plugin left behind, keeps the process running after the daemon released its socket and lockfile. It writes its PID, the time it stopped and the active resources that Node.js lists to the daemon log (`onLog`), and keeps `process.exitCode`. It does not wait for uploads that a plugin's `flushTelemetry` hook started, so an upload that is still running then is cut off. An embedded daemon (`shutdownSignal`) never exits the process. Messages that come right before the daemon exits are now written synchronously when no `onError` or `onLog` callback is given.", "type": "patch" } ], diff --git a/libraries/rush-daemon/src/serveRushDaemon.ts b/libraries/rush-daemon/src/serveRushDaemon.ts index f47ff7c194..bc3e2d4781 100644 --- a/libraries/rush-daemon/src/serveRushDaemon.ts +++ b/libraries/rush-daemon/src/serveRushDaemon.ts @@ -24,7 +24,9 @@ export interface IRushDaemonServeOptions extends IRushDaemonHostOptions { * clean shutdown, and {@link IRushDaemonHostOptions.shutdownDeadlineMs} defaults to 10 seconds. If the shutdown * does not finish by then, or another signal arrives first, the daemon reports why, releases what it safely can * and exits the process with code 1. Once it stops, if something else keeps the process running for 2 seconds, - * it reports the active resources that Node.js lists and exits the process, keeping `process.exitCode`. + * it writes its PID, when it stopped and the active resources that Node.js lists to the daemon log + * ({@link IRushDaemonHostOptions.onLog}, or stderr without it) and exits the process, keeping + * `process.exitCode`. An embedded daemon never exits the process. */ readonly shutdownSignal?: AbortSignal; } @@ -138,17 +140,18 @@ function exitAfterShutdownDeadline( /** * Exits the process {@link EXIT_AFTER_STOP_MS} after the daemon stopped serving, if it is still running then, and - * reports what kept it running. The timer does not keep the process running itself. `process.exit()` keeps an exit + * logs what kept it running. The timer does not keep the process running itself. `process.exit()` keeps an exit * code that the caller set, and the 'exit' hook of SubprocessTerminator kills the child processes that it still * tracks. A successor daemon is not one of them: it is started detached and untracked. */ function exitIfStillRunning(options: IRushDaemonServeOptions): void { + const stoppedAt: string = new Date().toISOString(); setTimeout(() => { - reportBeforeExit( - new Error( - `The Rush daemon stopped, but something kept its process running for ${EXIT_AFTER_STOP_MS / 1000} s, ` + - `so it exits now. Active resources that Node.js reports: ${describeActiveResources()}.` - ), + // Not a failure, so no stack. The log is shared by every daemon of the workspace, hence the PID and the time. + logBeforeExit( + `rushd (PID ${process.pid}) stopped at ${stoppedAt}, but something kept its process running for ` + + `${EXIT_AFTER_STOP_MS / 1000} s, so it exits now. Active resources that Node.js reports: ` + + `${describeActiveResources()}.`, options ); process.exit(); @@ -168,16 +171,29 @@ function describeActiveResources(): string { } /** - * Reports an error right before `process.exit()`. The report must be written synchronously: `process.emitWarning` - * prints on the next tick, which never comes. + * Reports an error right before `process.exit()`. Without `onError`, the report is written synchronously: + * `process.emitWarning` prints on the next tick, which never comes. */ function reportBeforeExit(report: Error, options: IRushDaemonServeOptions): void { if (options.onError) { options.onError(report); return; } + writeToStderrBeforeExit(report.stack ?? report.message); +} + +/** Writes a message for the daemon log right before `process.exit()`, synchronously without `onLog`. */ +function logBeforeExit(message: string, options: IRushDaemonServeOptions): void { + if (options.onLog) { + options.onLog(message); + return; + } + writeToStderrBeforeExit(message); +} + +function writeToStderrBeforeExit(text: string): void { try { - fs.writeSync(process.stderr.fd, `${report.stack ?? report.message}\n`); + fs.writeSync(process.stderr.fd, `${text}\n`); } catch { // The process exits either way. } diff --git a/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts b/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts index f05a509f7f..4e8a61ade3 100644 --- a/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts +++ b/libraries/rush-daemon/src/test/DaemonProcessExit.test.ts @@ -10,7 +10,7 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; import { readDaemonLockfile, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; import { DaemonRequestWireClient } from './DaemonRequestWireTestUtilities'; -import type { LeftBehind, Reporter } from './fixtures/LingeringDaemon'; +import type { LeftBehind, Ownership, Reporter } from './fixtures/LingeringDaemon'; import { createTemporaryRepo } from './TemporaryRepoWorkspaceSession'; const FIXTURE_PATH: string = path.join(__dirname, 'fixtures', 'LingeringDaemon.js'); @@ -18,8 +18,10 @@ const POLL_INTERVAL_MS: number = 20; const START_TIMEOUT_MS: number = 15000; // Far beyond the 2 s after which the daemon exits anyway, and far below the test's timeout. const EXIT_TIMEOUT_MS: number = 8000; +// Well past the 2 s after which a daemon that owns its process exits it. +const EMBEDDED_WAIT_MS: number = 3000; const LINGER_REPORT: RegExp = - /The Rush daemon stopped, but something kept its process running for 2 s, so it exits now\. Active resources that Node\.js reports: [^\n]*\bTimeout\b/; + /rushd \(PID (\d+)\) stopped at (\S+), but something kept its process running for 2 s, so it exits now\. Active resources that Node\.js reports: [^\n]*\bTimeout\b/; interface IProcessExit { readonly code: number | undefined; @@ -64,10 +66,10 @@ type StopRoute = 'daemon stop' | 'SIGTERM'; fs.rmSync(folder, { force: true, recursive: true }); }); - function startDaemon(leftBehind: LeftBehind, reporter: Reporter): IFixtureDaemon { + function startDaemon(leftBehind: LeftBehind, reporter: Reporter, ownership: Ownership): IFixtureDaemon { const child: ChildProcess = spawn( process.execPath, - [FIXTURE_PATH, repoRoot, controlFolder, leftBehind, reporter], + [FIXTURE_PATH, repoRoot, controlFolder, leftBehind, reporter, ownership], { // Like a daemon that rush-client launches: the leader of its own process group. detached: true, @@ -115,8 +117,10 @@ type StopRoute = 'daemon stop' | 'SIGTERM'; } } - /** Resolves with the exit and how long after the daemon stopped it came. */ - async function waitForExitAsync(fixture: IFixtureDaemon): Promise { + /** Resolves with the exit, when the daemon stopped and how long after that the exit came. */ + async function waitForExitAsync( + fixture: IFixtureDaemon + ): Promise { const { stoppedAtMs } = await waitForJsonAsync<{ stoppedAtMs: number }>(fixture, 'stopped.json'); const timeout: AbortController = new AbortController(); const exit: IProcessExit | undefined = await Promise.race([ @@ -130,24 +134,42 @@ type StopRoute = 'daemon stop' | 'SIGTERM'; `Its stderr:\n${fixture.getStderr()}` ); } - return { ...exit, afterStopMs: Date.now() - stoppedAtMs }; + return { ...exit, stoppedAtMs, afterStopMs: Date.now() - stoppedAtMs }; } - it.each<[StopRoute, Reporter]>([ - ['daemon stop', 'onError'], - ['SIGTERM', 'default'] + /** Checks the report of what kept the process running: its PID, and the time the daemon stopped. */ + function expectLingerReport(fixture: IFixtureDaemon, stoppedAtMs: number): void { + const match: RegExpExecArray | null = LINGER_REPORT.exec(fixture.getStderr()); + if (!match) { + throw new Error(`The fixture did not report what kept it running. Its stderr:\n${fixture.getStderr()}`); + } + expect(Number(match[1])).toBe(fixture.process.pid); + // The daemon stopped, and then the fixture wrote stopped.json. + const reportedStopMs: number = Date.parse(match[2]); + expect(new Date(reportedStopMs).toISOString()).toBe(match[2]); + expect(reportedStopMs).toBeLessThanOrEqual(stoppedAtMs); + expect(stoppedAtMs - reportedStopMs).toBeLessThan(1000); + } + + it.each<[StopRoute, Reporter, RegExp]>([ + ['daemon stop', 'callbacks', /^fixture log: rushd \(PID /m], + ['SIGTERM', 'default', /^rushd \(PID /m] ])( - 'exits 2 s after %s if a timer that it did not start keeps it running, and reports that through %s', - async (route: StopRoute, reporter: Reporter) => { - const fixture: IFixtureDaemon = startDaemon('timer', reporter); + 'exits 2 s after %s if a timer that it did not start keeps it running, and logs that (%s)', + async (route: StopRoute, reporter: Reporter, reportLine: RegExp) => { + const fixture: IFixtureDaemon = startDaemon('timer', reporter, 'process'); const { paths } = await waitForJsonAsync<{ paths: IDaemonPaths }>(fixture, 'ready.json'); await stopAsync(fixture, paths, route); - const { code, signal, afterStopMs } = await waitForExitAsync(fixture); + const { code, signal, stoppedAtMs, afterStopMs } = await waitForExitAsync(fixture); expect({ code, signal }).toEqual({ code: 0, signal: undefined }); expect(afterStopMs).toBeGreaterThanOrEqual(1900); expect(afterStopMs).toBeLessThan(3000); - expect(fixture.getStderr()).toMatch(LINGER_REPORT); + expectLingerReport(fixture, stoppedAtMs); + // A message for the daemon log, through onLog when there is one: not an error, and no stack. + expect(fixture.getStderr()).toMatch(reportLine); + expect(fixture.getStderr().match(/rushd \(PID /g)).toHaveLength(1); + expect(fixture.getStderr()).not.toMatch(/fixture error: |Error: rushd|^\s+at /m); expect(fs.existsSync(paths.socketPath)).toBe(false); expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); }, @@ -155,17 +177,34 @@ type StopRoute = 'daemon stop' | 'SIGTERM'; ); it('keeps the exit code that its caller set when it failed', async () => { - const fixture: IFixtureDaemon = startDaemon('failing', 'onError'); + const fixture: IFixtureDaemon = startDaemon('failing', 'callbacks', 'process'); - const { code, signal, afterStopMs } = await waitForExitAsync(fixture); + const { code, signal, stoppedAtMs, afterStopMs } = await waitForExitAsync(fixture); expect({ code, signal }).toEqual({ code: 1, signal: undefined }); expect(afterStopMs).toBeGreaterThanOrEqual(1900); expect(fixture.getStderr()).toMatch(/The fixture failed after it started\./); - expect(fixture.getStderr()).toMatch(LINGER_REPORT); + expectLingerReport(fixture, stoppedAtMs); + }, 30000); + + it('never exits the process when it is embedded, even if a timer keeps the process running', async () => { + const fixture: IFixtureDaemon = startDaemon('timer', 'callbacks', 'embedded'); + const { paths } = await waitForJsonAsync<{ paths: IDaemonPaths }>(fixture, 'ready.json'); + await stopAsync(fixture, paths, 'daemon stop'); + await waitForJsonAsync<{ stoppedAtMs: number }>(fixture, 'stopped.json'); + + await delayAsync(EMBEDDED_WAIT_MS); + // afterEach ends the fixture process, which only its own timer keeps running now. + expect({ code: fixture.process.exitCode, signal: fixture.process.signalCode }).toEqual({ + code: null, + signal: null + }); + expect(fixture.getStderr()).not.toMatch(/kept its process running/); + expect(fs.existsSync(paths.socketPath)).toBe(false); + expect(readDaemonLockfile(paths.lockfilePath)).toBeUndefined(); }, 30000); it('exits as soon as it stops if nothing keeps it running', async () => { - const fixture: IFixtureDaemon = startDaemon('nothing', 'onError'); + const fixture: IFixtureDaemon = startDaemon('nothing', 'callbacks', 'process'); const { paths } = await waitForJsonAsync<{ paths: IDaemonPaths }>(fixture, 'ready.json'); await stopAsync(fixture, paths, 'daemon stop'); diff --git a/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts b/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts index 5da51032d0..ef8ddd62b5 100644 --- a/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts +++ b/libraries/rush-daemon/src/test/fixtures/LingeringDaemon.ts @@ -15,10 +15,18 @@ import { TemporaryRepoWorkspaceSession } from '../TemporaryRepoWorkspaceSession' export type LeftBehind = 'timer' | 'failing' | 'nothing'; /** - * - `onError`: reports through `onError`, as the Rush daemon's own entry points do. - * - `default`: no `onError`. + * - `callbacks`: passes `onError` and `onLog`, as the Rush daemon's own entry points do. Each one writes with its + * own prefix, `fixture error: ` or `fixture log: `, so that a test can tell the paths apart. + * - `default`: neither. */ -export type Reporter = 'onError' | 'default'; +export type Reporter = 'callbacks' | 'default'; + +/** + * - `process`: no `shutdownSignal`, so the daemon owns its process and handles SIGINT and SIGTERM itself. + * - `embedded`: the fixture passes a `shutdownSignal` that it never aborts. It owns the process, and the daemon + * stops only when a client stops it. + */ +export type Ownership = 'process' | 'embedded'; function writeJson(filename: string, value: unknown): void { fs.writeFileSync(`${filename}.tmp`, JSON.stringify(value)); @@ -37,25 +45,30 @@ function onReady(leftBehind: LeftBehind, controlFolder: string): IRushDaemonServ }; } +function getReporterOptions(reporter: Reporter): Pick { + if (reporter === 'default') return {}; + return { + onError: (error: Error) => process.stderr.write(`fixture error: ${error.stack ?? error.message}\n`), + onLog: (message: string) => process.stderr.write(`fixture log: ${message}\n`) + }; +} + async function runAsync(): Promise { - const [repoRoot, controlFolder, leftBehind, reporter] = process.argv.slice(2); - if (!repoRoot || !controlFolder || !leftBehind || !reporter) { + const [repoRoot, controlFolder, leftBehind, reporter, ownership] = process.argv.slice(2); + if (!repoRoot || !controlFolder || !leftBehind || !reporter || !ownership) { throw new Error( - 'The fixture needs a repository folder, a control folder, what it leaves behind and a reporter.' + 'The fixture needs a repository, a control folder, what it leaves behind, a reporter and an owner.' ); } try { - // Process mode: no shutdownSignal, so the daemon handles SIGINT and SIGTERM itself. await serveRushDaemonAsync({ repoRoot, rushVersion: '5.178.1', daemonVersion: 'lingering-fixture', createWorkspaceSessionAsync: () => Promise.resolve(new TemporaryRepoWorkspaceSession(repoRoot)), onReady: onReady(leftBehind as LeftBehind, controlFolder), - onError: - (reporter as Reporter) === 'onError' - ? (error: Error) => process.stderr.write(`${error.stack ?? error.message}\n`) - : undefined + ...getReporterOptions(reporter as Reporter), + shutdownSignal: (ownership as Ownership) === 'embedded' ? new AbortController().signal : undefined }); } finally { writeJson(path.join(controlFolder, 'stopped.json'), { stoppedAtMs: Date.now() }); From 7c93aea35098163476ee8cc19d996582ff80c2ba Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 02:37:43 +0000 Subject: [PATCH 056/265] [rush-cli-client] The client prints a cancelling line at once, agent output shows an operation's error after its own output, and a cancelled request reports never-started operations as not run Swarm integration step 41; original commit cba3c1f17b (merge of swarm/r04-t132-153-int at 4b495d5f81). Scope: tasks 132, 142 and 153. Brings r04's tasks 132 (after Ctrl-C, SIGTERM or SIGHUP during a daemon request the client prints one line at once, plus a notice when the daemon doesn't confirm), 142 (agent output prints a failed operation's error even when the operation wrote output first) and 153 (a cancelled request no longer reports never-started operations with an earlier request's result), re-tipped on d87702c813. Second agents: t05 board 2323 (132), t07 board 2623 (142), t05 board 2524 (153). ch01's e2e on ch01-sB: the cancelling line and the unconfirmed notice appear, and c is reported aborted, not SUCCESS. s16 batch B, item 1 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree 6453dd938e) Commits folded into this step (3): - 85ed03e693 Say at once that a cancelled request waits for rushd to stop it (task 132) - 892c1592eb Print a failed operation's error even after its output in agent output (task 142) - 52b9d3fe3b Report never-started operations of a cancelled request as aborted (task 153) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 14 +- .../src/AgentProgressRenderer.ts | 122 ++++++++-- .../rush-cli-client/src/clientCancellation.ts | 21 +- apps/rush-cli-client/src/launchClient.ts | 36 ++- .../src/test/AgentProgressRenderer.test.ts | 175 +++++++++++++++ .../src/test/cancellationNotice.test.ts | 212 ++++++++++++++++++ .../test/persistentIpcCancellation.test.ts | 8 +- ...4-t132-cancel-notice_2026-09-28-22-10.json | 11 + ...t142-operation-error_2026-09-28-22-50.json | 11 + ...4-t132-cancel-notice_2026-09-28-22-10.json | 11 + ...-t153-cancel-results_2026-09-28-23-20.json | 11 + common/reviews/api/rush-client-core.api.md | 1 + .../rush-client-core/src/DaemonClient.ts | 10 +- .../src/test/DaemonClient.test.ts | 56 ++++- .../rush-daemon/src/PhasedRequestRouter.ts | 24 +- .../test/PhasedRequestCancellation.test.ts | 203 ++++++++++++++++- 16 files changed, 871 insertions(+), 55 deletions(-) create mode 100644 apps/rush-cli-client/src/test/cancellationNotice.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t142-operation-error_2026-09-28-22-50.json create mode 100644 common/changes/@rushstack/rush-client-core/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t153-cancel-results_2026-09-28-23-20.json diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index e83a16ca03..12c589e650 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -156,11 +156,15 @@ error count that the shown errors account for, and, when the first error shown n location, the lines before it. Up to three operations are reported. Two kinds are reported just before the summary line instead: a failed operation that wrote no output, with the error from the daemon's result, and, when no operation failed, the operations whose warnings failed the request -(`warnings: …`). On a pipe, a status line names a failed operation that wrote no output 1 s after -it failed, unless the result came first. The summary line names up to five failed (or warning) -operations. Every operation's full output is in its project's `rush-logs/` folder, whether or not -it was printed. When a request falls back to in-process Rush, agent mode stops and native output -follows. +(`warnings: …`). The error of a reported operation that wrote output is printed too, unless its +excerpt shows it or it only gives the exit code (`Returned error code: 1`): for example an error +thrown while the operation's build cache entry was restored. Only the daemon's result carries it, +so for an operation reported as it failed it comes just before the summary line, as an +`error: ` line followed by the error. On a pipe, a status line names a failed +operation that wrote no output 1 s after it failed, unless the result came first. The summary +line names up to five failed (or warning) operations. Every operation's full output is in its +project's `rush-logs/` folder, whether or not it was printed. When a request falls back to +in-process Rush, agent mode stops and native output follows. In agent mode a failed `rush build` doesn't wait for all of its work. Its result comes once an operation failed and none of the selected projects that no other selected project depends on (for diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index faa7918fca..d73665441f 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -33,6 +33,8 @@ const PIPE_STATUS_INTERVAL_MS: number = 25_000; const PIPE_UNREPORTED_FAILURE_DELAY_MS: number = 1_000; const SENT_PHASE: string = 'sent to rushd; preparing the workspace graph'; const STARTING_PHASE: string = 'rushd is still starting; waiting for it'; +/** Ends the summary line of a cancelled request when the client stopped waiting before rushd confirmed the stop. */ +const UNCONFIRMED_STOP: string = 'rushd did not confirm that the request stopped; it may still be stopping'; const FAILURE_STATUS: string = 'FAILURE'; const TTY_INTERVAL_MS: number = 100; /** The most failed (or warning) operations whose output excerpt is printed. */ @@ -51,6 +53,14 @@ const MAX_MESSAGE_LENGTH: number = 300; */ const ERROR_DETAIL_HEAD_LINES: number = 2; const ERROR_DETAIL_TAIL_LINES: number = 5; +/** + * The error of an operation whose process exited with a nonzero code. It says nothing that the operation's output + * does not, so it is printed only for an operation that wrote no output. + */ +const EXIT_CODE_ERROR_PATTERN: RegExp = /^Returned error code: \d+$/; +/** How much of an operation error's first line is looked for in the output shown for it, which clips long lines. */ +const SHOWN_ERROR_KEY_LENGTH: number = 100; +const WHITESPACE_PATTERN: RegExp = /\s+/g; /** * Summary labels that differ from the native status name. A daemon reports an operation that is unchanged since * it last ran as `NO OP` (when no operation in the iteration had to run) or as `SKIPPED`; both mean up to date. @@ -90,6 +100,11 @@ export interface IAgentFinalResult { readonly errorMessage?: string; /** Whether the command was cancelled (for example with Ctrl+C); reported as `CANCELLED`, not `FAILURE`. */ readonly cancelled?: boolean; + /** + * For a cancelled command: the client stopped waiting before rushd confirmed that the request stopped, so it may + * still be stopping. + */ + readonly stopUnconfirmed?: boolean; /** The daemon's final operation results, which may report statuses that no event carried. */ readonly operationResults?: ReadonlyArray; /** Why the daemon did not admit the request, if it did not. */ @@ -122,6 +137,37 @@ function getErrorDetail(lines: ReadonlyArray): string[] { ]; } +function getMatchKey(line: string): string { + return line.replace(WHITESPACE_PATTERN, ' ').trim().toLowerCase(); +} + +/** + * The lines that report a failed operation's error from the daemon's result, given the lines already shown for + * the operation: its first line, then its further lines as for a request's error. None when the lines shown + * include the error's first line, or when the error only gives the exit code of a process that wrote output. + */ +function getOperationErrorLines( + errorMessage: string | undefined, + shownLines: ReadonlyArray +): string[] { + const [firstLine, ...detail] = (errorMessage ?? '') + .split('\n') + .map((line) => line.trimEnd()) + .filter((line) => line.trim()); + if (firstLine === undefined) { + return []; + } + const message: string = firstLine.trim(); + const key: string = getMatchKey(message).slice(0, SHOWN_ERROR_KEY_LENGTH); + if ( + (shownLines.length && EXIT_CODE_ERROR_PATTERN.test(message)) || + shownLines.some((line) => getMatchKey(line).includes(key)) + ) { + return []; + } + return [message, ...getErrorDetail(detail)].map((line) => clipLine(line, MAX_MESSAGE_LENGTH)); +} + /** * Compact progress for agents on the daemon path: at most three live rows (TTY) or one line when the request is * sent (pipes), each failed operation's log file and a short excerpt of its output as soon as it fails, and a @@ -131,11 +177,14 @@ function getErrorDetail(lines: ReadonlyArray): string[] { * On a pipe, a request that takes less than 25 s writes the line that says it was sent, its failures and its * summary line, and nothing else. Status lines keep a longer request from looking hung: whenever nothing was * written for 25 s, a status line with the counts and the running operations follows, and a connection that - * takes longer than 10 s gets one. A wait for a daemon that is still starting also gets a line, once. Only the - * first three failed operations are reported. Whether warnings fail the request is only known at its end, so - * operations with warnings are reported before the summary line; so is a failed operation that wrote no output, - * whose error only the daemon's result carries. On a pipe, the next status line, which names that operation, is - * then due 1 s after it failed, so that a result that the daemon returns early can come first and make it moot. + * takes longer than 10 s gets one. A wait for a daemon that is still starting also gets a line, once, and so + * does a cancellation, as soon as the client asks rushd to stop the request. Only the first three failed + * operations are reported. Whether warnings fail the request is only known at its end, so operations with + * warnings are reported before the summary line; so is a failed operation that wrote no output, whose error + * only the daemon's result carries. On a pipe, the next status line, which names that operation, is then due + * 1 s after it failed, so that a result that the daemon returns early can come first and make it moot. An error + * that the output shown for a reported operation leaves out, for example one thrown while its build cache entry + * was restored, is written before the summary line too, as `error: ` and the error. */ export class AgentProgressRenderer { readonly #options: IAgentProgressRendererOptions; @@ -143,8 +192,8 @@ export class AgentProgressRenderer { readonly #startTimeMs: number; readonly #tracker: AgentOperationTracker = new AgentOperationTracker(); readonly #notices: AgentNotices = new AgentNotices(); - /** The operations whose log file and excerpt were written, in that order. */ - readonly #reported: Set = new Set(); + /** The operations whose log file and excerpt were written, in that order, with the output lines shown for each. */ + readonly #reported: Map> = new Map(); #lastActivity: string = ''; #phase: string = 'connecting to rushd (auto-starts if needed)'; #painted: number = 0; @@ -159,6 +208,8 @@ export class AgentProgressRenderer { /** The first queue position, for the summary line. */ #firstQueued: IQueuePosition | undefined; #stopped: boolean = false; + /** The client asked rushd to cancel the request; the progress line says so until the end. */ + #cancelling: boolean = false; /** On a pipe: an operation that wrote no output failed, and no line has named it yet. */ #unnamedFailure: boolean = false; /** The error message that the summary line contains in full, once written. */ @@ -190,7 +241,7 @@ export class AgentProgressRenderer { } public setPhase(phase: string): void { - if (phase === this.#phase) { + if (phase === this.#phase || this.#cancelling) { return; } this.#phase = phase; @@ -248,6 +299,22 @@ export class AgentProgressRenderer { this.setPhase(`queued behind another request (position ${position})`); } + /** + * The client asked rushd to cancel the request, and waits up to `timeoutMs` for rushd to stop it, which can take + * seconds while rushd prepares the workspace graph. Says so at once, and on a TTY until the end; on a pipe, in one + * line. + */ + public onCancelRequested(timeoutMs: number): void { + if (this.#cancelling) { + return; + } + this.setPhase(`cancelling; waiting up to ${Math.round(timeoutMs / 1000)}s for rushd to stop the request`); + this.#cancelling = true; + if (!this.#options.isTTY) { + this.#writePipeLine(this.#rows()[0]); + } + } + public onEvent(event: IDaemonEventEnvelope): void { if (this.#stopped) { return; @@ -277,8 +344,7 @@ export class AgentProgressRenderer { this.#notices.add(payload, event.scope?.operationId); if (typeof payload.text === 'string' && payload.text.trim()) { this.#lastActivity = payload.text.trim().split('\n')[0]; - this.#phase = 'running'; - this.#queued = undefined; + this.#onRunning(); } break; } @@ -329,6 +395,8 @@ export class AgentProgressRenderer { let summary: string = this.#getSummaryLine(verdict, emptySelection); if (verdict === 'FAILURE') { summary += formatUnfinishedOperations(result?.operationResults); + } else if (verdict === 'CANCELLED' && result?.stopUnconfirmed) { + summary += ` · ${UNCONFIRMED_STOP}`; } // An admission failure says that the request waited, and why it stopped waiting. if (this.#firstQueued && !result?.admissionErrorCode) { @@ -370,18 +438,25 @@ export class AgentProgressRenderer { status, logFilePath: typeof logFilePath === 'string' ? logFilePath : undefined }); - this.#phase = 'running'; - this.#queued = undefined; + this.#onRunning(); if (status === FAILURE_STATUS) { this.#reportFailure(operationId); } } + /** The request runs, so it no longer waits in a queue. The phase says so, unless the request is being cancelled. */ + #onRunning(): void { + if (!this.#cancelling) { + this.#phase = 'running'; + } + this.#queued = undefined; + } + /** * Writes a failed operation's log file and output excerpt as soon as it fails, while the rest of the request * runs on. The operation's output all arrived before its status. An operation that wrote nothing is left to * the failure report, which has the error from the daemon's result; on a pipe, the next status line names it - * sooner. + * sooner. The error of an operation reported here follows with the failure report, if the excerpt lacks it. */ #reportFailure(operationId: string): void { if (this.#stopped || this.#reported.has(operationId) || this.#reported.size >= MAX_REPORTED_OPERATIONS) { @@ -442,7 +517,8 @@ export class AgentProgressRenderer { * The log file and output excerpt of each failed operation not yet reported. Without failed operations: * operations with warnings (they fail a build unless the command allows warnings), or else output that belongs * to no operation. A cancelled command reports only failed operations. At most three operations are reported - * in all, with the operations reported as they failed. + * in all, with the operations reported as they failed. The daemon's result carries the error of each failed + * operation, so an operation reported as it failed gets its error here, unless the output shown for it had it. */ #getFailureReport(verdict: Verdict): string[] { const tracker: AgentOperationTracker = this.#tracker; @@ -457,7 +533,12 @@ export class AgentProgressRenderer { const lines: string[] = []; let hidden: number = 0; for (const problem of problems) { - if (this.#reported.has(problem.operationId)) { + const shownLines: ReadonlyArray | undefined = this.#reported.get(problem.operationId); + if (shownLines) { + const errorLines: string[] = getOperationErrorLines(problem.errorMessage, shownLines); + if (errorLines.length) { + lines.push(`error: ${problem.operationId}`, ...errorLines.map((line) => ` ${line}`)); + } continue; } if (this.#reported.size < MAX_REPORTED_OPERATIONS) { @@ -475,21 +556,20 @@ export class AgentProgressRenderer { /** * An operation's report: its log file, then its output excerpt, which is longer for the first reported - * operation (most often the root cause). Records that the operation was reported. + * operation (most often the root cause), then its error from the daemon's result, if known and not shown + * already. Records that the operation was reported, and the lines shown for it. */ #getProblemLines(label: string, problem: IAgentProblemOperation): string[] { const maxLines: number = this.#reported.size ? OTHER_OPERATION_EXCERPT_LINES : FIRST_OPERATION_EXCERPT_LINES; - this.#reported.add(problem.operationId); const excerpt: string[] = problem.excerpt?.getExcerpt(maxLines) ?? []; - if (!excerpt.length && problem.errorMessage) { - excerpt.push(clipLine(problem.errorMessage.trim().split('\n')[0], MAX_MESSAGE_LENGTH)); - } + const shownLines: string[] = [...excerpt, ...getOperationErrorLines(problem.errorMessage, excerpt)]; + this.#reported.set(problem.operationId, shownLines); const logFile: string = problem.logFilePath ? ` · full log: ${problem.logFilePath}` : ''; return [ `${label}: ${problem.operationId}${logFile}`, - ...(excerpt.length ? excerpt : ['(no output)']).map((line) => ` ${line}`) + ...(shownLines.length ? shownLines : ['(no output)']).map((line) => ` ${line}`) ]; } diff --git a/apps/rush-cli-client/src/clientCancellation.ts b/apps/rush-cli-client/src/clientCancellation.ts index f3a2102bff..3cd5648516 100644 --- a/apps/rush-cli-client/src/clientCancellation.ts +++ b/apps/rush-cli-client/src/clientCancellation.ts @@ -18,9 +18,24 @@ export function getSignalExitCode(signal: NodeJS.Signals): number { return SIGNAL_EXIT_CODE_BASE + (os.constants.signals[signal] ?? os.constants.signals.SIGINT); } -/** Formats the notice printed when a daemon-routed command is cancelled. */ -export function formatCancellationMessage(commandName: string): string { - return `rush-client: ${commandName} cancelled.\n`; +/** + * Formats the notice printed as soon as the client asks the daemon to cancel a daemon-routed command, which can take + * the daemon seconds, for example while it prepares the workspace graph. + */ +export function formatCancellingMessage(commandName: string, timeoutMs: number): string { + const seconds: number = Math.round(timeoutMs / 1000); + return `rush-client: cancelling ${commandName}; waiting up to ${seconds} s for rushd to stop the request.\n`; +} + +/** + * Formats the notice printed when a daemon-routed command is cancelled. `stopUnconfirmed` says that the client + * stopped waiting before the daemon confirmed that the request stopped. + */ +export function formatCancellationMessage(commandName: string, stopUnconfirmed: boolean = false): string { + return stopUnconfirmed + ? `rush-client: ${commandName} cancelled, but rushd did not confirm that the request stopped; ` + + 'it may still be stopping.\n' + : `rush-client: ${commandName} cancelled.\n`; } /** diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 9bd42f0ae0..095ea16714 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -29,6 +29,7 @@ import type { AgentProgressRenderer } from './AgentProgressRenderer'; import { CANCELLATION_SIGNALS, formatCancellationMessage, + formatCancellingMessage, getSignalExitCode, isCancelledOutcome } from './clientCancellation'; @@ -173,12 +174,27 @@ export async function launchClientAsync( return; } const abort: AbortController = new AbortController(); + const commandName: string = route.commandName; let cancellationSignal: NodeJS.Signals | undefined; + // Whether the client asked the daemon to cancel the request: after a signal, or a raw Ctrl+C, which raises none. + let cancelRequested: boolean = false; // Windows test harnesses emit signals without a name; treat those as Ctrl+C. const onSignal = (signal?: NodeJS.Signals): void => { cancellationSignal ??= signal ?? 'SIGINT'; abort.abort(); }; + const onCancelRequested = (timeoutMs: number): void => { + cancelRequested = true; + if (agentRenderer) { + agentRenderer.onCancelRequested(timeoutMs); + return; + } + // After SIGHUP the terminal may be gone. + writeStreamAsync(process.stderr, Buffer.from(formatCancellingMessage(commandName, timeoutMs))).catch( + () => undefined + ); + }; + const isCancelled = (): boolean => abort.signal.aborted || cancelRequested; for (const signal of CANCELLATION_SIGNALS) process.on(signal, onSignal); const renderer: ClientOperationRenderer = new ClientOperationRenderer({ requestId: request.requestId, @@ -235,6 +251,7 @@ export async function launchClientAsync( stdin: process.stdin, requiresStdinEnd: !process.stdin.isTTY, cancelOnCtrlC: !!process.stdin.isTTY, + onCancelRequested, initialRawMode: !!process.stdin.isRaw, setRawMode: process.stdin.isTTY ? (enabled) => { @@ -244,7 +261,7 @@ export async function launchClientAsync( }); } catch (error) { // After cancellation, a transport failure (e.g. the cancellation deadline) still means "cancelled". - if (!abort.signal.aborted || !(error instanceof DaemonClientError)) throw error; + if (!isCancelled() || !(error instanceof DaemonClientError)) throw error; outcome = undefined; } finally { for (const signal of CANCELLATION_SIGNALS) process.removeListener(signal, onSignal); @@ -254,18 +271,23 @@ export async function launchClientAsync( await client.closeAsync(); } } - if (outcome === undefined || isCancelledOutcome(outcome, abort.signal.aborted)) { + if (outcome === undefined || isCancelledOutcome(outcome, isCancelled())) { const exitCode: number = getSignalExitCode(cancellationSignal ?? 'SIGINT'); + // The client stopped waiting (at the cancellation deadline, or when the connection closed) before the daemon + // confirmed that the request stopped. The daemon also cancels a request whose client disconnects. + const stopUnconfirmed: boolean = outcome === undefined && cancelRequested; agentRenderer?.finish( outcome?.kind === 'result' ? { ...outcome.result, exitCode, cancelled: true } - : { exitCode, cancelled: true } + : { exitCode, cancelled: true, stopUnconfirmed } ); process.exitCode = exitCode; - // After SIGHUP the terminal may be gone; the exit code is what matters. - await writeStreamAsync(process.stderr, Buffer.from(formatCancellationMessage(route.commandName))).catch( - () => undefined - ); + // After SIGHUP the terminal may be gone; the exit code is what matters. Agent output's summary line already + // says whether the daemon confirmed the stop. + await writeStreamAsync( + process.stderr, + Buffer.from(formatCancellationMessage(commandName, stopUnconfirmed && !agentRenderer)) + ).catch(() => undefined); } else if (outcome.kind === 'result') { // In agent mode the summary line may already carry the complete error message; do not repeat it. const reportedByAgent: boolean = agentRenderer?.finish(outcome.result) ?? false; diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts index b8cfc5f1b9..9531de11f5 100644 --- a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -253,6 +253,61 @@ describe(AgentProgressRenderer.name, () => { ]); }); + it('says at once that it waits for rushd to stop a cancelled request, and whether rushd confirmed (task 132)', () => { + const confirmed: ITestRenderer = createRenderer(false); + confirmed.renderer.onEvent(registered('a (build)')); + confirmed.renderer.onEvent(status('a (build)', 'EXECUTING')); + confirmed.clock.ms = 7_600; + confirmed.renderer.onCancelRequested(5_000); + expect(confirmed.lines()).toEqual([ + 'rush build 0/1 · 7.6s · cancelling; waiting up to 5s for rushd to stop the request' + ]); + // Written once, and the operations that rushd stops do not make the request look as if it runs on. + confirmed.renderer.onCancelRequested(5_000); + confirmed.renderer.onEvent(status('a (build)', 'ABORTED')); + confirmed.clock.ms = 8_100; + confirmed.renderer.finish({ exitCode: 130, cancelled: true }); + expect(confirmed.lines()).toEqual([ + 'rush build 0/1 · 7.6s · cancelling; waiting up to 5s for rushd to stop the request', + 'rush build: CANCELLED 1/1 operations (1 aborted) in 8.1s' + ]); + + const unconfirmed: ITestRenderer = createRenderer(false); + unconfirmed.renderer.onCancelRequested(5_000); + unconfirmed.clock.ms = 5_000; + unconfirmed.renderer.finish({ exitCode: 130, cancelled: true, stopUnconfirmed: true }); + expect(unconfirmed.lines()).toEqual([ + 'rush build · 0.0s · cancelling; waiting up to 5s for rushd to stop the request', + 'rush build: CANCELLED in 5.0s · rushd did not confirm that the request stopped; it may still be stopping' + ]); + }); + + it('keeps showing on a TTY that it cancels', () => { + jest.useFakeTimers(); + try { + const { renderer, output } = createRenderer(true, 'build', 120); + const firstRow = (): string => output[output.length - 1].replace(ANSI_ESCAPE, '').split('\n')[0]; + renderer.start(); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onCancelRequested(5_000); + // Repainted at once, rather than on the next tick of the timer. + expect(firstRow()).toBe( + '⠙ rush build 0/1 · 0.0s · cancelling; waiting up to 5s for rushd to stop the request' + ); + renderer.onEvent(status('a (build)', 'ABORTED')); + renderer.onQueuePosition(1); + jest.advanceTimersByTime(100); + // Neither the operation that rushd stopped nor a queue position makes the request look as if it runs on. + expect(firstRow()).toBe( + '⠹ rush build 1/1 · 0.0s · cancelling; waiting up to 5s for rushd to stop the request' + ); + renderer.dispose(); + } finally { + jest.useRealTimers(); + } + }); + it('applies final statuses from the daemon result that no event reported', () => { const { renderer, lines } = createRenderer(false); const reason: string = @@ -647,6 +702,126 @@ describe(AgentProgressRenderer.name, () => { ]); }); + it('prints the error of an operation reported as it failed, when its output lacks the error (task 142)', () => { + const { renderer, lines } = createRenderer(false); + const querying: string = + 'This project was not found in the local build cache. Querying the cloud build cache.'; + const sasError: string = + "An Azure Storage SAS credential hasn't been provided, or has expired. Update the credentials by " + + 'running "rush update-cloud-credentials", or provide a SAS in the RUSH_BUILD_CACHE_CREDENTIAL ' + + 'environment variable'; + renderer.onEvent(registered('mini-a (build)')); + renderer.onEvent(registered('mini-b (build)')); + renderer.onEvent(status('mini-a (build)', 'EXECUTING')); + renderer.onLog(Buffer.from(`${querying}\n`), 'mini-a (build)', 'stdout'); + renderer.onEvent(status('mini-a (build)', 'FAILURE')); + renderer.onEvent(status('mini-b (build)', 'BLOCKED')); + expect(lines()).toEqual(['failed: mini-a (build)', ` ${querying}`]); + renderer.finish({ + exitCode: 1, + operationResults: [ + { operationId: 'mini-a (build)', status: 'FAILURE', errorMessage: `${sasError}\n` }, + { operationId: 'mini-b (build)', status: 'BLOCKED' } + ] + }); + expect(lines()).toEqual([ + 'failed: mini-a (build)', + ` ${querying}`, + 'error: mini-a (build)', + ` ${sasError}`, + 'rush build: FAILURE 2/2 operations (1 failure, 1 blocked) in 0.0s · failed: mini-a (build)' + ]); + }); + + it("prints a failed operation's error after its excerpt, with the further lines of a multi-line error", () => { + const { renderer, lines } = createRenderer(false); + renderer.onEvent(registered('a (build)')); + renderer.onEvent(status('a (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('Restoring from the build cache\n'), 'a (build)', 'stdout'); + const detail: string[] = Array.from({ length: 9 }, (unused, i) => ` detail ${i}`); + renderer.finish({ + exitCode: 1, + operationResults: [ + { + operationId: 'a (build)', + status: 'FAILURE', + errorMessage: [' Could not read the cache entry', ...detail].join('\r\n') + } + ] + }); + expect(lines()).toEqual([ + 'failed: a (build)', + ' Restoring from the build cache', + ' Could not read the cache entry', + ' detail 0', + ' detail 1', + ' … 2 more lines …', + ' detail 4', + ' detail 5', + ' detail 6', + ' detail 7', + ' detail 8', + 'rush build: FAILURE 1/1 operations (1 failure) in 0.0s · failed: a (build)' + ]); + }); + + it("does not repeat a failed operation's error that its output shows, or the exit code of a process that wrote output", () => { + const { renderer, lines } = createRenderer(false); + const readiness: string = 'The explicit daemon Node tool exited without completing IPC readiness.'; + fail(renderer, 'a (build)', ['src/a.ts:1:1 - error TS2322: a']); + fail(renderer, 'b (build)', [ + ` The explicit daemon node tool exited without completing IPC readiness.` + ]); + renderer.onEvent(status('c (build)', 'EXECUTING')); + renderer.onLog(Buffer.from('c output\n'), 'c (build)', 'stderr'); + renderer.finish({ + exitCode: 1, + operationResults: [ + { operationId: 'a (build)', status: 'FAILURE', errorMessage: 'Returned error code: 2' }, + { operationId: 'b (build)', status: 'FAILURE', errorMessage: readiness }, + { operationId: 'c (build)', status: 'FAILURE', errorMessage: 'Returned error code: 127' } + ] + }); + expect(lines()).toEqual([ + 'failed: a (build) · full log: /repo/a/rush-logs/x.log', + ' src/a.ts:1:1 - error TS2322: a', + 'failed: b (build) · full log: /repo/b/rush-logs/x.log', + ' The explicit daemon node tool exited without completing IPC readiness.', + 'failed: c (build)', + ' c output', + 'rush build: FAILURE 3/3 operations (3 failures) in 0.0s · failed: a (build), b (build), c (build)' + ]); + }); + + it('prints the signal that ended an operation, and clips a long error unless the output shows its start', () => { + const { renderer, lines } = createRenderer(false); + const long: string = `Cache entry rejected: ${'x'.repeat(400)}`; + const other: string = `Cache entry rejected: ${'y'.repeat(400)}`; + fail(renderer, 'a (build)', ['a output']); + fail(renderer, 'b (build)', [long]); + fail(renderer, 'c (build)', ['c output']); + renderer.finish({ + exitCode: 1, + operationResults: [ + { operationId: 'a (build)', status: 'FAILURE', errorMessage: 'Terminated by signal: SIGKILL' }, + { operationId: 'b (build)', status: 'FAILURE', errorMessage: long }, + { operationId: 'c (build)', status: 'FAILURE', errorMessage: other } + ] + }); + expect(lines()).toHaveLength(11); + expect(lines()[3]).toMatch(/^ {2}Cache entry rejected: x+…x+$/); + expect(lines().slice(6, 9)).toEqual([ + 'error: a (build)', + ' Terminated by signal: SIGKILL', + 'error: c (build)' + ]); + expect(lines()[9]).toMatch(/^ {2}Cache entry rejected: y+…y+$/); + expect(lines()[9]).toHaveLength(2 + 300); + expect(lines()[10]).toBe( + 'rush build: FAILURE 3/3 operations (3 failures) in 0.0s · failed: a (build), b (build), c (build)' + ); + }); + it('reports at most three operations in all, as they failed or before the summary line', () => { const { renderer, lines } = createRenderer(false); fail(renderer, 'a (build)', ['a error']); diff --git a/apps/rush-cli-client/src/test/cancellationNotice.test.ts b/apps/rush-cli-client/src/test/cancellationNotice.test.ts new file mode 100644 index 0000000000..aea86c74d0 --- /dev/null +++ b/apps/rush-cli-client/src/test/cancellationNotice.test.ts @@ -0,0 +1,212 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('@microsoft/rush/lib/start', () => ({})); +jest.mock('@rushstack/rush-client-core', () => ({ + ...jest.requireActual('@rushstack/rush-client-core'), + connectOrAwaitDaemonStartupAsync: jest.fn(), + executeWithDaemonRestartAsync: jest.fn() +})); + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { + DaemonClientError, + connectOrAwaitDaemonStartupAsync, + executeWithDaemonRestartAsync, + type DaemonClient, + type DaemonClientOutcome, + type IDaemonClientExecuteOptions +} from '@rushstack/rush-client-core'; + +import { AgentProgressRenderer } from '../AgentProgressRenderer'; +import * as connectionOptions from '../daemonConnectionOptions'; +import { launchClientAsync } from '../launchClient'; +import { getTestProcessEnvironment } from './TestProcessEnvironment'; + +const CANCELLING: string = + 'rush-client: cancelling build; waiting up to 5 s for rushd to stop the request.\n'; +const CANCELLED: string = 'rush-client: build cancelled.\n'; +const UNCONFIRMED: string = + 'rush-client: build cancelled, but rushd did not confirm that the request stopped; it may still be stopping.\n'; + +type Execution = (options: IDaemonClientExecuteOptions) => Promise; + +describe('the cancellation of a daemon request (task 132)', () => { + let folder: string; + let originalArgv: string[]; + let originalEnvironment: NodeJS.ProcessEnv; + let originalExitCode: typeof process.exitCode; + let listenersBefore: Map; + let stderr: string[]; + /** What the client had written to stderr when the daemon heard of the cancellation. */ + let stderrAtCancel: string[] | undefined; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-client-cancellation-')); + originalArgv = process.argv; + originalEnvironment = process.env; + originalExitCode = process.exitCode; + listenersBefore = new Map( + (['SIGINT', 'SIGTERM'] as const).map((signal) => [signal, process.listeners(signal)]) + ); + stderr = []; + stderrAtCancel = undefined; + fs.writeFileSync( + path.join(folder, 'rush.json'), + JSON.stringify({ rushVersion: '5.178.1', pnpmVersion: '10.27.0', projects: [] }) + ); + jest.spyOn(connectionOptions, 'getDaemonConnectionOptionsAsync').mockResolvedValue({ + paths: { + runtimeDir: folder, + socketPath: path.join(folder, 'd.sock'), + lockfilePath: path.join(folder, 'daemon.pid.json') + } + }); + jest.spyOn(process.stderr, 'write').mockImplementation((( + chunk: string | Uint8Array, + ...rest: unknown[] + ): boolean => { + stderr.push(Buffer.from(chunk).toString()); + (rest.find((argument) => typeof argument === 'function') as (() => void) | undefined)?.(); + return true; + }) as typeof process.stderr.write); + jest.spyOn(process.stdout, 'write').mockReturnValue(true); + jest.spyOn(process, 'cwd').mockReturnValue(folder); + jest.mocked(connectOrAwaitDaemonStartupAsync).mockResolvedValue({ + closeAsync: async () => undefined, + status: Promise.resolve({ pid: process.pid }) + } as unknown as DaemonClient); + process.argv = [process.execPath, 'rush-client', 'build', '--to', 'project']; + const environment: NodeJS.ProcessEnv = getTestProcessEnvironment(originalEnvironment); + for (const name of Object.keys(environment)) { + if (name.startsWith('RUSH_')) delete environment[name]; + } + process.env = { ...environment, RUSH_DAEMON: '1' }; + }); + + afterEach(() => { + process.argv = originalArgv; + process.env = originalEnvironment; + process.exitCode = originalExitCode; + jest.restoreAllMocks(); + jest.mocked(connectOrAwaitDaemonStartupAsync).mockReset(); + jest.mocked(executeWithDaemonRestartAsync).mockReset(); + fs.rmSync(folder, { recursive: true }); + }); + + /** Delivers a signal to the listener that the client installed, without signalling the test process. */ + function deliverSignal(name: 'SIGINT' | 'SIGTERM'): void { + const installed: unknown[] = process + .listeners(name) + .filter((listener) => !listenersBefore.get(name)!.includes(listener)); + expect(installed).toHaveLength(1); + (installed[0] as (signal: NodeJS.Signals) => void)(name); + } + + /** Runs the request on a daemon that `execute` plays, as `DaemonClient` reports it to the client. */ + function execute(execution: Execution): void { + jest + .mocked(executeWithDaemonRestartAsync) + .mockImplementation((client, connection, options) => execution(options)); + } + + /** The daemon hears of the cancellation, with the default deadline of `DaemonClient`. */ + function requestCancel(options: IDaemonClientExecuteOptions): void { + options.onCancelRequested!(5000); + stderrAtCancel = [...stderr]; + } + + function aborted(options: IDaemonClientExecuteOptions): DaemonClientOutcome { + return { + kind: 'result', + result: { requestId: options.request.requestId, exitCode: 130, outcome: 'aborted', aborted: true } + }; + } + + function cancellationDeadline(): DaemonClientError { + return new DaemonClientError( + 'timeout', + 'Daemon did not finish cancellation; disconnected without retrying the command.' + ); + } + + it('says at once that it waits for rushd, and keeps the final line when rushd confirms the stop', async () => { + execute(async (options) => { + deliverSignal('SIGINT'); + expect(options.abortSignal!.aborted).toBe(true); + requestCancel(options); + return aborted(options); + }); + await launchClientAsync(false); + expect(stderrAtCancel).toEqual([CANCELLING]); + expect(stderr).toEqual([CANCELLING, CANCELLED]); + expect(process.exitCode).toBe(130); + }); + + it('says so when rushd does not confirm the stop before the cancellation deadline', async () => { + execute(async (options) => { + deliverSignal('SIGTERM'); + requestCancel(options); + throw cancellationDeadline(); + }); + await launchClientAsync(false); + expect(stderr).toEqual([CANCELLING, UNCONFIRMED]); + expect(process.exitCode).toBe(143); + }); + + it('reports a raw Ctrl+C, which raises no signal, like SIGINT', async () => { + execute(async (options) => { + requestCancel(options); + expect(options.abortSignal!.aborted).toBe(false); + throw cancellationDeadline(); + }); + await launchClientAsync(false); + expect(stderr).toEqual([CANCELLING, UNCONFIRMED]); + expect(process.exitCode).toBe(130); + }); + + it('adds nothing when the request never reached rushd, which has nothing to stop', async () => { + execute(async (options) => { + deliverSignal('SIGINT'); + return aborted(options); + }); + await launchClientAsync(false); + expect(stderr).toEqual([CANCELLED]); + expect(process.exitCode).toBe(130); + }); + + it('writes both notices in the agent output, once', async () => { + const output: string[] = []; + const renderer: AgentProgressRenderer = new AgentProgressRenderer({ + commandName: 'build', + isTTY: false, + columns: 80, + write: (text: string) => output.push(text) + }); + let outputAtCancel: string[] | undefined; + execute(async (options) => { + deliverSignal('SIGINT'); + requestCancel(options); + outputAtCancel = [...output]; + throw cancellationDeadline(); + }); + await launchClientAsync(false, renderer); + const lines: string[] = output.join('').split('\n').slice(0, -1); + expect(outputAtCancel).toHaveLength(2); + expect(lines).toEqual([ + expect.stringMatching(/^rush build · \d+\.\ds · sent to rushd; preparing the workspace graph /), + expect.stringMatching( + /^rush build · \d+\.\ds · cancelling; waiting up to 5s for rushd to stop the request$/ + ), + expect.stringMatching( + /^rush build: CANCELLED in \d+\.\ds · rushd did not confirm that the request stopped; it may still be stopping$/ + ) + ]); + // The summary line says whether rushd confirmed the stop, so the legacy notices add nothing to it. + expect(stderr).toEqual([CANCELLED]); + expect(process.exitCode).toBe(130); + }); +}); diff --git a/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts b/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts index 883d2f83ab..1ba423e27e 100644 --- a/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts +++ b/apps/rush-cli-client/src/test/persistentIpcCancellation.test.ts @@ -57,7 +57,13 @@ describe('public-client cancellation of an admitted Node operation', () => { else client.kill('SIGINT'); // Cancellation terminates the client like a native signal (128 + SIGINT). expect(await closed).toEqual([130, null]); - expect(stderr).toContain('rush-client: build cancelled.'); + // The client says at once that it waits for rushd, which then confirms the stop in time. + const cancelling: number = stderr.indexOf( + 'rush-client: cancelling build; waiting up to 5 s for rushd' + ); + expect(cancelling).toBeGreaterThanOrEqual(0); + expect(stderr.indexOf('rush-client: build cancelled.')).toBeGreaterThan(cancelling); + expect(stderr).not.toContain('did not confirm'); expect(stderr).not.toMatch(/using in-process|not retried|timed out/i); expect(fixture.events().filter((event) => event.kind === 'ready')).toHaveLength(1); expect(fixture.events().filter((event) => event.kind === 'complete')).toHaveLength(2); diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json new file mode 100644 index 0000000000..412a9b9374 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "When a daemon-routed command is cancelled (SIGINT, SIGTERM, SIGHUP, or a raw Ctrl+C in watch mode), say at once that the client waits for rushd to stop the request, and for how long, in both legacy and agent output. If rushd does not confirm the stop in that time, the final line says so. A raw Ctrl+C is now reported like SIGINT, with exit code 130.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t142-operation-error_2026-09-28-22-50.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t142-operation-error_2026-09-28-22-50.json new file mode 100644 index 0000000000..96c2b0d51c --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t142-operation-error_2026-09-28-22-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "In agent output, print a failed operation's error from the daemon's result even when the operation wrote output first, unless the output shows it or it only gives the exit code; for example an error thrown while the operation's build cache entry was restored. An operation reported as it failed gets an `error: ` line and the error just before the summary line.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json b/common/changes/@rushstack/rush-client-core/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json new file mode 100644 index 0000000000..5cc468838e --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/swarm-r04-t132-cancel-notice_2026-09-28-22-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Add `IDaemonClientExecuteOptions.onCancelRequested`. The client calls it once, synchronously, when it asks the daemon to cancel a request (after the abort signal, or a raw Ctrl+C with `cancelOnCtrlC`), and passes the time it waits for the daemon to finish cancelling.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t153-cancel-results_2026-09-28-23-20.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t153-cancel-results_2026-09-28-23-20.json new file mode 100644 index 0000000000..baaab8e31a --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t153-cancel-results_2026-09-28-23-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A cancelled phased request reports each operation as it ended in the request's own iteration, and an operation that the cancel kept from starting as aborted, instead of with the result that an earlier request left for it (for example a failure that the change being built had fixed).", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 6955908922..48c0cafbdb 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -131,6 +131,7 @@ export interface IDaemonClientExecuteOptions { readonly cancelOnCtrlC?: boolean; // (undocumented) readonly initialRawMode?: boolean; + readonly onCancelRequested?: (timeoutMs: number) => void; // (undocumented) readonly onEventAsync?: (event: IDaemonEventEnvelope) => Promise; readonly onQueuePositionAsync?: (position: number, restartReason?: DaemonRestartReason) => Promise; diff --git a/libraries/rush-client-core/src/DaemonClient.ts b/libraries/rush-client-core/src/DaemonClient.ts index 2bf2a57f1b..ec2c66af68 100644 --- a/libraries/rush-client-core/src/DaemonClient.ts +++ b/libraries/rush-client-core/src/DaemonClient.ts @@ -67,6 +67,12 @@ export interface IDaemonClientExecuteOptions { readonly cancelOnCtrlC?: boolean; /** Time allowed to finish cancellation. Defaults to 5000 milliseconds. */ readonly cancellationTimeoutMs?: number; + /** + * Called once, synchronously, when the client asks the daemon to cancel the request: after `abortSignal` aborts, + * or on a raw Ctrl+C with `cancelOnCtrlC`. The daemon then has `timeoutMs` to deliver its final result; after + * that, the client disconnects without it. Not called for a request that was never sent. + */ + readonly onCancelRequested?: (timeoutMs: number) => void; } /** Only explicit, pre-execution rejections permit in-process fallback. @beta */ @@ -307,6 +313,7 @@ export class DaemonClient { if (this.#cancelSent || this.#finished || !this.#execution) return; this.#cancelSent = true; this.#stopInput(); + const timeoutMs: number = this.#execution.cancellationTimeoutMs ?? 5000; this.#cancelTimer = setTimeout(() => { this.#connection.abort( new DaemonClientError( @@ -314,11 +321,12 @@ export class DaemonClient { 'Daemon did not finish cancellation; disconnected without retrying the command.' ) ); - }, this.#execution.cancellationTimeoutMs ?? 5000); + }, timeoutMs); void this.#sendControlAsync({ kind: 'requestCancel', payload: { requestId: this.#execution.request.requestId } }).catch((error: Error) => this.#fail(error)); + this.#execution.onCancelRequested?.(timeoutMs); } async #onFrameAsync(frame: IDaemonFrame): Promise { diff --git a/libraries/rush-client-core/src/test/DaemonClient.test.ts b/libraries/rush-client-core/src/test/DaemonClient.test.ts index 5498a3d4ea..a9b8ea38ec 100644 --- a/libraries/rush-client-core/src/test/DaemonClient.test.ts +++ b/libraries/rush-client-core/src/test/DaemonClient.test.ts @@ -312,9 +312,12 @@ describe('DaemonClient', () => { it('cancels on abort and waits for the authoritative result', async () => { const abort = new AbortController(); const envelope = request(); + const cancelRequests: number[] = []; + let cancelRequestsBeforeCancel: number | undefined; onRequest = async (message) => { if (message.kind === 'requestStart') abort.abort(); if (message.kind === 'requestCancel') { + cancelRequestsBeforeCancel = cancelRequests.length; await sendAsync({ kind: 'requestResult', payload: { requestId: envelope.requestId, exitCode: 130, aborted: true, outcome: 'aborted' } @@ -322,11 +325,42 @@ describe('DaemonClient', () => { } }; const client = await DaemonClient.connectAsync({ socketPath: address }); - expect(await client.executeAsync({ request: envelope, abortSignal: abort.signal })).toMatchObject({ + expect( + await client.executeAsync({ + request: envelope, + abortSignal: abort.signal, + onCancelRequested: (timeoutMs) => cancelRequests.push(timeoutMs) + }) + ).toMatchObject({ kind: 'result', result: { exitCode: 130 } }); expect(controls.filter((message) => message.kind === 'requestCancel')).toHaveLength(1); + // The caller hears of the cancellation, and its default deadline, before the daemon does. + expect(cancelRequests).toEqual([5000]); + expect(cancelRequestsBeforeCancel).toBe(1); + }); + + it('disconnects without a result when the daemon does not finish cancellation in time', async () => { + const abort = new AbortController(); + const cancelRequests: number[] = []; + onRequest = async (message) => { + if (message.kind === 'requestStart') abort.abort(); + }; + const client = await DaemonClient.connectAsync({ socketPath: address }); + await expect( + client.executeAsync({ + request: request(), + abortSignal: abort.signal, + cancellationTimeoutMs: 50, + onCancelRequested: (timeoutMs) => cancelRequests.push(timeoutMs) + }) + ).rejects.toMatchObject({ + code: 'timeout', + message: expect.stringContaining('did not finish cancellation') + }); + expect(cancelRequests).toEqual([50]); + expect(controls.filter((message) => message.kind === 'requestCancel')).toHaveLength(1); }); it('reports why a queued request waits when the daemon restarts after the requests ahead of it', async () => { @@ -647,13 +681,15 @@ describe('DaemonClient', () => { it('does not send an already cancelled request or leak a rejected completion promise', async () => { const client = await DaemonClient.connectAsync({ socketPath: address }); - expect(await client.executeAsync({ request: request(), abortSignal: AbortSignal.abort() })).toMatchObject( - { - kind: 'result', - result: { exitCode: 130, aborted: true } - } - ); + const onCancelRequested: jest.Mock = jest.fn(); + expect( + await client.executeAsync({ request: request(), abortSignal: AbortSignal.abort(), onCancelRequested }) + ).toMatchObject({ + kind: 'result', + result: { exitCode: 130, aborted: true } + }); expect(controls.some((message) => message.kind === 'requestStart')).toBe(false); + expect(onCancelRequested).not.toHaveBeenCalled(); }); it('turns raw Ctrl+C into cancellation when enabled by the CLI', async () => { @@ -672,14 +708,18 @@ describe('DaemonClient', () => { } }; const client = await DaemonClient.connectAsync({ socketPath: address }); + const cancelRequests: number[] = []; expect( await client.executeAsync({ request: envelope, stdin, cancelOnCtrlC: true, - setRawMode: () => {} + setRawMode: () => {}, + onCancelRequested: (timeoutMs) => cancelRequests.push(timeoutMs) }) ).toMatchObject({ kind: 'result', result: { exitCode: 130 } }); + // A raw Ctrl+C raises no signal, so only the client can say that it cancels. + expect(cancelRequests).toEqual([5000]); }); }); diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index cf783fec90..27a9299eb7 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -1378,20 +1378,34 @@ function applySelections(graph: IOperationGraph, selections: ReadonlyArray, graph: IOperationGraph, requestSink: PhasedRequestEventSink, - fillMissingAsAborted: boolean = false, + aborted: boolean = false, report: OperationReport = 'final' ): ReadonlyArray { const outcomes: IPhasedOperationOutcome[] = []; for (const operation of [...activeOperations].sort(compareOperations)) { const observed: ReturnType = requestSink.getObservedResult(operation); - const retained: IOperationExecutionResult | undefined = graph.resultByOperation.get(operation); + const retained: IOperationExecutionResult | undefined = aborted + ? undefined + : graph.resultByOperation.get(operation); const current: IOperationExecutionResult | undefined = - report === 'running' ? requestSink.getScheduledResult(operation) : undefined; + report === 'running' || (aborted && report === 'final') + ? requestSink.getScheduledResult(operation) + : undefined; let status: string | undefined; let errorMessage: string | undefined; if (current !== undefined) { @@ -1416,8 +1430,8 @@ function collectOperationOutcomes( status = retained?.status ?? observed?.status; errorMessage = retained?.error?.message ?? observed?.executionResult.error?.message; } - status ??= fillMissingAsAborted ? OperationStatus.Aborted : undefined; - if (fillMissingAsAborted && status !== undefined && IN_PROGRESS_STATUSES.has(status)) { + status ??= aborted ? OperationStatus.Aborted : undefined; + if (aborted && status !== undefined && IN_PROGRESS_STATUSES.has(status)) { // The client stopped observing before this operation finished, e.g. because it was terminated. status = OperationStatus.Aborted; } diff --git a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts index 7941f76a7f..7664b94116 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestCancellation.test.ts @@ -17,6 +17,7 @@ import type { ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; const OPERATION_A: string = 'project-a (_phase:test)'; const OPERATION_B: string = 'project-b (_phase:test)'; const OPERATION_C: string = 'project-c (_phase:test)'; +const OPERATION_D: string = 'project-d (_phase:test)'; const PROMPT_CANCELLATION_MS: number = 1000; const TIMED_OUT: 'timed out' = 'timed out'; @@ -55,8 +56,11 @@ interface IHangingOperation { readonly release: () => void; } -/** An operation that only finishes when released, or when its hard-abort signal fires (like a killed process). */ -function createHangingOperation(): { +/** + * An operation that only finishes when released, or when its hard-abort signal fires (like a killed process), unless + * it ignores that signal. + */ +function createHangingOperation(ignoresTermination: boolean = false): { hanging: IHangingOperation; actionAsync: (terminal: ITerminal, context: IOperationRunnerContext) => Promise; } { @@ -82,8 +86,8 @@ function createHangingOperation(): { const aborted: Promise = new Promise((resolve) => abortSignal.addEventListener('abort', () => resolve(), { once: true }) ); - await Promise.race([aborted, released]); - if (abortSignal.aborted) { + await (ignoresTermination ? released : Promise.race([aborted, released])); + if (abortSignal.aborted && !ignoresTermination) { onTerminated(); return OperationStatus.Aborted; } @@ -104,6 +108,66 @@ function createFixture( ); } +interface IUpstreamFixFixture { + readonly fixture: ITestRoutingFixture; + readonly hanging: IHangingOperation; + /** Changes A, which then runs until it is released, or terminated unless it ignores that. C stays up to date. */ + readonly changeA: () => void; +} + +/** + * B and D depend on A, and D also depends on C. B fails until A is changed, like a downstream error that is fixed + * in the upstream project. After `changeA()`, the warm graph skips C as unchanged. + */ +function createUpstreamFixFixture(ignoresTermination: boolean = false): IUpstreamFixFixture { + const { hanging, actionAsync: hangingActionAsync } = createHangingOperation(ignoresTermination); + let changed: boolean = false; + const fixture: ITestRoutingFixture = createRoutingFixture( + new Map([ + [ + OPERATION_A, + new TestOperationRunner(OPERATION_A, OperationStatus.Success, async (terminal, context) => + changed ? await hangingActionAsync(terminal, context) : undefined + ) + ], + [ + OPERATION_B, + new TestOperationRunner(OPERATION_B, OperationStatus.Success, async () => + changed ? undefined : OperationStatus.Failure + ) + ], + [OPERATION_C, new TestOperationRunner(OPERATION_C)], + [OPERATION_D, new TestOperationRunner(OPERATION_D)] + ]), + [ + [OPERATION_B, OPERATION_A], + [OPERATION_D, OPERATION_A], + [OPERATION_D, OPERATION_C] + ], + { supportsTerminateRunning: true } + ); + // A and C start together, so C is skipped while A runs. + fixture.graph.parallelism = 2; + fixture.graph.hooks.configureIteration.tap('unchanged C', (records) => { + for (const record of records.values()) { + if (changed && record.operation.name === OPERATION_C) { + record.enabled = false; + } + } + }); + const changeA = (): void => { + changed = true; + fixture.session.operationGraph.invalidateOperations([fixture.operations.get(OPERATION_A)!], 'changed'); + }; + return { fixture, hanging, changeA }; +} + +function getStatuses(result: IDaemonPhasedRequestResult): Record { + return Object.fromEntries( + result.operationResults.map(({ operationId, status }) => [operationId, status] as const) + ); +} + describe('phased request client cancellation', () => { it('terminates running operations when the last client cancels and releases the graph promptly', async () => { const { hanging, actionAsync } = createHangingOperation(); @@ -353,4 +417,135 @@ describe('phased request client cancellation', () => { expect(orphan.signals.map((signal: AbortSignal) => signal.aborted)).toEqual([true]); await orphan.terminated; }); + + function createUpstreamFixRequest(requestId: string): IDaemonPhasedRequest { + return { + ...createRequest(requestId, OPERATION_B), + operationSelection: [ + { enabledState: true, operationId: OPERATION_B }, + { enabledState: true, operationId: OPERATION_D } + ] + }; + } + + it('reports operations that a cancel kept from starting as aborted, not with the results of an earlier request', async () => { + const { fixture, hanging, changeA } = createUpstreamFixFixture(); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const first: IDaemonPhasedRequestResult = await router.executeAsync( + createUpstreamFixRequest('first'), + new TestPhasedRequestClient('one') + ); + expect(getStatuses(first)).toEqual({ + [OPERATION_A]: OperationStatus.Success, + [OPERATION_B]: OperationStatus.Failure, + [OPERATION_C]: OperationStatus.Success, + [OPERATION_D]: OperationStatus.Success + }); + + // The fix goes into A, and the build is cancelled while A runs, before B and D can start. + changeA(); + const client: TestPhasedRequestClient = new TestPhasedRequestClient('two'); + const cancelled: Promise = router.executeAsync( + createUpstreamFixRequest('cancelled'), + client + ); + await hanging.started; + client.abortController.abort(); + const result: IDaemonPhasedRequestResult = await cancelled; + + await hanging.terminated; + expect(result).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(getStatuses(result)).toEqual({ + [OPERATION_A]: OperationStatus.Aborted, + [OPERATION_B]: OperationStatus.Aborted, + [OPERATION_C]: OperationStatus.Skipped, + [OPERATION_D]: OperationStatus.Aborted + }); + expect(result.operationResults.every(({ errorMessage }) => errorMessage === undefined)).toBe(true); + expect(fixture.runners.get(OPERATION_B)?.runCount).toBe(1); + expect(fixture.runners.get(OPERATION_D)?.runCount).toBe(1); + }); + + it('reports an operation that finished in the iteration of a cancelled request with that result', async () => { + // A ignores termination, so it finishes after the cancel, and its result is this request's. + const { fixture, hanging, changeA } = createUpstreamFixFixture(true); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + await router.executeAsync(createUpstreamFixRequest('first'), new TestPhasedRequestClient('one')); + + changeA(); + const client: TestPhasedRequestClient = new TestPhasedRequestClient('two'); + const cancelled: Promise = router.executeAsync( + createUpstreamFixRequest('cancelled'), + client + ); + await hanging.started; + client.abortController.abort(); + hanging.release(); + const result: IDaemonPhasedRequestResult = await cancelled; + + expect(result).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(getStatuses(result)).toEqual({ + [OPERATION_A]: OperationStatus.Success, + [OPERATION_B]: OperationStatus.Aborted, + [OPERATION_C]: OperationStatus.Skipped, + [OPERATION_D]: OperationStatus.Aborted + }); + expect(fixture.runners.get(OPERATION_B)?.runCount).toBe(1); + }); + + it('reports operations that a detached cancelling client never saw start as aborted, not with earlier results', async () => { + const { fixture, hanging, changeA } = createUpstreamFixFixture(); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + await router.executeAsync(createUpstreamFixRequest('first'), new TestPhasedRequestClient('one')); + + changeA(); + const cancelledClient: TestPhasedRequestClient = new TestPhasedRequestClient('two'); + const cancelled: Promise = router.executeAsync( + createUpstreamFixRequest('cancelled'), + cancelledClient + ); + const continuing: Promise = router.executeAsync( + createRequest('continuing', OPERATION_A), + new TestPhasedRequestClient('three') + ); + await hanging.started; + cancelledClient.abortController.abort(); + const result: IDaemonPhasedRequestResult = await cancelled; + hanging.release(); + + expect(result).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(getStatuses(result)).toEqual({ + [OPERATION_A]: OperationStatus.Aborted, + [OPERATION_B]: OperationStatus.Aborted, + [OPERATION_C]: OperationStatus.Skipped, + [OPERATION_D]: OperationStatus.Aborted + }); + expect(getStatuses(await continuing)).toEqual({ [OPERATION_A]: OperationStatus.Success }); + expect(fixture.runners.get(OPERATION_B)?.runCount).toBe(1); + }); + + it('reports every operation as aborted when the cancel comes while the iteration is being scheduled', async () => { + const { fixture, changeA } = createUpstreamFixFixture(); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + await router.executeAsync(createUpstreamFixRequest('first'), new TestPhasedRequestClient('one')); + + changeA(); + const client: TestPhasedRequestClient = new TestPhasedRequestClient('two'); + fixture.graph.hooks.configureIteration.tap('cancel while scheduling', () => { + client.abortController.abort(); + }); + const result: IDaemonPhasedRequestResult = await router.executeAsync( + createUpstreamFixRequest('cancelled'), + client + ); + + expect(result).toMatchObject({ aborted: true, outcome: 'aborted' }); + expect(getStatuses(result)).toEqual({ + [OPERATION_A]: OperationStatus.Aborted, + [OPERATION_B]: OperationStatus.Aborted, + [OPERATION_C]: OperationStatus.Aborted, + [OPERATION_D]: OperationStatus.Aborted + }); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + }); }); From 147473350669608e0ef3baf1171eb6741686bb82 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 02:37:43 +0000 Subject: [PATCH 057/265] [rush-daemon] Tests for a cancel while continuing work, and for the early failure offer Swarm integration step 42; original commit c4693204c1 (merge of swarm/r04-t108-nits-int at 97e473c08e). Scope: task 108(a) NITs. Brings r04's tests for the three gate NITs of task 108(a) (ch01 board 2678). Tests only; they pass on the old product, and ch01's mutants show they're live. s16 batch B, item 2 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree b21b6b1f0b) Commits folded into this step (1): - 97e473c08e Test the three gate NITs of task 108 (a): a cancel while continuing work stops, and the early failure offer's silent and aborted operations (ch01 board 2678) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...m-r04-t108-gate-nits_2026-09-29-01-10.json | 11 +++ .../test/PhasedRequestEarlyFailure.test.ts | 34 +++++-- .../src/test/PhasedRequestEventSink.test.ts | 71 +++++++++++++++ .../test/WorkspaceEarlyFailureResult.test.ts | 90 +++++++++++++++++-- 4 files changed, 192 insertions(+), 14 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t108-gate-nits_2026-09-29-01-10.json diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t108-gate-nits_2026-09-29-01-10.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t108-gate-nits_2026-09-29-01-10.json new file mode 100644 index 0000000000..c84e7ddfc6 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t108-gate-nits_2026-09-29-01-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Test that a client which cancels while the work that continues is stopping gets its answer at once, and that the early failure offer ignores silent operations and gives up once an operation was aborted.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts b/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts index f24c64d808..4daab514cb 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts @@ -69,13 +69,19 @@ interface IEarlyFailureFixture { readonly startedC: Promise; } +/** Runs nothing that the user sees, like a phase that the project does not define. */ +class SilentTestOperationRunner extends TestOperationRunner { + public override readonly silent: boolean = true; +} + /** * B fails and C is slow. Unless `failBeforeCStarts` is set, B fails only once C runs, so C is executing when B's * failure decides a request's result. */ function createEarlyFailureFixture( dependencies: ReadonlyArray = A_CONSUMES_B_AND_C, - failBeforeCStarts: boolean = false + failBeforeCStarts: boolean = false, + silentC: boolean = false ): IEarlyFailureFixture { const startedC: IDeferred = createDeferred(); const releaseC: IDeferred = createDeferred(); @@ -93,10 +99,14 @@ function createEarlyFailureFixture( ], [ OPERATION_C, - new TestOperationRunner(OPERATION_C, OperationStatus.Success, async (): Promise => { - startedC.resolve(); - await releaseC.promise; - }) + new (silentC ? SilentTestOperationRunner : TestOperationRunner)( + OPERATION_C, + OperationStatus.Success, + async (): Promise => { + startedC.resolve(); + await releaseC.promise; + } + ) ] ]), dependencies, @@ -174,6 +184,7 @@ interface IOrdinaryCase { readonly failBeforeCStarts: boolean; readonly name: string; readonly selection: ReadonlyArray; + readonly silentC?: boolean; } /** Requests that ask to return early on failure, but whose failed result can only be written at the end. */ @@ -198,6 +209,15 @@ const ORDINARY_CASES: ReadonlyArray = [ failBeforeCStarts: false, name: 'the request is not a shared build', selection: [OPERATION_A] + }, + { + // An early result would not list C, so it could not say that C continues. + commandName: 'build', + dependencies: A_CONSUMES_B_AND_C, + failBeforeCStarts: false, + name: 'only a silent operation is unfinished', + selection: [OPERATION_A], + silentC: true } ]; @@ -259,8 +279,8 @@ describe('phased requests that return early on failure', () => { it.each(ORDINARY_CASES)( 'writes the result after the iteration when $name', - async ({ commandName, dependencies, failBeforeCStarts, selection }: IOrdinaryCase) => { - const setup: IEarlyFailureFixture = createEarlyFailureFixture(dependencies, failBeforeCStarts); + async ({ commandName, dependencies, failBeforeCStarts, selection, silentC }: IOrdinaryCase) => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(dependencies, failBeforeCStarts, silentC); const tracked: ITrackedClient = trackClient('agent', setup); const resultPromise: Promise = trackResult( setup.router.executeAsync( diff --git a/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts b/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts index 78c41e666a..84eb6ab80b 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestEventSink.test.ts @@ -134,3 +134,74 @@ it('points at the full log of failed operations and operations with warnings, an { operationId: ACTIVE_OPERATION, previousStatus: 'READY', status: 'SUCCESS' } ]); }); + +const TARGET_OPERATION: string = 'project-a (_phase:test)'; +const FAILED_OPERATION: string = 'project-b (_phase:test)'; +const RUNNING_OPERATION: string = 'project-c (_phase:test)'; +const QUEUED_OPERATION: string = 'project-d (_phase:test)'; + +function createEarlyFailureSink(onSettled: (unfinishedOperations: number) => void): PhasedRequestEventSink { + const client: TestPhasedRequestClient = new TestPhasedRequestClient(); + return new PhasedRequestEventSink({ + activeOperationIds: new Set([TARGET_OPERATION, FAILED_OPERATION, RUNNING_OPERATION, QUEUED_OPERATION]), + client, + getNextSequence: () => client.getNextEventSequence(), + onWriteFailure: () => undefined, + rushVersion: '5.178.1', + earlyFailure: { targetOperationIds: new Set([TARGET_OPERATION]), onSettled } + }); +} + +function setStatus(record: IOperationExecutionResult, status: OperationStatus): void { + (record as { status: OperationStatus }).status = status; +} + +/** Schedules the records, then fails FAILED_OPERATION, which blocks the target while RUNNING_OPERATION runs. */ +function failWhileRunning( + sink: PhasedRequestEventSink, + queued: IOperationExecutionResult, + beforeFailure?: () => void +): void { + const target: IOperationExecutionResult = createRecord(TARGET_OPERATION, OperationStatus.Waiting); + const failed: IOperationExecutionResult = createRecord(FAILED_OPERATION, OperationStatus.Executing); + sink.onIterationScheduled([ + target, + failed, + createRecord(RUNNING_OPERATION, OperationStatus.Executing), + queued + ]); + beforeFailure?.(); + setStatus(failed, OperationStatus.Failure); + sink.onOperationStatusChanged(failed, OperationStatus.Executing); + // Blocked operations complete only when the iteration ends. + setStatus(target, OperationStatus.Blocked); + sink.onOperationCompleted(failed); +} + +describe('the early failure offer', () => { + it('counts the unfinished operations that are not silent', () => { + const onSettled: jest.Mock = jest.fn(); + // Like a phase that the project does not define: the result does not list it while it is unfinished. + const silent: IOperationExecutionResult = { + ...createRecord(QUEUED_OPERATION, OperationStatus.Ready), + silent: true + } as IOperationExecutionResult; + + failWhileRunning(createEarlyFailureSink(onSettled), silent); + + expect(onSettled.mock.calls).toEqual([[1]]); + }); + + it('offers nothing once one of its operations was aborted, even before that operation completed', () => { + const onSettled: jest.Mock = jest.fn(); + const queued: IOperationExecutionResult = createRecord(QUEUED_OPERATION, OperationStatus.Ready); + + // An aborted iteration marks the queued operations that it skips as aborted at once, but they complete only + // when the iteration ends, after the operations that were already running. + failWhileRunning(createEarlyFailureSink(onSettled), queued, () => + setStatus(queued, OperationStatus.Aborted) + ); + + expect(onSettled).not.toHaveBeenCalled(); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts b/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts index 2cda8f5ef8..b5b2443584 100644 --- a/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts @@ -11,7 +11,7 @@ import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; -import type { ITerminalExchange } from './DaemonRequestWireTestUtilities'; +import type { DaemonRequestWireClient, ITerminalExchange } from './DaemonRequestWireTestUtilities'; import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; @@ -19,11 +19,29 @@ jest.setTimeout(60_000); const BUILD_B: string[] = ['build', '--to', 'b', '--parallelism', '3']; +interface IEarlyFailureFixtureOptions { + readonly restartable?: boolean; + readonly slowToStop?: boolean; +} + +/** Keeps the output that it inherits open until the test removes the `hold` marker. */ +const HOLD_OUTPUT_SCRIPT: string = + "const fs=require('node:fs');const t=setInterval(()=>{if(!fs.existsSync('../hold'))clearInterval(t);},20);"; + /** * b consumes a and c. a fails, and c holds its build open until the test removes the `hold` marker, so a build of b - * that returns early on failure leaves c running. + * that returns early on failure leaves c running. With `slowToStop`, c first starts a detached process that shares + * its output, like a stray watcher: stopping c kills c's process group but not that process, so c's operation ends + * only when the test removes the marker. */ -function createEarlyFailureFixtureAsync(restartable: boolean = false): Promise { +function createEarlyFailureFixtureAsync({ + restartable = false, + slowToStop = false +}: IEarlyFailureFixtureOptions = {}): Promise { + const startOutputHolder: string = slowToStop + ? `require('node:child_process').spawn(process.execPath,['-e',${JSON.stringify(HOLD_OUTPUT_SCRIPT)}],` + + "{detached:true,stdio:['ignore','inherit','inherit']}).unref();" + : ''; return DaemonGraphTestFixture.createAsync((created: DaemonGraphTestFixture) => { if (restartable) { setDaemonPolicy(created, {}); @@ -45,7 +63,9 @@ function createEarlyFailureFixtureAsync(restartable: boolean = false): Promise{if(!fs.existsSync('../hold')){clearInterval(t);console.log('finished-c');}},20);" ); }); @@ -55,9 +75,21 @@ function countRuns(fixture: DaemonGraphTestFixture, name: string): number { return fixture.runs().filter((run: string) => run === name).length; } -async function waitForRunsAsync(fixture: DaemonGraphTestFixture, name: string, count: number): Promise { +async function waitUntilAsync(condition: () => boolean): Promise { const deadline: number = Date.now() + 30_000; - while (countRuns(fixture, name) < count && Date.now() < deadline) await delayAsync(20); + while (!condition() && Date.now() < deadline) await delayAsync(20); +} + +async function waitForRunsAsync(fixture: DaemonGraphTestFixture, name: string, count: number): Promise { + await waitUntilAsync(() => countRuns(fixture, name) >= count); +} + +/** Counts the stops that the daemon requested; every iteration's start also aborts, without options. */ +function countTerminatingAborts(abortSpy: jest.SpyInstance): number { + return abortSpy.mock.calls.filter( + ([options]: ReadonlyArray<{ terminateRunning?: boolean } | undefined>) => + options?.terminateRunning === true + ).length; } /** Whether an in-process Rush command could take the Rush lock that the daemon holds while it executes. */ @@ -178,6 +210,50 @@ describe('a failed build that returns early', () => { } }); + it('answers a client that cancels while the work that continues is stopping, without waiting for it to stop', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync({ slowToStop: true }); + const hold: string = path.join(fixture.folder, 'hold'); + try { + await returnEarlyAsync(fixture); + const abortSpy: jest.SpyInstance = jest.spyOn( + fixture.session.operationGraph!, + 'abortCurrentIterationAsync' + ); + const client: DaemonRequestWireClient = await fixture.connectAsync(); + try { + const custom: IDaemonRequestEnvelope = fixture.envelope(['test', '--to', 'c'], { + commandOrigin: 'custom' + }); + await client.sendControlAsync({ kind: 'requestStart', payload: custom }); + // Before it rejects `test`, the daemon stops c, which takes until the test removes the marker. + await waitUntilAsync(() => countTerminatingAborts(abortSpy) > 0); + expect(countTerminatingAborts(abortSpy)).toBe(1); + const answer: Promise = client.readTerminalAsync(custom.requestId); + expect(await Promise.race([answer, delayAsync(500).then(() => undefined)])).toBeUndefined(); + + // rush-client cancels on Ctrl+C, and gives up on the daemon if it does not answer within 5 seconds. + await client.sendControlAsync({ kind: 'requestCancel', payload: { requestId: custom.requestId } }); + const cancelled: ITerminalExchange | undefined = await Promise.race([ + answer, + delayAsync(3_000).then(() => undefined) + ]); + expect(cancelled?.terminal).toMatchObject({ + kind: 'requestRejected', + payload: { code: 'unsupported' } + }); + // c is still stopping, so the client must not run `test` in-process now; rush-client reports a command + // that it cancelled as cancelled instead. + expect(fixture.session.operationGraph?.status).toBe(OperationStatus.Executing); + expect(isNativeLockFree(fixture)).toBe(false); + } finally { + await client.closeAsync(); + } + } finally { + fs.rmSync(hold, { force: true }); + await fixture[Symbol.asyncDispose](); + } + }); + it('leaves the work that continues running for a read-only command or a rushx script that it rejects', async () => { const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); const hold: string = path.join(fixture.folder, 'hold'); @@ -216,7 +292,7 @@ describe('a failed build that returns early', () => { }); it('lets a restart for another environment proceed without waiting for the work that continues', async () => { - const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(true); + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync({ restartable: true }); const hold: string = path.join(fixture.folder, 'hold'); try { const before = await pongAsync(fixture); From e178316649e3423792ab0a018758f443634302d4 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 02:37:43 +0000 Subject: [PATCH 058/265] [rush-client-core] The client relaunches the daemon after a startup helper exited, once its relaunch time passed Swarm integration step 43; original commit e073bd0f4b (merge of swarm/r03-t126 at da64013877). Scope: task 126. Brings r03's task 126: a client takes over a startup reservation whose helper is provably gone and whose daemon never became ready, instead of running in-process until `daemon stop --force` (t05 board 1574 and board 1652, row 13a). Second agent: t07 board 2751. ch01's e2e: the reservation is taken over after the relaunch time. s16 batch B, item 3 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree a7707a430e) Commits folded into this step (1): - da64013877 rush-client-core: relaunch the daemon after a startup helper exited, once its relaunch time passed (task 126) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 15 +- apps/rush-cli-client/src/daemonCommands.ts | 5 +- .../src/test/launchClient.test.ts | 13 +- ...reservation-relaunch_2026-09-28-23-50.json | 11 + ...reservation-relaunch_2026-09-28-23-50.json | 11 + common/reviews/api/rush-client-core.api.md | 1 + libraries/rush-client-core/README.md | 36 ++- .../rush-client-core/src/DaemonOwnership.ts | 7 +- .../rush-client-core/src/DaemonStartup.ts | 12 +- .../src/DaemonStartupReservation.ts | 62 ++++- .../src/connectOrStartDaemon.ts | 56 +++- .../src/test/DaemonStartup.test.ts | 141 ++++++++++ .../test/connectOrAwaitDaemonStartup.test.ts | 5 +- .../src/test/connectOrStartDaemon.test.ts | 244 +++++++++++++++++- .../src/test/fixtures/daemon.ts | 12 +- 15 files changed, 573 insertions(+), 58 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json create mode 100644 common/changes/@rushstack/rush-client-core/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json create mode 100644 libraries/rush-client-core/src/test/DaemonStartup.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 12c589e650..75f5aa6bda 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -413,12 +413,16 @@ the next command: status then prints `state: "installationChanged"` with the pon A startup reservation (`.pid.json.starting`) refuses another daemon launch until the daemon it reserved becomes ready. Status reports one that remains as `startupReservation` with its `path`, the startup helper's `helperPid` when recorded, and -`helperState`: `running` (the helper still waits for readiness), `exited` (nothing else will +`helperState`: `running` (the helper still waits for readiness), `exited` (the helper will not release it), or `unknown` (written by an older client). Status never removes it. Next to a ready daemon, the next command that uses, stops or restarts that daemon removes it; when status -cannot connect, its diagnostic explains the reservation. After an `exited` helper, every automatic start is -refused at once unless that daemon still becomes ready: check `daemon logs`, and if the daemon -failed to start, run `daemon stop --force`. +cannot connect, its diagnostic explains the reservation. After an `exited` helper, status also +reports `relaunchAfter`, 15 seconds after that helper was launched. Until then every automatic +start is refused at once (the command runs in-process), so that a daemon that fails the same way +each time, for example because of a configuration error, is not launched by every command. The +first command after it that finds nothing listening at the endpoint takes the reservation over +and starts the daemon again, and `daemon logs` shows a line saying so. `daemon logs` may also show +why the daemon did not become ready. The optional workspace snapshot reports the provider generation/token, graph existence, and available warm accounting without initializing a graph. Missing fields are unknown, @@ -478,7 +482,8 @@ started/reused successor must pass hello/ping before reporting `state: "ready"`. When nothing listens at the endpoint, restart starts a daemon exactly like `daemon start`. Automatic and explicit startup reclaim stale artifacts only when that is provably -safe: while holding the start mutex with no `.starting` reservation, a socket +safe: while holding the start mutex with no `.starting` reservation (or after taking over +one whose helper exited, as described above), a socket without an ownership record, or an unreadable/corrupt record, is removed only after a connection attempt is refused (so no listener exists). On Linux, a record whose PID now belongs to a process that started after the record's `startedAt` (PID reuse) diff --git a/apps/rush-cli-client/src/daemonCommands.ts b/apps/rush-cli-client/src/daemonCommands.ts index adea0f631c..a81d671168 100644 --- a/apps/rush-cli-client/src/daemonCommands.ts +++ b/apps/rush-cli-client/src/daemonCommands.ts @@ -217,7 +217,8 @@ function explainExitedDaemon(error: unknown, paths: IDaemonPaths): unknown { /** * Reports a startup reservation, which refuses another daemon launch until it is resolved. Clients resolve it * once the daemon it reserved is ready, so the next command that uses, stops or restarts a ready daemon resolves - * a remaining one; status only reports it. + * a remaining one. Once its helper exited, a command that starts the daemon after `relaunchAfter` takes it over. + * Status only reports it. */ function getStartupReservationStatus(paths: IDaemonPaths): { startupReservation?: IDaemonStartupReservationInfo; @@ -238,7 +239,7 @@ function explainStartupReservation(error: unknown, paths: IDaemonPaths): unknown explanation = `A daemon is starting: ${helper} is still waiting for it to become ready; retry shortly.`; break; case 'exited': - explanation = `The startup reservation at ${reservation.path} remains, but ${helper} exited before the daemon became ready, so the reservation refuses every automatic start unless that daemon still becomes ready. Check "rush-client daemon logs"; if the daemon failed to start, run "rush-client daemon stop --force" to remove it.`; + explanation = `The startup reservation at ${reservation.path} remains, but ${helper} exited before the daemon became ready. A command that starts the daemon after ${reservation.relaunchAfter} takes the reservation over and launches the daemon again, provided that nothing listens at ${paths.socketPath} then. "rush-client daemon logs" may show why the daemon did not become ready.`; break; default: explanation = `The startup reservation at ${reservation.path} refuses another daemon launch. Check "rush-client daemon logs"; if no daemon is starting, run "rush-client daemon stop --force" to remove it.`; diff --git a/apps/rush-cli-client/src/test/launchClient.test.ts b/apps/rush-cli-client/src/test/launchClient.test.ts index 0e52e5a8bc..e7e0a2a7e7 100644 --- a/apps/rush-cli-client/src/test/launchClient.test.ts +++ b/apps/rush-cli-client/src/test/launchClient.test.ts @@ -400,16 +400,18 @@ describe('standalone rushx fallback', () => { await once(exited, 'close'); helperPid = exited.pid!; } + const helperStartedAt: Date = new Date(); fs.writeFileSync( reservation, - JSON.stringify({ token: 'fixture', helperPid, helperStartedAt: new Date().toISOString() }) + JSON.stringify({ token: 'fixture', helperPid, helperStartedAt: helperStartedAt.toISOString() }) ); + const relaunchAfter: string = new Date(helperStartedAt.getTime() + 15000).toISOString(); const status: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'status']); expect(status).toMatchObject({ code: 1, stdout: '' }); expect(status.stderr).toContain('Could not connect to daemon'); expect(status.stderr).toContain( helperState === 'exited' - ? `its startup helper (PID ${helperPid}) exited before the daemon became ready, so the reservation refuses every automatic start unless that daemon still becomes ready. Check "rush-client daemon logs"; if the daemon failed to start, run "rush-client daemon stop --force" to remove it.` + ? `its startup helper (PID ${helperPid}) exited before the daemon became ready. A command that starts the daemon after ${relaunchAfter} takes the reservation over and launches the daemon again, provided that nothing listens at ${paths.socketPath} then. "rush-client daemon logs" may show why the daemon did not become ready.` : `A daemon is starting: its startup helper (PID ${helperPid}) is still waiting for it to become ready; retry shortly.` ); const stopped: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop']); @@ -417,7 +419,12 @@ describe('standalone rushx fallback', () => { expect(JSON.parse(stopped.stdout)).toEqual({ state: 'notRunning', socketPath: paths.socketPath, - startupReservation: { path: reservation, helperPid, helperState } + startupReservation: { + path: reservation, + helperPid, + helperState, + ...(helperState === 'exited' ? { relaunchAfter } : {}) + } }); const reset: IInvocationResult = await invokeAsync(true, false, false, ['daemon', 'stop', '--force']); expect(reset).toMatchObject({ code: 0, stderr: '' }); diff --git a/common/changes/@rushstack/rush-cli-client/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json b/common/changes/@rushstack/rush-cli-client/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json new file mode 100644 index 0000000000..f9c2f5bdbd --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "After a daemon failed to start, the first command 15 seconds or more later starts it again instead of running in-process until `rush-client daemon stop --force`. `daemon status` and `daemon stop` report the `relaunchAfter` time of such a startup reservation.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json b/common/changes/@rushstack/rush-client-core/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json new file mode 100644 index 0000000000..042ae2be96 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-abandoned-reservation-relaunch_2026-09-28-23-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Take over a daemon startup reservation whose helper exited once 15 seconds have passed since that launch and nothing listens at the endpoint, and start the daemon again, instead of refusing every automatic start until `resetDaemonArtifactsAsync()`. Until then another launch is still refused at once. The startup helper now also releases its reservation for a daemon that another launch published while its own launcher exited. `inspectDaemonStartupReservation()` reports the `relaunchAfter` time.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 48c0cafbdb..76e0bf211f 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -171,6 +171,7 @@ export interface IDaemonStartupReservationInfo { readonly helperPid?: number; readonly helperState: DaemonStartupHelperState; readonly path: string; + readonly relaunchAfter?: string; } // @beta diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 41581c27bd..6461b1db0e 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -86,8 +86,8 @@ at least 120 seconds, even when the requesting client's own deadline is shorter, first start (for example while Windows scans newly installed files) is still handed off to later clients instead of leaving an abandoned reservation. Clients still await hello/pong under bounded backoff. Stdout/stderr go to `.log`. No PID -is killed. While holding the mutex with no startup reservation, stale leftovers are -reclaimed only when provably safe: a socket without an ownership record, or a corrupt +is killed. While holding the mutex with no startup reservation (or after taking over an +abandoned one, see below), stale leftovers are reclaimed only when provably safe: a socket without an ownership record, or a corrupt record, once a connection attempt is refused (no listener exists); and, on Linux, a record whose PID now belongs to a process that started after the record's `startedAt` (PID reuse, detected from `/proc`). Any other live PID with an unreachable socket fails @@ -96,14 +96,23 @@ which removes the record, socket and reservation after the same no-listener/no-l The helper uses a stable tool cwd, and the starting client awaits its exit after readiness. The explicit launcher's cwd is unchanged. -An unresolved startup reservation is never automatically reclaimed based on PID -liveness or elapsed time. If the helper cannot establish readiness, subsequent starts -fail closed instead of risking a second detached daemon. Only a known spawn failure -(no executable started) releases the reservation immediately. An arbitrary launcher -can spawn descendants, so its exit is not proof that another launch is safe. -Recovery of an abandoned reservation requires operator confirmation that the original -startup cannot still publish an endpoint; normal successful startup releases it -automatically. Cancellation stops the client waiting, not the detached handoff. +While its helper runs, a startup reservation is never taken over, however long startup +takes. Only a known spawn failure (no executable started) releases the reservation +immediately. If the helper cannot establish readiness (its launcher exited, or it timed +out), it exits and leaves the reservation. An arbitrary launcher can spawn descendants, so +neither exit proves that no daemon can still publish an endpoint. A reservation whose helper +is provably gone is therefore taken over only once its relaunch time has passed (15 seconds +after the helper was launched) and while nothing listens at the endpoint; the starting client +logs that it took the reservation over and launches the daemon again. This is safe even if a +daemon of the gone helper is still starting: a daemon listens before it publishes the endpoint, +publishes it only with link(2) (or as the first instance of a named pipe), and reclaims it +only from a dead owner that does not accept a connection. So of two such daemons, the one +that publishes second finds the other and exits, and the readiness of either releases the new +reservation. Before the relaunch time, clients refuse another launch at once, so that a +daemon that fails the same way each time (for example because of a configuration error) is +not launched by every client, and their callers can run without it. Normal successful startup +releases the reservation automatically. Cancellation stops the client waiting, not the +detached handoff. A client resolves a reservation on the same evidence the helper waits for, so a daemon that became ready after its helper stopped waiting (for example a first start slower than @@ -114,8 +123,11 @@ Reservations written by older clients, without a helper, are resolved the same w The recorded helper decides how long a refused launch waits: while it is alive, a starting client waits for it until the client's own deadline; once it is provably gone (its PID no longer exists or was reused), nothing else can release the reservation, so clients -refuse another launch at once. `inspectDaemonStartupReservation(paths)` reports the -reservation and its helper's state without changing it (`rush-client daemon status`), and +refuse another launch at once until the relaunch time, and then take the reservation over +as described above (while something listens at the endpoint, they wait for it until their +deadline instead). `inspectDaemonStartupReservation(paths)` reports the reservation, its +helper's state and, once the helper exited, the relaunch time (`relaunchAfter`) without +changing it (`rush-client daemon status`), and `requestDaemonShutdownAsync()` resolves a reservation for the attested daemon before it sends shutdown, so that its successor can start. `resolveDaemonStartupReservationAsync(client, paths)` does the same for a caller that stops the daemon without replacing it (`rush-client diff --git a/libraries/rush-client-core/src/DaemonOwnership.ts b/libraries/rush-client-core/src/DaemonOwnership.ts index c48949b75f..108d46712c 100644 --- a/libraries/rush-client-core/src/DaemonOwnership.ts +++ b/libraries/rush-client-core/src/DaemonOwnership.ts @@ -94,9 +94,10 @@ export function isOwnerProcessAlive(owner: { readonly pid: number; readonly star } /** - * Makes stale ownership reclaimable when that is provably safe. The caller must hold the start mutex and - * have observed no startup reservation, so no legitimate daemon can be between bind and record publication; - * a refused connection then proves no listener exists. + * Makes stale ownership reclaimable when that is provably safe. The caller must hold the start mutex and have + * observed no startup reservation, or taken over one whose helper is gone. A daemon publishes its endpoint only + * after it listens, so a refused connection then proves no listener exists. A daemon that a gone helper + * launched may still publish later; of two daemons that publish, the second finds the first and exits. */ export async function reclaimAbandonedOwnershipAsync(paths: IDaemonPaths): Promise { const state: OwnershipState = inspectOwnership(paths.lockfilePath); diff --git a/libraries/rush-client-core/src/DaemonStartup.ts b/libraries/rush-client-core/src/DaemonStartup.ts index 56b01abdd4..db21e00d76 100644 --- a/libraries/rush-client-core/src/DaemonStartup.ts +++ b/libraries/rush-client-core/src/DaemonStartup.ts @@ -153,9 +153,9 @@ function isNotFound(error: unknown): boolean { /** * Runs independently of the requesting client. Once spawn succeeds, only protocol readiness releases * the reservation: an arbitrary launcher may outlive its parent or spawn descendants. - * Failure before readiness deliberately leaves a durable reservation instead of guessing that a PID is safe. - * The reservation records this helper, so once it exits, later clients report the retained reservation at once - * instead of waiting for a release that cannot happen. + * Failure before readiness deliberately leaves the reservation instead of guessing that a PID is safe. + * The reservation records this helper, so once it exits, the next client that starts the daemon takes the + * reservation over instead of waiting for a release that cannot happen. */ export async function runDaemonStartupAsync(options: IDaemonStartupOptions): Promise { const { paths, startCommand: start, token, timeoutMs } = options; @@ -183,6 +183,10 @@ export async function runDaemonStartupAsync(options: IDaemonStartupOptions): Pro const deadline: number = Date.now() + timeoutMs; let backoffMs: number = 50; while (Date.now() < deadline) { + // Sampled before connecting: a launcher can exit because another daemon published this endpoint first (for + // example one that an abandoned reservation's helper launched before a client took the reservation over). + // That daemon already listens by then, so the attempt below finds it. + const launcherExited: boolean = child.exitCode !== null || child.signalCode !== null; let client: DaemonClient | undefined; try { client = await DaemonClient.connectAsync({ @@ -209,7 +213,7 @@ export async function runDaemonStartupAsync(options: IDaemonStartupOptions): Pro } return; } - if (child.exitCode !== null || child.signalCode !== null) { + if (launcherExited) { await closed; throw new DaemonClientError( 'startupFailed', diff --git a/libraries/rush-client-core/src/DaemonStartupReservation.ts b/libraries/rush-client-core/src/DaemonStartupReservation.ts index 301a21cff6..79d0a327ee 100644 --- a/libraries/rush-client-core/src/DaemonStartupReservation.ts +++ b/libraries/rush-client-core/src/DaemonStartupReservation.ts @@ -8,7 +8,7 @@ import { } from '@rushstack/rush-daemon-transport'; import type { DaemonClient } from './DaemonClient'; -import { isDaemonOwnership, isOwnerProcessAlive } from './DaemonOwnership'; +import { isDaemonOwnership, isEndpointUnboundAsync, isOwnerProcessAlive } from './DaemonOwnership'; import { getDaemonStartupFilePath, readDaemonStartupReservation, @@ -18,11 +18,20 @@ import { } from './DaemonStartup'; import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; +/** + * How long after a launch a client that starts the daemon may take over its reservation once the helper exited. + * It matches the default startup timeout. A daemon that fails to start the same way each time, for example + * because of a configuration error, is then launched at most once per interval however many clients start it; + * the others are refused at once and can run without it. + */ +const ABANDONED_STARTUP_RELAUNCH_DELAY_MS: number = 15000; + /** * What a startup reservation's recorded helper can still do. `running`: it may still release the reservation. - * `exited`: it is provably gone, so only a client that finds the daemon ready, or - * `resetDaemonArtifactsAsync()`, removes the reservation. `unknown`: the reservation records no helper, - * for example because an older client wrote it. + * `exited`: it is provably gone, so it never will. A client that finds the daemon ready removes the reservation, + * and so does a client that starts the daemon after `relaunchAfter` while nothing listens at the endpoint (it + * takes the reservation over). `unknown`: the reservation records no helper, for example because an older client + * wrote it. * @beta */ export type DaemonStartupHelperState = 'running' | 'exited' | 'unknown'; @@ -35,6 +44,11 @@ export interface IDaemonStartupReservationInfo { readonly helperPid?: number; /** Whether the helper can still release the reservation. */ readonly helperState: DaemonStartupHelperState; + /** + * When the helper exited: the time (ISO 8601) after which a client that starts the daemon takes the reservation + * over and launches the daemon again, provided that nothing listens at the endpoint then. + */ + readonly relaunchAfter?: string; } /** @@ -48,10 +62,14 @@ export function inspectDaemonStartupReservation( const reservation: IDaemonStartupReservation | undefined = readDaemonStartupReservation(paths); if (!reservation) return undefined; const { helper } = reservation; + const helperState: DaemonStartupHelperState = getStartupHelperState(reservation); return { path: getDaemonStartupFilePath(paths), ...(helper ? { helperPid: helper.pid } : {}), - helperState: getStartupHelperState(reservation) + helperState, + ...(helper && helperState === 'exited' + ? { relaunchAfter: new Date(getStartupRelaunchTime(helper)).toISOString() } + : {}) }; } @@ -60,6 +78,14 @@ export function getStartupHelperState(reservation: IDaemonStartupReservation): D return isStartupHelperAlive(reservation.helper) ? 'running' : 'exited'; } +/** + * The time (milliseconds since the epoch) after which a client that starts the daemon may take over a reservation + * of `helper` once that helper exited. The helper was recorded when it was launched. + */ +export function getStartupRelaunchTime(helper: IDaemonStartupHelper): number { + return Date.parse(helper.startedAt) + ABANDONED_STARTUP_RELAUNCH_DELAY_MS; +} + function isStartupHelperAlive(helper: IDaemonStartupHelper): boolean { try { return isOwnerProcessAlive(helper); @@ -86,6 +112,32 @@ export function resolveStartupReservationForReadyDaemon( return removeDaemonStartupIfUnchanged(paths, reservation); } +/** + * Removes a reservation whose helper is provably gone, once its relaunch time has passed and while nothing listens + * at the endpoint, so that the caller can launch the daemon again. Only the helper releases a reservation, so this + * one would otherwise refuse every automatic start. The caller must hold the start mutex, which keeps other clients + * from taking it over or from reserving startup at the same time. + * @remarks A daemon that the gone helper launched may still be starting. Launching another is still safe: a daemon + * publishes the endpoint only with link(2), or as the first instance of a named pipe, and only after it listens, + * and it reclaims the endpoint only from an owner that is dead and does not accept a connection. So whichever + * daemon publishes second finds the other and exits, and a helper releases its reservation once either daemon + * completes hello/ping. + * @returns true when this call removed `reservation`. + */ +export async function tryTakeOverAbandonedStartupReservationAsync( + paths: IDaemonPaths, + reservation: IDaemonStartupReservation +): Promise { + const { helper } = reservation; + if (!helper || getStartupHelperState(reservation) !== 'exited') return false; + if (Date.now() < getStartupRelaunchTime(helper)) return false; + if (!(await isEndpointUnboundAsync(paths.socketPath))) return false; + // The helper may have released its reservation just before it exited. + const current: IDaemonStartupReservation | undefined = readDaemonStartupReservation(paths); + if (!current || current.contents !== reservation.contents) return false; + return removeDaemonStartupIfUnchanged(paths, current); +} + /** * {@link resolveStartupReservationForReadyDaemon} for a connected client, taking the start mutex without waiting. * Returns false, keeping the reservation, while another client or this process holds the mutex. diff --git a/libraries/rush-client-core/src/connectOrStartDaemon.ts b/libraries/rush-client-core/src/connectOrStartDaemon.ts index 369ec81128..3d0dfadc6a 100644 --- a/libraries/rush-client-core/src/connectOrStartDaemon.ts +++ b/libraries/rush-client-core/src/connectOrStartDaemon.ts @@ -38,13 +38,16 @@ import { getDaemonStartupFilePath, readDaemonStartupReservation, reserveDaemonStartup, + type IDaemonStartupHelper, type IDaemonStartupOptions, type IDaemonStartupReservation } from './DaemonStartup'; import { getStartupHelperState, + getStartupRelaunchTime, resolveStartupReservationForReadyDaemon, tryResolveStartupReservationAsync, + tryTakeOverAbandonedStartupReservationAsync, type DaemonStartupHelperState } from './DaemonStartupReservation'; import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; @@ -180,7 +183,10 @@ async function startDaemonAsync( } if (!lock) throw startupError(options, 'timed out waiting for another starting client'); try { - await waitForStartupReservationAsync(options, deadline); + const abandonedHelper: IDaemonStartupHelper | undefined = await waitForStartupReservationAsync( + options, + deadline + ); const ready: DaemonClient | undefined = await tryConnectAsync(options, deadline); if (ready) return ready; const replacement: DaemonClient | undefined = await replaceMismatchedDaemonAsync(options, deadline); @@ -192,7 +198,7 @@ async function startDaemonAsync( await reclaimStaleDaemonAsync(options.paths); if (Date.now() >= deadline) throw startupError(options, 'exceeded its deadline before spawn'); options.abortSignal?.throwIfAborted(); - const helper: IStartupHelper = await spawnDetachedAsync(options, deadline); + const helper: IStartupHelper = await spawnDetachedAsync(options, deadline, abandonedHelper); const { child } = helper; backoffMs = 50; while (Date.now() < deadline) { @@ -229,24 +235,37 @@ async function startDaemonAsync( /** * Holding the start mutex, waits until no startup reservation remains. The helper releases its reservation once * the daemon completes hello/ping, and this client resolves it on the same evidence, so a daemon that became - * ready after its helper stopped waiting is still used. Another launch is refused while the reservation remains: - * at once when its helper exited, since nothing else will release it, and otherwise at the deadline. + * ready after its helper stopped waiting is still used. Once the helper is provably gone, nothing else will + * release the reservation: this client refuses another launch at once until the reservation's relaunch time, and + * then takes the reservation over as soon as nothing listens at the endpoint. Otherwise another launch is refused + * at the deadline. + * @returns the helper of a reservation that this client took over, if any. */ async function waitForStartupReservationAsync( options: IConnectOrStartDaemonOptions, deadline: number -): Promise { +): Promise { while (true) { options.abortSignal?.throwIfAborted(); const reservation: IDaemonStartupReservation | undefined = readDaemonStartupReservation(options.paths); - if (!reservation) return; + if (!reservation) return undefined; if (await tryResolveForReadyDaemonAsync(options, deadline)) continue; + if (await tryTakeOverAbandonedStartupReservationAsync(options.paths, reservation)) + return reservation.helper; const helperState: DaemonStartupHelperState = getStartupHelperState(reservation); - if (helperState === 'exited' || Date.now() >= deadline) { - // The helper may have released its reservation just before it exited. + const now: number = Date.now(); + const relaunchTime: number | undefined = + helperState === 'exited' ? getStartupRelaunchTime(reservation.helper!) : undefined; + const pendingRelaunchTime: number | undefined = + relaunchTime !== undefined && now < relaunchTime ? relaunchTime : undefined; + if (now >= deadline || pendingRelaunchTime !== undefined) { + // The helper may have released its reservation just before it exited or before the deadline. const current: IDaemonStartupReservation | undefined = readDaemonStartupReservation(options.paths); if (!current || current.contents !== reservation.contents) continue; - throw startupError(options, describeUnresolvedReservation(options.paths, reservation, helperState)); + throw startupError( + options, + describeUnresolvedReservation(options.paths, reservation, helperState, pendingRelaunchTime) + ); } await delayAsync(Math.min(100, Math.max(1, deadline - Date.now())), undefined, { signal: options.abortSignal @@ -279,16 +298,20 @@ async function tryResolveForReadyDaemonAsync( } } +/** `pendingRelaunchTime` is the relaunch time of a helper that exited, while that time has not passed yet. */ function describeUnresolvedReservation( paths: IDaemonPaths, reservation: IDaemonStartupReservation, - helperState: DaemonStartupHelperState + helperState: DaemonStartupHelperState, + pendingRelaunchTime: number | undefined ): string { const prefix: string = `has an unresolved startup handoff at ${getDaemonStartupFilePath(paths)}`; const helperPid: number | undefined = reservation.helper?.pid; switch (helperState) { case 'exited': - return `${prefix}: its startup helper (PID ${helperPid}) exited before the daemon became ready; refusing another launch. ${DAEMON_RESET_HINT}`; + return pendingRelaunchTime !== undefined + ? `${prefix}: its startup helper (PID ${helperPid}) exited before the daemon became ready; refusing another launch until ${new Date(pendingRelaunchTime).toISOString()}, so that a daemon that cannot start is not launched by every command.` + : `${prefix}: its startup helper (PID ${helperPid}) exited before the daemon became ready, but a process still accepts connections at ${paths.socketPath}; refusing another launch. ${DAEMON_RESET_HINT}`; case 'running': return `${prefix}: its startup helper (PID ${helperPid}) is still waiting for the daemon to become ready; refusing another launch`; default: @@ -604,7 +627,8 @@ async function waitForPreviousDaemonAsync( async function spawnDetachedAsync( options: IConnectOrStartDaemonOptions, - deadline: number + deadline: number, + abandonedHelper: IDaemonStartupHelper | undefined ): Promise { const start: IDaemonStartCommand = { ...options.startCommand!, @@ -633,6 +657,14 @@ async function spawnDetachedAsync( } fs.fchmodSync(logFd, 0o600); } + if (abandonedHelper) { + fs.writeSync( + logFd, + `${new Date().toISOString()} rush-client (PID ${process.pid}): took over the startup reservation of ` + + `startup helper PID ${abandonedHelper.pid} (started ${abandonedHelper.startedAt}), which exited ` + + 'before the daemon became ready; starting the daemon again.\n' + ); + } let helper: IStartupHelper | undefined; try { const child: ChildProcess = spawn(process.execPath, [path.join(__dirname, 'runDaemonStartup.js')], { diff --git a/libraries/rush-client-core/src/test/DaemonStartup.test.ts b/libraries/rush-client-core/src/test/DaemonStartup.test.ts new file mode 100644 index 0000000000..15d5eba954 --- /dev/null +++ b/libraries/rush-client-core/src/test/DaemonStartup.test.ts @@ -0,0 +1,141 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as net from 'node:net'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { + DAEMON_PROTOCOL_VERSION, + DaemonFrameType, + decodeDaemonControlMessage, + encodeDaemonControlMessage, + type DaemonControlMessage +} from '@rushstack/rush-daemon-protocol'; +import { DaemonFrameConnection, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { getDaemonStartupFilePath, reserveDaemonStartup, runDaemonStartupAsync } from '../DaemonStartup'; +import { removeTestFolderAsync } from './TestProcessExit'; + +async function waitUntilAsync(condition: () => boolean, description: string): Promise { + const deadline: number = Date.now() + 5000; + while (!condition()) { + if (Date.now() >= deadline) throw new Error(`Timed out waiting until ${description}.`); + await delayAsync(5); + } +} + +function isProcessGone(pid: number): boolean { + try { + process.kill(pid, 0); + return false; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ESRCH') return true; + throw error; + } +} + +describe(runDaemonStartupAsync.name, () => { + let folder: string; + let paths: IDaemonPaths; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-client-startup-')); + paths = { + runtimeDir: folder, + socketPath: + process.platform === 'win32' + ? `\\\\.\\pipe\\rush-client-${path.basename(folder)}` + : path.join(folder, 'd.sock'), + lockfilePath: path.join(folder, 'daemon.pid.json') + }; + }); + + afterEach(async () => { + await removeTestFolderAsync(folder); + }); + + it('releases its reservation for a ready daemon although its own launcher exited during the last attempt', async () => { + // Like a daemon that another launch published while this helper's launcher found it and exited: the launcher + // exits while a connection attempt is pending, that attempt fails, and the next one finds a ready daemon. + const pending: DaemonFrameConnection[] = []; + const closedConnections: Set = new Set(); + let answering: boolean = false; + const sockets: Set = new Set(); + const server: net.Server = net.createServer((socket) => { + sockets.add(socket); + socket.on('close', () => sockets.delete(socket)); + const connection: DaemonFrameConnection = new DaemonFrameConnection(socket); + connection.onClosed(() => closedConnections.add(connection)); + connection.onFrame(async (frame) => { + if (frame.kind !== DaemonFrameType.controlJson) return; + const message: DaemonControlMessage = decodeDaemonControlMessage(frame.payload); + let reply: DaemonControlMessage | undefined; + if (message.kind === 'hello') { + if (!answering) { + pending.push(connection); + return; + } + reply = { + kind: 'helloAck', + payload: { protocolVersion: DAEMON_PROTOCOL_VERSION, sessionId: 'test' } + }; + } else if (message.kind === 'ping') { + reply = { kind: 'pong', payload: { uptimeMs: 1, daemonVersion: 'test' } }; + } + if (reply) { + await connection.sendFrameAsync({ + kind: DaemonFrameType.controlJson, + payload: encodeDaemonControlMessage(reply) + }); + } + }); + }); + await new Promise((resolve) => server.listen(paths.socketPath, resolve)); + try { + const token: string = reserveDaemonStartup(paths, { + pid: process.pid, + startedAt: new Date().toISOString() + }); + const launcherPidPath: string = path.join(folder, 'launcher-pid'); + const exitNowPath: string = path.join(folder, 'exit-now'); + const launcher: string = [ + "const fs = require('fs');", + `fs.writeFileSync(${JSON.stringify(launcherPidPath)}, String(process.pid));`, + `setInterval(() => { if (fs.existsSync(${JSON.stringify(exitNowPath)})) process.exit(1); }, 5);` + ].join('\n'); + const outcome: Promise = runDaemonStartupAsync({ + paths, + token, + timeoutMs: 10000, + startCommand: { + command: process.execPath, + args: ['-e', launcher], + cwd: folder, + environment: { PATH: process.env.PATH ?? '', SystemRoot: process.env.SystemRoot ?? '' } + } + }).then( + () => 'resolved', + (error: Error) => error.message + ); + + await waitUntilAsync(() => pending.length > 0 && fs.existsSync(launcherPidPath), 'the first attempt'); + const attempt: DaemonFrameConnection = pending[pending.length - 1]; + const launcherPid: number = Number(fs.readFileSync(launcherPidPath, 'utf8')); + fs.writeFileSync(exitNowPath, ''); + // This process spawned the launcher, so once its PID is gone, the helper has observed its exit. + await waitUntilAsync(() => isProcessGone(launcherPid), 'the launcher is reaped'); + expect(closedConnections.has(attempt)).toBe(false); + answering = true; + await attempt.closeAsync(); + + expect(await outcome).toBe('resolved'); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + } finally { + for (const socket of sockets) socket.destroy(); + await new Promise((resolve) => server.close(() => resolve())); + } + }); +}); diff --git a/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts b/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts index c120d3416b..94fedd12f7 100644 --- a/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts +++ b/libraries/rush-client-core/src/test/connectOrAwaitDaemonStartup.test.ts @@ -294,7 +294,10 @@ describe('connectOrAwaitDaemonStartupAsync', () => { (error: unknown) => error ); expect(refusal).toBeInstanceOf(DaemonClientError); - expect((refusal as Error).message).toContain('exited before the daemon became ready'); + // Until the relaunch time, the refusal is at once, so that the caller can run without the daemon. + expect((refusal as Error).message).toContain( + 'exited before the daemon became ready; refusing another launch until ' + ); expect(Date.now() - started).toBeLessThan(options.startupTimeoutMs!); expect(onAwaitStartup).not.toHaveBeenCalled(); expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index dca59836f5..f7159c3594 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -8,6 +8,7 @@ import * as fs from 'node:fs'; import * as net from 'node:net'; import * as os from 'node:os'; import * as path from 'node:path'; +import { performance } from 'node:perf_hooks'; import { setTimeout as delayAsync } from 'node:timers/promises'; import { FileSystem } from '@rushstack/node-core-library'; @@ -28,7 +29,8 @@ import { connectToPlannedSuccessorAsync, requestDaemonShutdownAsync, resolveDaemonStartupReservationAsync, - type IConnectOrStartDaemonOptions + type IConnectOrStartDaemonOptions, + type IDaemonStartCommand } from '../connectOrStartDaemon'; import { executeWithDaemonRestartAsync, type IDaemonRestartNotice } from '../executeWithDaemonRestart'; import { getDaemonStartupFilePath, releaseDaemonStartup, reserveDaemonStartup } from '../DaemonStartup'; @@ -36,6 +38,13 @@ import { inspectDaemonStartupReservation } from '../DaemonStartupReservation'; import { tryAcquireStartupLockAsync, type IStartupLock } from '../StartupLock'; import { removeTestFolderAsync, waitForTestProcessExitAsync } from './TestProcessExit'; +/** How long after a launch a client may take over its reservation once the helper exited. */ +const RELAUNCH_DELAY_MS: number = 15000; + +function readIfPresent(filePath: string): string { + return fs.existsSync(filePath) ? fs.readFileSync(filePath, 'utf8') : ''; +} + describe('detached daemon startup', () => { let folder: string; let paths: IDaemonPaths; @@ -88,6 +97,8 @@ describe('detached daemon startup', () => { await Promise.all(pids.map((pid) => waitForTestProcessExitAsync(Number(pid)))); expect(pids.every((pid) => fs.existsSync(path.join(folder, `stopped-${pid}`)))).toBe(true); } + // A test that expects a fixture daemon to fail checks and removes this record. + expect(readIfPresent(path.join(folder, 'failures'))).toBe(''); if (fs.existsSync(path.join(folder, 'parents'))) { const parents = new Set(fs.readFileSync(path.join(folder, 'parents'), 'utf8').trim().split('\n')); await Promise.all([...parents].map((pid) => waitForTestProcessExitAsync(Number(pid)))); @@ -157,6 +168,89 @@ describe('detached daemon startup', () => { return contents; } + /** Records the reservation's launch as one relaunch delay earlier, so that its relaunch time has passed. */ + function ageReservation(): string { + const record: { helperStartedAt: string } = JSON.parse( + fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8') + ); + const helperStartedAt: string = new Date( + Date.parse(record.helperStartedAt) - RELAUNCH_DELAY_MS + ).toISOString(); + const contents: string = JSON.stringify({ ...record, helperStartedAt }); + fs.writeFileSync(getDaemonStartupFilePath(paths), contents); + return helperStartedAt; + } + + function readTakeOverLines(): string[] { + return readIfPresent(getDaemonLogFilePath(paths)) + .split('\n') + .filter((line) => line.includes('took over the startup reservation')); + } + + /** Unlike waitForTestProcessExitAsync, a zombie does not count: until it is reaped, its PID looks alive. */ + async function waitForProcessGoneAsync(pid: number): Promise { + const deadline: number = Date.now() + 5000; + while (true) { + try { + process.kill(pid, 0); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ESRCH') return; + throw error; + } + if (Date.now() >= deadline) throw new Error(`Process ${pid} was not reaped in time.`); + await delayAsync(10); + } + } + + async function waitForFileAsync(filePath: string): Promise { + const deadline: number = Date.now() + 5000; + while (!fs.existsSync(filePath) && Date.now() < deadline) await delayAsync(20); + return fs.readFileSync(filePath, 'utf8'); + } + + /** + * Leaves a daemon held before it listens, whose startup helper was killed after its client gave up, and a + * reservation whose relaunch time has passed. + */ + async function abandonStartupBeforeBindAsync(): Promise<{ daemonPid: number; helperPid: number }> { + fs.writeFileSync(path.join(folder, 'hold-prebind'), ''); + await expect(connectOrStartDaemonAsync({ ...options, startupTimeoutMs: 1000 })).rejects.toThrow( + 'timed out awaiting hello/ping readiness' + ); + const daemonPid: number = Number(await waitForFileAsync(path.join(folder, 'prebind'))); + const { helperPid } = JSON.parse(fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8')); + expect(fs.readFileSync(path.join(folder, 'parents'), 'utf8')).toBe(`${helperPid}\n`); + // This test started the helper (through connectOrStartDaemonAsync), so it may signal that PID. + process.kill(helperPid, 'SIGKILL'); + await waitForProcessGoneAsync(helperPid); + expect(inspectDaemonStartupReservation(paths)).toMatchObject({ helperPid, helperState: 'exited' }); + ageReservation(); + return { daemonPid, helperPid }; + } + + /** The start command of a second fixture daemon, which waits before it listens while "hold-prebind-b" exists. */ + function getSecondDaemonOptions(): IConnectOrStartDaemonOptions { + const startCommand: IDaemonStartCommand = options.startCommand!; + return { + ...options, + startCommand: { + ...startCommand, + environment: { ...startCommand.environment, FIXTURE_HOLD_PREBIND: 'hold-prebind-b' } + } + }; + } + + /** Waits for the fixture daemon that lost the race for the endpoint to exit, and checks why it failed. */ + async function expectLostEndpointRaceAsync(loserPid: number, winnerPid: number): Promise { + await waitForTestProcessExitAsync(loserPid); + const failuresPath: string = path.join(folder, 'failures'); + expect(readIfPresent(failuresPath)).toMatch( + new RegExp(`^${loserPid} (Daemon process ${winnerPid} still owns |A live daemon already listens at )`) + ); + expect(readIfPresent(failuresPath).trim().split('\n')).toHaveLength(1); + fs.unlinkSync(failuresPath); + } + async function startFixtureDaemonAsync(): Promise { const client: DaemonClient = await connectOrStartDaemonAsync(options); const { pid } = await client.status; @@ -249,7 +343,7 @@ describe('detached daemon startup', () => { expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); }); - it('refuses another launch at once when the startup helper exited without releasing its reservation', async () => { + it('refuses another launch at once after the startup helper exited, until the relaunch time', async () => { const startupPath: string = getDaemonStartupFilePath(paths); const failing: IConnectOrStartDaemonOptions = { ...options, @@ -257,11 +351,13 @@ describe('detached daemon startup', () => { }; await expect(connectOrStartDaemonAsync(failing)).rejects.toThrow('Unable to start'); const contents: string = fs.readFileSync(startupPath, 'utf8'); - const { helperPid } = JSON.parse(contents); + const { helperPid, helperStartedAt } = JSON.parse(contents); + const relaunchAfter: string = new Date(Date.parse(helperStartedAt) + RELAUNCH_DELAY_MS).toISOString(); expect(inspectDaemonStartupReservation(paths)).toEqual({ path: startupPath, helperPid, - helperState: 'exited' + helperState: 'exited', + relaunchAfter }); const started: number = Date.now(); const error: Error = await connectOrStartDaemonAsync({ ...options, startupTimeoutMs: 20000 }).then( @@ -270,14 +366,144 @@ describe('detached daemon startup', () => { ); expect(Date.now() - started).toBeLessThan(3000); expect(error.message).toContain( - `unresolved startup handoff at ${startupPath}: its startup helper (PID ${helperPid}) exited before the daemon became ready; refusing another launch.` + `unresolved startup handoff at ${startupPath}: its startup helper (PID ${helperPid}) exited before the daemon became ready; refusing another launch until ${relaunchAfter}, so that a daemon that cannot start is not launched by every command.` ); - expect(error.message).toContain('daemon stop --force'); expect(error.message).not.toContain('..'); expect(fs.readFileSync(startupPath, 'utf8')).toBe(contents); expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + expect(readTakeOverLines()).toEqual([]); + + // Once the relaunch time has passed, the next client takes the reservation over and starts the daemon. + const agedStartedAt: string = ageReservation(); + const daemonPid: number = await startFixtureDaemonAsync(); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n`); + const takeOvers: string[] = readTakeOverLines(); + expect(takeOvers).toHaveLength(1); + expect(takeOvers[0]).toMatch(/^\d{4}-\d\d-\d\dT[\d:.]+Z rush-client /); + expect(takeOvers[0]).toContain( + `rush-client (PID ${process.pid}): took over the startup reservation of startup helper PID ${helperPid} (started ${agedStartedAt}), which exited before the daemon became ready; starting the daemon again.` + ); + }); + + it('refuses a relaunch while a process accepts connections at the endpoint', async () => { + const sockets: Set = new Set(); + // Accepts connections but never completes hello, like a daemon that listens but is not ready. + const server: net.Server = net.createServer((socket) => { + sockets.add(socket); + socket.on('close', () => sockets.delete(socket)); + }); + await new Promise((resolve) => server.listen(paths.socketPath, resolve)); + try { + const helperPid: number = await getExitedPidAsync(); + const contents: string = writeReservation( + helperPid, + new Date(Date.now() - 2 * RELAUNCH_DELAY_MS).toISOString() + ); + const started: number = Date.now(); + const error: Error = await connectOrStartDaemonAsync({ + ...options, + startupTimeoutMs: 1500, + timeoutMs: 200 + }).then( + () => new Error('Expected startup to be refused.'), + (refusal: Error) => refusal + ); + // It keeps checking until the deadline, since that process may still become ready. + expect(Date.now() - started).toBeGreaterThanOrEqual(1400); + expect(error.message).toContain( + `its startup helper (PID ${helperPid}) exited before the daemon became ready, but a process still accepts connections at ${paths.socketPath}; refusing another launch.` + ); + expect(error.message).toContain('daemon stop --force'); + expect(fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8')).toBe(contents); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + expect(readTakeOverLines()).toEqual([]); + } finally { + for (const socket of sockets) socket.destroy(); + await new Promise((resolve) => server.close(() => resolve())); + } }); + it('never takes over a reservation whose startup helper still runs, however old', async () => { + // Recorded when this process started, so the reservation names this process, which still runs. + const helperStartedAt: number = performance.timeOrigin; + const age: number = Date.now() - helperStartedAt; + if (age <= RELAUNCH_DELAY_MS + 1000) await delayAsync(RELAUNCH_DELAY_MS + 1000 - age); + const contents: string = writeReservation(process.pid, new Date(helperStartedAt).toISOString()); + expect(inspectDaemonStartupReservation(paths)).toMatchObject({ helperState: 'running' }); + await expect(connectOrStartDaemonAsync({ ...options, startupTimeoutMs: 500 })).rejects.toThrow( + `its startup helper (PID ${process.pid}) is still waiting for the daemon to become ready; refusing another launch` + ); + expect(fs.readFileSync(getDaemonStartupFilePath(paths), 'utf8')).toBe(contents); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + expect(readTakeOverLines()).toEqual([]); + }, 30000); + + it('lets exactly one of several clients take over a reservation whose helper exited', async () => { + const helperPid: number = await getExitedPidAsync(); + writeReservation(helperPid, new Date(Date.now() - 2 * RELAUNCH_DELAY_MS).toISOString()); + const results = await Promise.all(Array.from({ length: 4 }, () => startClient().result)); + expect(results).toEqual(Array.from({ length: 4 }, () => ({ code: 0, stderr: '' }))); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8').trim().split('\n')).toHaveLength(1); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + const takeOvers: string[] = readTakeOverLines(); + expect(takeOvers).toHaveLength(1); + expect(takeOvers[0]).toContain(`took over the startup reservation of startup helper PID ${helperPid} `); + }, 15000); + + it('keeps one daemon when a relaunch outpaces the daemon of a killed startup helper', async () => { + const { daemonPid, helperPid } = await abandonStartupBeforeBindAsync(); + const client: DaemonClient = await connectOrStartDaemonAsync(getSecondDaemonOptions()); + let successorPid: number; + try { + successorPid = (await client.status).pid!; + } finally { + await client.closeAsync(); + } + expect(successorPid).not.toBe(daemonPid); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n${successorPid}\n`); + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + expect(readTakeOverLines()).toHaveLength(1); + expect(readTakeOverLines()[0]).toContain(`startup helper PID ${helperPid} `); + + // The first daemon finds the successor at the endpoint and exits. + fs.unlinkSync(path.join(folder, 'hold-prebind')); + await expectLostEndpointRaceAsync(daemonPid, successorPid); + const next: DaemonClient = await connectOrStartDaemonAsync(options); + try { + expect((await next.status).pid).toBe(successorPid); + } finally { + await next.closeAsync(); + } + }, 20000); + + it('keeps one daemon when the daemon of a killed startup helper outpaces the relaunch', async () => { + const { daemonPid } = await abandonStartupBeforeBindAsync(); + fs.writeFileSync(path.join(folder, 'hold-prebind-b'), ''); + const relaunch: Promise = connectOrStartDaemonAsync(getSecondDaemonOptions()); + const successorPid: number = Number(await waitForFileAsync(path.join(folder, 'prebind-b'))); + expect(successorPid).not.toBe(daemonPid); + + // The first daemon publishes the endpoint first, so the relaunch's helper finds it ready. + fs.unlinkSync(path.join(folder, 'hold-prebind')); + const client: DaemonClient = await relaunch; + try { + expect((await client.status).pid).toBe(daemonPid); + } finally { + await client.closeAsync(); + } + expect(fs.existsSync(getDaemonStartupFilePath(paths))).toBe(false); + + fs.unlinkSync(path.join(folder, 'hold-prebind-b')); + await expectLostEndpointRaceAsync(successorPid, daemonPid); + expect(fs.readFileSync(path.join(folder, 'starts'), 'utf8')).toBe(`${daemonPid}\n${successorPid}\n`); + const next: DaemonClient = await connectOrStartDaemonAsync(options); + try { + expect((await next.status).pid).toBe(daemonPid); + } finally { + await next.closeAsync(); + } + }, 20000); + it.each(['legacy', 'exited helper', 'no auto-start'])( 'uses and resolves a ready daemon next to a retained startup reservation (%s)', async (kind) => { @@ -430,11 +656,13 @@ describe('detached daemon startup', () => { helperState: 'running' }); const exitedPid: number = await getExitedPidAsync(); - writeReservation(exitedPid); + const helperStartedAt: Date = new Date(); + writeReservation(exitedPid, helperStartedAt.toISOString()); expect(inspectDaemonStartupReservation(paths)).toEqual({ path: startupPath, helperPid: exitedPid, - helperState: 'exited' + helperState: 'exited', + relaunchAfter: new Date(helperStartedAt.getTime() + RELAUNCH_DELAY_MS).toISOString() }); if (process.platform === 'linux') { // This process started after the recorded helper, so it merely reuses the PID. diff --git a/libraries/rush-client-core/src/test/fixtures/daemon.ts b/libraries/rush-client-core/src/test/fixtures/daemon.ts index faeac1e486..5b8f10234b 100644 --- a/libraries/rush-client-core/src/test/fixtures/daemon.ts +++ b/libraries/rush-client-core/src/test/fixtures/daemon.ts @@ -24,6 +24,8 @@ async function mainAsync(): Promise { const daemonVersion: string = process.argv[3] ?? 'fixture'; const mode: string | undefined = process.argv[4]; const restartMode: string | undefined = mode?.startsWith('restart-') ? mode : undefined; + // While this file exists, the daemon waits before it listens. It writes its PID to the file named without "hold-". + const holdPrebindName: string = process.env.FIXTURE_HOLD_PREBIND ?? 'hold-prebind'; const connections: Set = new Set(); let closing: Promise | undefined; let heldRequest: { connection: DaemonFrameConnection; requestId: string } | undefined; @@ -37,11 +39,11 @@ async function mainAsync(): Promise { fs.writeFileSync(path.join(folder, 'runtime-base'), process.env.RUSHD_RUNTIME_DIR ?? '(unset)'); process.stdout.write('launcher stdout\n'); process.stderr.write('launcher stderr\n'); - if (fs.existsSync(path.join(folder, 'hold-prebind'))) { + if (fs.existsSync(path.join(folder, holdPrebindName))) { const marker: string = path.join(folder, `prebind-${process.pid}.tmp`); fs.writeFileSync(marker, String(process.pid)); - fs.renameSync(marker, path.join(folder, 'prebind')); - while (fs.existsSync(path.join(folder, 'hold-prebind'))) { + fs.renameSync(marker, path.join(folder, holdPrebindName.replace(/^hold-/, ''))); + while (fs.existsSync(path.join(folder, holdPrebindName))) { if (fs.existsSync(path.join(folder, 'stop'))) { fs.writeFileSync(path.join(folder, `stopped-${process.pid}`), ''); return; @@ -241,6 +243,10 @@ function readNumber(folder: string, name: string, defaultValue: number): number } mainAsync().catch((error: Error) => { + // For example, the daemon found another one at the endpoint. The test expects this exit, so it counts as stopped. + const folder: string = path.dirname((JSON.parse(process.argv[2]) as IDaemonPaths).lockfilePath); + fs.appendFileSync(path.join(folder, 'failures'), `${process.pid} ${error.message}\n`); + fs.writeFileSync(path.join(folder, `stopped-${process.pid}`), ''); process.stderr.write(`${error.stack}\n`); process.exitCode = 1; }); From 2ace16d6e37bf78aaf971ab7cf2b3b691fc431e8 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 02:37:43 +0000 Subject: [PATCH 059/265] [rush-client-core] After a daemon crash the client reaps its orphaned operations before it returns Swarm integration step 44; original commit 80219e0892 (merge of swarm/r06-t154-int2 at 25c7f0135c). Scope: task 154. Brings r06's task 154 (board 2382), re-tipped on c0e17d82ab (board 2747): after a daemon crash, the client reaps the orphaned operations before returning, and before `--no-daemon` runs. Second agent: t05 board 2689 on 14baba4338. ch01's e2e: the orphan is reaped after a crash. s16 batch B, item 4 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree 2453314813) Commits folded into this step (2): - e79486fa37 [rush-client-core] Stop a crashed daemon's operations before reporting the crash (task 154) - 14baba4338 [rush-cli-client] Reclaim a crashed daemon before Rush runs in-process (task 154) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 10 +- .../src/daemonConnectionOptions.ts | 11 +- apps/rush-cli-client/src/launchClient.ts | 32 ++- .../src/test/nativeBuild.test.ts | 63 +++++- ...54-inprocess-reclaim_2026-09-28-23-56.json | 11 + ...54-inprocess-reclaim_2026-09-28-23-56.json | 11 + ...-reclaim-after-crash_2026-09-28-23-50.json | 11 + common/reviews/api/rush-client-core.api.md | 3 + libraries/rush-client-core/README.md | 17 ++ .../rush-client-core/src/DaemonDisconnect.ts | 9 +- .../src/ExitedDaemonReclaim.ts | 101 +++++++++ libraries/rush-client-core/src/index.ts | 1 + .../src/test/DaemonDisconnect.test.ts | 196 ++++++++++++++++-- .../src/test/ExitedDaemonReclaim.test.ts | 136 ++++++++++++ .../src/test/OrphanedOperation.ts | 90 ++++++++ 15 files changed, 678 insertions(+), 24 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json create mode 100644 common/changes/@rushstack/rush-client-core/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json create mode 100644 common/changes/@rushstack/rush-client-core/swarm-r06-t154-reclaim-after-crash_2026-09-28-23-50.json create mode 100644 libraries/rush-client-core/src/ExitedDaemonReclaim.ts create mode 100644 libraries/rush-client-core/src/test/ExitedDaemonReclaim.test.ts create mode 100644 libraries/rush-client-core/src/test/OrphanedOperation.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 75f5aa6bda..1f4dc9e3f7 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -285,8 +285,14 @@ and is not retried. The diagnostic keeps "Daemon disconnected before delivering command was not retried." and says what happened to rushd. If its process exited (a crash, an out-of-memory kill or a signal), it names the PID, points to `rush-client daemon logs` and, if the daemon exits again, to `--no-daemon` (`rushx-client --no-daemon` for Rushx), and quotes on a -second line the fatal error that the launcher log recorded after the command was sent. If -rushd still runs, it says that only the connection closed. Ctrl+C and an orderly `daemon stop` +second line the fatal error that the launcher log recorded after the command was sent. Before it +prints that, the client stops the operations that the exited daemon left running and removes its +ownership record and socket, as the next daemon start would, so a rerun, with or without +`--no-daemon`, does not race them. A `RUSH_DAEMON_ORPHANS_REAPED` warning says what it stopped. +Rush run in-process, with `--no-daemon` or as a fallback, first does the same when the ownership +record names a daemon that no longer runs, for example when the client that ran the command was +killed along with the daemon. +If rushd still runs, it says that only the connection closed. Ctrl+C and an orderly `daemon stop` or `daemon restart` still end a command as cancelled (exit code 130). Piped input uses protocol 0.7's negotiated stdin admission and EOF. The client does diff --git a/apps/rush-cli-client/src/daemonConnectionOptions.ts b/apps/rush-cli-client/src/daemonConnectionOptions.ts index 26f7158a9d..d088352dde 100644 --- a/apps/rush-cli-client/src/daemonConnectionOptions.ts +++ b/apps/rush-cli-client/src/daemonConnectionOptions.ts @@ -33,9 +33,7 @@ export function getDaemonConnectionOptions( 'The synchronous launcher only supports its installed engine; use asynchronous version selection.' ); } - const paths: IDaemonPaths = resolveDaemonPathsFromProcess( - computeDaemonWorkspaceKey({ canonicalRepoRoot, rushVersion }) - ); + const paths: IDaemonPaths = getDaemonPaths(canonicalRepoRoot, rushVersion); // Every daemon command trusts files in this folder: the socket, the lockfile, the log and the reservation. assertDaemonRuntimeFolderIsPrivate(paths); return { @@ -87,3 +85,10 @@ export async function getDaemonConnectionOptionsAsync( }); return { ...options, expectedDaemonVersion: launch.daemonVersion, startCommand: launch.startCommand }; } + +/** The runtime files of the workspace's daemon for this Rush version: its socket and ownership record. */ +export function getDaemonPaths(repoRoot: string, rushVersion: string): IDaemonPaths { + return resolveDaemonPathsFromProcess( + computeDaemonWorkspaceKey({ canonicalRepoRoot: fs.realpathSync.native(repoRoot), rushVersion }) + ); +} diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 095ea16714..308dba115b 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -15,11 +15,13 @@ import { captureDaemonRequest, connectOrAwaitDaemonStartupAsync, executeWithDaemonRestartAsync, + reclaimCrashedDaemonAsync, type DaemonClient, type DaemonClientOutcome, type IConnectOrStartDaemonOptions } from '@rushstack/rush-client-core'; import type { DaemonVerbosity, IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; +import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; import { ConsoleTerminalProvider } from '@rushstack/terminal'; import { executeDaemonCommandAsync } from './daemonCommands'; @@ -33,7 +35,7 @@ import { getSignalExitCode, isCancelledOutcome } from './clientCancellation'; -import { getDaemonConnectionOptionsAsync } from './daemonConnectionOptions'; +import { getDaemonConnectionOptionsAsync, getDaemonPaths } from './daemonConnectionOptions'; import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; import { getResultStderr } from './resultDiagnostics'; @@ -93,7 +95,7 @@ export async function launchClientAsync( } if (!route.daemon || !rushJsonPath || route.commandName === undefined) { agentRenderer?.dispose(); - launchInProcess(route.argv, rushx, selectedVersion); + await launchInProcessAsync(route.argv, rushx, selectedVersion, rushJsonPath); return; } const terminal: ConsoleTerminalProvider = new ConsoleTerminalProvider(); @@ -170,7 +172,7 @@ export async function launchClientAsync( throw error; agentRenderer?.dispose(); process.stderr.write(`rush-client: ${error.message} Using in-process Rush.\n`); - launchInProcess(route.argv, rushx, selectedVersion); + await launchInProcessAsync(route.argv, rushx, selectedVersion, rushJsonPath); return; } const abort: AbortController = new AbortController(); @@ -306,10 +308,32 @@ export async function launchClientAsync( } else { agentRenderer?.dispose(); process.stderr.write(`rush-client: ${outcome.message ?? outcome.reason}; using in-process Rush.\n`); - launchInProcess(route.argv, rushx, selectedVersion); + await launchInProcessAsync(route.argv, rushx, selectedVersion, rushJsonPath); } } +/** + * Runs Rush in-process. A daemon that crashed while it ran a command can leave its operations running, and + * they could overwrite this command's outputs, so they are stopped first, as the next daemon start would. + */ +async function launchInProcessAsync( + argv: ReadonlyArray, + rushx: boolean, + selectedVersion: string, + rushJsonPath: string | undefined +): Promise { + if (rushJsonPath) { + let paths: IDaemonPaths | undefined; + try { + paths = getDaemonPaths(path.dirname(rushJsonPath), selectedVersion); + } catch { + // Without the daemon's folder there is nothing to reclaim; Rush reports a workspace problem itself. + } + if (paths) await reclaimCrashedDaemonAsync(paths); + } + launchInProcess(argv, rushx, selectedVersion); +} + function launchInProcess(argv: ReadonlyArray, rushx: boolean, selectedVersion: string): void { const executable: string = rushx ? 'rushx' : 'rush'; const rushFolder: string = path.dirname(require.resolve('@microsoft/rush/package.json')); diff --git a/apps/rush-cli-client/src/test/nativeBuild.test.ts b/apps/rush-cli-client/src/test/nativeBuild.test.ts index 2703d68a41..efcbf6154a 100644 --- a/apps/rush-cli-client/src/test/nativeBuild.test.ts +++ b/apps/rush-cli-client/src/test/nativeBuild.test.ts @@ -1,14 +1,15 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { spawn } from 'node:child_process'; +import { spawn, type ChildProcess } from 'node:child_process'; import { once } from 'node:events'; import * as fs from 'node:fs'; import * as path from 'node:path'; import { Rush } from '@microsoft/rush-lib'; import { DaemonClient } from '@rushstack/rush-client-core'; -import { RUSHD_GRAPH_SNAPSHOT } from '@rushstack/rush-daemon-protocol'; +import { DAEMON_PROTOCOL_VERSION, RUSHD_GRAPH_SNAPSHOT } from '@rushstack/rush-daemon-protocol'; +import { writeDaemonLockfile, type IDaemonPaths } from '@rushstack/rush-daemon-transport'; import { createNativeBuildTestFixture, type INativeBuildTestFixture, @@ -17,6 +18,43 @@ import { // A pending CLI startup can take 15s; allow its join plus daemon stop/drain and fixture removal. const NATIVE_FIXTURE_CLEANUP_TIMEOUT_MS: number = 35_000; +// An operation process that a stand-in daemon starts; it exits by itself after a minute. +const OPERATION_SCRIPT: string = 'setTimeout(()=>{},60000)'; +// A stand-in daemon: like a phased operation, its operation process shares the daemon's process group. +const STAND_IN_DAEMON_SCRIPT: string = + "const c=require('node:child_process')" + + `.spawn(process.execPath,['-e','${OPERATION_SCRIPT}'],{stdio:'ignore'});` + + "process.stdout.write(String(c.pid)+'\\n');setInterval(()=>{},1000);"; + +/** Records a stand-in daemon as the workspace's daemon, then SIGKILLs only it, so its operation keeps running. */ +async function startCrashedDaemonAsync( + paths: IDaemonPaths +): Promise<{ daemonPid: number; operationPid: number }> { + const daemon: ChildProcess = spawn(process.execPath, ['-e', STAND_IN_DAEMON_SCRIPT], { + detached: true, + stdio: ['ignore', 'pipe', 'ignore'] + }); + const [chunk] = (await once(daemon.stdout!, 'data')) as [Buffer]; + daemon.stdout!.destroy(); + writeDaemonLockfile(paths.lockfilePath, { + pid: daemon.pid!, + protocolVersion: DAEMON_PROTOCOL_VERSION, + startedAt: new Date().toISOString(), + socketPath: paths.socketPath + }); + daemon.kill('SIGKILL'); + await once(daemon, 'exit'); + return { daemonPid: daemon.pid!, operationPid: Number(chunk.toString().trim()) }; +} + +/** False once the operation process has exited, even before it is reaped: a zombie's command line is empty. */ +function isOperationRunning(pid: number): boolean { + try { + return fs.readFileSync(`/proc/${pid}/cmdline`, 'utf8').includes(OPERATION_SCRIPT); + } catch { + return false; + } +} describe('native build through the standalone client', () => { let fixture: INativeBuildTestFixture | undefined; @@ -334,4 +372,25 @@ describe('native build through the standalone client', () => { 45000 ); }); + + (process.platform === 'linux' ? it : it.skip)( + 'stops the operations that a crashed daemon left running before it builds without the daemon', + () => + runWithFixtureAsync(async ({ folder, paths, invokeAsync }) => { + const { daemonPid, operationPid } = await startCrashedDaemonAsync(paths); + try { + expect(isOperationRunning(operationPid)).toBe(true); + const native: IResult = await invokeAsync(['--no-daemon', 'build']); + expect(native.code).toBe(0); + expect(native.stderr).toContain(`Reclaimed dead daemon ${daemonPid}:`); + expect(isOperationRunning(operationPid)).toBe(false); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + expect(fs.readFileSync(path.join(folder, 'runs.txt'), 'utf8')).toBe('a:one\nb:one\n'); + } finally { + // Only the operation process that this test started, not a process that reused its PID. + if (isOperationRunning(operationPid)) process.kill(operationPid, 'SIGKILL'); + } + }), + 30000 + ); }); diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json b/common/changes/@rushstack/rush-cli-client/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json new file mode 100644 index 0000000000..c8949b4ec2 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Before Rush runs in-process, with `--no-daemon` or as a fallback, stop the operations that a crashed daemon left running, so that they cannot overwrite the command's outputs.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json b/common/changes/@rushstack/rush-client-core/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json new file mode 100644 index 0000000000..289ae9a1b1 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/swarm-r06-t154-inprocess-reclaim_2026-09-28-23-56.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Add `reclaimCrashedDaemonAsync()`, which stops the operations that a crashed daemon left running, and removes its stale files, before a caller runs Rush in-process.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/swarm-r06-t154-reclaim-after-crash_2026-09-28-23-50.json b/common/changes/@rushstack/rush-client-core/swarm-r06-t154-reclaim-after-crash_2026-09-28-23-50.json new file mode 100644 index 0000000000..e1f5e55160 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/swarm-r06-t154-reclaim-after-crash_2026-09-28-23-50.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "When the daemon exits while it runs a request, the client now stops the operations that the daemon left running and removes its stale files before it reports the exit, as the next daemon start would, so rerunning the command with --no-daemon no longer races them.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 76e0bf211f..0b57acf773 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -182,6 +182,9 @@ export interface IExecuteWithDaemonRestartOptions extends IDaemonClientExecuteOp // @beta export function inspectDaemonStartupReservation(paths: IDaemonPaths): IDaemonStartupReservationInfo | undefined; +// @beta +export function reclaimCrashedDaemonAsync(paths: IDaemonPaths): Promise; + // @beta export function requestDaemonShutdownAsync(client: DaemonClient, paths: IDaemonPaths, timeoutMs?: number): Promise>; diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 6461b1db0e..57ae16f689 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -63,6 +63,14 @@ an exited process that is not reaped yet counts), the message names the PID, `ru daemon logs` and `--no-daemon` (`rushx-client` for Rushx requests), and adds a second line with the first fatal error that `.log` gained after the request was sent: a Node.js uncaught-exception report or a V8 `FATAL ERROR:` line, clipped to one printable line. +Before it returns that error, it reclaims the exited daemon as the next daemon start would, so +running the command again, with or without the daemon, does not race the operations the daemon +left running. While the ownership record names that process, it takes the start mutex and, unless +a startup is reserved, calls `reclaimStaleDaemonAsync()`, which terminates the orphaned operation +process groups (each reported as a `RUSH_DAEMON_ORPHANS_REAPED` process warning) and removes the +ownership record and socket. It waits up to 5 seconds while another client holds the start mutex, +or while the exited process is not reaped yet. If the reclaim fails or times out, the message is +the same, and the next daemon start reclaims the daemon instead. If the process still runs, the message says that only the connection closed. After the abort signal fires, the error is unchanged, so the caller reports the cancellation. @@ -161,6 +169,15 @@ only after binding. Other errors, such as `versionMismatch`, pass through unchan waiting, it calls the optional `onAwaitStartup(owner, waitMs)` once, with the live process and the remaining wait, so that the caller can say why the command has not started yet. +`reclaimCrashedDaemonAsync(paths)` is for a caller that is about to run Rush in-process, as the CLI +client does for `--no-daemon` and for each fallback. A daemon that crashed or was killed while it ran +a command leaves its operations running, and they could overwrite the in-process command's outputs. +When the ownership record names a PID that no longer exists (on Linux, also an exited process that is +not reaped yet), it reclaims that daemon as described above for a lost connection: under the start +mutex, only when no startup is reserved, and waiting up to 5 seconds. It does nothing when there is no +record, when a process with the recorded PID runs, or when the runtime folder is not private, and it +never throws. + `getDaemonLogFilePath(paths)` is the shared stable path used by both the launcher and the CLI's local `daemon logs` reader. Child stdout/stderr are appended across restarts, including startup failures; the parent always closes its descriptor diff --git a/libraries/rush-client-core/src/DaemonDisconnect.ts b/libraries/rush-client-core/src/DaemonDisconnect.ts index 14ad575411..fb40328d77 100644 --- a/libraries/rush-client-core/src/DaemonDisconnect.ts +++ b/libraries/rush-client-core/src/DaemonDisconnect.ts @@ -17,6 +17,7 @@ import type { DaemonClient } from './DaemonClient'; import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from './DaemonClientError'; import { getDaemonLogFilePath } from './DaemonLogFile'; import { isOwnerProcessAlive } from './DaemonOwnership'; +import { reclaimExitedDaemonAsync } from './ExitedDaemonReclaim'; import { isProcessDefunct } from './ProcessStartTime'; /** A process closes its connections while it exits, so it can briefly outlive them. */ @@ -41,6 +42,8 @@ export interface IServingDaemon { readonly startedAt: string | undefined; readonly logFilePath: string; readonly logOffset: number | undefined; + /** The workspace's daemon files, which are reclaimed if the process exits. */ + readonly paths: IDaemonPaths; } /** Identifies the daemon behind a ready client, or returns undefined for a peer that does not report its PID. */ @@ -56,7 +59,8 @@ export async function observeServingDaemonAsync( pid, startedAt: owner?.pid === pid ? owner.startedAt : undefined, logFilePath, - logOffset: tryGetFileSize(logFilePath) + logOffset: tryGetFileSize(logFilePath), + paths }; } @@ -64,6 +68,8 @@ export async function observeServingDaemonAsync( * Explains a connection lost before a request's result by what happened to the daemon process: whether it * exited, the fatal error its launcher log recorded, and how to recover. The request is never replayed. * Other errors are returned unchanged. + * @remarks If the daemon exited, this first reclaims it as the next daemon start would, so that running the + * command again, with or without the daemon, cannot race the operations the daemon left running. */ export async function explainLostConnectionAsync( error: unknown, @@ -81,6 +87,7 @@ export async function explainLostConnectionAsync( ); } const loggedError: string | undefined = readLoggedFatalError(daemon); + await reclaimExitedDaemonAsync(daemon); const client: string = request.invocationKind === 'rushx' ? 'rushx-client' : 'rush-client'; const message: string = `${DAEMON_DISCONNECTED_MESSAGE} rushd (PID ${daemon.pid}) exited while it ran the command; ` + diff --git a/libraries/rush-client-core/src/ExitedDaemonReclaim.ts b/libraries/rush-client-core/src/ExitedDaemonReclaim.ts new file mode 100644 index 0000000000..807cae4029 --- /dev/null +++ b/libraries/rush-client-core/src/ExitedDaemonReclaim.ts @@ -0,0 +1,101 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { + assertDaemonRuntimeDirIsPrivate, + readDaemonLockfile, + reclaimStaleDaemonAsync, + type IDaemonLockfile, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; + +import { isProcessAlive } from './DaemonOwnership'; +import { readDaemonStartupReservation } from './DaemonStartup'; +import { isProcessDefunct } from './ProcessStartTime'; +import { tryAcquireStartupLockAsync, type IStartupLock } from './StartupLock'; + +/** + * How long to wait while another client reclaims the exited daemon, or while its exited process is not reaped + * yet. A reclaim sends SIGTERM to orphaned operations, then SIGKILL to those still running 2 seconds later. + */ +const RECLAIM_WAIT_MS: number = 5000; +const RECLAIM_POLL_INTERVAL_MS: number = 50; + +/** A daemon process that has exited, and the workspace files that may still name it. */ +export interface IExitedDaemon { + readonly pid: number; + /** The ownership record's start time when known; a record with another start time names another process. */ + readonly startedAt: string | undefined; + readonly paths: IDaemonPaths; +} + +/** + * Reclaims the workspace's daemon when its ownership record names a process that no longer runs, such as a + * daemon that crashed or was killed while it ran a command. This stops the operations that the daemon left + * running, so that they cannot overwrite the outputs of a command that then runs without the daemon. + * + * @remarks + * Call it before Rush runs in-process. Like the next daemon start, it terminates the daemon's orphaned + * operation process groups (each reported as a `RUSH_DAEMON_ORPHANS_REAPED` process warning) and removes the + * ownership record and socket. It does so only under the start mutex and when no startup is reserved, and it + * waits up to 5 seconds while another client holds the mutex or while the exited process is not reaped yet. + * It does nothing when there is no record, when a process with the recorded PID runs, or when the runtime + * folder is not private, and it never throws. + * + * @beta + */ +export async function reclaimCrashedDaemonAsync(paths: IDaemonPaths): Promise { + let owner: IDaemonLockfile | undefined; + try { + // Every daemon command checks the folder before it trusts the records in it. + assertDaemonRuntimeDirIsPrivate(paths); + owner = readDaemonLockfile(paths.lockfilePath); + // The reclaim refuses a PID that still exists, except that an exited process may not be reaped yet. + if (!owner || (isProcessAlive(owner.pid) && !isProcessDefunct(owner.pid))) return; + } catch { + // For example EPERM: the PID exists but cannot be inspected. + return; + } + await reclaimExitedDaemonAsync({ pid: owner.pid, startedAt: owner.startedAt, paths }); +} + +/** + * Stops the operations that an exited daemon left running, and removes its ownership record and socket, as + * the next daemon start would (`RUSH_DAEMON_ORPHANS_REAPED` warnings say what was stopped). Best effort: it + * acts only while the ownership record names that daemon, under the start mutex, and when no startup is + * reserved. While another client holds the mutex, for example to reclaim the same daemon, it waits. + */ +export async function reclaimExitedDaemonAsync(daemon: IExitedDaemon): Promise { + const deadline: number = Date.now() + RECLAIM_WAIT_MS; + try { + while (isRecordedOwner(daemon)) { + // The reclaim refuses a PID that still exists, which includes an exited process that is not reaped yet. + if (!isProcessDefunct(daemon.pid)) { + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(daemon.paths); + if (lock) { + try { + if (isRecordedOwner(daemon) && !readDaemonStartupReservation(daemon.paths)) { + await reclaimStaleDaemonAsync(daemon.paths); + } + } finally { + await lock.releaseAsync(); + } + return; + } + } + if (Date.now() >= deadline) return; + await delayAsync(RECLAIM_POLL_INTERVAL_MS); + } + } catch { + // For example, the PID was reused, or another process holds the reclaim's mutex; the next start retries. + } +} + +function isRecordedOwner(daemon: IExitedDaemon): boolean { + const owner: IDaemonLockfile | undefined = readDaemonLockfile(daemon.paths.lockfilePath); + return ( + owner?.pid === daemon.pid && (daemon.startedAt === undefined || owner.startedAt === daemon.startedAt) + ); +} diff --git a/libraries/rush-client-core/src/index.ts b/libraries/rush-client-core/src/index.ts index 32bc6691f2..b6234ca642 100644 --- a/libraries/rush-client-core/src/index.ts +++ b/libraries/rush-client-core/src/index.ts @@ -26,6 +26,7 @@ export { type DaemonStartupHelperState, type IDaemonStartupReservationInfo } from './DaemonStartupReservation'; +export { reclaimCrashedDaemonAsync } from './ExitedDaemonReclaim'; export { executeWithDaemonRestartAsync, type IDaemonRestartNotice, diff --git a/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts b/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts index 39d6bde652..69bfd1f157 100644 --- a/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts +++ b/libraries/rush-client-core/src/test/DaemonDisconnect.test.ts @@ -2,17 +2,30 @@ // See LICENSE in the project root for license information. import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { setTimeout as delayAsync } from 'node:timers/promises'; import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; -import { DaemonTransportError, DaemonTransportErrorCode } from '@rushstack/rush-daemon-transport'; +import { + DaemonTransportError, + DaemonTransportErrorCode, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; import { captureDaemonRequest } from '../captureDaemonRequest'; import { DAEMON_DISCONNECTED_MESSAGE, DaemonClientError } from '../DaemonClientError'; import { explainLostConnectionAsync, findLoggedFatalError, type IServingDaemon } from '../DaemonDisconnect'; +import { reserveDaemonStartup } from '../DaemonStartup'; import { isProcessDefunct } from '../ProcessStartTime'; +import { tryAcquireStartupLockAsync, type IStartupLock } from '../StartupLock'; +import { + isRunning, + recordDaemonOwner, + startOrphanedOperationAsync, + stopOperationIfRunning +} from './OrphanedOperation'; import { withUnreapedChildAsync } from './UnreapedChildProcess'; const linuxIt: typeof it = process.platform === 'linux' ? it : it.skip; @@ -82,14 +95,38 @@ describe(explainLostConnectionAsync.name, () => { environment: {}, terminal: { isTTY: false, supportsColor: false } }); + let folder: string; + let paths: IDaemonPaths; + let operationPids: number[]; - function getServingDaemon(pid: number): IServingDaemon { - return { - pid, - startedAt: undefined, - logFilePath: path.join(os.tmpdir(), 'missing.log'), - logOffset: undefined + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-disconnect-')); + paths = { + runtimeDir: folder, + socketPath: path.join(folder, 'd.sock'), + lockfilePath: path.join(folder, 'daemon.pid.json') }; + operationPids = []; + }); + + afterEach(async () => { + operationPids.forEach(stopOperationIfRunning); + await fs.promises.rm(folder, { recursive: true, force: true }); + }); + + function getServingDaemon(pid: number, startedAt?: string): IServingDaemon { + return { pid, startedAt, logFilePath: path.join(folder, 'missing.log'), logOffset: undefined, paths }; + } + + function getLostConnection(): DaemonTransportError { + return new DaemonTransportError( + DaemonTransportErrorCode.transportClosed, + 'The daemon connection closed.' + ); + } + + function getExitMessage(pid: number): string { + return `${DAEMON_DISCONNECTED_MESSAGE} rushd (PID ${pid}) exited while it ran the command; "rush-client daemon logs" may show why. Run the command again; if the daemon exits again, run the command with "rush-client --no-daemon".`; } it('returns other failures, and failures without a known daemon, unchanged', async () => { @@ -103,10 +140,7 @@ describe(explainLostConnectionAsync.name, () => { await withUnreapedChildAsync(async (child) => { const deadline: number = Date.now() + 5000; while (!isProcessDefunct(child) && Date.now() < deadline) await delayAsync(20); - const closed: DaemonTransportError = new DaemonTransportError( - DaemonTransportErrorCode.transportClosed, - 'The daemon connection closed.' - ); + const closed: DaemonTransportError = getLostConnection(); const startedAt: number = Date.now(); const explained: unknown = await explainLostConnectionAsync(closed, getServingDaemon(child), request); expect(Date.now() - startedAt).toBeLessThan(500); @@ -114,8 +148,146 @@ describe(explainLostConnectionAsync.name, () => { expect(explained).toMatchObject({ code: 'disconnected', cause: closed, - message: `${DAEMON_DISCONNECTED_MESSAGE} rushd (PID ${child}) exited while it ran the command; "rush-client daemon logs" may show why. Run the command again; if the daemon exits again, run the command with "rush-client --no-daemon".` + message: getExitMessage(child) }); }); }); + + linuxIt( + 'stops the operations that an exited daemon left running, and removes its files, first', + async () => { + const warning: jest.SpyInstance = jest + .spyOn(process, 'emitWarning') + .mockImplementation(() => undefined); + try { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + const explained: unknown = await explainLostConnectionAsync( + getLostConnection(), + getServingDaemon(daemonPid), + request + ); + expect(explained).toMatchObject({ code: 'disconnected', message: getExitMessage(daemonPid) }); + expect(isRunning(operationPid)).toBe(false); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + expect(warning).toHaveBeenCalledWith( + expect.stringContaining(`Reclaimed dead daemon ${daemonPid}:`), + expect.objectContaining({ code: 'RUSH_DAEMON_ORPHANS_REAPED' }) + ); + } finally { + warning.mockRestore(); + } + } + ); + + linuxIt('reclaims an exited daemon that is not reaped yet once it is reaped', async () => { + await withUnreapedChildAsync(async (child, parentPid) => { + const deadline: number = Date.now() + 5000; + while (!isProcessDefunct(child) && Date.now() < deadline) await delayAsync(20); + recordDaemonOwner(paths, child); + let settled: boolean = false; + const explained: Promise = explainLostConnectionAsync( + getLostConnection(), + getServingDaemon(child), + request + ).finally(() => { + settled = true; + }); + await delayAsync(300); + expect(settled).toBe(false); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + // Once its parent exits, init or a subreaper reaps it. + process.kill(parentPid, 'SIGTERM'); + expect(await explained).toMatchObject({ message: getExitMessage(child) }); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + }); + }); + + linuxIt('waits while another client holds the start mutex, and leaves the reclaim to it', async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + expect(lock).toBeDefined(); + let settled: boolean = false; + const explained: Promise = explainLostConnectionAsync( + getLostConnection(), + getServingDaemon(daemonPid), + request + ).finally(() => { + settled = true; + }); + await delayAsync(300); + expect(settled).toBe(false); + // The other client's reclaim ends by removing the ownership record. + fs.unlinkSync(paths.lockfilePath); + await lock!.releaseAsync(); + expect(await explained).toMatchObject({ message: getExitMessage(daemonPid) }); + expect(isRunning(operationPid)).toBe(true); + }); + + linuxIt( + 'gives up after a few seconds while another client keeps the start mutex', + async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + expect(lock).toBeDefined(); + try { + const startedAt: number = Date.now(); + expect( + await explainLostConnectionAsync(getLostConnection(), getServingDaemon(daemonPid), request) + ).toMatchObject({ message: getExitMessage(daemonPid) }); + expect(Date.now() - startedAt).toBeGreaterThanOrEqual(4500); + expect(Date.now() - startedAt).toBeLessThan(10000); + expect(isRunning(operationPid)).toBe(true); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + } finally { + await lock!.releaseAsync(); + } + }, + 20000 + ); + + linuxIt('leaves an exited daemon to the startup that is reserved', async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + reserveDaemonStartup(paths, { pid: process.pid, startedAt: new Date().toISOString() }); + const startedAt: number = Date.now(); + expect( + await explainLostConnectionAsync(getLostConnection(), getServingDaemon(daemonPid), request) + ).toMatchObject({ message: getExitMessage(daemonPid) }); + expect(Date.now() - startedAt).toBeLessThan(1000); + expect(isRunning(operationPid)).toBe(true); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + }); + + linuxIt('leaves the files alone when the ownership record names another daemon', async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + // A process that has exited, which the record does not name. + const otherPid: number = spawnSync(process.execPath, ['-e', '']).pid!; + // The same PID, but a record written for another process. + for (const daemon of [ + getServingDaemon(otherPid), + getServingDaemon(daemonPid, '2000-01-01T00:00:00.000Z') + ]) { + expect(await explainLostConnectionAsync(getLostConnection(), daemon, request)).toMatchObject({ + message: getExitMessage(daemon.pid) + }); + expect(isRunning(operationPid)).toBe(true); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + } + }); + + linuxIt('still explains the exit when the reclaim fails', async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + // Another process, such as a starting daemon, holds the reclaim's own mutex, so the reclaim throws. + fs.writeFileSync(`${paths.lockfilePath}.reclaim`, JSON.stringify({ mutexPid: process.pid })); + expect( + await explainLostConnectionAsync(getLostConnection(), getServingDaemon(daemonPid), request) + ).toMatchObject({ code: 'disconnected', message: getExitMessage(daemonPid) }); + expect(isRunning(operationPid)).toBe(true); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + }); }); diff --git a/libraries/rush-client-core/src/test/ExitedDaemonReclaim.test.ts b/libraries/rush-client-core/src/test/ExitedDaemonReclaim.test.ts new file mode 100644 index 0000000000..424fd8c612 --- /dev/null +++ b/libraries/rush-client-core/src/test/ExitedDaemonReclaim.test.ts @@ -0,0 +1,136 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import type { IDaemonPaths } from '@rushstack/rush-daemon-transport'; + +import { reclaimCrashedDaemonAsync } from '../ExitedDaemonReclaim'; +import { isProcessDefunct } from '../ProcessStartTime'; +import { tryAcquireStartupLockAsync, type IStartupLock } from '../StartupLock'; +import { + isRunning, + recordDaemonOwner, + startOrphanedOperationAsync, + startStandInDaemonAsync, + stopOperationIfRunning, + type IStandInDaemon +} from './OrphanedOperation'; +import { withUnreapedChildAsync } from './UnreapedChildProcess'; + +const linuxIt: typeof it = process.platform === 'linux' ? it : it.skip; + +function getDaemonPaths(runtimeDir: string): IDaemonPaths { + return { + runtimeDir, + socketPath: path.join(runtimeDir, 'd.sock'), + lockfilePath: path.join(runtimeDir, 'daemon.pid.json') + }; +} + +describe(reclaimCrashedDaemonAsync.name, () => { + let folder: string; + let paths: IDaemonPaths; + let operationPids: number[]; + let warning: jest.SpyInstance; + + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-crash-reclaim-')); + paths = getDaemonPaths(folder); + operationPids = []; + warning = jest.spyOn(process, 'emitWarning').mockImplementation(() => undefined); + }); + + afterEach(async () => { + warning.mockRestore(); + operationPids.forEach(stopOperationIfRunning); + await fs.promises.rm(folder, { recursive: true, force: true }); + }); + + it('does nothing, and creates no runtime folder, when there is no ownership record', async () => { + const runtimeDir: string = path.join(folder, 'runtime'); + await reclaimCrashedDaemonAsync(getDaemonPaths(runtimeDir)); + expect(fs.existsSync(runtimeDir)).toBe(false); + expect(warning).not.toHaveBeenCalled(); + }); + + linuxIt('stops the operations that a crashed daemon left running, and removes its files', async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + await reclaimCrashedDaemonAsync(paths); + expect(isRunning(operationPid)).toBe(false); + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + expect(warning).toHaveBeenCalledWith( + expect.stringContaining(`Reclaimed dead daemon ${daemonPid}:`), + expect.objectContaining({ code: 'RUSH_DAEMON_ORPHANS_REAPED' }) + ); + }); + + linuxIt( + 'leaves a running daemon alone, without waiting while another client holds the start mutex', + async () => { + const { daemon, operationPid }: IStandInDaemon = await startStandInDaemonAsync(operationPids); + // For example, a client that starts or replaces the daemon. + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(paths); + try { + expect(lock).toBeDefined(); + recordDaemonOwner(paths, daemon.pid!); + const startedAt: number = Date.now(); + await reclaimCrashedDaemonAsync(paths); + expect(Date.now() - startedAt).toBeLessThan(1000); + expect(isRunning(daemon.pid!)).toBe(true); + expect(isRunning(operationPid)).toBe(true); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + expect(warning).not.toHaveBeenCalled(); + } finally { + await lock?.releaseAsync(); + daemon.kill('SIGKILL'); + } + } + ); + + linuxIt('reclaims a crashed daemon that is not reaped yet once it is reaped', async () => { + await withUnreapedChildAsync(async (child, parentPid) => { + const deadline: number = Date.now() + 5000; + while (!isProcessDefunct(child) && Date.now() < deadline) await delayAsync(20); + recordDaemonOwner(paths, child); + let settled: boolean = false; + const reclaimed: Promise = reclaimCrashedDaemonAsync(paths).finally(() => { + settled = true; + }); + await delayAsync(300); + expect(settled).toBe(false); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + // Once its parent exits, init or a subreaper reaps it. + process.kill(parentPid, 'SIGTERM'); + await reclaimed; + expect(fs.existsSync(paths.lockfilePath)).toBe(false); + }); + }); + + linuxIt('does not act on the records in a runtime folder that is not private', async () => { + const { daemonPid, operationPid } = await startOrphanedOperationAsync(operationPids); + recordDaemonOwner(paths, daemonPid); + // Another user could have created a link like this one, for example in /tmp. + const link: string = path.join(os.tmpdir(), `${path.basename(folder)}-link`); + fs.symlinkSync(folder, link); + const linkPaths: IDaemonPaths = getDaemonPaths(link); + // It does not even wait for the start mutex there. + const lock: IStartupLock | undefined = await tryAcquireStartupLockAsync(linkPaths); + try { + expect(lock).toBeDefined(); + const startedAt: number = Date.now(); + await reclaimCrashedDaemonAsync(linkPaths); + expect(Date.now() - startedAt).toBeLessThan(1000); + expect(isRunning(operationPid)).toBe(true); + expect(fs.existsSync(paths.lockfilePath)).toBe(true); + expect(warning).not.toHaveBeenCalled(); + } finally { + await lock?.releaseAsync(); + fs.unlinkSync(link); + } + }); +}); diff --git a/libraries/rush-client-core/src/test/OrphanedOperation.ts b/libraries/rush-client-core/src/test/OrphanedOperation.ts new file mode 100644 index 0000000000..561ec1fa7a --- /dev/null +++ b/libraries/rush-client-core/src/test/OrphanedOperation.ts @@ -0,0 +1,90 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import * as fs from 'node:fs'; +import type { Readable } from 'node:stream'; + +import { DAEMON_PROTOCOL_VERSION } from '@rushstack/rush-daemon-protocol'; +import { + isDaemonProcessAlive, + writeDaemonLockfile, + type IDaemonPaths +} from '@rushstack/rush-daemon-transport'; + +import { isProcessDefunct } from '../ProcessStartTime'; + +// An operation process that a stand-in daemon starts; it exits by itself after a minute. +const OPERATION_SCRIPT: string = 'setTimeout(()=>{},60000)'; +// A stand-in daemon: like a phased operation, which is spawned without `detached`, its operation process +// inherits the daemon's process group. It prints that process's PID. +const FAKE_DAEMON_SCRIPT: string = + "const c=require('node:child_process')" + + `.spawn(process.execPath,['-e','${OPERATION_SCRIPT}'],{stdio:'ignore'});` + + "process.stdout.write(String(c.pid)+'\\n');setInterval(()=>{},1000);"; + +/** A stand-in daemon that this test process started, and the operation process that it started. */ +export interface IStandInDaemon { + readonly daemon: ChildProcess; + readonly operationPid: number; +} + +/** + * Starts a stand-in daemon in its own process group, and waits for its operation process. The operation's PID + * is added to `operationPids` first, so that {@link stopOperationIfRunning} can clean it up. POSIX only. + */ +export async function startStandInDaemonAsync(operationPids: number[]): Promise { + const daemon: ChildProcess = spawn(process.execPath, ['-e', FAKE_DAEMON_SCRIPT], { + detached: true, + stdio: ['ignore', 'pipe', 'ignore'] + }); + const stdout: Readable = daemon.stdout!; + const [chunk] = (await once(stdout, 'data')) as [Buffer]; + stdout.destroy(); + const operationPid: number = Number(chunk.toString().trim()); + operationPids.push(operationPid); + return { daemon, operationPid }; +} + +/** Starts a stand-in daemon and SIGKILLs only it, so its operation process keeps running. POSIX only. */ +export async function startOrphanedOperationAsync( + operationPids: number[] +): Promise<{ daemonPid: number; operationPid: number }> { + const { daemon, operationPid } = await startStandInDaemonAsync(operationPids); + daemon.kill('SIGKILL'); + await once(daemon, 'exit'); + if (!isRunning(operationPid)) throw new Error(`The orphaned operation ${operationPid} is not running.`); + return { daemonPid: daemon.pid!, operationPid }; +} + +/** Writes the ownership record that a daemon with this PID would write. */ +export function recordDaemonOwner( + paths: IDaemonPaths, + pid: number, + startedAt: string = new Date().toISOString() +): void { + writeDaemonLockfile(paths.lockfilePath, { + pid, + protocolVersion: DAEMON_PROTOCOL_VERSION, + startedAt, + socketPath: paths.socketPath + }); +} + +/** True while the process exists and has not exited. */ +export function isRunning(pid: number): boolean { + return isDaemonProcessAlive(pid) && !isProcessDefunct(pid); +} + +/** SIGKILLs an operation process that a stand-in daemon started, unless it has exited. Linux only. */ +export function stopOperationIfRunning(pid: number): void { + try { + // Only the operation process that this test started, not a process that reused its PID. + if (fs.readFileSync(`/proc/${pid}/cmdline`, 'utf8').includes(OPERATION_SCRIPT)) { + process.kill(pid, 'SIGKILL'); + } + } catch { + // It has exited. + } +} From 62fc563aa1e92dcba21ae73e9f6e020578415dfa Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 02:37:43 +0000 Subject: [PATCH 060/265] [rush-daemon] Test that an expired protected project takes no warm-set lease Swarm integration step 45; original commit 1b18fcc3eb (merge of swarm/r07-t76-nit at 6c6f638a2e). Scope: task 76 NIT. Brings r07's test for the task 76 NIT. Test only. s16 batch B, item 5 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree 104ca5f27d) Commits folded into this step (1): - 6c6f638a2e [rush-daemon] Warm set: test that an expired protected project takes no lease Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...rotected-expiry-test_2026-09-29-01-45.json | 11 ++++++++ .../WorkspaceWarmSetMaintenanceCost.test.ts | 25 +++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 common/changes/@rushstack/rush-daemon/warm-set-protected-expiry-test_2026-09-29-01-45.json diff --git a/common/changes/@rushstack/rush-daemon/warm-set-protected-expiry-test_2026-09-29-01-45.json b/common/changes/@rushstack/rush-daemon/warm-set-protected-expiry-test_2026-09-29-01-45.json new file mode 100644 index 0000000000..5a9f7775c4 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/warm-set-protected-expiry-test_2026-09-29-01-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Test that a maintenance pass takes no native repository lock for a protected project that has expired while it holds resources.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts b/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts index f8ee0fa972..d6933f23cb 100644 --- a/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceWarmSetMaintenanceCost.test.ts @@ -371,4 +371,29 @@ describe('warm-set maintenance cost', () => { expect(status.protectedProjectNames).toEqual(['p0']); expect(watched.has('p0')).toBe(true); }); + + it('takes no native repository lease for an expired protected project that holds resources, pass after pass', async () => { + const { graph, operations, requestAll } = createGraph(4); + operations[0].runner = createResidentRunner(); + const acquire: jest.Mock, []> = createLease(); + const warm: WorkspaceWarmSet = attach( + graph, + { warmIdleTimeoutSeconds: 0.001 }, + { acquireExecutionLeaseAsync: acquire, getProtectedOperations: () => new Set([operations[0]]) } + ); + disposables.push(warm); + requestAll(); + await settleAsync(warm); + acquire.mockClear(); + + // Expiry never evicts a protected project, so no pass needs to own the repository for it + for (let pass: number = 0; pass < 2; pass++) { + const status: IWorkspaceWarmSetStatus = await warm.maintainAsync(); + expect(status.protectedProjectNames).toEqual(['p0']); + expect(status.deferredReason).toBeUndefined(); + } + expect(acquire).not.toHaveBeenCalled(); + expect(operations[0].runner?.isActive).toBe(true); + expect(graph.resultByOperation.size).toBe(4); + }); }); From 0b978371eaf30e5da00307b338e233d50d936e41 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:15 +0000 Subject: [PATCH 061/265] [package-deps-hash] A hot request with no file change since the previous one skips the repo-state work Swarm integration step 46; original commit a9e930338f (merge of swarm/r07-t114-int at e883c78766). Scope: task 114. Brings r07's task 114 (daemon L3b, D20.3): RepoStateCache keeps the last repo-state capture, and a hot request whose files haven't changed since reuses it instead of running git status, ls-tree, the inputs snapshot and each operation's own-state hash again. It also moves assertCompatibleInputs and isGraphDefinitionPath into WorkspaceInputsComparison.ts with the same path list. Second agents: m01 board 2842 and t04 board 2897 on d0f571a420 (t114verify: 0 STALE rows, board 2829); addenda t04 board 3042 and m01 board 3031. ch01's e2e on ch01-sC: 30 of 30 steps equal to --no-daemon, 0 STALE. s16 batch C, item 1 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 8299cd3e10) Commits folded into this step (4): - 5fbb206eb3 [package-deps-hash] Add RepoStateCache for processes that read the state of a repository repeatedly - b7904dd6d6 [rush-lib] Keep the state of the repository and reuse unchanged inputs snapshots in long-lived hosts - 7cb4c46d89 [rush-daemon] Skip comparing unchanged inputs snapshots on each request - d0f571a420 [package-deps-hash] Copy the index again when the sizes, attributes, configuration or filter change Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...unchanged-repo-state_2026-09-28-21-15.json | 11 + .../repo-state-cache_2026-09-28-21-15.json | 11 + ...ed-inputs-comparison_2026-09-28-21-15.json | 11 + common/reviews/api/package-deps-hash.api.md | 14 + common/reviews/api/rush-lib.api.md | 1 + libraries/package-deps-hash/README.md | 13 + libraries/package-deps-hash/src/FileStamp.ts | 33 + .../package-deps-hash/src/GitIndexFile.ts | 147 +++ .../package-deps-hash/src/RepoStateCache.ts | 859 +++++++++++++++++ .../package-deps-hash/src/getRepoState.ts | 180 +++- libraries/package-deps-hash/src/index.ts | 1 + .../src/test/GitIndexFile.test.ts | 228 +++++ .../src/test/RepoStateCache.test.ts | 912 ++++++++++++++++++ .../src/test/getRepoState.test.ts | 45 +- .../src/ProductionDaemonRequestResolver.ts | 40 +- .../src/WorkspaceInputsComparison.ts | 72 ++ .../test/WorkspaceInputsComparison.test.ts | 156 +++ .../cli/scriptActions/PhasedScriptAction.ts | 8 +- .../src/logic/ProjectChangeAnalyzer.ts | 123 ++- .../logic/test/ProjectChangeAnalyzer.test.ts | 203 +++- 20 files changed, 2971 insertions(+), 97 deletions(-) create mode 100644 common/changes/@microsoft/rush/reuse-unchanged-repo-state_2026-09-28-21-15.json create mode 100644 common/changes/@rushstack/package-deps-hash/repo-state-cache_2026-09-28-21-15.json create mode 100644 common/changes/@rushstack/rush-daemon/skip-unchanged-inputs-comparison_2026-09-28-21-15.json create mode 100644 libraries/package-deps-hash/src/FileStamp.ts create mode 100644 libraries/package-deps-hash/src/GitIndexFile.ts create mode 100644 libraries/package-deps-hash/src/RepoStateCache.ts create mode 100644 libraries/package-deps-hash/src/test/GitIndexFile.test.ts create mode 100644 libraries/package-deps-hash/src/test/RepoStateCache.test.ts create mode 100644 libraries/rush-daemon/src/WorkspaceInputsComparison.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceInputsComparison.test.ts diff --git a/common/changes/@microsoft/rush/reuse-unchanged-repo-state_2026-09-28-21-15.json b/common/changes/@microsoft/rush/reuse-unchanged-repo-state_2026-09-28-21-15.json new file mode 100644 index 0000000000..9c2ac4d92c --- /dev/null +++ b/common/changes/@microsoft/rush/reuse-unchanged-repo-state_2026-09-28-21-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Long-lived hosts such as the Rush daemon now keep the state of the Git repository between requests, using the new `RepoStateCache` from `@rushstack/package-deps-hash`, and reuse the previous inputs snapshot while the repository state, the additional files and the environment are unchanged.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/package-deps-hash/repo-state-cache_2026-09-28-21-15.json b/common/changes/@rushstack/package-deps-hash/repo-state-cache_2026-09-28-21-15.json new file mode 100644 index 0000000000..c17aaeb7a6 --- /dev/null +++ b/common/changes/@rushstack/package-deps-hash/repo-state-cache_2026-09-28-21-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/package-deps-hash", + "comment": "Add a (beta) `RepoStateCache` class for long-lived processes. It returns the same result as `getDetailedRepoStateAsync`, apart from rare cases that its documentation lists, but runs `git status` against a private copy of the Git index, which it copies again when the files that the index records or their recorded sizes change, when the Git configuration or attributes change, or when the filter changes. It runs `git ls-files` only when the files that the index records change, reuses the hash of a file while neither the file nor the `.gitattributes` files in the folders that contain it have changed since they settled, and returns the same object while nothing changes. It falls back to `getDetailedRepoStateAsync` when it fails.", + "type": "minor" + } + ], + "packageName": "@rushstack/package-deps-hash", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/skip-unchanged-inputs-comparison_2026-09-28-21-15.json b/common/changes/@rushstack/rush-daemon/skip-unchanged-inputs-comparison_2026-09-28-21-15.json new file mode 100644 index 0000000000..32d10a1664 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/skip-unchanged-inputs-comparison_2026-09-28-21-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Skip comparing the inputs of a request with those of the engine when the snapshot or its file hashes are the same objects, so an unchanged workspace no longer compares every file and operation hash on each request.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/package-deps-hash.api.md b/common/reviews/api/package-deps-hash.api.md index 1de2d1b706..67a8ebd6e7 100644 --- a/common/reviews/api/package-deps-hash.api.md +++ b/common/reviews/api/package-deps-hash.api.md @@ -48,4 +48,18 @@ export interface IFileDiffStatus { status: 'A' | 'D' | 'M'; } +// @beta +export interface IRepoStateCacheOptions { + gitPath?: string; + rootDirectory: string; + temporaryFolderPath?: string; +} + +// @beta +export class RepoStateCache { + constructor(options: IRepoStateCacheOptions); + dispose(): void; + getDetailedRepoStateAsync(additionalRelativePathsToHash?: ReadonlyArray, filterPath?: ReadonlyArray): Promise; +} + ``` diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 045cf759e9..f96c5f2059 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -1663,6 +1663,7 @@ export class ProjectChangeAnalyzer { // @internal _tryGetSnapshotProviderAsync(projectConfigurations: ReadonlyMap, terminal: ITerminal, projectSelection?: ReadonlySet, options?: { readonly throwOnMissingProjectShrinkwrapFile?: boolean; + readonly reuseUnchangedInputs?: boolean; }): Promise; } diff --git a/libraries/package-deps-hash/README.md b/libraries/package-deps-hash/README.md index 5386749d7a..53a4479465 100644 --- a/libraries/package-deps-hash/README.md +++ b/libraries/package-deps-hash/README.md @@ -29,6 +29,19 @@ if (_.isEqual(deps, existingDeps)) { } ``` +A long-lived process that reads the state of the same repository repeatedly can use a `RepoStateCache` (beta). +It returns the same result as `getDetailedRepoStateAsync()`, but keeps a private copy of the Git index and the +hashes of files that haven't changed between calls, and returns the same object while nothing changes: + +```ts +import { RepoStateCache } from '@rushstack/package-deps-hash'; + +const cache = new RepoStateCache({ rootDirectory: '/path/to/repo' }); +const state = await cache.getDetailedRepoStateAsync(); +// ... +cache.dispose(); +``` + ## Links - [CHANGELOG.md]( diff --git a/libraries/package-deps-hash/src/FileStamp.ts b/libraries/package-deps-hash/src/FileStamp.ts new file mode 100644 index 0000000000..aa3b0c68bd --- /dev/null +++ b/libraries/package-deps-hash/src/FileStamp.ts @@ -0,0 +1,33 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type * as fs from 'node:fs'; + +/** + * A file whose ctime or mtime is newer than this many milliseconds must not be memoized by its stamp. + * + * @remarks + * Every write updates a file's ctime, and userspace can't set it. A stamp recorded for a file whose last change + * was already this old can't hide a later write, because that write gets a newer ctime even on a filesystem + * whose timestamps have a coarse granularity (a jiffy on Linux, 2 seconds on FAT). The argument assumes that the + * filesystem's clock agrees with this process's clock to within this margin, as a local filesystem's does. + */ +export const SETTLED_FILE_AGE_MS: number = 3000; + +/** + * The ctime and mtime a file must be older than for its stamp to be memoized. Take it before examining any of + * the files, so that a write made after a file's examination gets a later ctime. + */ +export function getSettledBeforeNs(): bigint { + return BigInt(Date.now() - SETTLED_FILE_AGE_MS) * BigInt(1e6); +} + +/** Identifies a file's content by the identity, size, nanosecond mtime and ctime of the file. */ +export function getFileStamp(stats: fs.BigIntStats): string { + return `${stats.dev}:${stats.ino}:${stats.size}:${stats.mtimeNs}:${stats.ctimeNs}`; +} + +/** Whether a file examined after `settledBeforeNs` was taken may be memoized by its stamp. */ +export function isFileStatSettled(stats: fs.BigIntStats, settledBeforeNs: bigint): boolean { + return stats.ctimeNs < settledBeforeNs && stats.mtimeNs < settledBeforeNs; +} diff --git a/libraries/package-deps-hash/src/GitIndexFile.ts b/libraries/package-deps-hash/src/GitIndexFile.ts new file mode 100644 index 0000000000..4f600b9ce7 --- /dev/null +++ b/libraries/package-deps-hash/src/GitIndexFile.ts @@ -0,0 +1,147 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// The index format packs flags into bits +/* eslint-disable no-bitwise */ + +import { createHash } from 'node:crypto'; + +// The format is described in https://git-scm.com/docs/index-format +const INDEX_SIGNATURE: string = 'DIRC'; +const HEADER_LENGTH: number = 12; +// ctime, mtime, dev, ino, mode, uid, gid and size, 4 bytes each except the 8-byte times +const ENTRY_STAT_LENGTH: number = 40; +const ENTRY_MODE_OFFSET: number = 24; +const ENTRY_MODE_LENGTH: number = 4; +const ENTRY_SIZE_OFFSET: number = 36; +const ENTRY_SIZE_LENGTH: number = 4; +const ENTRY_FLAGS_LENGTH: number = 2; +const ENTRY_EXTENDED_FLAG: number = 0x4000; +const EXTENSION_HEADER_LENGTH: number = 8; +const SPLIT_INDEX_EXTENSION_SIGNATURE: string = 'link'; + +/** + * A summary of a Git index file. + */ +export interface IGitIndexSummary { + readonly entryCount: number; + /** + * A digest of the header and of every entry's mode, object ID, flags and path. It omits the file system data + * (times, device, inode, owner and size) that refreshing the index updates, and the extensions, such as the + * untracked cache and the file system monitor's token. Two indexes with the same digest describe the same files. + */ + readonly entriesDigest: string; + /** + * A digest of the size that the index records for each entry. Git considers a file modified, without examining + * its content, if the recorded size is not 0 and differs from the size of the file. So two indexes with the same + * entries digest may still disagree about which files are modified, unless their sizes digests are the same too. + */ + readonly sizesDigest: string; + /** + * Whether the index is split: its entries are then completed by a shared index file. + */ + readonly isSplit: boolean; +} + +/** + * Returns the number of entries that the header of a Git index file declares, or `undefined` if the data doesn't + * start with the header of a Git index file. + */ +export function tryGetGitIndexEntryCount(header: Buffer): number | undefined { + if ( + header.length < HEADER_LENGTH || + header.toString('latin1', 0, INDEX_SIGNATURE.length) !== INDEX_SIGNATURE + ) { + return undefined; + } + + return header.readUInt32BE(8); +} + +/** + * Summarizes the content of a Git index file of version 2, 3 or 4. + * + * @param content - The content of the index file + * @param objectIdLength - The length of an object ID in bytes: 20 for SHA-1, or 32 for SHA-256 + */ +export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGitIndexSummary { + const entryCount: number | undefined = tryGetGitIndexEntryCount(content); + if (entryCount === undefined) { + throw new Error('The file is not a Git index'); + } + + const version: number = content.readUInt32BE(4); + if (version < 2 || version > 4) { + throw new Error(`Unsupported Git index version ${version}`); + } + + const checksumOffset: number = content.length - objectIdLength; + // Each entry has at least its file system data, object ID and flags, and a NUL after its path + if ( + HEADER_LENGTH + entryCount * (ENTRY_STAT_LENGTH + objectIdLength + ENTRY_FLAGS_LENGTH + 1) > + checksumOffset + ) { + throw new Error('The Git index ends within an entry'); + } + + // A copy of the entries in which the file system data of each entry is cleared, except for the mode + const entries: Buffer = Buffer.from(content.subarray(0, checksumOffset)); + const sizes: Buffer = Buffer.alloc(entryCount * ENTRY_SIZE_LENGTH); + let offset: number = HEADER_LENGTH; + for (let i: number = 0; i < entryCount; i++) { + const flagsOffset: number = offset + ENTRY_STAT_LENGTH + objectIdLength; + if (flagsOffset + ENTRY_FLAGS_LENGTH > checksumOffset) { + throw new Error('The Git index ends within an entry'); + } + + sizes.writeUInt32BE(content.readUInt32BE(offset + ENTRY_SIZE_OFFSET), i * ENTRY_SIZE_LENGTH); + entries.fill(0, offset, offset + ENTRY_MODE_OFFSET); + entries.fill(0, offset + ENTRY_MODE_OFFSET + ENTRY_MODE_LENGTH, offset + ENTRY_STAT_LENGTH); + + const flags: number = content.readUInt16BE(flagsOffset); + let pathOffset: number = flagsOffset + ENTRY_FLAGS_LENGTH; + if (flags & ENTRY_EXTENDED_FLAG) { + // Extended flags follow, for example "skip-worktree" and "intent-to-add" + pathOffset += ENTRY_FLAGS_LENGTH; + } + + if (version === 4) { + // The path is prefix-compressed: a variable-length integer, then the rest of the path and a NUL + while (pathOffset < checksumOffset && content[pathOffset] & 0x80) { + pathOffset++; + } + pathOffset++; + } + + const pathEnd: number = content.indexOf(0, pathOffset); + if (pathEnd < 0 || pathEnd >= checksumOffset) { + throw new Error('The Git index ends within an entry'); + } + + // Before version 4, each entry is padded with 1-8 NULs to a multiple of 8 bytes + offset = version === 4 ? pathEnd + 1 : offset + ((pathEnd - offset + 8) & ~7); + } + + const entriesDigest: string = createHash('sha1').update(entries.subarray(4, offset)).digest('hex'); + const sizesDigest: string = createHash('sha1').update(sizes).digest('hex'); + + let isSplit: boolean = false; + while (offset + EXTENSION_HEADER_LENGTH <= checksumOffset) { + const signature: string = content.toString('latin1', offset, offset + 4); + if (signature === SPLIT_INDEX_EXTENSION_SIGNATURE) { + isSplit = true; + } + offset += EXTENSION_HEADER_LENGTH + content.readUInt32BE(offset + 4); + } + + if (offset !== checksumOffset) { + throw new Error('The extensions of the Git index are malformed'); + } + + return { + entryCount, + entriesDigest, + sizesDigest, + isSplit + }; +} diff --git a/libraries/package-deps-hash/src/RepoStateCache.ts b/libraries/package-deps-hash/src/RepoStateCache.ts new file mode 100644 index 0000000000..2ff67fd0b3 --- /dev/null +++ b/libraries/package-deps-hash/src/RepoStateCache.ts @@ -0,0 +1,859 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { FileSystem } from '@rushstack/node-core-library/lib/FileSystem'; + +import { getFileStamp, getSettledBeforeNs, isFileStatSettled } from './FileStamp'; +import { type IGitIndexSummary, summarizeGitIndex, tryGetGitIndexEntryCount } from './GitIndexFile'; +import { + classifyLocallyModifiedFiles, + getCleanGitEnvironment, + getDetailedRepoStateAsync, + getGitLsFilesArgs, + getGitStatusArgs, + hashFilesAsync, + type IDetailedRepoState, + type IGitTreeState, + type ILocallyModifiedFiles, + parseGitLsTree, + parseGitStatus, + spawnGitAsync, + STANDARD_GIT_OPTIONS +} from './getRepoState'; + +/** + * Options for {@link RepoStateCache}. + * @beta + */ +export interface IRepoStateCacheOptions { + /** + * The root directory of the Git repository. + */ + rootDirectory: string; + /** + * The path to the Git executable. + */ + gitPath?: string; + /** + * The folder in which to create the folder that holds the copy of the Git index. + * @defaultValue The temporary folder of the operating system + */ + temporaryFolderPath?: string; +} + +// After this many consecutive failures, stop trying to use the cache +const MAX_CONSECUTIVE_FAILURES: number = 3; +const SHA256_OBJECT_ID_LENGTH: number = 32; +const SHA1_OBJECT_ID_LENGTH: number = 20; +const INDEX_HEADER_LENGTH: number = 12; +const NANOSECONDS_PER_SECOND: bigint = BigInt(1e9); +const PRIVATE_FOLDER_PREFIX: string = 'package-deps-hash-'; +const PRIVATE_INDEX_NAME: string = 'index'; +const ATTRIBUTES_FILE_NAME: string = '.gitattributes'; + +// Unlike the other commands, "git status" doesn't pass "--no-optional-locks", so that it can save its refreshed +// copy of the index and the next "git status" doesn't need to examine the files that haven't changed since. +const PRIVATE_INDEX_STATUS_OPTIONS: readonly string[] = [ + // Ensure that commands don't run automatic maintenance, since performance of the command itself is paramount + '-c', + 'maintenance.auto=false', + // Git only keeps an untracked cache that matches this setting, and "git status -u" lists all untracked files + '-c', + 'status.showUntrackedFiles=all', + // Git would save the shared part of a split index in the Git folder of the repository + '-c', + 'core.splitIndex=false' +]; + +interface IGitPaths { + readonly indexPath: string; + readonly objectIdLength: number; + /** + * The configuration and attributes files that the hashes of files may depend on. + */ + readonly configurationPaths: ReadonlyArray; +} + +interface IPrivateIndex { + readonly path: string; + readonly entryCount: number; + readonly entriesDigest: string; + readonly sizesDigest: string; + /** + * Identifies the filter of the calls that use the copy. "git status" only refreshes the files within the filter, + * and only lists the attributes files within it. + */ + readonly filterKey: string; + realIndexStamp: string; + isRealIndexStampSettled: boolean; + /** + * Identifies the attributes files under which the first "git status" refreshed the copy, once it ran. + */ + attributesFingerprint: string | undefined; + /** + * Whether the attributes files changed since the first "git status", so that the index must be copied again. + */ + hasAttributesChanged: boolean; +} + +interface ITree { + readonly filterKey: string; + readonly state: IGitTreeState; +} + +interface IFileHash { + readonly stamp: string; + readonly hash: string; +} + +interface IResult { + readonly tree: ITree; + readonly additionalPaths: ReadonlyArray; + readonly statusOutput: string; + readonly additionalHashes: ReadonlyMap; + readonly modifiedHashes: ReadonlyMap; + readonly state: IDetailedRepoState; +} + +function noop(): void {} + +/** + * Computes the same state of a Git repository as {@link getDetailedRepoStateAsync}, but faster when a process + * computes it repeatedly. + * + * @remarks + * The commands that {@link getDetailedRepoStateAsync} runs must not update the Git index, so `git status` + * examines every file each time. This class keeps a private copy of the index that `git status` updates, so that + * each call only examines the files that changed since the previous call. It copies the index again when the + * files that the index records, or the sizes that it records for them, change, but not when Git merely refreshes + * the index. While the files that the index records don't change, it also reuses the list of files in the index. + * It reuses the hash of a file while the identity, size and times of the file and of the `.gitattributes` files in + * the folders that contain it don't change, and returns the same state as the previous call if nothing changed. + * + * A call may return the same state as an earlier call, so the state must not be modified. + * + * The state also depends on the attributes and configuration of Git, under which `git status` refreshed the copy. + * The index is copied again, and the files hashed again, when the repository's configuration or `info/attributes` + * file changes, or when the user's `.gitconfig` file or the Git configuration or attributes file in the user's + * configuration folder changes. The index is copied again when a `.gitattributes` file changes that `git status` + * lists as modified or untracked, or that is in a folder that contains a filter path; the call that detects the + * change computes the state without the cache. The index is also copied again when the filter changes. Changes to + * the system configuration, to a configuration file included by another, to a custom `core.attributesFile`, or to + * an ignored `.gitattributes` file that applies to files that the index records aren't detected. In rare cases, + * the state reports uncommitted changes that {@link getDetailedRepoStateAsync} doesn't: when a file whose recorded + * size Git refreshed in the copy but not in the index is rewritten with content that Git converts to the same + * object, for example with other line endings. + * + * Repositories with submodules or a split index fall back to {@link getDetailedRepoStateAsync}, as does a cache + * that fails repeatedly. + * @beta + */ +export class RepoStateCache { + readonly #rootDirectory: string; + readonly #gitPath: string | undefined; + readonly #temporaryFolderPath: string; + // Calls run one at a time, in order + #queue: Promise = Promise.resolve(); + #isDisabled: boolean = false; + #consecutiveFailureCount: number = 0; + #privateFolderPath: string | undefined; + #gitPaths: IGitPaths | undefined; + #privateIndex: IPrivateIndex | undefined; + #tree: ITree | undefined; + #previousResult: IResult | undefined; + #configurationFingerprint: string = ''; + #unsettledFingerprintCount: number = 0; + readonly #fileHashes: Map = new Map(); + + public constructor(options: IRepoStateCacheOptions) { + const { rootDirectory, gitPath, temporaryFolderPath = os.tmpdir() } = options; + this.#rootDirectory = rootDirectory; + this.#gitPath = gitPath; + this.#temporaryFolderPath = temporaryFolderPath; + } + + /** + * Gets the object hashes for all files in the Git repo, combining the current commit with working tree state. + * Returns the same result as {@link getDetailedRepoStateAsync}. + * + * @param additionalRelativePathsToHash - Root-relative file paths to have Git hash and include in the results + * @param filterPath - The paths to which to limit the results + * @returns The state of the repository. It may be the same object that an earlier call returned, so it must not + * be modified. + */ + public async getDetailedRepoStateAsync( + additionalRelativePathsToHash: ReadonlyArray = [], + filterPath?: ReadonlyArray + ): Promise { + const resultPromise: Promise = this.#queue.then(() => + this.#getStateAsync(additionalRelativePathsToHash, filterPath) + ); + this.#queue = resultPromise.then(noop, noop); + return await resultPromise; + } + + /** + * Deletes the copy of the Git index. Later calls compute the state without the cache. + */ + public dispose(): void { + this.#disable(); + } + + async #getStateAsync( + additionalPaths: ReadonlyArray, + filterPath: ReadonlyArray | undefined + ): Promise { + if (!this.#isDisabled) { + let state: IDetailedRepoState | undefined; + try { + state = await this.#tryGetCachedStateAsync(additionalPaths, filterPath); + } catch { + this.#reset(); + // If the state can't be computed without the cache either, report that error instead + state = await this.#getUncachedStateAsync(additionalPaths, filterPath); + if (++this.#consecutiveFailureCount >= MAX_CONSECUTIVE_FAILURES) { + this.#disable(); + } + + return state; + } + + if (state) { + this.#consecutiveFailureCount = 0; + return state; + } + } + + return await this.#getUncachedStateAsync(additionalPaths, filterPath); + } + + async #getUncachedStateAsync( + additionalPaths: ReadonlyArray, + filterPath: ReadonlyArray | undefined + ): Promise { + return await getDetailedRepoStateAsync( + this.#rootDirectory, + Array.from(additionalPaths), + this.#gitPath, + filterPath && Array.from(filterPath) + ); + } + + /** + * Returns `undefined` if the cache can't compute the state of this repository. + */ + async #tryGetCachedStateAsync( + additionalPaths: ReadonlyArray, + filterPath: ReadonlyArray | undefined + ): Promise { + const settledBeforeNs: bigint = getSettledBeforeNs(); + const gitPaths: IGitPaths = await this.#getGitPathsAsync(); + const filterKey: string = JSON.stringify(filterPath ?? []); + const configurationFingerprint: string = this.#getFingerprint( + gitPaths.configurationPaths, + settledBeforeNs + ); + const hasConfigurationChanged: boolean = configurationFingerprint !== this.#configurationFingerprint; + if (hasConfigurationChanged) { + this.#configurationFingerprint = configurationFingerprint; + this.#fileHashes.clear(); + } + + // Git refreshed the copy of the index under the previous configuration, so it could trust the recorded data of + // a file that the new configuration converts differently + const privateIndex: IPrivateIndex | undefined = await this.#tryUpdatePrivateIndexAsync( + gitPaths, + settledBeforeNs, + filterKey, + hasConfigurationChanged + ); + if (!privateIndex) { + return undefined; + } + + const environment: NodeJS.ProcessEnv = { + ...getCleanGitEnvironment(), + GIT_INDEX_FILE: privateIndex.path + }; + // No other process uses the copy of the index, so "git status" may always save it + delete environment.GIT_OPTIONAL_LOCKS; + // The stamps of the attributes files in the folders that contain the files to hash + const attributesStamps: Map = new Map(); + // Hash the additional files while Git reads the index. Wait for both even if one fails, so that no command + // still uses the copy of the index after this call. + const [[tree, statusOutput], additionalHashes] = await waitForBothAsync( + this.#getTreeAndStatusAsync(privateIndex, filterPath, filterKey, environment), + this.#hashFilesAsync(additionalPaths, settledBeforeNs, attributesStamps) + ); + if (!tree) { + return undefined; + } + + // "git status" and "git ls-files" silently treat a missing index as an empty one + if (tryReadGitIndexEntryCount(privateIndex.path) !== privateIndex.entryCount) { + throw new Error(`The copy of the Git index at "${privateIndex.path}" was modified by another process`); + } + + const locallyModified: Map = parseGitStatus(statusOutput); + const attributesFingerprint: string = this.#getFingerprint( + getAttributesFilePaths(this.#rootDirectory, locallyModified.keys(), filterPath), + settledBeforeNs + ); + if (privateIndex.attributesFingerprint === undefined) { + privateIndex.attributesFingerprint = attributesFingerprint; + } else if (attributesFingerprint !== privateIndex.attributesFingerprint) { + // Git refreshed the copy under the previous attributes, so it could trust the recorded data of a file that + // the new attributes convert differently. The next call copies the index again. + privateIndex.hasAttributesChanged = true; + return undefined; + } + + const { filesToHash, filesToRemove }: ILocallyModifiedFiles = classifyLocallyModifiedFiles( + locallyModified, + tree.state.symlinks + ); + const modifiedHashes: Map = await this.#hashFilesAsync( + filesToHash, + settledBeforeNs, + attributesStamps + ); + + const previousResult: IResult | undefined = this.#previousResult; + if ( + previousResult && + previousResult.tree === tree && + previousResult.statusOutput === statusOutput && + areArraysEqual(previousResult.additionalPaths, additionalPaths) && + areMapsEqual(previousResult.additionalHashes, additionalHashes) && + areMapsEqual(previousResult.modifiedHashes, modifiedHashes) + ) { + return previousResult.state; + } + + const files: Map = new Map(tree.state.files); + const symlinks: Map = new Map(tree.state.symlinks); + for (const filePath of filesToRemove) { + files.delete(filePath); + symlinks.delete(filePath); + } + + for (const [filePath, hash] of additionalHashes) { + files.set(filePath, hash); + } + + for (const [filePath, hash] of modifiedHashes) { + files.set(filePath, hash); + } + + const state: IDetailedRepoState = { + hasSubmodules: false, + hasUncommittedChanges: locallyModified.size > 0, + files, + symlinks + }; + this.#previousResult = { + tree, + additionalPaths: Array.from(additionalPaths), + statusOutput, + additionalHashes, + modifiedHashes, + state + }; + return state; + } + + /** + * Runs `git status` and, unless the list from an earlier call is still current, lists the files in the index. + * Returns an undefined tree if the repository has submodules. + */ + async #getTreeAndStatusAsync( + privateIndex: IPrivateIndex, + filterPath: ReadonlyArray | undefined, + filterKey: string, + environment: NodeJS.ProcessEnv + ): Promise<[ITree | undefined, string]> { + const currentTree: ITree | undefined = this.#tree; + let treePromise: Promise; + // An interrupted "git status" may have left its lock file behind, which would stop the next one from saving + await fs.promises.rm(`${privateIndex.path}.lock`, { force: true }); + if (currentTree?.filterKey === filterKey) { + treePromise = Promise.resolve(currentTree); + } else { + this.#tree = undefined; + // "git status" may save its refreshed copy of the index while this reads it. Git replaces the file rather + // than writing to it, and a refresh changes only what the index records about the files, not which files + // it records, so this lists the same files from either version. + treePromise = spawnGitAsync( + this.#gitPath, + STANDARD_GIT_OPTIONS.concat(getGitLsFilesArgs(filterPath)), + this.#rootDirectory, + undefined, + environment + ).then((lsFilesOutput: string): ITree => { + this.#tree = { filterKey, state: parseGitLsTree(lsFilesOutput) }; + return this.#tree; + }); + } + + const statusPromise: Promise = spawnGitAsync( + this.#gitPath, + PRIVATE_INDEX_STATUS_OPTIONS.concat(getGitStatusArgs(filterPath)), + this.#rootDirectory, + undefined, + environment + ); + const [tree, statusOutput] = await waitForBothAsync(treePromise, statusPromise); + if (tree.state.submodules.size > 0 && FileSystem.exists(`${this.#rootDirectory}/.gitmodules`)) { + this.#disable(); + return [undefined, '']; + } + + return [tree, statusOutput]; + } + + async #getGitPathsAsync(): Promise { + if (!this.#gitPaths) { + const output: string = await spawnGitAsync( + this.#gitPath, + STANDARD_GIT_OPTIONS.concat([ + 'rev-parse', + '--show-object-format', + '--git-path', + 'index', + '--git-path', + 'info/attributes', + '--git-path', + 'config', + '--git-path', + 'config.worktree' + ]), + this.#rootDirectory + ); + const [objectFormat, indexPath, ...repositoryConfigurationPaths] = output.trimEnd().split('\n'); + const homeFolderPath: string = os.homedir(); + const userConfigurationFolderPath: string = + process.env.XDG_CONFIG_HOME || path.join(homeFolderPath, '.config'); + this.#gitPaths = { + indexPath: path.resolve(this.#rootDirectory, indexPath), + objectIdLength: objectFormat === 'sha256' ? SHA256_OBJECT_ID_LENGTH : SHA1_OBJECT_ID_LENGTH, + configurationPaths: [ + ...repositoryConfigurationPaths.map((relativePath: string) => + path.resolve(this.#rootDirectory, relativePath) + ), + path.join(homeFolderPath, '.gitconfig'), + path.join(userConfigurationFolderPath, 'git', 'config'), + path.join(userConfigurationFolderPath, 'git', 'attributes') + ] + }; + } + + return this.#gitPaths; + } + + /** + * Ensures that the private copy of the index records the same files, with the same sizes, as the index of the + * repository, and that Git refreshed it under the current configuration and attributes, and for the same filter. + * Returns `undefined` if the repository has no index, or a split index. + */ + async #tryUpdatePrivateIndexAsync( + gitPaths: IGitPaths, + settledBeforeNs: bigint, + filterKey: string, + mustCopy: boolean + ): Promise { + let handle: fs.promises.FileHandle; + try { + handle = await fs.promises.open(gitPaths.indexPath, 'r'); + } catch (error) { + if (FileSystem.isNotExistError(error as Error)) { + return undefined; + } + + throw error; + } + + try { + const stats: fs.BigIntStats = await handle.stat({ bigint: true }); + const stamp: string = getFileStamp(stats); + const privateIndex: IPrivateIndex | undefined = this.#privateIndex; + const isCopyCurrent: boolean = + privateIndex !== undefined && + !mustCopy && + !privateIndex.hasAttributesChanged && + privateIndex.filterKey === filterKey && + tryReadGitIndexEntryCount(privateIndex.path) === privateIndex.entryCount; + // A stamp recorded before the index settled could also match a later version of the index + if (isCopyCurrent && privateIndex?.isRealIndexStampSettled && privateIndex.realIndexStamp === stamp) { + return privateIndex; + } + + // Git replaces the index rather than writing to it, so the open file doesn't change + const content: Buffer = await handle.readFile(); + const summary: IGitIndexSummary = summarizeGitIndex(content, gitPaths.objectIdLength); + if (summary.isSplit) { + this.#disable(); + return undefined; + } + + const isRealIndexStampSettled: boolean = isFileStatSettled(stats, settledBeforeNs); + if (privateIndex?.entriesDigest === summary.entriesDigest) { + // Refreshing the index, as "git status" does, changes the stamp. It changes the recorded sizes only of files + // whose content Git examined, such as a file that was rewritten with other line endings. + if (isCopyCurrent && privateIndex.sizesDigest === summary.sizesDigest) { + privateIndex.realIndexStamp = stamp; + privateIndex.isRealIndexStampSettled = isRealIndexStampSettled; + return privateIndex; + } + } else { + // The hashes of files don't depend on the index: "git hash-object" doesn't read it, not even for a + // ".gitattributes" file that is missing from the working tree + this.#tree = undefined; + } + + return await this.#writePrivateIndexAsync(content, stats, summary, filterKey, isRealIndexStampSettled); + } finally { + await handle.close(); + } + } + + async #writePrivateIndexAsync( + content: Buffer, + stats: fs.BigIntStats, + summary: IGitIndexSummary, + filterKey: string, + isRealIndexStampSettled: boolean + ): Promise { + this.#privateIndex = undefined; + + const folderPath: string = this.#getPrivateFolderPath(); + const indexPath: string = path.join(folderPath, PRIVATE_INDEX_NAME); + const temporaryPath: string = `${indexPath}.new`; + await fs.promises.writeFile(temporaryPath, content); + // Git doesn't trust the recorded times and size of a file that changed in the same second as the index was + // written, because the file may have changed again after it was recorded. Make the copy older than the index, + // so that Git doesn't trust any file in the copy that it wouldn't trust in the index. + const timeInSeconds: number = Number(stats.mtimeNs / NANOSECONDS_PER_SECOND) - 1; + await fs.promises.utimes(temporaryPath, timeInSeconds, timeInSeconds); + await fs.promises.rename(temporaryPath, indexPath); + + const privateIndex: IPrivateIndex = { + path: indexPath, + entryCount: summary.entryCount, + entriesDigest: summary.entriesDigest, + sizesDigest: summary.sizesDigest, + filterKey, + realIndexStamp: getFileStamp(stats), + isRealIndexStampSettled, + attributesFingerprint: undefined, + hasAttributesChanged: false + }; + this.#privateIndex = privateIndex; + return privateIndex; + } + + /** + * Hashes the files with `git hash-object`, except those whose stamps match the stamps they had when they were + * hashed before. The hashes are in the same order as the files. + */ + async #hashFilesAsync( + filePaths: ReadonlyArray, + settledBeforeNs: bigint, + attributesStamps: Map + ): Promise> { + const hashes: Map = new Map(); + if (filePaths.length === 0) { + return hashes; + } + + const statsList: (fs.BigIntStats | undefined)[] = await Promise.all( + filePaths.map(async (filePath: string) => { + try { + return await fs.promises.lstat(path.resolve(this.#rootDirectory, filePath), { bigint: true }); + } catch { + // "git hash-object" reports the error + return undefined; + } + }) + ); + + const filesToHash: string[] = []; + const stampsToRecord: Map = new Map(); + for (let i: number = 0; i < filePaths.length; i++) { + const filePath: string = filePaths[i]; + const stats: fs.BigIntStats | undefined = statsList[i]; + // A symbolic link, or a folder, is hashed afresh each time. The hash of a file also depends on the attributes + // files in the folders that contain it, including ignored ones, which "git status" doesn't list. + let stamp: string | undefined; + if (stats?.isFile()) { + const attributesStamp: string | undefined = this.#getAttributesStamp( + getParentFolderPath(filePath), + settledBeforeNs, + attributesStamps + ); + if (attributesStamp !== undefined) { + stamp = `${getFileStamp(stats)}\n${attributesStamp}`; + } + } + + const fileHash: IFileHash | undefined = this.#fileHashes.get(filePath); + if (stamp !== undefined && fileHash?.stamp === stamp) { + hashes.set(filePath, fileHash.hash); + } else { + // Reserve the position of the file + hashes.set(filePath, ''); + filesToHash.push(filePath); + stampsToRecord.set(filePath, stats && isFileStatSettled(stats, settledBeforeNs) ? stamp : undefined); + } + } + + if (filesToHash.length > 0) { + for (const [filePath, hash] of await hashFilesAsync(this.#rootDirectory, filesToHash, this.#gitPath)) { + hashes.set(filePath, hash); + // The file was examined before it was hashed, so a later change gives it a different stamp + const stamp: string | undefined = stampsToRecord.get(filePath); + if (stamp === undefined) { + this.#fileHashes.delete(filePath); + } else { + this.#fileHashes.set(filePath, { stamp, hash }); + } + } + } + + return hashes; + } + + /** + * Identifies the versions of the attributes files in the folder and in the folders that contain it, whether or not + * they exist. Returns `undefined` if one of them changed recently. + */ + #getAttributesStamp( + folderPath: string, + settledBeforeNs: bigint, + attributesStamps: Map + ): string | undefined { + if (attributesStamps.has(folderPath)) { + return attributesStamps.get(folderPath); + } + + let stamp: string | undefined = this.#getAttributesFileStamp(folderPath, settledBeforeNs); + if (stamp !== undefined && folderPath) { + const parentStamp: string | undefined = this.#getAttributesStamp( + getParentFolderPath(folderPath), + settledBeforeNs, + attributesStamps + ); + stamp = parentStamp === undefined ? undefined : `${parentStamp}/${stamp}`; + } + + attributesStamps.set(folderPath, stamp); + return stamp; + } + + #getAttributesFileStamp(folderPath: string, settledBeforeNs: bigint): string | undefined { + let stats: fs.BigIntStats | undefined; + try { + stats = fs.lstatSync(path.resolve(this.#rootDirectory, folderPath, ATTRIBUTES_FILE_NAME), { + bigint: true, + throwIfNoEntry: false + }); + } catch (error) { + // For example, a file in place of a folder + return `!${(error as NodeJS.ErrnoException).code}`; + } + + if (!stats) { + return ''; + } + + return isFileStatSettled(stats, settledBeforeNs) ? getFileStamp(stats) : undefined; + } + + /** + * Identifies the versions of the files. A file that changed recently gets a fingerprint that matches no other. + */ + #getFingerprint(filePaths: Iterable, settledBeforeNs: bigint): string { + let fingerprint: string = ''; + for (const filePath of filePaths) { + let stats: fs.BigIntStats | undefined; + try { + stats = fs.statSync(filePath, { bigint: true }); + } catch (error) { + fingerprint += `${filePath}\0${(error as NodeJS.ErrnoException).code}\n`; + continue; + } + + if (!isFileStatSettled(stats, settledBeforeNs)) { + // The file may change again without changing its stamp + return `${++this.#unsettledFingerprintCount}`; + } + + fingerprint += `${filePath}\0${getFileStamp(stats)}\n`; + } + + return fingerprint; + } + + #getPrivateFolderPath(): string { + if (this.#isDisabled) { + throw new Error('The cache is disabled'); + } + + if (!this.#privateFolderPath) { + fs.mkdirSync(this.#temporaryFolderPath, { recursive: true }); + this.#privateFolderPath = fs.mkdtempSync(path.join(this.#temporaryFolderPath, PRIVATE_FOLDER_PREFIX)); + addPrivateFolder(this.#privateFolderPath); + } + + return this.#privateFolderPath; + } + + #reset(): void { + this.#privateIndex = undefined; + this.#tree = undefined; + this.#previousResult = undefined; + this.#configurationFingerprint = ''; + this.#fileHashes.clear(); + // Start over in a new folder, in case something deleted this one + if (this.#privateFolderPath) { + deletePrivateFolder(this.#privateFolderPath); + this.#privateFolderPath = undefined; + } + } + + #disable(): void { + this.#isDisabled = true; + this.#reset(); + } +} + +// The folders of all caches share one listener, which is registered while any folder exists +const privateFolderPaths: Set = new Set(); + +function addPrivateFolder(folderPath: string): void { + if (privateFolderPaths.size === 0) { + process.on('exit', deletePrivateFolders); + } + + privateFolderPaths.add(folderPath); +} + +function deletePrivateFolder(folderPath: string): void { + privateFolderPaths.delete(folderPath); + if (privateFolderPaths.size === 0) { + process.off('exit', deletePrivateFolders); + } + + fs.rmSync(folderPath, { recursive: true, force: true }); +} + +function deletePrivateFolders(): void { + for (const folderPath of privateFolderPaths) { + fs.rmSync(folderPath, { recursive: true, force: true }); + } +} + +/** + * Reads the number of entries from the header of a Git index file, or returns `undefined` if the file doesn't + * exist or isn't a Git index. + */ +function tryReadGitIndexEntryCount(indexPath: string): number | undefined { + let fileDescriptor: number; + try { + fileDescriptor = fs.openSync(indexPath, 'r'); + } catch (error) { + if (FileSystem.isNotExistError(error as Error)) { + return undefined; + } + + throw error; + } + + try { + const header: Buffer = Buffer.alloc(INDEX_HEADER_LENGTH); + const bytesRead: number = fs.readSync(fileDescriptor, header, 0, INDEX_HEADER_LENGTH, 0); + return tryGetGitIndexEntryCount(header.subarray(0, bytesRead)); + } finally { + fs.closeSync(fileDescriptor); + } +} + +/** + * Returns the path of the folder that contains the file or folder, or '' for the root folder of the repository. + */ +function getParentFolderPath(relativePath: string): string { + const separatorIndex: number = Math.max(relativePath.lastIndexOf('/'), relativePath.lastIndexOf('\\')); + return separatorIndex < 0 ? '' : relativePath.slice(0, separatorIndex); +} + +/** + * Lists the attributes files that affect how "git status" refreshes the files within the filter: those that it + * lists as modified or untracked, and those in the folders that contain the filter paths, which it doesn't list. + */ +function* getAttributesFilePaths( + rootDirectory: string, + modifiedFilePaths: Iterable, + filterPath: ReadonlyArray | undefined +): IterableIterator { + for (const filePath of modifiedFilePaths) { + if (filePath === ATTRIBUTES_FILE_NAME || filePath.endsWith(`/${ATTRIBUTES_FILE_NAME}`)) { + yield path.resolve(rootDirectory, filePath); + } + } + + const rootPath: string = path.resolve(rootDirectory); + const folderPaths: Set = new Set(); + for (const filterEntry of filterPath ?? []) { + let folderPath: string = path.resolve(rootPath, filterEntry); + while (!folderPaths.has(folderPath)) { + folderPaths.add(folderPath); + const parentFolderPath: string = path.dirname(folderPath); + if (folderPath === rootPath || parentFolderPath === folderPath) { + break; + } + + folderPath = parentFolderPath; + } + } + + for (const folderPath of folderPaths) { + yield path.join(folderPath, ATTRIBUTES_FILE_NAME); + } +} + +/** + * Waits for both promises to settle, then fails with the first error, if any. + */ +async function waitForBothAsync(promise1: Promise, promise2: Promise): Promise<[T1, T2]> { + const [result1, result2] = await Promise.allSettled([promise1, promise2]); + if (result1.status === 'rejected') { + throw result1.reason; + } + + if (result2.status === 'rejected') { + throw result2.reason; + } + + return [result1.value, result2.value]; +} + +function areArraysEqual(a: ReadonlyArray, b: ReadonlyArray): boolean { + return a.length === b.length && a.every((value: string, i: number) => value === b[i]); +} + +function areMapsEqual(a: ReadonlyMap, b: ReadonlyMap): boolean { + if (a.size !== b.size) { + return false; + } + + for (const [key, value] of a) { + if (b.get(key) !== value) { + return false; + } + } + + return true; +} diff --git a/libraries/package-deps-hash/src/getRepoState.ts b/libraries/package-deps-hash/src/getRepoState.ts index e88cf1900c..348a3ecf47 100644 --- a/libraries/package-deps-hash/src/getRepoState.ts +++ b/libraries/package-deps-hash/src/getRepoState.ts @@ -20,7 +20,11 @@ const MINIMUM_GIT_VERSION: IGitVersion = { patch: 0 }; -const STANDARD_GIT_OPTIONS: readonly string[] = [ +/** + * The options for every read-only Git command. + * @internal + */ +export const STANDARD_GIT_OPTIONS: readonly string[] = [ // Don't request any optional file locks '--no-optional-locks', // Ensure that commands don't run automatic maintenance, since performance of the command itself is paramount @@ -79,7 +83,10 @@ const OBJECTMODE_FILE_EXECUTABLE: '100755' = '100755'; // e.g. 10644 blob \t const GIT_LSTREE_FORMAT: string = '%(objectmode) type %(objectname)%x09%(path)'; -interface IGitTreeState { +/** + * @internal + */ +export interface IGitTreeState { files: Map; // type "blob" symlinks: Map; // type "link" submodules: Map; // type "commit" @@ -273,8 +280,12 @@ export function parseGitStatus(output: string): Map { const repoRootCache: Map = new Map(); -// Strip GIT_DIR/GIT_WORK_TREE: git hooks in linked worktrees set GIT_DIR to the per-worktree metadata dir, causing rev-parse --show-toplevel to return CWD instead of the worktree root. -function getCleanGitEnvironment(): NodeJS.ProcessEnv { +/** + * Strip GIT_DIR/GIT_WORK_TREE: git hooks in linked worktrees set GIT_DIR to the per-worktree metadata dir, causing + * rev-parse --show-toplevel to return CWD instead of the worktree root. + * @internal + */ +export function getCleanGitEnvironment(): NodeJS.ProcessEnv { // eslint-disable-next-line @typescript-eslint/no-unused-vars const { GIT_DIR, GIT_WORK_TREE, ...trimmedEnv } = process.env; return trimmedEnv; @@ -323,17 +334,20 @@ export function getRepoRoot(currentWorkingDirectory: string, gitPath?: string): * @param args - The process arguments * @param currentWorkingDirectory - The working directory. Should be the repository root. * @param stdin - An optional Readable stream to use as stdin to the process. + * @param environment - The environment of the process. Defaults to {@link getCleanGitEnvironment}. + * @internal */ -async function spawnGitAsync( +export async function spawnGitAsync( gitPath: string | undefined, args: string[], currentWorkingDirectory: string, - stdin?: Readable + stdin?: Readable, + environment: NodeJS.ProcessEnv = getCleanGitEnvironment() ): Promise { const spawnOptions: IExecutableSpawnOptions = { currentWorkingDirectory, stdio: ['pipe', 'pipe', 'pipe'], - environment: getCleanGitEnvironment() + environment }; let stdout: string = ''; @@ -363,14 +377,29 @@ async function spawnGitAsync( if (status !== 0) { ensureGitMinimumVersion(gitPath); - // Name the git command itself, not the first of the STANDARD_GIT_OPTIONS in front of it - const command: string | undefined = args.find((arg: string) => !STANDARD_GIT_OPTIONS.includes(arg)); - throw new Error(`git ${command} exited with code ${status}:\n${stderr}`); + throw new Error(`git ${getGitCommandName(args)} exited with code ${status}:\n${stderr}`); } return stdout; } +/** + * Returns the Git command in the arguments, rather than the first of the options in front of it. + */ +function getGitCommandName(args: ReadonlyArray): string | undefined { + for (let i: number = 0; i < args.length; i++) { + const arg: string = args[i]; + if (arg === '-c') { + // Skip the value of a configuration option + i++; + } else if (!arg.startsWith('-')) { + return arg; + } + } + + return undefined; +} + function isIterable(value: Iterable | AsyncIterable): value is Iterable { return Symbol.iterator in value; } @@ -426,6 +455,87 @@ export async function hashFilesAsync( return parseGitHashObject(hashObjectResult, hashPaths); } +/** + * The arguments of the `git ls-files` command that lists the files in the index in the format of `git ls-tree`. + * @internal + */ +export function getGitLsFilesArgs(filterPath: ReadonlyArray | undefined): string[] { + return [ + 'ls-files', + // Read from the index only + '--cached', + // Use NUL as the separator + '-z', + // Specify the full path to files relative to the root + '--full-name', + // Match the format of "git ls-tree". The %(objecttype) placeholder requires git 2.51.0+, so not using yet. + `--format=${GIT_LSTREE_FORMAT}`, + '--', + ...(filterPath ?? []) + ]; +} + +/** + * The arguments of the `git status` command that lists the files that differ from the index or from HEAD, and the + * untracked files. Its output is parsed by {@link parseGitStatus}. + * @internal + */ +export function getGitStatusArgs(filterPath: ReadonlyArray | undefined): string[] { + return [ + 'status', + // Use NUL as the separator + '-z', + // Include untracked files + '-u', + // Disable rename detection so that renames show up as add + delete + '--no-renames', + // Don't process submodules with this command; they'll be handled individually + '--ignore-submodules', + // Don't compare against the remote + '--no-ahead-behind', + '--', + ...(filterPath ?? []) + ]; +} + +/** + * The locally modified files that `git status` reports, split into those to hash with `git hash-object` and those + * to remove from the state of the repository. + * @internal + */ +export interface ILocallyModifiedFiles { + filesToHash: string[]; + filesToRemove: string[]; +} + +/** + * Splits the locally modified files that `git status` reports into those to hash and those to remove from the + * state of the repository: the deleted files and the files that the index records as symbolic links. + * @internal + */ +export function classifyLocallyModifiedFiles( + locallyModified: ReadonlyMap, + symlinks: ReadonlyMap +): ILocallyModifiedFiles { + const filesToHash: string[] = []; + const filesToRemove: string[] = []; + const isWindows: boolean = process.platform === 'win32'; + for (const [filePath, exists] of locallyModified) { + if (exists && !symlinks.has(filePath)) { + // Skip Windows reserved device names. `git hash-object` cannot open them and would abort + // the entire repo-state computation. These are almost always stray artifacts (e.g. a `nul` + // file produced by a misdirected shell redirect) rather than meaningful inputs. + if (!isWindows || !isWindowsReservedPath(filePath)) { + filesToHash.push(filePath); + } + } else { + filesToRemove.push(filePath); + } + } + + return { filesToHash, filesToRemove }; +} + /** * Gets the object hashes for all files in the Git repo, combining the current commit with working tree state. * Uses async operations and runs all primary Git calls in parallel. @@ -489,38 +599,12 @@ export async function getDetailedRepoStateAsync( ): Promise { const statePromise: Promise = spawnGitAsync( gitPath, - STANDARD_GIT_OPTIONS.concat([ - 'ls-files', - // Read from the index only - '--cached', - // Use NUL as the separator - '-z', - // Specify the full path to files relative to the root - '--full-name', - // Match the format of "git ls-tree". The %(objecttype) placeholder requires git 2.51.0+, so not using yet. - `--format=${GIT_LSTREE_FORMAT}`, - '--', - ...(filterPath ?? []) - ]), + STANDARD_GIT_OPTIONS.concat(getGitLsFilesArgs(filterPath)), rootDirectory ).then(parseGitLsTree); const locallyModifiedPromise: Promise> = spawnGitAsync( gitPath, - STANDARD_GIT_OPTIONS.concat([ - 'status', - // Use NUL as the separator - '-z', - // Include untracked files - '-u', - // Disable rename detection so that renames show up as add + delete - '--no-renames', - // Don't process submodules with this command; they'll be handled individually - '--ignore-submodules', - // Don't compare against the remote - '--no-ahead-behind', - '--', - ...(filterPath ?? []) - ]), + STANDARD_GIT_OPTIONS.concat(getGitStatusArgs(filterPath)), rootDirectory ).then(parseGitStatus); @@ -533,21 +617,13 @@ export async function getDetailedRepoStateAsync( const [{ files, symlinks }, locallyModified] = await Promise.all([statePromise, locallyModifiedPromise]); - const isWindows: boolean = process.platform === 'win32'; - for (const [filePath, exists] of locallyModified) { - if (exists && !symlinks.has(filePath)) { - // Skip Windows reserved device names. `git hash-object` cannot open them and would abort - // the entire repo-state computation. These are almost always stray artifacts (e.g. a `nul` - // file produced by a misdirected shell redirect) rather than meaningful inputs. - if (isWindows && isWindowsReservedPath(filePath)) { - continue; - } - yield filePath; - } else { - files.delete(filePath); - symlinks.delete(filePath); - } + const { filesToHash, filesToRemove } = classifyLocallyModifiedFiles(locallyModified, symlinks); + for (const filePath of filesToRemove) { + files.delete(filePath); + symlinks.delete(filePath); } + + yield* filesToHash; } const hashObjectPromise: Promise> = hashFilesAsync( diff --git a/libraries/package-deps-hash/src/index.ts b/libraries/package-deps-hash/src/index.ts index 8558d210a4..4d54497791 100644 --- a/libraries/package-deps-hash/src/index.ts +++ b/libraries/package-deps-hash/src/index.ts @@ -24,3 +24,4 @@ export { ensureGitMinimumVersion, hashFilesAsync } from './getRepoState'; +export { type IRepoStateCacheOptions, RepoStateCache } from './RepoStateCache'; diff --git a/libraries/package-deps-hash/src/test/GitIndexFile.test.ts b/libraries/package-deps-hash/src/test/GitIndexFile.test.ts new file mode 100644 index 0000000000..b55b0f34e5 --- /dev/null +++ b/libraries/package-deps-hash/src/test/GitIndexFile.test.ts @@ -0,0 +1,228 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { type IGitIndexSummary, summarizeGitIndex, tryGetGitIndexEntryCount } from '../GitIndexFile'; + +const SHA1_OBJECT_ID_LENGTH: number = 20; +const SHA256_OBJECT_ID_LENGTH: number = 32; +const LONG_FILE_PATH: string = `dir/sub/${'long-name-'.repeat(12)}.txt`; + +function getGitEnvironment(): NodeJS.ProcessEnv { + const environment: NodeJS.ProcessEnv = { ...process.env }; + delete environment.GIT_DIR; + delete environment.GIT_WORK_TREE; + delete environment.GIT_INDEX_FILE; + // Let "git status" save the index that it refreshes + delete environment.GIT_OPTIONAL_LOCKS; + return environment; +} + +describe(summarizeGitIndex.name, () => { + let repoPath: string; + + function runGit(...args: string[]): string { + return execFileSync('git', args, { cwd: repoPath, env: getGitEnvironment(), encoding: 'utf8' }); + } + + function writeFile(relativePath: string, content: string): void { + const filePath: string = path.join(repoPath, relativePath); + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, content); + } + + function readIndex(): Buffer { + return fs.readFileSync(path.join(repoPath, '.git', 'index')); + } + + function summarize(objectIdLength: number = SHA1_OBJECT_ID_LENGTH): IGitIndexSummary { + return summarizeGitIndex(readIndex(), objectIdLength); + } + + function createRepo(...initArgs: string[]): void { + runGit('init', '--quiet', ...initArgs); + runGit('config', 'core.fsmonitor', 'false'); + runGit('config', 'core.untrackedCache', 'true'); + runGit('config', 'index.skipHash', 'false'); + writeFile('a.txt', 'a\n'); + writeFile('dir/b.txt', 'b\n'); + writeFile('dir/sub/c.txt', 'c\n'); + writeFile(LONG_FILE_PATH, 'd\n'); + runGit('add', '.'); + } + + // Git writes a version 3 index only if an entry has extended flags, and writes a version 2 index otherwise + function setIndexVersion(version: number): void { + if (version === 3) { + runGit('update-index', '--skip-worktree', LONG_FILE_PATH); + } + + runGit('update-index', `--index-version=${version}`); + expect(readIndex().readUInt32BE(4)).toBe(version); + } + + beforeEach(() => { + repoPath = fs.mkdtempSync(path.join(os.tmpdir(), 'git-index-file-test-')); + }); + + afterEach(() => { + fs.rmSync(repoPath, { recursive: true, force: true }); + }); + + it.each([2, 3, 4])('summarizes a version %i index', (version: number) => { + createRepo(); + setIndexVersion(version); + + const summary: IGitIndexSummary = summarize(); + expect(summary.entryCount).toBe(4); + expect(summary.entriesDigest).toMatch(/^[0-9a-f]{40}$/); + expect(summary.isSplit).toBe(false); + }); + + it('summarizes the index of a SHA-256 repository', () => { + createRepo('--object-format=sha256'); + runGit('update-index', '--index-version=4'); + + const summary: IGitIndexSummary = summarize(SHA256_OBJECT_ID_LENGTH); + expect(summary.entryCount).toBe(4); + expect(() => summarize(SHA1_OBJECT_ID_LENGTH)).toThrow(); + }); + + it.each([2, 3, 4])('ignores what refreshing a version %i index changes', (version: number) => { + createRepo(); + setIndexVersion(version); + runGit('-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '--quiet', '-m', 'Initial'); + const initialIndex: Buffer = readIndex(); + const initialSummary: IGitIndexSummary = summarize(); + + // Change the recorded times of a file, and the untracked cache + const time: number = Math.floor(Date.now() / 1000) - 100; + fs.utimesSync(path.join(repoPath, 'a.txt'), time, time); + writeFile('untracked.txt', 'untracked\n'); + runGit('status', '--porcelain'); + + expect(readIndex().equals(initialIndex)).toBe(false); + expect(summarize()).toEqual(initialSummary); + }); + + it.each([2, 3, 4])('changes when the entries of a version %i index change', (version: number) => { + createRepo(); + setIndexVersion(version); + const initialDigest: string = summarize().entriesDigest; + const digests: Set = new Set([initialDigest]); + + runGit('update-index', '--assume-unchanged', 'a.txt'); + digests.add(summarize().entriesDigest); + runGit('update-index', '--no-assume-unchanged', 'a.txt'); + expect(summarize().entriesDigest).toBe(initialDigest); + + // Extended flags + runGit('update-index', '--skip-worktree', 'dir/b.txt'); + digests.add(summarize().entriesDigest); + runGit('update-index', '--no-skip-worktree', 'dir/b.txt'); + expect(summarize().entriesDigest).toBe(initialDigest); + + runGit('update-index', '--chmod=+x', 'a.txt'); + digests.add(summarize().entriesDigest); + runGit('update-index', '--chmod=-x', 'a.txt'); + expect(summarize().entriesDigest).toBe(initialDigest); + + writeFile('a.txt', 'changed\n'); + runGit('add', 'a.txt'); + digests.add(summarize().entriesDigest); + + runGit('rm', '--cached', '--quiet', 'dir/sub/c.txt'); + const summary: IGitIndexSummary = summarize(); + expect(summary.entryCount).toBe(3); + digests.add(summary.entriesDigest); + + expect(digests.size).toBe(6); + }); + + it.each([2, 3, 4])('digests the recorded sizes of a version %i index separately', (version: number) => { + createRepo(); + setIndexVersion(version); + runGit('-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '--quiet', '-m', 'Initial'); + const initialSummary: IGitIndexSummary = summarize(); + expect(initialSummary.sizesDigest).toMatch(/^[0-9a-f]{40}$/); + + // Git writes the file with other line endings, and records its new size + runGit('config', 'core.autocrlf', 'true'); + fs.unlinkSync(path.join(repoPath, 'a.txt')); + runGit('checkout', '--', 'a.txt'); + expect(fs.readFileSync(path.join(repoPath, 'a.txt'), 'utf8')).toBe('a\r\n'); + let summary: IGitIndexSummary = summarize(); + expect(summary.entriesDigest).toBe(initialSummary.entriesDigest); + expect(summary.sizesDigest).not.toBe(initialSummary.sizesDigest); + + runGit('config', 'core.autocrlf', 'false'); + fs.unlinkSync(path.join(repoPath, 'a.txt')); + runGit('checkout', '--', 'a.txt'); + summary = summarize(); + expect(summary.entriesDigest).toBe(initialSummary.entriesDigest); + expect(summary.sizesDigest).toBe(initialSummary.sizesDigest); + }); + + it('detects a split index', () => { + createRepo(); + runGit('update-index', '--split-index'); + expect(summarize().isSplit).toBe(true); + + runGit('update-index', '--no-split-index'); + const summary: IGitIndexSummary = summarize(); + expect(summary.isSplit).toBe(false); + expect(summary.entryCount).toBe(4); + }); + + it('rejects data that is not a supported index', () => { + createRepo(); + const index: Buffer = readIndex(); + expect(() => summarizeGitIndex(Buffer.from('not an index'), SHA1_OBJECT_ID_LENGTH)).toThrow( + 'The file is not a Git index' + ); + expect(() => summarizeGitIndex(index.subarray(0, index.length - 30), SHA1_OBJECT_ID_LENGTH)).toThrow(); + + const unsupportedIndex: Buffer = Buffer.from(index); + unsupportedIndex.writeUInt32BE(5, 4); + expect(() => summarizeGitIndex(unsupportedIndex, SHA1_OBJECT_ID_LENGTH)).toThrow( + 'Unsupported Git index version 5' + ); + }); + + it('rejects an entry count that the index is too short to hold, before allocating memory for it', () => { + createRepo(); + const index: Buffer = readIndex(); + index.writeUInt32BE(0xffffffff, 8); + const allocSpy: jest.SpyInstance = jest.spyOn(Buffer, 'alloc').mockImplementation(() => { + throw new Error('Unexpected allocation'); + }); + try { + expect(() => summarizeGitIndex(index, SHA1_OBJECT_ID_LENGTH)).toThrow( + 'The Git index ends within an entry' + ); + expect(allocSpy).not.toHaveBeenCalled(); + } finally { + allocSpy.mockRestore(); + } + }); +}); + +describe(tryGetGitIndexEntryCount.name, () => { + it('reads the entry count from the header of an index', () => { + const header: Buffer = Buffer.alloc(12); + header.write('DIRC', 0, 'latin1'); + header.writeUInt32BE(2, 4); + header.writeUInt32BE(236074, 8); + expect(tryGetGitIndexEntryCount(header)).toBe(236074); + }); + + it('returns undefined for data that is not the header of an index', () => { + expect(tryGetGitIndexEntryCount(Buffer.alloc(0))).toBeUndefined(); + expect(tryGetGitIndexEntryCount(Buffer.from('DIRC'))).toBeUndefined(); + expect(tryGetGitIndexEntryCount(Buffer.from('PACK\0\0\0\x02\0\0\0\x01'))).toBeUndefined(); + }); +}); diff --git a/libraries/package-deps-hash/src/test/RepoStateCache.test.ts b/libraries/package-deps-hash/src/test/RepoStateCache.test.ts new file mode 100644 index 0000000000..162c7ac7a9 --- /dev/null +++ b/libraries/package-deps-hash/src/test/RepoStateCache.test.ts @@ -0,0 +1,912 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { type ChildProcess, execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { Executable, type IExecutableSpawnOptions } from '@rushstack/node-core-library'; + +import * as GitIndexFile from '../GitIndexFile'; +import { getDetailedRepoStateAsync, type IDetailedRepoState } from '../getRepoState'; +import { RepoStateCache } from '../RepoStateCache'; + +const originalDateNow: () => number = Date.now; +const originalSpawn: typeof Executable.spawn = Executable.spawn; + +function getGitEnvironment(): NodeJS.ProcessEnv { + const environment: NodeJS.ProcessEnv = { ...process.env }; + delete environment.GIT_DIR; + delete environment.GIT_WORK_TREE; + delete environment.GIT_INDEX_FILE; + // Let "git status" save the index that it refreshes + delete environment.GIT_OPTIONAL_LOCKS; + return environment; +} + +interface IComparableState { + hasSubmodules: boolean; + hasUncommittedChanges: boolean; + files: [string, string][]; + symlinks: [string, string][]; +} + +function toComparable(state: IDetailedRepoState): IComparableState { + return { + hasSubmodules: state.hasSubmodules, + hasUncommittedChanges: state.hasUncommittedChanges, + files: Array.from(state.files), + symlinks: Array.from(state.symlinks) + }; +} + +interface IGitCommand { + command: string | undefined; + usesPrivateIndex: boolean; +} + +function getGitCommand( + args: ReadonlyArray, + options: IExecutableSpawnOptions | undefined +): IGitCommand { + let command: string | undefined; + for (let i: number = 0; i < args.length && !command; i++) { + if (args[i] === '-c') { + // Skip the value of a configuration option + i++; + } else if (!args[i].startsWith('-')) { + command = args[i]; + } + } + + return { command, usesPrivateIndex: !!options?.environment?.GIT_INDEX_FILE }; +} + +describe(RepoStateCache.name, () => { + let repoPath: string; + let temporaryFolderPath: string; + let cache: RepoStateCache; + let gitCommands: IGitCommand[]; + let isRecordingGitCommands: boolean; + let beforeSpawn: ((gitCommand: IGitCommand) => void) | undefined; + let afterSpawn: ((gitCommand: IGitCommand, childProcess: ChildProcess) => void) | undefined; + + function runGit(...args: string[]): string { + return execFileSync('git', args, { cwd: repoPath, env: getGitEnvironment(), encoding: 'utf8' }); + } + + function writeFile(relativePath: string, content: string): void { + const filePath: string = path.join(repoPath, relativePath); + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, content); + } + + function commit(): void { + runGit('add', '--all'); + runGit('commit', '--quiet', '--allow-empty', '-m', 'Commit'); + } + + function getPrivateIndexPath(): string { + const [folderName] = fs.readdirSync(temporaryFolderPath); + return path.join(temporaryFolderPath, folderName, 'index'); + } + + // Treat every file as settled, so that the cache memoizes the hashes of files that just changed + function settleFiles(): void { + jest.spyOn(Date, 'now').mockImplementation(() => originalDateNow() + 10000); + } + + // Treat every file as changed recently, however slowly the test runs + function unsettleFiles(): void { + jest.spyOn(Date, 'now').mockImplementation(() => originalDateNow() - 60000); + } + + // Changes the recorded times of a file, but not its content, so that "git status" saves the refreshed index + function touchSettledFile(relativePath: string): void { + const time: number = Math.floor(originalDateNow() / 1000) - 100; + fs.utimesSync(path.join(repoPath, relativePath), time, time); + } + + function takeGitCommands(): IGitCommand[] { + const commands: IGitCommand[] = gitCommands; + gitCommands = []; + return commands; + } + + // The commands that the cache ran, sorted, since some run concurrently + function takeGitCommandNames(): (string | undefined)[] { + return takeGitCommands() + .map(({ command }: IGitCommand) => command) + .sort(); + } + + function takeUsesPrivateIndex(): boolean { + return takeGitCommands().some(({ usesPrivateIndex }: IGitCommand) => usesPrivateIndex); + } + + async function getStateAsync( + additionalRelativePathsToHash?: string[], + filterPath?: string[] + ): Promise { + return await cache.getDetailedRepoStateAsync(additionalRelativePathsToHash, filterPath); + } + + async function expectUncachedStateAsync( + state: IDetailedRepoState, + additionalRelativePathsToHash?: string[], + filterPath?: string[] + ): Promise { + isRecordingGitCommands = false; + try { + const expected: IDetailedRepoState = await getDetailedRepoStateAsync( + repoPath, + additionalRelativePathsToHash, + undefined, + filterPath + ); + expect(toComparable(state)).toEqual(toComparable(expected)); + } finally { + isRecordingGitCommands = true; + } + } + + beforeEach(() => { + repoPath = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'repo-state-cache-test-'))); + temporaryFolderPath = fs.mkdtempSync(path.join(os.tmpdir(), 'repo-state-cache-private-')); + runGit('init', '--quiet'); + runGit('config', 'user.name', 'Test'); + runGit('config', 'user.email', 'test@example.com'); + runGit('config', 'commit.gpgSign', 'false'); + runGit('config', 'core.fsmonitor', 'false'); + runGit('config', 'core.untrackedCache', 'true'); + runGit('config', 'maintenance.auto', 'false'); + writeFile('a.txt', 'a\n'); + writeFile('b.txt', 'b\n'); + writeFile('dir/c.txt', 'c\n'); + commit(); + + cache = new RepoStateCache({ rootDirectory: repoPath, temporaryFolderPath }); + gitCommands = []; + isRecordingGitCommands = true; + beforeSpawn = undefined; + afterSpawn = undefined; + jest + .spyOn(Executable, 'spawn') + .mockImplementation((filename: string, args: string[], options?: IExecutableSpawnOptions) => { + const gitCommand: IGitCommand = getGitCommand(args, options); + beforeSpawn?.(gitCommand); + if (isRecordingGitCommands) { + gitCommands.push(gitCommand); + } + + const childProcess: ChildProcess = originalSpawn.call(Executable, filename, args, options); + afterSpawn?.(gitCommand, childProcess); + return childProcess; + }); + }); + + afterEach(() => { + cache.dispose(); + jest.restoreAllMocks(); + fs.rmSync(repoPath, { recursive: true, force: true }); + fs.rmSync(temporaryFolderPath, { recursive: true, force: true }); + }); + + it('returns the same state as getDetailedRepoStateAsync, and the same object while nothing changes', async () => { + writeFile('a.txt', 'modified\n'); + writeFile('untracked.txt', 'untracked\n'); + settleFiles(); + + const state: IDetailedRepoState = await getStateAsync(['untracked.txt']); + expect( + takeGitCommands() + .map(({ command, usesPrivateIndex }: IGitCommand) => `${command} ${usesPrivateIndex}`) + .sort() + ).toEqual(['hash-object false', 'hash-object false', 'ls-files true', 'rev-parse false', 'status true']); + await expectUncachedStateAsync(state, ['untracked.txt']); + expect(state.hasUncommittedChanges).toBe(true); + + await expect(getStateAsync(['untracked.txt'])).resolves.toBe(state); + expect(takeGitCommandNames()).toEqual(['status']); + }); + + it('follows changes to the working tree', async () => { + let previousState: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(previousState); + expect(previousState.hasUncommittedChanges).toBe(false); + + const changes: (() => void)[] = [ + () => writeFile('a.txt', 'modified\n'), + () => writeFile('a.txt', 'modified again\n'), + () => writeFile('new.txt', 'new\n'), + () => fs.unlinkSync(path.join(repoPath, 'b.txt')), + () => runGit('checkout', '--', 'b.txt'), + () => fs.unlinkSync(path.join(repoPath, 'new.txt')), + () => runGit('checkout', '--', 'a.txt') + ]; + for (const change of changes) { + change(); + const state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state).not.toBe(previousState); + previousState = state; + } + + expect(previousState.hasUncommittedChanges).toBe(false); + }); + + it('hashes a file that changed recently each time', async () => { + unsettleFiles(); + writeFile('a.txt', 'modified\n'); + await getStateAsync(); + takeGitCommands(); + + // The file may change again without changing its stamp + await getStateAsync(); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + }); + + it('hashes a settled file only while its stamp changes', async () => { + settleFiles(); + writeFile('a.txt', 'modified\n'); + writeFile('untracked.txt', 'untracked\n'); + await getStateAsync(['untracked.txt']); + takeGitCommands(); + + await getStateAsync(['untracked.txt']); + expect(takeGitCommandNames()).toEqual(['status']); + + writeFile('untracked.txt', 'untracked and modified\n'); + const state: IDetailedRepoState = await getStateAsync(['untracked.txt']); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + await expectUncachedStateAsync(state, ['untracked.txt']); + }); + + it('hashes a file that changed recently again even when its stamp stays the same', async () => { + settleFiles(); + writeFile('untracked.txt', 'one\n'); + const filePath: string = path.join(repoPath, 'untracked.txt'); + // Only this file changed recently. Its stamp stays the same when it changes again, as it can when the file + // changes twice within the granularity of the file times. + const recentTimeNs: bigint = BigInt(Date.now()) * BigInt(1e6); + const recentStats: fs.BigIntStats = Object.create(fs.lstatSync(filePath, { bigint: true }), { + mtimeNs: { value: recentTimeNs }, + ctimeNs: { value: recentTimeNs } + }); + const lstatAsync: typeof fs.promises.lstat = fs.promises.lstat; + jest + .spyOn(fs.promises, 'lstat') + .mockImplementation(async (lstatPath: fs.PathLike, options?: fs.StatOptions) => + lstatPath === filePath ? recentStats : await lstatAsync(lstatPath, options) + ); + await getStateAsync(['untracked.txt']); + + writeFile('untracked.txt', 'two\n'); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(['untracked.txt']); + expect(takeGitCommandNames()).toEqual(['hash-object', 'hash-object', 'status']); + expect(state.files.get('untracked.txt')).toBe(hashText('two\n')); + }); + + it('copies the index again when the files that it records change', async () => { + settleFiles(); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + await getStateAsync(); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + + writeFile('a.txt', 'modified\n'); + runGit('add', 'a.txt'); + let state: IDetailedRepoState = await getStateAsync(); + expect(takeGitCommandNames()).toContain('ls-files'); + await expectUncachedStateAsync(state); + expect(writeFileSpy).toHaveBeenCalledTimes(2); + + // Committing doesn't change the files that the index records + runGit('commit', '--quiet', '-m', 'Modify'); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.hasUncommittedChanges).toBe(false); + expect(writeFileSpy).toHaveBeenCalledTimes(2); + + runGit('rm', '--cached', '--quiet', 'b.txt'); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(writeFileSpy).toHaveBeenCalledTimes(3); + }); + + it('lists the files in the copy of the index while "git status" refreshes it', async () => { + await getStateAsync(); + writeFile('a.txt', 'modified\n'); + runGit('add', 'a.txt'); + const events: string[] = []; + afterSpawn = ({ command }: IGitCommand, childProcess: ChildProcess) => { + if (command === 'ls-files' || command === 'status') { + events.push(`start ${command}`); + childProcess.once('exit', () => events.push(`exit ${command}`)); + } + }; + + const state: IDetailedRepoState = await getStateAsync(); + afterSpawn = undefined; + await expectUncachedStateAsync(state); + // Both commands start before either one exits + expect(events.slice(0, 2).sort()).toEqual(['start ls-files', 'start status']); + expect(events.slice(2).sort()).toEqual(['exit ls-files', 'exit status']); + }); + + it('keeps the copy of the index when Git only refreshes the index', async () => { + settleFiles(); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + await getStateAsync(); + + const indexPath: string = path.join(repoPath, '.git', 'index'); + const indexInode: number = fs.statSync(indexPath).ino; + const time: number = Math.floor(originalDateNow() / 1000) - 100; + fs.utimesSync(path.join(repoPath, 'a.txt'), time, time); + runGit('status', '--porcelain'); + // Git replaced the index + expect(fs.statSync(indexPath).ino).not.toBe(indexInode); + + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(); + expect(takeGitCommandNames()).toEqual(['status']); + await expectUncachedStateAsync(state); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + }); + + it('detects a change that the recorded times and size of a file cannot reveal', async () => { + // Git then compares only the whole seconds of the modification time, and the size + runGit('config', 'core.trustctime', 'false'); + runGit('config', 'core.checkStat', 'minimal'); + const time: number = Math.floor(originalDateNow() / 1000) - 100; + const filePath: string = path.join(repoPath, 'a.txt'); + fs.utimesSync(filePath, time, time); + runGit('update-index', '--refresh'); + // The index was saved in the same second as the file changed, so Git must not trust the recorded times + fs.utimesSync(path.join(repoPath, '.git', 'index'), time, time); + fs.writeFileSync(filePath, 'A\n'); + fs.utimesSync(filePath, time, time); + + const state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(runGit('hash-object', 'a.txt').trim()); + }); + + it('detects a change that the recorded times and size of a file cannot reveal, even where Git trusts them', async () => { + runGit('config', 'core.trustctime', 'false'); + runGit('config', 'core.checkStat', 'minimal'); + const time: number = Math.floor(originalDateNow() / 1000) - 100; + const filePath: string = path.join(repoPath, 'a.txt'); + fs.utimesSync(filePath, time, time); + runGit('update-index', '--refresh'); + // The index was saved a second after the file changed, so Git trusts the recorded times and size, and + // getDetailedRepoStateAsync misses the change. The copy of the index is older. + fs.utimesSync(path.join(repoPath, '.git', 'index'), time + 1, time + 1); + fs.writeFileSync(filePath, 'A\n'); + fs.utimesSync(filePath, time, time); + + const state: IDetailedRepoState = await getStateAsync(); + expect(state.files.get('a.txt')).toBe(runGit('hash-object', 'a.txt').trim()); + }); + + it('copies the index again when the recorded size of a file changes', async () => { + settleFiles(); + runGit('config', 'core.autocrlf', 'true'); + const additionalPaths: string[] = ['b.txt']; + await getStateAsync(additionalPaths); + writeFile('a.txt', 'modified\n'); + await expectUncachedStateAsync(await getStateAsync(additionalPaths), additionalPaths); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + + // Git writes the file with other line endings, and records its new size in the index but not in the copy + runGit('checkout', '--', 'a.txt'); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + expect(state.hasUncommittedChanges).toBe(false); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + // The files that the index records didn't change, so the list of files and the hashes are reused + expect(takeGitCommandNames()).toEqual(['status']); + }); + + it('copies the index again when the configuration of the repository changes', async () => { + settleFiles(); + runGit('config', 'core.autocrlf', 'true'); + // Git writes the file with CRLF line endings + fs.unlinkSync(path.join(repoPath, 'a.txt')); + runGit('checkout', '--', 'a.txt'); + await getStateAsync(); + // Git refreshes the recorded times of the file in the copy of the index, but not in the index + touchSettledFile('a.txt'); + let state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.hasUncommittedChanges).toBe(false); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + + // Git no longer converts the line endings, so the file no longer matches the index + runGit('config', 'core.autocrlf', 'false'); + takeGitCommands(); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(hashText('a\r\n')); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + }); + + it('copies the index again when a .gitattributes file changes, after computing the state without the cache', async () => { + settleFiles(); + writeFile('crlf.txt', 'a\r\n'); + commit(); + await getStateAsync(); + // Git refreshes the recorded times of the file in the copy of the index, but not in the index + touchSettledFile('crlf.txt'); + await getStateAsync(); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + + // Git now converts the line endings of the file, so it no longer matches the index + writeFile('.gitattributes', '*.txt text eol=lf\n'); + let state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('crlf.txt')).toBe(hashText('a\n')); + expect(writeFileSpy).not.toHaveBeenCalled(); + + takeGitCommands(); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + }); + + it('copies the index again when a .gitattributes file in a folder that contains the filter changes', async () => { + settleFiles(); + writeFile('sub/.gitattributes', '*.txt -text\n'); + writeFile('sub/dir/crlf.txt', 'a\r\n'); + writeFile('sub/dir/modified.txt', 'm\n'); + commit(); + const filterPath: string[] = ['sub/dir']; + await getStateAsync([], filterPath); + // Git refreshes the recorded times of the file in the copy of the index, but not in the index + touchSettledFile('sub/dir/crlf.txt'); + writeFile('sub/dir/modified.txt', 'modified\r\n'); + let state: IDetailedRepoState = await getStateAsync([], filterPath); + await expectUncachedStateAsync(state, [], filterPath); + + // "git status" doesn't list the file, which is outside the filter + writeFile('sub/.gitattributes', '*.txt text eol=lf\n'); + state = await getStateAsync([], filterPath); + await expectUncachedStateAsync(state, [], filterPath); + expect(state.files.get('sub/dir/crlf.txt')).toBe(hashText('a\n')); + expect(state.files.get('sub/dir/modified.txt')).toBe(hashText('modified\n')); + + takeGitCommands(); + state = await getStateAsync([], filterPath); + await expectUncachedStateAsync(state, [], filterPath); + expect(takeUsesPrivateIndex()).toBe(true); + }); + + it('copies the index again when the filter changes, instead of computing the state without the cache', async () => { + settleFiles(); + writeFile('dir/crlf.txt', 'a\r\n'); + commit(); + await getStateAsync(); + touchSettledFile('dir/crlf.txt'); + await getStateAsync(); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + + // An untracked file outside the filter, which "git status" doesn't list + writeFile('.gitattributes', '*.txt text eol=lf\n'); + const state: IDetailedRepoState = await getStateAsync([], ['dir']); + await expectUncachedStateAsync(state, [], ['dir']); + expect(state.files.get('dir/crlf.txt')).toBe(hashText('a\n')); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + }); + + it('hashes a file again when an ignored .gitattributes file in a folder that contains it changes', async () => { + settleFiles(); + writeFile('.gitignore', 'ignored/\n'); + commit(); + writeFile('ignored/dir/crlf.txt', 'a\r\n'); + const additionalPaths: string[] = ['ignored/dir/crlf.txt']; + let state: IDetailedRepoState = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + expect(state.files.get('ignored/dir/crlf.txt')).toBe(hashText('a\r\n')); + + // "git status" doesn't list the ignored file + writeFile('ignored/.gitattributes', '*.txt text eol=lf\n'); + takeGitCommands(); + state = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + expect(state.files.get('ignored/dir/crlf.txt')).toBe(hashText('a\n')); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + + await getStateAsync(additionalPaths); + expect(takeGitCommandNames()).toEqual(['status']); + }); + + it('hashes a file each time while a .gitattributes file in a folder that contains it may still change', async () => { + settleFiles(); + writeFile('.gitignore', 'ignored/\n'); + writeFile('.gitattributes', '*.txt -text\n'); + commit(); + writeFile('ignored/crlf.txt', 'a\r\n'); + // The attributes file changes after the calls start, but "git status" doesn't list it + const time: number = Math.floor(originalDateNow() / 1000) + 60; + fs.utimesSync(path.join(repoPath, '.gitattributes'), time, time); + const additionalPaths: string[] = ['ignored/crlf.txt']; + await getStateAsync(additionalPaths); + takeGitCommands(); + + await getStateAsync(additionalPaths); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + }); + + it('hashes files again when a modified .gitattributes file changes', async () => { + settleFiles(); + writeFile('.gitattributes', '# No attributes\n'); + // "git status" doesn't report an ignored file, so only the additional files include it + writeFile('.gitignore', 'ignored.txt\n'); + commit(); + writeFile('a.txt', 'modified\r\n'); + writeFile('untracked.txt', 'untracked\r\n'); + writeFile('ignored.txt', 'ignored\r\n'); + const additionalPaths: string[] = ['untracked.txt', 'ignored.txt']; + await expectUncachedStateAsync(await getStateAsync(additionalPaths), additionalPaths); + + writeFile('.gitattributes', '*.txt text eol=lf\n'); + let state: IDetailedRepoState = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + expect(state.files.get('a.txt')).toBe(hashText('modified\n')); + expect(state.files.get('ignored.txt')).toBe(hashText('ignored\n')); + + runGit('checkout', '--', '.gitattributes'); + state = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + expect(state.files.get('a.txt')).toBe(hashText('modified\r\n')); + expect(state.files.get('ignored.txt')).toBe(hashText('ignored\r\n')); + }); + + it('hashes files again when the configuration of the repository changes', async () => { + settleFiles(); + writeFile('a.txt', 'modified\r\n'); + let state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(hashText('modified\r\n')); + + runGit('config', 'core.autocrlf', 'true'); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(hashText('modified\n')); + }); + + it('reads the index again while its stamp may still change', async () => { + settleFiles(); + const summarizeSpy: jest.SpyInstance = jest.spyOn(GitIndexFile, 'summarizeGitIndex'); + // The index changes after the calls start, but the configuration doesn't + const indexPath: string = path.join(repoPath, '.git', 'index'); + const futureTime: number = Math.floor(originalDateNow() / 1000) + 60; + fs.utimesSync(indexPath, futureTime, futureTime); + await getStateAsync(); + await getStateAsync(); + expect(summarizeSpy).toHaveBeenCalledTimes(2); + + const pastTime: number = Math.floor(originalDateNow() / 1000) - 100; + fs.utimesSync(indexPath, pastTime, pastTime); + await getStateAsync(); + expect(summarizeSpy).toHaveBeenCalledTimes(3); + await getStateAsync(); + expect(summarizeSpy).toHaveBeenCalledTimes(3); + }); + + it('reuses the hashes of files when the files that the index records change', async () => { + settleFiles(); + const additionalPaths: string[] = ['b.txt']; + writeFile('a.txt', 'modified\n'); + await getStateAsync(additionalPaths); + + runGit('add', 'a.txt'); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + // "git status" still lists the staged file + expect(takeGitCommandNames()).toEqual(['ls-files', 'status']); + }); + + it('reuses the hash of a file when only the index has the .gitattributes file that applies to it', async () => { + settleFiles(); + writeFile('.gitignore', 'ignored/\n'); + writeFile('.gitattributes', '*.txt -text\n'); + commit(); + writeFile('ignored/crlf.txt', 'a\r\n'); + fs.unlinkSync(path.join(repoPath, '.gitattributes')); + const additionalPaths: string[] = ['ignored/crlf.txt']; + await getStateAsync(additionalPaths); + + // "git status" uses the version in the index in place of the missing file, but "git hash-object" doesn't + writeFile('attributes.tmp', '*.txt text eol=lf\n'); + const objectId: string = runGit('hash-object', '-w', 'attributes.tmp').trim(); + fs.unlinkSync(path.join(repoPath, 'attributes.tmp')); + runGit('update-index', '--cacheinfo', `100644,${objectId},.gitattributes`); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(additionalPaths); + await expectUncachedStateAsync(state, additionalPaths); + expect(state.files.get('ignored/crlf.txt')).toBe(hashText('a\r\n')); + expect(takeGitCommandNames()).toEqual(['ls-files', 'status']); + }); + + it('copies the index again when its copy is deleted or emptied', async () => { + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + settleFiles(); + writeFile('a.txt', 'modified\n'); + await getStateAsync(); + await getStateAsync(); + + fs.unlinkSync(getPrivateIndexPath()); + await expectUncachedStateAsync(await getStateAsync()); + expect(writeFileSpy).toHaveBeenCalledTimes(2); + + runGit('rm', '--cached', '--quiet', '-r', '.'); + fs.copyFileSync(path.join(repoPath, '.git', 'index'), getPrivateIndexPath()); + runGit('reset', '--quiet'); + await expectUncachedStateAsync(await getStateAsync()); + expect(writeFileSpy).toHaveBeenCalledTimes(3); + }); + + it('copies the index to a new folder when its folder is deleted', async () => { + await getStateAsync(); + fs.rmSync(path.dirname(getPrivateIndexPath()), { recursive: true }); + + takeGitCommands(); + await expectUncachedStateAsync(await getStateAsync()); + expect(takeUsesPrivateIndex()).toBe(false); + await expectUncachedStateAsync(await getStateAsync()); + expect(takeUsesPrivateIndex()).toBe(true); + expect(fs.readdirSync(temporaryFolderPath)).toHaveLength(1); + }); + + it('creates the temporary folder', async () => { + fs.rmSync(temporaryFolderPath, { recursive: true }); + await getStateAsync(); + expect(takeUsesPrivateIndex()).toBe(true); + expect(fs.readdirSync(temporaryFolderPath)).toHaveLength(1); + }); + + it('falls back to getDetailedRepoStateAsync when the copy of the index disappears while Git reads it', async () => { + await getStateAsync(); + let isPrivateIndexDeleted: boolean = false; + beforeSpawn = ({ command, usesPrivateIndex }: IGitCommand) => { + if (command === 'status' && usesPrivateIndex && !isPrivateIndexDeleted) { + isPrivateIndexDeleted = true; + fs.rmSync(getPrivateIndexPath()); + } + }; + + await expectUncachedStateAsync(await getStateAsync()); + expect(isPrivateIndexDeleted).toBe(true); + takeGitCommands(); + await expectUncachedStateAsync(await getStateAsync()); + expect(takeUsesPrivateIndex()).toBe(true); + }); + + it('removes a lock file that an interrupted "git status" left behind', async () => { + await getStateAsync(); + const privateIndexContent: Buffer = fs.readFileSync(getPrivateIndexPath()); + const lockPath: string = `${getPrivateIndexPath()}.lock`; + fs.writeFileSync(lockPath, ''); + touchSettledFile('a.txt'); + + await expectUncachedStateAsync(await getStateAsync()); + expect(fs.existsSync(lockPath)).toBe(false); + expect(fs.readFileSync(getPrivateIndexPath())).not.toEqual(privateIndexContent); + }); + + it('lets "git status" save the copy of the index when the environment disables optional locks', async () => { + const optionalLocks: string | undefined = process.env.GIT_OPTIONAL_LOCKS; + process.env.GIT_OPTIONAL_LOCKS = '0'; + try { + await getStateAsync(); + const privateIndexContent: Buffer = fs.readFileSync(getPrivateIndexPath()); + touchSettledFile('a.txt'); + + await expectUncachedStateAsync(await getStateAsync()); + expect(fs.readFileSync(getPrivateIndexPath())).not.toEqual(privateIndexContent); + } finally { + if (optionalLocks === undefined) { + delete process.env.GIT_OPTIONAL_LOCKS; + } else { + process.env.GIT_OPTIONAL_LOCKS = optionalLocks; + } + } + }); + + it('limits the state to the filter', async () => { + settleFiles(); + writeFile('dir/d.txt', 'd\n'); + writeFile('e.txt', 'e\n'); + const state: IDetailedRepoState = await getStateAsync([], ['dir/']); + takeGitCommands(); + await expectUncachedStateAsync(state, [], ['dir/']); + expect(Array.from(state.files.keys())).toEqual(['dir/c.txt', 'dir/d.txt']); + + await expect(getStateAsync([], ['dir/'])).resolves.toBe(state); + expect(takeGitCommandNames()).toEqual(['status']); + + // The files outside the filter + await expectUncachedStateAsync(await getStateAsync()); + expect(takeGitCommandNames()).toEqual(['hash-object', 'ls-files', 'status']); + + await expectUncachedStateAsync(await getStateAsync([], ['dir/']), [], ['dir/']); + expect(takeGitCommandNames()).toEqual(['ls-files', 'status']); + }); + + it('computes the state of a repository without an index', async () => { + fs.unlinkSync(path.join(repoPath, '.git', 'index')); + let state: IDetailedRepoState = await getStateAsync(); + expect(takeUsesPrivateIndex()).toBe(false); + await expectUncachedStateAsync(state); + + runGit('reset', '--quiet'); + state = await getStateAsync(); + expect(takeUsesPrivateIndex()).toBe(true); + await expectUncachedStateAsync(state); + }); + + it('stops using the cache for a split index', async () => { + await getStateAsync(); + runGit('update-index', '--split-index'); + writeFile('a.txt', 'modified\n'); + await expectUncachedStateAsync(await getStateAsync()); + expect(fs.readdirSync(temporaryFolderPath)).toEqual([]); + + runGit('update-index', '--no-split-index'); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(); + expect(takeUsesPrivateIndex()).toBe(false); + await expectUncachedStateAsync(state); + }); + + it('stops using the cache for a repository with submodules', async () => { + const submodulePath: string = fs.mkdtempSync(path.join(os.tmpdir(), 'repo-state-cache-submodule-')); + try { + execFileSync('git', ['init', '--quiet'], { cwd: submodulePath, env: getGitEnvironment() }); + fs.writeFileSync(path.join(submodulePath, 'inner.txt'), 'inner\n'); + execFileSync('git', ['add', '--all'], { cwd: submodulePath, env: getGitEnvironment() }); + execFileSync( + 'git', + ['-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '--quiet', '-m', 'Inner'], + { cwd: submodulePath, env: getGitEnvironment() } + ); + runGit('-c', 'protocol.file.allow=always', 'submodule', '--quiet', 'add', submodulePath, 'sub'); + commit(); + + const state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.hasSubmodules).toBe(true); + expect(state.files.has('sub/inner.txt')).toBe(true); + expect(fs.readdirSync(temporaryFolderPath)).toEqual([]); + } finally { + fs.rmSync(submodulePath, { recursive: true, force: true }); + } + }); + + it('falls back to getDetailedRepoStateAsync when it fails, and stops after repeated failures', async () => { + // Read the index on every call + unsettleFiles(); + const summarizeSpy: jest.SpyInstance = jest + .spyOn(GitIndexFile, 'summarizeGitIndex') + .mockImplementation(() => { + throw new Error('Test failure'); + }); + writeFile('a.txt', 'modified\n'); + await expectUncachedStateAsync(await getStateAsync()); + await expectUncachedStateAsync(await getStateAsync()); + + // A success resets the count + summarizeSpy.mockRestore(); + await getStateAsync(); + jest.spyOn(GitIndexFile, 'summarizeGitIndex').mockImplementation(() => { + throw new Error('Test failure'); + }); + await expectUncachedStateAsync(await getStateAsync()); + await expectUncachedStateAsync(await getStateAsync()); + await expectUncachedStateAsync(await getStateAsync()); + expect(GitIndexFile.summarizeGitIndex).toHaveBeenCalledTimes(3); + + await expectUncachedStateAsync(await getStateAsync()); + expect(GitIndexFile.summarizeGitIndex).toHaveBeenCalledTimes(3); + expect(fs.readdirSync(temporaryFolderPath)).toEqual([]); + }); + + it('reports the error of getDetailedRepoStateAsync, without counting it as a failure of the cache', async () => { + const expectedError: Error = await getDetailedRepoStateAsync(repoPath, ['missing.txt']).then( + () => new Error('Expected an error'), + (error: Error) => error + ); + expect(expectedError.message).toMatch(/^git hash-object exited with code/); + + for (let i: number = 0; i < 4; i++) { + await expect(getStateAsync(['missing.txt'])).rejects.toThrow(expectedError.message); + } + + takeGitCommands(); + await getStateAsync(); + expect(takeUsesPrivateIndex()).toBe(true); + }); + + it('computes the state without the cache after it is disposed', async () => { + const exitListenerCount: number = process.listenerCount('exit'); + await getStateAsync(); + expect(fs.readdirSync(temporaryFolderPath)).toHaveLength(1); + expect(process.listenerCount('exit')).toBe(exitListenerCount + 1); + + cache.dispose(); + expect(fs.readdirSync(temporaryFolderPath)).toEqual([]); + expect(process.listenerCount('exit')).toBe(exitListenerCount); + + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(); + expect(takeUsesPrivateIndex()).toBe(false); + await expectUncachedStateAsync(state); + }); + + it('shares one exit listener between caches', async () => { + const exitListeners: NodeJS.ExitListener[] = process.listeners('exit'); + const otherCache: RepoStateCache = new RepoStateCache({ rootDirectory: repoPath, temporaryFolderPath }); + try { + await getStateAsync(); + await otherCache.getDetailedRepoStateAsync(); + expect(fs.readdirSync(temporaryFolderPath)).toHaveLength(2); + const addedExitListeners: NodeJS.ExitListener[] = process + .listeners('exit') + .filter((listener: NodeJS.ExitListener) => !exitListeners.includes(listener)); + expect(addedExitListeners).toHaveLength(1); + + addedExitListeners[0](0); + expect(fs.readdirSync(temporaryFolderPath)).toEqual([]); + + cache.dispose(); + expect(process.listenerCount('exit')).toBe(exitListeners.length + 1); + } finally { + otherCache.dispose(); + } + + expect(process.listenerCount('exit')).toBe(exitListeners.length); + }); + + it('runs one call at a time', async () => { + writeFile('a.txt', 'modified\n'); + const states: IDetailedRepoState[] = await Promise.all([ + getStateAsync(), + getStateAsync(), + getStateAsync() + ]); + await expectUncachedStateAsync(states[0]); + expect(states[1]).toBe(states[0]); + expect(states[2]).toBe(states[0]); + }); + + if (process.platform !== 'win32') { + it('removes a symbolic link that was replaced by a file', async () => { + fs.symlinkSync('a.txt', path.join(repoPath, 'link')); + commit(); + let state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.symlinks.has('link')).toBe(true); + + fs.unlinkSync(path.join(repoPath, 'link')); + writeFile('link', 'not a link\n'); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.symlinks.has('link')).toBe(false); + }); + } + + function hashText(text: string): string { + return execFileSync('git', ['hash-object', '--stdin', '--no-filters'], { + cwd: repoPath, + env: getGitEnvironment(), + input: text, + encoding: 'utf8' + }).trim(); + } +}); diff --git a/libraries/package-deps-hash/src/test/getRepoState.test.ts b/libraries/package-deps-hash/src/test/getRepoState.test.ts index 573657feb3..227898b5c0 100644 --- a/libraries/package-deps-hash/src/test/getRepoState.test.ts +++ b/libraries/package-deps-hash/src/test/getRepoState.test.ts @@ -1,7 +1,12 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { isWindowsReservedPath, parseGitStatus, parseGitVersion } from '../getRepoState'; +import { + classifyLocallyModifiedFiles, + isWindowsReservedPath, + parseGitStatus, + parseGitVersion +} from '../getRepoState'; describe(parseGitVersion.name, () => { it('Can parse valid git version responses', () => { @@ -116,3 +121,41 @@ describe(isWindowsReservedPath.name, () => { expect(isWindowsReservedPath('packages/nul-suffix/index.ts')).toBe(false); }); }); + +describe(classifyLocallyModifiedFiles.name, () => { + const platformDescriptor: PropertyDescriptor = Object.getOwnPropertyDescriptor(process, 'platform')!; + + // "nul" is a reserved name on Windows + const locallyModified: ReadonlyMap = new Map([ + ['modified.txt', true], + ['deleted.txt', false], + ['link', true], + ['deleted-link', false], + ['apps/nul', true], + ['apps/con.txt', false] + ]); + const symlinks: ReadonlyMap = new Map([ + ['link', 'e69de29bb2d1d6434b8b29ae775ad8c2e48c5391'], + ['deleted-link', 'e69de29bb2d1d6434b8b29ae775ad8c2e48c5391'] + ]); + + afterEach(() => { + Object.defineProperty(process, 'platform', platformDescriptor); + }); + + it('hashes the files that exist, and removes the deleted files and the symbolic links', () => { + Object.defineProperty(process, 'platform', { ...platformDescriptor, value: 'linux' }); + expect(classifyLocallyModifiedFiles(locallyModified, symlinks)).toEqual({ + filesToHash: ['modified.txt', 'apps/nul'], + filesToRemove: ['deleted.txt', 'link', 'deleted-link', 'apps/con.txt'] + }); + }); + + it('neither hashes nor removes a file with a reserved name on Windows', () => { + Object.defineProperty(process, 'platform', { ...platformDescriptor, value: 'win32' }); + expect(classifyLocallyModifiedFiles(locallyModified, symlinks)).toEqual({ + filesToHash: ['modified.txt'], + filesToRemove: ['deleted.txt', 'link', 'deleted-link', 'apps/con.txt'] + }); + }); +}); diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 27f5e135e5..f58300645d 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -1,8 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import * as path from 'node:path'; - import { AlreadyReportedError } from '@rushstack/node-core-library'; import type { LockFile } from '@rushstack/node-core-library'; import { @@ -31,7 +29,6 @@ import { import { WorkspaceEngineComponentFactory, WorkspaceEngineRecreationRequiredError, - type IMapWorkspaceInvalidationsOptions, type IWorkspaceEngineShape, type IWorkspaceInvalidationReconciliation } from './WorkspaceEngineComponentFactory'; @@ -40,6 +37,7 @@ import { EngineTerminalProvider } from './EngineTerminalProvider'; import { OperationOutputFingerprints } from './OperationOutputFingerprints'; import { getDaemonShutdownReason } from './DaemonShutdownError'; import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; +import { createInputsCompatibilityCheck, getOperationsWithChangedInputs } from './WorkspaceInputsComparison'; import { createDaemonRequestTelemetrySink, type IDaemonEngineCreationTiming } from './DaemonRequestTelemetry'; /** @@ -293,6 +291,9 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { const outputFingerprints: OperationOutputFingerprints = new OperationOutputFingerprints( engine.operationGraph ); + const checkInputsCompatibility: (snapshot: IInputsSnapshot) => void = createInputsCompatibilityCheck( + engine.inputsSnapshot + ); const factory: WorkspaceEngineComponentFactory = new WorkspaceEngineComponentFactory({ createEngineComponentsAsync: async () => ({ ...engine, @@ -306,8 +307,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { } throw error; } - if (snapshot && !this.#validateGraphInputsAsync) - assertCompatibleInputs(engine.inputsSnapshot, snapshot); + if (snapshot && !this.#validateGraphInputsAsync) checkInputsCompatibility(snapshot); return snapshot; } }), @@ -315,7 +315,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { refreshInputsOnEveryRequest: true, validateGraphInputsAsync: this.#validateGraphInputsAsync, mapInvalidationsToOperationsAsync: async (invalidationOptions) => [ - ...getChangedOperations(invalidationOptions), + ...getOperationsWithChangedInputs(invalidationOptions), // Outputs are git-ignored and absent from state hashes, so check them separately. ...outputFingerprints.getOperationsWithChangedOutputs() ] @@ -386,31 +386,3 @@ function getFingerprintValue(name: string, value: string | undefined): string | ? undefined : getWorkspaceFingerprintEnvironmentEntries({ [name]: value })[0]?.[1]; } - -function getChangedOperations(options: IMapWorkspaceInvalidationsOptions): Iterable { - const { currentInputsSnapshot: current, nextInputsSnapshot: next, operationGraph } = options; - return Array.from(operationGraph.operations).filter( - (operation) => - current.getOperationOwnStateHash(operation.associatedProject, operation.associatedPhase.name) !== - next.getOperationOwnStateHash(operation.associatedProject, operation.associatedPhase.name) - ); -} - -function assertCompatibleInputs(current: IInputsSnapshot, next: IInputsSnapshot): void { - const paths: Set = new Set([...current.hashes.keys(), ...next.hashes.keys()]); - for (const filePath of paths) { - if (isGraphDefinitionPath(filePath) && current.hashes.get(filePath) !== next.hashes.get(filePath)) { - throw new WorkspaceEngineRecreationRequiredError(); - } - } -} - -function isGraphDefinitionPath(filePath: string): boolean { - const normalized: string = filePath.replace(/\\/g, '/'); - return ( - /(^|\/)config\//.test(normalized) || - ['rush.json', 'package.json', '.gitignore', '.npmrc', '.env', 'pnpm-lock.yaml'].includes( - path.basename(filePath) - ) - ); -} diff --git a/libraries/rush-daemon/src/WorkspaceInputsComparison.ts b/libraries/rush-daemon/src/WorkspaceInputsComparison.ts new file mode 100644 index 0000000000..c2545d155c --- /dev/null +++ b/libraries/rush-daemon/src/WorkspaceInputsComparison.ts @@ -0,0 +1,72 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as path from 'node:path'; + +import type { IInputsSnapshot, Operation } from '@microsoft/rush-lib'; + +import { + WorkspaceEngineRecreationRequiredError, + type IMapWorkspaceInvalidationsOptions +} from './WorkspaceEngineComponentFactory'; + +/** + * Returns the operations whose own state hashes differ between the current and the next inputs snapshots. + */ +export function getOperationsWithChangedInputs(options: IMapWorkspaceInvalidationsOptions): Operation[] { + const { currentInputsSnapshot: current, nextInputsSnapshot: next, operationGraph } = options; + if (current === next) { + // A snapshot computes the same hashes each time + return []; + } + + return Array.from(operationGraph.operations).filter( + (operation) => + current.getOperationOwnStateHash(operation.associatedProject, operation.associatedPhase.name) !== + next.getOperationOwnStateHash(operation.associatedProject, operation.associatedPhase.name) + ); +} + +/** + * Returns a function that checks that an inputs snapshot is compatible with the snapshot that an engine was + * created from, as {@link assertCompatibleInputs} does. It checks a snapshot only if it differs from the last + * compatible one. + */ +export function createInputsCompatibilityCheck( + initialSnapshot: IInputsSnapshot +): (snapshot: IInputsSnapshot) => void { + let compatibleSnapshot: IInputsSnapshot = initialSnapshot; + return (snapshot: IInputsSnapshot): void => { + if (snapshot !== compatibleSnapshot) { + assertCompatibleInputs(initialSnapshot, snapshot); + compatibleSnapshot = snapshot; + } + }; +} + +/** + * Throws {@link WorkspaceEngineRecreationRequiredError} if a file that may define the operation graph has + * different hashes in the two snapshots. + */ +export function assertCompatibleInputs(current: IInputsSnapshot, next: IInputsSnapshot): void { + if (current.hashes === next.hashes) { + return; + } + + const paths: Set = new Set([...current.hashes.keys(), ...next.hashes.keys()]); + for (const filePath of paths) { + if (isGraphDefinitionPath(filePath) && current.hashes.get(filePath) !== next.hashes.get(filePath)) { + throw new WorkspaceEngineRecreationRequiredError(); + } + } +} + +function isGraphDefinitionPath(filePath: string): boolean { + const normalized: string = filePath.replace(/\\/g, '/'); + return ( + /(^|\/)config\//.test(normalized) || + ['rush.json', 'package.json', '.gitignore', '.npmrc', '.env', 'pnpm-lock.yaml'].includes( + path.basename(filePath) + ) + ); +} diff --git a/libraries/rush-daemon/src/test/WorkspaceInputsComparison.test.ts b/libraries/rush-daemon/src/test/WorkspaceInputsComparison.test.ts new file mode 100644 index 0000000000..6718f3c8cf --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceInputsComparison.test.ts @@ -0,0 +1,156 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IInputsSnapshot, IOperationGraph, Operation } from '@microsoft/rush-lib'; + +import { WorkspaceEngineRecreationRequiredError } from '../WorkspaceEngineComponentFactory'; +import { + assertCompatibleInputs, + createInputsCompatibilityCheck, + getOperationsWithChangedInputs +} from '../WorkspaceInputsComparison'; + +const PHASE_NAME: string = '_phase:build'; + +function createOperation(packageName: string): Operation { + return { + associatedPhase: { name: PHASE_NAME }, + associatedProject: { packageName } + } as unknown as Operation; +} + +function createInputsSnapshot( + hashes: ReadonlyMap, + stateHashByPackageName: Readonly> = {} +): IInputsSnapshot { + return { + getOperationOwnStateHash: jest.fn((project: unknown, phaseName?: string) => { + const { packageName } = project as { packageName: string }; + return `${phaseName}:${stateHashByPackageName[packageName]}`; + }), + getTrackedFileHashesForOperation: () => hashes, + hasUncommittedChanges: false, + hashes, + rootDirectory: '/repo' + }; +} + +// Hashes that report whether they were read +function createObservedHashes(entries: [string, string][]): Map { + const hashes: Map = new Map(entries); + jest.spyOn(hashes, 'keys'); + jest.spyOn(hashes, 'get'); + return hashes; +} + +function wereRead(hashes: Map): boolean { + return (hashes.keys as jest.Mock).mock.calls.length > 0 || (hashes.get as jest.Mock).mock.calls.length > 0; +} + +const INITIAL_HASHES: [string, string][] = [ + ['common/config/rush/experiments.json', 'experiments'], + ['a/package.json', 'a-package'], + ['a/src/index.ts', 'a-index'], + ['b/src/index.ts', 'b-index'] +]; + +describe(getOperationsWithChangedInputs.name, () => { + const operationA: Operation = createOperation('a'); + const operationB: Operation = createOperation('b'); + const operationGraph: IOperationGraph = { + operations: new Set([operationA, operationB]) + } as unknown as IOperationGraph; + + it('returns the operations whose own state hashes differ', () => { + const hashes: Map = new Map(INITIAL_HASHES); + expect( + getOperationsWithChangedInputs({ + changedPaths: [], + currentInputsSnapshot: createInputsSnapshot(hashes, { a: '1', b: '2' }), + nextInputsSnapshot: createInputsSnapshot(hashes, { a: '1', b: '3' }), + operationGraph + }) + ).toEqual([operationB]); + }); + + it('computes no hashes to compare a snapshot with itself', () => { + const snapshot: IInputsSnapshot = createInputsSnapshot(new Map(INITIAL_HASHES), { a: '1', b: '2' }); + expect( + getOperationsWithChangedInputs({ + changedPaths: ['a/src/index.ts'], + currentInputsSnapshot: snapshot, + nextInputsSnapshot: snapshot, + operationGraph + }) + ).toEqual([]); + expect(snapshot.getOperationOwnStateHash).not.toHaveBeenCalled(); + }); +}); + +describe(assertCompatibleInputs.name, () => { + it('accepts snapshots that differ only in files that do not define the graph', () => { + const current: IInputsSnapshot = createInputsSnapshot(new Map(INITIAL_HASHES)); + const next: IInputsSnapshot = createInputsSnapshot( + new Map([...INITIAL_HASHES, ['a/src/index.ts', 'a-index-2'], ['b/src/new.ts', 'b-new']]) + ); + expect(() => assertCompatibleInputs(current, next)).not.toThrow(); + }); + + it.each([ + ['a changed', ['a/package.json', 'a-package-2']], + ['an added', ['c/package.json', 'c-package']] + ])('requires a new engine for %s file that may define the graph', (name: string, entry: string[]) => { + const current: IInputsSnapshot = createInputsSnapshot(new Map(INITIAL_HASHES)); + const next: IInputsSnapshot = createInputsSnapshot( + new Map([...INITIAL_HASHES, entry as [string, string]]) + ); + expect(() => assertCompatibleInputs(current, next)).toThrow(WorkspaceEngineRecreationRequiredError); + expect(() => assertCompatibleInputs(next, current)).toThrow(WorkspaceEngineRecreationRequiredError); + }); + + it('reads no hashes of snapshots that share them', () => { + const hashes: Map = createObservedHashes(INITIAL_HASHES); + assertCompatibleInputs(createInputsSnapshot(hashes), createInputsSnapshot(hashes)); + expect(wereRead(hashes)).toBe(false); + }); +}); + +describe(createInputsCompatibilityCheck.name, () => { + it('compares each snapshot with the initial snapshot, unless it was the last compatible one', () => { + const initialHashes: Map = createObservedHashes(INITIAL_HASHES); + const initialSnapshot: IInputsSnapshot = createInputsSnapshot(initialHashes); + const checkInputsCompatibility: (snapshot: IInputsSnapshot) => void = + createInputsCompatibilityCheck(initialSnapshot); + + checkInputsCompatibility(initialSnapshot); + expect(wereRead(initialHashes)).toBe(false); + + const compatibleHashes: Map = createObservedHashes([ + ...INITIAL_HASHES, + ['a/src/index.ts', 'a-index-2'] + ]); + const compatibleSnapshot: IInputsSnapshot = createInputsSnapshot(compatibleHashes); + checkInputsCompatibility(compatibleSnapshot); + expect(wereRead(compatibleHashes)).toBe(true); + + jest.clearAllMocks(); + checkInputsCompatibility(compatibleSnapshot); + expect(wereRead(compatibleHashes)).toBe(false); + + const incompatibleSnapshot: IInputsSnapshot = createInputsSnapshot( + new Map([...INITIAL_HASHES, ['a/package.json', 'a-package-2']]) + ); + expect(() => checkInputsCompatibility(incompatibleSnapshot)).toThrow( + WorkspaceEngineRecreationRequiredError + ); + expect(() => checkInputsCompatibility(incompatibleSnapshot)).toThrow( + WorkspaceEngineRecreationRequiredError + ); + + // Only the last compatible snapshot is skipped + checkInputsCompatibility(initialSnapshot); + jest.clearAllMocks(); + checkInputsCompatibility(compatibleSnapshot); + expect(wereRead(compatibleHashes)).toBe(true); + }); +}); diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index 3d084b3973..605c5ef016 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -814,8 +814,12 @@ export class PhasedScriptAction extends BaseScriptAction i terminal, // We need to include all dependencies, otherwise build cache id calculation will be incorrect relevantProjects, - // An engine cannot continue without a snapshot, so it reports why none could be taken. - { throwOnMissingProjectShrinkwrapFile: !!onEngine } + { + // An engine cannot continue without a snapshot, so it reports why none could be taken. + throwOnMissingProjectShrinkwrapFile: !!onEngine, + // An engine takes a snapshot for each request + reuseUnchangedInputs: !!onEngine + } ); const innerInitialSnapshot: IInputsSnapshot | undefined = innerGetInputsSnapshotAsync ? await innerGetInputsSnapshotAsync() diff --git a/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts b/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts index ed5353620a..1c00206dc4 100644 --- a/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts +++ b/libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts @@ -21,7 +21,9 @@ import { getRepoRoot, getDetailedRepoStateAsync, hashFilesAsync, - type IFileDiffStatus + type IDetailedRepoState, + type IFileDiffStatus, + RepoStateCache } from '@rushstack/package-deps-hash'; import type { ITerminal } from '@rushstack/terminal'; @@ -334,13 +336,20 @@ export class ProjectChangeAnalyzer { * `throwOnMissingProjectShrinkwrapFile`, a missing project dependency file (shrinkwrap-deps.json) instead * throws its error, whether it is detected when the provider is created or when a snapshot fails. A host that * cannot continue without a snapshot can then report that actionable cause. + * + * With `reuseUnchangedInputs`, the provider keeps the state of the Git repository between snapshots, and + * returns its previous snapshot while the inputs are unchanged. This suits a long-lived host, such as the Rush + * daemon, that takes a snapshot for each request. * @internal */ public async _tryGetSnapshotProviderAsync( projectConfigurations: ReadonlyMap, terminal: ITerminal, projectSelection?: ReadonlySet, - options?: { readonly throwOnMissingProjectShrinkwrapFile?: boolean } + options?: { + readonly throwOnMissingProjectShrinkwrapFile?: boolean; + readonly reuseUnchangedInputs?: boolean; + } ): Promise { const throwOnMissingProjectShrinkwrapFile: boolean = !!options?.throwOnMissingProjectShrinkwrapFile; try { @@ -444,12 +453,19 @@ export class ProjectChangeAnalyzer { filterPath = Array.from(projectSelection, ({ projectFolder }) => projectFolder); } + const repoStateCache: RepoStateCache | undefined = options?.reuseUnchangedInputs + ? getRepoStateCache(rootDirectory, gitPath, rushConfiguration.commonTempFolder) + : undefined; + let previousSnapshot: IReusableInputsSnapshot | undefined; + return async function tryGetSnapshotAsync(): Promise { // Recorded before Git reads the working tree: a file saved after this may be newer than its hash. const workingTreeReadStartTimeMs: number = Date.now(); try { - const [{ files: hashes, symlinks, hasUncommittedChanges }, additionalFiles] = await Promise.all([ - getDetailedRepoStateAsync(rootDirectory, additionalRelativePathsToHash, gitPath, filterPath), + const [repoState, additionalFiles] = await Promise.all([ + repoStateCache + ? repoStateCache.getDetailedRepoStateAsync(additionalRelativePathsToHash, filterPath) + : getDetailedRepoStateAsync(rootDirectory, additionalRelativePathsToHash, gitPath, filterPath), getAdditionalFilesFromRushProjectConfigurationAsync( additionalGlobs, lookupByPath, @@ -457,6 +473,7 @@ export class ProjectChangeAnalyzer { terminal ) ]); + const { files: hashes, symlinks, hasUncommittedChanges } = repoState; if (symlinks.size > 0) { terminal.writeWarningLine( @@ -472,11 +489,27 @@ export class ProjectChangeAnalyzer { } const additionalHashes: Map = new Map( - await hashFilesAsync(rootDirectory, additionalFiles, gitPath) + additionalFiles.size > 0 ? await hashFilesAsync(rootDirectory, additionalFiles, gitPath) : [] ); + const environment: Record = { ...process.env }; + // The files that each operation depends on accumulate across snapshots + const operationAdditionalFileCount: number = countOperationAdditionalFiles(projectMap); + + if ( + previousSnapshot && + previousSnapshot.repoState === repoState && + previousSnapshot.operationAdditionalFileCount === operationAdditionalFileCount && + areMapsEqual(previousSnapshot.additionalHashes, additionalHashes) && + areEnvironmentsEqual(previousSnapshot.environment, environment) + ) { + // Every hash that a new snapshot would compute is the same. The previous snapshot began reading the + // working tree earlier, which only widens the window in which a saved file may be newer than its hash. + return previousSnapshot.snapshot; + } - return new InputsSnapshot({ + const snapshot: IInputsSnapshot = new InputsSnapshot({ additionalHashes, + environment, globalAdditionalFiles, hashes, hasUncommittedChanges, @@ -485,6 +518,17 @@ export class ProjectChangeAnalyzer { rootDir: rootDirectory, workingTreeReadStartTimeMs }); + if (repoStateCache) { + previousSnapshot = { + additionalHashes, + environment, + operationAdditionalFileCount, + repoState, + snapshot + }; + } + + return snapshot; } catch (e) { // The files were checked once, when this provider was created. A file removed since then fails // "git hash-object" with an obscure message, so check them again, only after a failure. @@ -761,6 +805,73 @@ interface IAdditionalGlob { pattern: string; } +/** + * A snapshot, and the inputs that it was computed from. + */ +interface IReusableInputsSnapshot { + readonly additionalHashes: ReadonlyMap; + readonly environment: Readonly>; + readonly operationAdditionalFileCount: number; + readonly repoState: IDetailedRepoState; + readonly snapshot: IInputsSnapshot; +} + +/** + * The caches of the state of each Git repository, by the path of Git and the root of the repository. They outlive + * the snapshot providers, so that a long-lived host, such as the Rush daemon, keeps using a cache when it creates + * a provider again. + */ +const repoStateCacheByRepository: Map = new Map(); + +function getRepoStateCache( + rootDirectory: string, + gitPath: string, + temporaryFolderPath: string +): RepoStateCache { + const key: string = `${gitPath}\0${rootDirectory}`; + let repoStateCache: RepoStateCache | undefined = repoStateCacheByRepository.get(key); + if (!repoStateCache) { + repoStateCache = new RepoStateCache({ rootDirectory, gitPath, temporaryFolderPath }); + repoStateCacheByRepository.set(key, repoStateCache); + } + + return repoStateCache; +} + +function countOperationAdditionalFiles( + projectMap: ReadonlyMap +): number { + let count: number = 0; + for (const { additionalFilesByOperationName } of projectMap.values()) { + for (const additionalFiles of additionalFilesByOperationName?.values() ?? []) { + count += additionalFiles.size; + } + } + + return count; +} + +function areMapsEqual(a: ReadonlyMap, b: ReadonlyMap): boolean { + if (a.size !== b.size) { + return false; + } + + for (const [key, value] of a) { + if (b.get(key) !== value || !b.has(key)) { + return false; + } + } + + return true; +} + +function areEnvironmentsEqual( + a: Readonly>, + b: Readonly> +): boolean { + return areMapsEqual(new Map(Object.entries(a)), new Map(Object.entries(b))); +} + async function getAdditionalFilesFromRushProjectConfigurationAsync( additionalGlobs: IAdditionalGlob[], rootRelativeLookupByPath: IReadonlyLookupByPath, diff --git a/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts b/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts index 5d85cffa1d..a6e9e6c5a9 100644 --- a/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts +++ b/libraries/rush-lib/src/logic/test/ProjectChangeAnalyzer.test.ts @@ -26,6 +26,15 @@ const mockHashes: Map = new Map([ const mockGetRepoChanges: jest.MockedFunction = jest.fn(); const mockOnGetDetailedRepoState: jest.Mock = jest.fn(); +const mockHashFilesAsync: jest.Mock, [string, Iterable]> = jest.fn( + (rootDirectory: string, filePaths: Iterable) => + new Map(Array.from(filePaths, (filePath: string) => [filePath, filePath])) +); +const mockCreateRepoStateCache: jest.Mock = jest.fn(); +const mockGetCachedRepoStateAsync: jest.Mock< + Promise, + [ReadonlyArray | undefined, ReadonlyArray | undefined] +> = jest.fn(); jest.mock(`@rushstack/package-deps-hash`, () => { return { @@ -48,7 +57,19 @@ jest.mock(`@rushstack/package-deps-hash`, () => { return new Map(Array.from(filePaths, (filePath: string) => [filePath, filePath])); }, hashFilesAsync(rootDirectory: string, filePaths: Iterable): ReadonlyMap { - return new Map(Array.from(filePaths, (filePath: string) => [filePath, filePath])); + return mockHashFilesAsync(rootDirectory, filePaths); + }, + RepoStateCache: class MockRepoStateCache { + public constructor(options: IRepoStateCacheOptions) { + mockCreateRepoStateCache(options); + } + + public async getDetailedRepoStateAsync( + additionalRelativePathsToHash?: ReadonlyArray, + filterPath?: ReadonlyArray + ): Promise { + return await mockGetCachedRepoStateAsync(additionalRelativePathsToHash, filterPath); + } }, getRepoChanges( currentWorkingDirectory: string, @@ -117,7 +138,11 @@ jest.mock('../incremental/InputsSnapshot', () => { import * as fs from 'node:fs'; import { resolve } from 'node:path'; -import type { IDetailedRepoState, IFileDiffStatus } from '@rushstack/package-deps-hash'; +import type { + IDetailedRepoState, + IFileDiffStatus, + IRepoStateCacheOptions +} from '@rushstack/package-deps-hash'; import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import { @@ -127,6 +152,8 @@ import { tryGetMissingProjectShrinkwrapFileErrorAsync } from '../ProjectChangeAnalyzer'; import { RushConfiguration } from '../../api/RushConfiguration'; +import type { RushConfigurationProject } from '../../api/RushConfigurationProject'; +import type { RushProjectConfiguration } from '../../api/RushProjectConfiguration'; import type { IInputsSnapshot, GetInputsSnapshotAsyncFn, @@ -268,6 +295,178 @@ describe(ProjectChangeAnalyzer.name, () => { expect(terminalProvider.getWarningOutput()).toContain('git hash-object failed'); }); + describe('when reusing unchanged inputs', () => { + const reuseOptions: { reuseUnchangedInputs: boolean } = { reuseUnchangedInputs: true }; + let stateCount: number; + + function createRepoState(): IDetailedRepoState { + stateCount++; + return { + hasSubmodules: false, + hasUncommittedChanges: false, + files: new Map([['a/package.json', `hash${stateCount}`]]), + symlinks: new Map() + }; + } + + function getProjectConfigurations( + dependsOnAdditionalFiles: string[] + ): Map { + const projectConfiguration: RushProjectConfiguration = { + operationSettingsByOperationName: new Map([ + ['_phase:build', { operationName: '_phase:build', dependsOnAdditionalFiles }] + ]) + } as unknown as RushProjectConfiguration; + return new Map([[rushConfiguration.getProjectByName('a')!, projectConfiguration]]); + } + + beforeEach(() => { + stateCount = 0; + mockSnapshot.mockImplementation(() => ({})); + mockGetCachedRepoStateAsync.mockResolvedValue(createRepoState()); + }); + + afterEach(() => { + mockGetCachedRepoStateAsync.mockReset(); + delete process.env.PROJECT_CHANGE_ANALYZER_TEST; + }); + + it('returns the previous snapshot while its inputs are unchanged', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const provider: GetInputsSnapshotAsyncFn | undefined = await analyzer._tryGetSnapshotProviderAsync( + new Map(), + terminal, + undefined, + reuseOptions + ); + const snapshot: IInputsSnapshot | undefined = await provider!(); + expect(snapshot).toBeDefined(); + await expect(provider!()).resolves.toBe(snapshot); + expect(mockSnapshot).toHaveBeenCalledTimes(1); + expect(mockOnGetDetailedRepoState).not.toHaveBeenCalled(); + expect(mockGetCachedRepoStateAsync).toHaveBeenCalledTimes(2); + const [additionalRelativePathsToHash, filterPath] = mockGetCachedRepoStateAsync.mock.calls[1]; + expect(Array.from(additionalRelativePathsToHash!).sort()).toEqual([ + 'a/.rush/temp/shrinkwrap-deps.json', + 'b/.rush/temp/shrinkwrap-deps.json' + ]); + expect(filterPath).toEqual([]); + // There are no additional files to hash + expect(mockHashFilesAsync).not.toHaveBeenCalled(); + + // The state of the repository changed + mockGetCachedRepoStateAsync.mockResolvedValue(createRepoState()); + const nextSnapshot: IInputsSnapshot | undefined = await provider!(); + expect(nextSnapshot).not.toBe(snapshot); + await expect(provider!()).resolves.toBe(nextSnapshot); + expect(mockSnapshot).toHaveBeenCalledTimes(2); + expect(mockSnapshot.mock.calls[1][0].hashes).toBe( + (await mockGetCachedRepoStateAsync.mock.results[2].value).files + ); + }); + + it('takes a new snapshot when the environment changes', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const provider: GetInputsSnapshotAsyncFn | undefined = await analyzer._tryGetSnapshotProviderAsync( + new Map(), + terminal, + undefined, + reuseOptions + ); + const snapshot: IInputsSnapshot | undefined = await provider!(); + expect(mockSnapshot.mock.calls[0][0].environment).toEqual(process.env); + + process.env.PROJECT_CHANGE_ANALYZER_TEST = '1'; + const nextSnapshot: IInputsSnapshot | undefined = await provider!(); + expect(nextSnapshot).not.toBe(snapshot); + expect(mockSnapshot.mock.calls[1][0].environment).toEqual(process.env); + await expect(provider!()).resolves.toBe(nextSnapshot); + + process.env.PROJECT_CHANGE_ANALYZER_TEST = '2'; + await expect(provider!()).resolves.not.toBe(nextSnapshot); + expect(mockSnapshot).toHaveBeenCalledTimes(3); + }); + + it('takes a new snapshot when the additional files or their hashes change', async () => { + fs.writeFileSync(resolve(folder, 'a/extra.txt'), 'extra'); + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const provider: GetInputsSnapshotAsyncFn | undefined = await analyzer._tryGetSnapshotProviderAsync( + getProjectConfigurations(['*.txt']), + terminal, + undefined, + reuseOptions + ); + const snapshot: IInputsSnapshot | undefined = await provider!(); + expect(mockSnapshot.mock.calls[0][0].additionalHashes).toEqual( + new Map([['a/extra.txt', 'a/extra.txt']]) + ); + await expect(provider!()).resolves.toBe(snapshot); + + mockHashFilesAsync.mockImplementationOnce(() => new Map([['a/extra.txt', 'changed']])); + const changedHashSnapshot: IInputsSnapshot | undefined = await provider!(); + expect(changedHashSnapshot).not.toBe(snapshot); + + fs.writeFileSync(resolve(folder, 'a/other.txt'), 'other'); + const addedFileSnapshot: IInputsSnapshot | undefined = await provider!(); + expect(addedFileSnapshot).not.toBe(changedHashSnapshot); + await expect(provider!()).resolves.toBe(addedFileSnapshot); + expect(mockSnapshot).toHaveBeenCalledTimes(3); + }); + + it('takes a new snapshot when an operation depends on another tracked file', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const provider: GetInputsSnapshotAsyncFn | undefined = await analyzer._tryGetSnapshotProviderAsync( + getProjectConfigurations(['*.md']), + terminal, + undefined, + reuseOptions + ); + const snapshot: IInputsSnapshot | undefined = await provider!(); + + // A tracked file is hashed with the rest of the repository, not as an additional file + fs.writeFileSync(resolve(folder, 'a/README.md'), 'readme'); + mockGetCachedRepoStateAsync.mockImplementation(async () => { + const state: IDetailedRepoState = await mockGetCachedRepoStateAsync.mock.results[0].value; + state.files.set('a/README.md', 'readme'); + return state; + }); + await expect(provider!()).resolves.not.toBe(snapshot); + expect(mockHashFilesAsync).not.toHaveBeenCalled(); + expect(mockSnapshot).toHaveBeenCalledTimes(2); + }); + + it('shares the cache of the repository between providers', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + await analyzer._tryGetSnapshotProviderAsync(new Map(), terminal, undefined, reuseOptions); + await new ProjectChangeAnalyzer(rushConfiguration)._tryGetSnapshotProviderAsync( + new Map(), + terminal, + undefined, + reuseOptions + ); + expect(mockCreateRepoStateCache).toHaveBeenCalledTimes(1); + expect(mockCreateRepoStateCache).toHaveBeenCalledWith({ + rootDirectory: folder, + gitPath: expect.any(String), + temporaryFolderPath: rushConfiguration.commonTempFolder + }); + }); + + it('takes a new snapshot each time without the option', async () => { + const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(rushConfiguration); + const provider: GetInputsSnapshotAsyncFn | undefined = await analyzer._tryGetSnapshotProviderAsync( + new Map(), + terminal + ); + const snapshot: IInputsSnapshot | undefined = await provider!(); + await expect(provider!()).resolves.not.toBe(snapshot); + expect(mockSnapshot).toHaveBeenCalledTimes(2); + expect(mockOnGetDetailedRepoState).toHaveBeenCalledTimes(2); + expect(mockGetCachedRepoStateAsync).not.toHaveBeenCalled(); + expect(mockCreateRepoStateCache).not.toHaveBeenCalled(); + }); + }); + it('finds the first missing file in project order', async () => { await expect( tryGetMissingProjectShrinkwrapFileErrorAsync(rushConfiguration) From 071dc6587233b93ac5a6069702d1c5c56e66ff2e Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:15 +0000 Subject: [PATCH 062/265] [rush-daemon] An invalid build command line fails with native Rush's error and exit code instead of running in-process Swarm integration step 47; original commit f84ce3a3cb (merge of swarm/r04-t78-int at cf5921e40f). Scope: task 78. Brings r04's task 78: when a warm daemon's parser rejects a build command line, the client prints the daemon's parse error once and exits 2, where it used to fall back in-process and print about 29 lines. Second agent: o04 board 2801. ch01's e2e on ch01-sC: a warm `build --nope` with agent output returns rc 2, 2 lines, in 0.16 s, answered by the daemon. s16 batch C, item 2 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree d80d2a2ccb) Commits folded into this step (1): - cf5921e40f Fail an invalid build command line with native Rush's error and exit code instead of running it in-process (task 78) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...-r04-t78-usage-error_2026-09-29-00-14.json | 11 ++++ ...-r04-t78-usage-error_2026-09-29-00-14.json | 11 ++++ common/reviews/api/rush-lib.api.md | 7 ++- .../rush-daemon/src/DaemonControlSession.ts | 12 ++++ .../src/DaemonRequestUsageError.ts | 17 ++++++ .../src/ProductionDaemonRequestResolver.ts | 6 ++ .../src/WorkspaceRequestLifecycle.ts | 45 ++++++++++---- .../src/test/DaemonRequestUsageError.test.ts | 61 +++++++++++++++++++ .../rush-lib/src/api/PhasedCommandEngine.ts | 17 +++++- .../src/api/PhasedCommandEngineUsageError.ts | 19 ++++++ .../src/api/test/PhasedCommandEngine.test.ts | 20 ++++++ libraries/rush-lib/src/index.ts | 1 + 12 files changed, 214 insertions(+), 13 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r04-t78-usage-error_2026-09-29-00-14.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t78-usage-error_2026-09-29-00-14.json create mode 100644 libraries/rush-daemon/src/DaemonRequestUsageError.ts create mode 100644 libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts create mode 100644 libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts diff --git a/common/changes/@microsoft/rush/swarm-r04-t78-usage-error_2026-09-29-00-14.json b/common/changes/@microsoft/rush/swarm-r04-t78-usage-error_2026-09-29-00-14.json new file mode 100644 index 0000000000..a10aa44110 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r04-t78-usage-error_2026-09-29-00-14.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add `PhasedCommandEngineUsageError` (alpha). `PhasedCommandEngine.parseAsync` throws it with native Rush's message and exit code for a command line that native Rush rejects as invalid, such as an unknown parameter, so that a host can report the error instead of running the command in-process.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t78-usage-error_2026-09-29-00-14.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t78-usage-error_2026-09-29-00-14.json new file mode 100644 index 0000000000..af618148ec --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t78-usage-error_2026-09-29-00-14.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A build or rebuild request whose command line native Rush rejects as invalid (for example `rush-client build --nope`) now fails with native Rush's error message and exit code 2, instead of a rejection that made the client run the same command line in-process only to print the usage and the same error. If the workspace configuration changed since the daemon loaded it, the client still runs the command in-process, which parses it with the new configuration.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index f96c5f2059..10cbe380ac 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -1561,7 +1561,6 @@ export class PhasedCommandEngine { createTelemetryData(options: IPhasedCommandEngineTelemetryOptions): ITelemetryData; // (undocumented) readonly parameterIdentity: string; - // (undocumented) static parseAsync(options: IParsePhasedCommandOptions): Promise; get requestSettings(): IPhasedCommandEngineRequestSettings; selectOperationsAsync(graph: IOperationGraph): Promise>; @@ -1584,6 +1583,12 @@ export class PhasedCommandEngineProjectConfigurationError extends Error { readonly projectName: string; } +// @alpha +export class PhasedCommandEngineUsageError extends Error { + constructor(message: string, exitCode: number, options?: ErrorOptions); + readonly exitCode: number; +} + // @alpha export class PhasedCommandHooks { readonly createOperationsAsync: AsyncSeriesWaterfallHook<[ diff --git a/libraries/rush-daemon/src/DaemonControlSession.ts b/libraries/rush-daemon/src/DaemonControlSession.ts index a6995237c1..296a3b3410 100644 --- a/libraries/rush-daemon/src/DaemonControlSession.ts +++ b/libraries/rush-daemon/src/DaemonControlSession.ts @@ -33,6 +33,7 @@ import { DaemonInteractiveConnection } from './DaemonInteractiveConnection'; import type { IDaemonInteractiveConnection } from './DaemonInteractiveConnection'; import { MAX_REQUESTS_PER_CONNECTION } from './DaemonConnectionLimits'; import { DaemonRequestDispatchError } from './DaemonRequestDispatcher'; +import { DaemonRequestUsageError } from './DaemonRequestUsageError'; import type { DaemonRequestDispatcher } from './DaemonRequestDispatcher'; import { DaemonShutdownError } from './DaemonShutdownError'; import { DaemonWireRequestClient } from './DaemonWireRequestClient'; @@ -389,6 +390,17 @@ export class DaemonControlSession { dispatchError = combineErrors(dispatchError, cleanupError); } if (dispatchError !== undefined && !state.client.terminalOutcomeSent && !this.#connectionClosed) { + if (dispatchError instanceof DaemonRequestUsageError) { + // Native Rush reports an invalid command line and exits, so the client must not run it in-process. + await state.client.writeResultAsync({ + requestId: envelope.requestId, + exitCode: dispatchError.exitCode, + outcome: 'failure', + aborted: state.abortController.signal.aborted, + errorMessage: dispatchError.message + }); + return; + } const rejection: IClassifiedRejection = classifyRejection(dispatchError); // The client prints only the message; keep the rest where `rush-client daemon logs` finds it. this.#options.onLog?.( diff --git a/libraries/rush-daemon/src/DaemonRequestUsageError.ts b/libraries/rush-daemon/src/DaemonRequestUsageError.ts new file mode 100644 index 0000000000..8b5eecbcaf --- /dev/null +++ b/libraries/rush-daemon/src/DaemonRequestUsageError.ts @@ -0,0 +1,17 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * A command line that native Rush rejects as invalid. The request fails with native Rush's exit code, and the + * client does not run the command in-process, which would only report the same error again. + */ +export class DaemonRequestUsageError extends Error { + /** The exit code of native Rush for this command line. */ + public readonly exitCode: number; + + public constructor(message: string, exitCode: number, options?: ErrorOptions) { + super(message, options); + this.name = 'DaemonRequestUsageError'; + this.exitCode = exitCode; + } +} diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index f58300645d..30affd7246 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -10,6 +10,7 @@ import { PhasedCommandEngineBusyError, PhasedCommandEngineConfigurationChangedError, PhasedCommandEngineProjectConfigurationError, + PhasedCommandEngineUsageError, type IPhasedCommandEngine, type IPhasedCommandEngineLogTelemetryOptions, type IInputsSnapshot, @@ -26,6 +27,7 @@ import { type IResolveDaemonRequestOptions, type ResolvedDaemonRequest } from './DaemonRequestDispatcher'; +import { DaemonRequestUsageError } from './DaemonRequestUsageError'; import { WorkspaceEngineComponentFactory, WorkspaceEngineRecreationRequiredError, @@ -224,6 +226,10 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { terminalProvider: terminal }); } catch (error) { + // Answer only for a command line of the requested command; in-process Rush reports any other one. + if (error instanceof PhasedCommandEngineUsageError && envelope.argv[0] === envelope.commandName) { + throw new DaemonRequestUsageError(terminal.describeError(error), error.exitCode, { cause: error }); + } throw new DaemonRequestDispatchError('unsupported', terminal.describeError(error), { cause: error }); } if (command.commandName !== envelope.commandName) { diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index dc25a53b33..156664e949 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -31,8 +31,10 @@ import { type DispatchWorkspaceRequestAsync, type IDaemonRequestDispatchClient, type IDaemonRequestLifecycle, - type IDaemonRequestResolver + type IDaemonRequestResolver, + type IResolveDaemonRequestOptions } from './DaemonRequestDispatcher'; +import { DaemonRequestUsageError } from './DaemonRequestUsageError'; import { createNativeMutationResolver } from './NativeMutationRequest'; import { parseDaemonGraphRequest, type IDaemonGraphRequest } from './DaemonGraphRequest'; import { isRushxInvocation, type IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; @@ -546,11 +548,11 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { let commandIdentity: string | undefined; let projectFingerprint: string | undefined; if (tier !== WorkspaceInputChangeTier.Restart && !isMutation(envelope)) { - commandIdentity = await getResolverLifecycle(this.#resolver).getCommandParameterIdentityAsync({ - envelope, - workspaceSession: session, - abortSignal: client.abortSignal - }); + commandIdentity = await getCommandParameterIdentityAsync( + this.#resolver, + { envelope, workspaceSession: session, abortSignal: client.abortSignal }, + tier + ); if (tier === WorkspaceInputChangeTier.Reuse) { projectFingerprint = await this.#tryCaptureProjectFingerprintAsync(session, receivedTimeMs); if ( @@ -637,11 +639,11 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { }; } - commandIdentity = await getResolverLifecycle(this.#resolver).getCommandParameterIdentityAsync({ - envelope, - workspaceSession: session, - abortSignal: client.abortSignal - }); + commandIdentity = await getCommandParameterIdentityAsync( + this.#resolver, + { envelope, workspaceSession: session, abortSignal: client.abortSignal }, + tier + ); if ( this.#boundSession === session && this.#commandIdentity === commandIdentity && @@ -1125,6 +1127,27 @@ function getResolverLifecycle(resolver: IDaemonRequestResolver): IWorkspaceResol return resolver.workspaceLifecycle; } +/** + * Parses the command with the session's configuration. A command line that this configuration rejects fails as + * native Rush would, unless the configuration changed since the session loaded it (`tier` is not + * `WorkspaceInputChangeTier.Reuse`): native Rush might accept the command line then, so the client runs it + * in-process instead. + */ +async function getCommandParameterIdentityAsync( + resolver: IDaemonRequestResolver, + options: IResolveDaemonRequestOptions, + tier: WorkspaceInputChangeTier +): Promise { + try { + return await getResolverLifecycle(resolver).getCommandParameterIdentityAsync(options); + } catch (error) { + if (error instanceof DaemonRequestUsageError && tier !== WorkspaceInputChangeTier.Reuse) { + throw new DaemonRequestDispatchError('unsupported', error.message, { cause: error }); + } + throw error; + } +} + /** A request rejected as invalid after the resolver bound a graph to the session: its selection failed. */ function isSelectionRejection( error: unknown, diff --git a/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts b/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts new file mode 100644 index 0000000000..1c2966e172 --- /dev/null +++ b/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts @@ -0,0 +1,61 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { IOperationGraph } from '@microsoft/rush-lib'; + +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; + +// These cases start real hosts and run Git/build subprocesses rather than mocked unit work. +jest.setTimeout(30_000); + +const message: string = 'rush build: error: Unrecognized arguments: --nope.'; +const invalid: string[] = ['build', '--to', 'b', '--nope']; +const usageFailure: object = { + kind: 'requestResult', + payload: { exitCode: 2, outcome: 'failure', aborted: false, errorMessage: message } +}; +const inProcess: object = { kind: 'requestRejected', payload: { code: 'unsupported', message } }; +const success: object = { kind: 'requestResult', payload: { exitCode: 0 } }; + +it('fails an invalid command line with the exit code of native Rush instead of handing it to in-process Rush', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); + try { + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(usageFailure); + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + const generation: number = fixture.host.workspaceGeneration; + const graph: IOperationGraph | undefined = fixture.session.operationGraph; + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(usageFailure); + expect(fixture.host.workspaceGeneration).toBe(generation); + expect(fixture.session.operationGraph).toBe(graph); + // The daemon answers only for a command line that starts with the requested command. + const globalFlag: string[] = ['--debug', ...invalid]; + expect((await fixture.runAsync(globalFlag, { commandName: 'build' })).terminal).toMatchObject(inProcess); + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + expect(fixture.session.operationGraph).toBe(graph); + expect(fixture.runs()).toEqual(['a', 'b']); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); + +it('hands an invalid command line to in-process Rush after a configuration change, which parses it again', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); + try { + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + // The warm session's configuration may be stale, for example without a parameter that experiments.json adds. + fixture.write( + 'a/package.json', + JSON.stringify({ + name: 'a', + version: '1.0.0', + scripts: { '_phase:compile': 'node build.cjs --changed' } + }) + ); + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(inProcess); + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(usageFailure); + expect(fixture.runs()).toEqual(['a', 'b', 'a', 'b']); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); diff --git a/libraries/rush-lib/src/api/PhasedCommandEngine.ts b/libraries/rush-lib/src/api/PhasedCommandEngine.ts index 8172ff36c0..66b56080ed 100644 --- a/libraries/rush-lib/src/api/PhasedCommandEngine.ts +++ b/libraries/rush-lib/src/api/PhasedCommandEngine.ts @@ -7,6 +7,8 @@ import type { PerformanceEntry } from 'node:perf_hooks'; import { FileSystem, LockFile } from '@rushstack/node-core-library'; import { Terminal, type ITerminalProvider } from '@rushstack/terminal'; import type { CommandLineAction } from '@rushstack/ts-command-line'; +// The package does not export the error that its parsers throw for an invalid command line. +import { CommandLineParserExitError } from '@rushstack/ts-command-line/lib/providers/CommandLineParserExitError'; import { RushCommandLineParser } from '../cli/RushCommandLineParser'; import { PhasedScriptAction } from '../cli/scriptActions/PhasedScriptAction'; @@ -22,6 +24,7 @@ import type { RushSession } from '../pluginFramework/RushSession'; import type { RushConfiguration } from './RushConfiguration'; import { RushUserConfiguration } from './RushUserConfiguration'; import { PhasedCommandEngineBusyError } from './PhasedCommandEngineBusyError'; +import { PhasedCommandEngineUsageError } from './PhasedCommandEngineUsageError'; import { resolvePhasedCommandCwdAsync } from '../utilities/resolvePhasedCommandCwd'; /** How long disposing an engine waits for `flushTelemetry` taps that are still running. */ @@ -167,6 +170,10 @@ export class PhasedCommandEngine { this.unmatchedCompatiblePluginNames = unmatchedCompatiblePluginNames; } + /** + * Parses a native build or rebuild command line. Throws a {@link PhasedCommandEngineUsageError} for a command + * line that native Rush rejects as invalid. + */ public static async parseAsync(options: IParsePhasedCommandOptions): Promise { const { rushConfiguration, terminalProvider, cwd, argv, environment = process.env } = options; const resolvedCwd: string = await resolvePhasedCommandCwdAsync(cwd, rushConfiguration.rushJsonFolder); @@ -182,7 +189,15 @@ export class PhasedCommandEngine { cwd: resolvedCwd, engine: { rushConfiguration, terminalProvider, environment } }); - await parser.executeWithoutErrorHandlingAsync([...argv]); + try { + await parser.executeWithoutErrorHandlingAsync([...argv]); + } catch (error) { + // Native Rush prints this message and exits with this exit code. + if (error instanceof CommandLineParserExitError && error.exitCode !== 0) { + throw new PhasedCommandEngineUsageError(error.message.trim(), error.exitCode, { cause: error }); + } + throw error; + } const action: CommandLineAction | undefined = parser.selectedAction; if (!(action instanceof PhasedScriptAction) || !['build', 'rebuild'].includes(action.actionName)) { throw new Error('The production daemon engine currently supports native build and rebuild only.'); diff --git a/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts b/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts new file mode 100644 index 0000000000..51a8afa15e --- /dev/null +++ b/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts @@ -0,0 +1,19 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * The command line is not valid for the command, for example because it names an unknown parameter. + * Native Rush prints the message and exits with {@link PhasedCommandEngineUsageError.exitCode}. + * No engine preparation has begun. + * @alpha + */ +export class PhasedCommandEngineUsageError extends Error { + /** The exit code of native Rush for this command line. */ + public readonly exitCode: number; + + public constructor(message: string, exitCode: number, options?: ErrorOptions) { + super(message, options); + this.name = 'PhasedCommandEngineUsageError'; + this.exitCode = exitCode; + } +} diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts index 8b3f371b9b..acef36cc56 100644 --- a/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts @@ -8,6 +8,7 @@ import * as path from 'node:path'; import { NoOpTerminalProvider, StringBufferTerminalProvider } from '@rushstack/terminal'; import { PhasedCommandEngine } from '../PhasedCommandEngine'; +import { PhasedCommandEngineUsageError } from '../PhasedCommandEngineUsageError'; import { RushConfiguration } from '../RushConfiguration'; const PACKAGE_NAME: string = '@example/rush-example-plugin'; @@ -186,6 +187,25 @@ describe(PhasedCommandEngine.name, () => { expect(command.commandName).toBe('build'); }); + it('reports an invalid command line as a usage error with the exit code of native Rush', async () => { + const folder: string = createTestRepo({ associatedCommands: [] }); + // The parser also prints the usage, as native Rush does. + const stderrWrite: jest.SpyInstance = jest.spyOn(process.stderr, 'write').mockReturnValue(true); + try { + for (const [argv, message] of [ + [['build', '--nope'], 'rush build: error: Unrecognized arguments: --nope.'], + [['rebuild', '--to'], 'rush rebuild: error: argument "-t/--to": Expected one argument. null'] + ] as const) { + const error: unknown = await parseBuildAsync(folder, [...argv]).catch((e: unknown) => e); + expect(error).toBeInstanceOf(PhasedCommandEngineUsageError); + const { message: actualMessage, exitCode } = error as PhasedCommandEngineUsageError; + expect({ message: actualMessage, exitCode }).toEqual({ message, exitCode: 2 }); + } + } finally { + stderrWrite.mockRestore(); + } + }); + it('rejects an unassociated plugin, which Rush initializes for every command', async () => { const folder: string = createTestRepo({ commandLineJson: COMMAND_SCOPED_COMMAND_LINE_JSON }); await expect(parseBuildAsync(folder)).rejects.toThrow( diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index 7abdab3c2b..431949409e 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -187,6 +187,7 @@ export { export { PhasedCommandEngineConfigurationChangedError } from './api/PhasedCommandEngineConfigurationChangedError'; export { PhasedCommandEngineProjectConfigurationError } from './api/PhasedCommandEngineProjectConfigurationError'; export { PhasedCommandEngineBusyError } from './api/PhasedCommandEngineBusyError'; +export { PhasedCommandEngineUsageError } from './api/PhasedCommandEngineUsageError'; export { captureWorkspaceInputFingerprintAsync, captureProjectConfigurationFingerprintAsync, From 4388b98bae19cd5595475158fa380a991f3b316a Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:15 +0000 Subject: [PATCH 063/265] [rush-daemon] Coalesced requests run with each requester's environment, and each gets its own telemetry key Swarm integration step 48; original commit b572eb50a0 (merge of swarm/r06-t90-int2 at 4b9a258212). Scope: tasks 81, 89 and 90. Brings r06's tasks 81 (coalesced requests no longer run once with the first requester's environment when an operation hashes an ignored variable they differ on), 89 (operations see process.env writes made by the same iteration's beforeExecuteIterationAsync, through a lazy per-participant environment) and 90 (a per-request telemetry key for operations in a coalesced batch), re-tipped on 1b51e7af08 (board 2747). Second agents: o04 board 783 and board 906, o05 board 1020, m03 board 1091 on adcb6f6e0f. It lands after r02's e7b848d6a7 (board 1166), which integration has. ch01's e2e on ch01-sC: two queued requests that differ in WT_SESSION run [ax, ay, c], as --no-daemon does. s16 batch C, item 3 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 8a64bfc100) Commits folded into this step (6): - 5565ff57ce [rush-daemon] Don't coalesce requests that disagree on a hashed per-session variable - 56ef144e1b [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks - e4d4e90711 [rush-daemon] Let requests share a run when a hashed variable is unset in one and empty in the other - aec3ac8433 [rush-daemon] Start each operation from process.env as it is when the operation starts - 36d9a3a16f [rush-lib] Add getOperationRequestId to the operation graph's iteration options - 3099d5945f [rush-daemon] Give each operation the request id of the request whose environment it gets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...rushd-operation-request-id_2026-09-28.json | 11 + ...ushd-coalesced-request-env_2026-09-28.json | 11 + .../rushd-lazy-op-env_2026-09-28.json | 11 + ...rushd-operation-request-id_2026-09-28.json | 11 + common/reviews/api/rush-lib.api.md | 1 + libraries/rush-daemon/README.md | 22 +- .../rush-daemon/src/PhasedRequestRouter.ts | 115 ++++++- .../src/test/PhasedRequestBatching.test.ts | 323 +++++++++++++++++- .../src/logic/operations/IOperationGraph.ts | 12 + .../src/logic/operations/OperationGraph.ts | 12 +- ...OperationGraphOperationEnvironment.test.ts | 29 ++ 11 files changed, 530 insertions(+), 28 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-operation-request-id_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-coalesced-request-env_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-lazy-op-env_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-daemon/rushd-operation-request-id_2026-09-28.json diff --git a/common/changes/@microsoft/rush/rushd-operation-request-id_2026-09-28.json b/common/changes/@microsoft/rush/rushd-operation-request-id_2026-09-28.json new file mode 100644 index 0000000000..775de7bf45 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-operation-request-id_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add `getOperationRequestId` to the iteration options of an operation graph, so that a host that serves several requests in one iteration can tell `configureIteration`, `beforeExecuteIterationAsync` and `afterExecuteIterationAsync` which request each operation was run for.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/rushd-coalesced-request-env_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-coalesced-request-env_2026-09-28.json new file mode 100644 index 0000000000..baa6e161ca --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-coalesced-request-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Give queued phased requests separate iterations when they disagree on a per-session variable, such as `WT_SESSION`, that an operation lists in `dependsOnEnvVars`. Previously they shared one iteration, so the shared operation ran only with the first request's value and every other requester was told it succeeded.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} \ No newline at end of file diff --git a/common/changes/@rushstack/rush-daemon/rushd-lazy-op-env_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-lazy-op-env_2026-09-28.json new file mode 100644 index 0000000000..b121793292 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-lazy-op-env_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Start each phased operation from the daemon's `process.env` as it is when the operation starts, so it sees the variables that a plugin sets, changes or deletes in the same iteration's `beforeExecuteIterationAsync`. Previously the router copied `process.env` when it scheduled the iteration, so operations saw the values from the previous request's hooks.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} \ No newline at end of file diff --git a/common/changes/@rushstack/rush-daemon/rushd-operation-request-id_2026-09-28.json b/common/changes/@rushstack/rush-daemon/rushd-operation-request-id_2026-09-28.json new file mode 100644 index 0000000000..148431ca0e --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/rushd-operation-request-id_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Give the iteration hooks of a phased request the `requestId` of the request whose environment each operation gets, through `getOperationRequestId`, so that a plugin can join per-operation telemetry with the telemetry of the request that selected the operation.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} \ No newline at end of file diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 10cbe380ac..8fa1c09a04 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -778,6 +778,7 @@ export interface _IOperationGraphEventSink { // @alpha export interface IOperationGraphIterationOptions { getOperationEnvironment?: (operation: Operation) => Readonly>; + getOperationRequestId?: (operation: Operation) => string | undefined; // (undocumented) inputsSnapshot?: IInputsSnapshot; startTime?: number; diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 24f1f68c86..5179afee7e 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -93,11 +93,15 @@ comparisons (the tier-2 fingerprint and the production resolver's startup-enviro `getWorkspaceFingerprintEnvironmentEntries()`, which omits `workspaceFingerprintIgnoredEnvironmentVariables`: volatile per-shell, terminal, session and client-routing variables such as `PWD`, `OLDPWD`, `SHLVL`, `_`, `TERM`, `COLUMNS`, `WSL_INTEROP`, `SSH_*`, `INIT_CWD`, `RUSH_DAEMON`, `RUSH_DAEMON_AUTO_START` and -`RUSH_DAEMON_EXPERIMENTAL`. Rush does not read these to configure the engine, build the graph or hash operations, +`RUSH_DAEMON_EXPERIMENTAL`. Rush does not read these to configure the engine or build the graph, so running a command from a project subfolder or another shell reuses the warm workspace. All other environment inputs, including every other `RUSH_*` variable, `NODE_*`, npm/pnpm configuration, `PATH` and `HOME`, remain -unchanged and are checked normally. Phased operation processes inherit the daemon's own environment, so they -see the daemon's startup values for the ignored variables rather than the submitting shell's values. +unchanged and are checked normally. Each phased operation process takes the ignored variables from the request +that selected it (`getWorkspaceRequestOperationEnvironment()`) and hashes its `dependsOnEnvVars` from that +environment, so it sees the submitting shell's values, as a native command would. The rest of that environment is +the daemon's `process.env` when the operation starts, so it includes variables that a plugin sets in the same +iteration's `beforeExecuteIterationAsync`; state hashes use the values from when the iteration was scheduled, before +those hooks run, as native Rush does. Compatible selections reuse the same graph and records. An unchanged successful build schedules no work; rebuild still invalidates the graph on each request. Every execution refreshes operation inputs under its native lease. With the build cache enabled, a cacheable operation whose tracked input files change while the inputs snapshot is @@ -535,9 +539,15 @@ drains. The result translates only that client's operation subset to Rush's succ semantics. Warning-only builds honor the operation's configured `allowWarningsInSuccessfulBuild` state plus the request's immutable `RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD` environment override without mutating `process.env`. Compatible phased `SHARED-BUILD` requests admitted before the next graph iteration starts are coalesced at a -deterministic event-loop-turn boundary. The router reconciles retained invalidations once, unions the clients' enabled -dependency closures, and schedules one iteration. Shared operations execute once, while each client subscribes only -to its own closure and derives its final result only from that subset. A client does not wait for the other clients' +deterministic event-loop-turn boundary. Requests are compatible when they have the same request settings and the same +value of every environment variable that an operation of the graph lists in `dependsOnEnvVars` (an unset variable and +an empty one hash alike, so they count as the same value), because +a shared operation runs once, in the environment of the first request that selected it. Iteration hooks get the +same attribution: `getOperationRequestId` returns the `requestId` of the request whose environment +`getOperationEnvironment` returns for an operation. The router reconciles +retained invalidations once, unions the clients' enabled dependency closures, and schedules one iteration. Shared +operations execute once, while each client subscribes only to its own closure and derives its final result only from +that subset. A client does not wait for the other clients' larger selections: once every operation of its own closure that the iteration scheduled has completed and its output has drained, its result is published while the iteration, graph lease, and native execution lease continue for the remaining clients. The last client that still needs the iteration receives its result after iteration end and lease diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index 27a9299eb7..e7bb86417b 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -4,6 +4,7 @@ import type { IOperationExecutionResult, IOperationGraph, + IOperationGraphIterationOptions, IPhasedCommandEngineRequestSettings, Operation, _IOperationGraphEventSink @@ -79,8 +80,12 @@ interface IPreparedPhasedRequest { /** Lets requests that cannot run alongside this one preempt its workspace admission; see `#settleEntry`. */ readonly markAdmissionPreemptible: (onPreempted: () => void) => void; readonly request: IDaemonPhasedRequest; - /** Only requests with the same settings share one graph iteration. */ + /** The settings that the request's iteration applies to the graph. */ readonly requestSettings: IPhasedCommandEngineRequestSettings | undefined; + /** + * Only requests with the same settings, and the same values for every variable that the graph's operations hash, + * share one graph iteration. + */ readonly requestSettingsKey: string; readonly selection: IResolvedSelection; /** The `performance.now()` timestamp at which the daemon received the request; see `#canJoinCurrentBatch`. */ @@ -272,7 +277,7 @@ export class PhasedRequestRouter { receivedTimeMs: receivedTimeMs ?? startTimeMs, request, requestSettings, - requestSettingsKey: JSON.stringify(requestSettings ?? null), + requestSettingsKey: getRequestSettingsKey(graph, request, requestSettings), selection, startTimeMs, telemetry, @@ -569,7 +574,7 @@ class PhasedRequestBatchCoordinator { timings.scheduleStartTimeMs = performance.now(); scheduled = await this.#graph.scheduleIterationAsync({ inputsSnapshot: this.#workspaceSession.inputsSnapshot, - getOperationEnvironment: createOperationEnvironmentLookup(participants) + ...createOperationParticipantLookups(participants) }); timings.scheduledTimeMs = performance.now(); if (scheduled) { @@ -1176,28 +1181,102 @@ function validateRequestIdentity(request: IDaemonPhasedRequest): void { } /** - * Gives each operation of an iteration the environment of the first participant that selected it, as a native - * command would run it in its invoker's environment. An operation that several participants share runs once, in - * the first participant's environment. + * Returns the key that decides which requests can share one graph iteration. + * + * @remarks + * An operation that several participants select runs once, in the first participant's environment, and hashes its + * `dependsOnEnvVars` from that environment. Variables that do not select a daemon, such as `WT_SESSION`, reach the + * operation from each request, so the key includes this request's value of every variable that an operation of the + * graph lists in `dependsOnEnvVars`. Requests that disagree on one of them get separate iterations, and each + * operation runs and is hashed as it would be for its own requester. The key takes each value as the operation hashes + * it, so an unset variable and an empty one are the same value. + */ +function getRequestSettingsKey( + graph: IOperationGraph, + request: IDaemonPhasedRequest, + requestSettings: IPhasedCommandEngineRequestSettings | undefined +): string { + const names: ReadonlyArray = getGraphDependsOnEnvVars(graph); + const environment: Readonly> = + names.length > 0 ? getWorkspaceRequestOperationEnvironment(process.env, request.environment) : {}; + // InputsSnapshot hashes `environment[name] || ''`. + const values: ReadonlyArray = names.map((name: string) => [ + name, + environment[name] || '' + ]); + return JSON.stringify([requestSettings ?? null, values]); +} + +/** Returns the names of the environment variables that the graph's operations hash, sorted. */ +function getGraphDependsOnEnvVars(graph: IOperationGraph): ReadonlyArray { + const names: Set = new Set(); + for (const operation of graph.operations) { + for (const name of operation.settings?.dependsOnEnvVars ?? []) { + names.add(name); + } + } + return Array.from(names).sort(Sort.compareByValue); +} + +/** + * The lookups that attribute each operation of an iteration to one of its participants. */ -function createOperationEnvironmentLookup( +type IOperationParticipantLookups = Required< + Pick +>; + +/** + * Attributes each operation of an iteration to the first participant that selected it, as a native command would + * run it for its invoker, or to the first participant if no participant selected it. `getOperationEnvironment` + * returns that participant's environment and `getOperationRequestId` its request id, so a plugin can attribute the + * operation to the request whose environment it ran in. An operation that several participants share runs once, in + * the first participant's environment; the participants agree on the hashed value of every variable that an + * operation hashes (see `getRequestSettingsKey`). + * + * @remarks + * Each environment is a copy of the daemon's `process.env` in which the variables that do not select a daemon take + * the participant's values. The copy is taken when it is asked for, so an operation starts from `process.env` as it + * is when the operation starts, as a native command's operation does. That includes the variables that a plugin + * sets, changes or deletes in the same iteration's `beforeExecuteIterationAsync`. Reading `process.env` is slow, and + * the graph hashes every operation while it schedules an iteration, so the calls made before the next microtask + * share one copy for each participant. + */ +function createOperationParticipantLookups( participants: ReadonlyArray -): (operation: Operation) => Readonly> { - const environmentByOperation: Map>> = new Map(); - let firstEnvironment: Readonly> | undefined; +): IOperationParticipantLookups { + const entryByOperation: Map = new Map(); for (const entry of participants) { - const environment: Readonly> = getWorkspaceRequestOperationEnvironment( - process.env, - entry.request.environment - ); - firstEnvironment ??= environment; for (const operation of entry.selection.activeOperations) { - if (!environmentByOperation.has(operation)) { - environmentByOperation.set(operation, environment); + if (!entryByOperation.has(operation)) { + entryByOperation.set(operation, entry); } } } - return (operation: Operation) => environmentByOperation.get(operation) ?? firstEnvironment ?? process.env; + const firstEntry: IBatchEntry | undefined = participants[0]; + const getEntry = (operation: Operation): IBatchEntry | undefined => + entryByOperation.get(operation) ?? firstEntry; + let environmentByEntry: Map>> | undefined; + return { + getOperationEnvironment: (operation: Operation) => { + const entry: IBatchEntry | undefined = getEntry(operation); + if (!entry) { + return process.env; + } + if (!environmentByEntry) { + environmentByEntry = new Map(); + queueMicrotask(() => { + environmentByEntry = undefined; + }); + } + let environment: Readonly> | undefined = environmentByEntry.get(entry); + if (!environment) { + environment = getWorkspaceRequestOperationEnvironment(process.env, entry.request.environment); + environmentByEntry.set(entry, environment); + } + return environment; + }, + getOperationRequestId: (operation: Operation) => getEntry(operation)?.request.requestId + }; } function validateNonemptyName(value: string, kind: string): void { diff --git a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts index dea724566b..5e4b77751a 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts @@ -10,7 +10,12 @@ import type { } from '@rushstack/rush-daemon-protocol'; import { RUSHD_OPERATION_HEADER, RUSHD_OPERATION_STREAM_CLOSED } from '@rushstack/rush-daemon-protocol'; import { OperationStatus } from '@microsoft/rush-lib'; -import type { IOperationRunnerContext, IPhasedCommandEngineRequestSettings } from '@microsoft/rush-lib'; +import type { + IInputsSnapshot, + IOperationRunnerContext, + IPhasedCommandEngineRequestSettings, + IRushConfigurationProjectForSnapshot +} from '@microsoft/rush-lib'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; import { @@ -20,6 +25,7 @@ import { createRoutingFixture } from './PhasedRequestRouterTestUtilities'; import type { ITestClientWrite, ITestRoutingFixture } from './PhasedRequestRouterTestUtilities'; +import { TEST_REPO_ROOT } from './TestWorkspaceSession'; const OPERATION_A: string = 'project-a (_phase:test)'; const OPERATION_B: string = 'project-b (_phase:test)'; @@ -275,6 +281,321 @@ describe('shared phased request batching', () => { ); }); + it('gives each operation the request id of the request whose environment it gets', async () => { + const fixture: ITestRoutingFixture = createFixture(); + const iterations: string[][] = []; + fixture.graph.hooks.beforeExecuteIterationAsync.tap('test plugin', (records, options) => { + const lines: string[] = []; + for (const operation of records.keys()) { + const session: string | undefined = options.getOperationEnvironment?.(operation)[SESSION_VARIABLE]; + lines.push(`${operation.name}: ${options.getOperationRequestId?.(operation)} ${session}`); + } + iterations.push(lines.sort()); + }); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + const withSession = (request: IDaemonPhasedRequest, session: string): IDaemonPhasedRequest => ({ + ...request, + environment: { [SESSION_VARIABLE]: session } + }); + + await Promise.all([ + router.executeAsync( + withSession(createRequest('x', OPERATION_A, OPERATION_C), 'session-X'), + new TestPhasedRequestClient('x') + ), + router.executeAsync( + withSession(createRequest('y', OPERATION_B), 'session-Y'), + new TestPhasedRequestClient('y') + ) + ]); + await router.executeAsync( + withSession(createRequest('z', OPERATION_A), 'session-Z'), + new TestPhasedRequestClient('z') + ); + + // An operation that no participant selected gets the first participant's environment and request id. + expect(iterations).toEqual([ + [`${OPERATION_A}: x session-X`, `${OPERATION_B}: y session-Y`, `${OPERATION_C}: x session-X`], + [`${OPERATION_A}: z session-Z`, `${OPERATION_B}: z session-Z`, `${OPERATION_C}: z session-Z`] + ]); + }); + + describe('with a plugin that changes process.env in beforeExecuteIterationAsync', () => { + const ADDED_VARIABLE: string = 'RUSHD_TEST_ADDED'; + const CHANGED_VARIABLE: string = 'RUSHD_TEST_CHANGED'; + const REMOVED_VARIABLE: string = 'RUSHD_TEST_REMOVED'; + const NAMES: ReadonlyArray = [ + SESSION_VARIABLE, + ADDED_VARIABLE, + CHANGED_VARIABLE, + REMOVED_VARIABLE + ]; + let savedValues: ReadonlyArray = []; + + beforeEach(() => { + savedValues = NAMES.map((name: string) => process.env[name]); + process.env[SESSION_VARIABLE] = 'daemon'; + delete process.env[ADDED_VARIABLE]; + process.env[CHANGED_VARIABLE] = 'original'; + process.env[REMOVED_VARIABLE] = 'original'; + }); + + afterEach(() => { + NAMES.forEach((name: string, index: number) => { + const value: string | undefined = savedValues[index]; + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + }); + }); + + function createFixtureWithPlugin(options?: Parameters[0]): ITestRoutingFixture { + const fixture: ITestRoutingFixture = createFixture(options); + let iteration: number = 0; + fixture.graph.hooks.beforeExecuteIterationAsync.tap('test plugin', () => { + iteration++; + process.env[ADDED_VARIABLE] = `added-${iteration}`; + process.env[CHANGED_VARIABLE] = `changed-${iteration}`; + delete process.env[REMOVED_VARIABLE]; + }); + return fixture; + } + + function withSession(request: IDaemonPhasedRequest, session: string): IDaemonPhasedRequest { + return { ...request, environment: { [SESSION_VARIABLE]: session } }; + } + + function recordEnvironment(seen: string[], operationId: string): TestOperationAction { + return async (terminal: ITerminal, context: IOperationRunnerContext): Promise => { + const values: string[] = NAMES.map((name: string) => context.environment?.[name] ?? ''); + seen.push(`${operationId}: ${values.join(' ')}`); + }; + } + + it("starts each operation from what the same iteration's hook set, changed and deleted, with its requester's session", async () => { + const seen: string[] = []; + const fixture: ITestRoutingFixture = createFixtureWithPlugin({ + actionAAsync: recordEnvironment(seen, OPERATION_A), + actionCAsync: recordEnvironment(seen, OPERATION_C) + }); + const scheduleSpy: jest.SpyInstance = jest.spyOn(fixture.graph, 'scheduleIterationAsync'); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + + await Promise.all([ + router.executeAsync( + withSession(createRequest('a', OPERATION_A), 'session-A'), + new TestPhasedRequestClient('a') + ), + router.executeAsync( + withSession(createRequest('c', OPERATION_C), 'session-C'), + new TestPhasedRequestClient('c') + ) + ]); + await router.executeAsync( + withSession(createRequest('a-again', OPERATION_A), 'session-A-again'), + new TestPhasedRequestClient('a-again') + ); + + expect(scheduleSpy).toHaveBeenCalledTimes(2); + expect([...seen].sort()).toEqual([ + `${OPERATION_A}: session-A added-1 changed-1 `, + `${OPERATION_A}: session-A-again added-2 changed-2 `, + `${OPERATION_C}: session-C added-1 changed-1 ` + ]); + }); + + it('hashes from one copy of each requester environment while scheduling and starts from a copy taken after the hook', async () => { + const hashedEnvironments: Map< + IRushConfigurationProjectForSnapshot, + Readonly> | undefined + > = new Map(); + const seen: string[] = []; + const fixture: ITestRoutingFixture = createFixtureWithPlugin({ + actionAAsync: recordEnvironment(seen, OPERATION_A), + actionBAsync: recordEnvironment(seen, OPERATION_B), + actionCAsync: recordEnvironment(seen, OPERATION_C) + }); + const inputsSnapshot: IInputsSnapshot = { + getOperationOwnStateHash: ( + project: IRushConfigurationProjectForSnapshot, + operationName?: string, + environment?: Readonly> + ): string => { + hashedEnvironments.set(project, environment); + return 'hash'; + }, + getTrackedFileHashesForOperation: () => new Map(), + hasUncommittedChanges: false, + hashes: new Map(), + rootDirectory: TEST_REPO_ROOT + }; + Object.assign(fixture.session, { inputsSnapshot }); + const router: PhasedRequestRouter = new PhasedRequestRouter(fixture.session); + + await Promise.all([ + router.executeAsync( + withSession(createRequest('x', OPERATION_A, OPERATION_C), 'session-X'), + new TestPhasedRequestClient('x') + ), + router.executeAsync( + withSession(createRequest('y', OPERATION_B), 'session-Y'), + new TestPhasedRequestClient('y') + ) + ]); + + const [hashedA, hashedB, hashedC] = [OPERATION_A, OPERATION_B, OPERATION_C].map((operationId: string) => + hashedEnvironments.get(fixture.operations.get(operationId)!.associatedProject) + ); + expect(hashedA?.[SESSION_VARIABLE]).toBe('session-X'); + expect(hashedB?.[SESSION_VARIABLE]).toBe('session-Y'); + expect(hashedC).toBe(hashedA); + expect(hashedA?.[CHANGED_VARIABLE]).toBe('original'); + expect([...seen].sort()).toEqual([ + `${OPERATION_A}: session-X added-1 changed-1 `, + `${OPERATION_B}: session-Y added-1 changed-1 `, + `${OPERATION_C}: session-X added-1 changed-1 ` + ]); + }); + }); + + describe('with an operation that hashes a variable that does not select a daemon', () => { + const TERMINAL_VARIABLE: string = 'WT_SESSION'; + const HOST_VARIABLE: string = 'RUSHD_TEST_HOST_VARIABLE'; + + interface IHashedVariableFixture { + readonly fixture: ITestRoutingFixture; + readonly router: PhasedRequestRouter; + readonly runs: ReadonlyArray; + readonly scheduleSpy: jest.SpyInstance; + } + + function createHashedVariableFixture( + dependsOnEnvVars: string[], + actionCAsync?: TestOperationAction + ): IHashedVariableFixture { + const runs: (string | undefined)[] = []; + const fixture: ITestRoutingFixture = createFixture({ + actionAAsync: async (terminal: ITerminal, context: IOperationRunnerContext): Promise => { + runs.push(context.environment?.[TERMINAL_VARIABLE]); + }, + actionCAsync + }); + fixture.operations.get(OPERATION_A)!.settings = { operationName: '_phase:test', dependsOnEnvVars }; + return { + fixture, + router: new PhasedRequestRouter(fixture.session), + runs, + scheduleSpy: jest.spyOn(fixture.graph, 'scheduleIterationAsync') + }; + } + + function createRequestWithEnvironment( + requestId: string, + environment: Record, + operationId: string + ): IDaemonPhasedRequest { + return { ...createRequest(requestId, operationId), environment }; + } + + it('schedules separate iterations for queued requests that disagree on its value', async () => { + const occupierStarted: IDeferred = createDeferred(); + const releaseOccupier: IDeferred = createDeferred(); + const { fixture, router, runs, scheduleSpy } = createHashedVariableFixture( + [TERMINAL_VARIABLE], + async (): Promise => { + occupierStarted.resolve(); + await releaseOccupier.promise; + } + ); + const occupier: Promise = router.executeAsync( + createRequest('occupier', OPERATION_C), + new TestPhasedRequestClient('occupier') + ); + await occupierStarted.promise; + const results: Promise = Promise.all([ + router.executeAsync( + createRequestWithEnvironment('x', { [TERMINAL_VARIABLE]: 'wt-X' }, OPERATION_A), + new TestPhasedRequestClient('x') + ), + router.executeAsync( + createRequestWithEnvironment('y', { [TERMINAL_VARIABLE]: 'wt-Y' }, OPERATION_A), + new TestPhasedRequestClient('y') + ) + ]); + releaseOccupier.resolve(); + await occupier; + const [x, y] = await results; + + expect(scheduleSpy).toHaveBeenCalledTimes(3); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(2); + expect(runs).toEqual(['wt-X', 'wt-Y']); + expect(x).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(y).toMatchObject({ exitCode: 0, outcome: 'success' }); + expect(getResultOperationIds(x)).toEqual([OPERATION_A]); + expect(getResultOperationIds(y)).toEqual([OPERATION_A]); + }); + + it('shares one iteration for requests that agree on its value', async () => { + const { fixture, router, runs, scheduleSpy } = createHashedVariableFixture([TERMINAL_VARIABLE]); + + await Promise.all([ + router.executeAsync( + createRequestWithEnvironment( + 'x', + { [TERMINAL_VARIABLE]: 'wt-X', [SESSION_VARIABLE]: 'session-X' }, + OPERATION_A + ), + new TestPhasedRequestClient('x') + ), + router.executeAsync( + createRequestWithEnvironment( + 'y', + { [TERMINAL_VARIABLE]: 'wt-X', [SESSION_VARIABLE]: 'session-Y' }, + OPERATION_A + ), + new TestPhasedRequestClient('y') + ) + ]); + + expect(scheduleSpy).toHaveBeenCalledTimes(1); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + expect(runs).toEqual(['wt-X']); + }); + + it('shares one iteration for an unset value and an empty one, which operations hash alike', async () => { + const { fixture, router, runs, scheduleSpy } = createHashedVariableFixture([TERMINAL_VARIABLE]); + + await Promise.all([ + router.executeAsync(createRequest('unset', OPERATION_A), new TestPhasedRequestClient('unset')), + router.executeAsync( + createRequestWithEnvironment('empty', { [TERMINAL_VARIABLE]: '' }, OPERATION_A), + new TestPhasedRequestClient('empty') + ) + ]); + + expect(scheduleSpy).toHaveBeenCalledTimes(1); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + expect(runs).toEqual([undefined]); + }); + + it('keeps sharing one iteration when requests differ only in a variable that operations take from the daemon', async () => { + const { fixture, router, scheduleSpy } = createHashedVariableFixture([HOST_VARIABLE]); + + await Promise.all([ + router.executeAsync( + createRequestWithEnvironment('x', { [HOST_VARIABLE]: 'x' }, OPERATION_A), + new TestPhasedRequestClient('x') + ), + router.executeAsync( + createRequestWithEnvironment('y', { [HOST_VARIABLE]: 'y' }, OPERATION_A), + new TestPhasedRequestClient('y') + ) + ]); + + expect(scheduleSpy).toHaveBeenCalledTimes(1); + expect(fixture.runners.get(OPERATION_A)?.runCount).toBe(1); + }); + }); + it('shares one iteration for disjoint selections while isolating streams, events, and results', async () => { const fixture: ITestRoutingFixture = createFixture({ actionAAsync: async (terminal: ITerminal): Promise => terminal.writeLine('only-a'), diff --git a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts index 7da9225e19..9d78c3bede 100644 --- a/libraries/rush-lib/src/logic/operations/IOperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/IOperationGraph.ts @@ -33,6 +33,18 @@ export interface IOperationGraphIterationOptions { * (see `getWorkspaceRequestOperationEnvironment`). */ getOperationEnvironment?: (operation: Operation) => Readonly>; + + /** + * Returns the identifier of the request whose environment `getOperationEnvironment` returns for an operation of + * this iteration, or `undefined` if there is no such request. When omitted, the iteration serves no identified + * request. + * + * @remarks + * A long-lived host can serve several requests in one iteration. A plugin can use this identifier to attribute + * each operation to the request that selected it, for example to join per-operation telemetry with that + * request's own telemetry entry. + */ + getOperationRequestId?: (operation: Operation) => string | undefined; } /** diff --git a/libraries/rush-lib/src/logic/operations/OperationGraph.ts b/libraries/rush-lib/src/logic/operations/OperationGraph.ts index c78fcf862a..3983c886b1 100644 --- a/libraries/rush-lib/src/logic/operations/OperationGraph.ts +++ b/libraries/rush-lib/src/logic/operations/OperationGraph.ts @@ -97,6 +97,8 @@ interface IExecutionIterationContext extends IOperationExecutionRecordContext { startTime?: number; + getOperationRequestId?: (operation: Operation) => string | undefined; + completedOperations: number; totalOperations: number; } @@ -676,12 +678,14 @@ export class OperationGraph implements IOperationGraph { const { startTime = performance.now(), inputsSnapshot = await getInputsSnapshotAsync?.(), - getOperationEnvironment + getOperationEnvironment, + getOperationRequestId } = iterationOptions; const iterationOptionsForCallbacks: IOperationGraphIterationOptions = { startTime, inputsSnapshot, - getOperationEnvironment + getOperationEnvironment, + getOperationRequestId }; const { hooks } = this; @@ -731,6 +735,7 @@ export class OperationGraph implements IOperationGraph { onOperationStateChanged: undefined, createEnvironment: createEnvironmentForOperation, getOperationEnvironment, + getOperationRequestId, invalidate: graph.invalidateOperations.bind(graph), get debugMode(): boolean { return graph.debugMode; @@ -893,7 +898,8 @@ export class OperationGraph implements IOperationGraph { const iterationOptions: IOperationGraphIterationOptions = { inputsSnapshot: iterationContext.inputsSnapshot, startTime: iterationContext.startTime, - getOperationEnvironment: iterationContext.getOperationEnvironment + getOperationEnvironment: iterationContext.getOperationEnvironment, + getOperationRequestId: iterationContext.getOperationRequestId }; const executionQueue: AsyncOperationQueue = new AsyncOperationQueue( diff --git a/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts index 061de0a2b8..1cd94685df 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationGraphOperationEnvironment.test.ts @@ -196,4 +196,33 @@ describe('OperationGraph operation environment', () => { ['afterExecuteIterationAsync', undefined] ]); }); + + it('gives every iteration hook the request id lookup that the iteration was scheduled with', async () => { + const graph: OperationGraph = createGraph([new EnvironmentRecordingRunner('operation')]); + type GetOperationRequestId = IOperationGraphIterationOptions['getOperationRequestId']; + const received: [string, GetOperationRequestId][] = []; + graph.hooks.configureIteration.tap('test', (records, lastResults, options) => { + received.push(['configureIteration', options.getOperationRequestId]); + }); + graph.hooks.beforeExecuteIterationAsync.tapPromise('test', async (records, options) => { + received.push(['beforeExecuteIterationAsync', options.getOperationRequestId]); + }); + graph.hooks.afterExecuteIterationAsync.tapPromise('test', async (status, records, options) => { + received.push(['afterExecuteIterationAsync', options.getOperationRequestId]); + return status; + }); + const getOperationRequestId: GetOperationRequestId = () => 'request-A'; + + expect((await graph.executeAsync({ getOperationRequestId })).status).toBe(OperationStatus.Success); + expect((await graph.executeAsync({})).status).toBe(OperationStatus.Success); + + expect(received).toEqual([ + ['configureIteration', getOperationRequestId], + ['beforeExecuteIterationAsync', getOperationRequestId], + ['afterExecuteIterationAsync', getOperationRequestId], + ['configureIteration', undefined], + ['beforeExecuteIterationAsync', undefined], + ['afterExecuteIterationAsync', undefined] + ]); + }); }); From 0f2d00c1c690752a10b92db8d2267d97e8761db9 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:16 +0000 Subject: [PATCH 064/265] [rush-lib] A build-cache hit no longer restores an earlier failing run's stderr Swarm integration step 49; original commit c3f9fa2ac7 (merge of swarm/o03-t157 at 5b2a400f29). Scope: task 157. Brings o03's task 157: a cache hit no longer writes an older failing run's stderr into rush-logs/.error.log, so a cache entry written after a failure isn't poisoned (ch02 board 2284, t05 board 2456). Second agent: t05 board 2704. ch01's e2e on ch01-sC: no error.log after either hit in the OK, ERR, OK sequence, and none in any cache entry. s16 batch C, item 4 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 09d5eabe82) Commits folded into this step (1): - 5b2a400f29 Keep cache hits from restoring an earlier run's stderr (task 157) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...-cache-hit-error-log_2026-09-29-00-20.json | 11 ++ .../operations/OperationMetadataManager.ts | 80 +++++++-- .../test/OperationMetadataManager.test.ts | 162 +++++++++++++++++- 3 files changed, 241 insertions(+), 12 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-o03-t157-cache-hit-error-log_2026-09-29-00-20.json diff --git a/common/changes/@microsoft/rush/swarm-o03-t157-cache-hit-error-log_2026-09-29-00-20.json b/common/changes/@microsoft/rush/swarm-o03-t157-cache-hit-error-log_2026-09-29-00-20.json new file mode 100644 index 0000000000..dcc65945a8 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-o03-t157-cache-hit-error-log_2026-09-29-00-20.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "A build cache hit no longer writes the stderr of an earlier failed run to the operation's error log in rush-logs. When an operation runs without writing to stderr, Rush deletes the error log that an earlier run left in the operation's metadata folder, so later cache entries don't store it. On a cache hit, Rush writes the error log from the entry's log chunks, or deletes it when they hold no stderr, so entries saved by earlier versions can't restore a stale error log either.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-lib/src/logic/operations/OperationMetadataManager.ts b/libraries/rush-lib/src/logic/operations/OperationMetadataManager.ts index 51007b65a6..b8ece54487 100644 --- a/libraries/rush-lib/src/logic/operations/OperationMetadataManager.ts +++ b/libraries/rush-lib/src/logic/operations/OperationMetadataManager.ts @@ -3,11 +3,18 @@ import * as fs from 'node:fs'; -import { Async, FileSystem, type IFileSystemCopyFileOptions } from '@rushstack/node-core-library'; +import { + Async, + FileSystem, + type IFileSystemCopyFileOptions, + NewlineKind +} from '@rushstack/node-core-library'; import { type ITerminalChunk, TerminalChunkKind, TerminalProviderSeverity, + TerminalWritable, + TextRewriterTransform, type ITerminal, type ITerminalProvider } from '@rushstack/terminal'; @@ -125,6 +132,10 @@ export class OperationMetadataManager { if (!FileSystem.isNotExistError(e)) { throw e; } + + // This run didn't write the file (a run that writes nothing to stderr has no error log), so delete + // the copy an earlier run left. The metadata folder is a cached output and must describe only this run. + await FileSystem.deleteFileAsync(options.destinationPath); } }); } @@ -149,6 +160,7 @@ export class OperationMetadataManager { this.stateFile.state?.cobuildContextId === cobuildContextId && this.stateFile.state?.cobuildRunnerId !== cobuildRunnerId; + let errorLogText: string | undefined; try { const rawLogChunks: string = await FileSystem.readFileAsync(this.#logChunksPath); const chunks: ITerminalChunk[] = []; @@ -164,6 +176,7 @@ export class OperationMetadataManager { terminalProvider.write(text, TerminalProviderSeverity.log); } } + errorLogText = getErrorLogText(chunks); } catch (e) { if (FileSystem.isNotExistError(e)) { // Log chunks file doesn't exist, try to restore log file @@ -173,15 +186,29 @@ export class OperationMetadataManager { } } - // Try to restore cached error log as error log file - try { - await FileSystem.copyFileAsync({ - sourcePath: this.#errorLogPath, - destinationPath: errorLogPath - }); - } catch (e) { - if (!FileSystem.isNotExistError(e)) { - throw e; + // The error log file shows the stderr of the run that produced the cache entry. When the entry has log + // chunks, write it from them rather than copying the cached error log: an entry saved before saveAsync + // deleted stale files can hold the error log of an earlier run that failed. + if (errorLogText !== undefined) { + if (errorLogText) { + await FileSystem.writeFileAsync(errorLogPath, errorLogText, { ensureFolderExists: true }); + } else { + await FileSystem.deleteFileAsync(errorLogPath); + } + } else { + // Try to restore cached error log as error log file + try { + await FileSystem.copyFileAsync({ + sourcePath: this.#errorLogPath, + destinationPath: errorLogPath + }); + } catch (e) { + if (!FileSystem.isNotExistError(e)) { + throw e; + } + + // The entry has no error log, so don't leave the one from the last run that executed. + await FileSystem.deleteFileAsync(errorLogPath); } } } @@ -199,6 +226,39 @@ export class OperationMetadataManager { } } +/** + * Collects the text of the stderr chunks written to it. + */ +class StderrTextWritable extends TerminalWritable { + public text: string = ''; + + protected onWriteChunk(chunk: ITerminalChunk): void { + if (chunk.kind === TerminalChunkKind.Stderr) { + this.text += chunk.text; + } + } +} + +/** + * Returns the error log file text for an operation's log chunks: the stderr chunks, rewritten the same way + * as the error log that `initializeProjectLogFilesAsync` writes when the operation runs. + */ +function getErrorLogText(chunks: ReadonlyArray): string { + const stderrText: StderrTextWritable = new StderrTextWritable(); + const textRewriter: TextRewriterTransform = new TextRewriterTransform({ + destination: stderrText, + removeColors: true, + normalizeNewlines: NewlineKind.OsDefault + }); + for (const chunk of chunks) { + if (chunk.kind === TerminalChunkKind.Stderr) { + textRewriter.writeChunk(chunk); + } + } + textRewriter.close(); + return stderrText.text; +} + async function restoreFromLogFile(terminal: ITerminal, path: string): Promise { let logReadStream: fs.ReadStream | undefined; diff --git a/libraries/rush-lib/src/logic/operations/test/OperationMetadataManager.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationMetadataManager.test.ts index 6443e873d6..19025b7040 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationMetadataManager.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationMetadataManager.test.ts @@ -7,10 +7,10 @@ jest.mock('node:fs'); import { MockWritable, StringBufferTerminalProvider, Terminal, TerminalChunkKind } from '@rushstack/terminal'; import type { IPhase } from '../../../api/CommandLineConfiguration'; import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; -import { OperationMetadataManager } from '../OperationMetadataManager'; +import { type IOperationMetaData, OperationMetadataManager } from '../OperationMetadataManager'; import { CollatedTerminalProvider } from '../../../utilities/CollatedTerminalProvider'; import { CollatedTerminal } from '@rushstack/stream-collator'; -import { FileSystem } from '@rushstack/node-core-library'; +import { FileSystem, type IFileSystemCopyFileOptions, NewlineKind, Text } from '@rushstack/node-core-library'; import * as fs from 'node:fs'; import { Readable } from 'node:stream'; import { Operation } from '../Operation'; @@ -32,11 +32,24 @@ const manager: OperationMetadataManager = new OperationMetadataManager({ operation }); +const cachedErrorLogPath: string = '/path/to/project/.rush/temp/operation/identifier/error.log'; + +function createNotExistError(): NodeJS.ErrnoException { + return Object.assign(new Error('ENOENT: no such file or directory'), { + code: 'ENOENT', + errno: -2, + syscall: 'copyfile', + path: '/path/to/file' + }); +} + describe(OperationMetadataManager.name, () => { let mockTerminalProvider: StringBufferTerminalProvider; beforeEach(() => { mockTerminalProvider = new StringBufferTerminalProvider(false); jest.spyOn(FileSystem, 'copyFileAsync').mockResolvedValue(); + jest.spyOn(FileSystem, 'deleteFileAsync').mockResolvedValue(); + jest.spyOn(FileSystem, 'writeFileAsync').mockResolvedValue(); }); function toJsonLines(data: object[]): string { @@ -134,4 +147,149 @@ describe(OperationMetadataManager.name, () => { expect(mockClose).toHaveBeenCalledTimes(1); expect(mockWritable.chunks).toMatchSnapshot(); }); + + it('should write the error log from the stderr chunks instead of copying the cached error log', async () => { + const data = [ + { + text: 'logged to stdout\n', + kind: TerminalChunkKind.Stdout + }, + { + text: '\u001b[31merror TS2345: first\u001b[39m\r\n', + kind: TerminalChunkKind.Stderr + }, + { + text: 'more stdout\n', + kind: TerminalChunkKind.Stdout + }, + { + text: 'error TS2304: second\n', + kind: TerminalChunkKind.Stderr + } + ]; + + jest.spyOn(FileSystem, 'readFileAsync').mockResolvedValue(toJsonLines(data)); + + await manager.tryRestoreAsync({ + terminal: mockTerminal, + terminalProvider: mockTerminalProvider, + errorLogPath: '/path/to/errorLog' + }); + + expect(FileSystem.writeFileAsync).toHaveBeenCalledTimes(1); + expect(FileSystem.writeFileAsync).toHaveBeenCalledWith( + '/path/to/errorLog', + Text.convertTo('error TS2345: first\nerror TS2304: second\n', NewlineKind.OsDefault), + { ensureFolderExists: true } + ); + expect(FileSystem.copyFileAsync).not.toHaveBeenCalled(); + expect(FileSystem.deleteFileAsync).not.toHaveBeenCalled(); + }); + + it('should delete the error log when the chunks have no stderr, even if the cache entry has an error log', async () => { + // An entry saved after a failed run could hold that run's error log, even though its own run wrote no stderr + const data = [ + { + text: 'built without errors\n', + kind: TerminalChunkKind.Stdout + } + ]; + + jest.spyOn(FileSystem, 'readFileAsync').mockResolvedValue(toJsonLines(data)); + + await manager.tryRestoreAsync({ + terminal: mockTerminal, + terminalProvider: mockTerminalProvider, + errorLogPath: '/path/to/errorLog' + }); + + expect(FileSystem.deleteFileAsync).toHaveBeenCalledTimes(1); + expect(FileSystem.deleteFileAsync).toHaveBeenCalledWith('/path/to/errorLog'); + expect(FileSystem.copyFileAsync).not.toHaveBeenCalled(); + expect(FileSystem.writeFileAsync).not.toHaveBeenCalled(); + }); + + describe('without log chunks', () => { + beforeEach(() => { + jest.spyOn(FileSystem, 'readFileAsync').mockRejectedValue(createNotExistError()); + const mockReadStream: fs.ReadStream = Readable.from([]) as fs.ReadStream; + mockReadStream.close = jest.fn(); + jest.spyOn(fs, 'createReadStream').mockReturnValue(mockReadStream); + }); + + it('should copy the cached error log', async () => { + await manager.tryRestoreAsync({ + terminal: mockTerminal, + terminalProvider: mockTerminalProvider, + errorLogPath: '/path/to/errorLog' + }); + + expect(FileSystem.copyFileAsync).toHaveBeenCalledTimes(1); + expect(FileSystem.copyFileAsync).toHaveBeenCalledWith({ + sourcePath: cachedErrorLogPath, + destinationPath: '/path/to/errorLog' + }); + expect(FileSystem.deleteFileAsync).not.toHaveBeenCalled(); + expect(FileSystem.writeFileAsync).not.toHaveBeenCalled(); + }); + + it('should delete the error log when the cache entry has none', async () => { + jest.spyOn(FileSystem, 'copyFileAsync').mockRejectedValue(createNotExistError()); + + await manager.tryRestoreAsync({ + terminal: mockTerminal, + terminalProvider: mockTerminalProvider, + errorLogPath: '/path/to/errorLog' + }); + + expect(FileSystem.deleteFileAsync).toHaveBeenCalledTimes(1); + expect(FileSystem.deleteFileAsync).toHaveBeenCalledWith('/path/to/errorLog'); + }); + }); + + describe('saveAsync', () => { + const metadata: IOperationMetaData = { + durationInSeconds: 1, + cobuildContextId: undefined, + cobuildRunnerId: undefined, + logPath: '/path/to/rush-logs/project.identifier.log', + errorLogPath: '/path/to/rush-logs/project.identifier.error.log', + logChunksPath: '/path/to/project/.rush/temp/chunked-rush-logs/project.identifier.chunks.jsonl' + }; + + it('should copy the log files of the run', async () => { + await manager.saveAsync(metadata); + + expect(FileSystem.copyFileAsync).toHaveBeenCalledTimes(3); + expect(FileSystem.copyFileAsync).toHaveBeenCalledWith({ + sourcePath: metadata.errorLogPath, + destinationPath: cachedErrorLogPath + }); + expect(FileSystem.deleteFileAsync).not.toHaveBeenCalled(); + }); + + it('should delete the copy an earlier run left when the run did not write an error log', async () => { + jest + .spyOn(FileSystem, 'copyFileAsync') + .mockImplementation(async ({ sourcePath }: IFileSystemCopyFileOptions) => { + if (sourcePath === metadata.errorLogPath) { + throw createNotExistError(); + } + }); + + await manager.saveAsync(metadata); + + expect(FileSystem.copyFileAsync).toHaveBeenCalledTimes(3); + expect(FileSystem.deleteFileAsync).toHaveBeenCalledTimes(1); + expect(FileSystem.deleteFileAsync).toHaveBeenCalledWith(cachedErrorLogPath); + }); + + it('should rethrow other errors', async () => { + const error: Error = new Error('EACCES: permission denied'); + jest.spyOn(FileSystem, 'copyFileAsync').mockRejectedValue(error); + + await expect(manager.saveAsync(metadata)).rejects.toThrow(error); + expect(FileSystem.deleteFileAsync).not.toHaveBeenCalled(); + }); + }); }); From 08ad2aca97a97f3b4ec83978d22baee471a58d42 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:16 +0000 Subject: [PATCH 065/265] [rush-lib] An iteration checks an operation's build-cache inputs only when it needs them Swarm integration step 50; original commit 633640c81c (merge of swarm/r07-t152 at 7e98034976). Scope: task 152. Brings r07's task 152: an executed iteration no longer computes the build-cache disabled reason for all of the all-project graph's operations (about 0.7 s at odsp-web) or splices finished records out of the queue one at a time (about 0.1 s). Second agent: m01 board 2727. ch01's e2e on ch01-sC: the cache arm's hits and misses equal --no-daemon. s16 batch C, item 5 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 86cb10cdf4) Commits folded into this step (1): - 7e98034976 [rush-lib] Check the build cache inputs of an operation only when it's needed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...ache-disabled-reason_2026-09-28-23-08.json | 11 +++ .../logic/operations/AsyncOperationQueue.ts | 11 ++- .../operations/CacheableOperationPlugin.ts | 42 +++++++++-- .../test/AsyncOperationQueue.test.ts | 46 ++++++++++++ .../test/CacheableOperationPlugin.test.ts | 71 ++++++++++++++++++- 5 files changed, 170 insertions(+), 11 deletions(-) create mode 100644 common/changes/@microsoft/rush/lazy-cache-disabled-reason_2026-09-28-23-08.json diff --git a/common/changes/@microsoft/rush/lazy-cache-disabled-reason_2026-09-28-23-08.json b/common/changes/@microsoft/rush/lazy-cache-disabled-reason_2026-09-28-23-08.json new file mode 100644 index 0000000000..97a438445b --- /dev/null +++ b/common/changes/@microsoft/rush/lazy-cache-disabled-reason_2026-09-28-23-08.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Only check the tracked files of an operation for the build cache when the operation executes, or when cobuilds need to cluster the operations, and stop scanning the execution queue once it has found an operation for each waiting iterator. This speeds up builds of long-lived operation graphs such as the Rush daemon in large workspaces when only a few operations need to run.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-lib/src/logic/operations/AsyncOperationQueue.ts b/libraries/rush-lib/src/logic/operations/AsyncOperationQueue.ts index 035eb53395..bc0add0f9c 100644 --- a/libraries/rush-lib/src/logic/operations/AsyncOperationQueue.ts +++ b/libraries/rush-lib/src/logic/operations/AsyncOperationQueue.ts @@ -108,8 +108,14 @@ export class AsyncOperationQueue const readyOperations: OperationExecutionRecord[] = []; + // Operations that were never assigned are assigned first, in the order in which they are found, so the + // scan can stop once it has found one for each waiting iterator. This method runs each time an operation + // is requested, completes or becomes ready, and the queue of a long-lived graph (such as the Rush + // daemon's) can hold thousands of operations, so scanning all of them each time would be quadratic. + let untriedReadyCount: number = 0; + // By iterating in reverse order we do less array shuffling when removing operations - for (let i: number = queue.length - 1; waitingIterators.length > 0 && i >= 0; i--) { + for (let i: number = queue.length - 1; untriedReadyCount < waitingIterators.length && i >= 0; i--) { const record: OperationExecutionRecord = queue[i]; if ( @@ -137,6 +143,9 @@ export class AsyncOperationQueue throw new Error(`Unexpected status "${record.status}" for queued operation: ${record.name}`); } else { readyOperations.push(record); + if (!timesQueued.has(record)) { + untriedReadyCount++; + } } // Otherwise operation is still waiting } diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index 0a5de47256..485ff10710 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -68,7 +68,8 @@ export interface IOperationBuildCacheContext { isCacheReadAllowed: boolean; operationBuildCache: OperationBuildCache | undefined; - cacheDisabledReason: string | undefined; + // Computed when first read. See the beforeExecuteIterationAsync tap. + readonly cacheDisabledReason: string | undefined; outputFolderNames: ReadonlyArray; cobuildLock: CobuildLock | undefined; @@ -268,10 +269,23 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { const fileHashes: ReadonlyMap | undefined = inputsSnapshot.getTrackedFileHashesForOperation(associatedProject, phaseName); - const cacheDisabledReason: string | undefined = projectConfiguration - ? projectConfiguration.getCacheDisabledReason(fileHashes.keys(), phaseName, operation.isNoOp) - : `Project does not have a ${RushConstants.rushProjectConfigFilename} configuration file, ` + - 'or one provided by a rig, so it does not support caching.'; + // Computing the reason checks each tracked file of the project, and an iteration of a long-lived graph + // (e.g. the Rush daemon) holds every operation of the workspace. It is only read for the operations + // that execute and for cobuild clustering, so it is computed once, when it is first read. + let cacheDisabledReason: string | undefined; + let isCacheDisabledReasonComputed: boolean = false; + const getCacheDisabledReason = (): string | undefined => { + if (!isCacheDisabledReasonComputed) { + cacheDisabledReason = getCacheDisabledReasonForOperation( + projectConfiguration, + fileHashes, + phaseName, + operation.isNoOp + ); + isCacheDisabledReasonComputed = true; + } + return cacheDisabledReason; + }; const outputFolderNames: string[] = [record.metadataFolderPath]; const configuredOutputFolderNames: string[] | undefined = operationSettings?.outputFolderNames; @@ -286,7 +300,7 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { // Captured even if cache writes are disabled, since a long-lived graph (e.g. the Rush daemon) must not // retain outputs that were built from input files that changed during the iteration. const inputFilesState: IInputFilesState | undefined = - !cacheDisabledReason && record.enabled + record.enabled && !getCacheDisabledReason() ? captureInputFilesState( inputsSnapshot.rootDirectory, fileHashes.keys(), @@ -301,7 +315,9 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { isCacheReadAllowed: isIncrementalBuildAllowed, operationBuildCache: undefined, outputFolderNames, - cacheDisabledReason, + get cacheDisabledReason(): string | undefined { + return getCacheDisabledReason(); + }, cobuildLock: undefined, cobuildClusterId: undefined, buildCacheTerminal: undefined, @@ -1065,3 +1081,15 @@ export function clusterOperations( } } } + +function getCacheDisabledReasonForOperation( + projectConfiguration: RushProjectConfiguration | undefined, + fileHashes: ReadonlyMap, + phaseName: string, + isNoOp: boolean +): string | undefined { + return projectConfiguration + ? projectConfiguration.getCacheDisabledReason(fileHashes.keys(), phaseName, isNoOp) + : `Project does not have a ${RushConstants.rushProjectConfigFilename} configuration file, ` + + 'or one provided by a rig, so it does not support caching.'; +} diff --git a/libraries/rush-lib/src/logic/operations/test/AsyncOperationQueue.test.ts b/libraries/rush-lib/src/logic/operations/test/AsyncOperationQueue.test.ts index e93441a49f..c9aab3e37f 100644 --- a/libraries/rush-lib/src/logic/operations/test/AsyncOperationQueue.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/AsyncOperationQueue.test.ts @@ -238,4 +238,50 @@ describe(AsyncOperationQueue.name, () => { const rEnd: IteratorResult = await queue.next(); expect(rEnd.done).toBe(true); }); + + it('keeps the order of the remaining operations when it removes finished operations', async () => { + const operations: OperationExecutionRecord[] = []; + for (let i: number = 0; i < 10; i++) { + operations.push(createRecord(`r${i}`)); + } + const queue: AsyncOperationQueue = new AsyncOperationQueue(operations, nullSort); + for (let i: number = 1; i < operations.length; i += 2) { + operations[i].status = OperationStatus.Skipped; + } + + const actualOrder: string[] = []; + for await (const operation of queue) { + actualOrder.push(operation.name); + operation.status = OperationStatus.Success; + queue.complete(operation); + } + + // Without a preference, the ready operations are assigned from the end of the queue + expect(actualOrder).toEqual(['r8', 'r6', 'r4', 'r2', 'r0']); + }); + + it('stops scanning the queue once it has found a new operation for each waiting iterator', async () => { + const a: OperationExecutionRecord = createRecord('a'); + const b: OperationExecutionRecord = createRecord('b'); + const queue: AsyncOperationQueue = new AsyncOperationQueue([a, b], nullSort); + + // The queue is scanned from its end, so "b" is found before "a" + let status: OperationStatus = a.status; + let isStatusRead: boolean = false; + Object.defineProperty(a, 'status', { + get: () => { + isStatusRead = true; + return status; + }, + set: (value: OperationStatus) => { + status = value; + } + }); + + expect((await queue.next()).value).toBe(b); + expect(isStatusRead).toBe(false); + + expect((await queue.next()).value).toBe(a); + expect(isStatusRead).toBe(true); + }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts index b0d532c5c8..9a2281d00b 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPlugin.test.ts @@ -60,6 +60,7 @@ import { MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack import type { IPhase } from '../../../api/CommandLineConfiguration'; import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; import type { BuildCacheConfiguration } from '../../../api/BuildCacheConfiguration'; +import type { CobuildConfiguration } from '../../../api/CobuildConfiguration'; import type { RushProjectConfiguration } from '../../../api/RushProjectConfiguration'; import { PhasedCommandHooks, type IOperationGraphContext } from '../../../pluginFramework/PhasedCommandHooks'; import type { IInputsSnapshot } from '../../incremental/InputsSnapshot'; @@ -117,6 +118,10 @@ interface ITestGraph { localHashes: Map; // The tracked input file hashes of each operation in the inputs snapshot, by name trackedFileHashes: Map>; + // The reason that caching is disabled for each operation, by name. Undefined if caching is allowed. + cacheDisabledReasons: Map; + // The names of the operations whose reason that caching is disabled was computed, in order + cacheDisabledReasonComputations: string[]; executions: string[]; cacheWrites: string[]; // Called when an operation executes, e.g. to save one of its input files while it executes @@ -130,12 +135,15 @@ interface ITestGraph { async function createTestGraphAsync( names: string[], rootDirectory: string = '/repo', - cacheWriteEnabled: boolean = true + cacheWriteEnabled: boolean = true, + cobuildConfiguration: CobuildConfiguration | undefined = undefined ): Promise { const executions: string[] = []; const cacheWrites: string[] = []; const localHashes: Map = new Map(); const trackedFileHashes: Map> = new Map(); + const cacheDisabledReasons: Map = new Map(); + const cacheDisabledReasonComputations: string[] = []; const operations: Map = new Map(); const projectConfigurations: Map = new Map(); let onExecute: ((name: string) => void) | undefined; @@ -147,7 +155,10 @@ async function createTestGraphAsync( projectFolder: `${rootDirectory}/${name}` } as unknown as RushConfigurationProject; projectConfigurations.set(project, { - getCacheDisabledReason: () => undefined + getCacheDisabledReason: () => { + cacheDisabledReasonComputations.push(name); + return cacheDisabledReasons.get(name); + } } as unknown as RushProjectConfiguration); const operation: Operation = new Operation({ runner: new CacheableMockRunner(name, executions, (executedName: string) => onExecute?.(executedName)), @@ -183,7 +194,7 @@ async function createTestGraphAsync( buildCacheEnabled: true, cacheWriteEnabled } as unknown as BuildCacheConfiguration, - cobuildConfiguration: undefined, + cobuildConfiguration, terminal, excludeAppleDoubleFiles: false, useDirectFileTransfersForBuildCache: false @@ -207,6 +218,8 @@ async function createTestGraphAsync( operations, localHashes, trackedFileHashes, + cacheDisabledReasons, + cacheDisabledReasonComputations, executions, cacheWrites, get onExecute(): ((name: string) => void) | undefined { @@ -218,6 +231,7 @@ async function createTestGraphAsync( executeAsync: async (workingTreeReadStartTimeMs?: number) => { executions.length = 0; cacheWrites.length = 0; + cacheDisabledReasonComputations.length = 0; const inputsSnapshot: IInputsSnapshot = { hashes: new Map(), rootDirectory, @@ -507,4 +521,55 @@ describe(CacheableOperationPlugin.name, () => { } ); }); + + describe('reason that caching is disabled', () => { + const cacheDisabledReason: string = 'Caching has been disabled for this project.'; + + it('is only computed for the operations that execute', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); + await testGraph.executeAsync(); + expect(testGraph.cacheDisabledReasonComputations).toEqual(['a', 'b', 'c']); + + testGraph.localHashes.set('c', 'c-v2'); + const result: IExecutionResult = await testGraph.executeAsync(); + + expect(getStatus(testGraph, result, 'a')).toBe(OperationStatus.Skipped); + expect(getStatus(testGraph, result, 'b')).toBe(OperationStatus.Skipped); + expect(testGraph.executions).toEqual(['c']); + expect(testGraph.cacheWrites).toEqual(['c']); + expect(testGraph.cacheDisabledReasonComputations).toEqual(['c']); + }); + + it('is computed once for each operation that executes', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c']); + testGraph.cacheDisabledReasons.set('b', cacheDisabledReason); + + const result: IExecutionResult = await testGraph.executeAsync(); + + expect(result.status).toBe(OperationStatus.Success); + expect(testGraph.executions).toEqual(['a', 'b', 'c']); + expect(testGraph.cacheWrites).toEqual(['a', 'c']); + expect(testGraph.cacheDisabledReasonComputations).toEqual(['a', 'b', 'c']); + }); + + it('is computed for every operation if cobuilds are enabled, to cluster the operations', async () => { + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b', 'c'], '/repo', true, { + cobuildFeatureEnabled: true, + cobuildContextId: undefined + } as unknown as CobuildConfiguration); + // Without a build cache, an operation does not acquire a cobuild lock + for (const name of ['a', 'b', 'c']) { + testGraph.cacheDisabledReasons.set(name, cacheDisabledReason); + } + await testGraph.executeAsync(); + + testGraph.localHashes.set('c', 'c-v2'); + const result: IExecutionResult = await testGraph.executeAsync(); + + expect(getStatus(testGraph, result, 'a')).toBe(OperationStatus.Skipped); + expect(getStatus(testGraph, result, 'b')).toBe(OperationStatus.Skipped); + expect(testGraph.executions).toEqual(['c']); + expect([...testGraph.cacheDisabledReasonComputations].sort()).toEqual(['a', 'b', 'c']); + }); + }); }); From a3d3e65a1a9ec12c832f5d05e18de045ea308349 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:16 +0000 Subject: [PATCH 066/265] [rush-lib] A public beta API lets a plugin runner run :incremental under task 106's guard Swarm integration step 51; original commit e39062a035 (merge of swarm/r06-t131-int2 at 52c86b3174). Scope: task 131. Brings r06's task 131: rush-lib exposes a @beta API so that a plugin runner such as rush-fstrace-plugin can run an `:incremental` phase under task 106's L1a guard, re-tipped on 1b51e7af08 (board 2747). Second agent: o03 board 2360 on cb0891d79d. The test-file conflict in 7b8e8c665b keeps both sides. s16 batch C, item 6 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 07ac5f0139) Commits folded into this step (1): - cb0891d79d [rush-lib] Let operation runners from plugins use the incremental execution guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...incremental-plugin-runners_2026-09-28.json | 11 ++ common/reviews/api/rush-lib.api.md | 14 ++ docs/rush/plugin-migration-guide.md | 2 + libraries/rush-lib/src/index.ts | 4 + .../src/logic/operations/IOperationRunner.ts | 33 ++++ .../operations/IncrementalExecutionState.ts | 14 +- .../operations/OperationExecutionRecord.ts | 20 +++ .../IncrementalExecutionGuardPlugin.test.ts | 158 ++++++++++++++++-- .../test/OperationExecutionRecord.test.ts | 38 +++++ 9 files changed, 280 insertions(+), 14 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-incremental-plugin-runners_2026-09-28.json diff --git a/common/changes/@microsoft/rush/rushd-incremental-plugin-runners_2026-09-28.json b/common/changes/@microsoft/rush/rushd-incremental-plugin-runners_2026-09-28.json new file mode 100644 index 0000000000..66df6fac25 --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-incremental-plugin-runners_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add `IOperationRunnerContext.getIncrementalExecutionGuard()` and `IOperationRunnerContext.reportCommandExecution()` (beta), so that an operation runner from a Rush plugin can run its incremental command in the Rush daemon under the same guard as `:incremental` scripts. Results that it reports as incremental are never written to the build cache.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 8fa1c09a04..2a09f55a8f 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -618,6 +618,12 @@ export interface IGlobalCommand extends IRushCommand { setHandled(): void; } +// @beta +export interface IIncrementalExecutionGuard { + getBlockReasonAsync(): Promise; + verifyIncrementalResultAsync(): Promise; +} + // @public export interface IIndividualVersionJson extends IVersionPolicyJson { // (undocumented) @@ -714,6 +720,12 @@ export interface _IOperationChildProcessReporter { readonly stdio: child_process.StdioOptions; } +// @beta +export interface IOperationCommandExecution { + readonly hasIncrementalCommand: boolean; + readonly kind: 'initial' | 'incremental'; +} + // @alpha export interface IOperationExecutionResult extends IBaseOperationExecutionResult, IOperationLastState { readonly enabled: boolean; @@ -845,10 +857,12 @@ export interface IOperationRunnerContext { debugMode: boolean; environment: IEnvironment | undefined; error?: Error; + getIncrementalExecutionGuard?(): IIncrementalExecutionGuard | undefined; getInvalidateCallback(): (reason: string) => void; // @internal _operationMetadataManager: _OperationMetadataManager; quietMode: boolean; + reportCommandExecution?(execution: IOperationCommandExecution): void; runWithTerminalAsync(callback: (terminal: ITerminal, terminalProvider: ITerminalProvider, structuredChildOutputTerminalProvider: ITerminalProvider) => Promise, options: { createLogFile: boolean; logFileSuffix?: string; diff --git a/docs/rush/plugin-migration-guide.md b/docs/rush/plugin-migration-guide.md index 5ea09366bb..961a2432f2 100644 --- a/docs/rush/plugin-migration-guide.md +++ b/docs/rush/plugin-migration-guide.md @@ -322,6 +322,8 @@ class MyRunner implements IOperationRunner { This is how `ShellOperationRunner` selects between the `initialCommand` and `incrementalCommand` scripts defined in `rush-project.json`. +Outside watch mode, the Rush daemon can also give a runner a `lastState`, but `ShellOperationRunner` runs the incremental command there only when the operation's incremental execution guard allows it. A custom runner can use the same guard through `context.getIncrementalExecutionGuard?.()` and `context.reportCommandExecution?.()` (beta); the documentation of `getIncrementalExecutionGuard` gives the steps. Rush treats the outputs of a runner that doesn't report its commands as the outputs of its initial command, and may write them to the build cache, so such a runner must not run its incremental command outside watch mode. + > **Note:** `lastState` is only populated if the operation reached a completed terminal state (`Success`, `SuccessWithWarning`, `Failure`, `FromCache`, or `NoOp`) in a prior iteration. If the previous iteration was aborted before the operation began executing, or if the operation was `Skipped` or `Blocked`, `lastState` will still be `undefined` on the next call. Runners must not assume that a non-`undefined` `lastState` means the previous run succeeded — check `lastState.status` if the prior outcome matters for your incremental logic. > > To request re-execution from a long-lived runner, use `context.getInvalidateCallback()` on the `IOperationRunnerContext` to obtain a `(reason: string) => void` callback. This is available from the very first call to `executeAsync`, regardless of whether a previous result exists. diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index 431949409e..f9fa44afde 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -164,6 +164,10 @@ export type { IOperationRunnerContext, IOperationLastState } from './logic/operations/IOperationRunner'; +export type { + IIncrementalExecutionGuard, + IOperationCommandExecution +} from './logic/operations/IncrementalExecutionState'; export type { IConfigurableOperation, IBaseOperationExecutionResult, diff --git a/libraries/rush-lib/src/logic/operations/IOperationRunner.ts b/libraries/rush-lib/src/logic/operations/IOperationRunner.ts index f422083d85..b880671778 100644 --- a/libraries/rush-lib/src/logic/operations/IOperationRunner.ts +++ b/libraries/rush-lib/src/logic/operations/IOperationRunner.ts @@ -9,6 +9,7 @@ import type { OperationMetadataManager } from './OperationMetadataManager'; import type { IStopwatchResult } from '../../utilities/Stopwatch'; import type { IEnvironment } from '../../utilities/Utilities'; import type { IOperationChildProcessReporter } from './OperationEventSink'; +import type { IIncrementalExecutionGuard, IOperationCommandExecution } from './IncrementalExecutionState'; /** * A snapshot of a previous operation execution, passed to runners to inform incremental behavior. @@ -90,6 +91,38 @@ export interface IOperationRunnerContext { */ getInvalidateCallback(): (reason: string) => void; + /** + * Returns the guard that decides whether this operation may run its incremental command outside watch mode, or + * `undefined` if nothing guards incremental commands in this process, for example outside the Rush daemon. + * + * @remarks + * It is optional because older versions of Rush do not have it. A runner that has an incremental command and runs + * outside watch mode follows the same steps as Rush's own shell command runner: + * + * 1. If it has no last state, or there is no guard, it runs its initial command. + * + * 2. If `getBlockReasonAsync()` returns a reason, it writes + * `Not using the incremental command because .` and runs its initial command. + * + * 3. Otherwise it runs its incremental command. If that succeeds and `verifyIncrementalResultAsync()` returns a + * reason, it writes `Running the initial command, because .` and runs its initial command. + * + * If either method rejects, the runner treats the error like a reason. Before it starts each command, the runner + * calls {@link IOperationRunnerContext.reportCommandExecution}. The guard never allows the incremental command of an + * operation whose runner did not report the command of its last successful run. + */ + getIncrementalExecutionGuard?(): IIncrementalExecutionGuard | undefined; + + /** + * Records which command the runner is about to execute for this operation. Call it before starting each command, + * so that the outputs of a command that fails or is aborted are attributed to it too. + * + * @remarks + * It is optional because older versions of Rush do not have it. The outputs of an incremental command, and of + * operations built against them, are never written to the build cache. + */ + reportCommandExecution?(execution: IOperationCommandExecution): void; + /** * Allocates a negotiated reporter channel for a child process, when enabled. * diff --git a/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts b/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts index 76e0520809..dea5f04106 100644 --- a/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts +++ b/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts @@ -20,7 +20,10 @@ export const NATIVE_COMMAND_INVALIDATION_REASON: 'native-command-completed' = 'n /** * Decides whether an operation may run its `:incremental` command outside watch mode. - * Registered per execution record by {@link IncrementalExecutionGuardPlugin}. + * The Rush daemon registers one for each execution record. Runners get it from + * {@link IOperationRunnerContext.getIncrementalExecutionGuard}. + * + * @beta */ export interface IIncrementalExecutionGuard { /** @@ -35,10 +38,17 @@ export interface IIncrementalExecutionGuard { verifyIncrementalResultAsync(): Promise; } +/** + * The name that rush-lib's own runners and plugins use for {@link IOperationCommandExecution}. + */ +export type ICommandExecution = IOperationCommandExecution; + /** * Which command an operation runner executed for an operation in one iteration. + * + * @beta */ -export interface ICommandExecution { +export interface IOperationCommandExecution { /** * The command that produced the final outputs. */ diff --git a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts index f786b97113..310c99c409 100644 --- a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts +++ b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts @@ -39,6 +39,12 @@ import { type ILogFilePaths, initializeProjectLogFilesAsync } from './ProjectLogWritable'; +import { + getIncrementalExecutionGuard, + setCommandExecution, + type IIncrementalExecutionGuard, + type IOperationCommandExecution +} from './IncrementalExecutionState'; /** * @internal @@ -307,6 +313,20 @@ export class OperationExecutionRecord implements IOperationRunnerContext, IOpera return this.#context.eventSink?.createChildProcessReporter?.(this.name, this.iterationId); } + /** + * {@inheritdoc IOperationRunnerContext.getIncrementalExecutionGuard} + */ + public getIncrementalExecutionGuard(): IIncrementalExecutionGuard | undefined { + return getIncrementalExecutionGuard(this); + } + + /** + * {@inheritdoc IOperationRunnerContext.reportCommandExecution} + */ + public reportCommandExecution({ kind, hasIncrementalCommand }: IOperationCommandExecution): void { + setCommandExecution(this, { kind, hasIncrementalCommand }); + } + public get silent(): boolean { return !this.enabled || this.runner.silent; } diff --git a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts index 425705774f..a843903840 100644 --- a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts @@ -48,18 +48,26 @@ import { PassThrough } from 'node:stream'; import { LookupByPath } from '@rushstack/lookup-by-path'; import { SubprocessTerminator } from '@rushstack/node-core-library'; -import { MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; +import { type ITerminal, MockWritable, StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; import type { IPhase } from '../../../api/CommandLineConfiguration'; import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; import type { IOperationSettings, RushProjectConfiguration } from '../../../api/RushProjectConfiguration'; +import type { + IIncrementalExecutionGuard, + IOperationCommandExecution, + IOperationLastState, + IOperationRunner, + IOperationRunnerContext +} from '../../../index'; import { PhasedCommandHooks, type IOperationGraphContext } from '../../../pluginFramework/PhasedCommandHooks'; import { Utilities } from '../../../utilities/Utilities'; import { InputsSnapshot, type IInputsSnapshotProjectMetadata } from '../../incremental/InputsSnapshot'; import { IncrementalExecutionGuardPlugin } from '../IncrementalExecutionGuardPlugin'; import { INPUTS_CHANGED_INVALIDATION_REASON, - NATIVE_COMMAND_INVALIDATION_REASON + NATIVE_COMMAND_INVALIDATION_REASON, + wasExecutedIncrementally } from '../IncrementalExecutionState'; import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; import { LegacySkipPlugin } from '../LegacySkipPlugin'; @@ -110,6 +118,11 @@ interface IProjectSpec { * Files outside of the project that its build depends on, like `dependsOnAdditionalFiles` in rush-project.json */ readonly additionalFiles?: ReadonlyArray; + /** + * If set, the operation runs in a runner like that of a Rush plugin, which uses only the public API of Rush and runs + * the build itself. An `unreported` runner asks the guard, but never reports which command it runs. + */ + readonly pluginRunner?: 'reported' | 'unreported'; } interface IWorkspaceOptions { @@ -211,6 +224,73 @@ function build(projectFolder: string, isBundle: boolean, isIncremental: boolean) return 0; } +/** + * Runs the build itself, like the runner of a Rush plugin, and uses only the public API of Rush. It follows the steps + * that the documentation of `IOperationRunnerContext.getIncrementalExecutionGuard` describes. + */ +class PluginOperationRunner implements IOperationRunner { + public readonly name: string; + public readonly cacheable: boolean = true; + public readonly reportTiming: boolean = true; + public readonly silent: boolean = false; + public readonly warningsAreAllowed: boolean = false; + readonly #runBuild: (kind: IOperationCommandExecution['kind']) => number | undefined; + readonly #reportsCommandExecutions: boolean; + + public constructor( + name: string, + runBuild: (kind: IOperationCommandExecution['kind']) => number | undefined, + reportsCommandExecutions: boolean + ) { + this.name = name; + this.#runBuild = runBuild; + this.#reportsCommandExecutions = reportsCommandExecutions; + } + + public async executeAsync( + context: IOperationRunnerContext, + lastState?: IOperationLastState + ): Promise { + return await context.runWithTerminalAsync( + async (terminal: ITerminal): Promise => { + const guard: IIncrementalExecutionGuard | undefined = lastState + ? context.getIncrementalExecutionGuard?.() + : undefined; + if (!guard) { + return this.#run(context, 'initial'); + } + const blockReason: string | undefined = await guard.getBlockReasonAsync(); + if (blockReason !== undefined) { + terminal.writeLine(`Not using the incremental command because ${blockReason}.`); + return this.#run(context, 'initial'); + } + const status: OperationStatus = this.#run(context, 'incremental'); + if (status !== OperationStatus.Success) { + return status; + } + const rerunReason: string | undefined = await guard.verifyIncrementalResultAsync(); + if (rerunReason === undefined) { + return status; + } + terminal.writeLine(`Running the initial command, because ${rerunReason}.`); + return this.#run(context, 'initial'); + }, + { createLogFile: false } + ); + } + + public getConfigHash(): string { + return INITIAL_COMMAND; + } + + #run(context: IOperationRunnerContext, kind: IOperationCommandExecution['kind']): OperationStatus { + if (this.#reportsCommandExecutions) { + context.reportCommandExecution?.({ kind, hasIncrementalCommand: true }); + } + return this.#runBuild(kind) === 0 ? OperationStatus.Success : OperationStatus.Failure; + } +} + async function createWorkspaceAsync( projectSpecs: ReadonlyArray, { hasPassThroughPhase, hasLegacySkipDetection }: IWorkspaceOptions = {} @@ -325,21 +405,27 @@ async function createWorkspaceAsync( outputFolderByPrefix.set(name, outputFolderName); specByFolder.set(projectFolder, spec); + const runBuild = (kind: IOperationCommandExecution['kind']): number | undefined => { + commands.push(`${name}:${kind}`); + return build(projectFolder, !!isBundle, kind === 'incremental'); + }; const operation: Operation = new Operation({ phase: buildPhase, project, settings, logFilenameIdentifier: '_phase_build', - runner: new ShellOperationRunner({ - phase: buildPhase, - rushProject: project, - displayName: name, - initialCommand: INITIAL_COMMAND, - incrementalCommand: INCREMENTAL_COMMAND, - incrementalCommandRequiresGuard: true, - commandForHash: INITIAL_COMMAND, - ignoredParameterValues: [] - }) + runner: spec.pluginRunner + ? new PluginOperationRunner(name, runBuild, spec.pluginRunner === 'reported') + : new ShellOperationRunner({ + phase: buildPhase, + rushProject: project, + displayName: name, + initialCommand: INITIAL_COMMAND, + incrementalCommand: INCREMENTAL_COMMAND, + incrementalCommandRequiresGuard: true, + commandForHash: INITIAL_COMMAND, + ignoredParameterValues: [] + }) }); let dependent: Operation = operation; if (hasPassThroughPhase) { @@ -848,4 +934,52 @@ describe(IncrementalExecutionGuardPlugin.name, () => { 'Not using the incremental command because its outputs were not built by a successful run of its own command in this process.' ); }); + + describe('with a runner that uses the public API, like that of a Rush plugin', () => { + const wasIterationIncremental = (workspace: ITestWorkspace, iteration: ITestIteration): boolean => + wasExecutedIncrementally(iteration.result.operationResults.get(workspace.operations.get('a')!)!); + + it('runs its incremental command when the guard allows it, and its initial command otherwise', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a', pluginRunner: 'reported' }]); + expect((await workspace.executeAsync()).commands).toEqual(['a:initial']); + + workspace.writeFile('a/src/one.ts', 'one 2'); + const edited: ITestIteration = await workspace.executeAsync(); + expect(edited.commands).toEqual(['a:incremental']); + expect(edited.getStatus('a')).toBe(OperationStatus.Success); + // So the build cache and legacy skip detection ignore its outputs. + expect(wasIterationIncremental(workspace, edited)).toBe(true); + + workspace.writeFile('a/src/three.ts', 'three'); + const added: ITestIteration = await workspace.executeAsync(); + expect(added.commands).toEqual(['a:initial']); + expect(added.output).toContain( + 'Not using the incremental command because input files were added, deleted or renamed ("a/src/three.ts").' + ); + expect(wasIterationIncremental(workspace, added)).toBe(false); + + workspace.writeFile('a/src/one.ts', 'one emit:chunk'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental', 'a:initial']); + expect(changed.output).toContain( + 'Running the initial command, because the incremental command changed which output files it has: 1 added ("lib/chunk.js").' + ); + expect(wasIterationIncremental(workspace, changed)).toBe(false); + }); + + it('never runs its incremental command if it does not report which command it runs', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([ + { name: 'a', pluginRunner: 'unreported' } + ]); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'one 2'); + const edited: ITestIteration = await workspace.executeAsync(); + expect(edited.commands).toEqual(['a:initial']); + expect(edited.output).toContain( + 'Not using the incremental command because its outputs were not built by a successful run of its own command in this process.' + ); + expect(wasIterationIncremental(workspace, edited)).toBe(false); + }); + }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts index 6919cc704b..25d66cf0c7 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts @@ -6,6 +6,12 @@ import type { RushConfigurationProject } from '../../../api/RushConfigurationPro import type { IOperationSettings } from '../../../api/RushProjectConfiguration'; import { Operation } from '../Operation'; import { type IOperationExecutionRecordContext, OperationExecutionRecord } from '../OperationExecutionRecord'; +import { + getCommandExecution, + setIncrementalExecutionGuard, + wasExecutedIncrementally, + type IIncrementalExecutionGuard +} from '../IncrementalExecutionState'; import { MockOperationRunner } from './MockOperationRunner'; const MOCK_PHASE: IPhase = { @@ -125,4 +131,36 @@ describe(OperationExecutionRecord.name, () => { expect(record.weight).toBe(2); }); }); + + describe('incremental execution', () => { + it('returns the incremental execution guard that a plugin registered for the record', () => { + const operation: Operation = createOperation({ project: createProject('project-guarded') }); + const record: OperationExecutionRecord = createRecord(operation); + expect(record.getIncrementalExecutionGuard()).toBeUndefined(); + + const guard: IIncrementalExecutionGuard = { + getBlockReasonAsync: async () => undefined, + verifyIncrementalResultAsync: async () => undefined + }; + setIncrementalExecutionGuard(record, guard); + expect(record.getIncrementalExecutionGuard()).toBe(guard); + // The guard belongs to the record of one iteration, not to the operation. + expect(createRecord(operation).getIncrementalExecutionGuard()).toBeUndefined(); + }); + + it('records the command execution that the runner reports', () => { + const record: OperationExecutionRecord = createRecord( + createOperation({ project: createProject('project-reported') }) + ); + expect(getCommandExecution(record)).toBeUndefined(); + + record.reportCommandExecution({ kind: 'incremental', hasIncrementalCommand: true }); + expect(getCommandExecution(record)).toEqual({ kind: 'incremental', hasIncrementalCommand: true }); + expect(wasExecutedIncrementally(record)).toBe(true); + + // The last report counts, e.g. when the guard made the runner run its initial command after the incremental one. + record.reportCommandExecution({ kind: 'initial', hasIncrementalCommand: true }); + expect(wasExecutedIncrementally(record)).toBe(false); + }); + }); }); From 8e8621dbcda155918a9779cfa1dc199294616b1b Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 03:06:16 +0000 Subject: [PATCH 067/265] [rush-daemon] Answer a usage error only from a session that a build bound and checked Swarm integration step 52; original commit e12c8f3632 (merge of swarm/r04-t78-nit1 at 86a46e1d75). Scope: task 78 NIT 1. Brings r04's fix for o04's NIT 1 on task 78 (board 2801): the daemon answers a usage error only from a session that a build bound and checked. Second agent: o04 board 2948. s16 batch C, item 7 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 1af262606d) Commits folded into this step (1): - 86a46e1d75 Answer a usage error only from a session that a build bound and checked (task 78 follow-up, o04 board 2801 NIT 1) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...tartup-configuration_2026-09-29-01-40.json | 11 +++++ .../src/WorkspaceRequestLifecycle.ts | 22 ++++++--- .../src/test/DaemonGraphTestFixture.ts | 3 ++ .../src/test/DaemonRequestUsageError.test.ts | 48 ++++++++++++++++++- 4 files changed, 77 insertions(+), 7 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t78-startup-configuration_2026-09-29-01-40.json diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t78-startup-configuration_2026-09-29-01-40.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t78-startup-configuration_2026-09-29-01-40.json new file mode 100644 index 0000000000..2aa6af259f --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t78-startup-configuration_2026-09-29-01-40.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "An invalid build or rebuild command line that reaches the daemon before its first build now runs in-process, as it does after a configuration change. The configuration that the daemon loads when it starts can miss an edit made while it starts, for example to experiments.json, so only a build checks it.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index 156664e949..a0e615a4e8 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -551,7 +551,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { commandIdentity = await getCommandParameterIdentityAsync( this.#resolver, { envelope, workspaceSession: session, abortSignal: client.abortSignal }, - tier + this.#isConfigurationCurrent(session, tier) ); if (tier === WorkspaceInputChangeTier.Reuse) { projectFingerprint = await this.#tryCaptureProjectFingerprintAsync(session, receivedTimeMs); @@ -642,7 +642,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { commandIdentity = await getCommandParameterIdentityAsync( this.#resolver, { envelope, workspaceSession: session, abortSignal: client.abortSignal }, - tier + this.#isConfigurationCurrent(session, tier) ); if ( this.#boundSession === session && @@ -809,6 +809,16 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } } + /** + * Whether the session's configuration is known to match the workspace inputs: a reload bound this session after + * checking that its inputs did not change while it loaded, and they have not changed since. The session that the + * daemon loads at startup is not checked that way. An edit made after it loaded, while the startup capture runs, + * is in the startup fingerprint but not in the session. + */ + #isConfigurationCurrent(session: IWorkspaceSession, tier: WorkspaceInputChangeTier): boolean { + return tier === WorkspaceInputChangeTier.Reuse && this.#boundSession === session && !this.#forceReload; + } + #captureAsync( session: IWorkspaceSession, envelope: IDaemonRequestEnvelope, @@ -1129,19 +1139,19 @@ function getResolverLifecycle(resolver: IDaemonRequestResolver): IWorkspaceResol /** * Parses the command with the session's configuration. A command line that this configuration rejects fails as - * native Rush would, unless the configuration changed since the session loaded it (`tier` is not - * `WorkspaceInputChangeTier.Reuse`): native Rush might accept the command line then, so the client runs it + * native Rush would only if the configuration is current (`isConfigurationCurrent`). Otherwise native Rush might + * accept the command line, for example with a parameter that experiments.json adds, so the client runs it * in-process instead. */ async function getCommandParameterIdentityAsync( resolver: IDaemonRequestResolver, options: IResolveDaemonRequestOptions, - tier: WorkspaceInputChangeTier + isConfigurationCurrent: boolean ): Promise { try { return await getResolverLifecycle(resolver).getCommandParameterIdentityAsync(options); } catch (error) { - if (error instanceof DaemonRequestUsageError && tier !== WorkspaceInputChangeTier.Reuse) { + if (error instanceof DaemonRequestUsageError && !isConfigurationCurrent) { throw new DaemonRequestDispatchError('unsupported', error.message, { cause: error }); } throw error; diff --git a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts index d1c90fb8e2..bd78e3072a 100644 --- a/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts +++ b/libraries/rush-daemon/src/test/DaemonGraphTestFixture.ts @@ -41,6 +41,8 @@ export class DaemonGraphTestFixture implements AsyncDisposable { public readonly logs: string[] = []; /** Awaited before each workspace session is created, including a request's graph load or reload. */ public beforeCreateSessionAsync: (() => Promise) | undefined; + /** Awaited after each workspace session is created, before the host or the request that created it uses it. */ + public afterCreateSessionAsync: (() => Promise) | undefined; /** Also serves rushx package scripts, like the production host. Set it in `createAsync`'s `configure`. */ public servesRushx: boolean = false; public readonly folder: string = fs.realpathSync.native( @@ -170,6 +172,7 @@ export class DaemonGraphTestFixture implements AsyncDisposable { createWorkspaceSessionAsync: async (options) => { await this.beforeCreateSessionAsync?.(); this.session = await WorkspaceSession.createAsync(options); + await this.afterCreateSessionAsync?.(); return this.session; } }); diff --git a/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts b/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts index 1c2966e172..c333f2b618 100644 --- a/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts +++ b/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts @@ -17,10 +17,15 @@ const usageFailure: object = { const inProcess: object = { kind: 'requestRejected', payload: { code: 'unsupported', message } }; const success: object = { kind: 'requestResult', payload: { exitCode: 0 } }; +function experimentsJson(useIPCScriptsInWatchMode: boolean): string { + return JSON.stringify({ useIPCScriptsInWatchMode }); +} + it('fails an invalid command line with the exit code of native Rush instead of handing it to in-process Rush', async () => { const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); try { - expect((await fixture.runAsync(invalid)).terminal).toMatchObject(usageFailure); + // Until a build binds a session, the daemon has not checked the configuration that it loaded at startup. + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(inProcess); expect((await fixture.buildAsync()).terminal).toMatchObject(success); const generation: number = fixture.host.workspaceGeneration; const graph: IOperationGraph | undefined = fixture.session.operationGraph; @@ -59,3 +64,44 @@ it('hands an invalid command line to in-process Rush after a configuration chang await fixture[Symbol.asyncDispose](); } }); + +it('hands an invalid command line to in-process Rush when the configuration changed while the daemon started', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => { + // A build that has watch phases gets --no-ipc when experiments.json turns on useIPCScriptsInWatchMode. + created.write( + 'common/config/rush/command-line.json', + JSON.stringify({ + phases: [{ name: '_phase:compile', dependencies: { upstream: ['_phase:compile'] } }], + commands: [ + { + commandKind: 'phased', + name: 'build', + phases: ['_phase:compile'], + incremental: true, + enableParallelism: true, + watchOptions: { alwaysWatch: false, watchPhases: ['_phase:compile'] } + } + ] + }) + ); + created.write('common/config/rush/experiments.json', experimentsJson(false)); + created.afterCreateSessionAsync = async () => { + created.afterCreateSessionAsync = undefined; + // The startup capture reads this edit, but the session that the daemon loaded read experiments.json before it. + created.write('common/config/rush/experiments.json', experimentsJson(true)); + }; + }); + try { + const noIpc: string[] = ['build', '--to', 'b', '--no-ipc']; + expect((await fixture.runAsync(noIpc)).terminal).toMatchObject({ + kind: 'requestRejected', + payload: { code: 'unsupported', message: 'rush build: error: Unrecognized arguments: --no-ipc.' } + }); + // The first build binds a session that reads the edited experiments.json. + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + expect((await fixture.runAsync(noIpc)).terminal).toMatchObject(success); + expect(fixture.runs()).toEqual(['a', 'b']); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); From f5290071ccbbef5512db1af217802e0faaf5c191 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:05 +0000 Subject: [PATCH 068/265] [node-core-library] Test the edges of LockFile's time zone check Swarm integration step 53; original commit b122243826 (merge of o07/lockfile-tzrule-nits at 48fbc7ee88). Scope: task 121 NITs. Brings o07's fixes for ch01's 4 test NITs on task 121 (board 2972): tests, one doc sentence and a type-none change file. The branch is in , fetched read-only for this merge. s17 batch D1, item 1 of 10 (ch01 board 3310). Gate: ch01 GATE OK board 3104 and board 3310 (tree 20d42b41b6) Commits folded into this step (1): - 48fbc7ee88 [node-core-library] Test the edges of LockFile's time zone check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../lockfile-time-zone-tests_2026-09-29.json | 10 +++++++ libraries/node-core-library/src/LockFile.ts | 4 +++ .../src/test/LockFile.test.ts | 28 ++++++++++++++++++- 3 files changed, 41 insertions(+), 1 deletion(-) create mode 100644 common/changes/@rushstack/node-core-library/lockfile-time-zone-tests_2026-09-29.json diff --git a/common/changes/@rushstack/node-core-library/lockfile-time-zone-tests_2026-09-29.json b/common/changes/@rushstack/node-core-library/lockfile-time-zone-tests_2026-09-29.json new file mode 100644 index 0000000000..abd6bb1749 --- /dev/null +++ b/common/changes/@rushstack/node-core-library/lockfile-time-zone-tests_2026-09-29.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/node-core-library", + "comment": "", + "type": "none" + } + ], + "packageName": "@rushstack/node-core-library" +} diff --git a/libraries/node-core-library/src/LockFile.ts b/libraries/node-core-library/src/LockFile.ts index 881287ff34..bb113c8f91 100644 --- a/libraries/node-core-library/src/LockFile.ts +++ b/libraries/node-core-library/src/LockFile.ts @@ -613,6 +613,10 @@ const MAX_TIME_ZONE_OFFSET_MS: number = 14 * 60 * 60 * 1000; * On Linux, the start time that "ps" reports for a process moves with the system clock. So if the clock is * changed by more than START_TIME_TOLERANCE_MS while a process holds a lock, this can return false for the * lockfile of that process, which is then treated as stale. + * + * A custom POSIX TZ string can set an offset that no time zone has, such as TZ=XYZ-5:07. This returns false + * for a start time written with such an offset, so a process with another time zone treats that lockfile as + * stale. */ function _isStartTimeInSomeTimeZone(lockFileStartTime: string, startTimeMs: number): boolean { const lockFileStartTimeMs: number | undefined = _parseLstartAsUtcMs(lockFileStartTime); diff --git a/libraries/node-core-library/src/test/LockFile.test.ts b/libraries/node-core-library/src/test/LockFile.test.ts index bec5b86e03..1925c93e27 100644 --- a/libraries/node-core-library/src/test/LockFile.test.ts +++ b/libraries/node-core-library/src/test/LockFile.test.ts @@ -724,7 +724,10 @@ describe(LockFile.name, () => { test.each<[string, string, number]>([ ['5 hours and 45 minutes ahead of UTC', '20', (5 * 60 + 45) * 60 * 1000], ['12 hours behind UTC', '21', -12 * 60 * 60 * 1000], - ['7 hours behind UTC, printed 2 seconds off', '22', -7 * 60 * 60 * 1000 + 2000] + ['7 hours behind UTC, printed 2 seconds off', '22', -7 * 60 * 60 * 1000 + 2000], + ['7 hours behind UTC, printed 2 seconds early', '31', -7 * 60 * 60 * 1000 - 2000], + ['14 hours ahead of UTC, printed 3 seconds late', '32', 14 * 60 * 60 * 1000 + 3000], + ['12 hours behind UTC, printed 3 seconds early', '33', -12 * 60 * 60 * 1000 - 3000] ])( 'cannot acquire a lock if the other process wrote its start time in a time zone %s', (description: string, folderName: string, offsetMs: number) => { @@ -743,6 +746,7 @@ describe(LockFile.name, () => { test.each<[string, string, number]>([ ['7 minutes after', '23', 7 * 60 * 1000], + ['15 hours after', '34', 15 * 60 * 60 * 1000], ['25 hours after', '24', 25 * 60 * 60 * 1000], ['13 hours before', '25', -13 * 60 * 60 * 1000] ])( @@ -1098,6 +1102,28 @@ describe(LockFile.name, () => { expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [otherPids[0]]]); }); + test('keeps a lockfile with a start time from another time zone if /proc/[pid]/stat cannot be read', () => { + const testFolder: string = path.join(libTestFolder, '35'); + const otherPids: number[] = startProcesses(1); + // UTC+5:45, or UTC-12 if that is the local time zone + const otherTimeZone: string = new Date().getTimezoneOffset() === -345 ? 'XYZ+12' : 'XYZ-05:45'; + const otherPidStartTime: string = getLstartWithCLocale(otherPids[0], otherTimeZone); + const otherPidLockFileNames: string[] = createNewerLockFiles( + testFolder, + otherPids, + () => otherPidStartTime + ); + mockReadFile(`/proc/${otherPids[0]}/stat`, createEaccesError()); + const getStartTimeSpy: jest.Mock = jest.fn(getProcessStartTime); + setLockFileGetProcessStartTime(getStartTimeSpy); + + expectToAcquireAndKeep(testFolder, otherPidLockFileNames); + + // "ps" printed another start time for it, so the time zone check kept the lockfile. + expect(getStartTimeSpy.mock.calls).toEqual([[process.pid], [otherPids[0]]]); + expect(getStartTimeSpy.mock.results[1].value).not.toEqual(otherPidStartTime); + }); + test('does not run "ps" for them if they wrote their start times with other time zones or locales', () => { const testFolder: string = path.join(libTestFolder, '28'); const otherPids: number[] = startProcesses(3); From ef98cd239ab7c922df905c0499c50cc9f07648b0 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:05 +0000 Subject: [PATCH 069/265] [rush-daemon-protocol] Name the variables when a command's environment restarted the daemon Swarm integration step 54; original commit 898ff4dba1 (merge of swarm/r06-t88 at 04c6fb9d5a). Scope: task 88. Brings r06's fix for task 88 (board 2886): when a request restarts the daemon, or follows a restarted one, the client prints one line naming the differing variables (names only). Second agent: t07 board 2961. s17 batch D1, item 2 of 10. Gate: ch01 GATE OK board 3310 (tree ad9aa75f8c) Commits folded into this step (3): - ffbe43c152 [rush-daemon-protocol] Add the environmentChanged restart reason (task 88) - 8afd905246 [rush-daemon] Name the variables when the daemon restarts for an environment (task 88) - 04c6fb9d5a [rush-cli-client] Name the variables when a command's environment restarted the daemon (task 88) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 9 ++ .../src/daemonRestartNotice.ts | 35 ++++- .../src/test/daemonRestartNotice.test.ts | 32 ++++- .../src/test/nativeBuild.test.ts | 9 +- ...nment-restart-reason_2026-09-29-01-35.json | 11 ++ ...nment-restart-reason_2026-09-29-01-15.json | 11 ++ ...nment-restart-reason_2026-09-29-01-30.json | 11 ++ .../reviews/api/rush-daemon-protocol.api.md | 8 +- libraries/rush-daemon-protocol/README.md | 3 + .../src/DaemonEnvironmentChange.ts | 19 +++ .../src/DaemonInstallationChange.ts | 6 +- .../src/EnvironmentChangeValidation.ts | 18 +++ .../src/InstallationChangeValidation.ts | 3 + libraries/rush-daemon-protocol/src/index.ts | 1 + .../src/test/EnvironmentChange.test.ts | 78 +++++++++++ .../src/test/InstallationChange.test.ts | 2 +- .../src/test/QueuedRestartReason.test.ts | 2 +- libraries/rush-daemon/README.md | 8 ++ .../src/EnvironmentRestartReason.ts | 39 ++++++ .../src/WorkspaceRequestAdmission.ts | 4 +- .../src/WorkspaceRequestLifecycle.ts | 55 ++++++-- .../src/test/EnvironmentRestartReason.test.ts | 62 +++++++++ .../WorkspaceEnvironmentRestartReason.test.ts | 127 ++++++++++++++++++ 23 files changed, 533 insertions(+), 20 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/r06-t88-environment-restart-reason_2026-09-29-01-35.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/r06-t88-environment-restart-reason_2026-09-29-01-15.json create mode 100644 common/changes/@rushstack/rush-daemon/r06-t88-environment-restart-reason_2026-09-29-01-30.json create mode 100644 libraries/rush-daemon-protocol/src/DaemonEnvironmentChange.ts create mode 100644 libraries/rush-daemon-protocol/src/EnvironmentChangeValidation.ts create mode 100644 libraries/rush-daemon-protocol/src/test/EnvironmentChange.test.ts create mode 100644 libraries/rush-daemon/src/EnvironmentRestartReason.ts create mode 100644 libraries/rush-daemon/src/test/EnvironmentRestartReason.test.ts create mode 100644 libraries/rush-daemon/src/test/WorkspaceEnvironmentRestartReason.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 1f4dc9e3f7..8082733f1c 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -280,6 +280,15 @@ ends, resubmits the request, and prints one line on stderr (or above the agent progress rows): `rush-client: The daemon's installation at was removed; restarted the daemon (PID ).` +When the daemon restarts for a command's environment, the client prints a line of the +same kind that names the variables that differed, never their values: +`rush-client: A command's environment differed from the daemon's in NODE_OPTIONS; +restarted the daemon (PID ).` It names at most four variables and then says how +many more differed (`A, B, C, D and 2 more`). A command that was waiting when another +command's environment restarted the daemon prints that command's variables, because the +successor starts with that command's environment. A daemon that does not name the +variables gets no line. + When the connection is lost before a command's result, the command fails with exit code 1 and is not retried. The diagnostic keeps "Daemon disconnected before delivering a result; the command was not retried." and says what happened to rushd. If its process exited (a crash, an diff --git a/apps/rush-cli-client/src/daemonRestartNotice.ts b/apps/rush-cli-client/src/daemonRestartNotice.ts index e5d3eafe93..e1e7cd643a 100644 --- a/apps/rush-cli-client/src/daemonRestartNotice.ts +++ b/apps/rush-cli-client/src/daemonRestartNotice.ts @@ -4,18 +4,43 @@ import type { IDaemonRestartNotice } from '@rushstack/rush-client-core'; import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; +/** The most variables that a restart line names before it says how many more differed. */ +const MAX_NAMED_VARIABLES: number = 4; + +/** Lists names as `A`, `A and B`, `A, B and C`, or `A, B, C, D and 2 more`. */ +function formatVariableNames(names: readonly string[]): string { + const items: string[] = names.slice(0, MAX_NAMED_VARIABLES); + if (names.length > items.length) items.push(`${names.length - items.length} more`); + const last: string | undefined = items.pop(); + return items.length === 0 ? (last ?? '') : `${items.join(', ')} and ${last}`; +} + +/** Says why the daemon restarted, or returns `undefined` for a reason that this client does not know. */ +function formatRestartedCause(reason: DaemonRestartReason | undefined): string | undefined { + switch (reason?.kind) { + case 'installationChanged': + return `The daemon's installation at ${reason.folder} was ${reason.change}`; + case 'environmentChanged': { + // Names only: a value, such as NODE_OPTIONS's, can hold a secret. + const { variableNames } = reason; + const names: string = variableNames.length === 0 ? '' : ` in ${formatVariableNames(variableNames)}`; + return `A command's environment differed from the daemon's${names}`; + } + default: + return undefined; + } +} + /** * Returns the line that tells the user why the daemon restarted during a command, or `undefined` for a restart * that needs no explanation. */ export function formatDaemonRestartNotice(notice: IDaemonRestartNotice, rushx: boolean): string | undefined { const { reason, successorPid } = notice; - if (reason?.kind !== 'installationChanged') return undefined; + const cause: string | undefined = formatRestartedCause(reason); + if (cause === undefined) return undefined; const pid: string = successorPid === undefined ? '' : ` (PID ${successorPid})`; - return ( - `${rushx ? 'rushx-client' : 'rush-client'}: The daemon's installation at ${reason.folder} was ` + - `${reason.change}; restarted the daemon${pid}.` - ); + return `${rushx ? 'rushx-client' : 'rush-client'}: ${cause}; restarted the daemon${pid}.`; } /** diff --git a/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts b/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts index 1bed6ad88f..4ca8860fe3 100644 --- a/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts +++ b/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts @@ -13,7 +13,10 @@ import { } from '../daemonRestartNotice'; // A kind that only a newer daemon knows. -const NEWER_REASON: DaemonRestartReason = { kind: 'environmentChanged' } as unknown as DaemonRestartReason; +const NEWER_REASON: DaemonRestartReason = { + kind: 'newerReason', + detail: ['NODE_OPTIONS'] +} as unknown as DaemonRestartReason; describe(formatDaemonRestartNotice.name, () => { it('names the changed installation and the new daemon', () => { @@ -41,6 +44,33 @@ describe(formatDaemonRestartNotice.name, () => { ).toBe("rushx-client: The daemon's installation at /snapshots/s9 was removed; restarted the daemon."); }); + it("names the variables in which a command's environment differed, at most four of them", () => { + const format = ( + variableNames: string[], + successorPid: number | undefined, + rushx: boolean + ): string | undefined => + formatDaemonRestartNotice( + { restart: 1, reason: { kind: 'environmentChanged', variableNames }, successorPid }, + rushx + ); + expect(format(['NODE_OPTIONS'], 42, false)).toBe( + "rush-client: A command's environment differed from the daemon's in NODE_OPTIONS; " + + 'restarted the daemon (PID 42).' + ); + expect(format(['FOO', 'NODE_OPTIONS'], undefined, true)).toBe( + "rushx-client: A command's environment differed from the daemon's in FOO and NODE_OPTIONS; " + + 'restarted the daemon.' + ); + expect(format(['A', 'B', 'C', 'D'], 42, false)).toContain("the daemon's in A, B, C and D; restarted"); + expect(format(['A', 'B', 'C', 'D', 'E', 'F'], 42, false)).toContain( + "the daemon's in A, B, C, D and 2 more; restarted" + ); + expect(format([], 42, false)).toBe( + "rush-client: A command's environment differed from the daemon's; restarted the daemon (PID 42)." + ); + }); + it('says nothing about restarts that need no explanation', () => { expect( formatDaemonRestartNotice({ restart: 1, reason: undefined, successorPid: 42 }, false) diff --git a/apps/rush-cli-client/src/test/nativeBuild.test.ts b/apps/rush-cli-client/src/test/nativeBuild.test.ts index efcbf6154a..2f02fe76b6 100644 --- a/apps/rush-cli-client/src/test/nativeBuild.test.ts +++ b/apps/rush-cli-client/src/test/nativeBuild.test.ts @@ -360,10 +360,17 @@ describe('native build through the standalone client', () => { fs.writeFileSync(path.join(folder, 'a/input.txt'), 'two'); const changed: IResult = await invokeAsync(argv); expect(changed.code).toBe(0); - expect(changed.stderr).not.toMatch(/using in-process|restart/i); + expect(changed.stderr).not.toMatch(/using in-process/i); expect(changed.stdout).toContain('built-a-two'); const after = JSON.parse((await invokeAsync(['daemon', 'status'])).stdout); expect(after.pid).not.toBe(previousPid); + // One line names the variable that differed, never its value. + expect(changed.stderr).toContain( + "rush-client: A command's environment differed from the daemon's in RUSHD_TEST_RESTART_VALUE; " + + `restarted the daemon (PID ${after.pid}).\n` + ); + expect(changed.stderr.match(/restarted the daemon/g)).toHaveLength(1); + expect(changed.stderr).not.toContain('new-process-environment'); expect(fs.readFileSync(path.join(folder, 'runs.txt'), 'utf8')).toBe('a:one\nb:one\na:two\nb:one\n'); expect((await invokeAsync(argv)).code).toBe(0); expect(JSON.parse((await invokeAsync(['daemon', 'status'])).stdout).pid).toBe(after.pid); diff --git a/common/changes/@rushstack/rush-cli-client/r06-t88-environment-restart-reason_2026-09-29-01-35.json b/common/changes/@rushstack/rush-cli-client/r06-t88-environment-restart-reason_2026-09-29-01-35.json new file mode 100644 index 0000000000..23d8a8eaad --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/r06-t88-environment-restart-reason_2026-09-29-01-35.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Print one line that names the environment variables, never their values, when a command restarted the daemon because its environment differed from the daemon's.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/r06-t88-environment-restart-reason_2026-09-29-01-15.json b/common/changes/@rushstack/rush-daemon-protocol/r06-t88-environment-restart-reason_2026-09-29-01-15.json new file mode 100644 index 0000000000..eb5ee6032c --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/r06-t88-environment-restart-reason_2026-09-29-01-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "Add the `environmentChanged` restart reason, which names the environment variables that differ between a request and the daemon, never their values. Older clients ignore it.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/r06-t88-environment-restart-reason_2026-09-29-01-30.json b/common/changes/@rushstack/rush-daemon/r06-t88-environment-restart-reason_2026-09-29-01-30.json new file mode 100644 index 0000000000..600081f64f --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r06-t88-environment-restart-reason_2026-09-29-01-30.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "When a request's environment differs from the daemon's, the restart result, each result answered while that restart is pending, and the daemon log now name the variables that differ, never their values.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index 3e1b962294..ecca06700b 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -170,7 +170,7 @@ export type DaemonRequestAdmissionErrorCode = 'aborted' | 'no-wait' | 'wait-time export type DaemonRequestRejectionCode = 'invalidRequest' | 'routingFailed' | 'unsupported' | 'workspaceRecreationRequired'; // @beta -export type DaemonRestartReason = IDaemonInstallationChangedRestartReason; +export type DaemonRestartReason = IDaemonInstallationChangedRestartReason | IDaemonEnvironmentChangedRestartReason; // @beta export type DaemonRushCommandOrigin = 'built-in' | 'custom'; @@ -260,6 +260,12 @@ export interface IDaemonDiagnosticPayload { readonly severity: DaemonDiagnosticSeverity; } +// @beta +export interface IDaemonEnvironmentChangedRestartReason { + readonly kind: 'environmentChanged'; + readonly variableNames: readonly string[]; +} + // @beta export interface IDaemonErrorMessage { // (undocumented) diff --git a/libraries/rush-daemon-protocol/README.md b/libraries/rush-daemon-protocol/README.md index d95633c5db..d4d5f8e435 100644 --- a/libraries/rush-daemon-protocol/README.md +++ b/libraries/rush-daemon-protocol/README.md @@ -58,6 +58,9 @@ The engine-agnostic **wire layer** spoken by every client of the Rush daemon (`r with cancellation, admission errors or operation results. Clients may retry once after attested predecessor ownership release, never on transport loss or an error string. The ordinary mutation result has no retry flag and drains before restart. + A retry result may say why in `restartReason`: `installationChanged` names the folder that + was removed or replaced, and `environmentChanged` names the environment variables that + differ from the daemon's, never their values. Clients ignore kinds they do not know. - **Read-only workspace status** - optional `pong.payload.workspace` reports the provider generation, installed session token, graph existence and real warm accounting. An absent token means no session is installed; an absent `warmSet` means no controller is attached, not zero memory. Warm status includes diff --git a/libraries/rush-daemon-protocol/src/DaemonEnvironmentChange.ts b/libraries/rush-daemon-protocol/src/DaemonEnvironmentChange.ts new file mode 100644 index 0000000000..7fb66ec2b5 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/DaemonEnvironmentChange.ts @@ -0,0 +1,19 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * The request's environment differs from the one that the daemon started with, in variables that the daemon cannot + * take from each request. The daemon restarts with the request's environment. + * + * @beta + */ +export interface IDaemonEnvironmentChangedRestartReason { + /** Identifies this reason. */ + readonly kind: 'environmentChanged'; + /** + * The sorted names of the variables that differ: those that only one of the two environments sets, and those that + * they set to different values. Values are never sent, because a variable such as `NODE_OPTIONS` can carry a + * secret. A daemon may send an empty list. + */ + readonly variableNames: readonly string[]; +} diff --git a/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts b/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts index 0c9112e204..b97e4f2787 100644 --- a/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts +++ b/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { IDaemonEnvironmentChangedRestartReason } from './DaemonEnvironmentChange'; + /** * How a folder of a running daemon's installation changed after the daemon started. * @@ -37,4 +39,6 @@ export interface IDaemonInstallationChangedRestartReason extends IDaemonInstalla * * @beta */ -export type DaemonRestartReason = IDaemonInstallationChangedRestartReason; +export type DaemonRestartReason = + | IDaemonInstallationChangedRestartReason + | IDaemonEnvironmentChangedRestartReason; diff --git a/libraries/rush-daemon-protocol/src/EnvironmentChangeValidation.ts b/libraries/rush-daemon-protocol/src/EnvironmentChangeValidation.ts new file mode 100644 index 0000000000..246445b086 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/EnvironmentChangeValidation.ts @@ -0,0 +1,18 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { DaemonProtocolError } from './DaemonProtocolError'; + +const EMPTY_LENGTH: number = 0; + +/** Validates the variable names of an `environmentChanged` restart reason. @internal */ +export function validateEnvironmentChange(reason: Record): void { + const { variableNames } = reason; + if (!Array.isArray(variableNames) || !variableNames.every(isVariableName)) { + throw new DaemonProtocolError('malformedControlMessage', 'Invalid restartReason.variableNames.'); + } +} + +function isVariableName(value: unknown): boolean { + return typeof value === 'string' && value.length > EMPTY_LENGTH; +} diff --git a/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts b/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts index 77884d9583..afaf7d90af 100644 --- a/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts +++ b/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts @@ -3,9 +3,11 @@ import { isDaemonControlRecord } from './ControlRecord'; import { DaemonProtocolError } from './DaemonProtocolError'; +import { validateEnvironmentChange } from './EnvironmentChangeValidation'; const INSTALLATION_CHANGE_KINDS: ReadonlySet = new Set(['removed', 'replaced']); const INSTALLATION_CHANGED: string = 'installationChanged'; +const ENVIRONMENT_CHANGED: string = 'environmentChanged'; const EMPTY_LENGTH: number = 0; /** Validates an optional installation change, as reported by pong or by a restart reason. @internal */ @@ -32,6 +34,7 @@ function validateReason(reason: unknown): void { requireRecord(reason, 'restartReason'); requireKind(reason.kind); if (reason.kind === INSTALLATION_CHANGED) validateInstallationChange(reason, 'restartReason'); + if (reason.kind === ENVIRONMENT_CHANGED) validateEnvironmentChange(reason); } function requireRecord(value: unknown, field: string): asserts value is Record { diff --git a/libraries/rush-daemon-protocol/src/index.ts b/libraries/rush-daemon-protocol/src/index.ts index 4e984fe8ef..c90228393b 100644 --- a/libraries/rush-daemon-protocol/src/index.ts +++ b/libraries/rush-daemon-protocol/src/index.ts @@ -59,6 +59,7 @@ export type { DaemonCommandOutcome, IDaemonCommandResult } from './DaemonCommand export type { DaemonInstallationChangeKind, DaemonRestartReason } from './DaemonInstallationChange'; export type { IDaemonInstallationChange } from './DaemonInstallationChange'; export type { IDaemonInstallationChangedRestartReason } from './DaemonInstallationChange'; +export type { IDaemonEnvironmentChangedRestartReason } from './DaemonEnvironmentChange'; export { MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS } from './DaemonRequestAdmission'; export { validateDaemonRequestAdmissionOptions } from './DaemonRequestAdmission'; export type { DaemonRequestRejectionCode, IDaemonRequestCancelMessage } from './DaemonRequestControl'; diff --git a/libraries/rush-daemon-protocol/src/test/EnvironmentChange.test.ts b/libraries/rush-daemon-protocol/src/test/EnvironmentChange.test.ts new file mode 100644 index 0000000000..775d594509 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/test/EnvironmentChange.test.ts @@ -0,0 +1,78 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { decodeDaemonControlMessage, encodeDaemonControlMessage } from '../ControlFrameCodec'; +import type { IDaemonCommandResult } from '../DaemonCommandResult'; +import type { DaemonControlMessage } from '../DaemonControlMessage'; +import type { IDaemonEnvironmentChangedRestartReason } from '../DaemonEnvironmentChange'; +import { WIRE_TEXT_ENCODER } from '../DaemonWireText'; + +const FAILURE_EXIT_CODE: number = 1; +const FIRST_POSITION: number = 1; +const NOT_A_NAME: number = 7; +const QUEUED_REQUEST_ID: string = 'queued-before-restart'; +const REASON: IDaemonEnvironmentChangedRestartReason = { + kind: 'environmentChanged', + variableNames: ['FOO', 'NODE_OPTIONS'] +}; +const RESULT: IDaemonCommandResult = { + aborted: false, + exitCode: FAILURE_EXIT_CODE, + outcome: 'failure', + requestId: 'environment', + retryAfterRestart: true, + restartReason: REASON +}; + +function resultFrame(restartReason: unknown): Uint8Array { + return WIRE_TEXT_ENCODER.encode( + JSON.stringify({ kind: 'requestResult', payload: { ...RESULT, restartReason } }) + ); +} + +function queuePositionFrame(restartReason: unknown): Uint8Array { + return WIRE_TEXT_ENCODER.encode( + JSON.stringify({ + kind: 'queuePosition', + payload: { position: FIRST_POSITION, requestId: QUEUED_REQUEST_ID, restartReason } + }) + ); +} + +it('round-trips a restart result that names the variables that differ', () => { + const message: DaemonControlMessage = { kind: 'requestResult', payload: RESULT }; + expect(decodeDaemonControlMessage(encodeDaemonControlMessage(message))).toEqual(message); +}); + +it('round-trips a queue position whose restart is for an environment', () => { + const message: DaemonControlMessage = { + kind: 'queuePosition', + payload: { position: FIRST_POSITION, requestId: QUEUED_REQUEST_ID, restartReason: REASON } + }; + expect(decodeDaemonControlMessage(encodeDaemonControlMessage(message))).toEqual(message); +}); + +it('accepts an environment reason that names no variables', () => { + const restartReason: object = { kind: 'environmentChanged', variableNames: [] }; + expect(decodeDaemonControlMessage(resultFrame(restartReason))).toMatchObject({ + payload: { restartReason } + }); +}); + +const MALFORMED_REASONS: object[] = [ + { kind: 'environmentChanged' }, + { kind: 'environmentChanged', variableNames: null }, + { kind: 'environmentChanged', variableNames: 'NODE_OPTIONS' }, + { kind: 'environmentChanged', variableNames: [''] }, + { kind: 'environmentChanged', variableNames: ['NODE_OPTIONS', NOT_A_NAME] } +]; + +it.each(MALFORMED_REASONS)('rejects a restart result whose environment reason is %j', (reason: object) => { + expect(() => decodeDaemonControlMessage(resultFrame(reason))).toThrow(/restartReason\.variableNames/); +}); + +it.each(MALFORMED_REASONS)('rejects a queue position whose environment reason is %j', (reason: object) => { + expect(() => decodeDaemonControlMessage(queuePositionFrame(reason))).toThrow( + /restartReason\.variableNames/ + ); +}); diff --git a/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts b/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts index c9335c9c01..380b8d09f7 100644 --- a/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts +++ b/libraries/rush-daemon-protocol/src/test/InstallationChange.test.ts @@ -37,7 +37,7 @@ it('round-trips a restart result that names the changed installation', () => { it('accepts a restart reason kind from a newer daemon', () => { const payload: object = { ...RESULT, - restartReason: { kind: 'environmentChanged', names: ['NODE_OPTIONS'] } + restartReason: { kind: 'newerReason', detail: ['NODE_OPTIONS'] } }; expect(decodeDaemonControlMessage(resultFrame(payload))).toMatchObject({ payload }); }); diff --git a/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts b/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts index 05b4a02860..b694c303a5 100644 --- a/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts +++ b/libraries/rush-daemon-protocol/src/test/QueuedRestartReason.test.ts @@ -32,7 +32,7 @@ it('round-trips a queue position that says the daemon restarts once the requests }); it('accepts a queued restart reason kind from a newer daemon', () => { - const restartReason: object = { kind: 'environmentChanged', names: ['NODE_OPTIONS'] }; + const restartReason: object = { kind: 'newerReason', detail: ['NODE_OPTIONS'] }; expect(decodeDaemonControlMessage(queuePositionFrame(restartReason))).toMatchObject({ payload: { restartReason } }); diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 5179afee7e..383e4b6676 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -202,6 +202,14 @@ successor; otherwise it runs on this process. Its queue position is the number o to restart, and its wait timeout applies as it does to the drain, relative to the requests that were served when the script began to wait. +The `retryAfterRestart: true` result of the request that restarts the daemon for its environment carries +`restartReason: { kind: 'environmentChanged', variableNames }`: the sorted names of the variables that are set in only +one of the two environments, or set to different values, compared as the fingerprint's `environmentHash` compares them +(without the variables that it ignores, and with repeated `PATH` entries removed). Values are never sent, since a +variable such as `NODE_OPTIONS` can hold a secret. Each request that is answered while that restart is pending gets +the same reason, because the successor starts with the restarting request's environment, not its own. The daemon log +(`onLog`) names the request and the variables. + Protocol 0.10 (`DAEMON_WORKSPACE_RESTART_PROTOCOL_MINOR`) provides bounded, typed retry authorization. Only a pre-execution command result may carry `retryAfterRestart: true`. During a planned restart, accepted queued requests drain those typed results before disconnect rather than being reduced to ambiguous connection diff --git a/libraries/rush-daemon/src/EnvironmentRestartReason.ts b/libraries/rush-daemon/src/EnvironmentRestartReason.ts new file mode 100644 index 0000000000..1c205b3ea3 --- /dev/null +++ b/libraries/rush-daemon/src/EnvironmentRestartReason.ts @@ -0,0 +1,39 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { getWorkspaceFingerprintEnvironmentEntries } from '@microsoft/rush-lib'; +import type { IDaemonEnvironmentChangedRestartReason } from '@rushstack/rush-daemon-protocol'; + +/** + * The variables of an environment that a daemon's identity includes, by name, with the values that the workspace + * input fingerprint hashes: without the variables that it ignores, and with repeated `PATH` entries removed. + */ +export type EnvironmentIdentityEntries = ReadonlyMap; + +/** Returns the variables of `environment` that a daemon's identity includes. */ +export function getEnvironmentIdentityEntries( + environment: Readonly> +): EnvironmentIdentityEntries { + return new Map(getWorkspaceFingerprintEnvironmentEntries(environment)); +} + +/** + * Says why a daemon that started with `startupEntries` restarts for a request with `environment`: the sorted names + * of the variables that are set in only one of them or set to different values, never the values. Returns + * `undefined` when the environments do not differ, which is when their fingerprint `environmentHash` is the same. + */ +export function getEnvironmentRestartReason( + startupEntries: EnvironmentIdentityEntries, + environment: Readonly> +): IDaemonEnvironmentChangedRestartReason | undefined { + const requestEntries: EnvironmentIdentityEntries = getEnvironmentIdentityEntries(environment); + const variableNames: Set = new Set(); + for (const [name, value] of startupEntries) { + if (requestEntries.get(name) !== value) variableNames.add(name); + } + for (const name of requestEntries.keys()) { + if (!startupEntries.has(name)) variableNames.add(name); + } + if (variableNames.size === 0) return undefined; + return { kind: 'environmentChanged', variableNames: Array.from(variableNames).sort() }; +} diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index a8cef149f0..b73e4a0233 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -83,7 +83,9 @@ export function freezeDaemonRequestAdmissionOptions( /** Says why the daemon restarts, completing "the daemon could restart ". */ function formatRestartCause(restartReason: DaemonRestartReason): string { - return `because its installation at ${restartReason.folder} was ${restartReason.change}`; + return restartReason.kind === 'environmentChanged' + ? `because a command's environment differs from its own in ${restartReason.variableNames.join(', ')}` + : `because its installation at ${restartReason.folder} was ${restartReason.change}`; } class WorkspaceRequestScheduler extends RequestScheduler { diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index a0e615a4e8..ba7dec3e31 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -22,6 +22,7 @@ import { NoOpTerminalProvider, Terminal } from '@rushstack/terminal'; import type { DaemonRestartReason, IDaemonCommandResult, + IDaemonEnvironmentChangedRestartReason, IDaemonInstallationChange, IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; @@ -66,6 +67,11 @@ import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from './Workspa import { classifyRushCommand } from './RushCommandRequestPolicy'; import { FreshCaptureCoalescer } from './FreshCaptureCoalescer'; import type { CheckDaemonInstallation } from './DaemonInstallationMonitor'; +import { + getEnvironmentIdentityEntries, + getEnvironmentRestartReason, + type EnvironmentIdentityEntries +} from './EnvironmentRestartReason'; interface IExecutionState { began: boolean; @@ -154,9 +160,9 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { ); readonly #repoRoot: string; /** The daemon's environment before any engine ran. A plugin may add names to `process.env` later. */ - readonly #startupEnvironment: Record = Object.fromEntries( - Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) - ); + readonly #startupEnvironment: Record; + /** The variables of `#startupEnvironment` that the startup fingerprint's `environmentHash` includes. */ + readonly #startupEnvironmentEntries: EnvironmentIdentityEntries; readonly #startupFingerprint: IWorkspaceInputFingerprint; readonly #runtimeCache: WorkspaceRuntimeFingerprintCache; // Concurrent requests share captures; each capture still starts after the requests it serves arrived. @@ -174,6 +180,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { #forceReload: boolean = false; #closing: boolean = false; #restartPending: boolean = false; + /** Why the pending restart happens, when the request that it is for has a different environment. */ + #restartReason: IDaemonEnvironmentChangedRestartReason | undefined; #lastReloadTier: WorkspaceInputChangeTier = WorkspaceInputChangeTier.Reuse; #installationChange: IDaemonInstallationChange | undefined; #transitioning: boolean = false; @@ -186,7 +194,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { options: IWorkspaceRequestLifecycleOptions, repoRoot: string, fingerprint: IWorkspaceInputFingerprint, - runtimeCache: WorkspaceRuntimeFingerprintCache + runtimeCache: WorkspaceRuntimeFingerprintCache, + startupEnvironment: Record ) { this.#options = options; this.#repoRoot = repoRoot; @@ -194,6 +203,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { this.#resolver = options.resolver; this.#ownedResolvers.add(options.resolver); this.#runtimeCache = runtimeCache; + this.#startupEnvironment = startupEnvironment; + this.#startupEnvironmentEntries = getEnvironmentIdentityEntries(startupEnvironment); } public static async createAsync( @@ -201,13 +212,23 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { ): Promise { const session: IWorkspaceSession = await options.provider.getSessionAsync(); const runtimeCache: WorkspaceRuntimeFingerprintCache = new WorkspaceRuntimeFingerprintCache(); + // The startup fingerprint hashes this copy, so that a restart for another environment can name what differs. + const startupEnvironment: Record = Object.fromEntries( + Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) + ); const fingerprint: IWorkspaceInputFingerprint = await captureWorkspaceInputFingerprintAsync({ rushConfiguration: session.rushConfiguration, - environment: process.env, + environment: startupEnvironment, runtimePaths: [__dirname, path.resolve(__dirname, '../package.json')], runtimeCache }); - return new WorkspaceRequestLifecycle(options, session.metadata.repoRoot, fingerprint, runtimeCache); + return new WorkspaceRequestLifecycle( + options, + session.metadata.repoRoot, + fingerprint, + runtimeCache, + startupEnvironment + ); } /** The last applied input decision; reading status never changes or reloads the workspace. */ @@ -342,16 +363,27 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { continue; } if (error instanceof RestartBeforeExecution && !state.began && !state.terminalAttempted) { + // The request, and each request that is answered while the restart is pending, learns why it happens. + const restartReason: IDaemonEnvironmentChangedRestartReason | undefined = + getEnvironmentRestartReason(this.#startupEnvironmentEntries, error.plan.environment); try { await client.interactiveSession.finishAsync(); await client.writeResultAsync({ ...preExecutionFailure(envelope.requestId, error), - retryAfterRestart: true + retryAfterRestart: true, + ...(restartReason && { restartReason }) }); error.session.retire?.(); this.#lastReloadTier = WorkspaceInputChangeTier.Restart; + this.#restartReason = restartReason; this.#restartPending = true; this.#closing = true; + if (restartReason) { + this.#options.onLog?.( + `rushd: restarting for request ${envelope.requestId}, whose environment differs from this ` + + `daemon's in ${restartReason.variableNames.join(', ')}` + ); + } this.#options.onRestartRequested(error.plan); } finally { error.workspaceLease.release(); @@ -1061,7 +1093,14 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { #restartPendingResult(requestId: string, pending: RestartPendingBeforeExecution): IDaemonCommandResult { const change: IDaemonInstallationChange | undefined = this.#installationChange; - if (!change) return { ...preExecutionFailure(requestId, pending), retryAfterRestart: true }; + if (!change) { + const restartReason: IDaemonEnvironmentChangedRestartReason | undefined = this.#restartReason; + return { + ...preExecutionFailure(requestId, pending), + retryAfterRestart: true, + ...(restartReason && { restartReason }) + }; + } return { ...preExecutionFailure(requestId, new InstallationChangedBeforeExecution(change)), retryAfterRestart: true, diff --git a/libraries/rush-daemon/src/test/EnvironmentRestartReason.test.ts b/libraries/rush-daemon/src/test/EnvironmentRestartReason.test.ts new file mode 100644 index 0000000000..fcbf765054 --- /dev/null +++ b/libraries/rush-daemon/src/test/EnvironmentRestartReason.test.ts @@ -0,0 +1,62 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as path from 'node:path'; + +import { getEnvironmentIdentityEntries, getEnvironmentRestartReason } from '../EnvironmentRestartReason'; + +const STARTUP: Record = { + HOME: '/home/agent', + NODE_OPTIONS: '--max-old-space-size=8192', + PATH: ['/usr/local/bin', '/usr/bin'].join(path.delimiter), + TERM: 'xterm-256color' +}; + +describe(getEnvironmentRestartReason.name, () => { + const startupEntries = getEnvironmentIdentityEntries(STARTUP); + + it('returns undefined when only variables that the fingerprint ignores or normalizes differ', () => { + expect(getEnvironmentRestartReason(startupEntries, { ...STARTUP })).toBeUndefined(); + expect( + getEnvironmentRestartReason(startupEntries, { + TERM: 'dumb', + PWD: '/elsewhere', + PATH: [STARTUP.PATH, '/usr/bin'].join(path.delimiter), + NODE_OPTIONS: STARTUP.NODE_OPTIONS, + HOME: STARTUP.HOME, + UNSET: undefined + }) + ).toBeUndefined(); + }); + + it('names each variable that is changed, added or removed, sorted, and never a value', () => { + const requested: Record = { + ...STARTUP, + NODE_OPTIONS: '--inspect', + FOO: 'bar' + }; + delete requested.HOME; + const reason = getEnvironmentRestartReason(startupEntries, requested); + expect(reason).toEqual({ kind: 'environmentChanged', variableNames: ['FOO', 'HOME', 'NODE_OPTIONS'] }); + expect(JSON.stringify(reason)).not.toMatch(/bar|inspect|agent/); + }); + + it('names PATH when its entries differ, not only their repetition', () => { + expect( + getEnvironmentRestartReason(startupEntries, { + ...STARTUP, + PATH: ['/usr/bin', '/usr/local/bin'].join(path.delimiter) + }) + ).toEqual({ kind: 'environmentChanged', variableNames: ['PATH'] }); + }); + + it('treats a variable that is set to an empty string as set', () => { + expect(getEnvironmentRestartReason(startupEntries, { ...STARTUP, NODE_OPTIONS: '' })).toEqual({ + kind: 'environmentChanged', + variableNames: ['NODE_OPTIONS'] + }); + expect( + getEnvironmentRestartReason(getEnvironmentIdentityEntries({ ...STARTUP, EMPTY: '' }), STARTUP) + ).toEqual({ kind: 'environmentChanged', variableNames: ['EMPTY'] }); + }); +}); diff --git a/libraries/rush-daemon/src/test/WorkspaceEnvironmentRestartReason.test.ts b/libraries/rush-daemon/src/test/WorkspaceEnvironmentRestartReason.test.ts new file mode 100644 index 0000000000..003777a529 --- /dev/null +++ b/libraries/rush-daemon/src/test/WorkspaceEnvironmentRestartReason.test.ts @@ -0,0 +1,127 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { setTimeout as delayAsync } from 'node:timers/promises'; + +import { + DaemonFrameType, + decodeDaemonControlMessage, + type IDaemonFrame, + type IDaemonRequestEnvelope +} from '@rushstack/rush-daemon-protocol'; + +import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import type { DaemonRequestWireClient, ITerminalExchange } from './DaemonRequestWireTestUtilities'; +import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; +import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; + +jest.setTimeout(60_000); + +const BUILD_B: string[] = ['build', '--to', 'b', '--parallelism', '3']; +const BUILD_C: string[] = ['build', '--to', 'c', '--parallelism', '3']; + +interface IQueuedRequest { + readonly client: DaemonRequestWireClient; + readonly envelope: IDaemonRequestEnvelope; +} + +function isQueuePosition(frame: IDaemonFrame): boolean { + return ( + frame.kind === DaemonFrameType.controlJson && + decodeDaemonControlMessage(frame.payload).kind === 'queuePosition' + ); +} + +/** Starts a request and returns once the daemon reports that it waits. */ +async function startQueuedAsync( + fixture: DaemonGraphTestFixture, + environment: Record, + clients: DaemonRequestWireClient[] +): Promise { + const client: DaemonRequestWireClient = await fixture.connectAsync(); + clients.push(client); + const envelope: IDaemonRequestEnvelope = fixture.envelope(BUILD_B, { environment }); + await client.sendControlAsync({ kind: 'requestStart', payload: envelope }); + while (!isQueuePosition(await client.readFrameAsync())); + return { client, envelope }; +} + +it('tells the request that restarts the daemon for its environment, and each request answered while that restart is pending, which variables differ', async () => { + const fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + // Project c holds its build open until the test removes the marker. + created.write('hold', ''); + created.write( + 'c/build.cjs', + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','c\\n');" + + "const t=setInterval(()=>{if(!fs.existsSync('../hold')){clearInterval(t);console.log('finished-c');}},20);" + ); + }); + const clients: DaemonRequestWireClient[] = []; + try { + const before = await pongAsync(fixture); + const held: Promise = fixture.runAsync(BUILD_C); + const deadline: number = Date.now() + 30_000; + while (!fixture.runs().includes('c') && Date.now() < deadline) await delayAsync(20); + expect(fixture.runs()).toContain('c'); + + // Each request differs from the daemon in its own variable. Either one may restart the daemon; the other is + // answered while that restart is pending. + const requests: IQueuedRequest[] = []; + for (const [name, value] of [ + ['RUSHD_RESTART_REASON_FIRST', 'first-value'], + ['RUSHD_RESTART_REASON_SECOND', 'second-value'] + ]) { + requests.push(await startQueuedAsync(fixture, { ...fixture.environment, [name]: value }, clients)); + } + fs.rmSync(path.join(fixture.folder, 'hold')); + expect((await held).terminal).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + const exchanges: ITerminalExchange[] = await Promise.all( + requests.map(({ client, envelope }: IQueuedRequest) => client.readTerminalAsync(envelope.requestId)) + ); + + const restartLines: string[] = fixture.logs.filter((line: string) => + line.startsWith('rushd: restarting for request ') + ); + expect(restartLines).toHaveLength(1); + const restarter: number = requests.findIndex(({ envelope }: IQueuedRequest) => + restartLines[0].includes(`request ${envelope.requestId},`) + ); + expect(restarter).not.toBe(-1); + const variableName: string = + restarter === 0 ? 'RUSHD_RESTART_REASON_FIRST' : 'RUSHD_RESTART_REASON_SECOND'; + expect(restartLines[0]).toBe( + `rushd: restarting for request ${requests[restarter].envelope.requestId}, whose environment differs ` + + `from this daemon's in ${variableName}` + ); + // The request that was answered while the restart was pending learns the restart's reason, not its own. + for (const { terminal } of exchanges) { + expect(terminal).toMatchObject({ + kind: 'requestResult', + payload: { + exitCode: 1, + retryAfterRestart: true, + restartReason: { kind: 'environmentChanged', variableNames: [variableName] } + } + }); + } + expect(fixture.logs.join('\n')).not.toMatch(/first-value|second-value/); + const restarted = await fixture.host.restartCompleted; + expect(restarted?.pid).not.toBe(before.pid); + expect(fixture.runs()).toEqual(['c']); + } finally { + fs.rmSync(path.join(fixture.folder, 'hold'), { force: true }); + await Promise.all(clients.map((client: DaemonRequestWireClient) => client.closeAsync())); + try { + await fixture.host.closeAsync(); + await fixture.host.restartCompleted; + } finally { + await stopSuccessorAsync(fixture.host.paths); + await fixture[Symbol.asyncDispose](); + } + } +}); From ec6d7b80776791ef560df387b25b426f1117123d Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:05 +0000 Subject: [PATCH 070/265] [rush-daemon] A restart drain's waiter rechecks its tier Swarm integration step 55; original commit 075d2130b2 (merge of swarm/r05-t127-int at 236c184d94). Scope: task 127. Brings r05's task 127 follow-up (board 2436, re-tipped onto d87702c813 in board 2858): a Restart-tier change that is reverted under load no longer fails its waiter at 30 s. Second agent: o05 board 2552. s17 batch D1, item 3 of 10. Gate: ch01 GATE OK board 3310 (tree bdfad6af20) Commits folded into this step (2): - 961f007cae [rush-daemon] A request that waits for a restart drain rechecks whether it still needs the restart (task 127) - 0b1585667a [rush-daemon] Requests that wait for a restart drain share one recheck capture a second (task 127) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...estart-drain-recheck_2026-09-28-21-05.json | 10 + libraries/rush-daemon/README.md | 22 +- .../src/WorkspaceRequestAdmission.ts | 22 +- .../src/WorkspaceRequestLifecycle.ts | 77 +++++- .../src/WorkspaceRestartArbiter.ts | 112 +++++++- .../src/test/RestartDrainAdmission.test.ts | 35 ++- .../src/test/WorkspaceRestartArbiter.test.ts | 186 +++++++++++-- .../WorkspaceServedScriptAdmission.test.ts | 251 +++++++++++++++++- 8 files changed, 650 insertions(+), 65 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-restart-drain-recheck_2026-09-28-21-05.json diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-restart-drain-recheck_2026-09-28-21-05.json b/common/changes/@rushstack/rush-daemon/swarm-r05-restart-drain-recheck_2026-09-28-21-05.json new file mode 100644 index 0000000000..df2364e5f3 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-restart-drain-recheck_2026-09-28-21-05.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A request that waits for a restart drain rechecks its inputs every second, sharing one capture a second with the other requests that wait, and is admitted on the running daemon as if it had just arrived once the change that needed the restart is reverted.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 383e4b6676..04d65fa115 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -194,13 +194,21 @@ drain, which could otherwise keep it waiting for as long as they keep arriving. `waitTimeoutMs` limits the whole drain, and only its remaining time carries over to the successor. When a drain times out, its message names the time that did not count. -A restart is pending from when a request begins its restart drain until the request has planned the restart, or has -failed or been cancelled. A rushx script that arrives while a restart is pending does not start, since the restart -would then wait for it to exit: the script waits for the pending restart instead, and the drain does not count it. If -the restart was planned, the script's result carries `retryAfterRestart: true` so that the client runs it on the -successor; otherwise it runs on this process. Its queue position is the number of requests that are served or waiting -to restart, and its wait timeout applies as it does to the drain, relative to the requests that were served when the -script began to wait. +The change that needs a restart may be reverted during the drain, while requests that do not need one keep the drain +from finishing for as long as they keep arriving. A request that waits for the drain therefore captures its inputs +again every second, and once they no longer need a restart it stops waiting and is admitted as if it had just +arrived, on this process. The requests that wait share these captures: a request reuses the latest one until it is a +second old, so the drain costs one capture a second however many requests wait, and each request still sees a revert +within about two seconds. A request that needs a restart only for its environment does not capture again, since its +environment cannot change. + +A restart is pending from when a request begins its restart drain until the request has planned the restart, has found +that it no longer needs it, or has failed or been cancelled. A rushx script that arrives while a restart is pending +does not start, since the restart would then wait for it to exit: the script waits for the pending restart instead, +and the drain does not count it. If the restart was planned, the script's result carries `retryAfterRestart: true` so +that the client runs it on the successor; otherwise it runs on this process. Its queue position is the number of +requests that are served or waiting to restart, and its wait timeout applies as it does to the drain, relative to the +requests that were served when the script began to wait. The `retryAfterRestart: true` result of the request that restarts the daemon for its environment carries `restartReason: { kind: 'environmentChanged', variableNames }`: the sorted names of the variables that are set in only diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index b73e4a0233..54b4964d4f 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -24,7 +24,9 @@ import type { IWorkspaceSession } from './WorkspaceSession'; import { assertWorkspaceRequestResourcesHealthy } from './WorkspaceRequestResources'; import type { IWorkspaceRestartDrainOptions, + IWorkspaceRestartRecheck, IWorkspaceRestartTicket, + IWorkspaceRestartWaitResult, WorkspaceRestartArbiter } from './WorkspaceRestartArbiter'; @@ -467,16 +469,20 @@ export class RequestAdmissionController { * * A `restartReason` says that the daemon restarts for that reason rather than for the request's environment. Queue * positions then carry it, and admission errors name it. + * + * @returns true once the drain finishes, or false if `recheck` found that the request no longer needs the restart. */ public async waitForRestartDrainAsync( arbiter: WorkspaceRestartArbiter, ticket: IWorkspaceRestartTicket, - restartReason?: DaemonRestartReason - ): Promise { - await this.#waitForRestartArbiterAsync( - (options: IWorkspaceRestartDrainOptions) => arbiter.waitForDrainAsync(ticket, options), + restartReason?: DaemonRestartReason, + recheck?: IWorkspaceRestartRecheck + ): Promise { + const result: IWorkspaceRestartWaitResult = await this.#waitForRestartArbiterAsync( + (options: IWorkspaceRestartDrainOptions) => arbiter.waitForDrainAsync(ticket, options, recheck), restartReason ); + return !result.restartWithdrawn; } /** @@ -498,15 +504,15 @@ export class RequestAdmissionController { } async #waitForRestartArbiterAsync( - waitAsync: (options: IWorkspaceRestartDrainOptions) => Promise, + waitAsync: (options: IWorkspaceRestartDrainOptions) => Promise, restartReason?: DaemonRestartReason - ): Promise { + ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; const startMs: number = Date.now(); let waivedMs: number = 0; try { // The arbiter reports its own admission errors, so this does not depend on the scheduler error mapping. - waivedMs = await waitAsync({ + const result: IWorkspaceRestartWaitResult = await waitAsync({ abortSignal: this.#abortController.signal, noWait: this.#admission?.noWait, waitTimeoutMs: this.#remainingMs, @@ -514,6 +520,8 @@ export class RequestAdmissionController { restartCause: restartReason && formatRestartCause(restartReason), onServingCountChanged: writer ? (count: number) => writer.enqueue(count, restartReason) : undefined }); + waivedMs = result.waivedMs; + return result; } finally { await writer?.flushAsync(); this.#spend(Date.now() - startMs - waivedMs); diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index ba7dec3e31..fb2909f8f2 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -63,7 +63,11 @@ import type { IWorkspaceProcessRestartPlan, IWorkspaceSuccessorLaunch } from './WorkspaceProcessRestart'; -import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket } from './WorkspaceRestartArbiter'; +import { + WorkspaceRestartArbiter, + type IWorkspaceRestartRecheck, + type IWorkspaceRestartTicket +} from './WorkspaceRestartArbiter'; import { classifyRushCommand } from './RushCommandRequestPolicy'; import { FreshCaptureCoalescer } from './FreshCaptureCoalescer'; import type { CheckDaemonInstallation } from './DaemonInstallationMonitor'; @@ -73,6 +77,15 @@ import { type EnvironmentIdentityEntries } from './EnvironmentRestartReason'; +/** How often a request that waits for a restart drain checks whether it still needs the restart. */ +const RESTART_RECHECK_INTERVAL_MS: number = 1000; + +interface IRecheckCapture { + /** On the clock of `performance.now()`. */ + readonly startTimeMs: number; + readonly fingerprint: Promise; +} + interface IExecutionState { began: boolean; terminalAttempted: boolean; @@ -171,6 +184,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { new FreshCaptureCoalescer(); readonly #projectFingerprintCaptures: FreshCaptureCoalescer = new FreshCaptureCoalescer(); + /** The latest capture that a request waiting for a restart drain started, for each workspace configuration. */ + readonly #recheckCaptures: WeakMap = new WeakMap(); #fingerprint: IWorkspaceInputFingerprint; #projectFingerprint: string | undefined; #commandIdentity: string | undefined; @@ -517,8 +532,14 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { lease.release(); if (ticket) { // Like build requests, a graph-control restart must not preempt requests this process can serve. - await admission.waitForRestartDrainAsync(this.#restartArbiter, ticket); + const drained: boolean = await admission.waitForRestartDrainAsync( + this.#restartArbiter, + ticket, + undefined, + this.#createRestartRecheck(session, controlEnvelope, current, false) + ); if (this.#restartPending) throw new RestartPendingBeforeExecution(); + if (!drained) return await this.#prepareAsync(envelope, client, admission, ticket, receivedTimeMs); } this.#cancelObservers(); lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); @@ -613,8 +634,15 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { lease.release(); if (tier === WorkspaceInputChangeTier.Restart && ticket) { // Serve every queued or in-flight request that matches this process before restarting for another one. - await admission.waitForRestartDrainAsync(this.#restartArbiter, ticket); + const drained: boolean = await admission.waitForRestartDrainAsync( + this.#restartArbiter, + ticket, + undefined, + this.#createRestartRecheck(session, envelope, fingerprint, isMutation(envelope)) + ); if (this.#restartPending) throw new RestartPendingBeforeExecution(); + // The request no longer needs the restart, so it is admitted as it would be if it arrived now. + if (!drained) return await this.#prepareAsync(envelope, client, admission, ticket, receivedTimeMs); } if (this.#transitioning) { const shared: IRequestLease = await admission.acquireBehindTransitionAsync( @@ -653,6 +681,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { throw error; } } + // A request that drained for a restart it no longer needs must not keep rushx scripts waiting for the restart. + if (ticket) this.#restartArbiter.withdrawRestart(ticket); if (isMutation(envelope)) { if (!this.#options.getSuccessorLaunchAsync) { throw new DaemonRequestDispatchError( @@ -904,6 +934,47 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } } + /** + * The change that needs a restart may be reverted while the request waits for the restart drain, and requests that + * do not need a restart can keep the drain from finishing for as long as they keep arriving. The request therefore + * captures its inputs again while it waits, and stops waiting once they no longer need a restart. A request's + * environment does not change, so a request that needs a restart for its environment does not capture again. + */ + #createRestartRecheck( + session: IWorkspaceSession, + envelope: IDaemonRequestEnvelope, + fingerprint: IWorkspaceInputFingerprint, + mutation: boolean + ): IWorkspaceRestartRecheck | undefined { + if (fingerprint.environmentHash !== this.#startupFingerprint.environmentHash) return undefined; + return { + intervalMs: RESTART_RECHECK_INTERVAL_MS, + stillNeedsRestartAsync: async () => + this.#classify(await this.#recheckCaptureAsync(session, envelope), mutation) === + WorkspaceInputChangeTier.Restart + }; + } + + /** + * Every request that waits for a restart drain checks its inputs once per interval, so the waiters share the + * latest check's capture, whether it still runs or has settled, until it is one interval old. The checks then cost + * one capture per interval however many requests wait, and each waiter still sees a change within about two + * intervals. Only requests whose environments match this process's environment check again (see + * `#createRestartRecheck`), so every waiter of one workspace configuration requests the same capture. + */ + #recheckCaptureAsync( + session: IWorkspaceSession, + envelope: IDaemonRequestEnvelope + ): Promise { + const { rushConfiguration } = session; + const nowMs: number = performance.now(); + const latest: IRecheckCapture | undefined = this.#recheckCaptures.get(rushConfiguration); + if (latest && nowMs - latest.startTimeMs < RESTART_RECHECK_INTERVAL_MS) return latest.fingerprint; + const fingerprint: Promise = this.#captureAsync(session, envelope); + this.#recheckCaptures.set(rushConfiguration, { startTimeMs: nowMs, fingerprint }); + return fingerprint; + } + #classify(fingerprint: IWorkspaceInputFingerprint, mutation: boolean): WorkspaceInputChangeTier { if ( fingerprint.selectedRushVersion !== Rush.version || diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts index b23c4a364c..aa5a96975e 100644 --- a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -46,6 +46,8 @@ interface IWaitKind { readonly isBlocked: () => boolean; /** How many other requests the request waits for, reported as its queue position. */ readonly countWaitedFor: () => number; + /** Lets a request that waits to restart find that it no longer needs to. */ + readonly recheck: IWorkspaceRestartRecheck | undefined; readonly noWaitMessage: string; /** Begins the timeout message, which goes on to name the requests that the restart waits for. */ readonly timeoutPrefix: string; @@ -76,10 +78,83 @@ export interface IWorkspaceRestartDrainOptions { readonly onServingCountChanged?: (servingCount: number) => void; } +/** + * Lets a request that waits for a restart drain find that it no longer needs the restart. The change that needed it + * may be reverted during the drain, while requests that do not need a restart keep the drain from finishing. + */ +export interface IWorkspaceRestartRecheck { + /** + * Resolves whether the request still needs the restart. It is called every `intervalMs` while the request waits, + * each time after the previous call settles. A call that rejects counts as true. + */ + readonly stillNeedsRestartAsync: () => Promise; + readonly intervalMs: number; +} + +/** How a wait for a restart drain or a pending restart ended. */ +export interface IWorkspaceRestartWaitResult { + /** + * How many milliseconds of the wait did not spend `waitTimeoutMs` + * (see {@link IWorkspaceRestartDrainOptions.waivesTimeoutForServedWork}). + */ + readonly waivedMs: number; + /** + * The wait ended because {@link IWorkspaceRestartRecheck.stillNeedsRestartAsync} resolved false, so the request no + * longer counts as a pending restart. Otherwise the wait ended because the request no longer had to wait. + */ + readonly restartWithdrawn: boolean; +} + +/** Calls {@link IWorkspaceRestartRecheck.stillNeedsRestartAsync} while a restart candidate waits. */ +class RestartRecheck { + /** A call resolved false. */ + public withdrawn: boolean = false; + readonly #recheck: IWorkspaceRestartRecheck; + readonly #onWithdrawn: () => void; + #timer: ReturnType | undefined; + #stopped: boolean = false; + + public constructor(recheck: IWorkspaceRestartRecheck, onWithdrawn: () => void) { + this.#recheck = recheck; + this.#onWithdrawn = onWithdrawn; + this.#schedule(); + } + + /** Ignores a call that is still running, and makes no more. */ + public stop(): void { + this.#stopped = true; + clearTimeout(this.#timer); + } + + #schedule(): void { + this.#timer = setTimeout(() => { + void this.#recheckAsync(); + }, this.#recheck.intervalMs); + } + + async #recheckAsync(): Promise { + let stillNeeded: boolean = true; + try { + stillNeeded = await this.#recheck.stillNeedsRestartAsync(); + } catch { + // For example, a file that the capture reads was being rewritten. The next call tries again. + } + if (this.#stopped) return; + if (stillNeeded) { + this.#schedule(); + } else { + this.withdrawn = true; + this.#onWithdrawn(); + } + } +} + /** * Arbitrates process restarts between requests whose environments differ from the running daemon. * A request that needs a restart waits until every other request this process can serve has finished, * so a mismatched environment never preempts queued or in-flight work that matches the running process. + * A request that finds during or after the drain that it no longer needs the restart, since the change that needed it + * was reverted, withdraws the restart. * A rushx script that arrives while a restart is pending waits for the restart instead, since it may not exit until * it is stopped and the restart would otherwise wait for it. */ @@ -116,28 +191,37 @@ export class WorkspaceRestartArbiter { /** * Whether another tracked request needs to restart the daemon for its environment: it waits for the drain, or it * has drained and not yet left. A request that needs a restart leaves once the restart is planned, or once it - * fails or is cancelled. + * fails or is cancelled. A request that finds it no longer needs one withdraws its restart. */ public hasPendingRestart(ticket: IWorkspaceRestartTicket): boolean { return Array.from(this.#restartCandidates).some((candidate: IMutableTicket) => candidate !== ticket); } + /** + * Records that a request which drained for a restart no longer needs one, for example because the change was + * reverted, so that requests waiting for its restart stop waiting. Does nothing for other requests. + */ + public withdrawRestart(ticket: IWorkspaceRestartTicket): void { + if (this.#restartCandidates.delete(ticket as IMutableTicket)) this.#notifyChange(); + } + /** * Waits until no other tracked request is still being served by this process, then counts the ticket - * as served again so concurrent restart candidates proceed one at a time. Returns how many milliseconds of the - * wait did not spend `waitTimeoutMs` (see {@link IWorkspaceRestartDrainOptions.waivesTimeoutForServedWork}). - * From when the wait begins until the ticket leaves, {@link WorkspaceRestartArbiter.hasPendingRestart} reports - * the restart to other requests. + * as served again so concurrent restart candidates proceed one at a time. From when the wait begins until the + * ticket leaves, {@link WorkspaceRestartArbiter.hasPendingRestart} reports the restart to other requests. If + * `recheck` finds that the request no longer needs the restart, the wait ends then and the restart is withdrawn. */ public async waitForDrainAsync( ticket: IWorkspaceRestartTicket, - options: IWorkspaceRestartDrainOptions - ): Promise { + options: IWorkspaceRestartDrainOptions, + recheck?: IWorkspaceRestartRecheck + ): Promise { const { restartCause } = options; return await this.#waitAsync(ticket, options, { restarts: true, isBlocked: () => this.#serving.size > 0, countWaitedFor: () => this.#serving.size, + recheck, noWaitMessage: restartCause === undefined ? 'Another environment is still being served; the request did not wait for a restart.' @@ -159,11 +243,12 @@ export class WorkspaceRestartArbiter { public async waitForPendingRestartAsync( ticket: IWorkspaceRestartTicket, options: IWorkspaceRestartDrainOptions - ): Promise { + ): Promise { return await this.#waitAsync(ticket, options, { restarts: false, isBlocked: () => this.hasPendingRestart(ticket), countWaitedFor: () => new Set([...this.#serving, ...this.#restartCandidates]).size, + recheck: undefined, noWaitMessage: 'Another request is waiting to restart the daemon for its environment; the rushx script did not wait ' + 'for the restart.', @@ -178,7 +263,7 @@ export class WorkspaceRestartArbiter { ticket: IWorkspaceRestartTicket, options: IWorkspaceRestartDrainOptions, kind: IWaitKind - ): Promise { + ): Promise { const state: IMutableTicket = ticket as IMutableTicket; if (state.left || state.waitingForDrain) throw new Error('The restart ticket is not being served.'); if (kind.restarts) this.#restartCandidates.add(state); @@ -195,8 +280,12 @@ export class WorkspaceRestartArbiter { options.onServingCountChanged?.(count); } }; + // The ticket is a restart candidate for as long as the wait lasts, so withdrawing its restart wakes the wait. + const recheck: RestartRecheck | undefined = kind.recheck + ? new RestartRecheck(kind.recheck, () => this.withdrawRestart(ticket)) + : undefined; try { - while (kind.isBlocked()) { + while (!recheck?.withdrawn && kind.isBlocked()) { if (options.noWait) { throw new RequestSchedulerError(RequestSchedulerErrorCode.NoWait, kind.noWaitMessage); } @@ -218,8 +307,9 @@ export class WorkspaceRestartArbiter { else if (remainingMs !== undefined) remainingMs -= elapsedMs; } } - return waivedMs; + return { waivedMs, restartWithdrawn: recheck?.withdrawn === true }; } finally { + recheck?.stop(); state.waitingForDrain = false; this.#serve(state); } diff --git a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts index ccdf2c637a..5780afed79 100644 --- a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts +++ b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts @@ -20,7 +20,8 @@ import { RequestAdmissionController } from '../WorkspaceRequestAdmission'; import { WorkspaceRestartArbiter, type IWorkspaceRestartTicket, - type IWorkspaceRestartTicketOptions + type IWorkspaceRestartTicketOptions, + type IWorkspaceRestartWaitResult } from '../WorkspaceRestartArbiter'; const INSTALLATION_REMOVED: DaemonRestartReason = { @@ -74,11 +75,11 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { waitTimeoutMs: 50, waitTimeoutIsDefault: true }); - const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); await delayAsync(250); expect(await isSettledAsync(draining)).toBe(false); arbiter.leave(serving); - await draining; + expect(await draining).toBe(true); // The later admission steps keep what was left of the default when the drain began. expect(admission.remainingAdmission).toMatchObject({ waitTimeoutIsDefault: true }); expect(admission.remainingAdmission?.waitTimeoutMs).toBeGreaterThan(0); @@ -110,7 +111,7 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { waitTimeoutMs: 50, waitTimeoutIsDefault: true }); - const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); const late: IWorkspaceRestartTicket = arbiter.enter(); await delayAsync(150); expect(await isSettledAsync(draining)).toBe(false); @@ -143,7 +144,7 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { const { admission, arbiter, serving, ticket } = createDrainTest({ waitTimeoutMs: 10_000 }); // Such as capturing the request's inputs before the wait, and planning the restart after it. await delayAsync(1_000); - const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket); await delayAsync(300); arbiter.leave(serving); await draining; @@ -156,6 +157,26 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { expect(arbiter.servingCount).toBe(0); }); + it('ends the drain once a recheck finds that the restart is no longer needed, and spends the time waited', async () => { + const { admission, arbiter, serving, ticket } = createDrainTest({ waitTimeoutMs: 10_000 }); + const answers: boolean[] = [true, false]; + let calls: number = 0; + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket, undefined, { + intervalMs: 100, + stillNeedsRestartAsync: async () => answers[calls++] + }); + // The other request is still being served. + expect(await draining).toBe(false); + expect(calls).toBe(2); + const remainingMs: number | undefined = admission.remainingAdmission?.waitTimeoutMs; + expect(remainingMs).toBeLessThanOrEqual(9_810); + expect(remainingMs).toBeGreaterThan(9_000); + arbiter.leave(ticket); + arbiter.leave(serving); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); + it('applies no-wait to the drain', async () => { const { admission, arbiter, serving, ticket } = createDrainTest({ noWait: true }); const error: unknown = await admission @@ -273,7 +294,7 @@ interface IPendingRestartTest { readonly admission: RequestAdmissionController; readonly arbiter: WorkspaceRestartArbiter; readonly candidate: IWorkspaceRestartTicket; - readonly draining: Promise; + readonly draining: Promise; readonly positions: number[]; readonly script: IWorkspaceRestartTicket; readonly serving: IWorkspaceRestartTicket; @@ -287,7 +308,7 @@ function createPendingRestartTest( const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const serving: IWorkspaceRestartTicket = arbiter.enter(servingOptions); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const draining: Promise = arbiter.waitForDrainAsync(candidate, { + const draining: Promise = arbiter.waitForDrainAsync(candidate, { abortSignal: new AbortController().signal, noWait: undefined, waitTimeoutMs: undefined diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts index 3f91594741..74da42a9d0 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts @@ -7,7 +7,9 @@ import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestSche import { WorkspaceRestartArbiter, type IWorkspaceRestartDrainOptions, - type IWorkspaceRestartTicket + type IWorkspaceRestartRecheck, + type IWorkspaceRestartTicket, + type IWorkspaceRestartWaitResult } from '../WorkspaceRestartArbiter'; const WAIT: { abortSignal: AbortSignal; noWait: undefined; waitTimeoutMs: undefined } = { @@ -26,6 +28,23 @@ async function isSettledAsync(promise: Promise): Promise { return settled; } +/** Answers each recheck from a list, repeating the last answer. An `Error` answer rejects. */ +class ScriptedRecheck implements IWorkspaceRestartRecheck { + public readonly intervalMs: number = 20; + public calls: number = 0; + readonly #answers: ReadonlyArray; + + public constructor(answers: ReadonlyArray) { + this.#answers = answers; + } + + public readonly stillNeedsRestartAsync: () => Promise = async () => { + const answer: boolean | Error = this.#answers[Math.min(this.calls++, this.#answers.length - 1)]; + if (answer instanceof Error) throw answer; + return answer; + }; +} + function expectWaitTimeout(error: unknown): RequestSchedulerError { expect(error).toBeInstanceOf(RequestSchedulerError); expect((error as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); @@ -47,8 +66,8 @@ describe(WorkspaceRestartArbiter.name, () => { const serving: IWorkspaceRestartTicket = arbiter.enter(); const first: IWorkspaceRestartTicket = arbiter.enter(); const second: IWorkspaceRestartTicket = arbiter.enter(); - const firstWait: Promise = arbiter.waitForDrainAsync(first, WAIT); - const secondWait: Promise = arbiter.waitForDrainAsync(second, WAIT); + const firstWait: Promise = arbiter.waitForDrainAsync(first, WAIT); + const secondWait: Promise = arbiter.waitForDrainAsync(second, WAIT); const late: IWorkspaceRestartTicket = arbiter.enter(); arbiter.leave(serving); expect(await isSettledAsync(firstWait)).toBe(false); @@ -70,7 +89,7 @@ describe(WorkspaceRestartArbiter.name, () => { const serving: IWorkspaceRestartTicket = arbiter.enter(); const ticket: IWorkspaceRestartTicket = arbiter.enter(); const abort: AbortController = new AbortController(); - const waiting: Promise = arbiter.waitForDrainAsync(ticket, { + const waiting: Promise = arbiter.waitForDrainAsync(ticket, { abortSignal: abort.signal, noWait: mode === 'no-wait' ? true : undefined, waitTimeoutMs: mode === 'timeout' ? 10 : undefined @@ -91,7 +110,7 @@ describe(WorkspaceRestartArbiter.name, () => { const second: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); const counts: number[] = []; - const waiting: Promise = arbiter.waitForDrainAsync(candidate, { + const waiting: Promise = arbiter.waitForDrainAsync(candidate, { ...WAIT, onServingCountChanged: (count: number) => counts.push(count) }); @@ -103,7 +122,7 @@ describe(WorkspaceRestartArbiter.name, () => { // Another restart candidate stops counting once it waits too; it then waits for the first candidate. const other: IWorkspaceRestartTicket = arbiter.enter(); const otherCounts: number[] = []; - const otherWaiting: Promise = arbiter.waitForDrainAsync(other, { + const otherWaiting: Promise = arbiter.waitForDrainAsync(other, { ...WAIT, onServingCountChanged: (count: number) => otherCounts.push(count) }); @@ -153,11 +172,11 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); await delayAsync(150); expect(await isSettledAsync(waiting)).toBe(false); arbiter.leave(earlier); - expect(await waiting).toBeGreaterThanOrEqual(140); + expect((await waiting).waivedMs).toBeGreaterThanOrEqual(140); arbiter.leave(candidate); expect(arbiter.servingCount).toBe(0); }); @@ -166,7 +185,7 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); const late: IWorkspaceRestartTicket = arbiter.enter(); await delayAsync(150); expect(await isSettledAsync(waiting)).toBe(false); @@ -216,12 +235,12 @@ describe(WorkspaceRestartArbiter.name, () => { const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); + const waiting: Promise = arbiter.waitForDrainAsync(candidate, WAIVED); arbiter.leave(script); await delayAsync(150); expect(await isSettledAsync(waiting)).toBe(false); arbiter.leave(earlier); - expect(await waiting).toBeGreaterThanOrEqual(140); + expect((await waiting).waivedMs).toBeGreaterThanOrEqual(140); arbiter.leave(candidate); expect(arbiter.servingCount).toBe(0); }); @@ -249,13 +268,13 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const waiting: Promise = arbiter.waitForDrainAsync(candidate, { + const waiting: Promise = arbiter.waitForDrainAsync(candidate, { ...WAIT, waitTimeoutMs: 5_000 }); await delayAsync(20); arbiter.leave(earlier); - expect(await waiting).toBe(0); + expect((await waiting).waivedMs).toBe(0); arbiter.leave(candidate); expect(arbiter.servingCount).toBe(0); }); @@ -273,7 +292,7 @@ describe(WorkspaceRestartArbiter.name, () => { const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const candidate: IWorkspaceRestartTicket = arbiter.enter(); expect(arbiter.hasPendingRestart(running)).toBe(false); - const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); expect(arbiter.hasPendingRestart(running)).toBe(true); expect(arbiter.hasPendingRestart(candidate)).toBe(false); arbiter.leave(running); @@ -292,7 +311,7 @@ describe(WorkspaceRestartArbiter.name, () => { const serving: IWorkspaceRestartTicket = arbiter.enter(); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const counts: number[] = []; - const waivedMs: number = await arbiter.waitForPendingRestartAsync(script, { + const { waivedMs } = await arbiter.waitForPendingRestartAsync(script, { ...WAIT, noWait: true, onServingCountChanged: (count: number) => counts.push(count) @@ -308,10 +327,10 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const counts: number[] = []; - const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { ...WAIT, onServingCountChanged: (count: number) => counts.push(count) }); @@ -322,7 +341,7 @@ describe(WorkspaceRestartArbiter.name, () => { expect(counts).toEqual([2, 1]); expect(await isSettledAsync(waiting)).toBe(false); arbiter.leave(candidate); - expect(await waiting).toBe(0); + expect((await waiting).waivedMs).toBe(0); expect(arbiter.servingCount).toBe(1); arbiter.leave(script); expect(arbiter.servingCount).toBe(0); @@ -333,10 +352,10 @@ describe(WorkspaceRestartArbiter.name, () => { const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const first: IWorkspaceRestartTicket = arbiter.enter(); const second: IWorkspaceRestartTicket = arbiter.enter(); - const firstDrain: Promise = arbiter.waitForDrainAsync(first, WAIT); - const secondDrain: Promise = arbiter.waitForDrainAsync(second, WAIT); + const firstDrain: Promise = arbiter.waitForDrainAsync(first, WAIT); + const secondDrain: Promise = arbiter.waitForDrainAsync(second, WAIT); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); - const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIT); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIT); arbiter.leave(running); await firstDrain; arbiter.leave(first); @@ -365,10 +384,10 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const abort: AbortController = new AbortController(); - const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { abortSignal: abort.signal, noWait: mode === 'no-wait' ? true : undefined, waitTimeoutMs: mode === 'timeout' ? 10 : undefined @@ -392,15 +411,15 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); - const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIVED); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIVED); await delayAsync(150); expect(await isSettledAsync(waiting)).toBe(false); arbiter.leave(earlier); await draining; arbiter.leave(candidate); - expect(await waiting).toBeGreaterThanOrEqual(140); + expect((await waiting).waivedMs).toBeGreaterThanOrEqual(140); arbiter.leave(script); expect(arbiter.servingCount).toBe(0); }); @@ -409,7 +428,7 @@ describe(WorkspaceRestartArbiter.name, () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const outcome: Promise = arbiter .waitForPendingRestartAsync(script, WAIVED) @@ -432,7 +451,7 @@ describe(WorkspaceRestartArbiter.name, () => { const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const earlier: IWorkspaceRestartTicket = arbiter.enter(); const candidate: IWorkspaceRestartTicket = arbiter.enter(); - const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); const outcome: Promise = arbiter .waitForPendingRestartAsync(script, WAIVED) @@ -446,4 +465,115 @@ describe(WorkspaceRestartArbiter.name, () => { expect(arbiter.servingCount).toBe(0); }); }); + + describe('rechecks while a restart candidate drains', () => { + it('withdraws the restart once it is no longer needed, while other requests are served, and wakes scripts', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const recheck: ScriptedRecheck = new ScriptedRecheck([true, false]); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT, recheck); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIT); + expect(arbiter.hasPendingRestart(script)).toBe(true); + expect(await draining).toEqual({ waivedMs: 0, restartWithdrawn: true }); + expect(recheck.calls).toBe(2); + expect(arbiter.hasPendingRestart(script)).toBe(false); + await waiting; + expect(arbiter.servingCount).toBe(3); + for (const served of [serving, candidate, script]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + }); + + it('keeps waiting while the restart is still needed, and stops rechecking once the drain ends', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const recheck: ScriptedRecheck = new ScriptedRecheck([true]); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT, recheck); + await delayAsync(150); + expect(await isSettledAsync(draining)).toBe(false); + expect(recheck.calls).toBeGreaterThanOrEqual(2); + arbiter.leave(serving); + expect(await draining).toEqual({ waivedMs: 0, restartWithdrawn: false }); + const calls: number = recheck.calls; + await delayAsync(100); + expect(recheck.calls).toBe(calls); + // The restart stays pending until the candidate plans it and leaves. + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + expect(arbiter.hasPendingRestart(script)).toBe(true); + arbiter.leave(candidate); + expect(arbiter.hasPendingRestart(script)).toBe(false); + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + + it('counts a recheck that fails as still needing the restart, and tries again', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const recheck: ScriptedRecheck = new ScriptedRecheck([new Error('rush.json was being rewritten'), false]); + const draining: Promise = arbiter.waitForDrainAsync(candidate, WAIT, recheck); + expect(await draining).toEqual({ waivedMs: 0, restartWithdrawn: true }); + expect(recheck.calls).toBe(2); + arbiter.leave(candidate); + arbiter.leave(serving); + expect(arbiter.servingCount).toBe(0); + }); + + it.each(['timeout', 'abort'])( + 'still reports a %s while a recheck runs, and ignores the recheck once the wait has failed', + async (mode) => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const serving: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + let finishRecheck: ((stillNeeded: boolean) => void) | undefined; + const recheck: IWorkspaceRestartRecheck = { + intervalMs: 10, + stillNeedsRestartAsync: () => + new Promise((resolve) => { + finishRecheck = resolve; + }) + }; + const abort: AbortController = new AbortController(); + const waiting: Promise = arbiter.waitForDrainAsync( + candidate, + { ...WAIT, abortSignal: abort.signal, waitTimeoutMs: mode === 'timeout' ? 100 : undefined }, + recheck + ); + await delayAsync(50); + expect(finishRecheck).toBeDefined(); + if (mode === 'abort') abort.abort(); + const error: unknown = await waiting.catch((caught: unknown) => caught); + expect((error as RequestSchedulerError).code).toBe( + mode === 'timeout' ? RequestSchedulerErrorCode.WaitTimeout : RequestSchedulerErrorCode.Aborted + ); + finishRecheck!(false); + await delayAsync(50); + // The request fails and leaves; until then, its restart is still pending. + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + expect(arbiter.hasPendingRestart(script)).toBe(true); + for (const served of [candidate, serving, script]) arbiter.leave(served); + expect(arbiter.servingCount).toBe(0); + } + ); + + it('lets a request that drained withdraw its restart, which wakes scripts that wait for it', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + expect(await arbiter.waitForDrainAsync(candidate, WAIT)).toEqual({ waivedMs: 0, restartWithdrawn: false }); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, WAIT); + expect(await isSettledAsync(waiting)).toBe(false); + // Only a restart candidate has a restart to withdraw. + arbiter.withdrawRestart(script); + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.withdrawRestart(candidate); + expect(await waiting).toEqual({ waivedMs: 0, restartWithdrawn: false }); + expect(arbiter.hasPendingRestart(script)).toBe(false); + arbiter.leave(candidate); + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + }); }); diff --git a/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts index 4bff3d0a71..1e8ffdf642 100644 --- a/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts @@ -1,10 +1,18 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +jest.mock('@microsoft/rush-lib', () => { + const actual: typeof import('@microsoft/rush-lib') = jest.requireActual('@microsoft/rush-lib'); + return { + ...actual, + captureWorkspaceInputFingerprintAsync: jest.fn(actual.captureWorkspaceInputFingerprintAsync) + }; +}); + import * as fs from 'node:fs'; import * as path from 'node:path'; -import { WorkspaceInputChangeTier } from '@microsoft/rush-lib'; +import { captureWorkspaceInputFingerprintAsync, WorkspaceInputChangeTier } from '@microsoft/rush-lib'; import { DaemonFrameType, decodeDaemonControlMessage, @@ -14,7 +22,7 @@ import { } from '@rushstack/rush-daemon-protocol'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; -import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; +import { DaemonGraphTestFixture, responseSnapshot } from './DaemonGraphTestFixture'; import type { DaemonRequestWireClient, ITerminalExchange } from './DaemonRequestWireTestUtilities'; import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; @@ -24,6 +32,9 @@ jest.setTimeout(60_000); const BUILD_A: string[] = ['build', '--to', 'a', '--parallelism', '3']; const RELEASE_FILE: string = 'release-serve'; const LATE_RELEASE_FILE: string = 'release-serve2'; +const inputCaptureMock: jest.MockedFunction = jest.mocked( + captureWorkspaceInputFingerprintAsync +); function delayAsync(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); @@ -137,6 +148,27 @@ async function startRequestAsync( return { exchange, positions, settled: () => settled }; } +/** + * Changes an input that only a new daemon process picks up, as an install does, and returns a function that + * restores its content. + */ +function changeInstallation(fixture: DaemonGraphTestFixture): () => void { + const filePath: string = path.join(fixture.folder, 'common/config/rush/npm-shrinkwrap.json'); + const original: string = fs.readFileSync(filePath, 'utf8'); + fs.writeFileSync(filePath, `${original}\n`); + return () => fs.writeFileSync(filePath, original); +} + +async function closeRestartingFixtureAsync(fixture: DaemonGraphTestFixture): Promise { + try { + await fixture.host.closeAsync(); + await fixture.host.restartCompleted; + } finally { + await stopSuccessorAsync(fixture.host.paths); + await fixture[Symbol.asyncDispose](); + } +} + function changeProjectConfiguration(fixture: DaemonGraphTestFixture): void { const packageJsonPath: string = path.join(fixture.folder, 'c/package.json'); const packageJson: Record = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8')); @@ -303,3 +335,218 @@ describe('workspace admission while a served rushx script runs', () => { } }); }); + +describe('a restart drain whose change is reverted', () => { + const SERVE2: Partial = { commandOrigin: 'custom', invocationKind: 'rushx' }; + + it('serves the request on this process, while the script that holds the drain still runs', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + // A transition would cancel the watcher, and a request that arrives when nothing needs one does not transition. + const watch: IStreamedRequest = await startRequestAsync(fixture, ['daemon', 'graph', 'watch'], {}); + const script: IServedScript = await serveAsync(fixture); + + const revert: () => void = changeInstallation(fixture); + const build: IStreamedRequest = await startRequestAsync(fixture, BUILD_A, { + admission: { waitTimeoutMs: 20_000 } + }); + await waitForAsync(() => build.positions.length > 0 || build.settled(), 'the build to wait for the drain'); + const late: IStreamedRequest = await startRequestAsync(fixture, ['serve2'], { + ...SERVE2, + cwd: path.join(fixture.folder, 'a') + }); + await waitForAsync(() => late.positions.length > 0 || late.settled(), 'the script to wait for the restart'); + // Several rechecks find that the change is still there. + await delayAsync(2500); + expect(build.settled()).toBe(false); + expect(fixture.runs()).not.toContain('serve2-start'); + + const revertedAt: number = Date.now(); + revert(); + // The build used to wait for the script to exit, which a dev server never does (#127). + expectSuccess(await build.exchange); + expect(Date.now() - revertedAt).toBeLessThan(5000); + expect(script.settled()).toBe(false); + // The later script no longer waits for a restart either. + await waitForAsync(() => fixture.runs().includes('serve2-start'), 'the later script to start'); + expect((await pongAsync(fixture)).pid).toBe(before.pid); + expect(fixture.host.workspaceStatus.lastReloadTier).not.toBe(WorkspaceInputChangeTier.Restart); + expect(watch.settled()).toBe(false); + + fixture.write(RELEASE_FILE, ''); + fixture.write(LATE_RELEASE_FILE, ''); + expectSuccess(await script.exchange); + expectSuccess(await late.exchange); + } finally { + fixture.write(RELEASE_FILE, ''); + fixture.write(LATE_RELEASE_FILE, ''); + await closeRestartingFixtureAsync(fixture); + } + }); + + it('serves a graph control request on this process', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + const revert: () => void = changeInstallation(fixture); + const pause: IStreamedRequest = await startRequestAsync(fixture, ['daemon', 'graph', 'pause'], { + admission: { waitTimeoutMs: 20_000 } + }); + await waitForAsync(() => pause.positions.length > 0 || pause.settled(), 'the request to wait for the drain'); + await delayAsync(1500); + expect(pause.settled()).toBe(false); + + const revertedAt: number = Date.now(); + revert(); + expect(responseSnapshot(await pause.exchange)).toMatchObject({ pauseNextIteration: true }); + expect(Date.now() - revertedAt).toBeLessThan(5000); + expect(script.settled()).toBe(false); + expect((await pongAsync(fixture)).pid).toBe(before.pid); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + } finally { + fixture.write(RELEASE_FILE, ''); + await closeRestartingFixtureAsync(fixture); + } + }); + + it('still restarts after the drain when the change stays', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + changeInstallation(fixture); + const build: IStreamedRequest = await startRequestAsync(fixture, BUILD_A, { + admission: { waitTimeoutMs: 20_000 } + }); + await waitForAsync(() => build.positions.length > 0 || build.settled(), 'the build to wait for the drain'); + await delayAsync(2500); + expect(build.settled()).toBe(false); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + expect((await build.exchange).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true } + }); + const restarted = await fixture.host.restartCompleted; + expect(restarted?.pid).not.toBe(before.pid); + } finally { + fixture.write(RELEASE_FILE, ''); + await closeRestartingFixtureAsync(fixture); + } + }); + + it('does not keep scripts waiting when the request finds after its drain that it no longer needs the restart', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + // Project c builds until the test creates release-build. + created.write( + 'c/build.cjs', + "const fs=require('node:fs');fs.appendFileSync('../runs.txt','c-start\\n');" + + "const t=setInterval(()=>{if(fs.existsSync('../release-build')){clearInterval(t);" + + "fs.appendFileSync('../runs.txt','c-end\\n');}},20);" + ); + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + const revert: () => void = changeInstallation(fixture); + const build: IStreamedRequest = await startRequestAsync(fixture, ['build', '--to', 'c'], { + admission: { waitTimeoutMs: 20_000 } + }); + await waitForAsync(() => build.positions.length > 0 || build.settled(), 'the build to wait for the drain'); + const late: IStreamedRequest = await startRequestAsync(fixture, ['serve2'], { + ...SERVE2, + cwd: path.join(fixture.folder, 'a') + }); + await waitForAsync(() => late.positions.length > 0 || late.settled(), 'the script to wait for the restart'); + + // The drain ends as the script exits, so the build most likely finds the revert only after the drain. + revert(); + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + await waitForAsync(() => fixture.runs().includes('c-start'), 'the build to start'); + // The later script used to wait for the build to finish, since its restart stayed pending. + await waitForAsync(() => fixture.runs().includes('serve2-start'), 'the later script to start'); + expect(fixture.runs()).not.toContain('c-end'); + + fixture.write('release-build', ''); + fixture.write(LATE_RELEASE_FILE, ''); + expectSuccess(await build.exchange); + expectSuccess(await late.exchange); + expect((await pongAsync(fixture)).pid).toBe(before.pid); + } finally { + fixture.write(RELEASE_FILE, ''); + fixture.write(LATE_RELEASE_FILE, ''); + fixture.write('release-build', ''); + await closeRestartingFixtureAsync(fixture); + } + }); + + it('shares one capture a second between the requests that wait for the drain', async () => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + setDaemonPolicy(created, {}); + created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + try { + const before = await pongAsync(fixture); + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + const revert: () => void = changeInstallation(fixture); + const builds: IStreamedRequest[] = []; + // The requests arrive over about a second, so each one's checks would start at another time. + for (let i: number = 0; i < 16; i++) { + builds.push(await startRequestAsync(fixture, BUILD_A, { admission: { waitTimeoutMs: 20_000 } })); + await delayAsync(60); + } + await waitForAsync( + () => builds.every((build) => build.positions.length > 0 || build.settled()), + 'every build to wait for the drain' + ); + inputCaptureMock.mockClear(); + await delayAsync(3000); + // Each waiting request used to capture its inputs once a second, 16 captures a second in all (#2239). + const captures: number = inputCaptureMock.mock.calls.length; + expect(captures).toBeGreaterThanOrEqual(2); + expect(captures).toBeLessThanOrEqual(5); + expect(builds.some((build) => build.settled())).toBe(false); + + const revertedAt: number = Date.now(); + revert(); + for (const build of builds) { + expectSuccess(await build.exchange); + } + expect(Date.now() - revertedAt).toBeLessThan(10_000); + expect(script.settled()).toBe(false); + expect((await pongAsync(fixture)).pid).toBe(before.pid); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + } finally { + fixture.write(RELEASE_FILE, ''); + await closeRestartingFixtureAsync(fixture); + } + }); +}); From 1b715309390f321776b6f0d4b943a202adeca9df Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:05 +0000 Subject: [PATCH 071/265] [rush-daemon] ODSP_TELEMETRY_TAG no longer restarts the daemon Swarm integration step 56; original commit 442af4ed14 (merge of swarm/r02-t168-int at b9b7f9dedd). Scope: task 168. Brings r02's task 168 (board 2939, re-tipped onto batch B in board 3030): setting, changing or unsetting ODSP_TELEMETRY_TAG keeps the warm daemon. Second agents: t07 board 3004 and m03 board 2975. s17 batch D1, item 4 of 10. Gate: ch01 GATE OK board 3310 (tree eadadd0318) Commits folded into this step (3): - 4a2af75d2b [rush-lib] [rush-daemon] Keep the warm daemon when a request's telemetry tag changes (task 168) - efb14d246a [rush-daemon] Log an early failed result's telemetry once its continuing work settles (task 168) - c888d079b3 [rush-lib] Test that the telemetry flush wait clears its timer (task 168) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...try-tag-keeps-daemon_2026-09-29-01-38.json | 10 + ...lt-telemetry-settles_2026-09-29-01-54.json | 10 + ...try-tag-keeps-daemon_2026-09-29-01-38.json | 10 + .../rush-daemon/src/DaemonRequestTelemetry.ts | 6 +- .../rush-daemon/src/PhasedRequestRouter.ts | 97 +++++++--- .../rush-daemon/src/PhasedRequestTelemetry.ts | 21 +- .../test/PhasedRequestEarlyFailure.test.ts | 183 +++++++++++++++--- .../ProductionDaemonRequestResolver.test.ts | 45 +++++ .../VersionSelectedDaemonLauncher.test.ts | 1 + .../test/WorkspaceReloadTierStatus.test.ts | 2 + .../src/api/WorkspaceInputFingerprint.ts | 11 +- .../test/PhasedCommandEngineTelemetry.test.ts | 10 + .../test/WorkspaceInputFingerprint.test.ts | 5 + 13 files changed, 346 insertions(+), 65 deletions(-) create mode 100644 common/changes/@microsoft/rush/telemetry-tag-keeps-daemon_2026-09-29-01-38.json create mode 100644 common/changes/@rushstack/rush-daemon/early-result-telemetry-settles_2026-09-29-01-54.json create mode 100644 common/changes/@rushstack/rush-daemon/telemetry-tag-keeps-daemon_2026-09-29-01-38.json diff --git a/common/changes/@microsoft/rush/telemetry-tag-keeps-daemon_2026-09-29-01-38.json b/common/changes/@microsoft/rush/telemetry-tag-keeps-daemon_2026-09-29-01-38.json new file mode 100644 index 0000000000..f0d7724c01 --- /dev/null +++ b/common/changes/@microsoft/rush/telemetry-tag-keeps-daemon_2026-09-29-01-38.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add `ODSP_TELEMETRY_TAG` to `workspaceFingerprintIgnoredEnvironmentVariables` and `workspaceRequestScopedEnvironmentVariables`, so a request that sets, changes or unsets its telemetry tag keeps the warm Rush daemon instead of restarting it, and the daemon no longer inherits the tag of the client that started it.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-daemon/early-result-telemetry-settles_2026-09-29-01-54.json b/common/changes/@rushstack/rush-daemon/early-result-telemetry-settles_2026-09-29-01-54.json new file mode 100644 index 0000000000..76de199743 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/early-result-telemetry-settles_2026-09-29-01-54.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "When a failed build returns early while some of its operations still run, log its telemetry entry once the iteration ends, so those operations have their final status instead of `Aborted`. The entry keeps the durations and exit code of the result that the client received.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/common/changes/@rushstack/rush-daemon/telemetry-tag-keeps-daemon_2026-09-29-01-38.json b/common/changes/@rushstack/rush-daemon/telemetry-tag-keeps-daemon_2026-09-29-01-38.json new file mode 100644 index 0000000000..0a002c9007 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/telemetry-tag-keeps-daemon_2026-09-29-01-38.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "Test that requests that set, change or unset `ODSP_TELEMETRY_TAG` keep the warm daemon, and that each telemetry entry carries its own request's tag.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/DaemonRequestTelemetry.ts b/libraries/rush-daemon/src/DaemonRequestTelemetry.ts index 728d1a6a88..d3b5d8f4f5 100644 --- a/libraries/rush-daemon/src/DaemonRequestTelemetry.ts +++ b/libraries/rush-daemon/src/DaemonRequestTelemetry.ts @@ -28,8 +28,10 @@ const RUSH_MEASURE_PREFIX: string = 'rush:'; /** * Request environment variables that attribute an entry to its caller. They are read from the request, never from - * the daemon's own environment, and are not part of the daemon's environment identity. Native Rush rejects unknown - * `RUSH_` variables, so the tag variable does not use that prefix. + * the daemon's own environment. Each is in `workspaceFingerprintIgnoredEnvironmentVariables` and + * `workspaceRequestScopedEnvironmentVariables`, so a request that sets, changes or unsets one keeps the warm + * daemon, and the daemon's own `process.env` never has them. Native Rush rejects unknown `RUSH_` variables, so the + * tag variable does not use that prefix. */ const ATTRIBUTION_VARIABLES: Readonly> = { agentSessionId: 'COPILOT_AGENT_SESSION_ID', diff --git a/libraries/rush-daemon/src/PhasedRequestRouter.ts b/libraries/rush-daemon/src/PhasedRequestRouter.ts index e7bb86417b..852beef9ed 100644 --- a/libraries/rush-daemon/src/PhasedRequestRouter.ts +++ b/libraries/rush-daemon/src/PhasedRequestRouter.ts @@ -50,6 +50,7 @@ import { import { collectPhasedRequestTelemetryRecords, type IPhasedRequestTelemetryMeasure, + type IPhasedRequestTelemetryObservations, type IPhasedRequestTelemetryRecords, type IPhasedRequestTelemetrySink } from './PhasedRequestTelemetry'; @@ -129,6 +130,8 @@ interface IBatchEntry extends IPreparedPhasedRequest { finishPromise: Promise | undefined; /** The `performance.now()` timestamp at which the entry was taken into a batch. */ joinedTimeMs: number | undefined; + /** Logs the telemetry entry of an entry that continues after its result, once its iteration ended. */ + logTelemetryAfterIteration: (() => void) | undefined; outputError: unknown; participated: boolean; reject: (error: unknown) => void; @@ -358,6 +361,7 @@ class PhasedRequestBatchCoordinator { executionStarted: false, finishPromise: undefined, joinedTimeMs: undefined, + logTelemetryAfterIteration: undefined, outputError: undefined, participated: false, reject, @@ -855,6 +859,9 @@ class PhasedRequestBatchCoordinator { } #settleContinuingEntry(entry: IBatchEntry): void { + const logTelemetry: (() => void) | undefined = entry.logTelemetryAfterIteration; + entry.logTelemetryAfterIteration = undefined; + logTelemetry?.(); const settle: (() => void) | undefined = entry.settleAfterIteration; if (!settle) { return; @@ -966,41 +973,64 @@ class PhasedRequestBatchCoordinator { } } + /** + * Reports the request to its telemetry sink, with the timing of the result that the client receives. + * + * @remarks + * For an entry that continues after its result, the operations that its failure did not block still run, so + * the report waits until the iteration ended, and `#settleContinuingEntry` sends it with their final + * statuses. + */ #logTelemetry( entry: IBatchEntry, result: IDaemonPhasedRequestResult, batchScheduled: boolean, - iterationInProgress: boolean + earlyResult: boolean ): void { const { batchTimings: timings, requestSink, telemetry } = entry; if (!telemetry || !requestSink || !timings) { return; } - try { - const resultTimeMs: number = performance.now(); - const executionStartTimeMs: number = Math.max(timings.startTimeMs, entry.joinedTimeMs ?? 0); - const { records, countRetained }: IPhasedRequestTelemetryRecords = collectPhasedRequestTelemetryRecords({ - activeOperations: entry.selection.activeOperations, - graph: this.#graph, - observations: requestSink, - upToDateTimeMs: executionStartTimeMs - }); - telemetry.logRequest({ - request: entry.request, - result, - records, - countRetained, - batchSize: timings.batchSize, - scheduled: batchScheduled, - earlyResult: iterationInProgress, - receivedTimeMs: entry.startTimeMs, - executionStartTimeMs, - iterationStartTimeMs: batchScheduled ? timings.scheduleStartTimeMs : undefined, - resultTimeMs, - measures: createTelemetryMeasures(entry, timings, executionStartTimeMs, resultTimeMs) - }); - } catch { - // Telemetry never changes a request's result. + const resultTimeMs: number = performance.now(); + const executionStartTimeMs: number = Math.max(timings.startTimeMs, entry.joinedTimeMs ?? 0); + // Measured now, because the batch timings go on changing until the iteration ends. + const measures: IPhasedRequestTelemetryMeasure[] = createTelemetryMeasures( + entry, + timings, + executionStartTimeMs, + resultTimeMs + ); + const logRequest = (observations: IPhasedRequestTelemetryObservations): void => { + try { + const { records, countRetained }: IPhasedRequestTelemetryRecords = + collectPhasedRequestTelemetryRecords({ + activeOperations: entry.selection.activeOperations, + graph: this.#graph, + observations, + upToDateTimeMs: executionStartTimeMs + }); + telemetry.logRequest({ + request: entry.request, + result, + records, + countRetained, + batchSize: timings.batchSize, + scheduled: batchScheduled, + earlyResult, + receivedTimeMs: entry.startTimeMs, + executionStartTimeMs, + iterationStartTimeMs: batchScheduled ? timings.scheduleStartTimeMs : undefined, + resultTimeMs, + measures + }); + } catch { + // Telemetry never changes a request's result. + } + }; + if (entry.continuesAfterResult) { + entry.logTelemetryAfterIteration = () => logRequest(getIterationObservations(requestSink)); + } else { + logRequest(requestSink); } } @@ -1076,6 +1106,21 @@ function createTelemetryMeasures( return measures; } +/** + * The request's operations as the iteration's own records have them. The request's sink stops observing the + * iteration when it publishes an early result, so it has not seen what happened to them since. Once the + * iteration ended, these records have their final statuses. + */ +function getIterationObservations(requestSink: PhasedRequestEventSink): IPhasedRequestTelemetryObservations { + return { + getObservedResult: (operation: Operation) => { + const executionResult: IOperationExecutionResult | undefined = + requestSink.getScheduledResult(operation); + return executionResult ? { executionResult } : requestSink.getObservedResult(operation); + } + }; +} + function createBatchReleaseBarrier( batch: ReadonlyArray, releaseAsync: () => Promise diff --git a/libraries/rush-daemon/src/PhasedRequestTelemetry.ts b/libraries/rush-daemon/src/PhasedRequestTelemetry.ts index aa1240dc21..64734b1891 100644 --- a/libraries/rush-daemon/src/PhasedRequestTelemetry.ts +++ b/libraries/rush-daemon/src/PhasedRequestTelemetry.ts @@ -35,7 +35,9 @@ export interface IPhasedRequestTelemetryReport { readonly result: IDaemonPhasedRequestResult; /** * The request's non-silent selected operations. Operations that this request did not need to run, because - * the warm graph had them up to date, are reported as `Skipped` with a zero-length stopwatch. + * the warm graph had them up to date, are reported as `Skipped` with a zero-length stopwatch. For a failed + * result that was published while operations of the request still ran, the records are collected once the + * iteration ended, so those operations have their final statuses. */ readonly records: ReadonlyMap; /** How many of `records` were already up to date. */ @@ -44,7 +46,10 @@ export interface IPhasedRequestTelemetryReport { readonly batchSize: number; /** Whether the graph scheduled an iteration for the batch. */ readonly scheduled: boolean; - /** Whether the result was produced while the shared iteration was still running for other requests. */ + /** + * Whether the result was produced while the graph iteration was still running, for other requests or for + * operations of this request that its failure did not block. + */ readonly earlyResult: boolean; /** When the router received the request. */ readonly receivedTimeMs: number; @@ -62,13 +67,19 @@ export interface IPhasedRequestTelemetryReport { * Receives one report for each phased request that took part in a graph iteration or no-op check. * * @remarks - * The router invokes the sink before it writes the request's result. The sink must not throw; the router ignores - * its errors so that telemetry never changes a result. + * The router invokes the sink before it writes the request's result, except for a failed result that it + * publishes while operations of the request that the failure did not block still run: that request is reported + * once the iteration ended, so that those operations have their final statuses. The report still has the timing + * of the result that the client received. The sink must not throw; the router ignores its errors so that telemetry + * never changes a result. * * @beta */ export interface IPhasedRequestTelemetrySink { - /** Called once, before the result is written to the client. Errors are ignored. */ + /** + * Called once for each request, before its result is written to the client, or once the iteration ended for a + * failed result that was published while operations of the request still ran. Errors are ignored. + */ logRequest(report: IPhasedRequestTelemetryReport): void; } diff --git a/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts b/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts index 4daab514cb..800d4049b0 100644 --- a/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts +++ b/libraries/rush-daemon/src/test/PhasedRequestEarlyFailure.test.ts @@ -2,9 +2,11 @@ // See LICENSE in the project root for license information. import type { IDaemonPhasedRequest, IDaemonPhasedRequestResult } from '@rushstack/rush-daemon-protocol'; -import { OperationStatus } from '@microsoft/rush-lib'; +import { type IOperationRunnerContext, OperationStatus } from '@microsoft/rush-lib'; +import type { ITerminal } from '@rushstack/terminal'; import { PhasedRequestRouter } from '../PhasedRequestRouter'; +import type { IPhasedRequestTelemetryReport, IPhasedRequestTelemetrySink } from '../PhasedRequestTelemetry'; import { TEST_ENGINE_SHAPE, TestOperationRunner, @@ -16,6 +18,7 @@ import type { ITestClientWrite, ITestRoutingFixture } from './PhasedRequestRoute const OPERATION_A: string = 'project-a (_phase:test)'; const OPERATION_B: string = 'project-b (_phase:test)'; const OPERATION_C: string = 'project-c (_phase:test)'; +const OPERATION_D: string = 'project-d (_phase:test)'; /** A consumes B and C, so A is the only target of a request that selects A. */ const A_CONSUMES_B_AND_C: ReadonlyArray = [ [OPERATION_A, OPERATION_B], @@ -76,42 +79,54 @@ class SilentTestOperationRunner extends TestOperationRunner { /** * B fails and C is slow. Unless `failBeforeCStarts` is set, B fails only once C runs, so C is executing when B's - * failure decides a request's result. + * failure decides a request's result. If `silentC` is set, C is silent. If `terminable` is set, the graph can + * terminate running operations, and C stops with `Aborted` when it does, as a runner that kills its process does. + * If `dependencies` names D, D succeeds as soon as it runs. */ function createEarlyFailureFixture( dependencies: ReadonlyArray = A_CONSUMES_B_AND_C, failBeforeCStarts: boolean = false, - silentC: boolean = false + silentC: boolean = false, + terminable: boolean = false ): IEarlyFailureFixture { const startedC: IDeferred = createDeferred(); const releaseC: IDeferred = createDeferred(); const events: string[] = []; - const fixture: ITestRoutingFixture = createRoutingFixture( - new Map([ - [OPERATION_A, new TestOperationRunner(OPERATION_A)], - [ - OPERATION_B, - new TestOperationRunner(OPERATION_B, OperationStatus.Failure, async (): Promise => { - if (!failBeforeCStarts) { - await startedC.promise; - } - }) - ], - [ + const runners: Map = new Map([ + [OPERATION_A, new TestOperationRunner(OPERATION_A)], + [ + OPERATION_B, + new TestOperationRunner(OPERATION_B, OperationStatus.Failure, async (): Promise => { + if (!failBeforeCStarts) { + await startedC.promise; + } + }) + ], + [ + OPERATION_C, + new (silentC ? SilentTestOperationRunner : TestOperationRunner)( OPERATION_C, - new (silentC ? SilentTestOperationRunner : TestOperationRunner)( - OPERATION_C, - OperationStatus.Success, - async (): Promise => { - startedC.resolve(); - await releaseC.promise; - } - ) - ] - ]), - dependencies, - { parallelism: 2 } - ); + OperationStatus.Success, + async ( + terminal: ITerminal, + { abortSignal }: IOperationRunnerContext + ): Promise => { + startedC.resolve(); + const terminated: Promise = new Promise((resolve) => { + abortSignal?.addEventListener('abort', () => resolve(OperationStatus.Aborted), { once: true }); + }); + return await Promise.race([releaseC.promise, terminated]); + } + ) + ] + ]); + if (dependencies.some((pair: readonly [string, string]) => pair.includes(OPERATION_D))) { + runners.set(OPERATION_D, new TestOperationRunner(OPERATION_D)); + } + const fixture: ITestRoutingFixture = createRoutingFixture(runners, dependencies, { + parallelism: 2, + supportsTerminateRunning: terminable + }); fixture.session.acquireExecutionLeaseAsync = async (): Promise => { events.push('acquired'); return { @@ -178,6 +193,28 @@ function getWrittenResults(client: TestPhasedRequestClient): ReadonlyArray (result ? [result] : [])); } +/** Records the telemetry reports of a request, and when each was logged. */ +function recordTelemetry( + label: string, + events: string[], + reports: IPhasedRequestTelemetryReport[] +): IPhasedRequestTelemetrySink { + return { + logRequest: (report: IPhasedRequestTelemetryReport): void => { + events.push(`logged:${label}`); + reports.push(report); + } + }; +} + +function getRecordedStatuses(report: IPhasedRequestTelemetryReport): Record { + const statuses: Record = {}; + for (const [operation, record] of report.records) { + statuses[operation.name] = record.status; + } + return statuses; +} + interface IOrdinaryCase { readonly commandName: string; readonly dependencies: ReadonlyArray; @@ -423,6 +460,96 @@ describe('phased requests that return early on failure', () => { expect(getWrittenResults(agent.client)).toHaveLength(1); }); + it("logs a request that returned early once the work that continues settled, with that work's final status", async () => { + // A consumes B and D, and D consumes C, so D starts only after the request has its result. + const setup: IEarlyFailureFixture = createEarlyFailureFixture([ + [OPERATION_A, OPERATION_B], + [OPERATION_A, OPERATION_D], + [OPERATION_D, OPERATION_C] + ]); + const { events } = setup; + const agent: ITrackedClient = trackClient('agent', setup); + const reports: IPhasedRequestTelemetryReport[] = []; + + const resultPromise: Promise = trackResult( + setup.router.executeAsync( + createRequest('agent', true, OPERATION_A), + agent.client, + false, + undefined, + undefined, + recordTelemetry('agent', events, reports) + ), + 'agent', + events + ); + await agent.written; + await settleAsync(); + // C and D still run for this request, so its entry waits for them. + expect(events).toEqual(['acquired', 'wrote:agent']); + const [early] = getWrittenResults(agent.client); + expect(early.operationResults).toEqual([ + expect.objectContaining({ operationId: OPERATION_A, status: OperationStatus.Blocked }), + expect.objectContaining({ operationId: OPERATION_B, status: OperationStatus.Failure }), + expect.objectContaining({ operationId: OPERATION_C, status: OperationStatus.Executing }), + expect.objectContaining({ operationId: OPERATION_D, status: OperationStatus.Waiting }) + ]); + + setup.releaseC(); + expect(await resultPromise).toBe(early); + expect(events).toEqual(['acquired', 'wrote:agent', 'released', 'logged:agent', 'result:agent']); + expect(reports).toHaveLength(1); + const [report] = reports; + // The entry describes the result that the client received while C and D still ran. + expect(report.result).toBe(early); + expect(report).toMatchObject({ countRetained: 0, earlyResult: true, scheduled: true }); + expect(getRecordedStatuses(report)).toEqual({ + [OPERATION_A]: OperationStatus.Blocked, + [OPERATION_B]: OperationStatus.Failure, + [OPERATION_C]: OperationStatus.Success, + [OPERATION_D]: OperationStatus.Success + }); + for (const [operation, record] of report.records) { + if (operation.name === OPERATION_C || operation.name === OPERATION_D) { + expect(record.stopwatch.endTime).toBeGreaterThan(report.resultTimeMs); + } + } + }); + + it('logs the work that continues as aborted when the request is aborted after its result and stops it', async () => { + const setup: IEarlyFailureFixture = createEarlyFailureFixture(A_CONSUMES_B_AND_C, false, false, true); + const { events } = setup; + const agent: ITrackedClient = trackClient('agent', setup); + const reports: IPhasedRequestTelemetryReport[] = []; + + const resultPromise: Promise = trackResult( + setup.router.executeAsync( + createRequest('agent', true, OPERATION_A), + agent.client, + false, + undefined, + undefined, + recordTelemetry('agent', events, reports) + ), + 'agent', + events + ); + await agent.written; + await settleAsync(); + // The daemon stops C, which nobody waits for any more. + agent.client.abortController.abort(); + await resultPromise; + + expect(events).toEqual(['acquired', 'wrote:agent', 'released', 'logged:agent', 'result:agent']); + expect(reports).toHaveLength(1); + expect(reports[0].earlyResult).toBe(true); + expect(getRecordedStatuses(reports[0])).toEqual({ + [OPERATION_A]: OperationStatus.Blocked, + [OPERATION_B]: OperationStatus.Failure, + [OPERATION_C]: OperationStatus.Aborted + }); + }); + it('rejects a flag that is not a boolean', async () => { const setup: IEarlyFailureFixture = createEarlyFailureFixture(); await expect( diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 3a8426c049..b481de5dcf 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -1749,6 +1749,51 @@ process.exit(23); } }); + it("keeps serving when a request sets, changes or unsets its telemetry tag, and logs each request's own tag", async () => { + const fixture: IFixture = await createFixtureAsync(false, 'direct', { telemetryEnabled: true }); + try { + const tags: ReadonlyArray = [ + ['untagged', undefined], + ['tagged', 'nightly-7'], + ['retagged', 'nightly-8'], + ['untagged-again', undefined] + ]; + let session: WorkspaceSession | undefined; + let graph: IOperationGraph | undefined; + for (const [requestId, tag] of tags) { + const environment: Record = requestEnvironment(); + delete environment.ODSP_TELEMETRY_TAG; + if (tag !== undefined) environment.ODSP_TELEMETRY_TAG = tag; + expect( + (await runAsync(fixture, requestId, ['build', '--only', 'a'], { environment })).terminal + ).toMatchObject({ kind: 'requestResult', payload: { exitCode: 0 } }); + session ??= fixture.session; + graph ??= fixture.session.operationGraph; + expect(fixture.session).toBe(session); + expect(fixture.session.operationGraph).toBe(graph); + expect(readDaemonLockfile(fixture.host.paths.lockfilePath)?.pid).toBe(process.pid); + } + + const entries: ITelemetryData[] = readTelemetryEntries(fixture.repoRoot); + expect( + entries.map(({ extraData }) => [ + extraData?.requestId, + extraData?.requestIndex, + extraData?.graphWasInitialized, + extraData?.telemetryTag + ]) + ).toEqual([ + ['untagged', 1, false, undefined], + ['tagged', 2, true, 'nightly-7'], + ['retagged', 3, true, 'nightly-8'], + ['untagged-again', 4, true, undefined] + ]); + expect(runs(fixture)).toEqual(['a:one:']); + } finally { + await fixture[Symbol.asyncDispose](); + } + }); + it('releases the lockfile within seconds when a flushTelemetry tap never settles', async () => { const stalledUploads: string[] = []; const createEngineAsync = PhasedCommandEngine.prototype.createEngineAsync; diff --git a/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts b/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts index 5458cc0ad3..4b8554b5d4 100644 --- a/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts +++ b/libraries/rush-daemon/src/test/VersionSelectedDaemonLauncher.test.ts @@ -98,6 +98,7 @@ describe('version-selected daemon launcher', () => { HOME: '/home/user', RUSH_PARALLELISM: '48', COPILOT_AGENT_SESSION_ID: 'session-1', + ODSP_TELEMETRY_TAG: 'tag-1', RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', RUSHD_OUTPUT: 'agent', // The first client's session folders may disappear while the daemon lives on. diff --git a/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts b/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts index b8d770ac40..ae0b3cc932 100644 --- a/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceReloadTierStatus.test.ts @@ -97,6 +97,8 @@ it('reuses the warm generation when only volatile per-shell environment variable COPILOT_AGENT_SESSION_ID: 'another-session', VSCODE_IPC_HOOK_CLI: '/run/vscode-ipc.sock' }, + telemetryTag: { ...fixture.environment, ODSP_TELEMETRY_TAG: 'nightly-7' }, + anotherTelemetryTag: { ...fixture.environment, ODSP_TELEMETRY_TAG: 'nightly-8' }, repeatedPath: { ...fixture.environment, PATH: [fixture.environment.PATH, fixture.environment.PATH].join(path.delimiter) diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index 47d5df524c..4530644a53 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -65,6 +65,7 @@ export interface IWorkspaceInputFingerprintOptions { * `VSCODE_GIT_ASKPASS_EXTRA_ARGS`, `VSCODE_INJECTION`, `VSCODE_NONCE`, `GIT_ASKPASS`, `SSH_ASKPASS` * - coding agent session markers: `COPILOT_CLI`, `COPILOT_AGENT_SESSION_ID`, `COPILOT_LOADER_PID`, * `COPILOT_CLI_BINARY_VERSION`, `COPILOT_CLI_RESOLVED_DIST_DIR`, `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT` + * - `ODSP_TELEMETRY_TAG`, which tags the telemetry entry of one command with its caller's label * - `INIT_CWD`, which Rush removes from every lifecycle script environment and sets explicitly where needed, * and `RUSH_INVOKED_FOLDER`, which Rush assigns for each invocation * - client routing and presentation: `RUSH_DAEMON` and `RUSH_DAEMON_AUTO_START` only select and start a daemon, @@ -142,6 +143,7 @@ export const workspaceFingerprintIgnoredEnvironmentVariables: ReadonlySet = new Set([ 'RUSH_PARALLELISM', 'COPILOT_AGENT_SESSION_ID', + 'ODSP_TELEMETRY_TAG', 'TMPDIR', 'XDG_RUNTIME_DIR' ]); diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts index 3e9931220d..c304182674 100644 --- a/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngineTelemetry.test.ts @@ -236,6 +236,16 @@ describe(waitForTelemetryFlushAsync.name, () => { ); }); + it('clears its timer when the taps settle first, so that the timer does not keep the process alive', async () => { + jest.useFakeTimers(); + try { + await expect(waitForTelemetryFlushAsync(Promise.resolve(), 60_000)).resolves.toBe(true); + expect(jest.getTimerCount()).toBe(0); + } finally { + jest.useRealTimers(); + } + }); + it('stops waiting for taps that never settle, such as an upload over a stalled network', async () => { const startMs: number = Date.now(); diff --git a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts index f31d382118..203d82216f 100644 --- a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts +++ b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts @@ -220,6 +220,7 @@ describe('workspace input fingerprints', () => { COPILOT_LOADER_PID: '77', WT_SESSION: 'w' }, + { ODSP_TELEMETRY_TAG: 'nightly-7' }, { PATH: `${base.PATH}${path.delimiter}${base.PATH}` }, { TERM: undefined, PWD: undefined }, { TMPDIR: '/scratch/job-1', XDG_RUNTIME_DIR: '/run/user/1000' }, @@ -262,6 +263,7 @@ describe('workspace input fingerprints', () => { HOME: '/home/user', RUSH_PARALLELISM: '48', COPILOT_AGENT_SESSION_ID: 'session-1', + ODSP_TELEMETRY_TAG: 'tag-1', RUSHD_OUTPUT: 'agent', RUSH_DAEMON_IDLE_TIMEOUT_SECONDS: '86400', TMPDIR: '/scratch/job-1', @@ -288,6 +290,7 @@ describe('workspace input fingerprints', () => { PATH: '/usr/bin', NODE_OPTIONS: '--max-old-space-size=8192', COPILOT_AGENT_SESSION_ID: 'session-A', + ODSP_TELEMETRY_TAG: 'tag-A', COPILOT_CLI: '1', GIT_ASKPASS: '/window-A/askpass.sh', WT_SESSION: 'wt-A', @@ -297,6 +300,7 @@ describe('workspace input fingerprints', () => { HOME: '/home/other', PATH: '/other/bin', COPILOT_AGENT_SESSION_ID: 'session-B', + ODSP_TELEMETRY_TAG: 'tag-B', RUSH_PARALLELISM: '2', WT_SESSION: 'wt-B', RUSH_INVOKED_FOLDER: '/repo/apps/b', @@ -307,6 +311,7 @@ describe('workspace input fingerprints', () => { PATH: '/usr/bin', NODE_OPTIONS: '--max-old-space-size=8192', COPILOT_AGENT_SESSION_ID: 'session-B', + ODSP_TELEMETRY_TAG: 'tag-B', RUSH_PARALLELISM: '2', WT_SESSION: 'wt-B', RUSH_INVOKED_FOLDER: '/repo/apps/b' From c40f21d324c9c131051518186b9e9d61e78e1be3 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:05 +0000 Subject: [PATCH 072/265] [rush-daemon] The daemon's failure summary names the error, not the first buffered warning Swarm integration step 57; original commit edbe3f9065 (merge of swarm/r04-t170-int at 6219ed8e65). Scope: task 170. Brings r04's task 170 (board 2953) with its NIT fix 9630e0bd88 (board 3036), re-tipped onto batch B. Second agent: t01 board 2966 and board 3045. s17 batch D1, item 5 of 10. Gate: ch01 GATE OK board 3310 (tree cbdb7a5bb7) Commits folded into this step (3): - 781d08b278 Give the error first in a rejected request's message, not a warning written before it (task 170) - f1d8a34c90 Put the in-process fallback on the first line of the reason (task 170) - 9630e0bd88 Drop a colon that ends the fallback reason's first line (task 170, t01 NIT) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/src/inProcessFallback.ts | 16 ++++++ apps/rush-cli-client/src/launchClient.ts | 3 +- .../src/test/inProcessFallback.test.ts | 49 +++++++++++++++++++ ...4-t170-fallback-line_2026-09-29-02-10.json | 11 +++++ ...r04-t170-error-first_2026-09-29-02-05.json | 11 +++++ .../rush-daemon/src/EngineTerminalProvider.ts | 21 +++++--- .../src/test/EngineTerminalProvider.test.ts | 29 +++++++++-- .../src/test/NativeEngineTestFixture.ts | 5 +- .../ProductionDaemonRequestResolver.test.ts | 29 ++++++++++- 9 files changed, 161 insertions(+), 13 deletions(-) create mode 100644 apps/rush-cli-client/src/inProcessFallback.ts create mode 100644 apps/rush-cli-client/src/test/inProcessFallback.test.ts create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t170-fallback-line_2026-09-29-02-10.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t170-error-first_2026-09-29-02-05.json diff --git a/apps/rush-cli-client/src/inProcessFallback.ts b/apps/rush-cli-client/src/inProcessFallback.ts new file mode 100644 index 0000000000..e39f130268 --- /dev/null +++ b/apps/rush-cli-client/src/inProcessFallback.ts @@ -0,0 +1,16 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * Formats the stderr lines that say why the client runs a command in-process instead of on the daemon. The first + * line of the reason is the fallback line, which starts with `rush-client:` and ends with `using in-process Rush.` + * The other lines of a multi-line reason, such as a warning that the daemon wrote before its error, follow it as + * detail lines. + */ +export function formatInProcessFallbackMessage(reason: string): string { + const [firstLine = '', ...details] = reason.split(/\r?\n/).filter((line) => line.trim()); + // A reason is usually a sentence, or a line that introduces the lines after it; "; using" follows it without the + // period or the colon. + const fallbackLine: string = `rush-client: ${firstLine.replace(/[.:]$/, '')}; using in-process Rush.`; + return [fallbackLine, ...details.map((line) => ` ${line}`)].map((line) => `${line}\n`).join(''); +} diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index 308dba115b..e50e7f0355 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -41,6 +41,7 @@ import { selectClientRoute, type IClientRoute } from './routing'; import { getResultStderr } from './resultDiagnostics'; import { getTerminalColumns } from './terminalColumns'; import { createDaemonRequestNoticeHandlers } from './daemonRestartNotice'; +import { formatInProcessFallbackMessage } from './inProcessFallback'; import { writeStreamAsync } from './writeStreamAsync'; import { getBundledRushVersion, @@ -307,7 +308,7 @@ export async function launchClientAsync( throw new Error(message); } else { agentRenderer?.dispose(); - process.stderr.write(`rush-client: ${outcome.message ?? outcome.reason}; using in-process Rush.\n`); + process.stderr.write(formatInProcessFallbackMessage(outcome.message ?? outcome.reason)); await launchInProcessAsync(route.argv, rushx, selectedVersion, rushJsonPath); } } diff --git a/apps/rush-cli-client/src/test/inProcessFallback.test.ts b/apps/rush-cli-client/src/test/inProcessFallback.test.ts new file mode 100644 index 0000000000..128f86b5a1 --- /dev/null +++ b/apps/rush-cli-client/src/test/inProcessFallback.test.ts @@ -0,0 +1,49 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { formatInProcessFallbackMessage } from '../inProcessFallback'; + +describe(formatInProcessFallbackMessage.name, () => { + it('ends the reason with the fallback, without a period before the semicolon', () => { + expect( + formatInProcessFallbackMessage( + 'The daemon does not support explicit Rushx invocations; no request was sent.' + ) + ).toBe( + 'rush-client: The daemon does not support explicit Rushx invocations; no request was sent; ' + + 'using in-process Rush.\n' + ); + expect(formatInProcessFallbackMessage('controllingTerminalRequired')).toBe( + 'rush-client: controllingTerminalRequired; using in-process Rush.\n' + ); + }); + + it('gives the first line of a multi-line reason on the fallback line and the other lines as details', () => { + expect( + formatInProcessFallbackMessage( + 'Plugins must be declared daemon-compatible. Use --no-daemon.\n\n' + + 'The compatible plugin list names plugins that are not configured: "p".\n' + ) + ).toBe( + 'rush-client: Plugins must be declared daemon-compatible. Use --no-daemon; using in-process Rush.\n' + + ' The compatible plugin list names plugins that are not configured: "p".\n' + ); + }); + + it('drops the colon of a first line that introduces the lines after it', () => { + expect( + formatInProcessFallbackMessage( + 'Error reading "/repo/common/config/rush/command-line.json":\n Unexpected token } at 3:1\n}\n^' + ) + ).toBe( + 'rush-client: Error reading "/repo/common/config/rush/command-line.json"; using in-process Rush.\n' + + ' Unexpected token } at 3:1\n' + + ' }\n' + + ' ^\n' + ); + // JsonFile ends the line with os.EOL. + expect(formatInProcessFallbackMessage('Error reading "x.json":\r\n Unexpected token')).toBe( + 'rush-client: Error reading "x.json"; using in-process Rush.\n Unexpected token\n' + ); + }); +}); diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t170-fallback-line_2026-09-29-02-10.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t170-fallback-line_2026-09-29-02-10.json new file mode 100644 index 0000000000..f1e23d3f42 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t170-fallback-line_2026-09-29-02-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "When the daemon can't serve a request, the line that says the client uses in-process Rush now carries the first line of the reason, and further lines of the reason follow it, indented. The reason's final period or colon no longer comes before \"; using in-process Rush.\"", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t170-error-first_2026-09-29-02-05.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t170-error-first_2026-09-29-02-05.json new file mode 100644 index 0000000000..a56b58b37b --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t170-error-first_2026-09-29-02-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "When the daemon rejects a request, the error now comes first in the message, so that the client gives it as the reason. Warnings written before the error, such as one about the compatible plugin list, follow it instead of taking its place in the summary line.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/EngineTerminalProvider.ts b/libraries/rush-daemon/src/EngineTerminalProvider.ts index 403a5c51b8..8b42edaf82 100644 --- a/libraries/rush-daemon/src/EngineTerminalProvider.ts +++ b/libraries/rush-daemon/src/EngineTerminalProvider.ts @@ -25,21 +25,30 @@ export class EngineTerminalProvider implements ITerminalProvider { * Drains buffered diagnostics into the failure description, so that they belong to the failing request * and are never replayed into a later request. Like the request output, the description omits verbose and * debug messages unless the graph runs in debug mode: loading a large workspace writes thousands of them. + * + * The error comes first, because clients give the first line as the reason for the failure. The other + * diagnostics follow in the order they were written: a warning written while parsing the command line, such + * as one about the compatible plugin list, is not the reason for an error that selecting operations throws. */ public describeError(error: unknown): string { const lines: string[] = []; - let hasErrorLine: boolean = false; + let firstErrorLineIndex: number = -1; for (const { text, severity } of this.#messages.splice(0)) { const line: string = text.replace(/\r?\n$/, ''); if (this.#isHidden(severity) || !line.trim()) continue; - hasErrorLine ||= severity === TerminalProviderSeverity.error; + if (firstErrorLineIndex < 0 && severity === TerminalProviderSeverity.error) { + firstErrorLineIndex = lines.length; + } lines.push(line); } // An AlreadyReportedError only says "An error occurred."; the error lines written before it are the report. - if (!(hasErrorLine && error instanceof Error && error instanceof AlreadyReportedError)) { - lines.push(error instanceof Error ? error.message : String(error)); - } - return lines.join('\n'); + const reason: string = + firstErrorLineIndex >= 0 && error instanceof Error && error instanceof AlreadyReportedError + ? lines.splice(firstErrorLineIndex, 1)[0] + : error instanceof Error + ? error.message + : String(error); + return [reason, ...lines].join('\n'); } public get hasBufferedMessages(): boolean { diff --git a/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts b/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts index 549b8dd99c..ea9f908169 100644 --- a/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts +++ b/libraries/rush-daemon/src/test/EngineTerminalProvider.test.ts @@ -12,7 +12,7 @@ describe(EngineTerminalProvider.name, () => { const terminal: EngineTerminalProvider = new EngineTerminalProvider(); terminal.write('Permission denied', TerminalProviderSeverity.error); expect(terminal.hasBufferedMessages).toBe(true); - expect(terminal.describeError(new Error('snapshot failed'))).toBe('Permission denied\nsnapshot failed'); + expect(terminal.describeError(new Error('snapshot failed'))).toBe('snapshot failed\nPermission denied'); expect(terminal.hasBufferedMessages).toBe(false); expect(terminal.describeError(new Error('next request'))).toBe('next request'); }); @@ -35,7 +35,7 @@ describe(EngineTerminalProvider.name, () => { throw failure; }) ).rejects.toBe(failure); - expect(failure.message).toBe('binding request diagnostic\nPermission denied\ncould not capture'); + expect(failure.message).toBe('could not capture\nbinding request diagnostic\nPermission denied'); terminal.write('stale', TerminalProviderSeverity.warning); await expect(terminal.reconcileWithRequestDiagnosticsAsync(async () => 'ok')).resolves.toBe('ok'); @@ -54,7 +54,7 @@ describe(EngineTerminalProvider.name, () => { terminal.write('\n', TerminalProviderSeverity.log); terminal.write('Project "a" has no "build" script.\n', TerminalProviderSeverity.warning); expect(terminal.describeError(new Error('selection failed'))).toBe( - 'Project "a" has no "build" script.\nselection failed' + 'selection failed\nProject "a" has no "build" script.' ); }); @@ -71,10 +71,31 @@ describe(EngineTerminalProvider.name, () => { terminal.write('No error line was written.\n', TerminalProviderSeverity.warning); expect(terminal.describeError(new AlreadyReportedError())).toBe( - 'No error line was written.\nAn error occurred.' + 'An error occurred.\nNo error line was written.' ); }); + it('gives the error first, since clients give the first line as the reason, and the other lines in order', () => { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + const warning: string = + `The daemon's compatible plugin list names plugins that are not configured in rush-plugins.json: ` + + `"no-such-plugin".`; + const unknownProject: string = 'The project name "@x/nope" passed to "--to" does not exist in rush.json.'; + terminal.write(`${warning}\n`, TerminalProviderSeverity.warning); + terminal.write(`${unknownProject}\n`, TerminalProviderSeverity.error); + expect(terminal.describeError(new AlreadyReportedError())).toBe(`${unknownProject}\n${warning}`); + + terminal.write(`${warning}\n`, TerminalProviderSeverity.warning); + expect(terminal.describeError(new Error('Plugins must be daemon-compatible.'))).toBe( + `Plugins must be daemon-compatible.\n${warning}` + ); + + terminal.write(`${warning}\n`, TerminalProviderSeverity.warning); + terminal.write('first error\n', TerminalProviderSeverity.error); + terminal.write('second error\n', TerminalProviderSeverity.error); + expect(terminal.describeError(new AlreadyReportedError())).toBe(`first error\n${warning}\nsecond error`); + }); + it('drops diagnostics when the engine must be recreated', async () => { const terminal: EngineTerminalProvider = new EngineTerminalProvider(); const recreate: WorkspaceEngineRecreationRequiredError = new WorkspaceEngineRecreationRequiredError(); diff --git a/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts b/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts index 30eb93eb2a..5013e5cfdd 100644 --- a/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts +++ b/libraries/rush-daemon/src/test/NativeEngineTestFixture.ts @@ -54,6 +54,8 @@ export interface IFixtureOptions { /** Uses PNPM, which installs a dependency file (shrinkwrap-deps.json) that change detection hashes per project. */ readonly pnpm?: boolean; readonly telemetryEnabled?: boolean; + /** Sets `daemon.compatiblePlugins` in rush.json. */ + readonly compatiblePlugins?: ReadonlyArray; } export class DecoratedTestResolver implements IDaemonRequestResolver { @@ -117,7 +119,8 @@ export async function createFixtureAsync( // Retention assertions must not depend on the surrounding Jest worker's accumulated RSS. daemon: { warmMemoryBudgetMB: 100_000, - ...(options.incrementalBuilds === undefined ? {} : { incrementalBuilds: options.incrementalBuilds }) + ...(options.incrementalBuilds === undefined ? {} : { incrementalBuilds: options.incrementalBuilds }), + ...(options.compatiblePlugins === undefined ? {} : { compatiblePlugins: options.compatiblePlugins }) }, ...(options.telemetryEnabled ? { telemetryEnabled: true } : {}), projectFolderMinDepth: 2, diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index b481de5dcf..49cccacad6 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -191,6 +191,33 @@ describe('native production daemon engine', () => { } }); + it('gives the error as the reason for an invalid selection, before a warning that the parse wrote', async () => { + const fixture: IFixture = await createFixtureAsync(false, 'direct', { + compatiblePlugins: ['no-such-plugin'] + }); + // The binding request also writes the warning to the daemon's own stderr, which is the launcher log. + const stderr: jest.SpyInstance = jest.spyOn(process.stderr, 'write').mockReturnValue(true); + try { + const rejection: { kind: string; payload: { code: string; message: string } } = { + kind: 'requestRejected', + payload: { + code: 'invalidRequest', + message: + 'The project name "nope" passed to "--to" does not exist in rush.json.\n' + + `The daemon's compatible plugin list (rush.json "daemon.compatiblePlugins" or ` + + 'RUSH_DAEMON_COMPATIBLE_PLUGINS) names plugins that are not configured in rush-plugins.json: ' + + `"no-such-plugin". Check that each entry is the plugin's "pluginName".` + } + }; + expect((await runAsync(fixture, 'cold', ['build', '--to', 'nope'])).terminal).toMatchObject(rejection); + expect((await runAsync(fixture, 'warm', ['build', '--to', 'nope'])).terminal).toMatchObject(rejection); + expect(runs(fixture)).toEqual([]); + } finally { + stderr.mockRestore(); + await fixture[Symbol.asyncDispose](); + } + }); + it('keeps tier0 session and graph identity for unchanged content, including metadata touches', async () => { const fixture: IFixture = await createFixtureAsync(); try { @@ -1622,7 +1649,7 @@ process.exit(23); kind: 'requestRejected', payload: { message: expect.stringMatching( - /Permission denied[\s\S]*Rush could not capture the next workspace inputs snapshot\./ + /^Rush could not capture the next workspace inputs snapshot\.\n[\s\S]*Permission denied/ ) } }); From 4484f46654c2f8082a513dcf6217848571b41dd7 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:06 +0000 Subject: [PATCH 073/265] [rush-daemon] Check the installation again after a restart's waits Swarm integration step 58; original commit 13c4debd1f (merge of swarm/r03-t91g at b512743133). Scope: task 167. Brings r03's task 167 (board 2919): tests for ch01's surviving mutants on tasks 91 and 125, and rush-daemon checks the installation again after a restart's waits, before planning a successor. Second agent: t05 board 3064. s17 batch D1, item 6 of 10. Gate: ch01 GATE OK board 3310 (tree 6caa39900a) Commits folded into this step (1): - b512743133 rush-daemon: check the installation again after a restart's waits, before planning a successor (task 167) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...n-restart-late-check_2026-09-29-01-45.json | 11 ++ libraries/rush-daemon/README.md | 4 +- .../src/WorkspaceRequestLifecycle.ts | 5 +- .../src/test/DaemonInstallationChange.test.ts | 70 ++++++++++ .../test/WorkspaceEarlyFailureResult.test.ts | 129 +++++++++++++++++- 5 files changed, 216 insertions(+), 3 deletions(-) create mode 100644 common/changes/@rushstack/rush-daemon/r03-installation-restart-late-check_2026-09-29-01-45.json diff --git a/common/changes/@rushstack/rush-daemon/r03-installation-restart-late-check_2026-09-29-01-45.json b/common/changes/@rushstack/rush-daemon/r03-installation-restart-late-check_2026-09-29-01-45.json new file mode 100644 index 0000000000..5ccde590bb --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/r03-installation-restart-late-check_2026-09-29-01-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A build or graph control request that restarts the daemon for its inputs checks the daemon's installation again after its waits end, just before it plans a successor. If the installation was removed or replaced during those waits, the request now gets the typed restart result for that change, and the daemon exits without a successor, as it does for every other request, instead of starting one with its own launcher.", + "type": "patch" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 04d65fa115..48030e4ef6 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -240,7 +240,9 @@ for an environment: a client-default `waitTimeoutMs` does not limit waiting for being served when the wait began, as long as no rushx script is being served, and a timeout names the changed folder. A request that times out there, or that sets `noWait`, gets its admission error code and requests no restart. The first request that gets the result makes the daemon exit without selecting a successor, and each -client starts one with its own launcher. Embedded hosts opt in with `checkInstallation` +client starts one with its own launcher. A build or graph control request that waits to restart the daemon for its +inputs checks the installation again once those waits end, just before it would select a successor, so a change +during them gets the same result. Embedded hosts opt in with `checkInstallation` (`captureDaemonInstallation`). The daemon log (`onLog`) gets one line for the change and one for each rejected request, with its code, its message and, for an unexpected `routingFailed`, the stack. diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index fb2909f8f2..b028c0ea08 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -543,7 +543,6 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { } this.#cancelObservers(); lease = await admission.acquireAsync(this.#gate, RequestExclusivityClass.Exclusive); - this.#throwIfInstallationChanged(); await this.#waitForServedScriptsAsync(admission); session = await this.#options.provider.getSessionAsync(); await this.#quiesceWarmSetAsync(session); @@ -552,6 +551,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { RequestExclusivityClass.Exclusive ); try { + // After every wait: a daemon whose installation changed leaves selecting a successor to its clients. + this.#throwIfInstallationChanged(); const plan: IWorkspaceProcessRestartPlan = await this.#restartPlanAsync( session, controlEnvelope, @@ -670,6 +671,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { RequestExclusivityClass.Exclusive ); try { + // After every wait: a daemon whose installation changed leaves selecting a successor to its clients. + this.#throwIfInstallationChanged(); const plan: IWorkspaceProcessRestartPlan = await this.#restartPlanAsync( session, envelope, diff --git a/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts b/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts index febcf65b25..c03a3f920f 100644 --- a/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts +++ b/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts @@ -18,6 +18,8 @@ import { } from '@rushstack/rush-daemon-protocol'; import { captureDaemonInstallation, type CheckDaemonInstallation } from '../DaemonInstallationMonitor'; +import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; +import type { WorkspaceSession } from '../WorkspaceSession'; import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; import { createDeferred, @@ -27,6 +29,7 @@ import { } from './DaemonRequestWireTestUtilities'; import { assertSuccessfulNativeBuild } from './NativeBuildTestResult'; import { pongAsync, setDaemonPolicy } from './WarmGenerationTestUtilities'; +import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; jest.setTimeout(30_000); @@ -423,6 +426,73 @@ describe('a daemon whose installation changed', () => { ]); }); + it('answers a graph watch that arrives after the change with the restart, like any other request', async () => { + const current: { change?: IDaemonInstallationChange } = {}; + fixture = await DaemonGraphTestFixture.createAsync((created) => { + setDaemonPolicy(created, {}); + created.checkInstallation = () => current.change; + }); + await fixture.buildSuccessfullyAsync(); + current.change = { change: 'removed', folder: installation.folder }; + + // A watch has no restart ticket, since a restart cancels it, until it waits for this restart itself. + const { terminal } = await fixture.graphAsync('watch'); + expect(terminal).toMatchObject({ + kind: 'requestResult', + payload: { + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'removed', folder: installation.folder } + } + }); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + expect(fixture.runs()).toEqual(['a', 'b']); + }); + + it.each([ + { kind: 'a graph control request', argv: ['daemon', 'graph', 'pause'] }, + { kind: 'a build', argv: BUILD_B } + ])( + 'answers $kind that waited to restart the daemon for its inputs with the restart for a change during that wait', + async ({ argv }: { argv: string[] }) => { + const current: { change?: IDaemonInstallationChange } = {}; + const created: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((configured) => { + setDaemonPolicy(configured, {}); + configured.checkInstallation = () => current.change; + configured.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; + }); + fixture = created; + try { + await created.buildSuccessfullyAsync(); + // Another install state needs a new daemon process, so the request would select a successor. + created.write('common/temp/last-install.flag', '{}'); + // The installation changes after the request's last check before its waits end, while the warm set stops. + const session: WorkspaceSession = created.session; + const quiesceAsync: () => Promise = session.quiesceWarmSetAsync.bind(session); + jest.spyOn(session, 'quiesceWarmSetAsync').mockImplementation(async () => { + current.change ??= { change: 'replaced', folder: installation.folder }; + await quiesceAsync(); + }); + + const { terminal } = await created.runAsync(argv); + expect(terminal).toMatchObject({ + kind: 'requestResult', + payload: { + retryAfterRestart: true, + restartReason: { kind: 'installationChanged', change: 'replaced', folder: installation.folder } + } + }); + // The daemon exits without a successor, which each client then starts with its own launcher. + await expect(created.host.restartCompleted).resolves.toBeUndefined(); + expect(created.runs()).toEqual(['a', 'b']); + } finally { + jest.restoreAllMocks(); + await created.host.closeAsync(); + await created.host.restartCompleted; + await stopSuccessorAsync(created.host.paths); + } + } + ); + it('keeps serving while its installation is intact', async () => { fixture = await DaemonGraphTestFixture.createAsync((created) => { setDaemonPolicy(created, {}); diff --git a/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts b/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts index b5b2443584..b924f9480a 100644 --- a/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceEarlyFailureResult.test.ts @@ -7,8 +7,18 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; import { OperationStatus } from '@microsoft/rush-lib'; import { LockFile } from '@rushstack/node-core-library'; -import type { IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; +import { + DaemonFrameType, + decodeDaemonControlMessage, + type DaemonControlMessage, + type IDaemonFrame, + type IDaemonInstallationChange, + type IDaemonRequestEnvelope, + type IDaemonRequestQueuePositionMessage +} from '@rushstack/rush-daemon-protocol'; +import type { IResolveDaemonRequestOptions, ResolvedDaemonRequest } from '../DaemonRequestDispatcher'; +import { ProductionDaemonRequestResolver } from '../ProductionDaemonRequestResolver'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; import type { DaemonRequestWireClient, ITerminalExchange } from './DaemonRequestWireTestUtilities'; @@ -17,6 +27,9 @@ import { stopSuccessorAsync } from './WorkspaceLifecycleTestProcess'; jest.setTimeout(60_000); +/** What the daemon's check of its own installation reports; each test that sets it clears it again. */ +let installationChange: IDaemonInstallationChange | undefined; + const BUILD_B: string[] = ['build', '--to', 'b', '--parallelism', '3']; interface IEarlyFailureFixtureOptions { @@ -48,6 +61,7 @@ function createEarlyFailureFixtureAsync({ created.getSuccessorLaunchAsync = getInstalledWorkspaceSuccessorLaunchAsync; } created.write('hold', ''); + created.checkInstallation = () => installationChange; created.write( 'b/package.json', JSON.stringify({ @@ -112,6 +126,22 @@ async function isSettledAsync(promise: Promise): Promise { return settled; } +function queuePositions( + frames: ReadonlyArray +): IDaemonRequestQueuePositionMessage['payload'][] { + return frames + .filter((frame: IDaemonFrame) => frame.kind === DaemonFrameType.controlJson) + .map((frame: IDaemonFrame) => decodeDaemonControlMessage(frame.payload)) + .filter((message: DaemonControlMessage) => message.kind === 'queuePosition') + .map((message: DaemonControlMessage) => (message as IDaemonRequestQueuePositionMessage).payload); +} + +async function readUntilQueuedAsync(client: DaemonRequestWireClient): Promise { + const frames: IDaemonFrame[] = []; + while (queuePositions(frames).length === 0) frames.push(await client.readFrameAsync()); + return frames; +} + async function returnEarlyAsync(fixture: DaemonGraphTestFixture): Promise { const early: ITerminalExchange = await fixture.runAsync(BUILD_B, { returnEarlyOnFailure: true }); expect(early.terminal).toMatchObject({ @@ -326,4 +356,101 @@ describe('a failed build that returns early', () => { } } }); + + it('stops the work that continues before it answers with the restart for a changed installation', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); + const hold: string = path.join(fixture.folder, 'hold'); + try { + await returnEarlyAsync(fixture); + const change: IDaemonInstallationChange = { + change: 'replaced', + folder: path.join(fixture.folder, 'daemon') + }; + installationChange = change; + + const { frames, terminal } = await fixture.runAsync(['build', '--to', 'c', '--parallelism', '3']); + const restartReason: object = { kind: 'installationChanged', ...change }; + expect(terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true, restartReason } + }); + // It waited only while the held c stopped, and it said why it waited. + expect(queuePositions(frames)).toEqual([expect.objectContaining({ position: 1, restartReason })]); + expect(fs.existsSync(hold)).toBe(true); + expect(fixture.session.operationGraph?.status).not.toBe(OperationStatus.Executing); + expect(isNativeLockFree(fixture)).toBe(true); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + } finally { + installationChange = undefined; + fs.rmSync(hold, { force: true }); + await fixture[Symbol.asyncDispose](); + } + }); + + it('answers a command that it rejects with the restart once the installation changed, and leaves the work that continues running', async () => { + const fixture: DaemonGraphTestFixture = await createEarlyFailureFixtureAsync(); + const hold: string = path.join(fixture.folder, 'hold'); + let later: DaemonRequestWireClient | undefined; + let rejected: DaemonRequestWireClient | undefined; + try { + await returnEarlyAsync(fixture); + // A later build waits for the held c, so the daemon serves it until that c finishes. + later = await fixture.connectAsync(); + const build: IDaemonRequestEnvelope = fixture.envelope(['build', '--to', 'c', '--parallelism', '3']); + await later.sendControlAsync({ kind: 'requestStart', payload: build }); + await readUntilQueuedAsync(later); + + // The installation changes while the daemon rejects a command that the client would run in-process. + const change: IDaemonInstallationChange = { + change: 'replaced', + folder: path.join(fixture.folder, 'daemon') + }; + const custom: IDaemonRequestEnvelope = fixture.envelope(['test', '--to', 'c'], { + commandOrigin: 'custom' + }); + const resolveAsync: ProductionDaemonRequestResolver['resolveRequestAsync'] = + ProductionDaemonRequestResolver.prototype.resolveRequestAsync; + jest + .spyOn(ProductionDaemonRequestResolver.prototype, 'resolveRequestAsync') + .mockImplementation(async function ( + this: ProductionDaemonRequestResolver, + options: IResolveDaemonRequestOptions + ) { + if (options.envelope.requestId === custom.requestId) installationChange = change; + const resolved: ResolvedDaemonRequest = await resolveAsync.call(this, options); + return resolved; + }); + rejected = await fixture.connectAsync(); + await rejected.sendControlAsync({ kind: 'requestStart', payload: custom }); + + // So the client restarts the daemon instead of running the command in-process, and the restart waits for + // the later build, which still waits for the held c. + const restartReason: object = { kind: 'installationChanged', ...change }; + expect(queuePositions(await readUntilQueuedAsync(rejected))).toEqual([ + expect.objectContaining({ position: 1, restartReason }) + ]); + expect(fixture.session.operationGraph?.status).toBe(OperationStatus.Executing); + expect(countRuns(fixture, 'c')).toBe(1); + + fs.rmSync(hold); + expect((await later.readTerminalAsync(build.requestId)).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0 } + }); + expect((await rejected.readTerminalAsync(custom.requestId)).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, retryAfterRestart: true, restartReason } + }); + // The held c finished, and the later build found it done. + expect(countRuns(fixture, 'c')).toBe(1); + await expect(fixture.host.restartCompleted).resolves.toBeUndefined(); + } finally { + installationChange = undefined; + jest.restoreAllMocks(); + fs.rmSync(hold, { force: true }); + await later?.closeAsync(); + await rejected?.closeAsync(); + await fixture[Symbol.asyncDispose](); + } + }); }); From fe18240a4eee33dc4d9be940b4df08602beedfc3 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:06 +0000 Subject: [PATCH 074/265] [rush-lib] A parameter whose short name the tool also defines can be used by its long name Swarm integration step 59; original commit 49695c5d00 (merge of swarm/r03-t172 at 4526357153). Scope: task 172. Brings r03's task 172 (board 3008): rush update-cloud-credentials --delete no longer fails with an ambiguous -d. Second agent: o03 board 3054. s17 batch D1, item 7 of 10. Gate: ch01 GATE OK board 3310 (tree 29b72dd480) Commits folded into this step (1): - 4526357153 [ts-command-line] A parameter whose short name the tool also defines can be used by its long name (task 172) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...03-parent-short-name_2026-09-29-02-40.json | 11 ++ ...03-parent-short-name_2026-09-29-02-40.json | 11 ++ .../cli/test/RushCommandLineParser.test.ts | 59 +++++++ .../CommandLineHelp.test.ts.snap | 5 +- .../common/config/rush/command-line.json | 14 ++ .../providers/CommandLineParameterProvider.ts | 21 ++- .../test/AmbiguousCommandLineParser.test.ts | 165 ++++++++++++++++++ 7 files changed, 277 insertions(+), 9 deletions(-) create mode 100644 common/changes/@microsoft/rush/r03-parent-short-name_2026-09-29-02-40.json create mode 100644 common/changes/@rushstack/ts-command-line/r03-parent-short-name_2026-09-29-02-40.json diff --git a/common/changes/@microsoft/rush/r03-parent-short-name_2026-09-29-02-40.json b/common/changes/@microsoft/rush/r03-parent-short-name_2026-09-29-02-40.json new file mode 100644 index 0000000000..33a8818a74 --- /dev/null +++ b/common/changes/@microsoft/rush/r03-parent-short-name_2026-09-29-02-40.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Fix an issue where `rush update-cloud-credentials --delete` failed with `Ambiguous option: \"-d\"`, and where a custom command parameter whose short name is `-d` or `-q` couldn't be used by its long name either. After the command name, `-d` and `-q` are still ambiguous with the global `--debug` and `--quiet` parameters.", + "type": "patch" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/ts-command-line/r03-parent-short-name_2026-09-29-02-40.json b/common/changes/@rushstack/ts-command-line/r03-parent-short-name_2026-09-29-02-40.json new file mode 100644 index 0000000000..13671b14f6 --- /dev/null +++ b/common/changes/@rushstack/ts-command-line/r03-parent-short-name_2026-09-29-02-40.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "comment": "Fix an issue where an action parameter couldn't be used at all, not even by its long name, if the tool or a parent action also defines its short name. Now only the short name is ambiguous, and the action's help no longer shows it.", + "type": "patch", + "packageName": "@rushstack/ts-command-line" + } + ], + "packageName": "@rushstack/ts-command-line", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-lib/src/cli/test/RushCommandLineParser.test.ts b/libraries/rush-lib/src/cli/test/RushCommandLineParser.test.ts index 6a7920715e..e1427b708f 100644 --- a/libraries/rush-lib/src/cli/test/RushCommandLineParser.test.ts +++ b/libraries/rush-lib/src/cli/test/RushCommandLineParser.test.ts @@ -32,6 +32,7 @@ import type { SpawnOptions } from 'node:child_process'; import { FileSystem, JsonFile, LockFile, Path } from '@rushstack/node-core-library'; import type { IDetailedRepoState } from '@rushstack/package-deps-hash'; import type { IReporterEmitEventInput, IReporterEventSink } from '@rushstack/rush-reporter'; +import type { CommandLineAction } from '@rushstack/ts-command-line'; import { Autoinstaller } from '../../logic/Autoinstaller'; import type { ITelemetryData } from '../../logic/Telemetry'; import { @@ -296,6 +297,64 @@ describe('RushCommandLineParser', () => { }); }); + describe("'custom-short-name' action", () => { + it('accepts a custom parameter by its long name when its short name is also the global -d', async () => { + const { parser, repoPath } = await getCommandLineParserInstanceAsync( + 'basicAndRunBuildActionRepo', + 'custom-short-name' + ); + process.argv.push('--stale-after-days', '3'); + + await expect(parser.executeAsync()).resolves.toEqual(true); + + expect(JsonFile.load(`${repoPath}/custom-output-args.json`)).toEqual(['--stale-after-days', '3']); + }); + }); + + describe("'update-cloud-credentials' action", () => { + it('accepts --delete, whose short name -d is also the global --debug parameter', async () => { + const { parser } = await getCommandLineParserInstanceAsync( + 'basicAndRunBuildActionRepo', + 'update-cloud-credentials' + ); + process.argv.push('--delete'); + const action: CommandLineAction = parser.getAction('update-cloud-credentials'); + // Stop before the action loads the build cache configuration, which this repo doesn't have + const runSpy: jest.SpyInstance = jest + .spyOn(action as unknown as { runAsync(): Promise }, 'runAsync') + .mockResolvedValue(undefined); + + await expect(parser.executeAsync()).resolves.toEqual(true); + + expect(runSpy).toHaveBeenCalledTimes(1); + expect(action.getFlagParameter('--delete').value).toBe(true); + expect(parser.getFlagParameter('--debug').value).toBe(false); + }); + + it('reports -d after the action name as ambiguous', async () => { + const { parser } = await getCommandLineParserInstanceAsync( + 'basicAndRunBuildActionRepo', + 'update-cloud-credentials' + ); + process.argv.push('-d'); + const originalExitCode: string | number | undefined = process.exitCode; + const errorSpy: jest.SpyInstance = jest.spyOn(console, 'error').mockImplementation(() => undefined); + const logSpy: jest.SpyInstance = jest.spyOn(console, 'log').mockImplementation(() => undefined); + try { + await expect(parser.executeAsync()).resolves.toEqual(false); + + expect(process.exitCode).toBe(1); + expect(errorSpy).toHaveBeenCalledWith( + 'Error: rush update-cloud-credentials: error: Ambiguous option: "-d".\n' + ); + } finally { + errorSpy.mockRestore(); + logSpy.mockRestore(); + process.exitCode = originalExitCode; + } + }); + }); + describe("'rebuild' action", () => { it(`executes the package's 'build' script`, async () => { const repoName: string = 'basicAndRunRebuildActionRepo'; diff --git a/libraries/rush-lib/src/cli/test/__snapshots__/CommandLineHelp.test.ts.snap b/libraries/rush-lib/src/cli/test/__snapshots__/CommandLineHelp.test.ts.snap index efe3e717b7..bed4fdd30a 100644 --- a/libraries/rush-lib/src/cli/test/__snapshots__/CommandLineHelp.test.ts.snap +++ b/libraries/rush-lib/src/cli/test/__snapshots__/CommandLineHelp.test.ts.snap @@ -1445,7 +1445,8 @@ Optional arguments: exports[`CommandLineHelp prints the help for each action: update-cloud-credentials 1`] = ` "usage: rush update-cloud-credentials [-h] [-i] - [--credential CREDENTIAL_STRING] [-d] + [--credential CREDENTIAL_STRING] + [--delete] (EXPERIMENTAL) If the build caching feature is configured, this command @@ -1457,7 +1458,7 @@ Optional arguments: mode, if supported by the provider. --credential CREDENTIAL_STRING A static credential, to be cached. - -d, --delete If specified, delete stored credentials. + --delete If specified, delete stored credentials. " `; diff --git a/libraries/rush-lib/src/cli/test/basicAndRunBuildActionRepo/common/config/rush/command-line.json b/libraries/rush-lib/src/cli/test/basicAndRunBuildActionRepo/common/config/rush/command-line.json index c7d4e88c76..b73ec58834 100644 --- a/libraries/rush-lib/src/cli/test/basicAndRunBuildActionRepo/common/config/rush/command-line.json +++ b/libraries/rush-lib/src/cli/test/basicAndRunBuildActionRepo/common/config/rush/command-line.json @@ -5,6 +5,12 @@ "name": "custom-output", "summary": "Exercises custom parameters that overlap reporter controls.", "shellCommand": "node custom-output.js" + }, + { + "commandKind": "global", + "name": "custom-short-name", + "summary": "Exercises a custom parameter whose short name is also a global parameter's short name.", + "shellCommand": "node custom-output.js" } ], "parameters": [ @@ -34,6 +40,14 @@ "longName": "--verbose", "description": "Custom verbose flag.", "associatedCommands": ["custom-output"] + }, + { + "parameterKind": "integer", + "longName": "--stale-after-days", + "shortName": "-d", + "argumentName": "DAYS", + "description": "Custom parameter whose short name is also the short name of the global --debug parameter.", + "associatedCommands": ["custom-short-name"] } ] } diff --git a/libraries/ts-command-line/src/providers/CommandLineParameterProvider.ts b/libraries/ts-command-line/src/providers/CommandLineParameterProvider.ts index 2fe89023a3..e19e4c3a82 100644 --- a/libraries/ts-command-line/src/providers/CommandLineParameterProvider.ts +++ b/libraries/ts-command-line/src/providers/CommandLineParameterProvider.ts @@ -525,10 +525,13 @@ export abstract class CommandLineParameterProvider { // First, loop through all parameters with short names. If there are any duplicates, disable the short names // since we can't prefix scopes to short names in order to deduplicate them. The duplicate short names will - // be reported as errors if the user attempts to use them. + // be reported as errors if the user attempts to use them. A short name that the parent action or tool also + // defines is a duplicate too. Disabling it here gives it its own parser key, so the parameter can still be + // used by its long name. + const { parentParameterNames } = state; const parametersWithDuplicateShortNames: Set = new Set(); for (const [shortName, shortNameParameters] of this.#parametersByShortName.entries()) { - if (shortNameParameters.length > 1) { + if (shortNameParameters.length > 1 || parentParameterNames.has(shortName)) { for (const parameter of shortNameParameters) { this._defineAmbiguousParameter(shortName); parametersWithDuplicateShortNames.add(parameter); @@ -559,7 +562,6 @@ export abstract class CommandLineParameterProvider { // Register the existing parameters as ambiguous parameters. These are generally provided by the // parent action. - const { parentParameterNames } = state; for (const parentParameterName of parentParameterNames) { this._defineAmbiguousParameter(parentParameterName); } @@ -631,16 +633,21 @@ export abstract class CommandLineParameterProvider { // Search for any ambiguous parameters and throw an error if any are found for (const [parameterName, parserKey] of this._ambiguousParameterParserKeysByName) { if (data[parserKey]) { + const duplicateShortNameParameters: CommandLineParameterBase[] | undefined = + this.#parametersByShortName.get(parameterName); + // When the parser key matches the actually registered parameter, we know that this is an ambiguous - // parameter sourced from the parent action or tool - if (this._registeredParameterParserKeysByName.get(parameterName) === parserKey) { + // parameter sourced from the parent action or tool. The same is true for a short name that only one + // parameter here uses, since one parameter can't make its own short name ambiguous. + if ( + this._registeredParameterParserKeysByName.get(parameterName) === parserKey || + duplicateShortNameParameters?.length === 1 + ) { this.#throwParserExitError(parserOptions, data, 1, `Ambiguous option: "${parameterName}".`); } // Determine if the ambiguous parameter is a short name or a long name, since the process of finding // the non-ambiguous name is different for each. - const duplicateShortNameParameters: CommandLineParameterBase[] | undefined = - this.#parametersByShortName.get(parameterName); if (duplicateShortNameParameters) { // We also need to make sure we get the non-ambiguous long name for the parameter, since it is // possible for that the long name is ambiguous as well. diff --git a/libraries/ts-command-line/src/test/AmbiguousCommandLineParser.test.ts b/libraries/ts-command-line/src/test/AmbiguousCommandLineParser.test.ts index c6445201fc..1aeb6d734b 100644 --- a/libraries/ts-command-line/src/test/AmbiguousCommandLineParser.test.ts +++ b/libraries/ts-command-line/src/test/AmbiguousCommandLineParser.test.ts @@ -102,6 +102,37 @@ class AbbreviationAction extends CommandLineAction { } } +class ShortNameAction extends CommandLineAction { + public done: boolean = false; + public deleteFlag: CommandLineFlagParameter; + + public constructor() { + super({ + actionName: 'do:the-job', + summary: 'does the job', + documentation: 'a longer description' + }); + + this.deleteFlag = this.defineFlagParameter({ + parameterLongName: '--delete', + parameterShortName: '-d', + description: 'A flag whose short name the tool also declares' + }); + } + + protected override async onExecuteAsync(): Promise { + this.done = true; + } +} + +function defineToolDebugFlag(commandLineParser: CommandLineParser): CommandLineFlagParameter { + return commandLineParser.defineFlagParameter({ + parameterLongName: '--debug', + parameterShortName: '-d', + description: 'A flag whose short name the action also declares' + }); +} + class AliasAction extends AliasCommandLineAction { public constructor(targetActionClass: new () => CommandLineAction) { super({ @@ -250,6 +281,38 @@ class AbbreviationScopedAction extends ScopedCommandLineAction { } } +class ShortNameScopedAction extends ScopedCommandLineAction { + public done: boolean = false; + public deleteFlag: CommandLineFlagParameter | undefined; + + public constructor() { + super({ + actionName: 'scoped-action', + summary: 'does the scoped action', + documentation: 'a longer description' + }); + + // At least one scoping parameter is required to be defined on a scoped action + this.defineFlagParameter({ + parameterLongName: '--scoping', + description: 'The scoping parameter', + parameterGroup: SCOPING_PARAMETER_GROUP + }); + } + + protected override async onExecuteAsync(): Promise { + this.done = true; + } + + protected onDefineScopedParameters(scopedParameterProvider: CommandLineParameterProvider): void { + this.deleteFlag = scopedParameterProvider.defineFlagParameter({ + parameterLongName: '--delete', + parameterShortName: '-d', + description: 'A flag whose short name the tool also declares' + }); + } +} + describe(`Ambiguous ${CommandLineParser.name}`, () => { it('renders help text', () => { const commandLineParser: GenericCommandLine = new GenericCommandLine( @@ -357,6 +420,47 @@ describe(`Ambiguous ${CommandLineParser.name}`, () => { expect(action.abbreviationFlag.value).toBe(true); expect(toolAbbreviationFlag.value).toBe(false); }); + + it('can execute a parameter by its long name when the tool also declares its short name', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(ShortNameAction); + const toolDebugFlag: CommandLineFlagParameter = defineToolDebugFlag(commandLineParser); + + await commandLineParser.executeWithoutErrorHandlingAsync(['do:the-job', '--delete']); + + expect(commandLineParser.selectedAction).toBeDefined(); + expect(commandLineParser.selectedAction!.actionName).toEqual('do:the-job'); + + const action: ShortNameAction = commandLineParser.selectedAction as ShortNameAction; + expect(action.done).toBe(true); + expect(action.deleteFlag.value).toBe(true); + expect(toolDebugFlag.value).toBe(false); + + // The action's help doesn't offer the short name, since it can't be used after the action name + const helpText: string = action.renderHelpText(); + expect(helpText).toContain(' --delete '); + expect(helpText).not.toContain('-d, --delete'); + }); + + it('can use a short name declared in both the tool and the action before the action name', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(ShortNameAction); + const toolDebugFlag: CommandLineFlagParameter = defineToolDebugFlag(commandLineParser); + + await commandLineParser.executeWithoutErrorHandlingAsync(['-d', 'do:the-job', '--delete']); + + const action: ShortNameAction = commandLineParser.selectedAction as ShortNameAction; + expect(action.done).toBe(true); + expect(action.deleteFlag.value).toBe(true); + expect(toolDebugFlag.value).toBe(true); + }); + + it('fails when providing a short name to an action that was also declared in the tool', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(ShortNameAction); + defineToolDebugFlag(commandLineParser); + + await expect(commandLineParser.executeWithoutErrorHandlingAsync(['do:the-job', '-d'])).rejects.toThrow( + 'Error: example do:the-job: error: Ambiguous option: "-d".\n' + ); + }); }); describe(`Ambiguous aliased ${CommandLineParser.name}`, () => { @@ -478,6 +582,37 @@ describe(`Ambiguous aliased ${CommandLineParser.name}`, () => { expect(action.abbreviationFlag.value).toBe(true); expect(toolAbbreviationFlag.value).toBe(false); }); + + it('can execute a parameter by its long name when the tool also declares its short name', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(AliasAction, ShortNameAction); + commandLineParser.addAction( + (commandLineParser.getAction('do:the-job-alias')! as AliasAction).targetAction + ); + const toolDebugFlag: CommandLineFlagParameter = defineToolDebugFlag(commandLineParser); + + await commandLineParser.executeWithoutErrorHandlingAsync(['do:the-job-alias', '--delete']); + + expect(commandLineParser.selectedAction).toBeDefined(); + expect(commandLineParser.selectedAction!.actionName).toEqual('do:the-job-alias'); + + const action: ShortNameAction = (commandLineParser.selectedAction as AliasAction) + .targetAction as ShortNameAction; + expect(action.done).toBe(true); + expect(action.deleteFlag.value).toBe(true); + expect(toolDebugFlag.value).toBe(false); + }); + + it('fails when providing a short name to an action that was also declared in the tool', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(AliasAction, ShortNameAction); + commandLineParser.addAction( + (commandLineParser.getAction('do:the-job-alias')! as AliasAction).targetAction + ); + defineToolDebugFlag(commandLineParser); + + await expect( + commandLineParser.executeWithoutErrorHandlingAsync(['do:the-job-alias', '-d']) + ).rejects.toThrow('Error: example do:the-job-alias: error: Ambiguous option: "-d".\n'); + }); }); describe(`Ambiguous scoping ${CommandLineParser.name}`, () => { @@ -676,4 +811,34 @@ describe(`Ambiguous scoping ${CommandLineParser.name}`, () => { expect(targetAction.scopedAbbreviationFlag?.value).toBe(true); expect(targetAction.unscopedAbbreviationFlag?.value).toBe(false); }); + + it('can execute a scoped parameter by its long name when the tool also declares its short name', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(ShortNameScopedAction); + const toolDebugFlag: CommandLineFlagParameter = defineToolDebugFlag(commandLineParser); + const targetAction: ShortNameScopedAction = commandLineParser.getAction( + 'scoped-action' + ) as ShortNameScopedAction; + + await commandLineParser.executeWithoutErrorHandlingAsync([ + 'scoped-action', + '--scoping', + '--', + '--delete' + ]); + + expect(commandLineParser.selectedAction).toBeDefined(); + expect(commandLineParser.selectedAction!.actionName).toEqual('scoped-action'); + expect(targetAction.done).toBe(true); + expect(targetAction.deleteFlag?.value).toBe(true); + expect(toolDebugFlag.value).toBe(false); + }); + + it('fails when providing a short name to a scoped action that was also declared in the tool', async () => { + const commandLineParser: GenericCommandLine = new GenericCommandLine(ShortNameScopedAction); + defineToolDebugFlag(commandLineParser); + + await expect( + commandLineParser.executeWithoutErrorHandlingAsync(['scoped-action', '--scoping', '--', '-d']) + ).rejects.toThrow('Error: example scoped-action --scoping --: error: Ambiguous option: "-d".\n'); + }); }); From 2f235bcb83c390c79417da7bbb6e364873f17e7b Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:06 +0000 Subject: [PATCH 075/265] [rush-azure-storage-build-cache-plugin] Cache Azure credentials per storage endpoint Swarm integration step 60; original commit d950fba8d3 (merge of swarm/r03-t144b-x101 at 5e91eed724). Scope: task 164. Brings r03's task 164 (board 2765), merged with task 101: the Azure build cache keys a saved credential by its storage endpoint, and the schema says that http endpoints need a SAS. Second agent: ch05 board 2940. s17 batch D1, item 8 of 10. Gate: ch01 GATE OK board 3310 (tree b0ee664f20) Commits folded into this step (1): - 2a744907c2 [rush-azure-storage-build-cache-plugin] Save credentials per storageEndpoint, and require an http or https storageEndpoint (task 164) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...-endpoint-credential_2026-09-29-00-29.json | 11 +++ .../common/config/rush/build-cache.json | 8 +- .../api/test/BuildCacheConfiguration.test.ts | 30 +++++++- .../src/schemas/build-cache.schema.json | 5 +- .../src/AzureStorageAuthentication.ts | 12 ++- .../src/RushAzureStorageBuildCachePlugin.ts | 7 ++ .../azure-blob-storage-config.schema.json | 5 +- .../AzureStorageBuildCacheProvider.test.ts | 76 +++++++++++++++++++ 8 files changed, 147 insertions(+), 7 deletions(-) create mode 100644 common/changes/@microsoft/rush/azure-storage-endpoint-credential_2026-09-29-00-29.json diff --git a/common/changes/@microsoft/rush/azure-storage-endpoint-credential_2026-09-29-00-29.json b/common/changes/@microsoft/rush/azure-storage-endpoint-credential_2026-09-29-00-29.json new file mode 100644 index 0000000000..33f72bb77e --- /dev/null +++ b/common/changes/@microsoft/rush/azure-storage-endpoint-credential_2026-09-29-00-29.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "The Azure Storage build cache plugin now saves credentials for each `storageEndpoint`, so a credential saved for one endpoint is never sent to another. build-cache.json now rejects a `storageEndpoint` that doesn't start with `http://` or `https://`.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json b/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json index b9bed9937c..bd030586f4 100644 --- a/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json +++ b/libraries/rush-lib/assets/rush-init/common/config/rush/build-cache.json @@ -63,7 +63,13 @@ /** * An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, * private endpoints, or storage emulators. When specified, this overrides the default endpoint derived - * from "storageAccountName". + * from "storageAccountName". The URL must start with "http://" or "https://". + * + * A credential saved by "rush update-cloud-credentials" is used only with the endpoint that it was + * saved for. An Azure login ("rush update-cloud-credentials --interactive") needs an "https://" + * endpoint. For an "http://" endpoint, such as Azurite, provide a SAS token with + * "rush update-cloud-credentials --credential" or the RUSH_BUILD_CACHE_CREDENTIAL environment + * variable, or read without a credential. */ // "storageEndpoint": "http://127.0.0.1:10000/devstoreaccount1", diff --git a/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts b/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts index 189a1747d8..912d253167 100644 --- a/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/BuildCacheConfiguration.test.ts @@ -94,9 +94,37 @@ describe(BuildCacheConfiguration.name, () => { tryLoadAzureConfigurationAsync({ storageAccountName: 'example', storageContainerName: 'build-cache', - storageEndpoint: '127.0.0.1 port 10000' + storageEndpoint: 'http://127.0.0.1 port 10000' }) ).rejects.toThrow(/#\/azureBlobStorageConfiguration\/storageEndpoint\s+must match format "uri"/); expect(factory).not.toHaveBeenCalled(); }); + + it('rejects a storageEndpoint without an http or https scheme', async () => { + // "localhost:" parses as a URI scheme, so only the pattern catches this common mistake. + await expect( + tryLoadAzureConfigurationAsync({ + storageAccountName: 'example', + storageContainerName: 'build-cache', + storageEndpoint: 'localhost:10000/devstoreaccount1' + }) + ).rejects.toThrow( + /#\/azureBlobStorageConfiguration\/storageEndpoint\s+must match pattern "\^\[Hh\]\[Tt\]\[Tt\]\[Pp\]\[Ss\]\?:\/\/"/ + ); + expect(factory).not.toHaveBeenCalled(); + }); + + it.each([ + { kind: 'an https storageEndpoint', storageEndpoint: 'https://my-proxy.example.com/devstoreaccount1' }, + // URI schemes are case-insensitive, and Rush accepted this before the pattern existed. + { kind: 'a storageEndpoint with an uppercase scheme', storageEndpoint: 'HTTP://127.0.0.1:10000/x' } + ])('accepts $kind', async ({ storageEndpoint }: { storageEndpoint: string }) => { + await tryLoadAzureConfigurationAsync({ + storageAccountName: 'example', + storageContainerName: 'build-cache', + storageEndpoint + }); + expect(factory).toHaveBeenCalledTimes(1); + expect(factory.mock.calls[0][0]).toMatchObject({ azureBlobStorageConfiguration: { storageEndpoint } }); + }); }); diff --git a/libraries/rush-lib/src/schemas/build-cache.schema.json b/libraries/rush-lib/src/schemas/build-cache.schema.json index 8029a685a6..6eb4fe0855 100644 --- a/libraries/rush-lib/src/schemas/build-cache.schema.json +++ b/libraries/rush-lib/src/schemas/build-cache.schema.json @@ -73,8 +73,9 @@ }, "storageEndpoint": { "type": "string", - "description": "An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, private endpoints, or storage emulators. When specified, this overrides the default endpoint derived from storageAccountName. Example: \"http://127.0.0.1:10000/devstoreaccount1\" or \"https://my-proxy.example.com/devstoreaccount1\"", - "format": "uri" + "description": "An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, private endpoints, or storage emulators. When specified, this overrides the default endpoint derived from storageAccountName. The URL must start with \"http://\" or \"https://\". A credential saved by \"rush update-cloud-credentials\" is used only with the endpoint that it was saved for. An Azure login (\"rush update-cloud-credentials --interactive\") needs an \"https://\" endpoint. For an \"http://\" endpoint, such as Azurite, provide a SAS token with \"rush update-cloud-credentials --credential\" or the RUSH_BUILD_CACHE_CREDENTIAL environment variable, or read without a credential. Example: \"http://127.0.0.1:10000/devstoreaccount1\" or \"https://my-proxy.example.com/devstoreaccount1\"", + "format": "uri", + "pattern": "^[Hh][Tt][Tt][Pp][Ss]?://" }, "loginFlow": { "$ref": "#/definitions/entraLoginFlow" diff --git a/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageAuthentication.ts b/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageAuthentication.ts index 892cb0897b..62008d89a6 100644 --- a/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageAuthentication.ts +++ b/rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageAuthentication.ts @@ -30,6 +30,10 @@ export interface IAzureStorageAuthenticationOptions extends IAzureAuthentication const SAS_TTL_MILLISECONDS: number = 7 * 24 * 60 * 60 * 1000; // Seven days +function getDefaultStorageAccountUrl(storageAccountName: string): string { + return `https://${storageAccountName}.blob.core.windows.net/`; +} + /** * @public */ @@ -52,12 +56,18 @@ export class AzureStorageAuthentication extends AzureAuthenticationBase { ? storageEndpoint.endsWith('/') ? storageEndpoint : storageEndpoint + '/' - : `https://${storageAccountName}.blob.core.windows.net/`; + : getDefaultStorageAccountUrl(storageAccountName); } protected _getCacheIdParts(): string[] { const cacheIdParts: string[] = [this._storageAccountName, this._storageContainerName]; + // A saved credential is sent to the endpoint that it was saved for, and to no other. The default + // endpoint keeps the ID that it had before storageEndpoint existed. + if (this._storageAccountUrl !== getDefaultStorageAccountUrl(this._storageAccountName)) { + cacheIdParts.push(this._storageAccountUrl); + } + if (this._isCacheWriteAllowedByConfiguration) { cacheIdParts.push('cacheWriteAllowed'); } diff --git a/rush-plugins/rush-azure-storage-build-cache-plugin/src/RushAzureStorageBuildCachePlugin.ts b/rush-plugins/rush-azure-storage-build-cache-plugin/src/RushAzureStorageBuildCachePlugin.ts index 4955f96f27..298483de09 100644 --- a/rush-plugins/rush-azure-storage-build-cache-plugin/src/RushAzureStorageBuildCachePlugin.ts +++ b/rush-plugins/rush-azure-storage-build-cache-plugin/src/RushAzureStorageBuildCachePlugin.ts @@ -41,6 +41,13 @@ interface IAzureBlobStorageConfigurationJson { * An optional custom endpoint URL for the Azure Blob Storage account. * Use this to connect to Azurite, private endpoints, or storage emulators. * Overrides the default endpoint derived from storageAccountName. + * + * @remarks + * A credential saved by `rush update-cloud-credentials` is used only with the endpoint that it was + * saved for. An Azure login (`rush update-cloud-credentials --interactive`) needs an `https://` endpoint. + * For an `http://` endpoint, such as Azurite, provide a SAS token with + * `rush update-cloud-credentials --credential` or the `RUSH_BUILD_CACHE_CREDENTIAL` environment variable, + * or read without a credential. */ readonly storageEndpoint?: string; diff --git a/rush-plugins/rush-azure-storage-build-cache-plugin/src/schemas/azure-blob-storage-config.schema.json b/rush-plugins/rush-azure-storage-build-cache-plugin/src/schemas/azure-blob-storage-config.schema.json index 2ffcd58e4b..4d8efdc56d 100644 --- a/rush-plugins/rush-azure-storage-build-cache-plugin/src/schemas/azure-blob-storage-config.schema.json +++ b/rush-plugins/rush-azure-storage-build-cache-plugin/src/schemas/azure-blob-storage-config.schema.json @@ -97,8 +97,9 @@ "storageEndpoint": { "type": "string", - "description": "An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, private endpoints, or storage emulators. When specified, this overrides the default endpoint derived from storageAccountName. Example: \"http://127.0.0.1:10000/devstoreaccount1\" or \"https://my-proxy.example.com/devstoreaccount1\"", - "format": "uri" + "description": "An optional custom endpoint URL for the Azure Blob Storage account. Use this to connect to Azurite, private endpoints, or storage emulators. When specified, this overrides the default endpoint derived from storageAccountName. The URL must start with \"http://\" or \"https://\". A credential saved by \"rush update-cloud-credentials\" is used only with the endpoint that it was saved for. An Azure login (\"rush update-cloud-credentials --interactive\") needs an \"https://\" endpoint. For an \"http://\" endpoint, such as Azurite, provide a SAS token with \"rush update-cloud-credentials --credential\" or the RUSH_BUILD_CACHE_CREDENTIAL environment variable, or read without a credential. Example: \"http://127.0.0.1:10000/devstoreaccount1\" or \"https://my-proxy.example.com/devstoreaccount1\"", + "format": "uri", + "pattern": "^[Hh][Tt][Tt][Pp][Ss]?://" }, "isCacheWriteAllowed": { diff --git a/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts b/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts index 3b5e5e0b28..6dc03c12f7 100644 --- a/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts +++ b/rush-plugins/rush-azure-storage-build-cache-plugin/src/test/AzureStorageBuildCacheProvider.test.ts @@ -1,6 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + import { BlobServiceClient, type BlockBlobClient, type ContainerClient } from '@azure/storage-blob'; import { CredentialCache, type ICredentialCacheEntry } from '@rushstack/credential-cache'; @@ -67,6 +71,78 @@ describe(AzureStorageBuildCacheProvider.name, () => { }); }); + describe('a saved credential', () => { + const EMULATOR_ENDPOINT: string = 'http://127.0.0.1:10000/devstoreaccount1'; + const terminal: Terminal = new Terminal(new StringBufferTerminalProvider()); + let cacheFolder: string; + + beforeEach(() => { + cacheFolder = fs.mkdtempSync(path.join(os.tmpdir(), 'azure-storage-credentials-')); + const usingAsync: typeof CredentialCache.usingAsync = CredentialCache.usingAsync.bind(CredentialCache); + jest + .spyOn(CredentialCache, 'usingAsync') + .mockImplementation((options, doActionAsync) => + usingAsync({ ...options, cacheFilePath: `${cacheFolder}/credentials.json` }, doActionAsync) + ); + }); + + afterEach(() => { + jest.restoreAllMocks(); + fs.rmSync(cacheFolder, { recursive: true, force: true }); + }); + + function createProvider(storageEndpoint?: string): AzureStorageBuildCacheProvider { + return new AzureStorageBuildCacheProvider({ + storageAccountName: 'storage-account', + storageContainerName: 'container-name', + storageEndpoint, + isCacheWriteAllowed: true + }); + } + + async function tryGetSavedCredentialAsync(storageEndpoint?: string): Promise { + return (await createProvider(storageEndpoint).tryGetCachedCredentialAsync())?.credential; + } + + it('is used only with the endpoint that it was saved for', async () => { + await createProvider().updateCachedCredentialAsync(terminal, 'account-sas'); + expect(await tryGetSavedCredentialAsync()).toBe('account-sas'); + expect(await tryGetSavedCredentialAsync(EMULATOR_ENDPOINT)).toBeUndefined(); + + await createProvider(EMULATOR_ENDPOINT).updateCachedCredentialAsync(terminal, 'emulator-sas'); + expect(await tryGetSavedCredentialAsync(EMULATOR_ENDPOINT)).toBe('emulator-sas'); + expect(await tryGetSavedCredentialAsync(`${EMULATOR_ENDPOINT}/`)).toBe('emulator-sas'); + expect(await tryGetSavedCredentialAsync('https://proxy.example.com/devstoreaccount1')).toBeUndefined(); + expect(await tryGetSavedCredentialAsync()).toBe('account-sas'); + + const savedJson: { cacheEntries: Record } = JSON.parse( + fs.readFileSync(`${cacheFolder}/credentials.json`, 'utf8') + ); + expect(Object.keys(savedJson.cacheEntries).sort()).toEqual([ + 'azure-blob-storage|AzurePublicCloud|storage-account|container-name|cacheWriteAllowed', + `azure-blob-storage|AzurePublicCloud|storage-account|container-name|${EMULATOR_ENDPOINT}/|cacheWriteAllowed` + ]); + }); + + it("is shared with a storageEndpoint that names the account's default endpoint", async () => { + await createProvider().updateCachedCredentialAsync(terminal, 'account-sas'); + expect(await tryGetSavedCredentialAsync('https://storage-account.blob.core.windows.net')).toBe( + 'account-sas' + ); + expect(await tryGetSavedCredentialAsync('https://storage-account.blob.core.windows.net/')).toBe( + 'account-sas' + ); + }); + + it('is deleted only for the configured endpoint', async () => { + await createProvider().updateCachedCredentialAsync(terminal, 'account-sas'); + await createProvider(EMULATOR_ENDPOINT).updateCachedCredentialAsync(terminal, 'emulator-sas'); + await createProvider(EMULATOR_ENDPOINT).deleteCachedCredentialsAsync(terminal); + expect(await tryGetSavedCredentialAsync(EMULATOR_ENDPOINT)).toBeUndefined(); + expect(await tryGetSavedCredentialAsync()).toBe('account-sas'); + }); + }); + describe('isCacheWriteAllowed', () => { function prepareSubject( optionValue: boolean, From c8e43c1a53d6007c1afac7fc8b49e76030bd018a Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:06 +0000 Subject: [PATCH 076/265] [rush-cli-client] Test the cancellation notice when the client never asked rushd to stop Swarm integration step 61; original commit 4d06054101 (merge of swarm/r04-t132-nit1 at 8f0016995a). Scope: task 132 NIT 1. Brings r04's fix for ch01's NIT 1 on task 132 (board 3094). Second agent: t05 board 3172. s17 batch D1, item 9 of 10. Gate: ch01 GATE OK board 3310 (tree 7e80f904a2) Commits folded into this step (1): - 8f0016995a Test the cancellation notice when the client never asked rushd to stop the request (task 132 NIT 1, ch01 board 2984) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../src/test/cancellationNotice.test.ts | 41 +++++++++++++++++++ .../swarm-r04-t132-nit1_2026-09-29-03-10.json | 11 +++++ 2 files changed, 52 insertions(+) create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r04-t132-nit1_2026-09-29-03-10.json diff --git a/apps/rush-cli-client/src/test/cancellationNotice.test.ts b/apps/rush-cli-client/src/test/cancellationNotice.test.ts index aea86c74d0..c48e0c09a2 100644 --- a/apps/rush-cli-client/src/test/cancellationNotice.test.ts +++ b/apps/rush-cli-client/src/test/cancellationNotice.test.ts @@ -133,6 +133,13 @@ describe('the cancellation of a daemon request (task 132)', () => { ); } + function disconnected(): DaemonClientError { + return new DaemonClientError( + 'disconnected', + 'Daemon disconnected before delivering a result; the command was not retried.' + ); + } + it('says at once that it waits for rushd, and keeps the final line when rushd confirms the stop', async () => { execute(async (options) => { deliverSignal('SIGINT'); @@ -178,6 +185,18 @@ describe('the cancellation of a daemon request (task 132)', () => { expect(process.exitCode).toBe(130); }); + it('adds nothing when the connection fails before the client asked rushd to stop the request', async () => { + // For example, a signal while the client waits for a restarted daemon, whose connection then fails. + execute(async (options) => { + deliverSignal('SIGINT'); + expect(options.abortSignal!.aborted).toBe(true); + throw disconnected(); + }); + await launchClientAsync(false); + expect(stderr).toEqual([CANCELLED]); + expect(process.exitCode).toBe(130); + }); + it('writes both notices in the agent output, once', async () => { const output: string[] = []; const renderer: AgentProgressRenderer = new AgentProgressRenderer({ @@ -209,4 +228,26 @@ describe('the cancellation of a daemon request (task 132)', () => { expect(stderr).toEqual([CANCELLED]); expect(process.exitCode).toBe(130); }); + + it('ends the agent summary line without a stop to confirm when the client never asked rushd to stop', async () => { + const output: string[] = []; + const renderer: AgentProgressRenderer = new AgentProgressRenderer({ + commandName: 'build', + isTTY: false, + columns: 80, + write: (text: string) => output.push(text) + }); + execute(async () => { + deliverSignal('SIGINT'); + throw disconnected(); + }); + await launchClientAsync(false, renderer); + const lines: string[] = output.join('').split('\n').slice(0, -1); + expect(lines).toEqual([ + expect.stringMatching(/^rush build · \d+\.\ds · sent to rushd; preparing the workspace graph /), + expect.stringMatching(/^rush build: CANCELLED in \d+\.\ds$/) + ]); + expect(stderr).toEqual([CANCELLED]); + expect(process.exitCode).toBe(130); + }); }); diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r04-t132-nit1_2026-09-29-03-10.json b/common/changes/@rushstack/rush-cli-client/swarm-r04-t132-nit1_2026-09-29-03-10.json new file mode 100644 index 0000000000..a667d24642 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r04-t132-nit1_2026-09-29-03-10.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Test that a cancelled request whose connection fails before the client asked rushd to stop it is reported as cancelled, without saying that rushd did not confirm the stop.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} From 4602af13d5be50844abdda8fbdcddad67ce05a7e Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:12:06 +0000 Subject: [PATCH 077/265] [rush-client-core] Test that a takeover keeps a reservation that replaced the abandoned one Swarm integration step 62; original commit d06e2ded5e (merge of swarm/r03-t126-nit2 at 40ca8cd335). Scope: task 126 NIT 2. Brings r03's fix for ch01's NIT 2 on task 126 (board 2984, board 3089). ch01 checked it with mutant K126 in the gate. s17 batch D1, item 10 of 10. Gate: ch01 GATE OK board 3310 (tree 8061bc8677) Commits folded into this step (1): - 40ca8cd335 [rush-client-core] Test that a takeover keeps a reservation that replaced the abandoned one (task 126, ch01 NIT 2) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...akeover-recheck-test_2026-09-29-02-55.json | 11 +++++++ .../src/test/connectOrStartDaemon.test.ts | 29 +++++++++++++++++-- 2 files changed, 38 insertions(+), 2 deletions(-) create mode 100644 common/changes/@rushstack/rush-client-core/r03-reservation-takeover-recheck-test_2026-09-29-02-55.json diff --git a/common/changes/@rushstack/rush-client-core/r03-reservation-takeover-recheck-test_2026-09-29-02-55.json b/common/changes/@rushstack/rush-client-core/r03-reservation-takeover-recheck-test_2026-09-29-02-55.json new file mode 100644 index 0000000000..8e3f2a8416 --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/r03-reservation-takeover-recheck-test_2026-09-29-02-55.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "Test that a client takes over only the startup reservation that it found abandoned, not one that replaced it since.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts index f7159c3594..54533aa132 100644 --- a/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts +++ b/libraries/rush-client-core/src/test/connectOrStartDaemon.test.ts @@ -33,8 +33,17 @@ import { type IDaemonStartCommand } from '../connectOrStartDaemon'; import { executeWithDaemonRestartAsync, type IDaemonRestartNotice } from '../executeWithDaemonRestart'; -import { getDaemonStartupFilePath, releaseDaemonStartup, reserveDaemonStartup } from '../DaemonStartup'; -import { inspectDaemonStartupReservation } from '../DaemonStartupReservation'; +import { + getDaemonStartupFilePath, + readDaemonStartupReservation, + releaseDaemonStartup, + reserveDaemonStartup, + type IDaemonStartupReservation +} from '../DaemonStartup'; +import { + inspectDaemonStartupReservation, + tryTakeOverAbandonedStartupReservationAsync +} from '../DaemonStartupReservation'; import { tryAcquireStartupLockAsync, type IStartupLock } from '../StartupLock'; import { removeTestFolderAsync, waitForTestProcessExitAsync } from './TestProcessExit'; @@ -438,6 +447,22 @@ describe('detached daemon startup', () => { expect(readTakeOverLines()).toEqual([]); }, 30000); + it('takes over only the reservation that it found abandoned, not one that replaced it since', async () => { + const startupPath: string = getDaemonStartupFilePath(paths); + writeReservation(await getExitedPidAsync(), new Date(Date.now() - 2 * RELAUNCH_DELAY_MS).toISOString()); + const abandoned: IDaemonStartupReservation = readDaemonStartupReservation(paths)!; + // After the caller read it, the reservation was replaced by one whose helper still runs. + const replacement: string = writeReservation(process.pid); + expect(await tryTakeOverAbandonedStartupReservationAsync(paths, abandoned)).toBe(false); + expect(fs.readFileSync(startupPath, 'utf8')).toBe(replacement); + + // Unchanged, the same reservation is taken over. + fs.writeFileSync(startupPath, abandoned.contents!); + expect(await tryTakeOverAbandonedStartupReservationAsync(paths, abandoned)).toBe(true); + expect(fs.existsSync(startupPath)).toBe(false); + expect(fs.existsSync(path.join(folder, 'starts'))).toBe(false); + }); + it('lets exactly one of several clients take over a reservation whose helper exited', async () => { const helperPid: number = await getExitedPidAsync(); writeReservation(helperPid, new Date(Date.now() - 2 * RELAUNCH_DELAY_MS).toISOString()); From a4b4900f01358898e9c1163e885ad650dcc59a9f Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 05:04:01 +0000 Subject: [PATCH 078/265] [rush-lib] Type PhasedCommandEngineUsageError's options without ErrorOptions Swarm integration step 63; original commit f6d03bcffd (merge of swarm/r04-t78-erroropts at 17c6dacf70). Scope: task 196. Brings r04's fix for ch01's BUG board 3247 (board 3285): the constructor takes `options?: { cause?: unknown }` instead of `ErrorOptions`, so consumers of the rush-lib typings compiled with an older lib, such as the three es2017 build-tests projects, build again. Second agent: o04 board 3354. s17 batch D2, item 1 of 6. Gate: ch01 GATE OK board 3514 (tree 813312a4d4) Commits folded into this step (1): - 17c6dacf70 Type PhasedCommandEngineUsageError's options without ErrorOptions, so that consumers of the rush-lib typings with an older lib compile (task 78 fix, ch01 board 3247) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../swarm-r04-t78-erroropts_2026-09-29-03-57.json | 11 +++++++++++ common/reviews/api/rush-lib.api.md | 4 +++- .../rush-lib/src/api/PhasedCommandEngineUsageError.ts | 3 ++- 3 files changed, 16 insertions(+), 2 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r04-t78-erroropts_2026-09-29-03-57.json diff --git a/common/changes/@microsoft/rush/swarm-r04-t78-erroropts_2026-09-29-03-57.json b/common/changes/@microsoft/rush/swarm-r04-t78-erroropts_2026-09-29-03-57.json new file mode 100644 index 0000000000..b1abe0d517 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r04-t78-erroropts_2026-09-29-03-57.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "The constructor of `PhasedCommandEngineUsageError` (alpha) takes `options?: { cause?: unknown }` instead of `ErrorOptions`, so that projects that compile against the rush-lib or rush-sdk typings with a `lib` older than es2022 no longer fail with TS2304 \"Cannot find name 'ErrorOptions'\".", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 2a09f55a8f..3555800b90 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -1600,7 +1600,9 @@ export class PhasedCommandEngineProjectConfigurationError extends Error { // @alpha export class PhasedCommandEngineUsageError extends Error { - constructor(message: string, exitCode: number, options?: ErrorOptions); + constructor(message: string, exitCode: number, options?: { + cause?: unknown; + }); readonly exitCode: number; } diff --git a/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts b/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts index 51a8afa15e..f487c58c1a 100644 --- a/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts +++ b/libraries/rush-lib/src/api/PhasedCommandEngineUsageError.ts @@ -11,7 +11,8 @@ export class PhasedCommandEngineUsageError extends Error { /** The exit code of native Rush for this command line. */ public readonly exitCode: number; - public constructor(message: string, exitCode: number, options?: ErrorOptions) { + // The shape of ErrorOptions, which needs lib es2022. Consumers of these typings may use an older lib. + public constructor(message: string, exitCode: number, options?: { cause?: unknown }) { super(message, options); this.name = 'PhasedCommandEngineUsageError'; this.exitCode = exitCode; From 6121a8288bbe33e66ace39b72d89f7f464afd4a8 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 05:04:01 +0000 Subject: [PATCH 079/265] [rush-cli-client] A request that waits for a daemon restart says why Swarm integration step 64; original commit 12732cc337 (merge of swarm/r05-t166-int3 at 218ab4f8c3). Scope: task 166. Brings r05's task 166 (board 3041, board 3112), re-tipped with 170-int for batch D2 (board 3271): a request that waits for a daemon restart says why and what it waits for, within 0.5 s and every 25 s after that, and a waiting rushx-client says so with its own prefix. Errors name the real cause instead of "its environment". Second agent: o05 board 3168. s17 batch D2, item 2 of 6. Gate: ch01 GATE OK board 3514 (tree 17650f1da7) Commits folded into this step (6): - 2a0b2adb63 rush-daemon-protocol: a workspaceInputsChanged restart reason, and restart-wait details on queue positions (task 166) - a0c391d2ac rush-lib: WorkspaceRuntimeFingerprintCache.changedInstallationPaths (task 166) - 282d202be7 rush-client-core: name daemon restart causes and report restart waits and input admission (task 166) - b90a73b279 rush-daemon: say why a request waits for a daemon restart, in its queue positions and admission errors (task 166) - 20f173e57e rush-cli-client: say why and for what a request waits for a daemon restart, on pipes too, and repeat it every 25 s (task 166) - 2e3c61e239 rush-cli-client: no restart wait line once a cancellation is requested (task 166 with 132) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 44 ++- .../src/AgentProgressRenderer.ts | 17 + .../src/ClientAdmissionControls.ts | 8 +- .../src/daemonRestartNotice.ts | 213 +++++++++++-- apps/rush-cli-client/src/launchClient.ts | 34 +- apps/rush-cli-client/src/resultDiagnostics.ts | 14 +- .../src/test/AgentProgressRenderer.test.ts | 57 ++++ .../src/test/cancellationNotice.test.ts | 24 ++ .../src/test/daemonRestartNotice.test.ts | 299 +++++++++++++++--- .../src/test/resultDiagnostics.test.ts | 23 ++ ...restart-wait-reasons_2026-09-29-02-05.json | 11 + ...restart-wait-reasons_2026-09-29-02-05.json | 11 + ...restart-wait-reasons_2026-09-29-02-05.json | 11 + ...restart-wait-reasons_2026-09-29-02-05.json | 11 + ...restart-wait-reasons_2026-09-29-02-05.json | 11 + common/reviews/api/rush-client-core.api.md | 15 +- .../reviews/api/rush-daemon-protocol.api.md | 12 +- common/reviews/api/rush-lib.api.md | 3 + libraries/rush-client-core/README.md | 13 +- .../rush-client-core/src/DaemonClient.ts | 38 ++- .../src/DaemonRestartCause.ts | 79 +++++ libraries/rush-client-core/src/index.ts | 4 +- .../src/test/DaemonClient.test.ts | 57 +++- .../src/test/DaemonRestartCause.test.ts | 89 ++++++ libraries/rush-daemon-protocol/README.md | 8 +- .../src/DaemonInstallationChange.ts | 7 +- .../src/DaemonRequestAdmission.ts | 9 +- .../src/DaemonWorkspaceInputsChange.ts | 27 ++ .../src/InstallationChangeValidation.ts | 14 +- .../src/RequestAdmissionControlValidation.ts | 19 ++ .../src/WorkspaceInputsChangeValidation.ts | 33 ++ libraries/rush-daemon-protocol/src/index.ts | 4 +- .../test/WorkspaceInputsRestartReason.test.ts | 82 +++++ libraries/rush-daemon/README.md | 9 +- .../src/WorkspaceRequestAdmission.ts | 59 ++-- .../src/WorkspaceRequestLifecycle.ts | 51 ++- .../src/WorkspaceRestartArbiter.ts | 122 ++++--- .../src/test/DaemonInstallationChange.test.ts | 2 +- .../src/test/RestartDrainAdmission.test.ts | 127 ++++++-- .../src/test/WorkspaceRestartArbiter.test.ts | 138 ++++++++ .../WorkspaceServedScriptAdmission.test.ts | 165 +++++++++- .../src/api/WorkspaceInputFingerprint.ts | 59 +++- .../test/WorkspaceInputFingerprint.test.ts | 8 + 43 files changed, 1805 insertions(+), 236 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json create mode 100644 common/changes/@rushstack/rush-cli-client/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json create mode 100644 common/changes/@rushstack/rush-client-core/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json create mode 100644 common/changes/@rushstack/rush-daemon-protocol/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json create mode 100644 libraries/rush-client-core/src/DaemonRestartCause.ts create mode 100644 libraries/rush-client-core/src/test/DaemonRestartCause.test.ts create mode 100644 libraries/rush-daemon-protocol/src/DaemonWorkspaceInputsChange.ts create mode 100644 libraries/rush-daemon-protocol/src/WorkspaceInputsChangeValidation.ts create mode 100644 libraries/rush-daemon-protocol/src/test/WorkspaceInputsRestartReason.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index 8082733f1c..c3d3cf8ba8 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -90,14 +90,16 @@ timeout, the client exits with code 1 and suggests `--wait-timeout`. It does not suggest exporting `RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS`, because Rush versions that do not recognize a `RUSH_` environment variable fail every command while it is set. In legacy output and in `rushx-client`, the admission failure line -(`rush-client: daemon admission failed (wait-timeout): …`, or `(no-wait)`) gives the -daemon's reason, as agent mode's summary line does, so it names what the request -waited for, such as a daemon restart. +(`rush-client: daemon admission failed (wait-timeout): …`, or `(no-wait)`; in +`rushx-client` it begins with `rushx-client:`) gives the daemon's reason, as agent +mode's summary line does, so it names what the request waited for, such as a daemon +restart, and why the daemon restarts. Admission controls also apply to experimental graph requests, but not `start|stop|restart|status|logs`. They affect daemon admission only; native fallback retains native command behavior. Waiting positions are shown on interactive stderr, -and admission failures report their typed reason and a nonzero exit code. +and a wait for a daemon restart on a pipe too (see below). Admission failures report +their typed reason and a nonzero exit code. Explicit reporter/output/log-level controls (`--reporter`, `--output`, `--log-level`, `RUSH_REPORTER` other than `legacy`, or `RUSH_LOG_LEVEL`) retain the native frontend @@ -266,20 +268,38 @@ results before the old connection closes. When the daemon's own installation was removed or replaced (for example a deleted snapshot folder or a reinstalled Rush release), the daemon lets its running requests finish, answers each other request with that typed restart once they have, and then -exits. While a command waits, the agent progress status (or stderr: on a terminal at -each position, on a pipe once) says why: -`rush-client: waiting for the running requests to finish (position 1); the daemon -(PID ) then restarts, because its installation at was removed.` The -timeout rules of a restart for the request's environment apply (see above): the -built-in default does not limit waiting for the requests that were running when the -command arrived, but still limits it while the daemon runs a `rushx` script, and -`--no-wait` and an explicit `--wait-timeout` limit the whole wait. A command that +exits. The timeout rules of a restart for the request's environment apply (see +above): the built-in default does not limit waiting for the requests that were running +when the command arrived, but still limits it while the daemon runs a `rushx` script, +and `--no-wait` and an explicit `--wait-timeout` limit the whole wait. A command that times out exits with code 1, names the changed folder and, if a script runs, suggests stopping it. Otherwise the client starts a daemon from its own launcher once the wait ends, resubmits the request, and prints one line on stderr (or above the agent progress rows): `rush-client: The daemon's installation at was removed; restarted the daemon (PID ).` +While a command waits for a daemon restart, for its installation, the command's +environment or the workspace's inputs, the agent progress status (or stderr, on a +terminal and on a pipe) says what it waits for and why the daemon restarts, as soon as +the daemon reports the wait: `rush-client: waiting for 2 running requests to finish, +including 1 rushx script; the daemon (PID ) then restarts, because +common/config/rush/pnpm-lock.yaml changed.` After `because`, the cause is `its +installation at was removed` (or `replaced`), `this request's environment +differs from the daemon's in NODE_OPTIONS` (variable names, never their values), +` changed` for the workspace's installation, `the code of Rush or a Rush plugin +changed ()`, or `this request selects Rush `. The count names `rushx` +scripts, because a script such as a dev server may run until it is stopped. A +`rushx-client` script that waits for another request's restart prints `rushx-client: +waiting for the daemon (PID ) to restart for another request (2 requests ahead), +because .` The line goes to the stderr that the script writes to, which is a +pipe, because a `rushx-client` with a terminal runs the script in-process. A terminal +gets a line whenever the wait changes, and a pipe when the wait begins or its cause +changes. Both get the line again with the time waited (`still waiting after 25s for +…`) whenever 25 seconds pass without one, until the command follows the restart, +starts, or ends. Once the client asks rushd to cancel the command (Ctrl+C) and says so, +it writes no more wait lines. In agent mode, the progress phase says it, and on a pipe +a status line is written at once when the wait begins or its cause changes. + When the daemon restarts for a command's environment, the client prints a line of the same kind that names the variables that differed, never their values: `rush-client: A command's environment differed from the daemon's in NODE_OPTIONS; diff --git a/apps/rush-cli-client/src/AgentProgressRenderer.ts b/apps/rush-cli-client/src/AgentProgressRenderer.ts index d73665441f..f72c577f90 100644 --- a/apps/rush-cli-client/src/AgentProgressRenderer.ts +++ b/apps/rush-cli-client/src/AgentProgressRenderer.ts @@ -299,6 +299,23 @@ export class AgentProgressRenderer { this.setPhase(`queued behind another request (position ${position})`); } + /** + * The request waits for a daemon restart, and `wait` says why and for what. The phase and the status lines say + * that instead of a queue position. On a pipe, `announce` writes a status line at once, so that the wait and its + * cause are known when the wait begins or its cause changes, rather than at the next status line. Once the client + * asked rushd to cancel the request, the cancelling phase stays, and nothing is written. + */ + public onRestartWait(wait: string, announce: boolean): void { + if (this.#cancelling) { + return; + } + this.#queued = undefined; + this.setPhase(wait); + if (announce && !this.#options.isTTY) { + this.#writePipeLine(this.#getStatusLine()); + } + } + /** * The client asked rushd to cancel the request, and waits up to `timeoutMs` for rushd to stop it, which can take * seconds while rushd prepares the workspace graph. Says so at once, and on a TTY until the end; on a pipe, in one diff --git a/apps/rush-cli-client/src/ClientAdmissionControls.ts b/apps/rush-cli-client/src/ClientAdmissionControls.ts index 9be7c67b63..cbb6ce7d63 100644 --- a/apps/rush-cli-client/src/ClientAdmissionControls.ts +++ b/apps/rush-cli-client/src/ClientAdmissionControls.ts @@ -75,6 +75,9 @@ export function getConfiguredAdmission(options: IConfiguredAdmissionOptions): ID // Only the per-invocation flag is offered: Rush versions that do not recognize the variable reject it. const WAIT_LONGER_REMEDY: string = 'To wait longer, pass --wait-timeout .'; +/** The client that writes a line, which begins with its name. */ +export type ClientName = 'rush-client' | 'rushx-client'; + /** * Explains a daemon admission failure and how to wait longer. * @@ -85,9 +88,10 @@ const WAIT_LONGER_REMEDY: string = 'To wait longer, pass --wait-timeout 0 ? `, including ${formatCount(scriptCount, 'rushx script')}` : ''; +} + /** - * Returns what a queued request waits for when the daemon answers it with a restart once the requests ahead of it - * finish, or `undefined` for a queue position that needs no explanation. + * Says what a request that waits for a daemon restart waits for and why the daemon restarts, for example + * "waiting for 2 running requests to finish, including 1 rushx script; the daemon (PID 41) then restarts, because + * common/config/rush/pnpm-lock.yaml changed". A rushx script that waits for another request's restart gets + * "waiting for the daemon (PID 41) to restart for another request (2 requests ahead), because ...". The line names + * environment variables, never their values, and leaves out a cause that this client cannot word. */ -export function formatDaemonRestartWait( - position: number, - reason: DaemonRestartReason | undefined, - daemonPid: number | undefined -): string | undefined { - if (reason?.kind !== 'installationChanged') return undefined; +export function formatDaemonRestartWait(options: IDaemonRestartWaitLineOptions): string { + const { position, reason, details, daemonPid, elapsedMs = 0 } = options; + const anotherRequest: boolean = !!details.restartsForAnotherRequest; + const cause: string | undefined = formatDaemonRestartCause( + reason, + anotherRequest ? 'anotherRequest' : 'thisRequest' + ); + const because: string = cause === undefined ? '' : `, ${cause}`; const pid: string = daemonPid === undefined ? '' : ` (PID ${daemonPid})`; - return ( - `waiting for the running requests to finish (position ${position}); the daemon${pid} then restarts, ` + - `because its installation at ${reason.folder} was ${reason.change}` + const waiting: string = + elapsedMs < 1000 ? 'waiting' : `still waiting after ${Math.round(elapsedMs / 1000)}s`; + const scriptCount: number = details.scriptCount ?? 0; + if (anotherRequest) { + const ahead: string = `${formatCount(position, 'request')} ahead${formatIncludedScripts(scriptCount)}`; + return `${waiting} for the daemon${pid} to restart for another request (${ahead})${because}`; + } + // Scripts are named, since a script such as a dev server may run until it is stopped. + const allScripts: boolean = scriptCount >= position; + const running: string = formatCount(position, allScripts ? 'running rushx script' : 'running request'); + const scripts: string = allScripts ? '' : formatIncludedScripts(scriptCount); + return `${waiting} for ${running} to finish${scripts}; the daemon${pid} then restarts${because}`; +} + +/** The part of a restart wait that a pipe gets a line for at once when it changes: whose restart, and why. */ +function getRestartWaitCause(wait: IDaemonRestartWait): string { + const anotherRequest: boolean = !!wait.details.restartsForAnotherRequest; + const cause: string | undefined = formatDaemonRestartCause( + wait.reason, + anotherRequest ? 'anotherRequest' : 'thisRequest' ); + return `${anotherRequest}:${cause ?? wait.reason.kind}`; +} + +/** The agent renderer methods that tell an agent why a request waits or restarted. */ +export interface IAgentRequestNoticeRenderer { + note(line: string): void; + setPhase(phase: string): void; + onQueuePosition(position: number): void; + /** See `AgentProgressRenderer.onRestartWait`. */ + onRestartWait(wait: string, announce: boolean): void; } /** * Where a request's queue positions and restart lines go. */ export interface IDaemonRequestNoticeTarget extends IDaemonRestartNoticeTarget { - readonly agentRenderer: - | { note(line: string): void; setPhase(phase: string): void; onQueuePosition(position: number): void } - | undefined; - /** On a pipe, only the first {@link formatDaemonRestartWait} line is written, and plain positions are not. */ + readonly agentRenderer: IAgentRequestNoticeRenderer | undefined; + /** + * Without an agent renderer, a terminal gets a line for every change of a restart wait and every queue position, + * and a pipe gets a line when a restart wait begins or its cause changes. Both get the restart wait line again + * with the time waited when {@link RESTART_WAIT_REPEAT_MS} passes without one. + */ readonly stderrIsTTY: boolean; /** The process ID of the daemon that serves the request first. */ readonly daemonPid: number | undefined; + /** Returns the current time in milliseconds. Defaults to `Date.now`. */ + readonly now?: () => number; } /** - * The `onRestartAsync` and `onQueuePositionAsync` callbacks of one request. + * The callbacks of one request that tell the user why it waits or restarted. */ export interface IDaemonRequestNoticeHandlers { readonly onRestartAsync: (notice: IDaemonRestartNotice) => Promise; - readonly onQueuePositionAsync: (position: number, restartReason?: DaemonRestartReason) => Promise; + readonly onQueuePositionAsync: ( + position: number, + restartReason?: DaemonRestartReason, + restartWait?: IDaemonRestartWaitDetails + ) => Promise; + /** The daemon admitted the request's input, so a rushx script starts, and no longer waits. */ + readonly onInputAdmittedAsync: () => Promise; + /** The request's output or an event arrived, so it no longer waits. */ + readonly onRequestProgress: () => void; + /** + * The request ended, or the client asked rushd to cancel it: stops repeating the restart wait line, and ignores + * later queue positions. + */ + readonly dispose: () => void; +} + +interface IRestartWaitState { + readonly startedAtMs: number; + wait: IDaemonRestartWait; + /** The cause of the last restart wait that was announced; see {@link getRestartWaitCause}. */ + announcedCause: string | undefined; + /** The last restart wait line without the time waited, so that a terminal gets a line for each change. */ + line: string | undefined; + timer: ReturnType | undefined; } /** - * Creates the callbacks that tell the user why a request waits or restarted. Queue positions name the daemon that - * serves the request, which changes when the request follows a restart. + * Creates the callbacks that tell the user why a request waits or restarted. The lines name the daemon that serves + * the request, which changes when the request follows a restart. A restart wait line is repeated until the request + * restarts, gets input, output or an event, or waits for plain admission instead. Once the request ends or the + * client asks rushd to cancel it (see `dispose`), nothing more is written. */ export function createDaemonRequestNoticeHandlers( target: IDaemonRequestNoticeTarget ): IDaemonRequestNoticeHandlers { - const { agentRenderer, stderrIsTTY } = target; + const { agentRenderer, stderrIsTTY, now = Date.now } = target; + const prefix: string = target.rushx ? 'rushx-client' : 'rush-client'; const writeRestartNoticeAsync: (notice: IDaemonRestartNotice) => Promise = createDaemonRestartNoticeHandler(target); let daemonPid: number | undefined = target.daemonPid; - let wroteRestartWait: boolean = false; + let restartWait: IRestartWaitState | undefined; + let showedRestartWait: boolean = false; + let disposed: boolean = false; + + const endRestartWait = (): void => { + clearTimeout(restartWait?.timer); + restartWait = undefined; + }; + const writeRestartWaitAsync = async (state: IRestartWaitState): Promise => { + clearTimeout(state.timer); + state.timer = setTimeout(() => { + if (restartWait === state) writeRestartWaitAsync(state).catch(() => undefined); + }, RESTART_WAIT_REPEAT_MS); + state.timer.unref?.(); + const elapsedMs: number = now() - state.startedAtMs; + const line: string = formatDaemonRestartWait({ ...state.wait, daemonPid, elapsedMs }); + await target.writeStderrAsync(`${prefix}: ${line}.\n`); + }; + return { onRestartAsync: async (notice: IDaemonRestartNotice): Promise => { + endRestartWait(); daemonPid = notice.successorPid; await writeRestartNoticeAsync(notice); // The phase still says that the request waits for the previous daemon. - if (agentRenderer && wroteRestartWait) agentRenderer.setPhase(RESUBMITTED_PHASE); - wroteRestartWait = false; + if (agentRenderer && showedRestartWait) agentRenderer.setPhase(RESUBMITTED_PHASE); + showedRestartWait = false; }, - onQueuePositionAsync: async (position: number, restartReason?: DaemonRestartReason): Promise => { - const wait: string | undefined = formatDaemonRestartWait(position, restartReason, daemonPid); + onQueuePositionAsync: async ( + position: number, + restartReason?: DaemonRestartReason, + details: IDaemonRestartWaitDetails = {} + ): Promise => { + if (disposed) return; + if (!restartReason) { + endRestartWait(); + if (agentRenderer) agentRenderer.onQueuePosition(position); + else if (stderrIsTTY) { + await target.writeStderrAsync(`${prefix}: waiting for daemon admission (position ${position}).\n`); + } + return; + } + const wait: IDaemonRestartWait = { position, reason: restartReason, details }; + const state: IRestartWaitState = (restartWait ??= { + startedAtMs: now(), + wait, + announcedCause: undefined, + line: undefined, + timer: undefined + }); + state.wait = wait; + const cause: string = getRestartWaitCause(wait); + const announce: boolean = cause !== state.announcedCause; + state.announcedCause = cause; + const line: string = formatDaemonRestartWait({ ...wait, daemonPid }); + const changed: boolean = line !== state.line; + state.line = line; if (agentRenderer) { - wroteRestartWait ||= wait !== undefined; - if (wait) agentRenderer.setPhase(wait); - else agentRenderer.onQueuePosition(position); - } else if (wait && (stderrIsTTY || !wroteRestartWait)) { - wroteRestartWait = true; - await target.writeStderrAsync(`${target.rushx ? 'rushx-client' : 'rush-client'}: ${wait}.\n`); - } else if (!wait && stderrIsTTY) { - await target.writeStderrAsync(`rush-client: waiting for daemon admission (position ${position}).\n`); + showedRestartWait = true; + // The agent renderer's own status lines repeat the phase. + agentRenderer.onRestartWait(line, announce); + } else if (announce || (stderrIsTTY && changed)) { + await writeRestartWaitAsync(state); } + }, + onInputAdmittedAsync: async (): Promise => endRestartWait(), + onRequestProgress: endRestartWait, + dispose: (): void => { + disposed = true; + endRestartWait(); } }; } diff --git a/apps/rush-cli-client/src/launchClient.ts b/apps/rush-cli-client/src/launchClient.ts index e50e7f0355..485d03ccad 100644 --- a/apps/rush-cli-client/src/launchClient.ts +++ b/apps/rush-cli-client/src/launchClient.ts @@ -40,7 +40,7 @@ import { readUseRushReporter } from './outputSelection'; import { selectClientRoute, type IClientRoute } from './routing'; import { getResultStderr } from './resultDiagnostics'; import { getTerminalColumns } from './terminalColumns'; -import { createDaemonRequestNoticeHandlers } from './daemonRestartNotice'; +import { createDaemonRequestNoticeHandlers, type IDaemonRequestNoticeHandlers } from './daemonRestartNotice'; import { formatInProcessFallbackMessage } from './inProcessFallback'; import { writeStreamAsync } from './writeStreamAsync'; import { @@ -186,8 +186,11 @@ export async function launchClientAsync( cancellationSignal ??= signal ?? 'SIGINT'; abort.abort(); }; + let notices: IDaemonRequestNoticeHandlers | undefined; const onCancelRequested = (timeoutMs: number): void => { cancelRequested = true; + // A line that the request still waits for a daemon restart would contradict the cancelling line. + notices?.dispose(); if (agentRenderer) { agentRenderer.onCancelRequested(timeoutMs); return; @@ -229,28 +232,36 @@ export async function launchClientAsync( } await renderer.initializeAsync(); agentRenderer?.onRequestSent(); + const requestNotices: IDaemonRequestNoticeHandlers = createDaemonRequestNoticeHandlers({ + rushx, + agentRenderer, + stderrIsTTY: !!process.stderr.isTTY, + daemonPid: (await client.status).pid, + writeStderrAsync: (text) => writeStreamAsync(process.stderr, Buffer.from(text)) + }); + notices = requestNotices; outcome = await executeWithDaemonRestartAsync(client, connection, { request, abortSignal: abort.signal, onStdoutAsync: async (bytes, operationId) => { + requestNotices.onRequestProgress(); if (agentRenderer) return agentRenderer.onLog(bytes, operationId, 'stdout'); await writeDiscoveryAsync(); await renderer.writeLogAsync(bytes, operationId, 'stdout'); }, onStderrAsync: async (bytes, operationId) => { + requestNotices.onRequestProgress(); if (agentRenderer) return agentRenderer.onLog(bytes, operationId, 'stderr'); await writeDiscoveryAsync(); await renderer.writeLogAsync(bytes, operationId, 'stderr'); }, - onEventAsync: async (event) => - agentRenderer ? agentRenderer.onEvent(event) : renderer.writeEventAsync(event), - ...createDaemonRequestNoticeHandlers({ - rushx, - agentRenderer, - stderrIsTTY: !!process.stderr.isTTY, - daemonPid: (await client.status).pid, - writeStderrAsync: (text) => writeStreamAsync(process.stderr, Buffer.from(text)) - }), + onEventAsync: async (event) => { + requestNotices.onRequestProgress(); + return agentRenderer ? agentRenderer.onEvent(event) : renderer.writeEventAsync(event); + }, + onRestartAsync: requestNotices.onRestartAsync, + onQueuePositionAsync: requestNotices.onQueuePositionAsync, + onInputAdmittedAsync: requestNotices.onInputAdmittedAsync, stdin: process.stdin, requiresStdinEnd: !process.stdin.isTTY, cancelOnCtrlC: !!process.stdin.isTTY, @@ -267,6 +278,7 @@ export async function launchClientAsync( if (!isCancelled() || !(error instanceof DaemonClientError)) throw error; outcome = undefined; } finally { + notices?.dispose(); for (const signal of CANCELLATION_SIGNALS) process.removeListener(signal, onSignal); try { await renderer.closeAsync(); @@ -298,7 +310,7 @@ export async function launchClientAsync( // When the agent summary line explains the failure, nothing more is printed. const stderr: string | undefined = reportedByAgent ? undefined - : getResultStderr(outcome.result, request.admission); + : getResultStderr(outcome.result, request.admission, rushx ? 'rushx-client' : 'rush-client'); if (stderr) { await writeStreamAsync(process.stderr, Buffer.from(stderr)); } diff --git a/apps/rush-cli-client/src/resultDiagnostics.ts b/apps/rush-cli-client/src/resultDiagnostics.ts index 6657102ad8..9a012c5309 100644 --- a/apps/rush-cli-client/src/resultDiagnostics.ts +++ b/apps/rush-cli-client/src/resultDiagnostics.ts @@ -3,7 +3,7 @@ import type { IDaemonCommandResult, IDaemonRequestAdmissionOptions } from '@rushstack/rush-daemon-protocol'; -import { formatAdmissionFailure } from './ClientAdmissionControls'; +import { formatAdmissionFailure, type ClientName } from './ClientAdmissionControls'; /** * Returns the stderr line that explains a failed daemon result, if any. @@ -15,13 +15,14 @@ import { formatAdmissionFailure } from './ClientAdmissionControls'; * explains. */ export function getResultDiagnostic( - result: Pick + result: Pick, + clientName: ClientName = 'rush-client' ): string | undefined { // A request aborted while waiting for admission carries the reason (such as a daemon shutdown) in its // error message; other admission failures are explained by `formatAdmissionFailure`. if (result.admissionErrorCode !== undefined && result.admissionErrorCode !== 'aborted') return undefined; if (result.exitCode !== 0 && result.errorMessage) { - return `rush-client: ${result.errorMessage}\n`; + return `${clientName}: ${result.errorMessage}\n`; } return undefined; } @@ -35,9 +36,10 @@ export function getResultDiagnostic( */ export function getResultStderr( result: Pick, - admission: IDaemonRequestAdmissionOptions | undefined + admission: IDaemonRequestAdmissionOptions | undefined, + clientName: ClientName = 'rush-client' ): string | undefined { - const diagnostic: string | undefined = getResultDiagnostic(result); + const diagnostic: string | undefined = getResultDiagnostic(result, clientName); if (diagnostic || !result.admissionErrorCode) return diagnostic; - return formatAdmissionFailure(result.admissionErrorCode, admission, result.errorMessage); + return formatAdmissionFailure(result.admissionErrorCode, admission, result.errorMessage, clientName); } diff --git a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts index 9531de11f5..bd93567469 100644 --- a/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts +++ b/apps/rush-cli-client/src/test/AgentProgressRenderer.test.ts @@ -1128,6 +1128,49 @@ describe(AgentProgressRenderer.name, () => { ]); }); + it('writes a restart wait at once when it is announced, and then in place of the queue position (task 166)', () => { + const { renderer, clock, lines } = createRenderer(false); + const wait = (count: string): string => + `waiting for ${count} to finish; the daemon (PID 41) then restarts, because x changed`; + renderer.start(); + renderer.onRequestSent(); + advance(clock, 400); + renderer.onQueuePosition(1); + advance(clock, 600); + renderer.onRestartWait(wait('2 running requests'), true); + advance(clock, 1000); + renderer.onRestartWait(wait('1 running request'), false); + advance(clock, 23_999); + expect(lines()).toHaveLength(2); + advance(clock, 1); + renderer.dispose(); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + `rush build · 1.0s · ${wait('2 running requests')}`, + `rush build · 26.0s · ${wait('1 running request')}` + ]); + }); + + it('writes no restart wait once the client asked rushd to cancel the request (tasks 166 and 132)', () => { + const { renderer, clock, lines } = createRenderer(false); + const wait: string = + 'waiting for 1 running request to finish; the daemon (PID 41) then restarts, because x changed'; + renderer.start(); + renderer.onRequestSent(); + renderer.onRestartWait(wait, true); + advance(clock, 2000); + renderer.onCancelRequested(5_000); + renderer.onRestartWait(`${wait} again`, true); + advance(clock, 25_000); + renderer.dispose(); + expect(lines()).toEqual([ + 'rush build · 0.0s · sent to rushd; preparing the workspace graph (status at least every 25s)', + `rush build · 0.0s · ${wait}`, + 'rush build · 2.0s · cancelling; waiting up to 5s for rushd to stop the request', + 'rush build · 27.0s · cancelling; waiting up to 5s for rushd to stop the request' + ]); + }); + it('writes a note like any other line, so the next status line is due 25 s after it', () => { const { renderer, clock, lines } = createRenderer(false); renderer.start(); @@ -1160,6 +1203,20 @@ describe(AgentProgressRenderer.name, () => { ]); }); + it('shows a restart wait as the phase of the live rows on a TTY, and writes no line for it', () => { + const { renderer, output } = createRenderer(true, 'build', 200); + renderer.start(); + renderer.onRestartWait( + 'waiting for 1 running request to finish; the daemon (PID 41) then restarts', + true + ); + expect(output).toHaveLength(2); + expect(output[1].replace(ANSI_ESCAPE, '')).toMatch( + /^. rush build · 0\.0s · waiting for 1 running request to finish; the daemon \(PID 41\) then restarts\n/ + ); + renderer.dispose(); + }); + it('writes a note above the live rows on a TTY and redraws them below it', () => { const { renderer, output } = createRenderer(true); renderer.start(); diff --git a/apps/rush-cli-client/src/test/cancellationNotice.test.ts b/apps/rush-cli-client/src/test/cancellationNotice.test.ts index c48e0c09a2..77b0588ee4 100644 --- a/apps/rush-cli-client/src/test/cancellationNotice.test.ts +++ b/apps/rush-cli-client/src/test/cancellationNotice.test.ts @@ -20,6 +20,7 @@ import { type DaemonClientOutcome, type IDaemonClientExecuteOptions } from '@rushstack/rush-client-core'; +import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; import { AgentProgressRenderer } from '../AgentProgressRenderer'; import * as connectionOptions from '../daemonConnectionOptions'; @@ -153,6 +154,29 @@ describe('the cancellation of a daemon request (task 132)', () => { expect(process.exitCode).toBe(130); }); + it('writes no restart wait line once it asks rushd to cancel (task 166)', async () => { + const lockfile: DaemonRestartReason = { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/pnpm-lock.yaml'] + }; + const environment: DaemonRestartReason = { kind: 'environmentChanged', variableNames: ['FOO'] }; + execute(async (options) => { + await options.onQueuePositionAsync!(1, lockfile, { scriptCount: 1 }); + deliverSignal('SIGINT'); + requestCancel(options); + // A new cause would get a line at once on a pipe. + await options.onQueuePositionAsync!(1, environment, { scriptCount: 1 }); + return aborted(options); + }); + await launchClientAsync(false); + expect(stderr).toEqual([ + `rush-client: waiting for 1 running rushx script to finish; the daemon (PID ${process.pid}) then restarts, ` + + 'because common/config/rush/pnpm-lock.yaml changed.\n', + CANCELLING, + CANCELLED + ]); + }); + it('says so when rushd does not confirm the stop before the cancellation deadline', async () => { execute(async (options) => { deliverSignal('SIGTERM'); diff --git a/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts b/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts index 4ca8860fe3..09732dc527 100644 --- a/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts +++ b/apps/rush-cli-client/src/test/daemonRestartNotice.test.ts @@ -1,9 +1,11 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { IDaemonRestartWaitDetails } from '@rushstack/rush-client-core'; import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; import { + RESTART_WAIT_REPEAT_MS, RESUBMITTED_PHASE, createDaemonRequestNoticeHandlers, createDaemonRestartNoticeHandler, @@ -125,26 +127,120 @@ const REMOVED: DaemonRestartReason = { change: 'removed', folder: '/snapshots/s9' }; +const LOCKFILE: DaemonRestartReason = { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/pnpm-lock.yaml'] +}; +const PLUGIN_CODE: DaemonRestartReason = { + kind: 'workspaceInputsChanged', + implementationFiles: ['common/autoinstallers/p/node_modules/q/lib/x.js'] +}; +const ANOTHER: IDaemonRestartWaitDetails = { restartsForAnotherRequest: true }; + +// One case for each reason that a daemon gives for a restart, and how a waiting request and a waiting rushx script +// word it. +const REASONS: [string, DaemonRestartReason, string, string][] = [ + [ + 'a removed installation', + REMOVED, + 'because its installation at /snapshots/s9 was removed', + "because the daemon's installation at /snapshots/s9 was removed" + ], + [ + 'an environment', + { kind: 'environmentChanged', variableNames: ['NODE_OPTIONS', 'RUSH_X'] }, + "because this request's environment differs from the daemon's in NODE_OPTIONS and RUSH_X", + "because its environment differs from the daemon's in NODE_OPTIONS and RUSH_X" + ], + [ + 'a lockfile', + LOCKFILE, + 'because common/config/rush/pnpm-lock.yaml changed', + 'because common/config/rush/pnpm-lock.yaml changed' + ], + [ + 'plugin code', + PLUGIN_CODE, + 'because the code of Rush or a Rush plugin changed (common/autoinstallers/p/node_modules/q/lib/x.js)', + 'because the code of Rush or a Rush plugin changed (common/autoinstallers/p/node_modules/q/lib/x.js)' + ], + [ + 'a Rush version', + { kind: 'workspaceInputsChanged', selectedRushVersion: '5.180.0' }, + 'because this request selects Rush 5.180.0', + 'because it selects Rush 5.180.0' + ] +]; describe(formatDaemonRestartWait.name, () => { - it('says what the request waits for, and why the daemon then restarts', () => { - expect(formatDaemonRestartWait(2, REMOVED, 41)).toBe( - 'waiting for the running requests to finish (position 2); the daemon (PID 41) then restarts, because ' + - 'its installation at /snapshots/s9 was removed' + it.each(REASONS)('says why the daemon restarts for %s', (name, reason, cause, anotherCause) => { + expect(formatDaemonRestartWait({ position: 1, reason, details: {}, daemonPid: 41 })).toBe( + `waiting for 1 running request to finish; the daemon (PID 41) then restarts, ${cause}` ); - expect(formatDaemonRestartWait(1, { ...REMOVED, change: 'replaced' }, undefined)).toBe( - 'waiting for the running requests to finish (position 1); the daemon then restarts, because its ' + - 'installation at /snapshots/s9 was replaced' + expect(formatDaemonRestartWait({ position: 2, reason, details: ANOTHER, daemonPid: 41 })).toBe( + `waiting for the daemon (PID 41) to restart for another request (2 requests ahead), ${anotherCause}` ); }); - it('says nothing about plain queue positions or reasons that it does not know', () => { - expect(formatDaemonRestartWait(1, undefined, 41)).toBeUndefined(); - expect(formatDaemonRestartWait(1, NEWER_REASON, 41)).toBeUndefined(); + it('says how many of the requests that it waits for run a rushx script', () => { + const format = (position: number, details: IDaemonRestartWaitDetails): string => + formatDaemonRestartWait({ position, reason: LOCKFILE, details, daemonPid: undefined }); + expect(format(3, { scriptCount: 1 })).toBe( + 'waiting for 3 running requests to finish, including 1 rushx script; the daemon then restarts, because ' + + 'common/config/rush/pnpm-lock.yaml changed' + ); + expect(format(3, { scriptCount: 2 })).toMatch( + /^waiting for 3 running requests to finish, including 2 rushx scripts;/ + ); + expect(format(1, { scriptCount: 1 })).toMatch( + /^waiting for 1 running rushx script to finish; the daemon then/ + ); + expect(format(2, { scriptCount: 2 })).toMatch(/^waiting for 2 running rushx scripts to finish;/); + expect(format(2, { scriptCount: 0 })).toMatch(/^waiting for 2 running requests to finish;/); + expect(format(1, { ...ANOTHER, scriptCount: 0 })).toMatch( + /restart for another request \(1 request ahead\), / + ); + expect(format(3, { ...ANOTHER, scriptCount: 1 })).toMatch( + /^waiting for the daemon to restart for another request \(3 requests ahead, including 1 rushx script\), / + ); + }); + + it('says how long the request has waited, from a second on', () => { + const format = (elapsedMs: number, details: IDaemonRestartWaitDetails): string => + formatDaemonRestartWait({ position: 1, reason: LOCKFILE, details, daemonPid: 41, elapsedMs }); + expect(format(999, {})).toMatch(/^waiting for 1 running request/); + expect(format(25_400, {})).toMatch( + /^still waiting after 25s for 1 running request to finish; the daemon/ + ); + expect(format(50_000, ANOTHER)).toMatch( + /^still waiting after 50s for the daemon \(PID 41\) to restart for/ + ); + }); + + it('leaves out a cause that it cannot word', () => { + expect(formatDaemonRestartWait({ position: 2, reason: NEWER_REASON, details: {}, daemonPid: 41 })).toBe( + 'waiting for 2 running requests to finish; the daemon (PID 41) then restarts' + ); + expect( + formatDaemonRestartWait({ position: 2, reason: NEWER_REASON, details: ANOTHER, daemonPid: 41 }) + ).toBe('waiting for the daemon (PID 41) to restart for another request (2 requests ahead)'); }); }); describe(createDaemonRequestNoticeHandlers.name, () => { + let clock: number; + + beforeEach(() => { + jest.useFakeTimers(); + clock = 0; + }); + afterEach(() => jest.useRealTimers()); + + function advance(ms: number): void { + clock += ms; + jest.advanceTimersByTime(ms); + } + function createHandlers(options: { agent: boolean; stderrIsTTY: boolean; rushx?: boolean }): { calls: string[]; handlers: IDaemonRequestNoticeHandlers; @@ -154,11 +250,14 @@ describe(createDaemonRequestNoticeHandlers.name, () => { rushx: !!options.rushx, stderrIsTTY: options.stderrIsTTY, daemonPid: 41, + now: () => clock, agentRenderer: options.agent ? { note: (line: string) => calls.push(`note: ${line}`), setPhase: (phase: string) => calls.push(`phase: ${phase}`), - onQueuePosition: (position: number) => calls.push(`position: ${position}`) + onQueuePosition: (position: number) => calls.push(`position: ${position}`), + onRestartWait: (wait: string, announce: boolean) => + calls.push(`${announce ? 'announce' : 'wait'}: ${wait}`) } : undefined, writeStderrAsync: async (text: string) => { @@ -168,18 +267,158 @@ describe(createDaemonRequestNoticeHandlers.name, () => { return { calls, handlers }; } - it('shows a restart wait as the agent phase, and names the new daemon after the restart', async () => { + const LOCKFILE_WAIT: string = + 'the daemon (PID 41) then restarts, because common/config/rush/pnpm-lock.yaml changed'; + + describe.each([ + ['rush-client', false], + ['rushx-client', true] + ])('%s without an agent renderer', (client: string, rushx: boolean) => { + it.each(REASONS)( + 'writes the first line of a wait for %s at once, on a pipe and a terminal', + async (name, reason, cause) => { + for (const stderrIsTTY of [false, true]) { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY, rushx }); + await handlers.onQueuePositionAsync(1, reason, { scriptCount: 0 }); + handlers.dispose(); + expect(calls).toEqual([ + `stderr: ${client}: waiting for 1 running request to finish; the daemon (PID 41) then restarts, ${cause}.\n` + ]); + } + } + ); + + it('repeats the line with the time waited when 25 s pass without one, until the restart', async () => { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: false, rushx }); + advance(3000); + await handlers.onQueuePositionAsync(2, LOCKFILE, { scriptCount: 1 }); + advance(10_000); + // On a pipe, a new count waits for the next line. + await handlers.onQueuePositionAsync(1, LOCKFILE, {}); + advance(RESTART_WAIT_REPEAT_MS - 10_001); + expect(calls).toHaveLength(1); + advance(1); + advance(RESTART_WAIT_REPEAT_MS); + await handlers.onRestartAsync({ restart: 1, reason: undefined, successorPid: 42 }); + advance(RESTART_WAIT_REPEAT_MS * 4); + expect(calls).toEqual([ + `stderr: ${client}: waiting for 2 running requests to finish, including 1 rushx script; ${LOCKFILE_WAIT}.\n`, + `stderr: ${client}: still waiting after 25s for 1 running request to finish; ${LOCKFILE_WAIT}.\n`, + `stderr: ${client}: still waiting after 50s for 1 running request to finish; ${LOCKFILE_WAIT}.\n` + ]); + }); + + it('writes a line on a pipe at once when the cause changes, and every change on a terminal', async () => { + for (const stderrIsTTY of [false, true]) { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY, rushx }); + await handlers.onQueuePositionAsync(1, LOCKFILE, ANOTHER); + advance(500); + await handlers.onQueuePositionAsync(2, LOCKFILE, ANOTHER); + await handlers.onQueuePositionAsync(2, LOCKFILE, ANOTHER); + advance(1500); + // Another cause of the same kind. + await handlers.onQueuePositionAsync(2, PLUGIN_CODE, ANOTHER); + advance(1000); + await handlers.onQueuePositionAsync(2, REMOVED, ANOTHER); + handlers.dispose(); + const another: string = 'for the daemon (PID 41) to restart for another request'; + const lockfile: string = 'because common/config/rush/pnpm-lock.yaml changed'; + expect(calls).toEqual([ + `stderr: ${client}: waiting ${another} (1 request ahead), ${lockfile}.\n`, + ...(stderrIsTTY + ? [`stderr: ${client}: waiting ${another} (2 requests ahead), ${lockfile}.\n`] + : []), + `stderr: ${client}: still waiting after 2s ${another} (2 requests ahead), because the code of Rush or a ` + + 'Rush plugin changed (common/autoinstallers/p/node_modules/q/lib/x.js).\n', + `stderr: ${client}: still waiting after 3s ${another} (2 requests ahead), because the daemon's ` + + 'installation at /snapshots/s9 was removed.\n' + ]); + } + }); + + it('writes a line on a pipe when whose restart the request waits for changes, even for the same cause', async () => { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: false, rushx }); + await handlers.onQueuePositionAsync(1, LOCKFILE, {}); + await handlers.onQueuePositionAsync(1, LOCKFILE, ANOTHER); + handlers.dispose(); + expect(calls).toEqual([ + `stderr: ${client}: waiting for 1 running request to finish; ${LOCKFILE_WAIT}.\n`, + `stderr: ${client}: waiting for the daemon (PID 41) to restart for another request (1 request ahead), ` + + 'because common/config/rush/pnpm-lock.yaml changed.\n' + ]); + }); + + it('stops when the request gets input, output or an event, or waits for plain admission', async () => { + const stops: [string, (handlers: IDaemonRequestNoticeHandlers) => Promise | void][] = [ + ['input', (handlers) => handlers.onInputAdmittedAsync()], + ['progress', (handlers) => handlers.onRequestProgress()], + ['plain position', (handlers) => handlers.onQueuePositionAsync(1)] + ]; + for (const [, stop] of stops) { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: false, rushx }); + await handlers.onQueuePositionAsync(1, LOCKFILE, ANOTHER); + await stop(handlers); + advance(RESTART_WAIT_REPEAT_MS * 3); + expect(calls).toHaveLength(1); + // A later wait is a new one. + await handlers.onQueuePositionAsync(1, LOCKFILE, ANOTHER); + handlers.dispose(); + expect(calls).toHaveLength(2); + expect(calls[1]).toBe(calls[0]); + } + }); + + it('writes nothing more once the request ends or the client asks rushd to cancel it', async () => { + for (const stderrIsTTY of [false, true]) { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY, rushx }); + await handlers.onQueuePositionAsync(1, LOCKFILE, ANOTHER); + handlers.dispose(); + advance(RESTART_WAIT_REPEAT_MS * 3); + await handlers.onQueuePositionAsync(1, REMOVED, ANOTHER); + await handlers.onQueuePositionAsync(2); + expect(calls).toHaveLength(1); + } + }); + + it('names the new daemon after a restart, and writes plain positions only to a terminal', async () => { + for (const stderrIsTTY of [false, true]) { + const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY, rushx }); + await handlers.onQueuePositionAsync(3); + await handlers.onQueuePositionAsync(1, REMOVED, {}); + await handlers.onRestartAsync({ restart: 1, reason: REMOVED, successorPid: 42 }); + await handlers.onQueuePositionAsync(1, LOCKFILE, {}); + handlers.dispose(); + expect(calls).toEqual([ + ...(stderrIsTTY ? [`stderr: ${client}: waiting for daemon admission (position 3).\n`] : []), + `stderr: ${client}: waiting for 1 running request to finish; the daemon (PID 41) then restarts, because ` + + 'its installation at /snapshots/s9 was removed.\n', + `stderr: ${client}: The daemon's installation at /snapshots/s9 was removed; restarted the daemon (PID 42).\n`, + `stderr: ${client}: waiting for 1 running request to finish; the daemon (PID 42) then restarts, because ` + + 'common/config/rush/pnpm-lock.yaml changed.\n' + ]); + } + }); + }); + + it('shows a restart wait as the agent phase, announces each cause, and names the new daemon after the restart', async () => { const { calls, handlers } = createHandlers({ agent: true, stderrIsTTY: false }); await handlers.onQueuePositionAsync(2); - await handlers.onQueuePositionAsync(1, REMOVED); + await handlers.onQueuePositionAsync(2, LOCKFILE, { scriptCount: 1 }); + await handlers.onQueuePositionAsync(1, LOCKFILE, {}); + advance(RESTART_WAIT_REPEAT_MS * 2); + await handlers.onQueuePositionAsync(1, REMOVED, {}); await handlers.onRestartAsync({ restart: 1, reason: REMOVED, successorPid: 42 }); - await handlers.onQueuePositionAsync(1, REMOVED); + await handlers.onQueuePositionAsync(1, REMOVED, {}); + handlers.dispose(); + const removed: string = 'because its installation at /snapshots/s9 was removed'; expect(calls).toEqual([ 'position: 2', - `phase: ${formatDaemonRestartWait(1, REMOVED, 41)}`, + `announce: waiting for 2 running requests to finish, including 1 rushx script; ${LOCKFILE_WAIT}`, + `wait: waiting for 1 running request to finish; ${LOCKFILE_WAIT}`, + `announce: waiting for 1 running request to finish; the daemon (PID 41) then restarts, ${removed}`, "note: rush-client: The daemon's installation at /snapshots/s9 was removed; restarted the daemon (PID 42).", `phase: ${RESUBMITTED_PHASE}`, - `phase: ${formatDaemonRestartWait(1, REMOVED, 42)}` + `announce: waiting for 1 running request to finish; the daemon (PID 42) then restarts, ${removed}` ]); }); @@ -190,28 +429,12 @@ describe(createDaemonRequestNoticeHandlers.name, () => { expect(calls).toEqual(['position: 1']); }); - it('writes every queue position to a terminal', async () => { - const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: true }); + it('gives the agent renderer no more positions once the request ends or the client asks rushd to cancel it', async () => { + const { calls, handlers } = createHandlers({ agent: true, stderrIsTTY: false }); + await handlers.onQueuePositionAsync(1, LOCKFILE, {}); + handlers.dispose(); + await handlers.onQueuePositionAsync(1, REMOVED, {}); await handlers.onQueuePositionAsync(2); - await handlers.onQueuePositionAsync(2, REMOVED); - await handlers.onQueuePositionAsync(1, REMOVED); - expect(calls).toEqual([ - 'stderr: rush-client: waiting for daemon admission (position 2).\n', - `stderr: rush-client: ${formatDaemonRestartWait(2, REMOVED, 41)}.\n`, - `stderr: rush-client: ${formatDaemonRestartWait(1, REMOVED, 41)}.\n` - ]); - }); - - it('writes only the first restart wait for each daemon to a pipe', async () => { - const { calls, handlers } = createHandlers({ agent: false, stderrIsTTY: false, rushx: true }); - await handlers.onQueuePositionAsync(3); - await handlers.onQueuePositionAsync(2, REMOVED); - await handlers.onQueuePositionAsync(1, REMOVED); - await handlers.onRestartAsync({ restart: 1, reason: undefined, successorPid: 42 }); - await handlers.onQueuePositionAsync(1, REMOVED); - expect(calls).toEqual([ - `stderr: rushx-client: ${formatDaemonRestartWait(2, REMOVED, 41)}.\n`, - `stderr: rushx-client: ${formatDaemonRestartWait(1, REMOVED, 42)}.\n` - ]); + expect(calls).toEqual([`announce: waiting for 1 running request to finish; ${LOCKFILE_WAIT}`]); }); }); diff --git a/apps/rush-cli-client/src/test/resultDiagnostics.test.ts b/apps/rush-cli-client/src/test/resultDiagnostics.test.ts index f64b6e2f72..95ec573665 100644 --- a/apps/rush-cli-client/src/test/resultDiagnostics.test.ts +++ b/apps/rush-cli-client/src/test/resultDiagnostics.test.ts @@ -63,4 +63,27 @@ describe(getResultStderr.name, () => { ).toBe('rush-client: daemon shut down\n'); expect(getResultStderr({ exitCode: 0 }, undefined)).toBeUndefined(); }); + + it('begins with rushx-client for a rushx script (task 166)', () => { + const reason: string = + 'The rushx script was not admitted before the daemon could restart for another request, because ' + + 'common/config/rush/pnpm-lock.yaml changed. Use --wait-timeout to wait longer.'; + expect( + getResultStderr( + { exitCode: 1, admissionErrorCode: 'wait-timeout', errorMessage: reason }, + { waitTimeoutMs: 5000 }, + 'rushx-client' + ) + ).toBe(`rushx-client: daemon admission failed (wait-timeout): ${reason}\n`); + expect( + getResultStderr({ exitCode: 1, admissionErrorCode: 'no-wait' }, { noWait: true }, 'rushx-client') + ).toMatch(/^rushx-client: daemon admission failed \(no-wait\): another daemon request/); + expect( + getResultStderr( + { exitCode: 1, admissionErrorCode: 'aborted', errorMessage: 'daemon shut down' }, + undefined, + 'rushx-client' + ) + ).toBe('rushx-client: daemon shut down\n'); + }); }); diff --git a/common/changes/@microsoft/rush/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json b/common/changes/@microsoft/rush/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json new file mode 100644 index 0000000000..3034a4de88 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "WorkspaceRuntimeFingerprintCache.changedInstallationPaths names the installation files, such as lockfiles, whose content or existence differs from the first capture, as changedPaths does for the files of Rush and its plugins.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-cli-client/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json b/common/changes/@rushstack/rush-cli-client/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json new file mode 100644 index 0000000000..27facc6dbb --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "A command that waits for a daemon restart now says what it waits for and why the daemon restarts, for every reason and on a pipe too, and repeats the line with the time waited every 25 seconds. A rushx script that waits for another request's restart says so, and the lines and admission failures of rushx-client begin with \"rushx-client:\". Once the command asks rushd to cancel the request, it no longer writes that the request waits.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-client-core/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json b/common/changes/@rushstack/rush-client-core/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json new file mode 100644 index 0000000000..9b4efb8d2a --- /dev/null +++ b/common/changes/@rushstack/rush-client-core/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-client-core", + "comment": "formatDaemonRestartCause says why the daemon restarts for each restart reason kind, for the waiting request or for a rushx script that waits for another request's restart. onQueuePositionAsync also gets the restart wait details (scriptCount, restartsForAnotherRequest), and onInputAdmittedAsync reports when the daemon first admits a request's input, which for a rushx script is when the script starts.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-client-core", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon-protocol/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json b/common/changes/@rushstack/rush-daemon-protocol/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json new file mode 100644 index 0000000000..028bdb3874 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon-protocol/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon-protocol", + "comment": "A queue position that waits for a daemon restart can say how many of the requests that it counts run a rushx script (scriptCount), and whether a rushx script waits for another request's restart (restartsForAnotherRequest). The new workspaceInputsChanged restart reason names the installation files or the files of Rush and its plugins that changed since the daemon started, or the Rush version that the request selects.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon-protocol", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json b/common/changes/@rushstack/rush-daemon/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json new file mode 100644 index 0000000000..3a54cef4d1 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-t166-restart-wait-reasons_2026-09-29-02-05.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "A request that waits for a daemon restart, and a rushx script that waits for another request's restart, now say why in their queue positions and admission errors: the environment variables that differ, the installation files or the files of Rush and its plugins that changed, or the Rush version that the request selects. The queue positions also say how many of the requests that they count run a rushx script.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-client-core.api.md b/common/reviews/api/rush-client-core.api.md index 0b57acf773..73420c612c 100644 --- a/common/reviews/api/rush-client-core.api.md +++ b/common/reviews/api/rush-client-core.api.md @@ -64,6 +64,9 @@ export type DaemonClientOutcome = { readonly rejection: IDaemonRequestRejectedMessage['payload']; }; +// @beta +export type DaemonRestartRequester = 'thisRequest' | 'anotherRequest'; + // @beta export type DaemonStartupHelperState = 'running' | 'exited' | 'unknown'; @@ -75,6 +78,9 @@ export class DaemonStartupPendingError extends Error { // @beta export function executeWithDaemonRestartAsync(client: DaemonClient, connection: IConnectOrStartDaemonOptions, options: IExecuteWithDaemonRestartOptions): Promise; +// @beta +export function formatDaemonRestartCause(reason: DaemonRestartReason, requester: DaemonRestartRequester): string | undefined; + // @beta export function getDaemonLogFilePath(paths: IDaemonPaths): string; @@ -134,7 +140,8 @@ export interface IDaemonClientExecuteOptions { readonly onCancelRequested?: (timeoutMs: number) => void; // (undocumented) readonly onEventAsync?: (event: IDaemonEventEnvelope) => Promise; - readonly onQueuePositionAsync?: (position: number, restartReason?: DaemonRestartReason) => Promise; + readonly onInputAdmittedAsync?: () => Promise; + readonly onQueuePositionAsync?: (position: number, restartReason?: DaemonRestartReason, restartWait?: IDaemonRestartWaitDetails) => Promise; // (undocumented) readonly onStderrAsync?: (bytes: Uint8Array, operationId: string) => Promise; // (undocumented) @@ -154,6 +161,12 @@ export interface IDaemonRestartNotice { readonly successorPid: number | undefined; } +// @beta +export interface IDaemonRestartWaitDetails { + readonly restartsForAnotherRequest?: boolean; + readonly scriptCount?: number; +} + // @beta export interface IDaemonStartCommand { // (undocumented) diff --git a/common/reviews/api/rush-daemon-protocol.api.md b/common/reviews/api/rush-daemon-protocol.api.md index ecca06700b..8869716e27 100644 --- a/common/reviews/api/rush-daemon-protocol.api.md +++ b/common/reviews/api/rush-daemon-protocol.api.md @@ -170,7 +170,7 @@ export type DaemonRequestAdmissionErrorCode = 'aborted' | 'no-wait' | 'wait-time export type DaemonRequestRejectionCode = 'invalidRequest' | 'routingFailed' | 'unsupported' | 'workspaceRecreationRequired'; // @beta -export type DaemonRestartReason = IDaemonInstallationChangedRestartReason | IDaemonEnvironmentChangedRestartReason; +export type DaemonRestartReason = IDaemonInstallationChangedRestartReason | IDaemonEnvironmentChangedRestartReason | IDaemonWorkspaceInputsChangedRestartReason; // @beta export type DaemonRushCommandOrigin = 'built-in' | 'custom'; @@ -572,6 +572,8 @@ export interface IDaemonRequestQueuePositionMessage { readonly position: number; readonly requestId: string; readonly restartReason?: DaemonRestartReason; + readonly scriptCount?: number; + readonly restartsForAnotherRequest?: boolean; }; } @@ -761,6 +763,14 @@ export interface IDaemonWarmSetStatus { readonly watchedProjectNames: ReadonlyArray; } +// @beta +export interface IDaemonWorkspaceInputsChangedRestartReason { + readonly implementationFiles?: ReadonlyArray; + readonly installationFiles?: ReadonlyArray; + readonly kind: 'workspaceInputsChanged'; + readonly selectedRushVersion?: string; +} + // @beta export interface IDaemonWorkspaceStatus { // (undocumented) diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 3555800b90..675d2b98ba 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -2202,9 +2202,12 @@ export const workspaceRequestScopedEnvironmentVariables: ReadonlySet; // @alpha export class WorkspaceRuntimeFingerprintCache { + get changedInstallationPaths(): ReadonlyArray; get changedPaths(): ReadonlyArray; // @internal _hashInputFilesAsync(filenames: Iterable): Promise; + // @internal + _hashInstallationFilesAsync(filenames: Iterable): Promise; // @internal (undocumented) _hashPaths(paths: ReadonlyArray): string; } diff --git a/libraries/rush-client-core/README.md b/libraries/rush-client-core/README.md index 57ae16f689..ae677fd118 100644 --- a/libraries/rush-client-core/README.md +++ b/libraries/rush-client-core/README.md @@ -50,10 +50,15 @@ A result may say why the daemon restarts (`restartReason`); for `installationCha the daemon's installation was removed or replaced, so it exits without a successor and the client's own `startCommand` starts one. Such a daemon answers only once the requests ahead of the request finish; meanwhile `onQueuePositionAsync` gets the reason as its second -argument. After each hand-off to a ready successor, -the optional `onRestartAsync` callback gets the restart number, the reason (`undefined` -when the daemon gave none, as older daemons do) and the successor's PID, before the -request is resubmitted. +argument. It does so for each request that waits for a restart, with the wait's details as its +third argument: `scriptCount`, how many of the requests ahead run a rushx script, and +`restartsForAnotherRequest`, set for a rushx script that waits for another request's restart. +`formatDaemonRestartCause` words a reason as the end of "the daemon restarts ...", for the +request or for such a script, and `onInputAdmittedAsync` reports when the daemon first admits +the request's input, which for a rushx script is when the script starts. After each hand-off +to a ready successor, the optional `onRestartAsync` callback gets the restart number, the +reason (`undefined` when the daemon gave none, as older daemons do) and the successor's PID, +before the request is resubmitted. A connection lost before the result stays a `disconnected` `DaemonClientError`. Its message starts with "Daemon disconnected before delivering a result; the command was not retried." diff --git a/libraries/rush-client-core/src/DaemonClient.ts b/libraries/rush-client-core/src/DaemonClient.ts index ec2c66af68..b79995055f 100644 --- a/libraries/rush-client-core/src/DaemonClient.ts +++ b/libraries/rush-client-core/src/DaemonClient.ts @@ -45,6 +45,19 @@ export interface IDaemonClientConnectOptions { readonly timeoutMs?: number; } +/** + * What a request that waits for a daemon restart waits for, as its queue position reports it. Older daemons omit + * these fields. + * + * @beta + */ +export interface IDaemonRestartWaitDetails { + /** How many of the requests that the queue position counts run a rushx script. */ + readonly scriptCount?: number; + /** The request, a rushx script, waits for another request's restart rather than its own. */ + readonly restartsForAnotherRequest?: boolean; +} + /** One request's backpressured destinations. Callback order is wire order. @beta */ export interface IDaemonClientExecuteOptions { readonly request: IDaemonRequestEnvelope; @@ -52,10 +65,19 @@ export interface IDaemonClientExecuteOptions { readonly onStderrAsync?: (bytes: Uint8Array, operationId: string) => Promise; readonly onEventAsync?: (event: IDaemonEventEnvelope) => Promise; /** - * Called with the request's one-based queue position whenever it changes. `restartReason` is set when the daemon - * answers the request with a restart result for that reason once the requests ahead of it finish. + * Called with the request's one-based queue position whenever it changes. `restartReason` is set while the request + * waits for a daemon restart for that reason, and `restartWait` then says more about the wait. */ - readonly onQueuePositionAsync?: (position: number, restartReason?: DaemonRestartReason) => Promise; + readonly onQueuePositionAsync?: ( + position: number, + restartReason?: DaemonRestartReason, + restartWait?: IDaemonRestartWaitDetails + ) => Promise; + /** + * Called once, when the daemon first admits the request's input. For a rushx script, that is when the script + * starts. Only daemons that negotiate the input lifecycle admit input. + */ + readonly onInputAdmittedAsync?: () => Promise; readonly abortSignal?: AbortSignal; /** Protocol 0.7 input waits for stdinReady credits; older peers use the legacy raw-mode/terminal policy. */ readonly stdin?: Readable; @@ -447,6 +469,7 @@ export class DaemonClient { if (!this.#inputAdmitted) { this.#inputAdmitted = true; this.#startInput(); + await execution.onInputAdmittedAsync?.(); } else if (this.#inputAcknowledgement) { const acknowledgement: IDeferred = this.#inputAcknowledgement; this.#inputAcknowledgement = undefined; @@ -455,9 +478,14 @@ export class DaemonClient { throw new DaemonProtocolError('malformedControlMessage', 'Unexpected stdin write acknowledgement.'); } return; - case 'queuePosition': - await execution.onQueuePositionAsync?.(message.payload.position, message.payload.restartReason); + case 'queuePosition': { + const { position, restartReason, scriptCount, restartsForAnotherRequest } = message.payload; + await execution.onQueuePositionAsync?.(position, restartReason, { + scriptCount, + restartsForAnotherRequest + }); return; + } default: throw new DaemonProtocolError( 'malformedControlMessage', diff --git a/libraries/rush-client-core/src/DaemonRestartCause.ts b/libraries/rush-client-core/src/DaemonRestartCause.ts new file mode 100644 index 0000000000..6b248c8500 --- /dev/null +++ b/libraries/rush-client-core/src/DaemonRestartCause.ts @@ -0,0 +1,79 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { + DaemonRestartReason, + IDaemonWorkspaceInputsChangedRestartReason +} from '@rushstack/rush-daemon-protocol'; + +/** A list names at most this many items, then says how many more there are. */ +const MAX_LISTED_NAMES: number = 4; + +/** + * Which request needs the restart that a cause explains: the request that waits for it, or another request that a + * rushx script waits for, so that the restart does not wait for the script. + * + * @beta + */ +export type DaemonRestartRequester = 'thisRequest' | 'anotherRequest'; + +/** + * Says why the daemon restarts, completing "the daemon restarts ", for example + * `because common/config/rush/pnpm-lock.yaml changed`. Names environment variables, never their values. + * + * @returns `undefined` for a reason kind that this version does not know. + * @beta + */ +export function formatDaemonRestartCause( + reason: DaemonRestartReason, + requester: DaemonRestartRequester +): string | undefined { + const anotherRequest: boolean = requester === 'anotherRequest'; + switch (reason.kind) { + case 'installationChanged': + return ( + `because ${anotherRequest ? "the daemon's" : 'its'} installation at ${reason.folder} was ` + + `${reason.change}` + ); + case 'environmentChanged': { + const names: string = reason.variableNames.length ? ` in ${formatList(reason.variableNames)}` : ''; + return `because ${anotherRequest ? 'its' : "this request's"} environment differs from the daemon's${names}`; + } + case 'workspaceInputsChanged': + return `because ${formatList(getWorkspaceInputClauses(reason, anotherRequest), Infinity)}`; + default: + return undefined; + } +} + +function getWorkspaceInputClauses( + reason: IDaemonWorkspaceInputsChangedRestartReason, + anotherRequest: boolean +): string[] { + const { installationFiles, implementationFiles, selectedRushVersion } = reason; + const clauses: string[] = []; + if (installationFiles) { + clauses.push( + installationFiles.length + ? `${formatList(installationFiles)} changed` + : "the workspace's installation changed" + ); + } + if (implementationFiles) { + const files: string = implementationFiles.length ? ` (${formatList(implementationFiles)})` : ''; + clauses.push(`the code of Rush or a Rush plugin changed${files}`); + } + if (selectedRushVersion !== undefined) { + clauses.push(`${anotherRequest ? 'it' : 'this request'} selects Rush ${selectedRushVersion}`); + } + if (!clauses.length) clauses.push('the inputs that the daemon started with changed'); + return clauses; +} + +/** Joins items as "a", "a and b" or "a, b and c", naming at most `maxListed` of them. */ +function formatList(items: ReadonlyArray, maxListed: number = MAX_LISTED_NAMES): string { + const listed: string[] = items.slice(0, maxListed); + const more: number = items.length - listed.length; + if (more > 0) listed.push(`${more} more`); + return listed.length > 1 ? `${listed.slice(0, -1).join(', ')} and ${listed[listed.length - 1]}` : listed[0]; +} diff --git a/libraries/rush-client-core/src/index.ts b/libraries/rush-client-core/src/index.ts index b6234ca642..fc7fff117e 100644 --- a/libraries/rush-client-core/src/index.ts +++ b/libraries/rush-client-core/src/index.ts @@ -11,8 +11,10 @@ export { DaemonClient, type DaemonClientOutcome, type IDaemonClientConnectOptions, - type IDaemonClientExecuteOptions + type IDaemonClientExecuteOptions, + type IDaemonRestartWaitDetails } from './DaemonClient'; +export { formatDaemonRestartCause, type DaemonRestartRequester } from './DaemonRestartCause'; export { DaemonClientError, type DaemonClientErrorCode } from './DaemonClientError'; export { getDaemonLogFilePath } from './DaemonLogFile'; export { diff --git a/libraries/rush-client-core/src/test/DaemonClient.test.ts b/libraries/rush-client-core/src/test/DaemonClient.test.ts index a9b8ea38ec..fa4c521fa4 100644 --- a/libraries/rush-client-core/src/test/DaemonClient.test.ts +++ b/libraries/rush-client-core/src/test/DaemonClient.test.ts @@ -369,10 +369,22 @@ describe('DaemonClient', () => { change: 'removed', folder: '/snapshots/s9' } as const; + const inputsReason = { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/pnpm-lock.yaml'] + } as const; onRequest = async (message) => { if (message.kind !== 'requestStart') return; const { requestId } = message.payload; await sendAsync({ kind: 'queuePosition', payload: { position: 2, requestId } }); + await sendAsync({ + kind: 'queuePosition', + payload: { position: 2, requestId, restartReason: inputsReason, scriptCount: 1 } + }); + await sendAsync({ + kind: 'queuePosition', + payload: { position: 1, requestId, restartReason: inputsReason, restartsForAnotherRequest: true } + }); await sendAsync({ kind: 'queuePosition', payload: { position: 1, requestId, restartReason } }); await sendAsync({ kind: 'requestResult', @@ -390,17 +402,54 @@ describe('DaemonClient', () => { const client = await DaemonClient.connectAsync({ socketPath: address }); const outcome = await client.executeAsync({ request: request(), - onQueuePositionAsync: async (position, reason) => { - positions.push([position, reason]); + onQueuePositionAsync: async (position, reason, wait) => { + positions.push([position, reason, wait]); } }); expect(positions).toEqual([ - [2, undefined], - [1, restartReason] + [2, undefined, {}], + [2, inputsReason, { scriptCount: 1 }], + [1, inputsReason, { restartsForAnotherRequest: true }], + [1, restartReason, {}] ]); expect(outcome).toMatchObject({ kind: 'result', result: { retryAfterRestart: true, restartReason } }); }); + it('reports once, before any input is forwarded, that the daemon admitted input', async () => { + const stdin = new PassThrough(); + const seen: string[] = []; + const envelope = { ...request(), terminal: { ...request().terminal, acceptsStdin: true } }; + onRequest = async (message) => { + if (message.kind !== 'requestStart') return; + await sendAsync({ kind: 'queuePosition', payload: { position: 1, requestId: envelope.requestId } }); + await sendAsync({ kind: 'stdinReady', payload: { requestId: envelope.requestId } }); + }; + onStdin = async (bytes) => { + seen.push(`stdin ${Buffer.from(bytes)}`); + if (seen.filter((entry) => entry.startsWith('stdin')).length < 2) return; + await sendAsync({ + kind: 'requestResult', + payload: { requestId: envelope.requestId, exitCode: 0, aborted: false, outcome: 'success' } + }); + }; + const client = await DaemonClient.connectAsync({ socketPath: address }); + const execution = client.executeAsync({ + request: envelope, + stdin, + onQueuePositionAsync: async () => { + seen.push('queued'); + }, + onInputAdmittedAsync: async () => { + seen.push('admitted'); + // Written only after admission, so the second chunk needs the first chunk's acknowledgement. + stdin.write('a'); + setTimeout(() => stdin.write('b'), 10); + } + }); + await execution; + expect(seen).toEqual(['queued', 'admitted', 'stdin a', 'stdin b']); + }); + it('forwards raw stdin only after acknowledgement and restores raw mode', async () => { const stdin = new PassThrough(); const rawModes: boolean[] = []; diff --git a/libraries/rush-client-core/src/test/DaemonRestartCause.test.ts b/libraries/rush-client-core/src/test/DaemonRestartCause.test.ts new file mode 100644 index 0000000000..e2a1e46be3 --- /dev/null +++ b/libraries/rush-client-core/src/test/DaemonRestartCause.test.ts @@ -0,0 +1,89 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; + +import { formatDaemonRestartCause } from '../DaemonRestartCause'; + +describe(formatDaemonRestartCause.name, () => { + it.each<[DaemonRestartReason, string, string]>([ + [ + { kind: 'installationChanged', change: 'removed', folder: '/snapshots/s9' }, + 'because its installation at /snapshots/s9 was removed', + "because the daemon's installation at /snapshots/s9 was removed" + ], + [ + { kind: 'environmentChanged', variableNames: ['NODE_OPTIONS'] }, + "because this request's environment differs from the daemon's in NODE_OPTIONS", + "because its environment differs from the daemon's in NODE_OPTIONS" + ], + [ + { kind: 'environmentChanged', variableNames: [] }, + "because this request's environment differs from the daemon's", + "because its environment differs from the daemon's" + ], + [ + { kind: 'workspaceInputsChanged', installationFiles: ['common/config/rush/pnpm-lock.yaml'] }, + 'because common/config/rush/pnpm-lock.yaml changed', + 'because common/config/rush/pnpm-lock.yaml changed' + ], + [ + { kind: 'workspaceInputsChanged', installationFiles: [] }, + "because the workspace's installation changed", + "because the workspace's installation changed" + ], + [ + { kind: 'workspaceInputsChanged', implementationFiles: ['common/autoinstallers/p/lib/index.js'] }, + 'because the code of Rush or a Rush plugin changed (common/autoinstallers/p/lib/index.js)', + 'because the code of Rush or a Rush plugin changed (common/autoinstallers/p/lib/index.js)' + ], + [ + { kind: 'workspaceInputsChanged', implementationFiles: [] }, + 'because the code of Rush or a Rush plugin changed', + 'because the code of Rush or a Rush plugin changed' + ], + [ + { kind: 'workspaceInputsChanged', selectedRushVersion: '5.180.0' }, + 'because this request selects Rush 5.180.0', + 'because it selects Rush 5.180.0' + ], + [ + { kind: 'workspaceInputsChanged' }, + 'because the inputs that the daemon started with changed', + 'because the inputs that the daemon started with changed' + ] + ])('explains %j', (reason, forThisRequest, forAnotherRequest) => { + expect(formatDaemonRestartCause(reason, 'thisRequest')).toBe(forThisRequest); + expect(formatDaemonRestartCause(reason, 'anotherRequest')).toBe(forAnotherRequest); + }); + + it('joins every changed input, and lists at most four names of each', () => { + expect( + formatDaemonRestartCause( + { + kind: 'workspaceInputsChanged', + installationFiles: ['a', 'b'], + implementationFiles: ['c'], + selectedRushVersion: '5.180.0' + }, + 'thisRequest' + ) + ).toBe( + 'because a and b changed, the code of Rush or a Rush plugin changed (c) and this request selects Rush 5.180.0' + ); + expect( + formatDaemonRestartCause( + { kind: 'environmentChanged', variableNames: ['A', 'B', 'C', 'D', 'E', 'F'] }, + 'thisRequest' + ) + ).toBe("because this request's environment differs from the daemon's in A, B, C, D and 2 more"); + expect( + formatDaemonRestartCause({ kind: 'environmentChanged', variableNames: ['A', 'B', 'C'] }, 'thisRequest') + ).toBe("because this request's environment differs from the daemon's in A, B and C"); + }); + + it('does not explain a reason kind that it does not know', () => { + const reason: unknown = { kind: 'futureInputsChanged' }; + expect(formatDaemonRestartCause(reason as DaemonRestartReason, 'thisRequest')).toBeUndefined(); + }); +}); diff --git a/libraries/rush-daemon-protocol/README.md b/libraries/rush-daemon-protocol/README.md index d4d5f8e435..66f554ecbe 100644 --- a/libraries/rush-daemon-protocol/README.md +++ b/libraries/rush-daemon-protocol/README.md @@ -26,8 +26,12 @@ The engine-agnostic **wire layer** spoken by every client of the Rush daemon (`r acknowledged raw-mode controls and typed terminal-policy results remain scoped to one request. - **Request admission contracts** — resolved no-wait and bounded-timeout options, typed admission failure codes, and capability-gated one-based queue-position control messages. A queue position - may carry the `restartReason` with which the daemon answers the request once the requests ahead - of it finish; older daemons omit it, and clients ignore reason kinds they do not know. + may carry the `restartReason` for which the daemon restarts once the requests that the position + counts finish, with `scriptCount`, how many of those requests run a rushx script, and + `restartsForAnotherRequest`, set for a rushx script that waits for another request's restart. + `workspaceInputsChanged` names the installation files or the files of Rush and its plugins that + changed since the daemon started, or the Rush version that the request selects. Older daemons + omit these fields, and clients ignore reason kinds they do not know. - **Request lifecycle contracts** — a validated presentation-free command envelope, cancellation, typed routing rejection/fallback, and one authoritative terminal result control. Command parsing and Rush action construction remain outside the protocol. diff --git a/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts b/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts index b97e4f2787..1e2596dba7 100644 --- a/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts +++ b/libraries/rush-daemon-protocol/src/DaemonInstallationChange.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import type { IDaemonEnvironmentChangedRestartReason } from './DaemonEnvironmentChange'; +import type { IDaemonWorkspaceInputsChangedRestartReason } from './DaemonWorkspaceInputsChange'; /** * How a folder of a running daemon's installation changed after the daemon started. @@ -35,10 +36,12 @@ export interface IDaemonInstallationChangedRestartReason extends IDaemonInstalla } /** - * Why a daemon asked the client to retry after a restart. Clients ignore kinds that they do not know. + * Why a daemon asked the client to retry after a restart, or why a queued request waits for a restart. Clients ignore + * kinds that they do not know. * * @beta */ export type DaemonRestartReason = | IDaemonInstallationChangedRestartReason - | IDaemonEnvironmentChangedRestartReason; + | IDaemonEnvironmentChangedRestartReason + | IDaemonWorkspaceInputsChangedRestartReason; diff --git a/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts b/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts index 5ec2f64646..fda7815f30 100644 --- a/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts +++ b/libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts @@ -32,10 +32,15 @@ export interface IDaemonRequestQueuePositionMessage { readonly position: number; readonly requestId: string; /** - * Set while the daemon holds the request until the requests ahead of it finish, and then answers it with a - * restart result for this reason instead of running it. Older daemons omit it; clients ignore unknown kinds. + * Set while the request waits for the requests that `position` counts to finish, since the daemon then + * restarts for this reason, and the request runs after the restart. Older daemons omit it; clients ignore + * unknown kinds. */ readonly restartReason?: DaemonRestartReason; + /** Set with `restartReason` if any of the requests that `position` counts run a rushx script: how many. */ + readonly scriptCount?: number; + /** Set with `restartReason` for a rushx script that waits for another request's restart, not its own. */ + readonly restartsForAnotherRequest?: boolean; }; } diff --git a/libraries/rush-daemon-protocol/src/DaemonWorkspaceInputsChange.ts b/libraries/rush-daemon-protocol/src/DaemonWorkspaceInputsChange.ts new file mode 100644 index 0000000000..201f08f3d4 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/DaemonWorkspaceInputsChange.ts @@ -0,0 +1,27 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +/** + * Workspace inputs that bind a daemon process differ from the ones that it started with, so the daemon restarts to + * serve a request that needs them. Each field is set only when that kind of input differs, and a list is empty when + * the daemon could not tell which files changed. A request whose environment differs is described by + * `environmentChanged` instead, whatever else differs. + * + * @beta + */ +export interface IDaemonWorkspaceInputsChangedRestartReason { + /** Identifies this reason. */ + readonly kind: 'workspaceInputsChanged'; + /** + * The installation files that changed since the daemon started, such as the lockfile, relative to the workspace + * root when they are inside it. + */ + readonly installationFiles?: ReadonlyArray; + /** + * Some of the files of Rush or of a Rush plugin whose code changed since the daemon started, relative to the + * workspace root when they are inside it. + */ + readonly implementationFiles?: ReadonlyArray; + /** The Rush version that the request selects, which the daemon does not run. */ + readonly selectedRushVersion?: string; +} diff --git a/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts b/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts index afaf7d90af..436bdcd26b 100644 --- a/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts +++ b/libraries/rush-daemon-protocol/src/InstallationChangeValidation.ts @@ -4,11 +4,22 @@ import { isDaemonControlRecord } from './ControlRecord'; import { DaemonProtocolError } from './DaemonProtocolError'; import { validateEnvironmentChange } from './EnvironmentChangeValidation'; +import { validateWorkspaceInputsChange } from './WorkspaceInputsChangeValidation'; const INSTALLATION_CHANGE_KINDS: ReadonlySet = new Set(['removed', 'replaced']); const INSTALLATION_CHANGED: string = 'installationChanged'; const ENVIRONMENT_CHANGED: string = 'environmentChanged'; +const WORKSPACE_INPUTS_CHANGED: string = 'workspaceInputsChanged'; const EMPTY_LENGTH: number = 0; +/** Validates the fields of each known restart reason kind; unknown kinds are accepted. */ +const REASON_VALIDATORS: ReadonlyMap) => void> = new Map([ + [ + INSTALLATION_CHANGED, + (reason: Record) => validateInstallationChange(reason, 'restartReason') + ], + [ENVIRONMENT_CHANGED, validateEnvironmentChange], + [WORKSPACE_INPUTS_CHANGED, validateWorkspaceInputsChange] +]); /** Validates an optional installation change, as reported by pong or by a restart reason. @internal */ export function validateInstallationChange(value: unknown, field: string): void { @@ -33,8 +44,7 @@ export function validateQueuedRestartReason(payload: Record): v function validateReason(reason: unknown): void { requireRecord(reason, 'restartReason'); requireKind(reason.kind); - if (reason.kind === INSTALLATION_CHANGED) validateInstallationChange(reason, 'restartReason'); - if (reason.kind === ENVIRONMENT_CHANGED) validateEnvironmentChange(reason); + REASON_VALIDATORS.get(reason.kind)?.(reason); } function requireRecord(value: unknown, field: string): asserts value is Record { diff --git a/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts b/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts index ead34a05a6..357b57f0fd 100644 --- a/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts +++ b/libraries/rush-daemon-protocol/src/RequestAdmissionControlValidation.ts @@ -6,6 +6,7 @@ import { validateQueuedRestartReason } from './InstallationChangeValidation'; const EMPTY_STRING_LENGTH: number = 0; const FIRST_QUEUE_POSITION: number = 1; +const NO_SCRIPTS: number = 0; /** Validates optional request-admission capability negotiation. @internal */ export function validateRequestAdmissionCapability(payload: Record): void { @@ -22,6 +23,24 @@ export function validateRequestQueuePositionControl(payload: Record= NO_SCRIPTS; } function validateRequestId(value: unknown): void { diff --git a/libraries/rush-daemon-protocol/src/WorkspaceInputsChangeValidation.ts b/libraries/rush-daemon-protocol/src/WorkspaceInputsChangeValidation.ts new file mode 100644 index 0000000000..a8b2dcbc46 --- /dev/null +++ b/libraries/rush-daemon-protocol/src/WorkspaceInputsChangeValidation.ts @@ -0,0 +1,33 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { DaemonProtocolError } from './DaemonProtocolError'; + +const FILE_LIST_FIELDS: ReadonlyArray = ['installationFiles', 'implementationFiles']; +const EMPTY_LENGTH: number = 0; + +/** Validates the optional fields of a `workspaceInputsChanged` restart reason. @internal */ +export function validateWorkspaceInputsChange(reason: Record): void { + for (const field of FILE_LIST_FIELDS) validateOptionalNames(reason[field], field); + validateOptionalVersion(reason.selectedRushVersion); +} + +function validateOptionalVersion(value: unknown): void { + if (value !== undefined && !isNonEmptyString(value)) fail('selectedRushVersion'); +} + +function validateOptionalNames(value: unknown, field: string): void { + if (value !== undefined && !isNameList(value)) fail(field); +} + +function isNameList(value: unknown): boolean { + return Array.isArray(value) && value.every(isNonEmptyString); +} + +function isNonEmptyString(value: unknown): boolean { + return typeof value === 'string' && value.length > EMPTY_LENGTH; +} + +function fail(field: string): never { + throw new DaemonProtocolError('malformedControlMessage', `Invalid restartReason.${field}.`); +} diff --git a/libraries/rush-daemon-protocol/src/index.ts b/libraries/rush-daemon-protocol/src/index.ts index c90228393b..648d0cf50a 100644 --- a/libraries/rush-daemon-protocol/src/index.ts +++ b/libraries/rush-daemon-protocol/src/index.ts @@ -38,9 +38,8 @@ export { DAEMON_CONTROL_MESSAGE_KINDS, isDaemonControlMessageKind } from './Daem export type { DaemonControlMessageKind } from './DaemonControlKinds'; export type { DaemonControlMessage, DaemonEmptyPayload } from './DaemonControlMessage'; export type { IDaemonErrorMessage, IDaemonHelloAckMessage } from './DaemonControlMessage'; -export type { IDaemonHelloMessage } from './DaemonControlMessage'; +export type { IDaemonHelloMessage, IDaemonUnsubscribeMessage } from './DaemonControlMessage'; export type { IDaemonPingMessage, IDaemonSubscribeMessage } from './DaemonControlMessage'; -export type { IDaemonUnsubscribeMessage } from './DaemonControlMessage'; export type { IDaemonRawModeChangedMessage, IDaemonSetRawModeMessage } from './DaemonInteractiveControl'; export type { IDaemonStdinEndMessage, IDaemonStdinReadyMessage } from './DaemonInteractiveControl'; export type { IDaemonTerminalPolicyMessage } from './DaemonInteractiveControl'; @@ -60,6 +59,7 @@ export type { DaemonInstallationChangeKind, DaemonRestartReason } from './Daemon export type { IDaemonInstallationChange } from './DaemonInstallationChange'; export type { IDaemonInstallationChangedRestartReason } from './DaemonInstallationChange'; export type { IDaemonEnvironmentChangedRestartReason } from './DaemonEnvironmentChange'; +export type { IDaemonWorkspaceInputsChangedRestartReason } from './DaemonWorkspaceInputsChange'; export { MAX_DAEMON_REQUEST_WAIT_TIMEOUT_MS } from './DaemonRequestAdmission'; export { validateDaemonRequestAdmissionOptions } from './DaemonRequestAdmission'; export type { DaemonRequestRejectionCode, IDaemonRequestCancelMessage } from './DaemonRequestControl'; diff --git a/libraries/rush-daemon-protocol/src/test/WorkspaceInputsRestartReason.test.ts b/libraries/rush-daemon-protocol/src/test/WorkspaceInputsRestartReason.test.ts new file mode 100644 index 0000000000..d74fc3d8cd --- /dev/null +++ b/libraries/rush-daemon-protocol/src/test/WorkspaceInputsRestartReason.test.ts @@ -0,0 +1,82 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { decodeDaemonControlMessage, encodeDaemonControlMessage } from '../ControlFrameCodec'; +import type { DaemonControlMessage } from '../DaemonControlMessage'; +import { WIRE_TEXT_ENCODER } from '../DaemonWireText'; +import type { IDaemonWorkspaceInputsChangedRestartReason } from '../DaemonWorkspaceInputsChange'; + +const FAILURE_EXIT_CODE: number = 1; +const POSITION: number = 2; +const SCRIPT_COUNT: number = 1; +const NO_SCRIPTS: number = 0; +const FRACTIONAL_COUNT: number = 1.5; +const NEGATIVE_COUNT: number = -1; +const REQUEST_ID: string = 'waits-for-inputs'; +const REASON: IDaemonWorkspaceInputsChangedRestartReason = { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/pnpm-lock.yaml'], + implementationFiles: ['common/autoinstallers/plugins/node_modules/p/lib/index.js'], + selectedRushVersion: '5.180.0' +}; + +function roundTrip(message: DaemonControlMessage): unknown { + return decodeDaemonControlMessage(encodeDaemonControlMessage(message)); +} + +function queuePositionFrame(fields: object): Uint8Array { + const payload: object = { position: POSITION, requestId: REQUEST_ID, ...fields }; + return WIRE_TEXT_ENCODER.encode(JSON.stringify({ kind: 'queuePosition', payload })); +} + +it('round-trips a queue position that waits for a restart for changed workspace inputs', () => { + const message: DaemonControlMessage = { + kind: 'queuePosition', + payload: { position: POSITION, requestId: REQUEST_ID, restartReason: REASON, scriptCount: SCRIPT_COUNT } + }; + expect(roundTrip(message)).toEqual(message); +}); + +it('round-trips a rushx script that waits for another request restart', () => { + const restartReason: IDaemonWorkspaceInputsChangedRestartReason = { kind: 'workspaceInputsChanged' }; + const payload: object = { restartReason, restartsForAnotherRequest: true, scriptCount: NO_SCRIPTS }; + expect(decodeDaemonControlMessage(queuePositionFrame(payload))).toMatchObject({ payload }); +}); + +it('round-trips a restart result for changed workspace inputs', () => { + const message: DaemonControlMessage = { + kind: 'requestResult', + payload: { + aborted: false, + exitCode: FAILURE_EXIT_CODE, + outcome: 'failure', + requestId: REQUEST_ID, + restartReason: REASON, + retryAfterRestart: true + } + }; + expect(roundTrip(message)).toEqual(message); +}); + +it.each([ + { installationFiles: 'common/config/rush/pnpm-lock.yaml' }, + { implementationFiles: [''] }, + { installationFiles: [FAILURE_EXIT_CODE] }, + { implementationFiles: null }, + { selectedRushVersion: '' }, + { selectedRushVersion: [] } +])('rejects a malformed workspace inputs reason %j', (fields: object) => { + const restartReason: object = { kind: 'workspaceInputsChanged', ...fields }; + expect(() => decodeDaemonControlMessage(queuePositionFrame({ restartReason }))).toThrow(/restartReason/); +}); + +it.each([ + { scriptCount: NEGATIVE_COUNT }, + { scriptCount: FRACTIONAL_COUNT }, + { scriptCount: '1' }, + { restartsForAnotherRequest: 'yes' } +])('rejects a malformed restart wait %j', (fields: object) => { + expect(() => decodeDaemonControlMessage(queuePositionFrame({ restartReason: REASON, ...fields }))).toThrow( + /scriptCount|restartsForAnotherRequest/ + ); +}); diff --git a/libraries/rush-daemon/README.md b/libraries/rush-daemon/README.md index 48030e4ef6..5161262ddc 100644 --- a/libraries/rush-daemon/README.md +++ b/libraries/rush-daemon/README.md @@ -185,7 +185,11 @@ restart can still take the startup mutex first. A request whose environment needs another process does not restart the daemon while it serves other requests. It first waits for the requests that this process is serving to finish (the restart drain), and its queue position is the -number of those requests. Like the graph-execution gate, waiting for the requests that were already being served when +number of those requests. The queue positions also say why the daemon restarts (`restartReason`): `environmentChanged` +names the variables that differ (see below), and `workspaceInputsChanged` names the installation files and the files +of Rush or its plugins that changed since the daemon started, as the latest capture found them, or the Rush version +that the request selects. They say how many of those requests run a rushx script (`scriptCount`), and the drain's +admission errors name the same reason. Like the graph-execution gate, waiting for the requests that were already being served when the drain began is progress rather than contention: while one of them is still being served and no rushx script is, a client-default `waitTimeoutMs` (`waitTimeoutIsDefault`) does not limit the drain and is not spent, and the client sends the request to the successor with its default again. The default still limits the drain while a rushx script is @@ -207,7 +211,8 @@ that it no longer needs it, or has failed or been cancelled. A rushx script that does not start, since the restart would then wait for it to exit: the script waits for the pending restart instead, and the drain does not count it. If the restart was planned, the script's result carries `retryAfterRestart: true` so that the client runs it on the successor; otherwise it runs on this process. Its queue position is the number of -requests that are served or waiting to restart, and its wait timeout applies as it does to the drain, relative to the +requests that are served or waiting to restart, with the reason of the first request that waits to restart and +`restartsForAnotherRequest: true`, and its wait timeout applies as it does to the drain, relative to the requests that were served when the script began to wait. The `retryAfterRestart: true` result of the request that restarts the daemon for its environment carries diff --git a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts index 54b4964d4f..6f53ed75e2 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestAdmission.ts @@ -11,6 +11,7 @@ import type { IDaemonRequestAdmissionOptions, IDaemonRequestQueuePositionMessage } from '@rushstack/rush-daemon-protocol'; +import { formatDaemonRestartCause, type IDaemonRestartWaitDetails } from '@rushstack/rush-client-core'; import { type IRequestLease, @@ -26,6 +27,7 @@ import type { IWorkspaceRestartDrainOptions, IWorkspaceRestartRecheck, IWorkspaceRestartTicket, + IWorkspaceRestartWaitReport, IWorkspaceRestartWaitResult, WorkspaceRestartArbiter } from './WorkspaceRestartArbiter'; @@ -83,13 +85,6 @@ export function freezeDaemonRequestAdmissionOptions( return copy; } -/** Says why the daemon restarts, completing "the daemon could restart ". */ -function formatRestartCause(restartReason: DaemonRestartReason): string { - return restartReason.kind === 'environmentChanged' - ? `because a command's environment differs from its own in ${restartReason.variableNames.join(', ')}` - : `because its installation at ${restartReason.folder} was ${restartReason.change}`; -} - class WorkspaceRequestScheduler extends RequestScheduler { readonly #session: IWorkspaceSession; @@ -130,12 +125,25 @@ class QueuePositionWriter { writeQueuePositionAsync.call(client, message); } - public enqueue(position: number, restartReason?: DaemonRestartReason): void { + public enqueue( + position: number, + restartReason?: DaemonRestartReason, + restartWait?: IDaemonRestartWaitDetails + ): void { + const { scriptCount, restartsForAnotherRequest } = restartWait ?? {}; this.#tail = this.#tail .then(() => this.#writeQueuePositionAsync({ kind: 'queuePosition', - payload: { position, requestId: this.#requestId, ...(restartReason && { restartReason }) } + payload: { + position, + requestId: this.#requestId, + ...(restartReason && { + restartReason, + ...(scriptCount ? { scriptCount } : undefined), + ...(restartsForAnotherRequest && { restartsForAnotherRequest }) + }) + } }) ) .catch((error: unknown) => { @@ -340,7 +348,8 @@ export class RequestAdmissionController { scheduler, RequestExclusivityClass.Exclusive, this.#remainingMs, - `the running requests to finish before the daemon restarts ${formatRestartCause(restartReason)}`, + 'the running requests to finish before the daemon restarts ' + + (formatDaemonRestartCause(restartReason, 'thisRequest') ?? 'for its environment'), this.#abortController.signal, restartReason ); @@ -467,8 +476,9 @@ export class RequestAdmissionController { * requests that arrived later, which could otherwise keep the request waiting for as long as they keep arriving. * An explicit `noWait` or `waitTimeoutMs` applies to the whole wait, using the same budget as workspace admission. * - * A `restartReason` says that the daemon restarts for that reason rather than for the request's environment. Queue - * positions then carry it, and admission errors name it. + * A `restartReason` says why the request needs the restart. Queue positions carry it, as do those of rushx scripts + * that wait for the restart, and admission errors name it. Without it, they say that the daemon restarts for the + * request's environment. Queue positions also say how many of the requests that they count run a rushx script. * * @returns true once the drain finishes, or false if `recheck` found that the request no longer needs the restart. */ @@ -486,9 +496,9 @@ export class RequestAdmissionController { } /** - * Waits, for a rushx script, until no other request needs to restart the daemon for its environment, so that the - * restart does not also wait for the script. The client is told how many requests are served or need a restart, as - * a queue position. + * Waits, for a rushx script, until no other request needs to restart the daemon, so that the restart does not also + * wait for the script. The client is told how many requests are served or need a restart, as a queue position, + * and why the first request that needs a restart needs it. * * @remarks * The wait timeout applies as it does to the restart drain, relative to the requests served when this wait began: @@ -498,14 +508,17 @@ export class RequestAdmissionController { arbiter: WorkspaceRestartArbiter, ticket: IWorkspaceRestartTicket ): Promise { - await this.#waitForRestartArbiterAsync((options: IWorkspaceRestartDrainOptions) => - arbiter.waitForPendingRestartAsync(ticket, options) + await this.#waitForRestartArbiterAsync( + (options: IWorkspaceRestartDrainOptions) => arbiter.waitForPendingRestartAsync(ticket, options), + undefined, + true ); } async #waitForRestartArbiterAsync( waitAsync: (options: IWorkspaceRestartDrainOptions) => Promise, - restartReason?: DaemonRestartReason + restartReason?: DaemonRestartReason, + restartsForAnotherRequest?: boolean ): Promise { const writer: QueuePositionWriter | undefined = this.#writer; const startMs: number = Date.now(); @@ -517,8 +530,14 @@ export class RequestAdmissionController { noWait: this.#admission?.noWait, waitTimeoutMs: this.#remainingMs, waivesTimeoutForServedWork: this.#admission?.waitTimeoutIsDefault === true, - restartCause: restartReason && formatRestartCause(restartReason), - onServingCountChanged: writer ? (count: number) => writer.enqueue(count, restartReason) : undefined + restartReason, + onServingCountChanged: writer + ? (count: number, report: IWorkspaceRestartWaitReport) => + writer.enqueue(count, report.restartReason, { + scriptCount: report.scriptCount, + restartsForAnotherRequest + }) + : undefined }); waivedMs = result.waivedMs; return result; diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index b028c0ea08..e9e4fd9223 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -79,6 +79,8 @@ import { /** How often a request that waits for a restart drain checks whether it still needs the restart. */ const RESTART_RECHECK_INTERVAL_MS: number = 1000; +/** A restart reason names at most this many of the changed files of Rush and its plugins. */ +const MAX_REASON_IMPLEMENTATION_FILES: number = 3; interface IRecheckCapture { /** On the clock of `performance.now()`. */ @@ -535,7 +537,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { const drained: boolean = await admission.waitForRestartDrainAsync( this.#restartArbiter, ticket, - undefined, + this.#getRestartReason(current, controlEnvelope.environment, false), this.#createRestartRecheck(session, controlEnvelope, current, false) ); if (this.#restartPending) throw new RestartPendingBeforeExecution(); @@ -638,7 +640,7 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { const drained: boolean = await admission.waitForRestartDrainAsync( this.#restartArbiter, ticket, - undefined, + this.#getRestartReason(fingerprint, envelope.environment, isMutation(envelope)), this.#createRestartRecheck(session, envelope, fingerprint, isMutation(envelope)) ); if (this.#restartPending) throw new RestartPendingBeforeExecution(); @@ -978,6 +980,51 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { return fingerprint; } + /** + * Says why a request whose inputs need a restart (see `#classify`) needs it, for the request and for rushx scripts + * that wait for the restart. A request whose environment differs from the daemon's gets the names of the variables + * that differ, whatever else differs. Otherwise the reason names the files that changed, as the latest capture + * found them, and the Rush version that the request selects if the daemon does not run it. + */ + #getRestartReason( + fingerprint: IWorkspaceInputFingerprint, + environment: Readonly>, + mutation: boolean + ): DaemonRestartReason { + const environmentReason: DaemonRestartReason | undefined = getEnvironmentRestartReason( + this.#startupEnvironmentEntries, + environment + ); + if (environmentReason) return environmentReason; + const startup: IWorkspaceInputFingerprint = this.#startupFingerprint; + const { selectedRushVersion } = fingerprint; + return { + kind: 'workspaceInputsChanged', + ...(!mutation && + fingerprint.installationHash !== startup.installationHash && { + installationFiles: this.#toWorkspacePaths(this.#runtimeCache.changedInstallationPaths) + }), + ...(fingerprint.runtimeHash !== startup.runtimeHash && { + implementationFiles: this.#toWorkspacePaths( + this.#runtimeCache.changedPaths.slice(0, MAX_REASON_IMPLEMENTATION_FILES) + ) + }), + ...((selectedRushVersion !== Rush.version || selectedRushVersion !== this.#options.rushVersion) && { + selectedRushVersion + }) + }; + } + + /** Makes the paths inside the workspace relative to its root, with forward slashes. */ + #toWorkspacePaths(filePaths: ReadonlyArray): string[] { + return filePaths.map((filePath: string) => { + const relativePath: string = path.relative(this.#repoRoot, filePath); + return relativePath && !relativePath.startsWith('..') && !path.isAbsolute(relativePath) + ? relativePath.split(path.sep).join('/') + : filePath; + }); + } + #classify(fingerprint: IWorkspaceInputFingerprint, mutation: boolean): WorkspaceInputChangeTier { if ( fingerprint.selectedRushVersion !== Rush.version || diff --git a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts index aa5a96975e..5979ec14dd 100644 --- a/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts +++ b/libraries/rush-daemon/src/WorkspaceRestartArbiter.ts @@ -1,6 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. +import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; +import { formatDaemonRestartCause, type DaemonRestartRequester } from '@rushstack/rush-client-core'; + import { RequestSchedulerError, RequestSchedulerErrorCode } from './RequestScheduler'; const MAX_TIMER_DELAY_MS: number = 0x7fffffff; @@ -10,6 +13,14 @@ const SCRIPT_TIMEOUT_CLAUSE: string = ', including a rushx script that may not e const SCRIPT_TIMEOUT_REMEDY: string = 'Stop the script, or use --wait-timeout to wait longer.'; const TIMEOUT_REMEDY: string = 'Use --wait-timeout to wait longer.'; +/** Says why the daemon restarts, as the end of "the daemon restarts ", if the reason is known. */ +function formatCause( + restartReason: DaemonRestartReason | undefined, + requester: DaemonRestartRequester +): string | undefined { + return restartReason && formatDaemonRestartCause(restartReason, requester); +} + /** Names the waived time, since a drain that waived time times out that much later than the wait timeout. */ function formatWaivedTime(waivedMs: number): string { const seconds: number = Math.round(waivedMs / 100) / 10; @@ -34,6 +45,8 @@ interface IMutableTicket { waitingForDrain: boolean; left: boolean; readonly runsScript: boolean; + /** Why the request needs a restart, while it is a restart candidate. */ + restartReason: DaemonRestartReason | undefined; } interface IWaitKind { @@ -44,13 +57,26 @@ interface IWaitKind { readonly restarts: boolean; /** Whether the request must keep waiting. */ readonly isBlocked: () => boolean; - /** How many other requests the request waits for, reported as its queue position. */ - readonly countWaitedFor: () => number; + /** The other requests that the request waits for. Their number is reported as its queue position. */ + readonly getWaitedFor: () => ReadonlySet; + /** Why the daemon restarts, if that is known. */ + readonly getRestartReason: () => DaemonRestartReason | undefined; /** Lets a request that waits to restart find that it no longer needs to. */ readonly recheck: IWorkspaceRestartRecheck | undefined; - readonly noWaitMessage: string; + readonly getNoWaitMessage: () => string; /** Begins the timeout message, which goes on to name the requests that the restart waits for. */ - readonly timeoutPrefix: string; + readonly getTimeoutPrefix: () => string; +} + +/** What a request that waits for a restart waits for, besides how many requests. */ +export interface IWorkspaceRestartWaitReport { + /** How many of the requests that the request waits for run a rushx script. */ + readonly scriptCount: number; + /** + * Why the daemon restarts: for a restart drain, the request's own reason, and for a rushx script that waits for a + * pending restart, the reason of the first request that needs one. Undefined if that request gave no reason. + */ + readonly restartReason: DaemonRestartReason | undefined; } /** Options for {@link WorkspaceRestartArbiter.waitForDrainAsync}, supplied by request admission. */ @@ -66,16 +92,16 @@ export interface IWorkspaceRestartDrainOptions { */ readonly waivesTimeoutForServedWork?: boolean; /** - * Why the daemon restarts, as the admission errors of {@link WorkspaceRestartArbiter.waitForDrainAsync} say it: - * "the daemon could restart ", for example `because its installation at /x was removed`. The default is - * `for its environment`. + * Why the request needs the restart that {@link WorkspaceRestartArbiter.waitForDrainAsync} waits for. The admission + * errors of the drain name it, as do those of rushx scripts that wait for the restart, and their reports. Without + * it, they say that the daemon restarts for the request's environment. */ - readonly restartCause?: string; + readonly restartReason?: DaemonRestartReason; /** * Called while the request waits with the number of other requests that it waits for, when the wait begins and - * whenever that number changes, so that the client can report the wait as a queue position. + * whenever that number or the report changes, so that the client can report the wait as a queue position. */ - readonly onServingCountChanged?: (servingCount: number) => void; + readonly onServingCountChanged?: (servingCount: number, report: IWorkspaceRestartWaitReport) => void; } /** @@ -174,7 +200,8 @@ export class WorkspaceRestartArbiter { const ticket: IMutableTicket = { waitingForDrain: false, left: false, - runsScript: options?.runsScript === true + runsScript: options?.runsScript === true, + restartReason: undefined }; this.#serve(ticket); return ticket; @@ -216,20 +243,22 @@ export class WorkspaceRestartArbiter { options: IWorkspaceRestartDrainOptions, recheck?: IWorkspaceRestartRecheck ): Promise { - const { restartCause } = options; + const cause: string | undefined = formatCause(options.restartReason, 'thisRequest'); return await this.#waitAsync(ticket, options, { restarts: true, isBlocked: () => this.#serving.size > 0, - countWaitedFor: () => this.#serving.size, + getWaitedFor: () => this.#serving, + getRestartReason: () => options.restartReason, recheck, - noWaitMessage: - restartCause === undefined + getNoWaitMessage: () => + cause === undefined ? 'Another environment is still being served; the request did not wait for a restart.' - : `The daemon is still serving other requests, which finish before it restarts ${restartCause}; ` + + : `The daemon is still serving other requests, which finish before it restarts ${cause}; ` + 'the request did not wait for a restart.', - timeoutPrefix: - `The request was not admitted before the daemon could restart ${restartCause ?? ENVIRONMENT_RESTART_CAUSE}, ` + - 'which waits for' + getTimeoutPrefix: () => + cause === undefined + ? `The request was not admitted before the daemon could restart ${ENVIRONMENT_RESTART_CAUSE}, which waits for` + : `The request was not admitted before the daemon could restart ${cause}. The restart waits for` }); } @@ -244,18 +273,29 @@ export class WorkspaceRestartArbiter { ticket: IWorkspaceRestartTicket, options: IWorkspaceRestartDrainOptions ): Promise { + const getRestartReason = (): DaemonRestartReason | undefined => + Array.from(this.#restartCandidates).find((candidate: IMutableTicket) => candidate !== ticket) + ?.restartReason; return await this.#waitAsync(ticket, options, { restarts: false, isBlocked: () => this.hasPendingRestart(ticket), - countWaitedFor: () => new Set([...this.#serving, ...this.#restartCandidates]).size, + getWaitedFor: () => new Set([...this.#serving, ...this.#restartCandidates]), + getRestartReason, recheck: undefined, - noWaitMessage: - 'Another request is waiting to restart the daemon for its environment; the rushx script did not wait ' + - 'for the restart.', - timeoutPrefix: - "The rushx script was not admitted before the daemon could restart for another request's environment. A " + - 'script waits for a pending restart so that the restart does not wait for the script, and the restart ' + - 'waits for' + getNoWaitMessage: () => { + const cause: string = formatCause(getRestartReason(), 'anotherRequest') ?? ENVIRONMENT_RESTART_CAUSE; + return `Another request is waiting to restart the daemon ${cause}; the rushx script did not wait for the restart.`; + }, + getTimeoutPrefix: () => { + const cause: string | undefined = formatCause(getRestartReason(), 'anotherRequest'); + return ( + (cause === undefined + ? "The rushx script was not admitted before the daemon could restart for another request's environment." + : `The rushx script was not admitted before the daemon could restart for another request, ${cause}.`) + + ' A script waits for a pending restart so that the restart does not wait for the script, and the ' + + 'restart waits for' + ); + } }); } @@ -266,18 +306,27 @@ export class WorkspaceRestartArbiter { ): Promise { const state: IMutableTicket = ticket as IMutableTicket; if (state.left || state.waitingForDrain) throw new Error('The restart ticket is not being served.'); - if (kind.restarts) this.#restartCandidates.add(state); + if (kind.restarts) { + state.restartReason = options.restartReason; + this.#restartCandidates.add(state); + } state.waitingForDrain = true; this.#stopServing(state); const waivedFor: IMutableTicket[] = options.waivesTimeoutForServedWork ? Array.from(this.#serving) : []; let remainingMs: number | undefined = options.waitTimeoutMs; let waivedMs: number = 0; - let reported: number | undefined; + let reported: string | undefined; const report = (): void => { - const count: number = kind.countWaitedFor(); - if (count > 0 && count !== reported) { - reported = count; - options.onServingCountChanged?.(count); + const waitedFor: ReadonlySet = kind.getWaitedFor(); + if (waitedFor.size === 0) return; + const waitReport: IWorkspaceRestartWaitReport = { + scriptCount: Array.from(waitedFor).filter((waited: IMutableTicket) => waited.runsScript).length, + restartReason: kind.getRestartReason() + }; + const key: string = JSON.stringify([waitedFor.size, waitReport.scriptCount, waitReport.restartReason]); + if (key !== reported) { + reported = key; + options.onServingCountChanged?.(waitedFor.size, waitReport); } }; // The ticket is a restart candidate for as long as the wait lasts, so withdrawing its restart wakes the wait. @@ -287,7 +336,7 @@ export class WorkspaceRestartArbiter { try { while (!recheck?.withdrawn && kind.isBlocked()) { if (options.noWait) { - throw new RequestSchedulerError(RequestSchedulerErrorCode.NoWait, kind.noWaitMessage); + throw new RequestSchedulerError(RequestSchedulerErrorCode.NoWait, kind.getNoWaitMessage()); } report(); const waived: boolean = @@ -339,8 +388,9 @@ export class WorkspaceRestartArbiter { const script: boolean = this.#isServingScript(); return new RequestSchedulerError( RequestSchedulerErrorCode.WaitTimeout, - `${kind.timeoutPrefix} the requests that the daemon is serving to finish${script ? SCRIPT_TIMEOUT_CLAUSE : ''}` + - `${formatWaivedTime(waivedMs)}. ${script ? SCRIPT_TIMEOUT_REMEDY : TIMEOUT_REMEDY}` + `${kind.getTimeoutPrefix()} the requests that the daemon is serving to finish` + + `${script ? SCRIPT_TIMEOUT_CLAUSE : ''}${formatWaivedTime(waivedMs)}. ` + + (script ? SCRIPT_TIMEOUT_REMEDY : TIMEOUT_REMEDY) ); } diff --git a/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts b/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts index c03a3f920f..ea3c9f482c 100644 --- a/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts +++ b/libraries/rush-daemon/src/test/DaemonInstallationChange.test.ts @@ -299,7 +299,7 @@ describe('a daemon whose installation changed', () => { const result: IDaemonCommandResult = timedOut.terminal.payload as IDaemonCommandResult; expect(result.retryAfterRestart).toBeUndefined(); expect(result.errorMessage).toContain( - `could restart because its installation at ${installation.folder} was replaced, which waits for the ` + + `could restart because its installation at ${installation.folder} was replaced. The restart waits for the ` + 'requests that the daemon is serving to finish, including a rushx script that may not exit until it is ' + 'stopped. Stop the script, or use --wait-timeout to wait longer.' ); diff --git a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts index 5780afed79..e122bba3b4 100644 --- a/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts +++ b/libraries/rush-daemon/src/test/RestartDrainAdmission.test.ts @@ -29,6 +29,12 @@ const INSTALLATION_REMOVED: DaemonRestartReason = { change: 'removed', folder: '/old/daemon' }; +const LOCKFILE_CHANGED: DaemonRestartReason = { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/pnpm-lock.yaml'] +}; + +type QueuePosition = IDaemonRequestQueuePositionMessage['payload']; interface IDrainTest { readonly admission: RequestAdmissionController; @@ -48,6 +54,25 @@ function createCandidate( }); } +/** A request whose client supports request admission and records its queue positions. */ +function createReportingAdmission( + options: IDaemonRequestAdmissionOptions, + requestId: string, + positions: QueuePosition[] +): RequestAdmissionController { + return new RequestAdmissionController({ + admission: options, + client: { + abortSignal: new AbortController().signal, + supportsRequestAdmission: true, + writeQueuePositionAsync: async (message: IDaemonRequestQueuePositionMessage) => { + positions.push(message.payload); + } + }, + requestId + }); +} + /** A restart candidate with the given admission options, and one other request that the daemon is serving. */ function createDrainTest( options: IDaemonRequestAdmissionOptions, @@ -194,19 +219,9 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { const serving: IWorkspaceRestartTicket = arbiter.enter(); const ticket: IWorkspaceRestartTicket = arbiter.enter(); const restartReason: DaemonRestartReason = INSTALLATION_REMOVED; - const positions: IDaemonRequestQueuePositionMessage['payload'][] = []; + const positions: QueuePosition[] = []; const createAdmission = (options: IDaemonRequestAdmissionOptions): RequestAdmissionController => - new RequestAdmissionController({ - admission: options, - client: { - abortSignal: new AbortController().signal, - supportsRequestAdmission: true, - writeQueuePositionAsync: async (message: IDaemonRequestQueuePositionMessage) => { - positions.push(message.payload); - } - }, - requestId: 'restart-candidate' - }); + createReportingAdmission(options, 'restart-candidate', positions); const notWaiting: RequestAdmissionController = createAdmission({ noWait: true }); const noWaitError: unknown = await notWaiting @@ -226,7 +241,7 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { expect((timeoutError as RequestSchedulerError).code).toBe(RequestSchedulerErrorCode.WaitTimeout); expect((timeoutError as Error).message).toBe( 'The request was not admitted before the daemon could restart because its installation at /old/daemon was ' + - 'removed, which waits for the requests that the daemon is serving to finish. Use --wait-timeout ' + + 'removed. The restart waits for the requests that the daemon is serving to finish. Use --wait-timeout ' + ' to wait longer.' ); expect(positions).toEqual([{ position: 1, requestId: 'restart-candidate', restartReason }]); @@ -235,6 +250,31 @@ describe('RequestAdmissionController.waitForRestartDrainAsync', () => { arbiter.leave(serving); expect(arbiter.servingCount).toBe(0); }); + + it('says in its queue positions how many of the requests that it waits for run a rushx script', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const build: IWorkspaceRestartTicket = arbiter.enter(); + const ticket: IWorkspaceRestartTicket = arbiter.enter(); + const positions: QueuePosition[] = []; + const admission: RequestAdmissionController = createReportingAdmission( + {}, + 'restart-candidate', + positions + ); + const draining: Promise = admission.waitForRestartDrainAsync(arbiter, ticket, LOCKFILE_CHANGED); + arbiter.leave(script); + arbiter.leave(build); + expect(await draining).toBe(true); + await delayAsync(0); + expect(positions).toEqual([ + { position: 2, requestId: 'restart-candidate', restartReason: LOCKFILE_CHANGED, scriptCount: 1 }, + { position: 1, requestId: 'restart-candidate', restartReason: LOCKFILE_CHANGED } + ]); + arbiter.leave(ticket); + admission.dispose(); + expect(arbiter.servingCount).toBe(0); + }); }); describe('RequestAdmissionController.acquireBeforeRestartAsync', () => { @@ -295,15 +335,19 @@ interface IPendingRestartTest { readonly arbiter: WorkspaceRestartArbiter; readonly candidate: IWorkspaceRestartTicket; readonly draining: Promise; - readonly positions: number[]; + readonly positions: QueuePosition[]; readonly script: IWorkspaceRestartTicket; readonly serving: IWorkspaceRestartTicket; } -/** A rushx script with the given admission options, which arrives while a restart candidate drains one request. */ +/** + * A rushx script with the given admission options, which arrives while a restart candidate, which gives the + * restart reason if any, drains one request. + */ function createPendingRestartTest( options: IDaemonRequestAdmissionOptions, - servingOptions?: IWorkspaceRestartTicketOptions + servingOptions?: IWorkspaceRestartTicketOptions, + restartReason?: DaemonRestartReason ): IPendingRestartTest { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const serving: IWorkspaceRestartTicket = arbiter.enter(servingOptions); @@ -311,22 +355,12 @@ function createPendingRestartTest( const draining: Promise = arbiter.waitForDrainAsync(candidate, { abortSignal: new AbortController().signal, noWait: undefined, - waitTimeoutMs: undefined + waitTimeoutMs: undefined, + restartReason }); const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); - const positions: number[] = []; - const admission: RequestAdmissionController = new RequestAdmissionController({ - admission: options, - client: { - abortSignal: new AbortController().signal, - supportsRequestAdmission: true, - writeQueuePositionAsync: (message: IDaemonRequestQueuePositionMessage) => { - positions.push(message.payload.position); - return Promise.resolve(); - } - }, - requestId: 'rushx-script' - }); + const positions: QueuePosition[] = []; + const admission: RequestAdmissionController = createReportingAdmission(options, 'rushx-script', positions); return { admission, arbiter, candidate, draining, positions, script, serving }; } @@ -352,13 +386,42 @@ describe('RequestAdmissionController.waitForPendingRestartAsync', () => { await test.draining; test.arbiter.leave(test.candidate); await waiting; - // The build and the candidate, then the candidate alone. - expect(test.positions).toEqual([2, 1]); + // The build and the candidate, then the candidate alone. The candidate gave no reason for its restart. + expect(test.positions).toEqual([ + { position: 2, requestId: 'rushx-script' }, + { position: 1, requestId: 'rushx-script' } + ]); // The script is then told to run on the successor, which gets its default again. expect(test.admission.remainingAdmission?.waitTimeoutMs).toBeGreaterThan(0); await finishPendingRestartTestAsync(test); }); + it('says in its queue positions why the daemon restarts for another request, and how many scripts it waits for', async () => { + const test: IPendingRestartTest = createPendingRestartTest({}, { runsScript: true }, LOCKFILE_CHANGED); + const waiting: Promise = test.admission.waitForPendingRestartAsync(test.arbiter, test.script); + test.arbiter.leave(test.serving); + await test.draining; + test.arbiter.leave(test.candidate); + await waiting; + await delayAsync(0); + expect(test.positions).toEqual([ + { + position: 2, + requestId: 'rushx-script', + restartReason: LOCKFILE_CHANGED, + scriptCount: 1, + restartsForAnotherRequest: true + }, + { + position: 1, + requestId: 'rushx-script', + restartReason: LOCKFILE_CHANGED, + restartsForAnotherRequest: true + } + ]); + await finishPendingRestartTestAsync(test); + }); + it('applies a client-default timeout while a rushx script is served, and says why', async () => { const test: IPendingRestartTest = createPendingRestartTest( { waitTimeoutMs: 50, waitTimeoutIsDefault: true }, diff --git a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts index 74da42a9d0..ead5608628 100644 --- a/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceRestartArbiter.test.ts @@ -3,12 +3,15 @@ import { setTimeout as delayAsync } from 'node:timers/promises'; +import type { DaemonRestartReason } from '@rushstack/rush-daemon-protocol'; + import { RequestSchedulerError, RequestSchedulerErrorCode } from '../RequestScheduler'; import { WorkspaceRestartArbiter, type IWorkspaceRestartDrainOptions, type IWorkspaceRestartRecheck, type IWorkspaceRestartTicket, + type IWorkspaceRestartWaitReport, type IWorkspaceRestartWaitResult } from '../WorkspaceRestartArbiter'; @@ -18,6 +21,21 @@ const WAIT: { abortSignal: AbortSignal; noWait: undefined; waitTimeoutMs: undefi waitTimeoutMs: undefined }; +const LOCKFILE_CHANGED: DaemonRestartReason = { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/pnpm-lock.yaml'] +}; +const ENVIRONMENT_CHANGED: DaemonRestartReason = { + kind: 'environmentChanged', + variableNames: ['NODE_OPTIONS', 'RUSH_BUILD_CACHE_ENABLED'] +}; +const VERSION_SELECTED: DaemonRestartReason = { + kind: 'workspaceInputsChanged', + selectedRushVersion: '5.180.0' +}; + +type WaitReports = [number, IWorkspaceRestartWaitReport][]; + async function isSettledAsync(promise: Promise): Promise { let settled: boolean = false; void promise.then( @@ -138,6 +156,91 @@ describe(WorkspaceRestartArbiter.name, () => { expect(arbiter.servingCount).toBe(0); }); + it('reports how many of the requests that a restart candidate waits for run a rushx script, and its reason', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const build: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const reports: WaitReports = []; + const waiting: Promise = arbiter.waitForDrainAsync(candidate, { + ...WAIT, + restartReason: LOCKFILE_CHANGED, + onServingCountChanged: (count: number, report: IWorkspaceRestartWaitReport) => + reports.push([count, report]) + }); + arbiter.leave(script); + arbiter.leave(build); + await waiting; + expect(reports).toEqual([ + [2, { scriptCount: 1, restartReason: LOCKFILE_CHANGED }], + [1, { scriptCount: 0, restartReason: LOCKFILE_CHANGED }] + ]); + arbiter.leave(candidate); + expect(arbiter.servingCount).toBe(0); + }); + + it.each([ + [ + 'drain', + 'no-wait', + LOCKFILE_CHANGED, + 'The daemon is still serving other requests, which finish before it restarts because ' + + 'common/config/rush/pnpm-lock.yaml changed; the request did not wait for a restart.' + ], + [ + 'drain', + 'timeout', + ENVIRONMENT_CHANGED, + "The request was not admitted before the daemon could restart because this request's environment differs " + + "from the daemon's in NODE_OPTIONS and RUSH_BUILD_CACHE_ENABLED. The restart waits for the requests that " + + 'the daemon is serving to finish. Use --wait-timeout to wait longer.' + ], + [ + 'pending restart', + 'no-wait', + ENVIRONMENT_CHANGED, + "Another request is waiting to restart the daemon because its environment differs from the daemon's in " + + 'NODE_OPTIONS and RUSH_BUILD_CACHE_ENABLED; the rushx script did not wait for the restart.' + ], + [ + 'pending restart', + 'timeout', + VERSION_SELECTED, + 'The rushx script was not admitted before the daemon could restart for another request, because it selects ' + + 'Rush 5.180.0. A script waits for a pending restart so that the restart does not wait for the script, and ' + + 'the restart waits for the requests that the daemon is serving to finish. Use --wait-timeout to ' + + 'wait longer.' + ] + ])('names the restart reason when a %s wait fails with %s', async (wait, mode, restartReason, message) => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const build: IWorkspaceRestartTicket = arbiter.enter(); + const candidate: IWorkspaceRestartTicket = arbiter.enter(); + const options: IWorkspaceRestartDrainOptions = { + ...WAIT, + noWait: mode === 'no-wait' ? true : undefined, + waitTimeoutMs: mode === 'timeout' ? 10 : undefined + }; + let error: unknown; + if (wait === 'drain') { + error = await arbiter + .waitForDrainAsync(candidate, { ...options, restartReason }) + .catch((caught: unknown) => caught); + } else { + const draining: Promise = arbiter.waitForDrainAsync(candidate, { + ...WAIT, + restartReason + }); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + error = await arbiter.waitForPendingRestartAsync(script, options).catch((caught: unknown) => caught); + arbiter.leave(script); + arbiter.leave(build); + await draining; + } + expect((error as Error).message).toBe(message); + for (const ticket of [build, candidate]) arbiter.leave(ticket); + expect(arbiter.servingCount).toBe(0); + }); + it('reports no count for a candidate that does not wait', async () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const ticket: IWorkspaceRestartTicket = arbiter.enter(); @@ -347,6 +450,41 @@ describe(WorkspaceRestartArbiter.name, () => { expect(arbiter.servingCount).toBe(0); }); + it('tells the script why the daemon restarts, and again when another request then needs the restart', async () => { + const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); + const first: IWorkspaceRestartTicket = arbiter.enter(); + await arbiter.waitForDrainAsync(first, { ...WAIT, restartReason: LOCKFILE_CHANGED }); + const build: IWorkspaceRestartTicket = arbiter.enter(); + const second: IWorkspaceRestartTicket = arbiter.enter(); + const secondDrain: Promise = arbiter.waitForDrainAsync(second, { + ...WAIT, + restartReason: ENVIRONMENT_CHANGED + }); + const script: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); + const reports: WaitReports = []; + const waiting: Promise = arbiter.waitForPendingRestartAsync(script, { + ...WAIT, + onServingCountChanged: (count: number, report: IWorkspaceRestartWaitReport) => + reports.push([count, report]) + }); + // The first candidate finds after its drain that it no longer needs the restart; it is still served. + arbiter.withdrawRestart(first); + arbiter.leave(first); + arbiter.leave(build); + await secondDrain; + expect(await isSettledAsync(waiting)).toBe(false); + arbiter.leave(second); + await waiting; + expect(reports).toEqual([ + [3, { scriptCount: 0, restartReason: LOCKFILE_CHANGED }], + [3, { scriptCount: 0, restartReason: ENVIRONMENT_CHANGED }], + [2, { scriptCount: 0, restartReason: ENVIRONMENT_CHANGED }], + [1, { scriptCount: 0, restartReason: ENVIRONMENT_CHANGED }] + ]); + arbiter.leave(script); + expect(arbiter.servingCount).toBe(0); + }); + it('waits for every pending restart', async () => { const arbiter: WorkspaceRestartArbiter = new WorkspaceRestartArbiter(); const running: IWorkspaceRestartTicket = arbiter.enter({ runsScript: true }); diff --git a/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts b/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts index 1e8ffdf642..b9ff7653b3 100644 --- a/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts +++ b/libraries/rush-daemon/src/test/WorkspaceServedScriptAdmission.test.ts @@ -12,13 +12,20 @@ jest.mock('@microsoft/rush-lib', () => { import * as fs from 'node:fs'; import * as path from 'node:path'; -import { captureWorkspaceInputFingerprintAsync, WorkspaceInputChangeTier } from '@microsoft/rush-lib'; +import { + captureWorkspaceInputFingerprintAsync, + type IWorkspaceInputFingerprint, + WorkspaceInputChangeTier +} from '@microsoft/rush-lib'; import { DaemonFrameType, decodeDaemonControlMessage, type DaemonControlMessage, + type DaemonRestartReason, + type IDaemonCommandResult, type IDaemonFrame, - type IDaemonRequestEnvelope + type IDaemonRequestEnvelope, + type IDaemonRequestQueuePositionMessage } from '@rushstack/rush-daemon-protocol'; import { getInstalledWorkspaceSuccessorLaunchAsync } from '../WorkspaceProcessRestart'; @@ -108,8 +115,11 @@ async function serveAsync(fixture: DaemonGraphTestFixture): Promise; + readonly requestId: string; /** The queue positions that the daemon has reported so far. */ readonly positions: ReadonlyArray; + /** The payloads of those queue position messages. */ + readonly positionPayloads: ReadonlyArray; readonly settled: () => boolean; } @@ -123,6 +133,7 @@ async function startRequestAsync( const payload: IDaemonRequestEnvelope = fixture.envelope(argv, overrides); await client.sendControlAsync({ kind: 'requestStart', payload }); const positions: number[] = []; + const positionPayloads: IDaemonRequestQueuePositionMessage['payload'][] = []; let settled: boolean = false; const readAsync = async (): Promise => { const frames: IDaemonFrame[] = []; @@ -132,7 +143,10 @@ async function startRequestAsync( frames.push(frame); if (frame.kind !== DaemonFrameType.controlJson) continue; const message: DaemonControlMessage = decodeDaemonControlMessage(frame.payload); - if (message.kind === 'queuePosition') positions.push(message.payload.position); + if (message.kind === 'queuePosition') { + positions.push(message.payload.position); + positionPayloads.push(message.payload); + } if (message.kind === 'requestResult' && message.payload.requestId === payload.requestId) { return { frames, terminal: message }; } @@ -145,7 +159,7 @@ async function startRequestAsync( const exchange: Promise = readAsync(); // A failed expectation leaves the exchange unread until the fixture closes the connection. exchange.catch(() => undefined); - return { exchange, positions, settled: () => settled }; + return { exchange, requestId: payload.requestId, positions, positionPayloads, settled: () => settled }; } /** @@ -308,6 +322,29 @@ describe('workspace admission while a served rushx script runs', () => { expect(fixture.runs()).not.toContain('serve2-start'); // It waits for the served script and the restart. expect(late.positions).toEqual([2]); + const restartReason: DaemonRestartReason = { + kind: 'environmentChanged', + variableNames: ['RUSHD_RELOAD_TIER_TEST'] + }; + await waitForAsync( + () => restart.positionPayloads.length >= 3 || restart.settled(), + 'the restart to stop counting the later script' + ); + expect(restart.positionPayloads).toEqual([ + { position: 1, requestId: restart.requestId, restartReason, scriptCount: 1 }, + // The later script counts until it finds that a restart is pending. + { position: 2, requestId: restart.requestId, restartReason, scriptCount: 2 }, + { position: 1, requestId: restart.requestId, restartReason, scriptCount: 1 } + ]); + expect(late.positionPayloads).toEqual([ + { + position: 2, + requestId: late.requestId, + restartReason, + scriptCount: 1, + restartsForAnotherRequest: true + } + ]); fixture.write(RELEASE_FILE, ''); expectSuccess(await script.exchange); @@ -319,6 +356,12 @@ describe('workspace admission while a served rushx script runs', () => { }); } expect(late.positions).toEqual([2, 1]); + expect(late.positionPayloads[1]).toEqual({ + position: 1, + requestId: late.requestId, + restartReason, + restartsForAnotherRequest: true + }); expect(fixture.runs()).not.toContain('serve2-start'); const restarted = await fixture.host.restartCompleted; expect(restarted?.pid).not.toBe(before.pid); @@ -406,6 +449,16 @@ describe('a restart drain whose change is reverted', () => { await waitForAsync(() => pause.positions.length > 0 || pause.settled(), 'the request to wait for the drain'); await delayAsync(1500); expect(pause.settled()).toBe(false); + // Like a build, the request is told why it waits (task 166). + expect(pause.positionPayloads[0]).toEqual({ + position: 1, + requestId: pause.requestId, + restartReason: { + kind: 'workspaceInputsChanged', + installationFiles: ['common/config/rush/npm-shrinkwrap.json'] + }, + scriptCount: 1 + }); const revertedAt: number = Date.now(); revert(); @@ -550,3 +603,107 @@ describe('a restart drain whose change is reverted', () => { } }); }); + +describe('the reason for a restart that waits for a served rushx script', () => { + const actualCaptureAsync: typeof captureWorkspaceInputFingerprintAsync = + jest.requireActual( + '@microsoft/rush-lib' + ).captureWorkspaceInputFingerprintAsync; + + /** Makes every later capture of the workspace inputs report the given change. */ + function changeCapturedInputs(change: Partial): void { + inputCaptureMock.mockImplementation(async (options) => ({ + ...(await actualCaptureAsync(options)), + ...change + })); + } + + afterEach(() => { + inputCaptureMock.mockImplementation(actualCaptureAsync); + }); + + it.each< + [ + string, + (fixture: DaemonGraphTestFixture) => Partial, + DaemonRestartReason, + string + ] + >([ + [ + 'a changed installation file', + (fixture: DaemonGraphTestFixture) => { + changeInstallation(fixture); + return {}; + }, + { kind: 'workspaceInputsChanged', installationFiles: ['common/config/rush/npm-shrinkwrap.json'] }, + 'because common/config/rush/npm-shrinkwrap.json changed' + ], + [ + 'changed code of Rush or a Rush plugin', + () => { + changeCapturedInputs({ runtimeHash: 'changed' }); + return {}; + }, + // The capture found no changed file, since the test only changed its hash. + { kind: 'workspaceInputsChanged', implementationFiles: [] }, + 'because the code of Rush or a Rush plugin changed' + ], + [ + 'another Rush version', + () => { + changeCapturedInputs({ selectedRushVersion: '9.9.9' }); + return {}; + }, + { kind: 'workspaceInputsChanged', selectedRushVersion: '9.9.9' }, + 'because this request selects Rush 9.9.9' + ], + [ + 'a changed environment', + (fixture: DaemonGraphTestFixture) => ({ + environment: { ...fixture.environment, RUSHD_RELOAD_TIER_TEST: 'changed' } + }), + { kind: 'environmentChanged', variableNames: ['RUSHD_RELOAD_TIER_TEST'] }, + "because this request's environment differs from the daemon's in RUSHD_RELOAD_TIER_TEST" + ] + ])( + 'names %s in the queue positions and the admission error of a build', + async (inputs, change, reason, cause) => { + const fixture: DaemonGraphTestFixture = await createServingFixtureAsync((created) => { + // The build is never admitted, so no successor is launched. + created.getSuccessorLaunchAsync = () => Promise.reject(new Error('No successor was expected.')); + }); + try { + expectSuccess(await fixture.runAsync(BUILD_A)); + const script: IServedScript = await serveAsync(fixture); + + const overrides: Partial = change(fixture); + const build: IStreamedRequest = await startRequestAsync(fixture, BUILD_A, { + ...overrides, + admission: { waitTimeoutMs: 1500 } + }); + const { terminal } = await build.exchange; + expect(terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 1, admissionErrorCode: 'wait-timeout' } + }); + expect((terminal.payload as IDaemonCommandResult).errorMessage).toContain( + `The request was not admitted before the daemon could restart ${cause}. The restart waits for the ` + + 'requests that the daemon is serving to finish, including a rushx script that may not exit until it ' + + 'is stopped. Stop the script, or use --wait-timeout to wait longer.' + ); + expect(build.positionPayloads).toEqual([ + { position: 1, requestId: build.requestId, restartReason: reason, scriptCount: 1 } + ]); + expect(script.settled()).toBe(false); + + fixture.write(RELEASE_FILE, ''); + expectSuccess(await script.exchange); + } finally { + inputCaptureMock.mockImplementation(actualCaptureAsync); + fixture.write(RELEASE_FILE, ''); + await fixture[Symbol.asyncDispose](); + } + } + ); +}); diff --git a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts index 4530644a53..1a1cbf7e44 100644 --- a/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts +++ b/libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts @@ -271,8 +271,9 @@ interface IFileDigest { * @remarks * Embedding hosts create one cache per workspace lifetime and pass it to * {@link captureWorkspaceInputFingerprintAsync} through `runtimeCache`. The capture function - * updates the cache; hosts can inspect {@link WorkspaceRuntimeFingerprintCache.changedPaths} - * when reporting why a process restart is required. + * updates the cache; hosts can inspect {@link WorkspaceRuntimeFingerprintCache.changedPaths} and + * {@link WorkspaceRuntimeFingerprintCache.changedInstallationPaths} when reporting why a process restart + * is required. * * The cache also memoizes the digests of workspace definition and installation files, which users edit while * a host is running. Such a digest is recorded only if the file's ctime and mtime were at least 3 seconds old @@ -292,6 +293,8 @@ export class WorkspaceRuntimeFingerprintCache { private readonly _inputFiles: Map = new Map(); private _baseline: ReadonlyMap | undefined; private _changedPaths: ReadonlyArray = []; + private _installationBaseline: ReadonlyMap | undefined; + private _changedInstallationPaths: ReadonlyArray = []; /** * Implementation paths whose content or existence differs from the first capture using this cache. @@ -301,6 +304,15 @@ export class WorkspaceRuntimeFingerprintCache { return this._changedPaths; } + /** + * Installation files, such as lockfiles and the flags that an install writes, whose content or existence + * differs from the first capture using this cache. Updated by each capture; metadata-only changes do not + * appear in this list. + */ + public get changedInstallationPaths(): ReadonlyArray { + return this._changedInstallationPaths; + } + /** @internal */ public _hashPaths(paths: ReadonlyArray): string { const filenames: Set = new Set(); @@ -334,13 +346,9 @@ export class WorkspaceRuntimeFingerprintCache { entries.push([filename, 'missing']); } } - const current: ReadonlyMap = new Map( - entries.map((entry) => [entry[0], JSON.stringify(entry)]) - ); + const current: ReadonlyMap = getEntryMap(entries); this._baseline ??= current; - this._changedPaths = Array.from(new Set([...this._baseline.keys(), ...current.keys()])).filter( - (filename) => this._baseline!.get(filename) !== current.get(filename) - ); + this._changedPaths = getChangedPaths(this._baseline, current); return hashText(JSON.stringify(entries)); } @@ -350,6 +358,23 @@ export class WorkspaceRuntimeFingerprintCache { * @internal */ public async _hashInputFilesAsync(filenames: Iterable): Promise { + return hashText(JSON.stringify(await this._getInputFileEntriesAsync(filenames))); + } + + /** + * Hashes installation files as {@link WorkspaceRuntimeFingerprintCache._hashInputFilesAsync} does, and + * updates {@link WorkspaceRuntimeFingerprintCache.changedInstallationPaths}. + * @internal + */ + public async _hashInstallationFilesAsync(filenames: Iterable): Promise { + const entries: ReadonlyArray[] = await this._getInputFileEntriesAsync(filenames); + const current: ReadonlyMap = getEntryMap(entries); + this._installationBaseline ??= current; + this._changedInstallationPaths = getChangedPaths(this._installationBaseline, current); + return hashText(JSON.stringify(entries)); + } + + private async _getInputFileEntriesAsync(filenames: Iterable): Promise[]> { const settledBeforeNs: bigint = getSettledBeforeNs(); const sortedFilenames: string[] = Array.from(filenames).sort(); const entries: ReadonlyArray[] = new Array(sortedFilenames.length); @@ -393,10 +418,24 @@ export class WorkspaceRuntimeFingerprintCache { }, { concurrency: 3 } ); - return hashText(JSON.stringify(entries)); + return entries; } } +function getEntryMap(entries: ReadonlyArray>): ReadonlyMap { + return new Map(entries.map((entry) => [entry[0], JSON.stringify(entry)])); +} + +/** Returns the paths whose entries differ between two captures, including paths that only one of them has. */ +function getChangedPaths( + baseline: ReadonlyMap, + current: ReadonlyMap +): string[] { + return Array.from(new Set([...baseline.keys(), ...current.keys()])).filter( + (filename) => baseline.get(filename) !== current.get(filename) + ); +} + /** The strongest action required by a workspace input change. @alpha */ export enum WorkspaceInputChangeTier { Reuse = 0, @@ -502,7 +541,7 @@ export async function captureWorkspaceInputFingerprintAsync( return { configurationHash: await cache._hashInputFilesAsync(definitions), environmentHash: hashText(JSON.stringify(getWorkspaceFingerprintEnvironmentEntries(environment))), - installationHash: await cache._hashInputFilesAsync(installation), + installationHash: await cache._hashInstallationFilesAsync(installation), runtimeHash: hashText(JSON.stringify([process.execPath, process.version, runtimeHash])), selectedRushVersion: environment.RUSH_PREVIEW_VERSION ?? rushJson.rushVersion }; diff --git a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts index 203d82216f..2f2638fe4a 100644 --- a/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts +++ b/libraries/rush-lib/src/api/test/WorkspaceInputFingerprint.test.ts @@ -527,10 +527,18 @@ describe('workspace input fingerprints', () => { expect(classifyWorkspaceInputChange(retargeted, removed)).toBe(WorkspaceInputChangeTier.Reload); write('a/package.json', '{"name":"a","version":"1.0.1"}'); expect(await captureAsync()).toEqual([retargeted, 1]); + expect(runtimeCache.changedInstallationPaths).toEqual([]); // An installation file write('common/config/rush/pnpm-lock.yaml', 'lockfileVersion: 10'); const [installed] = await captureAsync(); expect(classifyWorkspaceInputChange(retargeted, installed)).toBe(WorkspaceInputChangeTier.Restart); + const lockfilePath: string = path.join(rushConfiguration.commonRushConfigFolder, 'pnpm-lock.yaml'); + expect(runtimeCache.changedInstallationPaths).toEqual([lockfilePath]); + // Content, not the edit, identifies the installation. + write('common/config/rush/pnpm-lock.yaml', 'lockfileVersion: 1'); + const [restored] = await captureAsync(); + expect(restored.installationHash).toBe(first.installationHash); + expect(runtimeCache.changedInstallationPaths).toEqual([]); } finally { jest.restoreAllMocks(); fs.rmSync(folder, { recursive: true, force: true }); From 23c3e882588bc5eedf5dac1048a2979f23d60edc Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 05:04:01 +0000 Subject: [PATCH 080/265] [rush-lib] A warm daemon request parses its command line once Swarm integration step 65; original commit a15fb1c70a (merge of swarm/r05-t146-int3 at d46b1047e7). Scope: task 146. Brings r05's task 146 (board 2739, re-tipped in board 3163 and board 3403): a warm request parses its command line once instead of twice, and re-derives a JSON configuration file only when the file's text changed. 146-int3 adds 170-int and the one-line test fix that ch01's BLOCK board 3328 asked for. Second agent: t01 board 2912. s17 batch D2, item 3 of 6. Gate: ch01 GATE OK board 3514 (tree 0e2dfdba29) Commits folded into this step (3): - 475db85a16 [rush-lib] An engine host parses each JSON configuration file again only if its text changed (task 146) - 4937fb600d [rush-daemon] Parse the command line of a warm request once, not twice (task 146) - d46b1047e7 rush-daemon: expect the error before the reused parse's diagnostics, as task 170 orders them (task 146, ch01 board 3328) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...gine-json-load-cache_2026-09-28-23-50.json | 10 + ...identity-parse-reuse_2026-09-29-00-22.json | 10 + .../src/ProductionDaemonRequestResolver.ts | 83 ++++++-- .../test/ProductionDaemonRequestParse.test.ts | 195 ++++++++++++++++++ .../ProductionDaemonRequestResolver.test.ts | 17 ++ .../src/api/CommandLineConfiguration.ts | 68 +++--- .../src/api/test/PhasedCommandEngine.test.ts | 185 +++++++++++++++++ .../rush-lib/src/cli/RushCommandLineParser.ts | 15 +- .../cli/scriptActions/GlobalScriptAction.ts | 14 +- .../PluginLoader/PluginLoaderBase.ts | 18 +- .../src/pluginFramework/PluginManager.ts | 9 +- .../src/utilities/JsonFileLoadCache.ts | 78 +++++++ .../utilities/test/JsonFileLoadCache.test.ts | 147 +++++++++++++ 13 files changed, 790 insertions(+), 59 deletions(-) create mode 100644 common/changes/@microsoft/rush/swarm-r05-engine-json-load-cache_2026-09-28-23-50.json create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r05-identity-parse-reuse_2026-09-29-00-22.json create mode 100644 libraries/rush-daemon/src/test/ProductionDaemonRequestParse.test.ts create mode 100644 libraries/rush-lib/src/utilities/JsonFileLoadCache.ts create mode 100644 libraries/rush-lib/src/utilities/test/JsonFileLoadCache.test.ts diff --git a/common/changes/@microsoft/rush/swarm-r05-engine-json-load-cache_2026-09-28-23-50.json b/common/changes/@microsoft/rush/swarm-r05-engine-json-load-cache_2026-09-28-23-50.json new file mode 100644 index 0000000000..55b3fff124 --- /dev/null +++ b/common/changes/@microsoft/rush/swarm-r05-engine-json-load-cache_2026-09-28-23-50.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "A long-lived engine host parses and validates command-line.json, plugin manifests and plugin command-line.json files, and autoinstaller package.json files again only if their text changed since the host's last parse.", + "type": "none" + } + ], + "packageName": "@microsoft/rush" +} diff --git a/common/changes/@rushstack/rush-daemon/swarm-r05-identity-parse-reuse_2026-09-29-00-22.json b/common/changes/@rushstack/rush-daemon/swarm-r05-identity-parse-reuse_2026-09-29-00-22.json new file mode 100644 index 0000000000..6c0f329cc7 --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r05-identity-parse-reuse_2026-09-29-00-22.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "The production resolver parses the command line of a warm request once instead of twice. It resolves the request with the parse from the request's identity check, and parses again if the request's command line, environment or session changed.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon" +} diff --git a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts index 30affd7246..2fbcf08eee 100644 --- a/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts +++ b/libraries/rush-daemon/src/ProductionDaemonRequestResolver.ts @@ -19,7 +19,7 @@ import { type Operation, type OperationEnabledState } from '@microsoft/rush-lib'; -import type { IDaemonPhasedOperationSelection } from '@rushstack/rush-daemon-protocol'; +import type { IDaemonPhasedOperationSelection, IDaemonRequestEnvelope } from '@rushstack/rush-daemon-protocol'; import { DaemonRequestDispatchError, @@ -42,6 +42,14 @@ import type { IWorkspaceResolverLifecycle } from './WorkspaceResolverLifecycle'; import { createInputsCompatibilityCheck, getOperationsWithChangedInputs } from './WorkspaceInputsComparison'; import { createDaemonRequestTelemetrySink, type IDaemonEngineCreationTiming } from './DaemonRequestTelemetry'; +/** A native parse of the command line of a request. */ +interface IParsedCommand { + readonly command: PhasedCommandEngine; + readonly terminal: EngineTerminalProvider; + readonly envelope: IDaemonRequestEnvelope; + readonly workspaceSession: IWorkspaceSession; +} + /** * Binds the standalone host to a real native build/rebuild graph on its first request. * @@ -68,6 +76,9 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { /** The daemon's environment when it started. Replacement sessions keep it; see `createForSession`. */ readonly #startupEnvironment: Readonly>; readonly #validateGraphInputsAsync: (() => Promise) | undefined; + // The workspace lifecycle checks the command identity of a request, and then resolves the request under the same + // admission lease. Keyed by the abort signal of the request, a parse is kept only as long as its request. + readonly #identityParses: WeakMap = new WeakMap(); public constructor(options?: { readonly preparationLock?: LockFile; @@ -102,14 +113,19 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { /** Inspects the native command shape without constructing or executing an operation graph. */ public async getCommandParameterIdentityAsync(options: IResolveDaemonRequestOptions): Promise { - return (await this.#parseCommandAsync(options, new EngineTerminalProvider())).parameterIdentity; + const parsed: IParsedCommand = await this.#parseCommandAsync(options); + // Resolving the same request uses this parse instead of parsing the same command line again. + this.#identityParses.set(options.abortSignal, parsed); + return parsed.command.parameterIdentity; } public async resolveRequestAsync(options: IResolveDaemonRequestOptions): Promise { const resolveStartTimeMs: number = performance.now(); const { envelope, workspaceSession } = options; - const terminal: EngineTerminalProvider = new EngineTerminalProvider(); - const command: PhasedCommandEngine = await this.#parseCommandAsync(options, terminal); + const { command, terminal }: IParsedCommand = await this.#parseCommandAsync( + options, + this.#takeIdentityParse(options) + ); let bindingStartTimeMs: number | undefined; if (this.#binding) { if ( @@ -191,10 +207,17 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { }; } + /** Returns the parse of the identity check of this request, if its command line and session are unchanged. */ + #takeIdentityParse(options: IResolveDaemonRequestOptions): IParsedCommand | undefined { + const parsed: IParsedCommand | undefined = this.#identityParses.get(options.abortSignal); + this.#identityParses.delete(options.abortSignal); + return parsed && isSameCommandLine(parsed, options) ? parsed : undefined; + } + async #parseCommandAsync( options: IResolveDaemonRequestOptions, - terminal: EngineTerminalProvider - ): Promise { + identityParse?: IParsedCommand + ): Promise { const { envelope, workspaceSession, abortSignal } = options; if (!['build', 'rebuild'].includes(envelope.commandName) || envelope.commandOrigin !== 'built-in') { throw new DaemonRequestDispatchError( @@ -216,23 +239,27 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { 'may have changed process.env. Restart the daemon or use --no-daemon.' ); } - let command: PhasedCommandEngine; - try { - command = await PhasedCommandEngine.parseAsync({ - argv: envelope.argv, - cwd: envelope.cwd, - environment: envelope.environment, - rushConfiguration: workspaceSession.rushConfiguration, - terminalProvider: terminal - }); - } catch (error) { - // Answer only for a command line of the requested command; in-process Rush reports any other one. - if (error instanceof PhasedCommandEngineUsageError && envelope.argv[0] === envelope.commandName) { - throw new DaemonRequestUsageError(terminal.describeError(error), error.exitCode, { cause: error }); + let parsed: IParsedCommand | undefined = identityParse; + if (!parsed) { + const terminal: EngineTerminalProvider = new EngineTerminalProvider(); + try { + const command: PhasedCommandEngine = await PhasedCommandEngine.parseAsync({ + argv: envelope.argv, + cwd: envelope.cwd, + environment: envelope.environment, + rushConfiguration: workspaceSession.rushConfiguration, + terminalProvider: terminal + }); + parsed = { command, terminal, envelope, workspaceSession }; + } catch (error) { + // Answer only for a command line of the requested command; in-process Rush reports any other one. + if (error instanceof PhasedCommandEngineUsageError && envelope.argv[0] === envelope.commandName) { + throw new DaemonRequestUsageError(terminal.describeError(error), error.exitCode, { cause: error }); + } + throw new DaemonRequestDispatchError('unsupported', terminal.describeError(error), { cause: error }); } - throw new DaemonRequestDispatchError('unsupported', terminal.describeError(error), { cause: error }); } - if (command.commandName !== envelope.commandName) { + if (parsed.command.commandName !== envelope.commandName) { throw new DaemonRequestDispatchError( 'invalidRequest', 'The command name does not match the native parsed argv.' @@ -244,7 +271,7 @@ export class ProductionDaemonRequestResolver implements IDaemonRequestResolver { getDaemonShutdownReason(abortSignal)?.message ?? 'The request was cancelled before engine initialization.' ); - return command; + return parsed; } /** @@ -386,6 +413,18 @@ function environmentIdentity(environment: Readonly undefined + } as unknown as IWorkspaceSession; +} + +function createOptions( + workspaceSession: IWorkspaceSession, + abortController: AbortController = new AbortController() +): IResolveDaemonRequestOptions { + return { + abortSignal: abortController.signal, + envelope: createWireEnvelope('request', 'build', process.cwd(), { + commandOrigin: 'built-in', + environment: Object.fromEntries( + Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined) + ) + }), + workspaceSession + }; +} + +/** Checks the identity of a request, which is unreachable when this returns. */ +async function checkIdentityAsync( + resolver: ProductionDaemonRequestResolver, + session: IWorkspaceSession +): Promise { + await resolver.getCommandParameterIdentityAsync(createOptions(session)); +} + +/** The workspace lifecycle resolves a request with a copy of the envelope of its identity check. */ +function dispatched(options: IResolveDaemonRequestOptions): IResolveDaemonRequestOptions { + return { ...options, envelope: { ...options.envelope, admission: { waitTimeoutMs: 1000 } } }; +} + +describe('ProductionDaemonRequestResolver command line parsing', () => { + let parses: number; + let selectionError: Error | undefined; + let commands: WeakRef[]; + let parse: jest.SpyInstance; + + beforeEach(() => { + parses = 0; + selectionError = undefined; + commands = []; + parse = jest + .spyOn(PhasedCommandEngine, 'parseAsync') + .mockImplementation(async ({ terminalProvider }: IParsePhasedCommandOptions) => { + terminalProvider.write(`Warning from parse ${++parses}\n`, TerminalProviderSeverity.warning); + const command: PhasedCommandEngine = { + commandName: 'build', + parameterIdentity: 'parameters', + requestSettings: {}, + unmatchedCompatiblePluginNames: [], + selectOperationsAsync: async () => { + if (selectionError) throw selectionError; + return new Map(); + } + } as unknown as PhasedCommandEngine; + commands.push(new WeakRef(command)); + return command; + }); + }); + + afterEach(() => { + parse.mockRestore(); + }); + + it('resolves a request with the parse of its identity check', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(createSession()); + await expect(resolver.getCommandParameterIdentityAsync(options)).resolves.toBe('parameters'); + await expect(resolver.resolveRequestAsync(dispatched(options))).resolves.toMatchObject({ + kind: 'phased', + request: { requestId: 'request' } + }); + expect(parses).toBe(1); + }); + + it('reports the diagnostics of the parse that it reused with a selection error', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const session: IWorkspaceSession = createSession(); + await resolver.resolveRequestAsync(createOptions(session)); + const options: IResolveDaemonRequestOptions = createOptions(session); + await resolver.getCommandParameterIdentityAsync(options); + selectionError = new Error('The project name "nope" does not exist.'); + await expect(resolver.resolveRequestAsync(dispatched(options))).rejects.toMatchObject({ + code: 'invalidRequest', + message: 'The project name "nope" does not exist.\nWarning from parse 2' + }); + expect(parses).toBe(2); + }); + + it.each<[string, (envelope: IDaemonRequestEnvelope) => Partial]>([ + ['another request', () => ({ requestId: 'another' })], + ['another argv', ({ argv }) => ({ argv: [...argv] })], + ['another working folder', ({ cwd }) => ({ cwd: path.dirname(cwd) })], + ['another environment', ({ environment }) => ({ environment: { ...environment } })] + ])('parses again to resolve %s', async (name, change) => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(createSession()); + await resolver.getCommandParameterIdentityAsync(options); + await resolver.resolveRequestAsync({ + ...options, + envelope: { ...options.envelope, ...change(options.envelope) } + }); + expect(parses).toBe(2); + }); + + it('parses again to resolve a request for another session, or with another resolver', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(createSession()); + await resolver.getCommandParameterIdentityAsync(options); + await resolver.resolveRequestAsync({ ...options, workspaceSession: createSession() }); + expect(parses).toBe(2); + await resolver.getCommandParameterIdentityAsync(options); + await resolver.createForSession().resolveRequestAsync(dispatched(options)); + expect(parses).toBe(4); + }); + + it('uses the parse of an identity check for only one resolution', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(createSession()); + await resolver.getCommandParameterIdentityAsync(options); + await resolver.resolveRequestAsync(dispatched(options)); + await resolver.resolveRequestAsync(dispatched(options)); + expect(parses).toBe(2); + }); + + it('rejects a request that was cancelled after its identity check', async () => { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const abortController: AbortController = new AbortController(); + const options: IResolveDaemonRequestOptions = createOptions(createSession(), abortController); + await resolver.getCommandParameterIdentityAsync(options); + abortController.abort(); + await expect(resolver.resolveRequestAsync(dispatched(options))).rejects.toMatchObject({ + code: 'routingFailed', + message: 'The request was cancelled before engine initialization.' + }); + expect(parses).toBe(1); + }); + + it('rejects a request if the daemon environment changed after its identity check', async () => { + // A name that a plugin adds to the daemon's environment is not a change, so this changes a startup name. + process.env.RUSHD_TEST_REQUEST_PARSE = 'startup'; + try { + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + const options: IResolveDaemonRequestOptions = createOptions(createSession()); + await resolver.getCommandParameterIdentityAsync(options); + process.env.RUSHD_TEST_REQUEST_PARSE = 'changed'; + await expect(resolver.resolveRequestAsync(dispatched(options))).rejects.toMatchObject({ + code: 'unsupported', + message: expect.stringContaining( + "The daemon's own environment changed after it started (RUSHD_TEST_REQUEST_PARSE)" + ) + }); + expect(parses).toBe(1); + } finally { + delete process.env.RUSHD_TEST_REQUEST_PARSE; + } + }); + + it('keeps the parse of an identity check only as long as its request', async () => { + v8.setFlagsFromString('--expose-gc'); + const collectGarbage: () => void = vm.runInNewContext('gc'); + const resolver: ProductionDaemonRequestResolver = new ProductionDaemonRequestResolver(); + await checkIdentityAsync(resolver, createSession()); + parse.mockClear(); + // A WeakRef keeps its target until the current job ends. + await new Promise((resolve) => setImmediate(resolve)); + collectGarbage(); + expect(commands).toHaveLength(1); + expect(commands[0].deref()).toBeUndefined(); + }); +}); diff --git a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts index 49cccacad6..d98b5f0340 100644 --- a/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts +++ b/libraries/rush-daemon/src/test/ProductionDaemonRequestResolver.test.ts @@ -238,6 +238,23 @@ describe('native production daemon engine', () => { } }); + it('parses the command line of a warm request once, for its identity check and its resolution', async () => { + const fixture: IFixture = await createFixtureAsync(); + const parse: jest.SpyInstance = jest.spyOn(PhasedCommandEngine, 'parseAsync'); + try { + await runAsync(fixture, 'cold-parse', ['build', '--only', 'a']); + parse.mockClear(); + expect((await runAsync(fixture, 'warm-parse', ['build', '--only', 'a'])).terminal).toMatchObject({ + kind: 'requestResult', + payload: { exitCode: 0, scheduled: false } + }); + expect(parse).toHaveBeenCalledTimes(1); + } finally { + parse.mockRestore(); + await fixture[Symbol.asyncDispose](); + } + }); + it('serves client output and request-scoped settings in the same generation instead of restarting', async () => { const fixture: IFixture = await createFixtureAsync(); try { diff --git a/libraries/rush-lib/src/api/CommandLineConfiguration.ts b/libraries/rush-lib/src/api/CommandLineConfiguration.ts index 8ec4b5a3de..7423e3cf95 100644 --- a/libraries/rush-lib/src/api/CommandLineConfiguration.ts +++ b/libraries/rush-lib/src/api/CommandLineConfiguration.ts @@ -1,10 +1,11 @@ // Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. // See LICENSE in the project root for license information. -import { JsonFile, JsonSchema, FileSystem } from '@rushstack/node-core-library'; +import { JsonSchema, FileSystem } from '@rushstack/node-core-library'; import type { CommandLineParameter } from '@rushstack/ts-command-line'; import { RushConstants } from '../logic/RushConstants'; +import { type JsonFileLoadCache, loadJsonFile } from '../utilities/JsonFileLoadCache'; import type { CommandJson, ICommandLineJson, @@ -635,11 +636,19 @@ export class CommandLineConfiguration { * use {@see loadFromFileOrDefault} instead. * * If the file does not exist, this function returns `undefined` + * + * @param jsonFileLoadCache - The cache of a long-lived engine host, which parses the file again only if it changed. */ - public static tryLoadFromFile(jsonFilePath: string): CommandLineConfiguration | undefined { + public static tryLoadFromFile( + jsonFilePath: string, + jsonFileLoadCache?: JsonFileLoadCache + ): CommandLineConfiguration | undefined { let commandLineJson: ICommandLineJson | undefined; try { - commandLineJson = JsonFile.loadAndValidate(jsonFilePath, _jsonSchema); + commandLineJson = loadJsonFile(jsonFileLoadCache, jsonFilePath, (json) => { + _jsonSchema.validateObject(json, jsonFilePath); + return json as ICommandLineJson; + }); } catch (e) { if (!FileSystem.isNotExistError(e as Error)) { throw e; @@ -667,39 +676,25 @@ export class CommandLineConfiguration { * Loads the configuration from the specified file and applies any omitted default build * settings. If the file does not exist, then a default instance is returned. * If the file contains errors, then an exception is thrown. + * + * @param jsonFileLoadCache - The cache of a long-lived engine host, which parses the file again only if it changed. */ public static loadFromFileOrDefault( jsonFilePath?: string, - doNotIncludeDefaultBuildCommands?: boolean + doNotIncludeDefaultBuildCommands?: boolean, + jsonFileLoadCache?: JsonFileLoadCache ): CommandLineConfiguration { let commandLineJson: ICommandLineJson | undefined = undefined; if (jsonFilePath) { try { - commandLineJson = JsonFile.load(jsonFilePath); + commandLineJson = loadJsonFile(jsonFileLoadCache, jsonFilePath, (json) => + _prepareRepoCommandLineJson(json as ICommandLineJson, jsonFilePath) + ); } catch (e) { if (!FileSystem.isNotExistError(e as Error)) { throw e; } } - - // merge commands specified in command-line.json and default (re)build settings - // Ensure both build commands are included and preserve any other commands specified - if (commandLineJson?.commands) { - _applyBuildCommandDefaults(commandLineJson); - - _jsonSchema.validateObject(commandLineJson, jsonFilePath); - - // Validate that globalPlugin commands are not used in the repo's command-line.json - for (const { commandKind, name } of commandLineJson.commands) { - if (commandKind === RushConstants.globalPluginCommandKind) { - throw new Error( - `${RushConstants.commandLineFilename} defines a command "${name}" using ` + - `the command kind "${RushConstants.globalPluginCommandKind}". This command kind can only ` + - `be used in command-line.json files provided by Rush plugins.` - ); - } - } - } } return new CommandLineConfiguration(commandLineJson, { doNotIncludeDefaultBuildCommands }); @@ -751,6 +746,31 @@ export class CommandLineConfiguration { } } +function _prepareRepoCommandLineJson( + commandLineJson: ICommandLineJson, + jsonFilePath: string +): ICommandLineJson { + // merge commands specified in command-line.json and default (re)build settings + // Ensure both build commands are included and preserve any other commands specified + if (commandLineJson?.commands) { + _applyBuildCommandDefaults(commandLineJson); + + _jsonSchema.validateObject(commandLineJson, jsonFilePath); + + // Validate that globalPlugin commands are not used in the repo's command-line.json + for (const { commandKind, name } of commandLineJson.commands) { + if (commandKind === RushConstants.globalPluginCommandKind) { + throw new Error( + `${RushConstants.commandLineFilename} defines a command "${name}" using ` + + `the command kind "${RushConstants.globalPluginCommandKind}". This command kind can only ` + + `be used in command-line.json files provided by Rush plugins.` + ); + } + } + } + return commandLineJson; +} + function _applyBuildCommandDefaults(commandLineJson: ICommandLineJson): void { // merge commands specified in command-line.json and default (re)build settings // Ensure both build commands are included and preserve any other commands specified diff --git a/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts b/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts index acef36cc56..82669036fb 100644 --- a/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts +++ b/libraries/rush-lib/src/api/test/PhasedCommandEngine.test.ts @@ -5,11 +5,15 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; +import { FileSystem, JsonFile } from '@rushstack/node-core-library'; import { NoOpTerminalProvider, StringBufferTerminalProvider } from '@rushstack/terminal'; import { PhasedCommandEngine } from '../PhasedCommandEngine'; import { PhasedCommandEngineUsageError } from '../PhasedCommandEngineUsageError'; import { RushConfiguration } from '../RushConfiguration'; +import { EnvironmentConfiguration } from '../EnvironmentConfiguration'; +import { RushCommandLineParser } from '../../cli/RushCommandLineParser'; +import { JsonFileLoadCache } from '../../utilities/JsonFileLoadCache'; const PACKAGE_NAME: string = '@example/rush-example-plugin'; const PLUGIN_NAME: string = 'rush-example-plugin'; @@ -402,4 +406,185 @@ describe(PhasedCommandEngine.name, () => { ); }); }); + + describe('JSON configuration files of parses that share a workspace configuration', () => { + const REPO_COMMAND_LINE_PATH: string = 'common/config/rush/command-line.json'; + const AUTOINSTALLER_PACKAGE_JSON_PATH: string = 'common/autoinstallers/plugins/package.json'; + const PLUGIN_STORE_FOLDER: string = `common/autoinstallers/plugins/rush-plugins/${PACKAGE_NAME}`; + const MANIFEST_PATH: string = `${PLUGIN_STORE_FOLDER}/rush-plugin-manifest.json`; + const PLUGIN_COMMAND_LINE_PATH: string = `${PLUGIN_STORE_FOLDER}/${PLUGIN_NAME}/command-line.json`; + const EXAMPLE_MODE_PARAMETER: object = { + parameterKind: 'choice', + longName: '--example-mode', + description: 'An example mode.', + associatedCommands: ['build'], + alternatives: [ + { name: 'on', description: 'On.' }, + { name: 'off', description: 'Off.' } + ], + defaultValue: 'on' + }; + // A global command whose autoinstaller's package.json Rush reads while it parses any command line. + const GLOBAL_COMMAND_LINE_JSON: object = { + ...REPO_COMMAND_LINE_JSON, + commands: [ + ...(REPO_COMMAND_LINE_JSON as { commands: object[] }).commands, + { + commandKind: 'global', + name: 'example-global', + summary: 'An example.', + shellCommand: 'node example.js', + autoinstallerName: 'plugins' + } + ] + }; + + interface IManifestJson { + plugins: { associatedCommands: string[] }[]; + } + + function createSharedConfigurationRepo(): RushConfiguration { + const folder: string = createTestRepo( + { associatedCommands: [PLUGIN_COMMAND], commandLineJson: COMMAND_SCOPED_COMMAND_LINE_JSON }, + { commandLineJson: GLOBAL_COMMAND_LINE_JSON } + ); + return RushConfiguration.loadFromConfigurationFile(path.join(folder, 'rush.json')); + } + + function editJson(rushConfiguration: RushConfiguration, relativePath: string, edit: (json: T) => void): void { + const filePath: string = path.join(rushConfiguration.rushJsonFolder, relativePath); + const json: T = JSON.parse(fs.readFileSync(filePath, 'utf8')); + edit(json); + fs.writeFileSync(filePath, JSON.stringify(json)); + } + + async function parseAsync( + rushConfiguration: RushConfiguration, + argv: string[] = ['build'] + ): Promise { + return await PhasedCommandEngine.parseAsync({ + argv, + cwd: rushConfiguration.rushJsonFolder, + rushConfiguration, + terminalProvider: new NoOpTerminalProvider() + }); + } + + async function getParseErrorAsync(rushConfiguration: RushConfiguration): Promise { + return await parseAsync(rushConfiguration).then( + () => undefined, + (error: Error) => error + ); + } + + afterEach(() => { + jest.restoreAllMocks(); + }); + + it('sees each change to the repository command-line.json', async () => { + const rushConfiguration: RushConfiguration = createSharedConfigurationRepo(); + await expect(parseAsync(rushConfiguration, ['build', '--example-mode', 'off'])).rejects.toThrow( + '--example-mode' + ); + + editJson<{ parameters?: object[] }>(rushConfiguration, REPO_COMMAND_LINE_PATH, (json) => { + json.parameters = [EXAMPLE_MODE_PARAMETER]; + }); + const offIdentity: string = (await parseAsync(rushConfiguration, ['build', '--example-mode', 'off'])) + .parameterIdentity; + expect((await parseAsync(rushConfiguration)).parameterIdentity).not.toBe(offIdentity); + }); + + it('reports an invalid repository command-line.json for every parse, as a parse without the cache does', async () => { + const rushConfiguration: RushConfiguration = createSharedConfigurationRepo(); + await parseAsync(rushConfiguration); + + editJson<{ unknownSetting?: boolean }>(rushConfiguration, REPO_COMMAND_LINE_PATH, (json) => { + json.unknownSetting = true; + }); + const error: Error | undefined = await getParseErrorAsync(rushConfiguration); + expect(error?.message).toMatch(/command-line\.json/); + expect((await getParseErrorAsync(rushConfiguration))?.message).toBe(error?.message); + await expect(parseBuildAsync(rushConfiguration.rushJsonFolder)).rejects.toThrow(error?.message); + + editJson<{ unknownSetting?: boolean }>(rushConfiguration, REPO_COMMAND_LINE_PATH, (json) => { + delete json.unknownSetting; + }); + await expect(parseAsync(rushConfiguration)).resolves.toBeInstanceOf(PhasedCommandEngine); + }); + + it("sees each change to a plugin's manifest and command-line.json", async () => { + const rushConfiguration: RushConfiguration = createSharedConfigurationRepo(); + await expect(parseAsync(rushConfiguration)).resolves.toBeInstanceOf(PhasedCommandEngine); + + editJson(rushConfiguration, MANIFEST_PATH, (manifest) => { + manifest.plugins[0].associatedCommands.push('build'); + }); + await expect(parseAsync(rushConfiguration)).rejects.toThrow( + `"${PLUGIN_NAME}" (${PACKAGE_NAME}) is associated with "build"` + ); + + editJson(rushConfiguration, MANIFEST_PATH, (manifest) => { + manifest.plugins[0].associatedCommands.pop(); + }); + fs.writeFileSync( + path.join(rushConfiguration.rushJsonFolder, PLUGIN_COMMAND_LINE_PATH), + JSON.stringify(PHASE_SHAPING_COMMAND_LINE_JSON) + ); + await expect(parseAsync(rushConfiguration)).rejects.toThrow( + `"${PLUGIN_NAME}" (${PACKAGE_NAME}) associates "--example-flag" with the "_phase:build" phase` + ); + }); + + it("sees each change to the package.json of a global command's autoinstaller", async () => { + const rushConfiguration: RushConfiguration = createSharedConfigurationRepo(); + await expect(parseAsync(rushConfiguration)).resolves.toBeInstanceOf(PhasedCommandEngine); + + editJson<{ name: string }>(rushConfiguration, AUTOINSTALLER_PACKAGE_JSON_PATH, (packageJson) => { + packageJson.name = 'other'; + }); + await expect(parseAsync(rushConfiguration)).rejects.toThrow( + `specifies an "autoinstallerName" setting, but the package.json file's "name" field is not "plugins"` + ); + }); + + it('reads each file for every parse, but parses it only if it changed', async () => { + const rushConfiguration: RushConfiguration = createSharedConfigurationRepo(); + const filePaths: string[] = [ + REPO_COMMAND_LINE_PATH, + AUTOINSTALLER_PACKAGE_JSON_PATH, + MANIFEST_PATH, + PLUGIN_COMMAND_LINE_PATH + ].map((relativePath) => path.join(rushConfiguration.rushJsonFolder, relativePath)); + const texts: string[] = filePaths.map((filePath) => fs.readFileSync(filePath, 'utf8')); + await parseAsync(rushConfiguration); + + const readFileSpy: jest.SpyInstance = jest.spyOn(FileSystem, 'readFile'); + const loadSpy: jest.SpyInstance = jest.spyOn(JsonFile, 'load'); + const parseStringSpy: jest.SpyInstance = jest.spyOn(JsonFile, 'parseString'); + await parseAsync(rushConfiguration); + + const readFilePaths: Set = new Set(readFileSpy.mock.calls.map(([filePath]) => filePath)); + expect(filePaths.filter((filePath) => !readFilePaths.has(filePath))).toEqual([]); + expect(loadSpy.mock.calls.filter(([filePath]) => filePaths.includes(filePath))).toEqual([]); + expect(parseStringSpy.mock.calls.filter(([text]) => texts.includes(text))).toEqual([]); + }); + + it('does not cache the files for a native command line', async () => { + const rushConfiguration: RushConfiguration = createSharedConfigurationRepo(); + const cacheLoadSpy: jest.SpyInstance = jest.spyOn(JsonFileLoadCache.prototype, 'load'); + const loadSpy: jest.SpyInstance = jest.spyOn(JsonFile, 'load'); + // Engine parses in this process validated the environment, and a native parser must load .env files first. + EnvironmentConfiguration.reset(); + + const parser: RushCommandLineParser = new RushCommandLineParser({ cwd: rushConfiguration.rushJsonFolder }); + expect(parser.getAction('example-global')).toBeDefined(); + expect(cacheLoadSpy).not.toHaveBeenCalled(); + const loadedFilePaths: Set = new Set(loadSpy.mock.calls.map(([filePath]) => filePath)); + expect(loadedFilePaths).toContain(path.join(rushConfiguration.rushJsonFolder, REPO_COMMAND_LINE_PATH)); + expect(loadedFilePaths).toContain( + path.join(rushConfiguration.rushJsonFolder, AUTOINSTALLER_PACKAGE_JSON_PATH) + ); + }); + }); }); diff --git a/libraries/rush-lib/src/cli/RushCommandLineParser.ts b/libraries/rush-lib/src/cli/RushCommandLineParser.ts index 5c6cd6521e..70bbe7d2c8 100644 --- a/libraries/rush-lib/src/cli/RushCommandLineParser.ts +++ b/libraries/rush-lib/src/cli/RushCommandLineParser.ts @@ -71,6 +71,7 @@ import { type IRushSessionReporterOptions, RushSession } from '../pluginFramewor import type { IBuiltInPluginConfiguration } from '../pluginFramework/PluginLoader/BuiltInPluginLoader'; import { InitSubspaceAction } from './actions/InitSubspaceAction'; import { RushAlerts } from '../utilities/RushAlerts'; +import { getEngineJsonFileLoadCache, type JsonFileLoadCache } from '../utilities/JsonFileLoadCache'; import { initializeDotEnv } from '../logic/dotenv'; import { measureAsyncFn } from '../utilities/performance'; import { EnvironmentVariableNames } from '../api/EnvironmentConfiguration'; @@ -187,6 +188,8 @@ export class RushCommandLineParser extends CommandLineParser { readonly #quietParameter: CommandLineFlagParameter; readonly #restrictConsoleOutput: boolean = RushCommandLineParser.shouldRestrictConsoleOutput(); readonly #rushOptions: IRushCommandLineParserOptions; + /** For a parser that serves a long-lived engine host, the cache through which it reads JSON configuration files. */ + readonly #jsonFileLoadCache: JsonFileLoadCache | undefined; readonly #terminalProvider: ITerminalProvider; readonly #terminal: Terminal; readonly #autocreateBuildCommand: boolean; @@ -243,6 +246,9 @@ export class RushCommandLineParser extends CommandLineParser { this.#rushOptions = this.#normalizeOptions(options || {}); const { cwd, alreadyReportedNodeTooNewError, builtInPluginConfigurations, reporter } = this.#rushOptions; + this.#jsonFileLoadCache = this.#rushOptions.engine + ? getEngineJsonFileLoadCache(this.#rushOptions.engine.rushConfiguration) + : undefined; const reporterTerminalProvider: ReporterTerminalProvider | undefined = reporter?.operationStreamEnabled ? new ReporterTerminalProvider() : undefined; @@ -296,7 +302,8 @@ export class RushCommandLineParser extends CommandLineParser { terminal, builtInPluginConfigurations, restrictConsoleOutput: this.#restrictConsoleOutput, - rushGlobalFolder: this.rushGlobalFolder + rushGlobalFolder: this.rushGlobalFolder, + jsonFileLoadCache: this.#jsonFileLoadCache }); if (this.#initializationFailed) { this.#autocreateBuildCommand = true; @@ -583,7 +590,8 @@ export class RushCommandLineParser extends CommandLineParser { const commandLineConfiguration: CommandLineConfiguration = CommandLineConfiguration.loadFromFileOrDefault( commandLineConfigFilePath, - doNotIncludeDefaultBuildCommands + doNotIncludeDefaultBuildCommands, + this.#jsonFileLoadCache ); this.#addCommandLineConfigActions(commandLineConfiguration); } @@ -661,7 +669,8 @@ export class RushCommandLineParser extends CommandLineParser { shellCommand, autoinstallerName, - providedByPlugin + providedByPlugin, + jsonFileLoadCache: this.#jsonFileLoadCache }) ); } diff --git a/libraries/rush-lib/src/cli/scriptActions/GlobalScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/GlobalScriptAction.ts index 421bd5adea..ad541948cc 100644 --- a/libraries/rush-lib/src/cli/scriptActions/GlobalScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/GlobalScriptAction.ts @@ -9,7 +9,6 @@ import type { CommandLineParameter } from '@rushstack/ts-command-line'; import { FileSystem, type IPackageJson, - JsonFile, AlreadyReportedError, } from '@rushstack/node-core-library'; import { Colorize } from '@rushstack/terminal'; @@ -22,6 +21,7 @@ import { Autoinstaller } from '../../logic/Autoinstaller'; import { RushConstants } from '../../logic/RushConstants'; import type { IGlobalCommandConfig, IShellCommandTokenContext } from '../../api/CommandLineConfiguration'; import { measureAsyncFn } from '../../utilities/performance'; +import { type JsonFileLoadCache, loadJsonFile } from '../../utilities/JsonFileLoadCache'; /** * Constructor parameters for GlobalScriptAction. @@ -30,6 +30,8 @@ export interface IGlobalScriptActionOptions extends IBaseScriptActionOptions { public constructor(options: IGlobalScriptActionOptions) { super(options); - const { shellCommand, providedByPlugin, autoinstallerName = '' } = options; + const { shellCommand, providedByPlugin, autoinstallerName = '', jsonFileLoadCache } = options; this.#shellCommand = shellCommand; this.#providedByPlugin = providedByPlugin; this.#autoinstallerName = autoinstallerName; @@ -85,9 +87,13 @@ export class GlobalScriptAction extends BaseScriptAction { ); } - const packageJson: IPackageJson = JsonFile.load(packageJsonPath); + const packageName: string | undefined = loadJsonFile( + jsonFileLoadCache, + packageJsonPath, + (packageJson) => (packageJson as IPackageJson).name + ); - if (packageJson.name !== this.#autoinstallerName) { + if (packageName !== this.#autoinstallerName) { throw new Error( `The custom command "${this.actionName}" specifies an "autoinstallerName" setting,` + ` but the package.json file's "name" field is not "${this.#autoinstallerName}": ` + diff --git a/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts b/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts index 4c26791a62..a5c36b6bec 100644 --- a/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts +++ b/libraries/rush-lib/src/pluginFramework/PluginLoader/PluginLoaderBase.ts @@ -18,6 +18,7 @@ import { CommandLineConfiguration } from '../../api/CommandLineConfiguration'; import type { RushConfiguration } from '../../api/RushConfiguration'; import type { IRushPluginConfigurationBase } from '../../api/RushPluginsConfiguration'; import { RushConstants } from '../../logic/RushConstants'; +import { type JsonFileLoadCache, loadJsonFile } from '../../utilities/JsonFileLoadCache'; import type { IRushPlugin } from '../IRushPlugin'; import { RushSdk } from './RushSdk'; import schemaJson from '../../schemas/rush-plugin-manifest.schema.json'; @@ -42,6 +43,8 @@ export interface IPluginLoaderOptions | undefined; #packageVersionCache: string | undefined; + readonly #jsonFileLoadCache: JsonFileLoadCache | undefined; /** * The folder that should be used for resolving the plugin's NPM package. @@ -65,12 +69,14 @@ export abstract class PluginLoaderBase< public constructor({ pluginConfiguration, rushConfiguration, - terminal + terminal, + jsonFileLoadCache }: IPluginLoaderOptions) { this.packageName = pluginConfiguration.packageName; this.pluginName = pluginConfiguration.pluginName; this._rushConfiguration = rushConfiguration; this._terminal = terminal; + this.#jsonFileLoadCache = jsonFileLoadCache; } public load(): IRushPlugin | undefined { @@ -109,7 +115,7 @@ export abstract class PluginLoaderBase< return undefined; } const commandLineConfiguration: CommandLineConfiguration | undefined = - CommandLineConfiguration.tryLoadFromFile(commandLineJsonFilePath); + CommandLineConfiguration.tryLoadFromFile(commandLineJsonFilePath, this.#jsonFileLoadCache); if (!commandLineConfiguration) { return undefined; } @@ -230,9 +236,13 @@ export abstract class PluginLoaderBase< ); } - const rushPluginManifestJson: IRushPluginManifestJson = JsonFile.loadAndValidate( + const rushPluginManifestJson: IRushPluginManifestJson = loadJsonFile( + this.#jsonFileLoadCache, manifestPath, - PluginLoaderBase._jsonSchema + (json) => { + PluginLoaderBase._jsonSchema.validateObject(json, manifestPath); + return json as IRushPluginManifestJson; + } ); const pluginManifest: IRushPluginManifest | undefined = rushPluginManifestJson.plugins.find( diff --git a/libraries/rush-lib/src/pluginFramework/PluginManager.ts b/libraries/rush-lib/src/pluginFramework/PluginManager.ts index e5c2223cdc..715742f15b 100644 --- a/libraries/rush-lib/src/pluginFramework/PluginManager.ts +++ b/libraries/rush-lib/src/pluginFramework/PluginManager.ts @@ -15,6 +15,7 @@ import { Rush } from '../api/Rush'; import type { RushGlobalFolder } from '../api/RushGlobalFolder'; import { findNodeModulesPackageFolder } from '../utilities/RushLibPathHandoff'; import { rushLibPathHandoff } from '../utilities/SetRushLibPath'; +import type { JsonFileLoadCache } from '../utilities/JsonFileLoadCache'; export interface IPluginManagerOptions { terminal: ITerminal; @@ -23,6 +24,8 @@ export interface IPluginManagerOptions { builtInPluginConfigurations: IBuiltInPluginConfiguration[]; restrictConsoleOutput: boolean; rushGlobalFolder: RushGlobalFolder; + /** The cache of a long-lived engine host, through which plugin loaders read the plugins' JSON files. */ + jsonFileLoadCache?: JsonFileLoadCache; } export interface ICustomCommandLineConfigurationInfo { @@ -114,7 +117,8 @@ export class PluginManager { return new BuiltInPluginLoader({ pluginConfiguration, rushConfiguration: this.#rushConfiguration, - terminal: this.#terminal + terminal: this.#terminal, + jsonFileLoadCache: options.jsonFileLoadCache }); }); @@ -126,7 +130,8 @@ export class PluginManager { rushConfiguration: this.#rushConfiguration, terminal: this.#terminal, restrictConsoleOutput: this.#restrictConsoleOutput, - rushGlobalFolder: this.#rushGlobalFolder + rushGlobalFolder: this.#rushGlobalFolder, + jsonFileLoadCache: options.jsonFileLoadCache }); }); } diff --git a/libraries/rush-lib/src/utilities/JsonFileLoadCache.ts b/libraries/rush-lib/src/utilities/JsonFileLoadCache.ts new file mode 100644 index 0000000000..75814c86a2 --- /dev/null +++ b/libraries/rush-lib/src/utilities/JsonFileLoadCache.ts @@ -0,0 +1,78 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import { FileSystem, JsonFile, type JsonObject } from '@rushstack/node-core-library'; + +import type { RushConfiguration } from '../api/RushConfiguration'; + +interface ICacheEntry { + readonly text: string; + readonly value: unknown; +} + +/** + * Memoizes what a long-lived engine host derives from each JSON configuration file, keyed by the file's text. + * + * @remarks + * The host parses a command line for every request, and each parse loads the same files: command-line.json, each + * plugin's manifest and command-line.json, and the package.json of each autoinstaller that a command names. Parsing + * and validating them costs several times what reading them does. A load through this cache still reads the file, + * so it sees the file as it is now, but it parses the text and derives the value again only if the text differs from + * the text of the file's last successful load. Each load returns its own copy of the value. + */ +export class JsonFileLoadCache { + readonly #entries: Map = new Map(); + + /** + * Returns `deriveValue(JsonFile.load(filePath))`, and throws the errors that it would throw. + * + * @param deriveValue - Validates and transforms the file's contents. Its result must depend only on its argument + * (and on values that never change, such as the file path), and it must not keep a reference to its result. + */ + public load(filePath: string, deriveValue: (json: JsonObject) => T): T { + let text: string; + let value: T; + try { + text = FileSystem.readFile(filePath); + const entry: ICacheEntry | undefined = this.#entries.get(filePath); + if (entry?.text === text) { + return structuredClone(entry.value) as T; + } + value = deriveValue(JsonFile.parseString(text)); + } catch (error) { + if (FileSystem.isNotExistError(error as Error)) { + // JsonFile.load throws this error as it is. + throw error; + } + // Load the file as a caller without this cache does, so that the error and its message are the same. + return deriveValue(JsonFile.load(filePath)); + } + this.#entries.set(filePath, { text, value: structuredClone(value) }); + return value; + } +} + +/** + * Returns `deriveValue(JsonFile.load(filePath))`, through the cache if there is one. + */ +export function loadJsonFile( + cache: JsonFileLoadCache | undefined, + filePath: string, + deriveValue: (json: JsonObject) => T +): T { + return cache ? cache.load(filePath, deriveValue) : deriveValue(JsonFile.load(filePath)); +} + +const cachesByRushConfiguration: WeakMap = new WeakMap(); + +/** + * The cache that every command line which a long-lived engine host parses for one workspace configuration shares. + */ +export function getEngineJsonFileLoadCache(rushConfiguration: RushConfiguration): JsonFileLoadCache { + let cache: JsonFileLoadCache | undefined = cachesByRushConfiguration.get(rushConfiguration); + if (!cache) { + cache = new JsonFileLoadCache(); + cachesByRushConfiguration.set(rushConfiguration, cache); + } + return cache; +} diff --git a/libraries/rush-lib/src/utilities/test/JsonFileLoadCache.test.ts b/libraries/rush-lib/src/utilities/test/JsonFileLoadCache.test.ts new file mode 100644 index 0000000000..e45310d501 --- /dev/null +++ b/libraries/rush-lib/src/utilities/test/JsonFileLoadCache.test.ts @@ -0,0 +1,147 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +import { FileSystem, JsonFile, type JsonObject } from '@rushstack/node-core-library'; + +import type { RushConfiguration } from '../../api/RushConfiguration'; +import { getEngineJsonFileLoadCache, JsonFileLoadCache, loadJsonFile } from '../JsonFileLoadCache'; + +interface IExample { + items: number[]; +} + +function deriveExample(json: JsonObject): IExample { + if (!Array.isArray(json.items)) { + throw new Error('"items" must be an array'); + } + return { items: json.items }; +} + +function getError(fn: () => unknown): Error { + try { + fn(); + } catch (error) { + return error as Error; + } + throw new Error('Expected an error'); +} + +describe(JsonFileLoadCache.name, () => { + let folder: string; + let filePath: string; + beforeEach(() => { + folder = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-json-file-load-cache-')); + filePath = path.join(folder, 'example.json'); + }); + afterEach(() => { + fs.rmSync(folder, { recursive: true, force: true }); + jest.restoreAllMocks(); + }); + + it('derives the value once while the text is unchanged, and returns a copy for each load', () => { + fs.writeFileSync(filePath, '{ "items": [1, 2] }'); + const cache: JsonFileLoadCache = new JsonFileLoadCache(); + const derive: jest.Mock = jest.fn(deriveExample); + + const first: IExample = cache.load(filePath, derive); + first.items.push(3); + const second: IExample = cache.load(filePath, derive); + second.items.push(4); + + expect(cache.load(filePath, derive)).toEqual({ items: [1, 2] }); + expect(second).not.toBe(first); + expect(derive).toHaveBeenCalledTimes(1); + }); + + it('reads the file for every load, and derives the value again when the text changes', () => { + fs.writeFileSync(filePath, '{ "items": [1] }'); + const cache: JsonFileLoadCache = new JsonFileLoadCache(); + const derive: jest.Mock = jest.fn(deriveExample); + const { mtime } = fs.statSync(filePath); + + expect(cache.load(filePath, derive)).toEqual({ items: [1] }); + // The same size and modification time, which a cache keyed by file stamps could miss. + fs.writeFileSync(filePath, '{ "items": [2] }'); + fs.utimesSync(filePath, mtime, mtime); + expect(cache.load(filePath, derive)).toEqual({ items: [2] }); + expect(derive).toHaveBeenCalledTimes(2); + }); + + it('throws the errors that JsonFile.load and the derivation throw, and caches no failed load', () => { + const cache: JsonFileLoadCache = new JsonFileLoadCache(); + + const expectedMissingError: Error = getError(() => JsonFile.load(filePath)); + const readFileSpy: jest.SpyInstance = jest.spyOn(FileSystem, 'readFile'); + const missingError: Error = getError(() => cache.load(filePath, deriveExample)); + expect(FileSystem.isNotExistError(missingError)).toBe(true); + expect(missingError.message).toBe(expectedMissingError.message); + // The cache does not read a missing file twice. + expect(readFileSpy).toHaveBeenCalledTimes(1); + + const expectedFolderError: Error = getError(() => JsonFile.load(folder)); + expect(expectedFolderError.message).toContain('Error reading'); + expect(() => cache.load(folder, deriveExample)).toThrow(expectedFolderError.message); + + fs.writeFileSync(filePath, '{ "items": [1] '); + const expectedParseError: Error = getError(() => JsonFile.load(filePath)); + expect(expectedParseError.message).toContain('Error reading'); + expect(() => cache.load(filePath, deriveExample)).toThrow(expectedParseError.message); + + fs.writeFileSync(filePath, '{ "items": 1 }'); + for (let i: number = 0; i < 2; i++) { + expect(() => cache.load(filePath, deriveExample)).toThrow('"items" must be an array'); + } + + fs.writeFileSync(filePath, '{ "items": [1] }'); + expect(cache.load(filePath, deriveExample)).toEqual({ items: [1] }); + }); + + it('keeps an entry for each file', () => { + const otherFilePath: string = path.join(folder, 'other.json'); + fs.writeFileSync(filePath, '{ "items": [1] }'); + fs.writeFileSync(otherFilePath, '{ "items": [2] }'); + const cache: JsonFileLoadCache = new JsonFileLoadCache(); + const derive: jest.Mock = jest.fn(deriveExample); + + for (let i: number = 0; i < 2; i++) { + expect(cache.load(filePath, derive)).toEqual({ items: [1] }); + expect(cache.load(otherFilePath, derive)).toEqual({ items: [2] }); + } + expect(derive).toHaveBeenCalledTimes(2); + }); +}); + +describe(loadJsonFile.name, () => { + it('loads and derives the value for every call without a cache', () => { + const folder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-json-file-load-cache-')); + try { + const filePath: string = path.join(folder, 'example.json'); + fs.writeFileSync(filePath, '{ "items": [1] }'); + const derive: jest.Mock = jest.fn(deriveExample); + expect(loadJsonFile(undefined, filePath, derive)).toEqual({ items: [1] }); + expect(loadJsonFile(undefined, filePath, derive)).toEqual({ items: [1] }); + expect(derive).toHaveBeenCalledTimes(2); + + const cache: JsonFileLoadCache = new JsonFileLoadCache(); + expect(loadJsonFile(cache, filePath, derive)).toEqual({ items: [1] }); + expect(loadJsonFile(cache, filePath, derive)).toEqual({ items: [1] }); + expect(derive).toHaveBeenCalledTimes(3); + } finally { + fs.rmSync(folder, { recursive: true, force: true }); + } + }); +}); + +describe(getEngineJsonFileLoadCache.name, () => { + it('shares one cache for each workspace configuration', () => { + const configuration: RushConfiguration = {} as RushConfiguration; + const otherConfiguration: RushConfiguration = {} as RushConfiguration; + const cache: JsonFileLoadCache = getEngineJsonFileLoadCache(configuration); + expect(getEngineJsonFileLoadCache(configuration)).toBe(cache); + expect(getEngineJsonFileLoadCache(otherConfiguration)).not.toBe(cache); + }); +}); From 810f077e8412700ec7727cfc28ab26863a5eabb5 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 05:04:02 +0000 Subject: [PATCH 081/265] [package-deps-hash] Keep the untracked cache and fsmonitor token when RepoStateCache copies the index again Swarm integration step 66; original commit 4c28b44283 (merge of swarm/r07-t169 at effeec5b44). Scope: task 169. Brings r07's task 169 (board 3170): when the copied index records the same paths, the new copy keeps the old copy's untracked cache and fsmonitor token. The daemon's first git status after git add, restore --staged, reset --hard or checkout -- drops from 0.94-1.44 s to 0.28-0.45 s. Second agents: t04 board 3334 and m01 board 3440. s17 batch D2, item 4 of 6. Gate: ch01 GATE OK board 3514 (tree d362c9d055) Commits folded into this step (1): - effeec5b44 Keep the untracked cache and file system monitor state when RepoStateCache copies the index again Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...dex-caches-on-recopy_2026-09-29-02-37.json | 11 + .../package-deps-hash/src/GitIndexFile.ts | 492 +++++++++++++- .../package-deps-hash/src/RepoStateCache.ts | 85 ++- .../src/test/FsmonitorHook.ts | 45 ++ .../src/test/GitIndexFile.test.ts | 632 +++++++++++++++++- .../src/test/RepoStateCache.test.ts | 248 +++++++ 6 files changed, 1448 insertions(+), 65 deletions(-) create mode 100644 common/changes/@rushstack/package-deps-hash/keep-index-caches-on-recopy_2026-09-29-02-37.json create mode 100644 libraries/package-deps-hash/src/test/FsmonitorHook.ts diff --git a/common/changes/@rushstack/package-deps-hash/keep-index-caches-on-recopy_2026-09-29-02-37.json b/common/changes/@rushstack/package-deps-hash/keep-index-caches-on-recopy_2026-09-29-02-37.json new file mode 100644 index 0000000000..bd37d3432c --- /dev/null +++ b/common/changes/@rushstack/package-deps-hash/keep-index-caches-on-recopy_2026-09-29-02-37.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/package-deps-hash", + "comment": "When `RepoStateCache` copies the Git index again and the index records the same paths as the previous copy, for example after `git add`, the new copy keeps the untracked cache and the file system monitor's state of the previous copy, so that `git status` doesn't read every folder and file again.", + "type": "patch" + } + ], + "packageName": "@rushstack/package-deps-hash", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/package-deps-hash/src/GitIndexFile.ts b/libraries/package-deps-hash/src/GitIndexFile.ts index 4f600b9ce7..f5826d8d4c 100644 --- a/libraries/package-deps-hash/src/GitIndexFile.ts +++ b/libraries/package-deps-hash/src/GitIndexFile.ts @@ -4,21 +4,58 @@ // The index format packs flags into bits /* eslint-disable no-bitwise */ -import { createHash } from 'node:crypto'; +import { createHash, type Hash } from 'node:crypto'; // The format is described in https://git-scm.com/docs/index-format const INDEX_SIGNATURE: string = 'DIRC'; const HEADER_LENGTH: number = 12; +const MAX_INDEX_LENGTH: number = 2 ** 32 - 1; // ctime, mtime, dev, ino, mode, uid, gid and size, 4 bytes each except the 8-byte times const ENTRY_STAT_LENGTH: number = 40; const ENTRY_MODE_OFFSET: number = 24; const ENTRY_MODE_LENGTH: number = 4; +// The object type bits of the mode: a regular file, a symbolic link or a gitlink +const ENTRY_MODE_TYPE_SHIFT: number = 12; const ENTRY_SIZE_OFFSET: number = 36; const ENTRY_SIZE_LENGTH: number = 4; const ENTRY_FLAGS_LENGTH: number = 2; const ENTRY_EXTENDED_FLAG: number = 0x4000; +const ENTRY_STAGE_MASK: number = 0x3000; const EXTENSION_HEADER_LENGTH: number = 8; const SPLIT_INDEX_EXTENSION_SIGNATURE: string = 'link'; +const UNTRACKED_CACHE_EXTENSION_SIGNATURE: string = 'UNTR'; +const FSMONITOR_EXTENSION_SIGNATURE: string = 'FSMN'; +// Git ignores an extension whose signature starts with an uppercase letter if it doesn't understand it +const FIRST_OPTIONAL_SIGNATURE_CHARACTER_CODE: number = 0x41; +const LAST_OPTIONAL_SIGNATURE_CHARACTER_CODE: number = 0x5a; +// A new copy of the index takes the untracked cache and the file system monitor's state from the previous copy. It +// drops "EOIE", which describes the extensions that follow the entries, since those change, and so also "IEOT", which +// lets Git read the entries with several threads, since Git only finds it through "EOIE". +const REPLACED_EXTENSION_SIGNATURES: ReadonlySet = new Set([ + 'EOIE', + 'IEOT', + UNTRACKED_CACHE_EXTENSION_SIGNATURE, + FSMONITOR_EXTENSION_SIGNATURE +]); +const FSMONITOR_VERSION_1: number = 1; +const FSMONITOR_VERSION_2: number = 2; +const FSMONITOR_VERSION_LENGTH: number = 4; +// Version 1 identifies the state of the file system monitor with a time, and version 2 with a NUL-terminated token +const FSMONITOR_VERSION_1_TIME_LENGTH: number = 8; +const FSMONITOR_BITMAP_SIZE_LENGTH: number = 4; +const SHA256_OBJECT_ID_LENGTH: number = 32; + +// An EWAH bitmap is a sequence of 64-bit words, stored as two 32-bit halves in big-endian order. Each "marker" word +// has the value of a run of identical words (bit 0), the length of the run (bits 1-32) and the number of literal +// words that follow the marker (bits 33-63). See ewah/ewok_rlw.h in the Git source code. +const EWAH_HEADER_LENGTH: number = 8; +const EWAH_WORD_LENGTH: number = 8; +const EWAH_HALF_WORD_LENGTH: number = 4; +const EWAH_MARKER_POSITION_LENGTH: number = 4; +const BITS_PER_EWAH_WORD: number = 64; +const BITS_PER_EWAH_HALF_WORD: number = 32; +const MAX_EWAH_RUN_LENGTH: number = 2 ** 32 - 1; +const MAX_EWAH_LITERAL_WORD_COUNT: number = 2 ** 31 - 1; /** * A summary of a Git index file. @@ -43,6 +80,47 @@ export interface IGitIndexSummary { readonly isSplit: boolean; } +/** + * The location of an extension in a Git index file. + */ +export interface IGitIndexExtension { + readonly signature: string; + /** + * The offset of the extension's header. + */ + readonly start: number; + /** + * The offset after the extension's data. + */ + readonly end: number; +} + +/** + * The locations of the parts of a Git index file. + */ +export interface IGitIndexLayout { + readonly version: number; + readonly entryCount: number; + /** + * The offset of each entry, followed by the offset after the last entry. + */ + readonly entryOffsets: Uint32Array; + /** + * The offset of each entry's path. In a version 4 index, the path starts with the number of characters to remove + * from the end of the previous entry's path. + */ + readonly pathOffsets: Uint32Array; + /** + * The offset of the NUL that ends each entry's path. + */ + readonly pathEndOffsets: Uint32Array; + readonly extensions: ReadonlyArray; + /** + * The offset of the checksum, which ends the file. + */ + readonly checksumOffset: number; +} + /** * Returns the number of entries that the header of a Git index file declares, or `undefined` if the data doesn't * start with the header of a Git index file. @@ -59,12 +137,12 @@ export function tryGetGitIndexEntryCount(header: Buffer): number | undefined { } /** - * Summarizes the content of a Git index file of version 2, 3 or 4. + * Locates the entries and extensions of a Git index file of version 2, 3 or 4. * * @param content - The content of the index file * @param objectIdLength - The length of an object ID in bytes: 20 for SHA-1, or 32 for SHA-256 */ -export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGitIndexSummary { +export function parseGitIndexLayout(content: Buffer, objectIdLength: number): IGitIndexLayout { const entryCount: number | undefined = tryGetGitIndexEntryCount(content); if (entryCount === undefined) { throw new Error('The file is not a Git index'); @@ -75,6 +153,11 @@ export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGit throw new Error(`Unsupported Git index version ${version}`); } + // The offsets are stored in 32 bits + if (content.length > MAX_INDEX_LENGTH) { + throw new Error('The Git index is too large'); + } + const checksumOffset: number = content.length - objectIdLength; // Each entry has at least its file system data, object ID and flags, and a NUL after its path if ( @@ -84,20 +167,17 @@ export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGit throw new Error('The Git index ends within an entry'); } - // A copy of the entries in which the file system data of each entry is cleared, except for the mode - const entries: Buffer = Buffer.from(content.subarray(0, checksumOffset)); - const sizes: Buffer = Buffer.alloc(entryCount * ENTRY_SIZE_LENGTH); + const entryOffsets: Uint32Array = new Uint32Array(entryCount + 1); + const pathOffsets: Uint32Array = new Uint32Array(entryCount); + const pathEndOffsets: Uint32Array = new Uint32Array(entryCount); let offset: number = HEADER_LENGTH; for (let i: number = 0; i < entryCount; i++) { + entryOffsets[i] = offset; const flagsOffset: number = offset + ENTRY_STAT_LENGTH + objectIdLength; if (flagsOffset + ENTRY_FLAGS_LENGTH > checksumOffset) { throw new Error('The Git index ends within an entry'); } - sizes.writeUInt32BE(content.readUInt32BE(offset + ENTRY_SIZE_OFFSET), i * ENTRY_SIZE_LENGTH); - entries.fill(0, offset, offset + ENTRY_MODE_OFFSET); - entries.fill(0, offset + ENTRY_MODE_OFFSET + ENTRY_MODE_LENGTH, offset + ENTRY_STAT_LENGTH); - const flags: number = content.readUInt16BE(flagsOffset); let pathOffset: number = flagsOffset + ENTRY_FLAGS_LENGTH; if (flags & ENTRY_EXTENDED_FLAG) { @@ -105,6 +185,7 @@ export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGit pathOffset += ENTRY_FLAGS_LENGTH; } + pathOffsets[i] = pathOffset; if (version === 4) { // The path is prefix-compressed: a variable-length integer, then the rest of the path and a NUL while (pathOffset < checksumOffset && content[pathOffset] & 0x80) { @@ -118,30 +199,399 @@ export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGit throw new Error('The Git index ends within an entry'); } + pathEndOffsets[i] = pathEnd; // Before version 4, each entry is padded with 1-8 NULs to a multiple of 8 bytes offset = version === 4 ? pathEnd + 1 : offset + ((pathEnd - offset + 8) & ~7); } - const entriesDigest: string = createHash('sha1').update(entries.subarray(4, offset)).digest('hex'); - const sizesDigest: string = createHash('sha1').update(sizes).digest('hex'); - - let isSplit: boolean = false; + entryOffsets[entryCount] = offset; + const extensions: IGitIndexExtension[] = []; while (offset + EXTENSION_HEADER_LENGTH <= checksumOffset) { - const signature: string = content.toString('latin1', offset, offset + 4); - if (signature === SPLIT_INDEX_EXTENSION_SIGNATURE) { - isSplit = true; - } + const start: number = offset; offset += EXTENSION_HEADER_LENGTH + content.readUInt32BE(offset + 4); + extensions.push({ signature: content.toString('latin1', start, start + 4), start, end: offset }); } if (offset !== checksumOffset) { throw new Error('The extensions of the Git index are malformed'); } + return { version, entryCount, entryOffsets, pathOffsets, pathEndOffsets, extensions, checksumOffset }; +} + +/** + * Summarizes the content of a Git index file of version 2, 3 or 4. + * + * @param content - The content of the index file + * @param objectIdLength - The length of an object ID in bytes: 20 for SHA-1, or 32 for SHA-256 + */ +export function summarizeGitIndex(content: Buffer, objectIdLength: number): IGitIndexSummary { + const { entryCount, entryOffsets, extensions }: IGitIndexLayout = parseGitIndexLayout( + content, + objectIdLength + ); + const entriesEnd: number = entryOffsets[entryCount]; + // A copy of the entries in which the file system data of each entry is cleared, except for the mode + const entries: Buffer = Buffer.from(content.subarray(0, entriesEnd)); + const sizes: Buffer = Buffer.alloc(entryCount * ENTRY_SIZE_LENGTH); + for (let i: number = 0; i < entryCount; i++) { + const offset: number = entryOffsets[i]; + sizes.writeUInt32BE(content.readUInt32BE(offset + ENTRY_SIZE_OFFSET), i * ENTRY_SIZE_LENGTH); + entries.fill(0, offset, offset + ENTRY_MODE_OFFSET); + entries.fill(0, offset + ENTRY_MODE_OFFSET + ENTRY_MODE_LENGTH, offset + ENTRY_STAT_LENGTH); + } + return { entryCount, - entriesDigest, - sizesDigest, - isSplit + entriesDigest: createHash('sha1').update(entries.subarray(4, entriesEnd)).digest('hex'), + sizesDigest: createHash('sha1').update(sizes).digest('hex'), + isSplit: extensions.some( + ({ signature }: IGitIndexExtension) => signature === SPLIT_INDEX_EXTENSION_SIGNATURE + ) }; } + +/** + * Builds a new copy of a Git index that keeps the untracked cache and the state of the file system monitor of the + * previous copy, so that `git status` doesn't examine the folders and files that didn't change since it last + * examined the previous copy. The new copy has the header, entries and other extensions of the index. + * + * @remarks + * The untracked cache lists the untracked files in each folder, which depends on which paths the index records. + * So the copy keeps it only if the index records the same paths, with the same stages and object types, as the + * previous copy. The file system monitor's state says which entries Git needn't examine, because they didn't + * change since the monitor's token: the copy marks the entries that differ from those in the previous copy as + * changed, and keeps the token of the previous copy, which matches its untracked cache. + * + * @param content - The content of the index + * @param previousContent - The content of the previous copy of the index + * @param objectIdLength - The length of an object ID in bytes: 20 for SHA-1, or 32 for SHA-256 + * @returns The content of the new copy, or `undefined` if it can't keep the state of the previous copy: when the + * index and the previous copy record different paths, have different versions, or have an extension that Git + * requires to understand, such as that of a split or sparse index, or when the previous copy has no untracked + * cache. + */ +export function tryCarryOverGitIndexCaches( + content: Buffer, + previousContent: Buffer, + objectIdLength: number +): Buffer | undefined { + const layout: IGitIndexLayout = parseGitIndexLayout(content, objectIdLength); + const previousLayout: IGitIndexLayout = parseGitIndexLayout(previousContent, objectIdLength); + const untrackedCache: IGitIndexExtension | undefined = findExtension( + previousLayout, + UNTRACKED_CACHE_EXTENSION_SIGNATURE + ); + if ( + !untrackedCache || + layout.version !== previousLayout.version || + layout.entryCount !== previousLayout.entryCount || + hasRequiredExtension(layout) || + hasRequiredExtension(previousLayout) + ) { + return undefined; + } + + const changedEntries: Uint8Array | undefined = tryFindChangedEntries( + content, + layout, + previousContent, + previousLayout, + objectIdLength + ); + if (!changedEntries) { + return undefined; + } + + const parts: Buffer[] = [content.subarray(0, layout.entryOffsets[layout.entryCount])]; + for (const { signature, start, end } of layout.extensions) { + if (!REPLACED_EXTENSION_SIGNATURES.has(signature)) { + parts.push(content.subarray(start, end)); + } + } + + parts.push(previousContent.subarray(untrackedCache.start, untrackedCache.end)); + const fsmonitor: IGitIndexExtension | undefined = findExtension( + previousLayout, + FSMONITOR_EXTENSION_SIGNATURE + ); + if (fsmonitor) { + const fsmonitorExtension: Buffer | undefined = tryUpdateFsmonitorExtension( + previousContent.subarray(fsmonitor.start, fsmonitor.end), + changedEntries + ); + if (!fsmonitorExtension) { + return undefined; + } + + parts.push(fsmonitorExtension); + } + + // With "index.skipHash", Git writes zeros instead of the checksum + const checksum: Buffer = Buffer.alloc(objectIdLength); + if (content.subarray(layout.checksumOffset).some((byte: number) => byte !== 0)) { + const hash: Hash = createHash(objectIdLength === SHA256_OBJECT_ID_LENGTH ? 'sha256' : 'sha1'); + for (const part of parts) { + hash.update(part); + } + + hash.digest().copy(checksum); + } + + parts.push(checksum); + return Buffer.concat(parts); +} + +function findExtension(layout: IGitIndexLayout, signature: string): IGitIndexExtension | undefined { + return layout.extensions.find((extension: IGitIndexExtension) => extension.signature === signature); +} + +function hasRequiredExtension(layout: IGitIndexLayout): boolean { + return layout.extensions.some(({ signature }: IGitIndexExtension) => { + const characterCode: number = signature.charCodeAt(0); + return ( + characterCode < FIRST_OPTIONAL_SIGNATURE_CHARACTER_CODE || + characterCode > LAST_OPTIONAL_SIGNATURE_CHARACTER_CODE + ); + }); +} + +/** + * Returns a flag for each entry, which is 1 if the entry differs from the one in the previous copy, or `undefined` + * if an entry has a different path, stage or object type. + */ +function tryFindChangedEntries( + content: Buffer, + layout: IGitIndexLayout, + previousContent: Buffer, + previousLayout: IGitIndexLayout, + objectIdLength: number +): Uint8Array | undefined { + const changedEntries: Uint8Array = new Uint8Array(layout.entryCount); + for (let i: number = 0; i < layout.entryCount; i++) { + const start: number = layout.entryOffsets[i]; + const previousStart: number = previousLayout.entryOffsets[i]; + if ( + content.compare( + previousContent, + previousStart, + previousLayout.entryOffsets[i + 1], + start, + layout.entryOffsets[i + 1] + ) === 0 + ) { + continue; + } + + // All previous paths are the same, so in a version 4 index, the same path is compressed the same way + const flagsOffset: number = ENTRY_STAT_LENGTH + objectIdLength; + if ( + (content.readUInt16BE(start + flagsOffset) & ENTRY_STAGE_MASK) !== + (previousContent.readUInt16BE(previousStart + flagsOffset) & ENTRY_STAGE_MASK) || + content.readUInt32BE(start + ENTRY_MODE_OFFSET) >>> ENTRY_MODE_TYPE_SHIFT !== + previousContent.readUInt32BE(previousStart + ENTRY_MODE_OFFSET) >>> ENTRY_MODE_TYPE_SHIFT || + content.compare( + previousContent, + previousLayout.pathOffsets[i], + previousLayout.pathEndOffsets[i], + layout.pathOffsets[i], + layout.pathEndOffsets[i] + ) !== 0 + ) { + return undefined; + } + + changedEntries[i] = 1; + } + + return changedEntries; +} + +/** + * Builds a file system monitor extension with the token of the given one, in which the changed entries are also + * marked as changed. Returns `undefined` if the extension is malformed. + */ +function tryUpdateFsmonitorExtension(extension: Buffer, changedEntries: Uint8Array): Buffer | undefined { + const data: Buffer = extension.subarray(EXTENSION_HEADER_LENGTH); + if (data.length < FSMONITOR_VERSION_LENGTH) { + return undefined; + } + + let bitmapSizeOffset: number; + switch (data.readUInt32BE(0)) { + case FSMONITOR_VERSION_1: + bitmapSizeOffset = FSMONITOR_VERSION_LENGTH + FSMONITOR_VERSION_1_TIME_LENGTH; + break; + case FSMONITOR_VERSION_2: + bitmapSizeOffset = data.indexOf(0, FSMONITOR_VERSION_LENGTH) + 1; + if (bitmapSizeOffset === 0) { + return undefined; + } + break; + default: + return undefined; + } + + const bitmapOffset: number = bitmapSizeOffset + FSMONITOR_BITMAP_SIZE_LENGTH; + if (bitmapOffset > data.length) { + return undefined; + } + + const bitmapEnd: number = bitmapOffset + data.readUInt32BE(bitmapSizeOffset); + const changedEntriesOfPreviousCopy: Uint8Array | undefined = + bitmapEnd <= data.length + ? tryReadEwahBitmap(data.subarray(bitmapOffset, bitmapEnd), changedEntries.length) + : undefined; + if (!changedEntriesOfPreviousCopy) { + return undefined; + } + + for (let i: number = 0; i < changedEntries.length; i++) { + changedEntriesOfPreviousCopy[i] |= changedEntries[i]; + } + + const bitmap: Buffer = writeEwahBitmap(changedEntriesOfPreviousCopy); + const header: Buffer = Buffer.from(extension.subarray(0, EXTENSION_HEADER_LENGTH + bitmapSizeOffset)); + header.writeUInt32BE(bitmapOffset + bitmap.length, 4); + const bitmapSize: Buffer = Buffer.alloc(FSMONITOR_BITMAP_SIZE_LENGTH); + bitmapSize.writeUInt32BE(bitmap.length); + return Buffer.concat([header, bitmapSize, bitmap]); +} + +/** + * Reads an EWAH bitmap, as Git serializes it, into a flag for each bit. Returns `undefined` if the bitmap is + * malformed, or has a bit at or beyond the given count. + */ +export function tryReadEwahBitmap(data: Buffer, bitCount: number): Uint8Array | undefined { + if (data.length < EWAH_HEADER_LENGTH) { + return undefined; + } + + const bitSize: number = data.readUInt32BE(0); + const wordCount: number = data.readUInt32BE(4); + if ( + bitSize > bitCount || + data.length !== EWAH_HEADER_LENGTH + wordCount * EWAH_WORD_LENGTH + EWAH_MARKER_POSITION_LENGTH + ) { + return undefined; + } + + const bits: Uint8Array = new Uint8Array(bitCount); + let position: number = 0; + let wordIndex: number = 0; + while (wordIndex < wordCount) { + const markerOffset: number = EWAH_HEADER_LENGTH + wordIndex * EWAH_WORD_LENGTH; + const markerHigh: number = data.readUInt32BE(markerOffset); + const markerLow: number = data.readUInt32BE(markerOffset + EWAH_HALF_WORD_LENGTH); + const runBitCount: number = + ((markerLow >>> 1) + (markerHigh & 1) * 2 ** (BITS_PER_EWAH_HALF_WORD - 1)) * BITS_PER_EWAH_WORD; + const literalWordCount: number = markerHigh >>> 1; + if (markerLow & 1) { + if (position + runBitCount > bitSize) { + return undefined; + } + + bits.fill(1, position, position + runBitCount); + } + + position += runBitCount; + wordIndex++; + if (wordIndex + literalWordCount > wordCount) { + return undefined; + } + + for (let i: number = 0; i < literalWordCount; i++, wordIndex++) { + const wordOffset: number = EWAH_HEADER_LENGTH + wordIndex * EWAH_WORD_LENGTH; + if ( + !trySetBits(bits, data.readUInt32BE(wordOffset + EWAH_HALF_WORD_LENGTH), position, bitSize) || + !trySetBits(bits, data.readUInt32BE(wordOffset), position + BITS_PER_EWAH_HALF_WORD, bitSize) + ) { + return undefined; + } + + position += BITS_PER_EWAH_WORD; + } + } + + return bits; +} + +function trySetBits(bits: Uint8Array, halfWord: number, position: number, bitSize: number): boolean { + for (let bit: number = position; halfWord !== 0; bit++, halfWord >>>= 1) { + if (halfWord & 1) { + if (bit >= bitSize) { + return false; + } + + bits[bit] = 1; + } + } + + return true; +} + +/** + * Serializes a flag for each bit as an EWAH bitmap, as Git does. The bitmap ends at the last bit that is set. + */ +export function writeEwahBitmap(bits: Uint8Array): Buffer { + const bitSize: number = bits.lastIndexOf(1) + 1; + const literalWordCount: number = Math.ceil(bitSize / BITS_PER_EWAH_WORD); + const lowHalves: Uint32Array = new Uint32Array(literalWordCount); + const highHalves: Uint32Array = new Uint32Array(literalWordCount); + for (let bit: number = 0; bit < bitSize; bit++) { + if (bits[bit]) { + const halves: Uint32Array = bit % BITS_PER_EWAH_WORD < BITS_PER_EWAH_HALF_WORD ? lowHalves : highHalves; + halves[Math.floor(bit / BITS_PER_EWAH_WORD)] |= 1 << bit % BITS_PER_EWAH_HALF_WORD; + } + } + + // Each marker word is followed by the literal words that it counts. Only runs of words that are 0 are + // compressed, so the bitmap has no bit beyond its size. An empty bitmap is one marker word that is 0. + const words: number[] = []; + let lastMarkerIndex: number = 0; + let wordIndex: number = 0; + do { + const runStart: number = wordIndex; + while ( + wordIndex < literalWordCount && + lowHalves[wordIndex] === 0 && + highHalves[wordIndex] === 0 && + wordIndex - runStart < MAX_EWAH_RUN_LENGTH + ) { + wordIndex++; + } + + const runLength: number = wordIndex - runStart; + const literalStart: number = wordIndex; + while ( + wordIndex < literalWordCount && + (lowHalves[wordIndex] !== 0 || highHalves[wordIndex] !== 0) && + wordIndex - literalStart < MAX_EWAH_LITERAL_WORD_COUNT + ) { + wordIndex++; + } + + lastMarkerIndex = words.length / 2; + const runLengthHigh: number = Math.floor(runLength / 2 ** (BITS_PER_EWAH_HALF_WORD - 1)); + words.push( + ((wordIndex - literalStart) * 2 + runLengthHigh) >>> 0, + ((runLength % 2 ** (BITS_PER_EWAH_HALF_WORD - 1)) * 2) >>> 0 + ); + for (let i: number = literalStart; i < wordIndex; i++) { + words.push(highHalves[i], lowHalves[i]); + } + } while (wordIndex < literalWordCount); + + const wordCount: number = words.length / 2; + const data: Buffer = Buffer.alloc( + EWAH_HEADER_LENGTH + wordCount * EWAH_WORD_LENGTH + EWAH_MARKER_POSITION_LENGTH + ); + data.writeUInt32BE(bitSize, 0); + data.writeUInt32BE(wordCount, 4); + for (let i: number = 0; i < words.length; i++) { + data.writeUInt32BE(words[i], EWAH_HEADER_LENGTH + i * EWAH_HALF_WORD_LENGTH); + } + + data.writeUInt32BE(lastMarkerIndex, EWAH_HEADER_LENGTH + wordCount * EWAH_WORD_LENGTH); + return data; +} diff --git a/libraries/package-deps-hash/src/RepoStateCache.ts b/libraries/package-deps-hash/src/RepoStateCache.ts index 2ff67fd0b3..987422c033 100644 --- a/libraries/package-deps-hash/src/RepoStateCache.ts +++ b/libraries/package-deps-hash/src/RepoStateCache.ts @@ -8,7 +8,12 @@ import * as path from 'node:path'; import { FileSystem } from '@rushstack/node-core-library/lib/FileSystem'; import { getFileStamp, getSettledBeforeNs, isFileStatSettled } from './FileStamp'; -import { type IGitIndexSummary, summarizeGitIndex, tryGetGitIndexEntryCount } from './GitIndexFile'; +import { + type IGitIndexSummary, + summarizeGitIndex, + tryCarryOverGitIndexCaches, + tryGetGitIndexEntryCount +} from './GitIndexFile'; import { classifyLocallyModifiedFiles, getCleanGitEnvironment, @@ -105,6 +110,14 @@ interface ITree { readonly state: IGitTreeState; } +interface ICarriedOverCopy { + readonly content: Buffer; + /** + * The modification time of the previous copy. + */ + readonly previousTimeNs: bigint; +} + interface IFileHash { readonly stamp: string; readonly hash: string; @@ -130,7 +143,10 @@ function noop(): void {} * examines every file each time. This class keeps a private copy of the index that `git status` updates, so that * each call only examines the files that changed since the previous call. It copies the index again when the * files that the index records, or the sizes that it records for them, change, but not when Git merely refreshes - * the index. While the files that the index records don't change, it also reuses the list of files in the index. + * the index. While the index records the same paths, a new copy keeps the untracked cache and the file system + * monitor's state of the previous copy, so that `git status` doesn't examine every folder and file again after + * `git add`, for example. While the files that the index records don't change, it also reuses the list of files in + * the index. * It reuses the hash of a file while the identity, size and times of the file and of the `.gitattributes` files in * the folders that contain it don't change, and returns the same state as the previous call if nothing changed. * @@ -514,7 +530,16 @@ export class RepoStateCache { this.#tree = undefined; } - return await this.#writePrivateIndexAsync(content, stats, summary, filterKey, isRealIndexStampSettled); + return await this.#writePrivateIndexAsync( + content, + stats, + summary, + filterKey, + isRealIndexStampSettled, + // Git updated the current copy under the current configuration, and for the same filter + isCopyCurrent ? privateIndex : undefined, + gitPaths.objectIdLength + ); } finally { await handle.close(); } @@ -525,18 +550,31 @@ export class RepoStateCache { stats: fs.BigIntStats, summary: IGitIndexSummary, filterKey: string, - isRealIndexStampSettled: boolean + isRealIndexStampSettled: boolean, + previousIndex: IPrivateIndex | undefined, + objectIdLength: number ): Promise { this.#privateIndex = undefined; + // If the index records the same paths as the previous copy, the new copy keeps the untracked cache and the file + // system monitor's state of the previous copy, so that "git status" doesn't examine every folder again + const carriedOverCopy: ICarriedOverCopy | undefined = + previousIndex && (await tryCarryOverCachesAsync(previousIndex.path, content, objectIdLength)); const folderPath: string = this.#getPrivateFolderPath(); const indexPath: string = path.join(folderPath, PRIVATE_INDEX_NAME); const temporaryPath: string = `${indexPath}.new`; - await fs.promises.writeFile(temporaryPath, content); + await fs.promises.writeFile(temporaryPath, carriedOverCopy?.content ?? content); // Git doesn't trust the recorded times and size of a file that changed in the same second as the index was // written, because the file may have changed again after it was recorded. Make the copy older than the index, - // so that Git doesn't trust any file in the copy that it wouldn't trust in the index. - const timeInSeconds: number = Number(stats.mtimeNs / NANOSECONDS_PER_SECOND) - 1; + // so that Git doesn't trust any file in the copy that it wouldn't trust in the index. Git trusts the recorded + // times of a folder in the untracked cache by the same rule, so a copy that keeps the untracked cache of the + // previous copy must be older than that too. + let timeNs: bigint = stats.mtimeNs; + if (carriedOverCopy && carriedOverCopy.previousTimeNs < timeNs) { + timeNs = carriedOverCopy.previousTimeNs; + } + + const timeInSeconds: number = Number(timeNs / NANOSECONDS_PER_SECOND) - 1; await fs.promises.utimes(temporaryPath, timeInSeconds, timeInSeconds); await fs.promises.rename(temporaryPath, indexPath); @@ -548,7 +586,9 @@ export class RepoStateCache { filterKey, realIndexStamp: getFileStamp(stats), isRealIndexStampSettled, - attributesFingerprint: undefined, + // The file system monitor's state that the new copy keeps says which files Git found unchanged under the + // attributes of the previous copy, so the next call must detect a change since then + attributesFingerprint: carriedOverCopy ? previousIndex?.attributesFingerprint : undefined, hasAttributesChanged: false }; this.#privateIndex = privateIndex; @@ -756,6 +796,35 @@ function deletePrivateFolders(): void { } } +/** + * Builds a new copy of the index that keeps the untracked cache and the file system monitor's state of the previous + * copy. Returns `undefined` if the index records other paths than the previous copy, or if the previous copy can't + * be read, in which case the index is copied as it is. + */ +async function tryCarryOverCachesAsync( + previousPath: string, + content: Buffer, + objectIdLength: number +): Promise { + try { + const handle: fs.promises.FileHandle = await fs.promises.open(previousPath, 'r'); + try { + const previousStats: fs.BigIntStats = await handle.stat({ bigint: true }); + const previousContent: Buffer = await handle.readFile(); + const newContent: Buffer | undefined = tryCarryOverGitIndexCaches( + content, + previousContent, + objectIdLength + ); + return newContent && { content: newContent, previousTimeNs: previousStats.mtimeNs }; + } finally { + await handle.close(); + } + } catch { + return undefined; + } +} + /** * Reads the number of entries from the header of a Git index file, or returns `undefined` if the file doesn't * exist or isn't a Git index. diff --git a/libraries/package-deps-hash/src/test/FsmonitorHook.ts b/libraries/package-deps-hash/src/test/FsmonitorHook.ts new file mode 100644 index 0000000000..1693cd4437 --- /dev/null +++ b/libraries/package-deps-hash/src/test/FsmonitorHook.ts @@ -0,0 +1,45 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +/** + * A file system monitor hook for Git, which reports the paths that a test logs. + */ +export interface IFsmonitorHook { + /** + * The path of the hook, the value of Git's `core.fsmonitor` setting. + */ + readonly hookPath: string; + /** + * Logs a path that changed, which the hook then reports to Git. + */ + logChange(relativePath: string): void; +} + +/** + * Creates a file system monitor hook in the given folder. Its token is the number of paths that were logged when Git + * queried it, and it reports every path as changed for a token that it didn't create. + */ +export function createFsmonitorHook(folderPath: string): IFsmonitorHook { + const hookPath: string = path.join(folderPath, 'fsmonitor-hook'); + const logPath: string = path.join(folderPath, 'fsmonitor-hook.log'); + fs.writeFileSync(logPath, ''); + const script: string = [ + '#!/bin/sh', + `log='${logPath}'`, + `count=$(wc -l < "$log" | tr -d ' ')`, + `printf 't:%s\\0' "$count"`, + 'case "$2" in', + ` t:*) tail -n "+$((\${2#t:} + 1))" "$log" | tr '\\n' '\\0' ;;`, + ` *) printf '/\\0' ;;`, + 'esac', + '' + ].join('\n'); + fs.writeFileSync(hookPath, script, { mode: 0o755 }); + return { + hookPath, + logChange: (relativePath: string) => fs.appendFileSync(logPath, `${relativePath}\n`) + }; +} diff --git a/libraries/package-deps-hash/src/test/GitIndexFile.test.ts b/libraries/package-deps-hash/src/test/GitIndexFile.test.ts index b55b0f34e5..64f14f77db 100644 --- a/libraries/package-deps-hash/src/test/GitIndexFile.test.ts +++ b/libraries/package-deps-hash/src/test/GitIndexFile.test.ts @@ -2,11 +2,22 @@ // See LICENSE in the project root for license information. import { execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; -import { type IGitIndexSummary, summarizeGitIndex, tryGetGitIndexEntryCount } from '../GitIndexFile'; +import { + type IGitIndexExtension, + type IGitIndexSummary, + parseGitIndexLayout, + summarizeGitIndex, + tryCarryOverGitIndexCaches, + tryGetGitIndexEntryCount, + tryReadEwahBitmap, + writeEwahBitmap +} from '../GitIndexFile'; +import { createFsmonitorHook, type IFsmonitorHook } from './FsmonitorHook'; const SHA1_OBJECT_ID_LENGTH: number = 20; const SHA256_OBJECT_ID_LENGTH: number = 32; @@ -22,47 +33,63 @@ function getGitEnvironment(): NodeJS.ProcessEnv { return environment; } -describe(summarizeGitIndex.name, () => { - let repoPath: string; +let repoPath: string; - function runGit(...args: string[]): string { - return execFileSync('git', args, { cwd: repoPath, env: getGitEnvironment(), encoding: 'utf8' }); - } +function runGit(...args: string[]): string { + return execFileSync('git', args, { cwd: repoPath, env: getGitEnvironment(), encoding: 'utf8' }); +} - function writeFile(relativePath: string, content: string): void { - const filePath: string = path.join(repoPath, relativePath); - fs.mkdirSync(path.dirname(filePath), { recursive: true }); - fs.writeFileSync(filePath, content); - } +function runGitWithIndex(indexPath: string, ...args: string[]): string { + return execFileSync('git', args, { + cwd: repoPath, + env: { ...getGitEnvironment(), GIT_INDEX_FILE: indexPath }, + encoding: 'utf8' + }); +} - function readIndex(): Buffer { - return fs.readFileSync(path.join(repoPath, '.git', 'index')); - } +function writeFile(relativePath: string, content: string): void { + const filePath: string = path.join(repoPath, relativePath); + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, content); +} - function summarize(objectIdLength: number = SHA1_OBJECT_ID_LENGTH): IGitIndexSummary { - return summarizeGitIndex(readIndex(), objectIdLength); - } +function getIndexPath(): string { + return path.join(repoPath, '.git', 'index'); +} + +function readIndex(): Buffer { + return fs.readFileSync(getIndexPath()); +} + +function createRepo(...initArgs: string[]): void { + runGit('init', '--quiet', ...initArgs); + runGit('config', 'core.fsmonitor', 'false'); + runGit('config', 'core.untrackedCache', 'true'); + runGit('config', 'index.skipHash', 'false'); + writeFile('a.txt', 'a\n'); + writeFile('dir/b.txt', 'b\n'); + writeFile('dir/sub/c.txt', 'c\n'); + writeFile(LONG_FILE_PATH, 'd\n'); + runGit('add', '.'); +} - function createRepo(...initArgs: string[]): void { - runGit('init', '--quiet', ...initArgs); - runGit('config', 'core.fsmonitor', 'false'); - runGit('config', 'core.untrackedCache', 'true'); - runGit('config', 'index.skipHash', 'false'); - writeFile('a.txt', 'a\n'); - writeFile('dir/b.txt', 'b\n'); - writeFile('dir/sub/c.txt', 'c\n'); - writeFile(LONG_FILE_PATH, 'd\n'); - runGit('add', '.'); +function commit(): void { + runGit('-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '--quiet', '-m', 'Commit'); +} + +// Git writes a version 3 index only if an entry has extended flags, and writes a version 2 index otherwise +function setIndexVersion(version: number): void { + if (version === 3) { + runGit('update-index', '--skip-worktree', LONG_FILE_PATH); } - // Git writes a version 3 index only if an entry has extended flags, and writes a version 2 index otherwise - function setIndexVersion(version: number): void { - if (version === 3) { - runGit('update-index', '--skip-worktree', LONG_FILE_PATH); - } + runGit('update-index', `--index-version=${version}`); + expect(readIndex().readUInt32BE(4)).toBe(version); +} - runGit('update-index', `--index-version=${version}`); - expect(readIndex().readUInt32BE(4)).toBe(version); +describe(summarizeGitIndex.name, () => { + function summarize(objectIdLength: number = SHA1_OBJECT_ID_LENGTH): IGitIndexSummary { + return summarizeGitIndex(readIndex(), objectIdLength); } beforeEach(() => { @@ -95,7 +122,7 @@ describe(summarizeGitIndex.name, () => { it.each([2, 3, 4])('ignores what refreshing a version %i index changes', (version: number) => { createRepo(); setIndexVersion(version); - runGit('-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '--quiet', '-m', 'Initial'); + commit(); const initialIndex: Buffer = readIndex(); const initialSummary: IGitIndexSummary = summarize(); @@ -146,7 +173,7 @@ describe(summarizeGitIndex.name, () => { it.each([2, 3, 4])('digests the recorded sizes of a version %i index separately', (version: number) => { createRepo(); setIndexVersion(version); - runGit('-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '--quiet', '-m', 'Initial'); + commit(); const initialSummary: IGitIndexSummary = summarize(); expect(initialSummary.sizesDigest).toMatch(/^[0-9a-f]{40}$/); @@ -226,3 +253,536 @@ describe(tryGetGitIndexEntryCount.name, () => { expect(tryGetGitIndexEntryCount(Buffer.from('PACK\0\0\0\x02\0\0\0\x01'))).toBeUndefined(); }); }); + +interface ITrace2Event { + category?: string; + key?: string; + value?: string; +} + +interface IFsmonitorState { + token: string; + changedEntries: number[]; +} + +function getSetBits(bits: Uint8Array): number[] { + const setBits: number[] = []; + bits.forEach((bit: number, index: number) => { + if (bit) { + setBits.push(index); + } + }); + return setBits; +} + +function createBits(bitCount: number, setBits: ReadonlyArray): Uint8Array { + const bits: Uint8Array = new Uint8Array(bitCount); + for (const bit of setBits) { + bits[bit] = 1; + } + + return bits; +} + +function getRange(start: number, end: number): number[] { + const range: number[] = []; + for (let i: number = start; i < end; i++) { + range.push(i); + } + + return range; +} + +describe(tryCarryOverGitIndexCaches.name, () => { + const replacedSignatures: ReadonlySet = new Set(['EOIE', 'IEOT', 'UNTR', 'FSMN']); + let copyPath: string; + let newCopyPath: string; + + function getExtension( + content: Buffer, + signature: string, + objectIdLength: number = SHA1_OBJECT_ID_LENGTH + ): Buffer | undefined { + const extension: IGitIndexExtension | undefined = parseGitIndexLayout( + content, + objectIdLength + ).extensions.find((candidate: IGitIndexExtension) => candidate.signature === signature); + return extension && content.subarray(extension.start, extension.end); + } + + function getRequiredExtension(content: Buffer, signature: string): Buffer { + const extension: Buffer | undefined = getExtension(content, signature); + if (!extension) { + throw new Error(`The index has no "${signature}" extension`); + } + + return extension; + } + + function getSignatures(content: Buffer): string[] { + return parseGitIndexLayout(content, SHA1_OBJECT_ID_LENGTH).extensions.map( + ({ signature }: IGitIndexExtension) => signature + ); + } + + // Inserts an extension before the checksum, without updating the checksum + function insertExtension(content: Buffer, signature: string, data: Buffer): Buffer { + const header: Buffer = Buffer.alloc(8); + header.write(signature, 0, 'latin1'); + header.writeUInt32BE(data.length, 4); + const checksumOffset: number = content.length - SHA1_OBJECT_ID_LENGTH; + return Buffer.concat([ + content.subarray(0, checksumOffset), + header, + data, + content.subarray(checksumOffset) + ]); + } + + // Git doesn't trust the recorded times of a file or folder that changed in the same second as the index was saved + function settleWorkingTree(folderPath: string, time: number): void { + for (const entry of fs.readdirSync(folderPath, { withFileTypes: true })) { + if (entry.name !== '.git') { + const entryPath: string = path.join(folderPath, entry.name); + if (entry.isDirectory()) { + settleWorkingTree(entryPath, time); + } + + fs.utimesSync(entryPath, time, time); + } + } + + fs.utimesSync(folderPath, time, time); + } + + // Creates a repository with untracked files, and a copy of its index in which Git saved the untracked cache + function createRepoWithCopy(version: number, objectIdLength: number = SHA1_OBJECT_ID_LENGTH): void { + createRepo(...(objectIdLength === SHA256_OBJECT_ID_LENGTH ? ['--object-format=sha256'] : [])); + // Git then saves the "EOIE" extension + runGit('config', 'index.threads', 'true'); + setIndexVersion(version); + writeFile('untracked.txt', 'untracked\n'); + writeFile('dir/untracked.txt', 'untracked\n'); + settleWorkingTree(repoPath, Math.floor(Date.now() / 1000) - 100); + runGit('update-index', '--refresh'); + commit(); + fs.copyFileSync(getIndexPath(), copyPath); + runGitWithIndex(copyPath, 'status', '--porcelain'); + expect(getExtension(fs.readFileSync(copyPath), 'UNTR', objectIdLength)).toBeDefined(); + } + + function stageModifiedFile(): void { + writeFile('a.txt', 'modified\n'); + runGit('add', 'a.txt'); + } + + function carryOver(objectIdLength: number = SHA1_OBJECT_ID_LENGTH): Buffer { + const content: Buffer | undefined = tryCarryOverGitIndexCaches( + readIndex(), + fs.readFileSync(copyPath), + objectIdLength + ); + if (!content) { + throw new Error('The new copy did not keep the caches of the previous copy'); + } + + fs.writeFileSync(newCopyPath, content); + return content; + } + + // Runs "git status", and counts the folders that it read rather than finding their untracked files in the cache + function getStatus(indexPath: string, ...configArgs: string[]): [string, number[]] { + const tracePath: string = path.join(repoPath, '.git', 'trace2.json'); + const output: string = execFileSync('git', [...configArgs, 'status', '--porcelain'], { + cwd: repoPath, + env: { + ...getGitEnvironment(), + GIT_INDEX_FILE: indexPath, + GIT_TRACE2_EVENT: tracePath, + // Git reports the statistics of the untracked cache in a nested region + GIT_TRACE2_EVENT_NESTING: '10' + }, + encoding: 'utf8' + }); + const events: ITrace2Event[] = fs + .readFileSync(tracePath, 'utf8') + .split('\n') + .filter((line: string) => line) + .map((line: string) => JSON.parse(line)); + fs.unlinkSync(tracePath); + const openedFolderCounts: number[] = events + .filter(({ category, key }: ITrace2Event) => category === 'read_directory' && key === 'opendir') + .map(({ value }: ITrace2Event) => Number(value)); + return [output, openedFolderCounts]; + } + + beforeEach(() => { + repoPath = fs.mkdtempSync(path.join(os.tmpdir(), 'git-index-file-test-')); + copyPath = path.join(repoPath, '.git', 'copy'); + newCopyPath = path.join(repoPath, '.git', 'new-copy'); + }); + + afterEach(() => { + fs.rmSync(repoPath, { recursive: true, force: true }); + }); + + it.each([2, 3, 4])( + 'keeps the untracked cache of the previous copy of a version %i index', + (version: number) => { + createRepoWithCopy(version); + stageModifiedFile(); + writeFile('dir/b.txt', 'modified\n'); + const index: Buffer = readIndex(); + expect(getSignatures(index)).toContain('EOIE'); + + const content: Buffer = carryOver(); + expect(getSignatures(content)).toEqual([ + ...getSignatures(index).filter((signature: string) => !replacedSignatures.has(signature)), + 'UNTR' + ]); + expect(getExtension(content, 'UNTR')).toEqual(getExtension(fs.readFileSync(copyPath), 'UNTR')); + expect(content.subarray(-SHA1_OBJECT_ID_LENGTH)).toEqual( + createHash('sha1').update(content.subarray(0, -SHA1_OBJECT_ID_LENGTH)).digest() + ); + expect(runGitWithIndex(newCopyPath, 'ls-files', '--stage', '--debug')).toBe( + runGit('ls-files', '--stage', '--debug') + ); + const [output, openedFolderCounts] = getStatus(newCopyPath); + expect(openedFolderCounts).toEqual([0]); + expect(output).toBe(runGit('status', '--porcelain')); + expect(output).toBe('M a.txt\n M dir/b.txt\n?? dir/untracked.txt\n?? untracked.txt\n'); + } + ); + + it('keeps the untracked cache of the previous copy of the index of a SHA-256 repository', () => { + createRepoWithCopy(4, SHA256_OBJECT_ID_LENGTH); + stageModifiedFile(); + + const content: Buffer = carryOver(SHA256_OBJECT_ID_LENGTH); + expect(getExtension(content, 'UNTR', SHA256_OBJECT_ID_LENGTH)).toEqual( + getExtension(fs.readFileSync(copyPath), 'UNTR', SHA256_OBJECT_ID_LENGTH) + ); + expect(content.subarray(-SHA256_OBJECT_ID_LENGTH)).toEqual( + createHash('sha256').update(content.subarray(0, -SHA256_OBJECT_ID_LENGTH)).digest() + ); + expect(getStatus(newCopyPath)).toEqual([runGit('status', '--porcelain'), [0]]); + }); + + it('writes no checksum when Git writes none', () => { + createRepoWithCopy(4); + runGit('config', 'index.skipHash', 'true'); + stageModifiedFile(); + expect(readIndex().subarray(-SHA1_OBJECT_ID_LENGTH)).toEqual(Buffer.alloc(SHA1_OBJECT_ID_LENGTH)); + + const content: Buffer = carryOver(); + expect(content.subarray(-SHA1_OBJECT_ID_LENGTH)).toEqual(Buffer.alloc(SHA1_OBJECT_ID_LENGTH)); + expect(getStatus(newCopyPath)).toEqual([runGit('status', '--porcelain'), [0]]); + }); + + it.each<[string, () => void, number]>([ + ['adds a file', () => runGit('add', 'untracked.txt'), 5], + ['removes a file', () => runGit('rm', '--cached', '--quiet', 'a.txt'), 3], + ['renames a file', () => runGit('mv', 'a.txt', 'e.txt'), 4], + [ + 'records another file in place of one', + () => { + runGit('rm', '--cached', '--quiet', 'a.txt'); + runGit('add', 'dir/untracked.txt'); + }, + 4 + ], + [ + 'records a symbolic link in place of a file', + () => { + const objectId: string = runGit('rev-parse', 'HEAD:a.txt').trim(); + runGit('update-index', '--cacheinfo', `120000,${objectId},a.txt`); + }, + 4 + ], + [ + 'records a conflict in place of a file', + () => { + const objectId: string = runGit('rev-parse', 'HEAD:a.txt').trim(); + execFileSync('git', ['update-index', '--index-info'], { + cwd: repoPath, + env: getGitEnvironment(), + input: `0 ${'0'.repeat(40)}\ta.txt\n100644 ${objectId} 1\ta.txt\n` + }); + }, + 4 + ] + ])('returns undefined when the index %s', (description: string, change: () => void, entryCount: number) => { + createRepoWithCopy(4); + change(); + expect(tryGetGitIndexEntryCount(readIndex())).toBe(entryCount); + expect( + tryCarryOverGitIndexCaches(readIndex(), fs.readFileSync(copyPath), SHA1_OBJECT_ID_LENGTH) + ).toBeUndefined(); + }); + + it('returns undefined when the index and the previous copy have different versions', () => { + createRepoWithCopy(4); + runGit('update-index', '--index-version=2'); + expect( + tryCarryOverGitIndexCaches(readIndex(), fs.readFileSync(copyPath), SHA1_OBJECT_ID_LENGTH) + ).toBeUndefined(); + }); + + it('returns undefined when the previous copy has no untracked cache', () => { + createRepoWithCopy(4); + runGitWithIndex(copyPath, '-c', 'core.untrackedCache=false', 'status', '--porcelain'); + const previousContent: Buffer = fs.readFileSync(copyPath); + expect(getExtension(previousContent, 'UNTR')).toBeUndefined(); + stageModifiedFile(); + expect(tryCarryOverGitIndexCaches(readIndex(), previousContent, SHA1_OBJECT_ID_LENGTH)).toBeUndefined(); + }); + + it('keeps the extensions of the index that Git may ignore, but not those that it must understand', () => { + createRepoWithCopy(4); + stageModifiedFile(); + const index: Buffer = insertExtension(readIndex(), 'ZZZZ', Buffer.from('data')); + const previousContent: Buffer = fs.readFileSync(copyPath); + const content: Buffer | undefined = tryCarryOverGitIndexCaches( + index, + previousContent, + SHA1_OBJECT_ID_LENGTH + ); + expect(content && getSignatures(content)).toEqual(['TREE', 'ZZZZ', 'UNTR']); + expect(content && getExtension(content, 'ZZZZ')).toEqual(getExtension(index, 'ZZZZ')); + + // The extensions of a split index and of a sparse index + for (const signature of ['link', 'sdir']) { + const extensionData: Buffer = Buffer.alloc(signature === 'link' ? SHA1_OBJECT_ID_LENGTH : 0); + expect( + tryCarryOverGitIndexCaches( + insertExtension(index, signature, extensionData), + previousContent, + SHA1_OBJECT_ID_LENGTH + ) + ).toBeUndefined(); + expect( + tryCarryOverGitIndexCaches( + index, + insertExtension(previousContent, signature, extensionData), + SHA1_OBJECT_ID_LENGTH + ) + ).toBeUndefined(); + } + }); + + it('throws when the previous copy is not an index', () => { + createRepoWithCopy(4); + const previousContent: Buffer = fs.readFileSync(copyPath); + expect(() => + tryCarryOverGitIndexCaches(readIndex(), previousContent.subarray(0, 100), SHA1_OBJECT_ID_LENGTH) + ).toThrow('The Git index ends within an entry'); + }); + + if (process.platform !== 'win32') { + describe('with a file system monitor', () => { + let hook: IFsmonitorHook; + + function runGitWithFsmonitor(indexPath: string, ...args: string[]): string { + return runGitWithIndex(indexPath, '-c', `core.fsmonitor=${hook.hookPath}`, ...args); + } + + function readFsmonitorState(content: Buffer): IFsmonitorState { + const data: Buffer = getRequiredExtension(content, 'FSMN').subarray(8); + expect(data.readUInt32BE(0)).toBe(2); + const tokenEnd: number = data.indexOf(0, 4); + const bitmap: Buffer = data.subarray(tokenEnd + 5); + expect(bitmap.length).toBe(data.readUInt32BE(tokenEnd + 1)); + const bits: Uint8Array | undefined = tryReadEwahBitmap(bitmap, content.readUInt32BE(8)); + return { + token: data.toString('latin1', 4, tokenEnd), + changedEntries: bits ? getSetBits(bits) : [] + }; + } + + function modifyFile(relativePath: string): void { + writeFile(relativePath, 'modified\n'); + hook.logChange(relativePath); + } + + it('keeps the token of the previous copy, and marks the entries that differ from it as changed', () => { + createRepoWithCopy(4); + hook = createFsmonitorHook(path.join(repoPath, '.git')); + // Git asks the hook for every change, since the copy has no token, and then saves the token of the hook + runGitWithFsmonitor(copyPath, 'status', '--porcelain'); + runGitWithFsmonitor(copyPath, 'update-index', '--no-fsmonitor-valid', 'dir/b.txt'); + expect(readFsmonitorState(fs.readFileSync(copyPath))).toEqual({ token: 't:0', changedEntries: [1] }); + + modifyFile('a.txt'); + runGit('add', 'a.txt'); + const content: Buffer = carryOver(); + expect(readFsmonitorState(content)).toEqual({ token: 't:0', changedEntries: [0, 1] }); + // Git lists the entries that the file system monitor considers unchanged in lowercase + expect(runGitWithFsmonitor(newCopyPath, 'ls-files', '-f')).toBe( + `H a.txt\nH dir/b.txt\nh dir/sub/c.txt\nh ${LONG_FILE_PATH}\n` + ); + expect(getStatus(newCopyPath, '-c', `core.fsmonitor=${hook.hookPath}`)[0]).toBe( + runGit('status', '--porcelain') + ); + }); + + it('reads and writes runs of entries that are all marked as changed', () => { + const filePaths: string[] = getRange(0, 200).map( + (index: number) => `f/${String(index).padStart(3, '0')}` + ); + for (const filePath of filePaths) { + writeFile(filePath, `${filePath}\n`); + } + + createRepoWithCopy(4); + hook = createFsmonitorHook(path.join(repoPath, '.git')); + runGitWithFsmonitor(copyPath, 'status', '--porcelain'); + runGitWithFsmonitor(copyPath, 'update-index', '--no-fsmonitor-valid', ...filePaths); + const previousContent: Buffer = fs.readFileSync(copyPath); + const previousState: IFsmonitorState = readFsmonitorState(previousContent); + expect(previousState).toEqual({ token: 't:0', changedEntries: getRange(4, 204) }); + + modifyFile('a.txt'); + runGit('add', 'a.txt'); + const content: Buffer = carryOver(); + const state: IFsmonitorState = readFsmonitorState(content); + expect(state).toEqual({ token: 't:0', changedEntries: [0, ...getRange(4, 204)] }); + // Git compresses runs of words whose bits are all set, but the new copy doesn't + expect(getRequiredExtension(content, 'FSMN').length).toBeGreaterThan( + getRequiredExtension(previousContent, 'FSMN').length + ); + expect(runGitWithFsmonitor(newCopyPath, 'ls-files', '-f')).toBe( + [ + 'H a.txt', + 'h dir/b.txt', + 'h dir/sub/c.txt', + `h ${LONG_FILE_PATH}`, + ...filePaths.map((filePath: string) => `H ${filePath}`), + '' + ].join('\n') + ); + }); + + it('keeps the token of the previous copy rather than that of the index', () => { + createRepoWithCopy(4); + hook = createFsmonitorHook(path.join(repoPath, '.git')); + runGitWithFsmonitor(copyPath, 'status', '--porcelain'); + // Git saves the index with a later token, after the untracked file was created + writeFile('dir/new.txt', 'new\n'); + hook.logChange('dir/new.txt'); + runGitWithFsmonitor(getIndexPath(), 'status', '--porcelain'); + modifyFile('a.txt'); + runGitWithFsmonitor(getIndexPath(), 'add', 'a.txt'); + expect(readFsmonitorState(readIndex()).token).toBe('t:2'); + + const content: Buffer = carryOver(); + expect(readFsmonitorState(content).token).toBe('t:0'); + const [output] = getStatus(newCopyPath, '-c', `core.fsmonitor=${hook.hookPath}`); + expect(output).toBe(runGitWithFsmonitor(getIndexPath(), 'status', '--porcelain')); + expect(output).toContain('?? dir/new.txt\n'); + }); + + it('drops the state of the file system monitor of the index', () => { + createRepoWithCopy(4); + hook = createFsmonitorHook(path.join(repoPath, '.git')); + modifyFile('a.txt'); + runGitWithFsmonitor(getIndexPath(), 'add', 'a.txt'); + expect(getExtension(readIndex(), 'FSMN')).toBeDefined(); + + expect(getExtension(carryOver(), 'FSMN')).toBeUndefined(); + }); + + it('returns undefined when the state of the file system monitor of the previous copy is malformed', () => { + createRepoWithCopy(4); + hook = createFsmonitorHook(path.join(repoPath, '.git')); + runGitWithFsmonitor(copyPath, 'status', '--porcelain'); + stageModifiedFile(); + const previousContent: Buffer = fs.readFileSync(copyPath); + expect(tryCarryOverGitIndexCaches(readIndex(), previousContent, SHA1_OBJECT_ID_LENGTH)).toBeDefined(); + + // An unknown version + const malformedContent: Buffer = Buffer.from(previousContent); + const extensionOffset: number = previousContent.indexOf( + getRequiredExtension(previousContent, 'FSMN') + ); + malformedContent.writeUInt32BE(3, extensionOffset + 8); + expect( + tryCarryOverGitIndexCaches(readIndex(), malformedContent, SHA1_OBJECT_ID_LENGTH) + ).toBeUndefined(); + }); + }); + } +}); + +describe(writeEwahBitmap.name, () => { + it('writes an empty bitmap as Git does', () => { + expect(writeEwahBitmap(new Uint8Array(3)).toString('hex')).toBe( + ['00000000', '00000001', '0000000000000000', '00000000'].join('') + ); + }); + + it('writes a bitmap as Git does', () => { + // As Git saved it for an index of 4 entries, in which it marked the second entry as changed + expect(writeEwahBitmap(createBits(4, [1])).toString('hex')).toBe( + ['00000002', '00000002', '0000000200000000', '0000000000000002', '00000000'].join('') + ); + }); + + it('compresses runs of words that are 0', () => { + const bits: Uint8Array = createBits(100000, [5, 99999]); + const data: Buffer = writeEwahBitmap(bits); + // A marker and a literal word, then a marker for the run and a literal word + expect(data.length).toBe(8 + 4 * 8 + 4); + expect(tryReadEwahBitmap(data, bits.length)).toEqual(bits); + }); + + it('writes bitmaps that it reads back', () => { + let seed: number = 1; + for (const bitCount of [1, 31, 32, 63, 64, 65, 200, 5000]) { + for (const density of [0, 0.01, 0.5, 0.99, 1]) { + const bits: Uint8Array = new Uint8Array(bitCount); + for (let i: number = 0; i < bitCount; i++) { + seed = (seed * 1103515245 + 12345) % 2 ** 31; + bits[i] = seed / 2 ** 31 < density ? 1 : 0; + } + + expect(tryReadEwahBitmap(writeEwahBitmap(bits), bitCount)).toEqual(bits); + } + } + }); +}); + +describe(tryReadEwahBitmap.name, () => { + it('reads runs of words whose bits are all set', () => { + // As Git saved it for an index of 132 entries, in which it marked every entry as changed + const data: Buffer = Buffer.from( + ['00000084', '00000002', '0000000200000005', '000000000000000f', '00000000'].join(''), + 'hex' + ); + expect(tryReadEwahBitmap(data, 132)).toEqual(new Uint8Array(132).fill(1)); + expect(tryReadEwahBitmap(data, 131)).toBeUndefined(); + }); + + it('rejects a malformed bitmap', () => { + const data: Buffer = writeEwahBitmap(createBits(4, [1])); + expect(tryReadEwahBitmap(data, 4)).toEqual(createBits(4, [1])); + expect(tryReadEwahBitmap(data.subarray(0, 4), 4)).toBeUndefined(); + expect(tryReadEwahBitmap(data.subarray(0, data.length - 1), 4)).toBeUndefined(); + + // A bit beyond the size of the bitmap + const bitBeyondSize: Buffer = Buffer.from(data); + bitBeyondSize.writeUInt32BE(1, 0); + expect(tryReadEwahBitmap(bitBeyondSize, 4)).toBeUndefined(); + + // More literal words than the bitmap has + const missingLiteralWord: Buffer = Buffer.from(data); + missingLiteralWord.writeUInt32BE(4, 8); + expect(tryReadEwahBitmap(missingLiteralWord, 4)).toBeUndefined(); + + // A run of words whose bits are all set, beyond the size of the bitmap + const runBeyondSize: Buffer = Buffer.from( + ['00000004', '00000001', '0000000000000003', '00000000'].join(''), + 'hex' + ); + expect(tryReadEwahBitmap(runBeyondSize, 64)).toBeUndefined(); + }); +}); diff --git a/libraries/package-deps-hash/src/test/RepoStateCache.test.ts b/libraries/package-deps-hash/src/test/RepoStateCache.test.ts index 162c7ac7a9..a971b1866f 100644 --- a/libraries/package-deps-hash/src/test/RepoStateCache.test.ts +++ b/libraries/package-deps-hash/src/test/RepoStateCache.test.ts @@ -11,6 +11,7 @@ import { Executable, type IExecutableSpawnOptions } from '@rushstack/node-core-l import * as GitIndexFile from '../GitIndexFile'; import { getDetailedRepoStateAsync, type IDetailedRepoState } from '../getRepoState'; import { RepoStateCache } from '../RepoStateCache'; +import { createFsmonitorHook, type IFsmonitorHook } from './FsmonitorHook'; const originalDateNow: () => number = Date.now; const originalSpawn: typeof Executable.spawn = Executable.spawn; @@ -46,6 +47,38 @@ interface IGitCommand { usesPrivateIndex: boolean; } +interface ITrace2Event { + category?: string; + key?: string; + value?: string; +} + +function getExtension(content: Buffer, signature: string): Buffer | undefined { + const extension: GitIndexFile.IGitIndexExtension | undefined = GitIndexFile.parseGitIndexLayout( + content, + 20 + ).extensions.find((candidate: GitIndexFile.IGitIndexExtension) => candidate.signature === signature); + return extension && content.subarray(extension.start, extension.end); +} + +// Sets environment variables, and returns a function that restores their previous values +function setEnvironmentVariables(variables: Record): () => void { + const previousValues: [string, string | undefined][] = Object.keys(variables).map((name: string) => [ + name, + process.env[name] + ]); + Object.assign(process.env, variables); + return () => { + for (const [name, value] of previousValues) { + if (value === undefined) { + delete process.env[name]; + } else { + process.env[name] = value; + } + } + }; +} + function getGitCommand( args: ReadonlyArray, options: IExecutableSpawnOptions | undefined @@ -901,6 +934,221 @@ describe(RepoStateCache.name, () => { }); } + describe('when the index records the same paths as the copy', () => { + // Git doesn't trust the recorded times of a folder that changed in the same second as the index was saved + function settleFolders(...relativePaths: string[]): void { + const time: number = Math.floor(originalDateNow() / 1000) - 100; + for (const relativePath of relativePaths) { + fs.utimesSync(path.join(repoPath, relativePath), time, time); + } + } + + // Returns the state, and the numbers of folders that "git status" read rather than finding their untracked + // files in the untracked cache + async function getStateAndOpenedFolderCountsAsync(): Promise<[IDetailedRepoState, number[]]> { + const tracePath: string = path.join(repoPath, '.git', 'trace2.json'); + const restoreEnvironment: () => void = setEnvironmentVariables({ + GIT_TRACE2_EVENT: tracePath, + // Git reports the statistics of the untracked cache in a nested region + GIT_TRACE2_EVENT_NESTING: '10' + }); + let state: IDetailedRepoState; + try { + state = await getStateAsync(); + } finally { + restoreEnvironment(); + } + + const events: ITrace2Event[] = fs + .readFileSync(tracePath, 'utf8') + .split('\n') + .filter((line: string) => line) + .map((line: string) => JSON.parse(line)); + fs.unlinkSync(tracePath); + const openedFolderCounts: number[] = events + .filter(({ category, key }: ITrace2Event) => category === 'read_directory' && key === 'opendir') + .map(({ value }: ITrace2Event) => Number(value)); + return [state, openedFolderCounts]; + } + + beforeEach(() => { + // Otherwise the configuration files may still change, and the cache copies the index as it is + settleFiles(); + }); + + it('keeps the untracked cache of the previous copy', async () => { + writeFile('untracked.txt', 'untracked\n'); + writeFile('dir/untracked.txt', 'untracked\n'); + settleFolders('.', 'dir'); + await getStateAsync(); + const previousPath: string = getPrivateIndexPath(); + const previousContent: Buffer = fs.readFileSync(previousPath); + expect(getExtension(previousContent, 'UNTR')).toBeDefined(); + // Git saved the copy after the index + const previousTime: number = Math.floor(originalDateNow() / 1000) - 50; + fs.utimesSync(previousPath, previousTime, previousTime); + + writeFile('a.txt', 'modified\n'); + runGit('add', 'a.txt'); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + const utimesSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'utimes'); + const [state, openedFolderCounts] = await getStateAndOpenedFolderCountsAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('dir/untracked.txt')).toBe(hashText('untracked\n')); + expect(openedFolderCounts).toEqual([0]); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + expect(getExtension(writeFileSpy.mock.calls[0][1], 'UNTR')).toEqual( + getExtension(previousContent, 'UNTR') + ); + // The new copy is older than the previous copy, so that Git doesn't trust the recorded times of any folder + // that it didn't trust in the previous copy + expect(utimesSpy).toHaveBeenCalledWith(expect.any(String), previousTime - 1, previousTime - 1); + }); + + it('finds the untracked files that changed since the previous copy', async () => { + writeFile('untracked.txt', 'untracked\n'); + settleFolders('.', 'dir'); + await getStateAsync(); + + writeFile('dir/untracked.txt', 'untracked\n'); + fs.unlinkSync(path.join(repoPath, 'untracked.txt')); + writeFile('a.txt', 'modified\n'); + runGit('add', 'a.txt'); + const [state, openedFolderCounts] = await getStateAndOpenedFolderCountsAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('dir/untracked.txt')).toBe(hashText('untracked\n')); + expect(state.files.has('untracked.txt')).toBe(false); + expect(openedFolderCounts).toEqual([2]); + }); + + it('copies the index as it is when the index records other paths', async () => { + writeFile('untracked.txt', 'untracked\n'); + settleFolders('.', 'dir'); + await getStateAsync(); + + // The index records as many files as the copy + runGit('rm', '--cached', '--quiet', 'b.txt'); + runGit('add', 'untracked.txt'); + const carryOverSpy: jest.SpyInstance = jest.spyOn(GitIndexFile, 'tryCarryOverGitIndexCaches'); + const state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('b.txt')).toBe(hashText('b\n')); + expect(carryOverSpy.mock.results).toEqual([{ type: 'return', value: undefined }]); + }); + + it('returns the same state as getDetailedRepoStateAsync after each command', async () => { + const carryOverSpy: jest.SpyInstance = jest.spyOn(GitIndexFile, 'tryCarryOverGitIndexCaches'); + const commands: (() => void)[] = [ + () => writeFile('untracked.txt', 'untracked\n'), + () => writeFile('a.txt', 'modified\n'), + () => runGit('add', 'a.txt'), + () => runGit('restore', '--staged', 'a.txt'), + () => runGit('stash', '--quiet'), + () => runGit('stash', 'pop', '--quiet'), + () => runGit('checkout', '--', 'a.txt'), + () => writeFile('dir/sub/untracked.txt', 'untracked\n'), + () => runGit('add', 'dir/sub/untracked.txt'), + () => fs.unlinkSync(path.join(repoPath, 'untracked.txt')), + () => runGit('rm', '--cached', '--quiet', 'b.txt'), + () => runGit('mv', 'dir/c.txt', 'dir/d.txt'), + () => writeFile('dir/d.txt', 'modified\n'), + () => runGit('add', '--all') + ]; + await getStateAsync(); + for (const command of commands) { + command(); + await expectUncachedStateAsync(await getStateAsync()); + } + + expect( + carryOverSpy.mock.results.filter(({ value }: jest.MockResult) => value) + ).not.toHaveLength(0); + }); + + it('computes the state without the cache when the attributes changed since the previous copy', async () => { + writeFile('crlf.txt', 'a\r\n'); + commit(); + await getStateAsync(); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + + writeFile('.gitattributes', '*.txt text eol=lf\n'); + writeFile('a.txt', 'modified\n'); + runGit('add', 'a.txt'); + takeGitCommands(); + let state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + // The cache ran "git status" on the new copy, and then computed the state without it + expect(takeGitCommands().some(({ usesPrivateIndex }: IGitCommand) => !usesPrivateIndex)).toBe(true); + + takeGitCommands(); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(writeFileSpy).toHaveBeenCalledTimes(2); + expect(takeUsesPrivateIndex()).toBe(true); + }); + + if (process.platform !== 'win32') { + describe('with a file system monitor', () => { + let hook: IFsmonitorHook; + let restoreEnvironment: () => void; + + function modifyFile(relativePath: string): void { + writeFile(relativePath, 'modified\n'); + hook.logChange(relativePath); + } + + beforeEach(() => { + hook = createFsmonitorHook(path.join(repoPath, '.git')); + // Git applies the configuration in the environment after that of the repository + const count: number = Number(process.env.GIT_CONFIG_COUNT || 0); + restoreEnvironment = setEnvironmentVariables({ + [`GIT_CONFIG_KEY_${count}`]: 'core.fsmonitor', + [`GIT_CONFIG_VALUE_${count}`]: hook.hookPath, + GIT_CONFIG_COUNT: String(count + 1) + }); + }); + + afterEach(() => { + restoreEnvironment(); + }); + + it('marks the files whose entries changed as changed', async () => { + const carryOverSpy: jest.SpyInstance = jest.spyOn(GitIndexFile, 'tryCarryOverGitIndexCaches'); + await getStateAsync(); + modifyFile('a.txt'); + await expectUncachedStateAsync(await getStateAsync()); + runGit('add', 'a.txt'); + await expectUncachedStateAsync(await getStateAsync()); + // Git found the file unchanged since it was added, but the index no longer records its content + expect(runGit('ls-files', '-f', 'a.txt')).toBe('H a.txt\n'); + runGit('restore', '--staged', 'a.txt'); + + const state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(hashText('modified\n')); + expect( + carryOverSpy.mock.results.map(({ value }: jest.MockResult) => !!value) + ).toEqual([true, true]); + }); + + it('keeps the token of the previous copy rather than that of the index', async () => { + await getStateAsync(); + // Git saves the index with a later token, after the untracked file was created + writeFile('dir/new.txt', 'new\n'); + hook.logChange('dir/new.txt'); + runGit('status', '--porcelain'); + modifyFile('b.txt'); + runGit('add', 'b.txt'); + + const state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('dir/new.txt')).toBe(hashText('new\n')); + }); + }); + } + }); + function hashText(text: string): string { return execFileSync('git', ['hash-object', '--stdin', '--no-filters'], { cwd: repoPath, From 73968539150d17fcf6b60f0ca9dad97450cff451 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 05:04:02 +0000 Subject: [PATCH 082/265] [rush-daemon] Test that a usage error goes to in-process Rush while a reload is pending Swarm integration step 67; original commit 3ff87ea4b8 (merge of swarm/r04-t78-nit7 at aae9f29de0). Scope: task 78 NIT 7. Brings r04's fix for ch01's NIT 7 on task 78 (board 3099, board 3160): a new test pins the `!#forceReload` conjunct. No behavior change. Second agent: o04 board 3188. s17 batch D2, item 5 of 6. Gate: ch01 GATE OK board 3514 (tree 2fcaab2166) Commits folded into this step (1): - aae9f29de0 Test that a usage error goes to in-process Rush while a reload is pending (task 78 follow-up, ch01 board 3099 NIT 7) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- .../swarm-r04-t78-nit7_2026-09-29-03-45.json | 11 +++++++ .../src/WorkspaceRequestLifecycle.ts | 3 +- .../src/test/DaemonRequestUsageError.test.ts | 29 +++++++++++++++++++ 3 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 common/changes/@rushstack/rush-daemon/swarm-r04-t78-nit7_2026-09-29-03-45.json diff --git a/common/changes/@rushstack/rush-daemon/swarm-r04-t78-nit7_2026-09-29-03-45.json b/common/changes/@rushstack/rush-daemon/swarm-r04-t78-nit7_2026-09-29-03-45.json new file mode 100644 index 0000000000..166f004f7f --- /dev/null +++ b/common/changes/@rushstack/rush-daemon/swarm-r04-t78-nit7_2026-09-29-03-45.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-daemon", + "comment": "", + "type": "none" + } + ], + "packageName": "@rushstack/rush-daemon", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts index e9e4fd9223..29ab445144 100644 --- a/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts +++ b/libraries/rush-daemon/src/WorkspaceRequestLifecycle.ts @@ -880,7 +880,8 @@ export class WorkspaceRequestLifecycle implements IDaemonRequestLifecycle { * Whether the session's configuration is known to match the workspace inputs: a reload bound this session after * checking that its inputs did not change while it loaded, and they have not changed since. The session that the * daemon loads at startup is not checked that way. An edit made after it loaded, while the startup capture runs, - * is in the startup fingerprint but not in the session. + * is in the startup fingerprint but not in the session. Nor does a session count while a reload is pending, for + * example after a reload found the Rush lock busy; until a build reloads, a usage error goes to in-process Rush. */ #isConfigurationCurrent(session: IWorkspaceSession, tier: WorkspaceInputChangeTier): boolean { return tier === WorkspaceInputChangeTier.Reuse && this.#boundSession === session && !this.#forceReload; diff --git a/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts b/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts index c333f2b618..4b21f9946d 100644 --- a/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts +++ b/libraries/rush-daemon/src/test/DaemonRequestUsageError.test.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import type { IOperationGraph } from '@microsoft/rush-lib'; +import { LockFile } from '@rushstack/node-core-library'; import { DaemonGraphTestFixture } from './DaemonGraphTestFixture'; @@ -65,6 +66,34 @@ it('hands an invalid command line to in-process Rush after a configuration chang } }); +it('hands an invalid command line to in-process Rush while the warm set waits for a reload that found the Rush lock busy', async () => { + const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync(); + try { + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(usageFailure); + // Other parameters need a reload. It stops the warm set before it takes the Rush lock, which this test holds. + const native: LockFile | undefined = LockFile.tryAcquire( + fixture.session.rushConfiguration.commonTempFolder, + 'rush' + ); + expect(native).toBeDefined(); + try { + expect((await fixture.runAsync(['build', '--to', 'b', '--ignore-hooks'])).terminal).toMatchObject({ + kind: 'requestRejected', + payload: { message: expect.stringContaining('Another Rush command') } + }); + } finally { + native?.release(); + } + // The inputs did not change, but the daemon answers again only after a build reloads. + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(inProcess); + expect((await fixture.buildAsync()).terminal).toMatchObject(success); + expect((await fixture.runAsync(invalid)).terminal).toMatchObject(usageFailure); + } finally { + await fixture[Symbol.asyncDispose](); + } +}); + it('hands an invalid command line to in-process Rush when the configuration changed while the daemon started', async () => { const fixture: DaemonGraphTestFixture = await DaemonGraphTestFixture.createAsync((created) => { // A build that has watch phases gets --no-ipc when experiments.json turns on useIPCScriptsInWatchMode. From d719e2dc2b7fc3d0010a7e0978f7e63ab8601eec Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 05:04:02 +0000 Subject: [PATCH 083/265] [package-deps-hash] Test more RepoStateCache changes Swarm integration step 68; original commit ddf37c86ed (merge of swarm/r07-t114-nits at c4b24fad18). Scope: task 114 NITs 1-4. Brings r07's test-only fix for ch01's NITs 1-4 on task 114 (board 3099, board 3169): more RepoStateCache tests in package-deps-hash. No source changes. ch01 checked them with mutants P114N (4 of 4 killed) in the gate. s17 batch D2, item 6 of 6. Gate: ch01 GATE OK board 3514 (tree 0466deb3dd) Commits folded into this step (1): - c4b24fad18 [package-deps-hash] Test more RepoStateCache changes that only the file list, the change time or a new configuration reveal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- ...ate-cache-more-tests_2026-09-29-03-15.json | 11 +++ .../src/test/RepoStateCache.test.ts | 97 +++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 common/changes/@rushstack/package-deps-hash/repo-state-cache-more-tests_2026-09-29-03-15.json diff --git a/common/changes/@rushstack/package-deps-hash/repo-state-cache-more-tests_2026-09-29-03-15.json b/common/changes/@rushstack/package-deps-hash/repo-state-cache-more-tests_2026-09-29-03-15.json new file mode 100644 index 0000000000..0765baefd2 --- /dev/null +++ b/common/changes/@rushstack/package-deps-hash/repo-state-cache-more-tests_2026-09-29-03-15.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/package-deps-hash", + "comment": "Test that `RepoStateCache` follows a checkout between commits whose files have the same sizes, a rewrite that only the change time of a file reveals, and each change to the configuration while it may still change.", + "type": "none" + } + ], + "packageName": "@rushstack/package-deps-hash", + "email": "selarkin@microsoft.com" +} diff --git a/libraries/package-deps-hash/src/test/RepoStateCache.test.ts b/libraries/package-deps-hash/src/test/RepoStateCache.test.ts index a971b1866f..6f7799802d 100644 --- a/libraries/package-deps-hash/src/test/RepoStateCache.test.ts +++ b/libraries/package-deps-hash/src/test/RepoStateCache.test.ts @@ -322,6 +322,50 @@ describe(RepoStateCache.name, () => { expect(state.files.get('untracked.txt')).toBe(hashText('two\n')); }); + it('hashes a file that changed recently again even when only its change time is recent', async () => { + settleFiles(); + writeFile('untracked.txt', 'one\n'); + const filePath: string = path.join(repoPath, 'untracked.txt'); + // As after a write that put back the modification time of the file. The stamp stays the same when the file + // changes again within the granularity of the file times. + const recentTimeNs: bigint = BigInt(Date.now()) * BigInt(1e6); + const oldTimeNs: bigint = BigInt(originalDateNow() - 100000) * BigInt(1e6); + const recentStats: fs.BigIntStats = Object.create(fs.lstatSync(filePath, { bigint: true }), { + mtimeNs: { value: oldTimeNs }, + ctimeNs: { value: recentTimeNs } + }); + const lstatAsync: typeof fs.promises.lstat = fs.promises.lstat; + jest + .spyOn(fs.promises, 'lstat') + .mockImplementation(async (lstatPath: fs.PathLike, options?: fs.StatOptions) => + lstatPath === filePath ? recentStats : await lstatAsync(lstatPath, options) + ); + await getStateAsync(['untracked.txt']); + + writeFile('untracked.txt', 'two\n'); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(['untracked.txt']); + expect(takeGitCommandNames()).toEqual(['hash-object', 'hash-object', 'status']); + expect(state.files.get('untracked.txt')).toBe(hashText('two\n')); + }); + + it('hashes a settled file again when it changes at the same size and its modification time is put back', async () => { + settleFiles(); + const filePath: string = path.join(repoPath, 'untracked.txt'); + const time: number = Math.floor(originalDateNow() / 1000) - 100; + writeFile('untracked.txt', 'one\n'); + fs.utimesSync(filePath, time, time); + await getStateAsync(['untracked.txt']); + + // Only the change time of the file reveals the change, since userspace can't set it + writeFile('untracked.txt', 'two\n'); + fs.utimesSync(filePath, time, time); + takeGitCommands(); + const state: IDetailedRepoState = await getStateAsync(['untracked.txt']); + expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); + expect(state.files.get('untracked.txt')).toBe(hashText('two\n')); + }); + it('copies the index again when the files that it records change', async () => { settleFiles(); const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); @@ -348,6 +392,30 @@ describe(RepoStateCache.name, () => { expect(writeFileSpy).toHaveBeenCalledTimes(3); }); + it('follows a checkout between commits that record the same files with the same sizes', async () => { + settleFiles(); + const firstCommit: string = runGit('rev-parse', 'HEAD').trim(); + writeFile('a.txt', 'A\n'); + commit(); + const secondCommit: string = runGit('rev-parse', 'HEAD').trim(); + let state: IDetailedRepoState = await getStateAsync(); + expect(state.files.get('a.txt')).toBe(hashText('A\n')); + + // The working tree stays clean, so "git status" reports the same output each time. Only the list of the + // files in the index tells the states apart. + runGit('reset', '--quiet', '--hard', firstCommit); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.hasUncommittedChanges).toBe(false); + expect(state.files.get('a.txt')).toBe(hashText('a\n')); + + runGit('checkout', '--quiet', secondCommit); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.hasUncommittedChanges).toBe(false); + expect(state.files.get('a.txt')).toBe(hashText('A\n')); + }); + it('lists the files in the copy of the index while "git status" refreshes it', async () => { await getStateAsync(); writeFile('a.txt', 'modified\n'); @@ -467,6 +535,35 @@ describe(RepoStateCache.name, () => { expect(takeGitCommandNames()).toEqual(['hash-object', 'status']); }); + it('copies the index again on each call while the configuration of the repository may still change', async () => { + runGit('config', 'core.autocrlf', 'true'); + // Git writes the file with CRLF line endings + fs.unlinkSync(path.join(repoPath, 'a.txt')); + runGit('checkout', '--', 'a.txt'); + // Git refreshes the recorded times of the file in the copy of the index, but not in the index + touchSettledFile('a.txt'); + // The configuration may change again without changing its stamp + unsettleFiles(); + const writeFileSpy: jest.SpyInstance = jest.spyOn(fs.promises, 'writeFile'); + let state: IDetailedRepoState = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(hashText('a\n')); + expect(writeFileSpy).toHaveBeenCalledTimes(1); + + const changes: [string, string][] = [ + ['false', 'a\r\n'], + ['true', 'a\n'] + ]; + for (const [autocrlf, content] of changes) { + runGit('config', 'core.autocrlf', autocrlf); + state = await getStateAsync(); + await expectUncachedStateAsync(state); + expect(state.files.get('a.txt')).toBe(hashText(content)); + } + + expect(writeFileSpy).toHaveBeenCalledTimes(3); + }); + it('copies the index again when a .gitattributes file changes, after computing the state without the cache', async () => { settleFiles(); writeFile('crlf.txt', 'a\r\n'); From 26f85bab59cc92f55f496d2dd1bdee88f6a4aa86 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 06:15:45 +0000 Subject: [PATCH 084/265] [rush-lib] The daemon can keep warm heft run-watch workers for incremental builds, off by default Swarm integration step 69; original commit 053872af71 (merge of o07/l1b-on-c at 7088556062). Scope: task 107. Brings o07's task 107, L1b stage 1 (board 2762; landing candidate board 3259): with `daemon.warmWorkers` (or RUSH_DAEMON_WARM_WORKERS=1) and `daemon.incrementalBuilds`, the daemon keeps a `heft run-watch` worker alive for each operation whose rush-project.json settings set `allowDaemonWarmWorker`, and sends it the next incremental run, as rush start does. It is off by default, and swarm-dogfood sets none of these. 7088556062 is dd57043877 (board 3166) plus two commits for the verdicts' NITs. Second agent: t03 CONFIRMED board 2898. s17 batch D3, item 1 of 8. Gate: ch01 GATE OK board 3747 (tree 7bdb42fe64) Commits folded into this step (16): - cc3ad016da [heft-lint-plugin] Add a lintInWatchMode option - 6e403a9135 [rush-lib] Let incremental guard callers accept bundled outputs and skip build cache reads - 45565a0c3e [rush-lib] Keep warm workers alive between Rush daemon builds - 1a651f544b [rush-lib] Require an opt-in for Rush daemon warm workers - cfd3107c10 [rush-lib] Run the initial command after a failed run on a reused warm worker - 7cca36dd3a [heft-typescript-plugin] Never throw from the realpath function for missing paths - 2f8fd2cca0 [rush-lib] Report why a warm worker closed if the build cache restores its operation - 565e84c5bb [rush-lib] Say when a warm worker was closed between builds - e6bc0d81a5 [rush-lib] Drop an unreachable exception from the incremental guard's initial-only marks - 7a89294333 [rush-lib] Start warm workers with a TypeScript file watcher that sees replaced files - 26c35d188d [rush-lib] Compare content-hashed bundles when checking a warm worker's outputs - 31182e0002 [rush-lib] Stop using a warm worker after it renamed a content-hashed output - 89d541e9ab [rush-lib] Run the initial command after a warm worker's input folder was recreated - c2b58c25ad Document two more costs of Rush daemon warm workers - 085e3a6e69 Name only the top recreated input folders in the guard's reason - 7088556062 Document warm runs that succeed after a missing file, and what Heft's watchers miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 1 + .../rush/rushd-warm-workers_2026-09-28.json | 11 + .../lint-in-watch-mode_2026-09-28.json | 10 + .../realpath-missing-paths_2026-09-28.json | 10 + .../rushd-warm-workers_2026-09-28.json | 11 + common/reviews/api/rush-lib.api.md | 13 +- docs/rush/dogfooding-rush-daemon.md | 73 +- docs/rush/environment-variables.md | 1 + .../heft-lint-plugin/src/LintPlugin.ts | 28 +- .../src/schemas/heft-lint-plugin.schema.json | 6 + .../src/test/LintPlugin.test.ts | 245 +++++ .../src/loadTypeScriptTool.ts | 4 +- .../rush-lib/src/api/DaemonConfiguration.ts | 13 +- .../src/api/EnvironmentConfiguration.ts | 3 + .../src/api/RushProjectConfiguration.ts | 13 + .../src/api/test/DaemonConfiguration.test.ts | 19 +- .../RushProjectConfiguration.test.ts.snap | 1 + .../api/test/jsonFiles/rush-project-base.json | 3 +- .../cli/scriptActions/PhasedScriptAction.ts | 11 +- libraries/rush-lib/src/index.ts | 1 + .../operations/CacheableOperationPlugin.ts | 9 +- .../operations/DaemonWarmWorkerPlugin.ts | 127 +++ .../IncrementalExecutionGuardPlugin.ts | 173 +++- .../operations/IncrementalExecutionState.ts | 46 +- .../operations/OperationExecutionRecord.ts | 8 +- .../operations/OperationOutputManifest.ts | 48 +- .../logic/operations/ShellOperationRunner.ts | 34 +- .../operations/WarmWorkerOperationRunner.ts | 753 ++++++++++++++ ...ableOperationPluginRetainedResults.test.ts | 32 +- .../IncrementalExecutionGuardPlugin.test.ts | 211 +++- .../test/OperationExecutionRecord.test.ts | 16 + .../test/OperationOutputManifest.test.ts | 28 + .../test/WarmWorkerOperationRunner.test.ts | 936 ++++++++++++++++++ .../src/schemas/rush-project.schema.json | 5 + .../rush-lib/src/schemas/rush.schema.json | 5 + 35 files changed, 2848 insertions(+), 60 deletions(-) create mode 100644 common/changes/@microsoft/rush/rushd-warm-workers_2026-09-28.json create mode 100644 common/changes/@rushstack/heft-lint-plugin/lint-in-watch-mode_2026-09-28.json create mode 100644 common/changes/@rushstack/heft-typescript-plugin/realpath-missing-paths_2026-09-28.json create mode 100644 common/changes/@rushstack/rush-cli-client/rushd-warm-workers_2026-09-28.json create mode 100644 heft-plugins/heft-lint-plugin/src/test/LintPlugin.test.ts create mode 100644 libraries/rush-lib/src/logic/operations/DaemonWarmWorkerPlugin.ts create mode 100644 libraries/rush-lib/src/logic/operations/WarmWorkerOperationRunner.ts create mode 100644 libraries/rush-lib/src/logic/operations/test/WarmWorkerOperationRunner.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index c3d3cf8ba8..aad50dc10e 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -359,6 +359,7 @@ keys and unknown `RUSH_DAEMON*` variables fail validation. | `watch` | `RUSH_DAEMON_WATCH` | false | Persistent host observation of requested warm projects; false keeps root/config guards only. Never schedules builds | | `usePersistentIpcRunners` | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | false | Enables explicit per-operation `daemonIpc` Node launchers for unsharded incremental daemon builds | | `incrementalBuilds` | `RUSH_DAEMON_INCREMENTAL_BUILDS` | true | Runs an operation's `:incremental` script instead of its initial script when only files it builds were edited since its last successful run in the daemon and its output folders are unchanged. Additions, deletions, renames, configuration, tool, environment and command-line changes, bundled outputs, cache restores and native Rush commands run the initial script. Incremental results are never written to the build cache | +| `warmWorkers` | `RUSH_DAEMON_WARM_WORKERS` | false | With `incrementalBuilds`, keeps a watch-mode worker (the `:incremental:ipc` script) alive between builds for each operation whose `rush-project.json` operation settings set `allowDaemonWarmWorker`, and sends it the next incremental run. Opt in only if that script runs every task and check that the initial script runs, for Heft including lint and API Extractor. When an incremental run is not allowed, the worker is closed and the initial script runs. Workers count toward `warmMemoryBudgetMB` and `warmSetMaxProjects`, so raise both to keep them alive | | `warmIdleTimeoutSeconds` | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | 300 | Idle runner, project-watcher and retained-result eviction | | `warmMemoryBudgetMB` | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | 512 | Best-effort sampled RSS budget in MiB, not a hard ceiling. Compared against whole-daemon RSS plus measured child RSS, so keep it above the daemon baseline (~130-190 MiB) | | `warmSetMaxProjects` | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | 20 | Best-effort limit on projects holding warm resources (active runners, watchers); retained results of resource-free projects do not count. Never trims requested execution | diff --git a/common/changes/@microsoft/rush/rushd-warm-workers_2026-09-28.json b/common/changes/@microsoft/rush/rushd-warm-workers_2026-09-28.json new file mode 100644 index 0000000000..786e12971c --- /dev/null +++ b/common/changes/@microsoft/rush/rushd-warm-workers_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@microsoft/rush", + "comment": "Add `daemon.warmWorkers` (or `RUSH_DAEMON_WARM_WORKERS=1`), off by default: with `daemon.incrementalBuilds`, the Rush daemon runs the incremental builds of an operation whose project defines a `:incremental:ipc` script, and whose rush-project.json operation settings set the new `allowDaemonWarmWorker` field, in a worker started from that script. It keeps the worker alive between builds, so that the next build sends it another run instead of starting the build tool again. Opt in only if that script runs every task and check that the initial script runs; for Heft, that means heft-lint-plugin's `lintInWatchMode` and API Extractor's `runInWatchMode`. When the initial script must run, the worker is closed first, and the initial script runs in a shell, or in a new worker if the project defines `:ipc`. If a run on a reused worker fails, or a worker exits during a run, the worker is closed and the initial script runs in the same build, so state that a worker kept from its earlier runs cannot fail an operation that the initial script passes. A worker is also closed after its operation fails. If a run on a worker changes which output files exist, the initial script runs after it; if the run added or removed a file with a content hash in its name, as a bundler does when a chunk's content changes, the operation runs only its initial script until the daemon replaces its engine. A worker starts with `TSC_WATCHFILE=UseFsEventsOnParentDirectory` unless the operation's environment sets `TSC_WATCHFILE`, because TypeScript's default file watcher can lose track of a file that Git replaced, and the worker would then miss later edits to it. For the same reason, the initial script runs if a folder that held the operation's input files was deleted or recreated since the last run on its worker, e.g. by a branch switch, even if the files kept their content. The build cache is not read for a run on a live worker, and results of incremental runs are never written to it. A worker is closed after 25 runs or once its memory doubles, and workers count toward the warm set's memory and project limits.", + "type": "none" + } + ], + "packageName": "@microsoft/rush", + "email": "selarkin@microsoft.com" +} diff --git a/common/changes/@rushstack/heft-lint-plugin/lint-in-watch-mode_2026-09-28.json b/common/changes/@rushstack/heft-lint-plugin/lint-in-watch-mode_2026-09-28.json new file mode 100644 index 0000000000..3cde9768fa --- /dev/null +++ b/common/changes/@rushstack/heft-lint-plugin/lint-in-watch-mode_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/heft-lint-plugin", + "comment": "Add a `lintInWatchMode` option. When it is set, each run in watch mode lints the files that changed and the files that had lint failures, as a run that is not in watch mode does.", + "type": "minor" + } + ], + "packageName": "@rushstack/heft-lint-plugin" +} diff --git a/common/changes/@rushstack/heft-typescript-plugin/realpath-missing-paths_2026-09-28.json b/common/changes/@rushstack/heft-typescript-plugin/realpath-missing-paths_2026-09-28.json new file mode 100644 index 0000000000..26e268297d --- /dev/null +++ b/common/changes/@rushstack/heft-typescript-plugin/realpath-missing-paths_2026-09-28.json @@ -0,0 +1,10 @@ +{ + "changes": [ + { + "packageName": "@rushstack/heft-typescript-plugin", + "comment": "Fix an issue where, with `onlyResolveSymlinksInNodeModules`, every build in watch mode failed with an ENOENT error after the `package.json` file of a dependency was rewritten, for example by Git.", + "type": "patch" + } + ], + "packageName": "@rushstack/heft-typescript-plugin" +} diff --git a/common/changes/@rushstack/rush-cli-client/rushd-warm-workers_2026-09-28.json b/common/changes/@rushstack/rush-cli-client/rushd-warm-workers_2026-09-28.json new file mode 100644 index 0000000000..4c19502788 --- /dev/null +++ b/common/changes/@rushstack/rush-cli-client/rushd-warm-workers_2026-09-28.json @@ -0,0 +1,11 @@ +{ + "changes": [ + { + "packageName": "@rushstack/rush-cli-client", + "comment": "Document the `warmWorkers` daemon setting and `RUSH_DAEMON_WARM_WORKERS`.", + "type": "none" + } + ], + "packageName": "@rushstack/rush-cli-client", + "email": "selarkin@microsoft.com" +} diff --git a/common/reviews/api/rush-lib.api.md b/common/reviews/api/rush-lib.api.md index 675d2b98ba..0891385d4b 100644 --- a/common/reviews/api/rush-lib.api.md +++ b/common/reviews/api/rush-lib.api.md @@ -322,6 +322,7 @@ export const EnvironmentVariableNames: { readonly RUSH_DAEMON_WATCH: "RUSH_DAEMON_WATCH"; readonly RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: "RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS"; readonly RUSH_DAEMON_INCREMENTAL_BUILDS: "RUSH_DAEMON_INCREMENTAL_BUILDS"; + readonly RUSH_DAEMON_WARM_WORKERS: "RUSH_DAEMON_WARM_WORKERS"; readonly RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: "RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS"; readonly RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS: "RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS"; readonly RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: "RUSH_DAEMON_WARM_MEMORY_BUDGET_MB"; @@ -535,6 +536,7 @@ export interface IDaemonConfigurationJson { readonly warmIdleTimeoutSeconds?: number; readonly warmMemoryBudgetMB?: number; readonly warmSetMaxProjects?: number; + readonly warmWorkers?: boolean; readonly watch?: boolean; } @@ -620,8 +622,13 @@ export interface IGlobalCommand extends IRushCommand { // @beta export interface IIncrementalExecutionGuard { - getBlockReasonAsync(): Promise; - verifyIncrementalResultAsync(): Promise; + getBlockReasonAsync(options?: IIncrementalExecutionGuardOptions): Promise; + verifyIncrementalResultAsync(options?: IIncrementalExecutionGuardOptions): Promise; +} + +// @beta +export interface IIncrementalExecutionGuardOptions { + readonly outputsMayBeBundles?: boolean; } // @public @@ -724,6 +731,7 @@ export interface _IOperationChildProcessReporter { export interface IOperationCommandExecution { readonly hasIncrementalCommand: boolean; readonly kind: 'initial' | 'incremental'; + readonly watchesInputs?: boolean; } // @alpha @@ -875,6 +883,7 @@ export interface IOperationRunnerContext { // @alpha (undocumented) export interface IOperationSettings { allowCobuildWithoutCache?: boolean; + allowDaemonWarmWorker?: boolean; daemonIpc?: IDaemonIpcConfiguration; dependsOnAdditionalFiles?: string[]; dependsOnEnvVars?: string[]; diff --git a/docs/rush/dogfooding-rush-daemon.md b/docs/rush/dogfooding-rush-daemon.md index ef9af40482..c674e6a78e 100644 --- a/docs/rush/dogfooding-rush-daemon.md +++ b/docs/rush/dogfooding-rush-daemon.md @@ -188,9 +188,10 @@ Remove the snapshot with `rm -rf common/temp/rush-daemon-dogfood` (or `rush purg `--watch`, `--install`, `--variant`, and `--node-diagnostic-dir` stay native, as do build event-hook scripts and reporter controls such as `--output`. Keep using ordinary Rush for `install`, `update`, and other commands. `rushx-client` keeps scripts attached to a TTY in-process. -- **No persistent Heft or TypeScript workers.** Each operation still starts its Heft process; the daemon - saves Rush startup and graph construction, not compilation. The `usePersistentIpcRunners`/`daemonIpc` - mode requires a bundled, self-contained worker entry point, and Heft is not packaged that way. +- **No persistent Heft or TypeScript workers by default.** Each operation still starts its Heft process; the + daemon saves Rush startup and graph construction, not compilation. The `usePersistentIpcRunners`/`daemonIpc` + mode requires a bundled, self-contained worker entry point, and Heft is not packaged that way. See + `daemon.warmWorkers` below for Heft workers that stay alive between builds. - **`:incremental` scripts.** With `daemon.incrementalBuilds` (on by default), an operation whose project defines a `_phase::incremental` script runs it instead of the initial script when only files that the operation builds were edited since its last successful run in the daemon. The operation log then says @@ -199,6 +200,72 @@ Remove the snapshot with `rm -rf common/temp/rush-daemon-dogfood` (or `rush purg command make it run the initial script, and the log says why (`Not using the incremental command because ...`). Incremental results are never written to the build cache, and neither are the results of operations built against them. Set `RUSH_DAEMON_INCREMENTAL_BUILDS=0` to always run the initial script. +- **Warm workers.** With `daemon.warmWorkers` (or `RUSH_DAEMON_WARM_WORKERS=1`, off by default) and + `incrementalBuilds`, an operation whose project defines a `_phase::incremental:ipc` script (for Heft, + `heft run-watch --only --`) and whose `rush-project.json` operation settings set + `"allowDaemonWarmWorker": true` runs its incremental builds in a worker started from that script. Set it only + if the script, run in watch mode, runs every task and check that the initial script runs: the daemon reports + a run on the worker as if the initial script had run, and `rush start` may run the same script with fewer + tasks on purpose. Heft skips lint and API Extractor in watch mode unless the lint task sets heft-lint-plugin's + `lintInWatchMode` option and `config/api-extractor-task.json` sets `runInWatchMode`; without them, a lint + error does not fail the build, and consumers read a stale `.d.ts` rollup. A rig can set all three for its + projects. The worker stays alive between builds and keeps its last build in memory, so the next build sends + it another run instead of starting Heft again. The operation log says `Invoking (incremental): ...`, followed + by `Starting a warm worker for it.` or `Sending run to the warm worker (pid ).`. When the rules above + require the initial script, the worker is closed first (`Closing the warm worker (pid ), because ...`), + and that line is written even if the build cache then restores the operation. The initial script then runs + as usual, or in a new worker if the project defines `_phase::ipc` (for Heft, + `heft run-watch --only -- --clean`). A worker also accepts bundled outputs, because it keeps its + bundler state in memory; if a run changes which output files exist, the initial script runs after it. If the + run added or removed a file with a content hash in its name, as a bundler does when a chunk's content changes, + the operation runs only its initial script from then on, until a request replaces the engine, because each + later edit would rename the chunk again and leave the old one behind. + A worker starts with `TSC_WATCHFILE=UseFsEventsOnParentDirectory` unless the operation's environment sets + `TSC_WATCHFILE`. On Linux and macOS, TypeScript's default file watcher follows each file's inode, and after Git + replaces a file it can lose track of that file for the life of the process, so the worker would miss every later + edit to the file and keep its stale outputs, with no error. TypeScript reads the variable only if + `tsconfig.json` does not set `watchOptions.watchFile`. The variable's watcher fails loudly instead: on Linux, + if a source file is missing when a run on the worker updates the TypeScript program, for example because a + checkout deleted it during the build and then restored it, TypeScript never finds that file again in the + process. The run usually fails with error `TS6053` (file not found), or with `ENOENT` if a whole folder was + missing, and like any failed run it closes the worker (see below), so it costs a failed run but leaves no stale + outputs. But if Heft sees the change while the run is in progress, it runs its tasks again, and the last pass + decides the result. The run can then succeed with warnings, which fails the build unless the phase sets + `allowWarningsOnSuccess`, but nothing runs again and the worker stays alive. The next build does not reuse that + worker if a folder that held the operation's input files was recreated (see below), or if the build cache is + enabled for the operation, because input files that changed while an operation ran make its result + unverifiable. A single file that is deleted and restored keeps its folder, so neither check applies to it, but + a program that still misses the file reports `TS6053` again in the next run on the worker, which then fails + and closes the worker as above. + A watcher of a folder can also miss every later change in it once the folder is deleted and created again, for + example by a branch switch that removed the folder and then restored it. So the next build runs the initial + script if a folder that held the operation's input files was deleted or recreated since the last run on the + worker (`Not using the incremental command because folders that held its input files were deleted or + recreated since its last run (...)`), even if the files kept their content. The daemon compares each folder's + inode and creation time; a file that an editor saves by renaming a new file over it does not count. The + message names only the top folder of a recreated tree. + A worker's outputs are only as current as Heft's watchers. A change that a watcher misses is not built, and + because the run still succeeds, later builds report the operation as up to date and keep the stale outputs + until the file is edited again. The checks above cover the cases found in testing: a file that Git replaces, a + folder that is recreated, and a file that is missing while a run updates TypeScript. A case that none of them + covers would leave stale outputs without an error. + A reused worker can fail where the initial script passes, for example after Git rewrites the `package.json` of + a dependency, because some tools keep state between runs. So if a run on a reused worker fails, or the worker + exits during a run, the worker is closed and the initial script runs in the same build + (`Running the initial command, because the run on the warm worker failed.`), and the operation fails only if + the initial script fails too. A genuine error then costs an extra run of the initial script. The failed first + run of a new worker is reported as it stands, because that run had no earlier state. The build cache is not + read for a run on a live worker. A worker is closed after its operation fails, because the next build runs the + initial script, and after 25 runs or once its memory doubles. Builds that follow each other within seconds can + double a worker's memory in a few runs without a leak, because V8 collects each run's garbage later, while the + worker is idle: with API Extractor in watch mode, a small package's worker grew by 120 to 180 MB per run and was + replaced every 4 or 5 runs, but with 20 seconds between builds it stayed flat. Workers count toward + `warmMemoryBudgetMB` and `warmSetMaxProjects`. Over either limit, or once no build has included a project for + `warmIdleTimeoutSeconds`, the warm set closes the project's workers and drops its last results, so its next + build runs the initial script (`The warm worker (pid ) was closed after the operation last ran.`). The + default budget, 512 MiB, includes the daemon itself and leaves room for few workers, so raise both limits when + you turn workers on. A request that replaces the engine, e.g. a `rebuild` after a `build`, closes every worker + without a note, and each operation's next build runs the initial script. - **Plugins.** This repository's only configured plugin, `@rushstack/rush-published-versions-json-plugin`, is associated only with `record-published-versions` and is inert for builds. A plugin without `associatedCommands`, a plugin associated with `build` or `rebuild`, or a plugin command-line that defines diff --git a/docs/rush/environment-variables.md b/docs/rush/environment-variables.md index 09e141ae94..e89f8206d5 100644 --- a/docs/rush/environment-variables.md +++ b/docs/rush/environment-variables.md @@ -28,6 +28,7 @@ variables and invalid values are errors, not ignored settings. | `RUSH_DAEMON_WATCH` | `0` | Observe requested warm projects between requests. Never schedules builds and does not enable `--watch` mode. Root/config guards and request-time input reconciliation remain active when disabled. Overrides `watch`. | | `RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS` | `0` | Enable explicitly configured `operationSettings[].daemonIpc` Node workers for supported incremental daemon builds. Does not convert arbitrary shell scripts into persistent workers. Overrides `usePersistentIpcRunners`. | | `RUSH_DAEMON_INCREMENTAL_BUILDS` | `1` | Let daemon builds run an operation's `:incremental` script on top of the outputs of its last successful run in the daemon, when only files that it builds were edited and its output folders are unchanged. Otherwise the initial script runs, as it does for native Rush. `0` always runs the initial script. Incremental results are never written to the build cache. Overrides `incrementalBuilds`. | +| `RUSH_DAEMON_WARM_WORKERS` | `0` | With `RUSH_DAEMON_INCREMENTAL_BUILDS`, keep a watch-mode worker (the `:incremental:ipc` script) alive between daemon builds for each operation whose `rush-project.json` operation settings set `allowDaemonWarmWorker`, and send it the next incremental run. The worker keeps its last build in memory, so it rebuilds only what changed. When an incremental run is not allowed, the worker is closed and the initial script runs, in a new worker if the project defines `:ipc`. Workers count toward `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` and `RUSH_DAEMON_WARM_SET_MAX_PROJECTS`. Over either limit the daemon closes them and drops their projects' last results, so raise both for the workers to stay alive. Overrides `warmWorkers`. | | `RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS` | `300` | Idle expiration for retained runners and project watchers, together with those projects' results. Results of resource-free (shell/null) projects do not expire. Positive seconds, at most 2147483.647. Overrides `warmIdleTimeoutSeconds`. | | `RUSH_DAEMON_WARM_MEMORY_BUDGET_MB` | `512` | Best-effort sampled RSS budget in MiB. Positive number, at most 9007199254740991. Not a hard process-tree memory ceiling; active/protected work and results of resource-free projects are exempt. Overrides `warmMemoryBudgetMB`. | | `RUSH_DAEMON_WARM_SET_MAX_PROJECTS` | `20` | Best-effort retained-project limit. Positive safe integer, at most 9007199254740991; never trims the requested execution set. Overrides `warmSetMaxProjects`. | diff --git a/heft-plugins/heft-lint-plugin/src/LintPlugin.ts b/heft-plugins/heft-lint-plugin/src/LintPlugin.ts index b62d6afcea..b1b4c82551 100644 --- a/heft-plugins/heft-lint-plugin/src/LintPlugin.ts +++ b/heft-plugins/heft-lint-plugin/src/LintPlugin.ts @@ -32,6 +32,7 @@ const FIX_PARAMETER_NAME: string = '--fix'; interface ILintPluginOptions { alwaysFix?: boolean; + lintInWatchMode?: boolean; sarifLogPath?: string; } @@ -64,6 +65,10 @@ function checkFix(taskSession: IHeftTaskSession, pluginOptions?: ILintPluginOpti return fix; } +function getConfigFilePath(tsProgram: IExtendedProgram): string | undefined { + return tsProgram.getCompilerOptions().configFilePath as string | undefined; +} + function getSarifLogPath( heftConfiguration: HeftConfiguration, pluginOptions?: ILintPluginOptions @@ -87,9 +92,10 @@ export default class LintPlugin implements IHeftTaskPlugin { heftConfiguration: HeftConfiguration, pluginOptions?: ILintPluginOptions ): void { - // Disable linting in watch mode. Some lint rules require the context of multiple files, which - // may not be available in watch mode. - if (taskSession.parameters.watch) { + // Unless the "lintInWatchMode" option is set, disable linting in watch mode. Some lint rules require the + // context of multiple files, which may not be available in watch mode. + const { watch } = taskSession.parameters; + if (watch && !pluginOptions?.lintInWatchMode) { let warningPrinted: boolean = false; taskSession.hooks.run.tapPromise(PLUGIN_NAME, async () => { if (warningPrinted) { @@ -111,6 +117,10 @@ export default class LintPlugin implements IHeftTaskPlugin { // Use the changed files hook to collect the files and programs from TypeScript let typescriptChangedFiles: [IExtendedProgram, ReadonlySet][] = []; + // In watch mode, TypeScript reports a program only in the runs in which it emits that program. The latest + // program of each tsconfig file is linted again in the other runs, so that files with lint failures are + // reported again, as they would be by a run that is not in watch mode. + const lastProgramByConfigFilePath: Map = new Map(); taskSession.requestAccessToPluginByName( TYPESCRIPT_PLUGIN_PACKAGE_NAME, TYPESCRIPT_PLUGIN_NAME, @@ -137,6 +147,18 @@ export default class LintPlugin implements IHeftTaskPlugin { taskSession ); typescriptChangedFiles.push([tsProgram, new Set(tsProgram.getSourceFiles())]); + } else if (watch) { + const reportedConfigFilePaths: Set = new Set( + typescriptChangedFiles.map(([tsProgram]) => getConfigFilePath(tsProgram)) + ); + for (const [configFilePath, tsProgram] of lastProgramByConfigFilePath) { + if (!reportedConfigFilePaths.has(configFilePath)) { + typescriptChangedFiles.push([tsProgram, new Set()]); + } + } + for (const [tsProgram] of typescriptChangedFiles) { + lastProgramByConfigFilePath.set(getConfigFilePath(tsProgram), tsProgram); + } } // Run the linters to completion. Linters emit errors and warnings to the logger. diff --git a/heft-plugins/heft-lint-plugin/src/schemas/heft-lint-plugin.schema.json b/heft-plugins/heft-lint-plugin/src/schemas/heft-lint-plugin.schema.json index 072d2c70df..1304e3edc3 100644 --- a/heft-plugins/heft-lint-plugin/src/schemas/heft-lint-plugin.schema.json +++ b/heft-plugins/heft-lint-plugin/src/schemas/heft-lint-plugin.schema.json @@ -13,6 +13,12 @@ "type": "boolean" }, + "lintInWatchMode": { + "title": "Lint In Watch Mode", + "description": "If set to true, lint in watch mode too. Each run lints the files that TypeScript reports as changed and the files that had lint failures, as a run that is not in watch mode does. Defaults to false, which skips linting in watch mode.", + "type": "boolean" + }, + "sarifLogPath": { "title": "SARIF Log Path", "description": "If specified and using ESLint, a log describing the lint configuration and all messages (suppressed or not) will be emitted in the Static Analysis Results Interchange Format (https://sarifweb.azurewebsites.net/) at the provided path, relative to the project root.", diff --git a/heft-plugins/heft-lint-plugin/src/test/LintPlugin.test.ts b/heft-plugins/heft-lint-plugin/src/test/LintPlugin.test.ts new file mode 100644 index 0000000000..6fd9168142 --- /dev/null +++ b/heft-plugins/heft-lint-plugin/src/test/LintPlugin.test.ts @@ -0,0 +1,245 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +// ESLint 9 loads its configuration with a dynamic import, which Jest doesn't support without +// --experimental-vm-modules. This linter reports every file that contains "debugger" instead, and it keeps the +// real LinterBase, which decides which files to lint. +jest.mock('../Eslint', () => { + const { LinterBase } = jest.requireActual<{ LinterBase: typeof LinterBaseType }>('../LinterBase'); + + class MockEslint extends LinterBase { + public constructor(options: ILinterBaseOptions) { + super('eslint', options); + } + + public static async resolveEslintConfigFilePathAsync( + heftConfiguration: HeftConfiguration + ): Promise { + return `${heftConfiguration.buildFolderPath}/eslint.config.js`; + } + + public static async initializeAsync(options: ILinterBaseOptions): Promise { + return new MockEslint(options); + } + + public printVersionHeader(): void { + // Nothing to print + } + + protected async getCacheVersionAsync(): Promise { + return 'mock'; + } + + protected async lintFileAsync(sourceFile: IExtendedSourceFile | ISourceFileToLint): Promise { + return sourceFile.text.includes('debugger') ? [sourceFile.fileName] : []; + } + + protected async lintingFinishedAsync(lintResults: string[]): Promise { + for (const fileName of lintResults) { + this._scopedLogger.emitError(new Error(`(no-debugger) ${fileName}`)); + } + } + + protected hasLintFailures(lintResults: string[]): boolean { + return lintResults.length > 0; + } + + protected async isFileExcludedAsync(): Promise { + return false; + } + } + + return { Eslint: MockEslint }; +}); + +import path from 'node:path'; + +import * as ts from 'typescript'; + +import type { + HeftConfiguration, + IHeftTaskRunHookOptions, + IHeftTaskSession, + IScopedLogger +} from '@rushstack/heft'; +import type { IChangedFilesHookOptions, ITypeScriptPluginAccessor } from '@rushstack/heft-typescript-plugin'; +import { AlreadyReportedError, FileSystem } from '@rushstack/node-core-library'; +import { StringBufferTerminalProvider, Terminal } from '@rushstack/terminal'; + +import LintPlugin from '../LintPlugin'; +import type { ILinterBaseOptions, ISourceFileToLint, LinterBase as LinterBaseType } from '../LinterBase'; +import type { IExtendedProgram, IExtendedSourceFile } from '../internalTypings/TypeScriptInternals'; + +const PROJECT_FOLDER: string = path.resolve(__dirname, '../..'); +const FIXTURE_FOLDER: string = `${PROJECT_FOLDER}/temp/test/lint-plugin-watch`; +const FAILING_SOURCE: string = 'export function a() {\n debugger;\n}\n'; +const PASSING_SOURCE: string = 'export function a() {\n return;\n}\n'; + +interface IWatchSession { + readonly taskSession: IHeftTaskSession; + readonly heftConfiguration: HeftConfiguration; + readonly terminalProvider: StringBufferTerminalProvider; + readonly errors: Error[]; + readonly requestedPluginNames: string[]; + reportProgram(program: IExtendedProgram, changedFileNames: string[]): void; + runAsync(): Promise; +} + +function createWatchSession(): IWatchSession { + const terminalProvider: StringBufferTerminalProvider = new StringBufferTerminalProvider(); + const terminal: Terminal = new Terminal(terminalProvider); + const errors: Error[] = []; + const logger: Partial = { + terminal, + get hasErrors(): boolean { + return errors.length > 0; + }, + emitError: (error: Error) => errors.push(error), + emitWarning: () => undefined, + resetErrorsAndWarnings: () => { + errors.length = 0; + } + }; + + const runHooks: ((options: IHeftTaskRunHookOptions) => Promise)[] = []; + const changedFilesListeners: ((options: IChangedFilesHookOptions) => void)[] = []; + const requestedPluginNames: string[] = []; + const taskSession: Partial = { + logger: logger as IScopedLogger, + tempFolderPath: `${FIXTURE_FOLDER}/temp/lint`, + parameters: { + watch: true, + production: false, + getFlagParameter: () => ({ value: false }) + } as unknown as IHeftTaskSession['parameters'], + hooks: { + run: { + tapPromise: (name: string, fn: (options: IHeftTaskRunHookOptions) => Promise) => { + runHooks.push(fn); + } + } + } as unknown as IHeftTaskSession['hooks'], + requestAccessToPluginByName: (( + pluginPackageName: string, + pluginName: string, + callback: (accessor: ITypeScriptPluginAccessor) => void + ) => { + requestedPluginNames.push(pluginName); + callback({ + onChangedFilesHook: { + tap: (name: string, fn: (options: IChangedFilesHookOptions) => void) => { + changedFilesListeners.push(fn); + } + } + } as unknown as ITypeScriptPluginAccessor); + }) as IHeftTaskSession['requestAccessToPluginByName'] + }; + + const heftConfiguration: Partial = { + buildFolderPath: FIXTURE_FOLDER, + rigPackageResolver: { + resolvePackageAsync: async (packageName: string) => `${FIXTURE_FOLDER}/node_modules/${packageName}` + } as unknown as HeftConfiguration['rigPackageResolver'] + }; + + return { + taskSession: taskSession as IHeftTaskSession, + heftConfiguration: heftConfiguration as HeftConfiguration, + terminalProvider, + errors, + requestedPluginNames, + reportProgram: (program: IExtendedProgram, changedFileNames: string[]) => { + const changedFiles: Set = new Set( + changedFileNames.map((fileName: string) => program.getSourceFile(fileName)!) + ); + for (const listener of changedFilesListeners) { + listener({ program, changedFiles } as IChangedFilesHookOptions); + } + }, + // Like Heft's task runner: reset the logger, then run the task's run hooks. + runAsync: async () => { + logger.resetErrorsAndWarnings!(); + const options: IHeftTaskRunHookOptions = { + abortSignal: new AbortController().signal + } as IHeftTaskRunHookOptions; + await Promise.all(runHooks.map((runHook) => runHook(options))); + } + }; +} + +function writeSourceFile(name: string, text: string): string { + const filePath: string = `${FIXTURE_FOLDER}/src/${name}`; + FileSystem.writeFile(filePath, text, { ensureFolderExists: true }); + return filePath; +} + +function createProgram(rootNames: string[]): IExtendedProgram { + return ts.createProgram({ + rootNames, + options: { + configFilePath: `${FIXTURE_FOLDER}/tsconfig.json`, + noEmit: true, + types: [] + } + }) as IExtendedProgram; +} + +function getLintedFileCounts(verboseOutput: string): number[] { + return Array.from(verboseOutput.matchAll(/Lint: [\d.]+ms \((\d+) files\)/g), (match) => Number(match[1])); +} + +describe('LintPlugin in watch mode', () => { + beforeEach(() => { + FileSystem.ensureEmptyFolder(FIXTURE_FOLDER); + }); + + it('does not lint unless the lintInWatchMode option is set', async () => { + const session: IWatchSession = createWatchSession(); + new LintPlugin().apply(session.taskSession, session.heftConfiguration, {}); + + await session.runAsync(); + await session.runAsync(); + + expect(session.requestedPluginNames).toEqual([]); + expect(session.terminalProvider.getWarningOutput({ normalizeSpecialCharacters: false })).toBe( + "Linting isn't currently supported in watch mode\n" + ); + }); + + it('with the lintInWatchMode option, reports lint failures in every run until they are fixed', async () => { + const session: IWatchSession = createWatchSession(); + new LintPlugin().apply(session.taskSession, session.heftConfiguration, { lintInWatchMode: true }); + expect(session.requestedPluginNames).toEqual(['typescript-plugin']); + + const aPath: string = writeSourceFile('a.ts', FAILING_SOURCE); + const bPath: string = writeSourceFile('b.ts', 'export const b = 1;\n'); + + // The first run lints the files that TypeScript emitted. + session.reportProgram(createProgram([aPath, bPath]), [aPath, bPath]); + await expect(session.runAsync()).rejects.toBeInstanceOf(AlreadyReportedError); + expect(session.errors.map(String)).toEqual([expect.stringContaining('(no-debugger)')]); + + // TypeScript emits nothing, e.g. because only a file that it doesn't compile changed. The file with the + // lint failure is linted again, and the failure is reported again. + let verboseOutputLength: number = session.terminalProvider.getVerboseOutput().length; + await expect(session.runAsync()).rejects.toBeInstanceOf(AlreadyReportedError); + expect(session.errors.map(String)).toEqual([expect.stringContaining('(no-debugger)')]); + expect( + getLintedFileCounts(session.terminalProvider.getVerboseOutput().slice(verboseOutputLength)) + ).toEqual([1]); + + // Fixing the failure makes the next run pass. + writeSourceFile('a.ts', PASSING_SOURCE); + session.reportProgram(createProgram([aPath, bPath]), [aPath]); + await session.runAsync(); + expect(session.errors).toEqual([]); + + // Now every file is in the linter's cache, so a run in which TypeScript emits nothing lints no files. + verboseOutputLength = session.terminalProvider.getVerboseOutput().length; + await session.runAsync(); + expect(session.errors).toEqual([]); + expect( + getLintedFileCounts(session.terminalProvider.getVerboseOutput().slice(verboseOutputLength)) + ).toEqual([0]); + }, 60_000); +}); diff --git a/heft-plugins/heft-typescript-plugin/src/loadTypeScriptTool.ts b/heft-plugins/heft-typescript-plugin/src/loadTypeScriptTool.ts index 5fb9ad86fb..60d02a63ae 100644 --- a/heft-plugins/heft-typescript-plugin/src/loadTypeScriptTool.ts +++ b/heft-plugins/heft-typescript-plugin/src/loadTypeScriptTool.ts @@ -128,7 +128,9 @@ export async function loadTypeScriptToolAsync( let realpath: typeof ts.sys.realpath = ts.sys.realpath; if (onlyResolveSymlinksInNodeModules) { - const resolver: RealNodeModulePathResolver = new RealNodeModulePathResolver(); + // Like `ts.sys.realpath`, which returns the input for a path that does not exist, it must not throw: in watch + // mode, TypeScript also asks for the real path of module resolution lookups that failed. + const resolver: RealNodeModulePathResolver = new RealNodeModulePathResolver({ ignoreMissingPaths: true }); realpath = resolver.realNodeModulePath; } diff --git a/libraries/rush-lib/src/api/DaemonConfiguration.ts b/libraries/rush-lib/src/api/DaemonConfiguration.ts index 8edf596f7e..8bf935d082 100644 --- a/libraries/rush-lib/src/api/DaemonConfiguration.ts +++ b/libraries/rush-lib/src/api/DaemonConfiguration.ts @@ -20,6 +20,13 @@ export interface IDaemonConfigurationJson { * incremental script are not written to the build cache. Defaults to true. */ readonly incrementalBuilds?: boolean; + /** + * Keeps a watch-mode worker (the `:incremental:ipc` script) alive between daemon builds for each operation + * whose rush-project.json operation settings set `allowDaemonWarmWorker`, and sends it the next incremental run + * when `incrementalBuilds` allows one. Otherwise the worker is closed and the initial script runs. Requires + * `incrementalBuilds`. Defaults to false. + */ + readonly warmWorkers?: boolean; /** Maximum admission queue wait in seconds. Defaults to 30. */ readonly queueTimeoutSeconds?: number; /** @@ -53,6 +60,7 @@ const defaults: Required = { watch: false, usePersistentIpcRunners: false, incrementalBuilds: true, + warmWorkers: false, queueTimeoutSeconds: 30, warmIdleTimeoutSeconds: 300, warmMemoryBudgetMB: 512, @@ -70,6 +78,7 @@ export const daemonEnvironmentVariables: Readonly> ): boolean { diff --git a/libraries/rush-lib/src/api/EnvironmentConfiguration.ts b/libraries/rush-lib/src/api/EnvironmentConfiguration.ts index 926237dd35..723591e0f7 100644 --- a/libraries/rush-lib/src/api/EnvironmentConfiguration.ts +++ b/libraries/rush-lib/src/api/EnvironmentConfiguration.ts @@ -278,6 +278,8 @@ export const EnvironmentVariableNames = { RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: 'RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS', /** Lets daemon builds run an operation's guarded `:incremental` script outside watch mode. */ RUSH_DAEMON_INCREMENTAL_BUILDS: 'RUSH_DAEMON_INCREMENTAL_BUILDS', + /** Keeps `:incremental:ipc` watch-mode workers alive between daemon builds. */ + RUSH_DAEMON_WARM_WORKERS: 'RUSH_DAEMON_WARM_WORKERS', /** Overrides the request admission queue timeout. */ RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: 'RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS', /** Overrides idle eviction in an attached daemon warm set. */ @@ -697,6 +699,7 @@ export class EnvironmentConfiguration { case EnvironmentVariableNames.RUSH_DAEMON_WATCH: case EnvironmentVariableNames.RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: case EnvironmentVariableNames.RUSH_DAEMON_INCREMENTAL_BUILDS: + case EnvironmentVariableNames.RUSH_DAEMON_WARM_WORKERS: case EnvironmentVariableNames.RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS: case EnvironmentVariableNames.RUSH_DAEMON_WARM_IDLE_TIMEOUT_SECONDS: case EnvironmentVariableNames.RUSH_DAEMON_WARM_MEMORY_BUDGET_MB: diff --git a/libraries/rush-lib/src/api/RushProjectConfiguration.ts b/libraries/rush-lib/src/api/RushProjectConfiguration.ts index 17dc406e30..d8c539c21e 100644 --- a/libraries/rush-lib/src/api/RushProjectConfiguration.ts +++ b/libraries/rush-lib/src/api/RushProjectConfiguration.ts @@ -127,6 +127,19 @@ export interface IOperationSettings { */ daemonIpc?: IDaemonIpcConfiguration; + /** + * If true, and `daemon.warmWorkers` is enabled, the Rush daemon keeps the operation's `:incremental:ipc` + * script running as a warm worker between builds, and sends it the incremental runs that the incremental + * execution guard allows. + * + * @remarks + * Set this only if that script, run in watch mode, runs every task and check that the `` script runs, + * because the daemon reports a run on the worker as if the `` script had run. For Heft, that means the + * `lintInWatchMode` option of heft-lint-plugin and the `runInWatchMode` setting of API Extractor, which both + * default to skipping their task in watch mode. + */ + allowDaemonWarmWorker?: boolean; + /** * Specify the folders where this operation writes its output files. If enabled, the Rush build * cache will restore these folders from the cache. The strings are folder names under the project diff --git a/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts b/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts index af95f8278d..236f317d0c 100644 --- a/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts +++ b/libraries/rush-lib/src/api/test/DaemonConfiguration.test.ts @@ -12,6 +12,7 @@ describe('daemon configuration', () => { autoStart: true, usePersistentIpcRunners: false, incrementalBuilds: true, + warmWorkers: false, idleTimeoutSeconds: 900 }); expect( @@ -37,7 +38,8 @@ describe('daemon configuration', () => { { RUSH_DAEMON_AUTO_WARM_BY_TELEMETRY: '' }, { RUSH_DAEMON_EXPERIMENTAL: 'yes' }, { RUSH_DAEMON_USE_PERSISTENT_IPC_RUNNERS: 'yes' }, - { RUSH_DAEMON_INCREMENTAL_BUILDS: 'off' } + { RUSH_DAEMON_INCREMENTAL_BUILDS: 'off' }, + { RUSH_DAEMON_WARM_WORKERS: 'true' } ])('rejects invalid overrides %j', (environment) => { expect(() => resolveDaemonConfiguration({}, environment)).toThrow(); }); @@ -56,6 +58,7 @@ describe('daemon configuration', () => { { autoWarmByTelemetry: 1 }, { usePersistentIpcRunners: 'true' }, { incrementalBuilds: 'false' }, + { warmWorkers: 1 }, { compatiblePlugins: 'rush-example-plugin' }, { compatiblePlugins: [''] }, { compatiblePlugins: [' rush-example-plugin'] }, @@ -82,6 +85,14 @@ describe('daemon configuration', () => { ); }); + it('keeps warm workers only if the environment or configuration turns them on', () => { + expect(resolveDaemonConfiguration({ warmWorkers: true }, {}).warmWorkers).toBe(true); + expect( + resolveDaemonConfiguration({ warmWorkers: true }, { RUSH_DAEMON_WARM_WORKERS: '0' }).warmWorkers + ).toBe(false); + expect(resolveDaemonConfiguration({}, { RUSH_DAEMON_WARM_WORKERS: '1' }).warmWorkers).toBe(true); + }); + it('resolves compatible plugin names from the environment, then configuration, then no plugins', () => { expect(resolveDaemonConfiguration({}, {}).compatiblePlugins).toEqual([]); const configured: string[] = ['rush-a-plugin', 'rush-b-plugin']; @@ -97,8 +108,10 @@ describe('daemon configuration', () => { // An empty value is an explicit override that declares no plugins. for (const value of ['', ' ']) { expect( - resolveDaemonConfiguration({ compatiblePlugins: configured }, { RUSH_DAEMON_COMPATIBLE_PLUGINS: value }) - .compatiblePlugins + resolveDaemonConfiguration( + { compatiblePlugins: configured }, + { RUSH_DAEMON_COMPATIBLE_PLUGINS: value } + ).compatiblePlugins ).toEqual([]); } const resolved: readonly string[] = resolveDaemonConfiguration( diff --git a/libraries/rush-lib/src/api/test/__snapshots__/RushProjectConfiguration.test.ts.snap b/libraries/rush-lib/src/api/test/__snapshots__/RushProjectConfiguration.test.ts.snap index 8de9fe79d0..876fe54bae 100644 --- a/libraries/rush-lib/src/api/test/__snapshots__/RushProjectConfiguration.test.ts.snap +++ b/libraries/rush-lib/src/api/test/__snapshots__/RushProjectConfiguration.test.ts.snap @@ -41,6 +41,7 @@ exports[`RushProjectConfiguration operationSettingsByOperationName loads a rush- exports[`RushProjectConfiguration operationSettingsByOperationName loads a rush-project.json config that extends another config file 2`] = ` Map { "_phase:a" => Object { + "allowDaemonWarmWorker": true, "operationName": "_phase:a", "outputFolderNames": Array [ "a-a", diff --git a/libraries/rush-lib/src/api/test/jsonFiles/rush-project-base.json b/libraries/rush-lib/src/api/test/jsonFiles/rush-project-base.json index 2d9827001f..fcf2bb0c97 100644 --- a/libraries/rush-lib/src/api/test/jsonFiles/rush-project-base.json +++ b/libraries/rush-lib/src/api/test/jsonFiles/rush-project-base.json @@ -2,7 +2,8 @@ "operationSettings": [ { "operationName": "_phase:a", - "outputFolderNames": ["a-a"] + "outputFolderNames": ["a-a"], + "allowDaemonWarmWorker": true }, { "operationName": "_phase:b", diff --git a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts index 605c5ef016..a651ab08a1 100644 --- a/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts +++ b/libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts @@ -238,7 +238,9 @@ export class PhasedScriptAction extends BaseScriptAction i parameterShortName: '-p', argumentName: 'COUNT', // An engine host reads this default from the request's environment instead; see #getParallelism(). - environmentVariable: this.#engineEnvironment ? undefined : EnvironmentVariableNames.RUSH_PARALLELISM, + environmentVariable: this.#engineEnvironment + ? undefined + : EnvironmentVariableNames.RUSH_PARALLELISM, description: 'Specifies the maximum number of concurrent processes to launch during a build.' + ' The COUNT should be a positive integer, a percentage value (eg. "50%") or the word "max"' + @@ -688,6 +690,13 @@ export class PhasedScriptAction extends BaseScriptAction i /* webpackChunkName: 'IncrementalExecutionGuardPlugin' */ '../../logic/operations/IncrementalExecutionGuardPlugin' ); new IncrementalExecutionGuardPlugin().apply(this.hooks); + if (this.rushConfiguration.daemon.warmWorkers && !this.#noIPCParameter?.value) { + // Applied after DaemonIpcOperationRunnerPlugin, so that an explicit IPC tool keeps its runner. + const { DaemonWarmWorkerPlugin } = await import( + /* webpackChunkName: 'DaemonWarmWorkerPlugin' */ '../../logic/operations/DaemonWarmWorkerPlugin' + ); + new DaemonWarmWorkerPlugin().apply(this.hooks); + } } if (isWatch && this.#noIPCParameter?.value === false) { new ( diff --git a/libraries/rush-lib/src/index.ts b/libraries/rush-lib/src/index.ts index f9fa44afde..32811efbbb 100644 --- a/libraries/rush-lib/src/index.ts +++ b/libraries/rush-lib/src/index.ts @@ -166,6 +166,7 @@ export type { } from './logic/operations/IOperationRunner'; export type { IIncrementalExecutionGuard, + IIncrementalExecutionGuardOptions, IOperationCommandExecution } from './logic/operations/IncrementalExecutionState'; export type { diff --git a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts index 485ff10710..aec9ae0dba 100644 --- a/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts @@ -51,7 +51,7 @@ import type { BuildCacheConfiguration } from '../../api/BuildCacheConfiguration' import type { IConfigurableOperation, IOperationExecutionResult } from './IOperationExecutionResult'; import type { OperationExecutionRecord } from './OperationExecutionRecord'; import { enableUnverifiedRetainedOperations, markResultUnverifiable } from './RetainedResultVerification'; -import { wasExecutedIncrementally } from './IncrementalExecutionState'; +import { isBuildCacheReadSkipped, wasExecutedIncrementally } from './IncrementalExecutionState'; const PLUGIN_NAME: 'CacheablePhasedOperationPlugin' = 'CacheablePhasedOperationPlugin'; const PERIODIC_CALLBACK_INTERVAL_IN_SECONDS: number = 10; @@ -516,6 +516,9 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { } return !!restoreFromCacheSuccess; }; + // A runner that reuses outputs that it keeps in memory, e.g. a warm worker, can skip the read. + const isCacheReadAllowed: boolean = + buildCacheContext.isCacheReadAllowed && !isBuildCacheReadSkipped(record); if (cobuildLock) { // handling rebuilds. "rush rebuild" or "rush retest" command will save operations to // the build cache once completed, but does not retrieve them (since the "incremental" @@ -539,14 +542,14 @@ export class CacheableOperationPlugin implements IPhasedCommandPlugin { if (restoreFromCacheSuccess) { return status; } - } else if (!buildCacheContext.isCacheReadAttempted && buildCacheContext.isCacheReadAllowed) { + } else if (!buildCacheContext.isCacheReadAttempted && isCacheReadAllowed) { const restoreFromCacheSuccess: boolean = await restoreCacheAsync(operationBuildCache); if (restoreFromCacheSuccess) { return OperationStatus.FromCache; } } - } else if (buildCacheContext.isCacheReadAllowed) { + } else if (isCacheReadAllowed) { const restoreFromCacheSuccess: boolean = await restoreCacheAsync(operationBuildCache); if (restoreFromCacheSuccess) { diff --git a/libraries/rush-lib/src/logic/operations/DaemonWarmWorkerPlugin.ts b/libraries/rush-lib/src/logic/operations/DaemonWarmWorkerPlugin.ts new file mode 100644 index 0000000000..0af5e0f7f6 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/DaemonWarmWorkerPlugin.ts @@ -0,0 +1,127 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { + ICreateOperationsContext, + IOperationGraphContext, + IPhasedCommandPlugin, + PhasedCommandHooks +} from '../../pluginFramework/PhasedCommandHooks'; +import type { IOperationGraph } from './IOperationGraph'; +import type { IOperationExecutionResult } from './IOperationExecutionResult'; +import type { IOperationRunnerContext } from './IOperationRunner'; +import type { Operation } from './Operation'; +import type { OperationStatus } from './OperationStatus'; +import { + PLUGIN_NAME as ShellOperationPluginName, + formatCommand, + getCustomParameterValuesByOperation, + getDisplayName, + type ICustomParameterValuesForOperation +} from './ShellOperationRunnerPlugin'; +import { WarmWorkerOperationRunner } from './WarmWorkerOperationRunner'; + +const PLUGIN_NAME: 'DaemonWarmWorkerPlugin' = 'DaemonWarmWorkerPlugin'; + +// Before the default stage, so that a worker that must not be reused is closed before CacheableOperationPlugin +// restores the operation's outputs, and before the Rush daemon notes which runners were active. +const PREPARE_STAGE: number = -1; + +/** + * Runs the operations whose settings set `allowDaemonWarmWorker` and whose projects define a + * `:incremental:ipc` script in warm workers, see `WarmWorkerOperationRunner`. For the non-watch commands of + * the Rush daemon, with `IncrementalExecutionGuardPlugin`, which decides whether a worker may build on top of the + * outputs of its last run. + * + * @remarks + * The script alone is not enough, because `rush start` runs the same script in watch mode, where a project may + * intend to skip tasks such as lint. Operations that already have a runner, e.g. one for an explicit IPC tool, and + * operations with a shell command or shards are not changed. + */ +export class DaemonWarmWorkerPlugin implements IPhasedCommandPlugin { + public apply(hooks: PhasedCommandHooks): void { + hooks.createOperationsAsync.tap( + { + name: PLUGIN_NAME, + before: ShellOperationPluginName + }, + (operations: Set, context: ICreateOperationsContext): Set => { + const { isWatch, isIncrementalBuildAllowed } = context; + if (isWatch || !isIncrementalBuildAllowed) { + return operations; + } + + const getCustomParameterValues: (operation: Operation) => ICustomParameterValuesForOperation = + getCustomParameterValuesByOperation(); + + for (const operation of operations) { + const { associatedPhase: phase, associatedProject: project, runner, settings } = operation; + if ( + runner || + phase.shellCommand !== undefined || + settings?.sharding || + settings?.allowDaemonWarmWorker !== true + ) { + continue; + } + + const { scripts } = project.packageJson; + const { name: phaseName } = phase; + const initialScript: string | undefined = scripts?.[phaseName]; + const incrementalIpcScript: string | undefined = scripts?.[`${phaseName}:incremental:ipc`]; + if (!initialScript || !incrementalIpcScript) { + continue; + } + const initialIpcScript: string | undefined = scripts?.[`${phaseName}:ipc`]; + + const { parameterValues: customParameterValues, ignoredParameterValues } = + getCustomParameterValues(operation); + const initialCommand: string = formatCommand(initialScript, customParameterValues); + operation.runner = new WarmWorkerOperationRunner({ + phase, + rushProject: project, + displayName: getDisplayName(phase, project), + initialCommand, + initialIpcCommand: initialIpcScript + ? formatCommand(initialIpcScript, customParameterValues) + : undefined, + incrementalIpcCommand: formatCommand(incrementalIpcScript, customParameterValues), + // As for ShellOperationRunner, so that the build cache entries do not depend on warm workers. + commandForHash: initialCommand, + ignoredParameterValues + }); + } + + return operations; + } + ); + + hooks.onGraphCreatedAsync.tap(PLUGIN_NAME, (graph: IOperationGraph, context: IOperationGraphContext) => { + if (context.isWatch || !context.isIncrementalBuildAllowed) { + return; + } + graph.hooks.beforeExecuteOperationAsync.tapPromise( + { name: PLUGIN_NAME, stage: PREPARE_STAGE }, + async ( + record: IOperationRunnerContext & IOperationExecutionResult + ): Promise => { + const { operation } = record; + const { runner } = operation; + if (record.enabled && runner instanceof WarmWorkerOperationRunner) { + await runner.prepareAsync(record, graph.resultByOperation.has(operation)); + } + return undefined; + } + ); + graph.hooks.afterExecuteOperationAsync.tapPromise( + PLUGIN_NAME, + async (record: IOperationRunnerContext & IOperationExecutionResult): Promise => { + const { runner } = record.operation; + if (runner instanceof WarmWorkerOperationRunner) { + await runner.writeUnusedNotesAsync(record); + } + } + ); + }); + } +} diff --git a/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts b/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts index f643f8b135..7e9ac9856f 100644 --- a/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts +++ b/libraries/rush-lib/src/logic/operations/IncrementalExecutionGuardPlugin.ts @@ -2,6 +2,7 @@ // See LICENSE in the project root for license information. import { createHash, type Hash } from 'node:crypto'; +import * as fs from 'node:fs'; import * as path from 'node:path'; import { FileSystem, InternalError, Path } from '@rushstack/node-core-library'; @@ -24,10 +25,12 @@ import { NATIVE_COMMAND_INVALIDATION_REASON, setIncrementalExecutionGuard, type ICommandExecution, - type IIncrementalExecutionGuard + type IIncrementalExecutionGuard, + type IIncrementalExecutionGuardOptions } from './IncrementalExecutionState'; import { describeOutputFileChanges, + hasContentHashedOutputChange, readOperationOutputManifestAsync, type IOperationOutputManifest } from './OperationOutputManifest'; @@ -38,6 +41,9 @@ const PLUGIN_NAME: 'IncrementalExecutionGuardPlugin' = 'IncrementalExecutionGuar // Runs after the default-stage taps, e.g. CacheableOperationPlugin's input file checks, which can mark a result as // unverifiable. const RECORD_RESULT_STAGE: number = 1; +// Runs before those checks, so that a folder that was recreated before the input folders were read makes the result +// unverifiable, if the build cache checks the operation's input files. +const READ_INPUT_FOLDERS_STAGE: number = -1; const MAX_EXAMPLE_PATHS: number = 3; @@ -225,14 +231,19 @@ export function getIncrementalInputChangeReason( * failure, or a run that was interrupted or whose input files changed while it ran. A native Rush command that * ran in the workspace since then forgets every such result. * 2. Its inputs changed as {@link getIncrementalInputChangeReason} allows. - * 3. Its declared output folders hold the same files and folders as at the end of that run, and none of the folders + * 3. If that run was in a process that keeps watching the input files, such as a warm worker, none of the folders + * that held its input files was deleted or recreated since that run. A watcher can miss changes in such a folder, + * e.g. one that a branch switch recreated. + * 4. Its declared output folders hold the same files and folders as at the end of that run, and none of the folders * was recreated or had an entry added, removed or replaced since then. - * 4. Its outputs do not include bundles (JavaScript or CSS in a `dist` or `release` folder, or with a content hash in - * the name). A bundler can replace a chunk with a differently named one and leave the old one behind. + * 5. Its outputs do not include bundles (JavaScript or CSS in a `dist` or `release` folder, or with a content hash in + * the name). A bundler can replace a chunk with a differently named one and leave the old one behind. A runner + * whose incremental runs keep the previous build in memory can accept bundles with `outputsMayBeBundles`. * * When the incremental command succeeds, the operation's output files must be the files that it had before, or the - * initial command runs as well. Results of the incremental command are never written to the build cache (see - * `CacheableOperationPlugin`). + * initial command runs as well. The operation then never runs its incremental command again in this graph, unless + * the runner passed `outputsMayBeBundles` and no content-hashed output was added or removed. Results of the + * incremental command are never written to the build cache (see `CacheableOperationPlugin`). */ export class IncrementalExecutionGuardPlugin implements IPhasedCommandPlugin { public apply(hooks: PhasedCommandHooks): void { @@ -251,6 +262,9 @@ interface IOutputState { interface IIncrementalBase { readonly inputs: IIncrementalInputState; + // The identity of each folder that held an input file, if the last run was in a process that keeps watching the + // input files. Read before that run started, unless no check preceded it. + readonly inputFolders: ReadonlyMap | undefined; // Rejects if the output folders could not be read. readonly outputsPromise: Promise; } @@ -261,10 +275,13 @@ interface IRecordState { readonly getOperationEnvironment: IOperationGraphIterationOptions['getOperationEnvironment']; preRunOutputs?: IOperationOutputManifest; verifiedOutputs?: IOutputState; + inputFolders?: ReadonlyMap; } function applyToGraph(graph: IOperationGraph): void { const baseByOperation: Map = new Map(); + // Callers that pass `outputsMayBeBundles` add an entry only after their incremental command renamed a + // content-hashed output. An operation keeps its runner for the life of the graph. const cleanOnlyReasonByOperation: Map = new Map(); const stateByRecord: WeakMap = new WeakMap(); @@ -284,8 +301,10 @@ function applyToGraph(graph: IOperationGraph): void { const recordState: IRecordState = { records, inputsSnapshot, getOperationEnvironment }; stateByRecord.set(record, recordState); const guard: IIncrementalExecutionGuard = { - getBlockReasonAsync: () => getBlockReasonAsync(record, recordState), - verifyIncrementalResultAsync: () => verifyIncrementalResultAsync(record, recordState) + getBlockReasonAsync: (options?: IIncrementalExecutionGuardOptions) => + getBlockReasonAsync(record, recordState, options), + verifyIncrementalResultAsync: (options?: IIncrementalExecutionGuardOptions) => + verifyIncrementalResultAsync(record, recordState, options) }; setIncrementalExecutionGuard(record, guard); } @@ -294,7 +313,8 @@ function applyToGraph(graph: IOperationGraph): void { async function getBlockReasonAsync( record: IOperationExecutionResult, - recordState: IRecordState + recordState: IRecordState, + { outputsMayBeBundles = false }: IIncrementalExecutionGuardOptions = {} ): Promise { const { operation } = record; const cleanOnlyReason: string | undefined = cleanOnlyReasonByOperation.get(operation); @@ -305,6 +325,16 @@ function applyToGraph(graph: IOperationGraph): void { if (!base) { return 'its outputs were not built by a successful run of its own command in this process'; } + let changedFolders: string[] | undefined; + if (base.inputFolders) { + // Read before the command starts, so that the next check notices a folder that is recreated while it runs. + const inputFolders: ReadonlyMap = readInputFolderIdentities( + operation, + recordState.inputsSnapshot + ); + recordState.inputFolders = inputFolders; + changedFolders = getChangedFolders(base.inputFolders, inputFolders); + } const inputChangeReason: string | undefined = getIncrementalInputChangeReason( operation, @@ -319,6 +349,11 @@ function applyToGraph(graph: IOperationGraph): void { if (inputChangeReason) { return inputChangeReason; } + if (changedFolders?.length) { + return `folders that held its input files were deleted or recreated since its last run${formatExamplePaths( + changedFolders + )}`; + } const outputFolderNames: ReadonlyArray | undefined = operation.settings?.outputFolderNames; if (!outputFolderNames?.length) { @@ -330,7 +365,7 @@ function applyToGraph(graph: IOperationGraph): void { } catch (error) { return `its output folders could not be read after its last run: ${error}`; } - if (baseOutputs.cleanOnlyReason) { + if (baseOutputs.cleanOnlyReason && !outputsMayBeBundles) { cleanOnlyReasonByOperation.set(operation, baseOutputs.cleanOnlyReason); return baseOutputs.cleanOnlyReason; } @@ -348,7 +383,8 @@ function applyToGraph(graph: IOperationGraph): void { async function verifyIncrementalResultAsync( record: IOperationExecutionResult, - recordState: IRecordState + recordState: IRecordState, + { outputsMayBeBundles = false }: IIncrementalExecutionGuardOptions = {} ): Promise { const { operation } = record; const { preRunOutputs } = recordState; @@ -360,24 +396,39 @@ function applyToGraph(graph: IOperationGraph): void { operation.associatedProject.projectFolder, outputFolderNames ); + if (outputs.cleanOnlyReason && !outputsMayBeBundles) { + cleanOnlyReasonByOperation.set(operation, outputs.cleanOnlyReason); + return outputs.cleanOnlyReason; + } const changes: string | undefined = describeOutputFileChanges(preRunOutputs.files, outputs.files); if (changes) { - // Its outputs may be named after their content, so a later incremental run could leave stale files behind. - cleanOnlyReasonByOperation.set( - operation, - `its incremental command changed which output files it has in an earlier run: ${changes}` - ); + // Its outputs may be named after their content, so a later incremental run could leave stale files behind. A + // runner that keeps its build in memory can run again, unless a content-hashed output was renamed: a bundler + // renames it whenever its content changes, so the initial command would run after every incremental run. + if (!outputsMayBeBundles || hasContentHashedOutputChange(preRunOutputs.files, outputs.files)) { + cleanOnlyReasonByOperation.set( + operation, + `its incremental command changed which output files it has in an earlier run: ${changes}` + ); + } return `the incremental command changed which output files it has: ${changes}`; } - if (outputs.cleanOnlyReason) { - cleanOnlyReasonByOperation.set(operation, outputs.cleanOnlyReason); - return outputs.cleanOnlyReason; - } // eslint-disable-next-line require-atomic-updates -- The runner of the execution record calls the guard sequentially. recordState.verifiedOutputs = { signature: outputs.signature, cleanOnlyReason: undefined }; return undefined; } + graph.hooks.afterExecuteOperationAsync.tap( + { name: PLUGIN_NAME, stage: READ_INPUT_FOLDERS_STAGE }, + (record: IOperationRunnerContext & IOperationExecutionResult): void => { + const recordState: IRecordState | undefined = stateByRecord.get(record); + // E.g. the first run of the operation in this graph, which no check preceded + if (recordState && !recordState.inputFolders && getCommandExecution(record)?.watchesInputs) { + recordState.inputFolders = readInputFolderIdentities(record.operation, recordState.inputsSnapshot); + } + } + ); + graph.hooks.afterExecuteOperationAsync.tapPromise( { name: PLUGIN_NAME, stage: RECORD_RESULT_STAGE }, async (record: IOperationRunnerContext & IOperationExecutionResult): Promise => { @@ -427,6 +478,7 @@ function applyToGraph(graph: IOperationGraph): void { getEnvironment(record, recordState), recordState.records ), + inputFolders: execution.watchesInputs ? recordState.inputFolders : undefined, outputsPromise }); // Finish reading the outputs before anything else can change them, e.g. an operation that depends on this one. @@ -504,12 +556,87 @@ function describeInputFileChanges( if (changedFiles.length === 0) { return ''; } - changedFiles.sort(); - const examples: string = changedFiles + return formatExamplePaths(changedFiles); +} + +function formatExamplePaths(paths: string[]): string { + paths.sort(); + const examples: string = paths .slice(0, MAX_EXAMPLE_PATHS) .map((filePath: string) => JSON.stringify(filePath)) .join(', '); - return ` (${examples}${changedFiles.length > MAX_EXAMPLE_PATHS ? ', ...' : ''})`; + return ` (${examples}${paths.length > MAX_EXAMPLE_PATHS ? ', ...' : ''})`; +} + +/** + * Returns the identity of each folder of the operation's project that holds one of its input files, by the path of + * the folder relative to the root of the inputs snapshot. + */ +function readInputFolderIdentities( + operation: Operation, + inputsSnapshot: IInputsSnapshot +): ReadonlyMap { + const { associatedProject: project, associatedPhase: phase } = operation; + const projectPrefix: string = getProjectPrefix(inputsSnapshot, project); + const folderPaths: Set = new Set(); + for (const filePath of inputsSnapshot.getTrackedFileHashesForOperation(project, phase.name).keys()) { + if (filePath.startsWith(projectPrefix) && !path.isAbsolute(filePath)) { + folderPaths.add(filePath.slice(0, Math.max(filePath.lastIndexOf('/'), 0))); + } + } + const identities: Map = new Map(); + for (const folderPath of folderPaths) { + identities.set(folderPath, getFolderIdentity(path.resolve(inputsSnapshot.rootDirectory, folderPath))); + } + return identities; +} + +/** + * Returns the folders of `lastIdentities` whose identity changed, including folders that no longer hold input files. + * A changed folder inside another changed folder is left out, e.g. a recreated "src" is named without its subfolders. + */ +function getChangedFolders( + lastIdentities: ReadonlyMap, + currentIdentities: ReadonlyMap +): string[] { + const changedFolders: Set = new Set(); + for (const [folderPath, identity] of lastIdentities) { + if (currentIdentities.get(folderPath) !== identity) { + changedFolders.add(folderPath); + } + } + const topFolders: string[] = []; + for (const folderPath of changedFolders) { + if (!hasAncestorFolder(folderPath, changedFolders)) { + topFolders.push(folderPath); + } + } + return topFolders; +} + +// Folder paths are relative, with "/" separators, and "" is the root of the inputs snapshot. +function hasAncestorFolder(folderPath: string, folderPaths: ReadonlySet): boolean { + let parentPath: string = folderPath; + while (parentPath) { + parentPath = parentPath.slice(0, Math.max(parentPath.lastIndexOf('/'), 0)); + if (folderPaths.has(parentPath)) { + return true; + } + } + return false; +} + +// A folder that was deleted and recreated has another inode, or at least another birth time. Unlike its modification +// and status change times, neither changes when an entry of the folder is added, removed or replaced, e.g. by an +// editor that saves a file by renaming a new file over it. +function getFolderIdentity(folderPath: string): string { + try { + const stats: fs.Stats | undefined = fs.lstatSync(folderPath, { throwIfNoEntry: false }); + return stats ? `${stats.dev}:${stats.ino}:${stats.birthtimeMs}` : 'missing'; + } catch (error) { + // E.g. ENOTDIR, if a folder on its path was replaced by a file + return `${(error as NodeJS.ErrnoException).code}`; + } } /** diff --git a/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts b/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts index dea5f04106..6937c6bdfa 100644 --- a/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts +++ b/libraries/rush-lib/src/logic/operations/IncrementalExecutionState.ts @@ -18,6 +18,22 @@ export const INPUTS_CHANGED_INVALIDATION_REASON: 'workspace-inputs-changed' = 'w */ export const NATIVE_COMMAND_INVALIDATION_REASON: 'native-command-completed' = 'native-command-completed'; +/** + * Options that an operation runner passes to its {@link IIncrementalExecutionGuard}. + * + * @beta + */ +export interface IIncrementalExecutionGuardOptions { + /** + * Set by a runner whose incremental runs keep the previous build in memory, such as a watch-mode bundler in a + * warm worker. Outputs that look like bundles then do not require the initial command. If an incremental run + * changes which output files exist, that run is still followed by the initial command, but later runs of the + * operation may use the incremental command again, unless the run added or removed a content-hashed file. + * Defaults to false. + */ + readonly outputsMayBeBundles?: boolean; +} + /** * Decides whether an operation may run its `:incremental` command outside watch mode. * The Rush daemon registers one for each execution record. Runners get it from @@ -30,12 +46,12 @@ export interface IIncrementalExecutionGuard { * Returns `undefined` if the incremental command may run, otherwise why it may not, as a clause that completes * "Not using the incremental command because ...", e.g. `its command line changed`. */ - getBlockReasonAsync(): Promise; + getBlockReasonAsync(options?: IIncrementalExecutionGuardOptions): Promise; /** * Called after the incremental command succeeded. Returns `undefined` if its outputs can be kept, otherwise why * the initial command must run as well, as a clause that completes "Running the initial command, because ...". */ - verifyIncrementalResultAsync(): Promise; + verifyIncrementalResultAsync(options?: IIncrementalExecutionGuardOptions): Promise; } /** @@ -57,11 +73,19 @@ export interface IOperationCommandExecution { * Whether the runner has an incremental command that a later iteration could use. */ readonly hasIncrementalCommand: boolean; + /** + * Whether the command ran in a process that keeps watching the operation's input files after the command + * completed, to run the incremental command again, such as a warm worker. A watcher can miss changes in a folder + * that was deleted and recreated, so the next incremental run then requires that none of the folders that held + * input files was deleted or recreated since this run. Defaults to false. + */ + readonly watchesInputs?: boolean; } -// Both maps are keyed by the execution record, which is the runner's context and the hooks' argument. +// All are keyed by the execution record, which is the runner's context and the hooks' argument. const guardByRecord: WeakMap = new WeakMap(); const commandExecutionByRecord: WeakMap = new WeakMap(); +const recordsWithoutCacheRead: WeakSet = new WeakSet(); export function setIncrementalExecutionGuard(record: object, guard: IIncrementalExecutionGuard): void { guardByRecord.set(record, guard); @@ -91,3 +115,19 @@ export function getCommandExecution(record: object): ICommandExecution | undefin export function wasExecutedIncrementally(record: object): boolean { return commandExecutionByRecord.get(record)?.kind === 'incremental'; } + +/** + * Records that the runner of an execution record will run its incremental command without first trying to restore + * the operation from the build cache, e.g. because a restore would replace the outputs that a warm worker built and + * keeps in memory. Call it from a `beforeExecuteOperationAsync` tap that runs before `CacheableOperationPlugin`'s. + */ +export function skipBuildCacheRead(record: object): void { + recordsWithoutCacheRead.add(record); +} + +/** + * Returns true if `skipBuildCacheRead` was called for the execution record. + */ +export function isBuildCacheReadSkipped(record: object): boolean { + return recordsWithoutCacheRead.has(record); +} diff --git a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts index 310c99c409..71a9ef1dbd 100644 --- a/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts +++ b/libraries/rush-lib/src/logic/operations/OperationExecutionRecord.ts @@ -323,8 +323,12 @@ export class OperationExecutionRecord implements IOperationRunnerContext, IOpera /** * {@inheritdoc IOperationRunnerContext.reportCommandExecution} */ - public reportCommandExecution({ kind, hasIncrementalCommand }: IOperationCommandExecution): void { - setCommandExecution(this, { kind, hasIncrementalCommand }); + public reportCommandExecution({ + kind, + hasIncrementalCommand, + watchesInputs + }: IOperationCommandExecution): void { + setCommandExecution(this, { kind, hasIncrementalCommand, watchesInputs }); } public get silent(): boolean { diff --git a/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts b/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts index 006b1d6c05..adac51f772 100644 --- a/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts +++ b/libraries/rush-lib/src/logic/operations/OperationOutputManifest.ts @@ -118,8 +118,7 @@ export function getCleanOnlyReason(files: Iterable): string | undefined let bundleFile: string | undefined; for (const file of files) { const baseName: string = file.slice(file.lastIndexOf('/') + 1); - const match: RegExpExecArray | null = HASHED_BUNDLE_FILE_REGEXP.exec(baseName); - if (match && /\d/.test(match[1])) { + if (isContentHashedBundleFile(baseName)) { return `its outputs include the content-hashed file "${file}"`; } if (bundleFile === undefined && BUNDLE_FILE_REGEXP.test(baseName) && BUNDLE_FOLDER_REGEXP.test(file)) { @@ -129,9 +128,48 @@ export function getCleanOnlyReason(files: Iterable): string | undefined return bundleFile === undefined ? undefined : `its outputs include the bundle "${bundleFile}"`; } +function isContentHashedBundleFile(baseName: string): boolean { + const match: RegExpExecArray | null = HASHED_BUNDLE_FILE_REGEXP.exec(baseName); + return !!match && /\d/.test(match[1]); +} + +// A chunk or asset that a bundler named after its content, which it renames whenever the content changes. +function isContentHashedOutput(file: string): boolean { + return ( + isContentHashedBundleFile(file.slice(file.lastIndexOf('/') + 1)) || + (CONTENT_ADDRESSED_PATH_REGEXP.test(file) && BUNDLE_FOLDER_REGEXP.test(file)) + ); +} + +// A content-hashed file that a bundler emits is not a cache entry: a full build would not leave an old one behind. +function isCacheEntry(file: string): boolean { + return CONTENT_ADDRESSED_PATH_REGEXP.test(file) && !isContentHashedOutput(file); +} + +/** + * Whether a content-hashed chunk or asset, as bundlers emit, is among the files that were added or removed. + */ +export function hasContentHashedOutputChange( + before: ReadonlySet, + after: ReadonlySet +): boolean { + for (const file of after) { + if (!before.has(file) && isContentHashedOutput(file)) { + return true; + } + } + for (const file of before) { + if (!after.has(file) && isContentHashedOutput(file)) { + return true; + } + } + return false; +} + /** * Describes how two sets of output files differ, e.g. `2 added ("lib/a.js", ...), 1 removed ("lib/b.js")`. - * Files whose paths contain a content hash, such as cache entries, are ignored. + * Cache entries whose paths contain a content hash are ignored. Content-hashed bundles, and files in a `dist` or + * `release` folder, are not. */ export function describeOutputFileChanges( before: ReadonlySet, @@ -140,12 +178,12 @@ export function describeOutputFileChanges( const added: string[] = []; const removed: string[] = []; for (const file of after) { - if (!before.has(file) && !CONTENT_ADDRESSED_PATH_REGEXP.test(file)) { + if (!before.has(file) && !isCacheEntry(file)) { added.push(file); } } for (const file of before) { - if (!after.has(file) && !CONTENT_ADDRESSED_PATH_REGEXP.test(file)) { + if (!after.has(file) && !isCacheEntry(file)) { removed.push(file); } } diff --git a/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts b/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts index 9283810d9c..e4b827d42f 100644 --- a/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts +++ b/libraries/rush-lib/src/logic/operations/ShellOperationRunner.ts @@ -41,7 +41,10 @@ export interface IShellOperationRunnerOptions { ignoredParameterValues: ReadonlyArray; } -interface ICommandTerminals { +/** + * The terminals that `ShellOperationRunner.invokeCommandAsync` writes to. + */ +export interface ICommandTerminals { readonly terminal: ITerminal; readonly terminalProvider: ITerminalProvider; readonly structuredChildOutputTerminalProvider: ITerminalProvider; @@ -166,21 +169,40 @@ export class ShellOperationRunner implements IOperationRunner { async #invokeCommandAsync( context: IOperationRunnerContext, - { terminal, terminalProvider, structuredChildOutputTerminalProvider }: ICommandTerminals, + terminals: ICommandTerminals, kind: ICommandExecution['kind'], commandToRun: string ): Promise { - let hasWarningOrError: boolean = false; - if (this.#incrementalCommandRequiresGuard) { // Recorded before the command starts, so that outputs of a command that fails or is aborted are attributed to it. setCommandExecution(context, { kind, hasIncrementalCommand: this.#incrementalCommand !== undefined }); } + return await ShellOperationRunner.invokeCommandAsync( + context, + terminals, + this.#rushProject, + kind, + commandToRun + ); + } + + /** + * Runs a command of an operation in a shell, in the folder of its project, and returns the operation's status. + * It does not record the command execution for the incremental execution guard; the caller does that. + */ + public static async invokeCommandAsync( + context: IOperationRunnerContext, + { terminal, terminalProvider, structuredChildOutputTerminalProvider }: ICommandTerminals, + rushProject: RushConfigurationProject, + kind: ICommandExecution['kind'], + commandToRun: string + ): Promise { + let hasWarningOrError: boolean = false; // Run the operation terminal.writeLine(`Invoking (${kind}): ${commandToRun}`); - const { rushConfiguration, projectFolder } = this.#rushProject; + const { rushConfiguration, projectFolder } = rushProject; const { environment: initialEnvironment, abortSignal } = context; const childProcessReporter: IOperationChildProcessReporter | undefined = @@ -291,7 +313,7 @@ export class ShellOperationRunner implements IOperationRunner { /** * Returns what an incremental execution guard returned, or why it failed. */ -async function getGuardResultAsync( +export async function getGuardResultAsync( getResultAsync: () => Promise ): Promise { try { diff --git a/libraries/rush-lib/src/logic/operations/WarmWorkerOperationRunner.ts b/libraries/rush-lib/src/logic/operations/WarmWorkerOperationRunner.ts new file mode 100644 index 0000000000..065750fa86 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/WarmWorkerOperationRunner.ts @@ -0,0 +1,753 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +import type { ChildProcess } from 'node:child_process'; +import { StringDecoder } from 'node:string_decoder'; + +import type { + IAfterExecuteEventMessage, + IExitCommandMessage, + IRequestRunEventMessage, + IRunCommandMessage, + ISyncEventMessage +} from '@rushstack/operation-graph'; +import { SubprocessTerminator } from '@rushstack/node-core-library'; +import { TerminalProviderSeverity, type ITerminal, type ITerminalProvider } from '@rushstack/terminal'; + +import type { IPhase } from '../../api/CommandLineConfiguration'; +import { EnvironmentConfiguration } from '../../api/EnvironmentConfiguration'; +import type { RushConfigurationProject } from '../../api/RushConfigurationProject'; +import { Utilities, type IEnvironment } from '../../utilities/Utilities'; +import type { IOperationRunner, IOperationRunnerContext, IOperationLastState } from './IOperationRunner'; +import { OperationError } from './OperationError'; +import { OperationStatus } from './OperationStatus'; +import { + getIncrementalExecutionGuard, + setCommandExecution, + skipBuildCacheRead, + type ICommandExecution, + type IIncrementalExecutionGuard, + type IIncrementalExecutionGuardOptions +} from './IncrementalExecutionState'; +import { getGuardResultAsync, ShellOperationRunner, type ICommandTerminals } from './ShellOperationRunner'; + +const DEFAULT_MAX_RUNS_PER_WORKER: number = 25; +const DEFAULT_MAX_MEMORY_GROWTH: number = 2; +const DEFAULT_CHANGE_REPORT_TIMEOUT_MS: number = 1000; +const EXIT_TIMEOUT_MS: number = 10000; +const BYTES_PER_MB: number = 1024 * 1024; + +// A worker keeps its last build in memory and only writes the outputs that changed, so its outputs may be bundles. +const GUARD_OPTIONS: IIncrementalExecutionGuardOptions = { outputsMayBeBundles: true }; +const INITIAL_COMMAND_REASON: string = 'the initial command must run'; +const FAILED_OPERATION_REASON: string = 'the operation failed, so its next build runs the initial command'; + +// On Linux and macOS, TypeScript's default file watcher follows the inode of each file. After a file is replaced (git +// deletes and recreates each file that it writes), that watcher can lose track of the file for the life of the +// process, and a worker would then miss every later edit to the file and keep its stale outputs. A watcher on the +// file's folder sees the new file. TypeScript reads the variable only if the tsconfig.json does not set +// `watchOptions.watchFile`, and a value in the operation's environment is kept. +const TYPESCRIPT_WATCH_FILE_VARIABLE: string = 'TSC_WATCHFILE'; +const TYPESCRIPT_WATCH_FILE: string = 'UseFsEventsOnParentDirectory'; + +/** + * @internal + */ +export interface IWarmWorkerOperationRunnerOptions { + phase: IPhase; + rushProject: RushConfigurationProject; + displayName: string; + /** + * The initial command of the operation, which runs in a shell if `initialIpcCommand` is undefined. + */ + initialCommand: string; + /** + * The `:ipc` command of the operation, if it has one: a worker that builds from scratch and then waits for + * further runs. + */ + initialIpcCommand: string | undefined; + /** + * The `:incremental:ipc` command of the operation: a worker that builds on top of the existing outputs and + * then waits for further runs. + */ + incrementalIpcCommand: string; + commandForHash: string; + ignoredParameterValues: ReadonlyArray; + /** + * A worker is closed after this many runs. Defaults to 25. + */ + maxRunsPerWorker?: number; + /** + * A worker is closed once its resident memory is more than this multiple of its resident memory after its first + * run. Defaults to 2. + */ + maxMemoryGrowth?: number; + /** + * How long a reused worker may take to report a change that it saw, in milliseconds, before it is told to run + * anyway. A worker decides what to rebuild from the changes that its own file watcher reported, so a run that + * starts before the watcher saw an edit would not rebuild it. Defaults to 1000. + */ + changeReportTimeoutMs?: number; +} + +interface IRunPlan { + readonly kind: ICommandExecution['kind']; + // Printed to the operation's log before the command runs. + readonly notes: ReadonlyArray; +} + +interface IWorkerRunResult { + readonly status: OperationStatus; + readonly hasWarningOrError: boolean; +} + +interface IWorkerCommandResult { + readonly status: OperationStatus; + // Whether the run was sent to a worker that had run before, i.e. one that may keep state from earlier runs. + readonly wasWorkerReused: boolean; +} + +function isAfterExecuteEventMessage(message: unknown): message is IAfterExecuteEventMessage { + return ( + !!message && + typeof message === 'object' && + (message as IAfterExecuteEventMessage).event === 'after-execute' + ); +} + +function isRequestRunEventMessage(message: unknown): message is IRequestRunEventMessage { + return ( + !!message && typeof message === 'object' && (message as IRequestRunEventMessage).event === 'requestRun' + ); +} + +function isSyncEventMessage(message: unknown): message is ISyncEventMessage { + return !!message && typeof message === 'object' && (message as ISyncEventMessage).event === 'sync'; +} + +function formatMegabytes(bytes: number): string { + return `${Math.round(bytes / BYTES_PER_MB)} MB`; +} + +/** + * A long-lived process that runs the operation's command when it receives a "run" message, and keeps the state of + * its last build in memory between runs. + */ +class WarmWorker { + public readonly process: ChildProcess; + public readonly command: string; + public readonly closedPromise: Promise; + public readyPromise: Promise; + /** + * True once the process sent "sync", i.e. it supports the IPC protocol. + */ + public isReady: boolean = false; + /** + * True if the process reported a change since it was last told to run. + */ + public isRunRequested: boolean = false; + public runCount: number = 0; + public firstResidentMemoryBytes: number | undefined; + public residentMemoryBytes: number | undefined; + + #onRunRequested: (() => void) | undefined; + + public constructor(childProcess: ChildProcess, command: string) { + this.process = childProcess; + this.command = command; + this.closedPromise = new Promise((resolve: () => void) => { + childProcess.once('close', () => resolve()); + }); + let resolveReady!: () => void; + this.readyPromise = new Promise((resolve: () => void) => { + resolveReady = resolve; + }); + childProcess.on('message', (message: unknown) => { + if (isSyncEventMessage(message)) { + this.isReady = true; + resolveReady(); + } else if (isRequestRunEventMessage(message)) { + this.isRunRequested = true; + this.#onRunRequested?.(); + } + }); + // Without a listener, an 'error' event would be thrown; the 'close' event reports the outcome. + childProcess.on('error', () => undefined); + } + + public get pid(): number | undefined { + return this.process.pid; + } + + public get isAlive(): boolean { + return this.process.exitCode === null && this.process.signalCode === null; + } + + /** + * Waits until the process reports a change, for at most `timeoutMs`. Returns false if it did not. + */ + public async waitForRunRequestAsync( + timeoutMs: number, + abortSignal: AbortSignal | undefined + ): Promise { + if (this.isRunRequested || timeoutMs <= 0) { + return this.isRunRequested; + } + await new Promise((resolve: () => void) => { + const timer: { timeout?: NodeJS.Timeout } = {}; + const finish: () => void = () => { + clearTimeout(timer.timeout); + this.#onRunRequested = undefined; + abortSignal?.removeEventListener('abort', finish); + this.process.off('close', finish); + resolve(); + }; + timer.timeout = setTimeout(finish, timeoutMs); + this.#onRunRequested = finish; + abortSignal?.addEventListener('abort', finish, { once: true }); + this.process.once('close', finish); + }); + return this.isRunRequested; + } + + public terminate(): void { + if (this.isAlive) { + try { + SubprocessTerminator.killProcessTree(this.process, SubprocessTerminator.RECOMMENDED_OPTIONS); + } catch { + // It exited in the meantime. + } + } + } + + /** + * Asks the process to exit and waits until it closed. Terminates it if it cannot be asked, or if it does not + * exit in time. + */ + public async closeAsync(): Promise { + if (this.isAlive) { + if (this.process.connected) { + const exitCommand: IExitCommandMessage = { command: 'exit' }; + try { + this.process.send(exitCommand); + } catch { + this.terminate(); + } + const timeout: NodeJS.Timeout = setTimeout(() => this.terminate(), EXIT_TIMEOUT_MS); + await this.closedPromise; + clearTimeout(timeout); + } else { + this.terminate(); + } + } + // Even after it exited, the stdio of the process or of its descendants may still be draining. + await this.closedPromise; + } +} + +/** + * An `IOperationRunner` for the Rush daemon that runs the incremental builds of an operation in a warm worker: a + * long-lived process, started from the project's `:incremental:ipc` script, that keeps its last build in + * memory and rebuilds only what changed when it is told to run again. + * + * @remarks + * Whether a run may build on top of the outputs of the last one is decided by the operation's incremental + * execution guard, as for `ShellOperationRunner`, in `prepareAsync`. If the guard allows it and the worker is + * running, the build cache is not read, because restoring the outputs would not update the worker's memory. If the + * guard does not allow it, the worker is closed, and the initial command runs: in a new worker, if the project has + * a `:ipc` script, or else in a shell. The initial command also runs after a failed run on a reused worker, + * or on a worker that exited, so that state that a worker kept from its earlier runs cannot fail an operation that + * the initial command passes. The failed first run of a new worker is reported as it stands, because that run had + * no earlier state. After a failure, the worker is closed, because the next build runs the initial command. If + * `closeAsync` closes a worker between executions, e.g. for the warm set of the Rush daemon, the next execution + * says so. Results of incremental runs are never written to the build cache. + * + * @internal + */ +export class WarmWorkerOperationRunner implements IOperationRunner { + public readonly name: string; + public readonly reportTiming: boolean = true; + public readonly silent: boolean = false; + public readonly cacheable: boolean = true; + public readonly warningsAreAllowed: boolean; + public readonly isNoOp: boolean = false; + + readonly #rushProject: RushConfigurationProject; + readonly #initialCommand: string; + readonly #initialIpcCommand: string | undefined; + readonly #incrementalIpcCommand: string; + readonly #commandForHash: string; + readonly #ignoredParameterValues: ReadonlyArray; + readonly #maxRunsPerWorker: number; + readonly #maxMemoryGrowth: number; + readonly #changeReportTimeoutMs: number; + readonly #planByContext: WeakMap = new WeakMap(); + + #worker: WarmWorker | undefined; + // Resolves when the last worker that was closed has exited. + #lastClosePromise: Promise = Promise.resolve(); + // Set when `closeAsync` closes a worker between executions, e.g. for the warm set of the Rush daemon. The next + // plan writes it, because the next execution would otherwise not say why the worker is gone. + #closedWorkerNote: string | undefined; + + public constructor(options: IWarmWorkerOperationRunnerOptions) { + const { + phase, + rushProject, + displayName, + initialCommand, + initialIpcCommand, + incrementalIpcCommand, + commandForHash, + ignoredParameterValues, + maxRunsPerWorker = DEFAULT_MAX_RUNS_PER_WORKER, + maxMemoryGrowth = DEFAULT_MAX_MEMORY_GROWTH, + changeReportTimeoutMs = DEFAULT_CHANGE_REPORT_TIMEOUT_MS + } = options; + this.name = displayName; + this.warningsAreAllowed = + EnvironmentConfiguration.allowWarningsInSuccessfulBuild || phase.allowWarningsOnSuccess || false; + this.#rushProject = rushProject; + this.#initialCommand = initialCommand; + this.#initialIpcCommand = initialIpcCommand; + this.#incrementalIpcCommand = incrementalIpcCommand; + this.#commandForHash = commandForHash; + this.#ignoredParameterValues = ignoredParameterValues; + this.#maxRunsPerWorker = maxRunsPerWorker; + this.#maxMemoryGrowth = maxMemoryGrowth; + this.#changeReportTimeoutMs = changeReportTimeoutMs; + } + + public get isActive(): boolean { + return !!this.#worker?.isAlive; + } + + public get residentMemoryBytes(): number | undefined { + return this.isActive ? this.#worker!.residentMemoryBytes : undefined; + } + + /** + * The process ID of the worker, if one is running. + */ + public get workerPid(): number | undefined { + return this.isActive ? this.#worker!.pid : undefined; + } + + /** + * Decides how the next execution with this context runs. Call it before the build cache is read, i.e. from a + * `beforeExecuteOperationAsync` tap that runs before `CacheableOperationPlugin`'s, and call + * `writeUnusedNotesAsync` after the operation executed. + * + * @param context - The execution record of the operation + * @param hasLastState - Whether the operation has a result from an earlier execution in this graph + */ + public async prepareAsync(context: IOperationRunnerContext, hasLastState: boolean): Promise { + this.#planByContext.set(context, await this.#planAsync(context, hasLastState)); + } + + public async executeAsync( + context: IOperationRunnerContext, + lastState?: IOperationLastState + ): Promise { + const preparedPlan: IRunPlan | undefined = this.#planByContext.get(context); + this.#planByContext.delete(context); + // Without a last state, e.g. in an iteration that allows no incremental build, the initial command runs. + const plan: IRunPlan = + preparedPlan && (preparedPlan.kind === 'initial' || lastState) + ? preparedPlan + : await this.#planAsync(context, !!lastState); + return await context.runWithTerminalAsync( + async ( + terminal: ITerminal, + terminalProvider: ITerminalProvider, + structuredChildOutputTerminalProvider: ITerminalProvider + ): Promise => { + if (this.#ignoredParameterValues.length > 0) { + terminal.writeLine( + `These parameters were ignored for this operation by project-level configuration: ${this.#ignoredParameterValues.join(' ')}` + ); + } + for (const note of plan.notes) { + terminal.writeLine(note); + } + this.#closedWorkerNote = undefined; + + const terminals: ICommandTerminals = { + terminal, + terminalProvider, + structuredChildOutputTerminalProvider + }; + let status: OperationStatus; + if (plan.kind === 'initial') { + status = await this.#runInitialAsync(context, terminals); + } else { + const { status: workerStatus, wasWorkerReused } = await this.#runOnWorkerAsync( + context, + terminals, + 'incremental', + this.#incrementalIpcCommand + ); + status = workerStatus; + if (status === OperationStatus.Success || status === OperationStatus.SuccessWithWarning) { + const guard: IIncrementalExecutionGuard | undefined = getIncrementalExecutionGuard(context); + const rerunReason: string | undefined = guard + ? await getGuardResultAsync(() => guard.verifyIncrementalResultAsync(GUARD_OPTIONS)) + : undefined; + if (rerunReason !== undefined) { + terminal.writeLine(`Running the initial command, because ${rerunReason}.`); + await this.#closeWorkerAsync(terminal, INITIAL_COMMAND_REASON); + status = await this.#runInitialAsync(context, terminals); + } + } else if (status === OperationStatus.Failure && (wasWorkerReused || !this.isActive)) { + // A reused worker can fail where the initial command passes, because of state that it kept from its + // earlier runs, and so can a worker that crashed. The initial command then decides whether the + // operation fails. + if (context.error) { + terminal.writeLine(context.error.message); + context.error = undefined; + } + terminal.writeLine('Running the initial command, because the run on the warm worker failed.'); + await this.#closeWorkerAsync(terminal, INITIAL_COMMAND_REASON); + status = await this.#runInitialAsync(context, terminals); + } + } + + if (!context.shouldRunnerPersist) { + await this.#closeWorkerAsync(undefined, ''); + } else if (status === OperationStatus.Failure) { + await this.#closeWorkerAsync(terminal, FAILED_OPERATION_REASON); + } else { + await this.#recycleWorkerIfNeededAsync(terminal); + } + return status; + }, + { + createLogFile: true + } + ); + } + + public getConfigHash(): string { + return this.#commandForHash; + } + + /** + * Writes the notes of the plan that `prepareAsync` made for this context if `executeAsync` did not run, e.g. + * because the build cache restored the operation, so that a worker that the plan closed is not closed silently. + * Call it after the operation executed. + */ + public async writeUnusedNotesAsync(context: IOperationRunnerContext): Promise { + const plan: IRunPlan | undefined = this.#planByContext.get(context); + this.#planByContext.delete(context); + if (plan) { + this.#closedWorkerNote = undefined; + } + if (plan?.notes.length) { + // The log file, if any, is not this runner's, e.g. the one that the build cache restored. + await context.runWithTerminalAsync( + async (terminal: ITerminal): Promise => { + for (const note of plan.notes) { + terminal.writeLine(note); + } + }, + { createLogFile: false } + ); + } + } + + public async closeAsync(): Promise { + const worker: WarmWorker | undefined = this.#worker; + if (worker?.isAlive) { + this.#closedWorkerNote = `The warm worker (pid ${worker.pid}) was closed after the operation last ran.`; + } + await this.#closeWorkerAsync(undefined, ''); + } + + async #planAsync(context: IOperationRunnerContext, hasLastState: boolean): Promise { + // It stays until a plan's notes are written, e.g. if this plan is replaced. + const notes: string[] = this.#closedWorkerNote ? [this.#closedWorkerNote] : []; + // As in ShellOperationRunner, the guard is not consulted before the first execution in this graph. + const guard: IIncrementalExecutionGuard | undefined = hasLastState + ? getIncrementalExecutionGuard(context) + : undefined; + if (guard) { + const blockReason: string | undefined = await getGuardResultAsync(() => + guard.getBlockReasonAsync(GUARD_OPTIONS) + ); + if (blockReason === undefined) { + if (this.isActive) { + skipBuildCacheRead(context); + } + return { kind: 'incremental', notes }; + } + notes.push(`Not using the incremental command because ${blockReason}.`); + } + // The initial command must not run on top of what a worker keeps in memory. + const closeNote: string | undefined = await this.#closeWorkerAsync(undefined, INITIAL_COMMAND_REASON); + if (closeNote) { + notes.push(closeNote); + } + return { kind: 'initial', notes }; + } + + async #runInitialAsync( + context: IOperationRunnerContext, + terminals: ICommandTerminals + ): Promise { + if (this.#initialIpcCommand !== undefined) { + return (await this.#runOnWorkerAsync(context, terminals, 'initial', this.#initialIpcCommand)).status; + } + setCommandExecution(context, { kind: 'initial', hasIncrementalCommand: true }); + return await ShellOperationRunner.invokeCommandAsync( + context, + terminals, + this.#rushProject, + 'initial', + this.#initialCommand + ); + } + + async #runOnWorkerAsync( + context: IOperationRunnerContext, + { terminal, terminalProvider }: ICommandTerminals, + kind: ICommandExecution['kind'], + command: string + ): Promise { + // Recorded before the command starts, so that outputs of a command that fails or is aborted are attributed to it. + setCommandExecution(context, { kind, hasIncrementalCommand: true, watchesInputs: true }); + terminal.writeLine(`Invoking (${kind}): ${command}`); + + const { abortSignal } = context; + let worker: WarmWorker | undefined = this.#worker; + if (worker?.isAlive && kind === 'initial') { + // A build from scratch must not run on top of what a worker keeps in memory. + await this.#closeWorkerAsync(terminal, INITIAL_COMMAND_REASON); + worker = undefined; + } + let wasRunRequested: boolean = false; + if (worker?.isAlive) { + wasRunRequested = await worker.waitForRunRequestAsync(this.#changeReportTimeoutMs, abortSignal); + } + if (worker && !worker.isAlive) { + terminal.writeLine(`The warm worker (pid ${worker.pid}) exited after its last run.`); + await this.#closeWorkerAsync(undefined, ''); + worker = undefined; + } + + const wasWorkerReused: boolean = worker !== undefined && worker.runCount > 0; + if (!worker) { + terminal.writeLine('Starting a warm worker for it.'); + worker = this.#startWorker(context, command); + } else { + if (!wasRunRequested) { + terminal.writeVerboseLine( + `The warm worker did not report a change within ${this.#changeReportTimeoutMs} ms.` + ); + } + terminal.writeLine(`Sending run ${worker.runCount + 1} to the warm worker (pid ${worker.pid}).`); + } + + const { status, hasWarningOrError } = await this.#runWorkerAsync( + context, + worker, + terminal, + terminalProvider + ); + return { + status: + status === OperationStatus.Success && hasWarningOrError ? OperationStatus.SuccessWithWarning : status, + wasWorkerReused + }; + } + + #startWorker(context: IOperationRunnerContext, command: string): WarmWorker { + const { rushConfiguration, projectFolder } = this.#rushProject; + const environment: IEnvironment | undefined = context.environment; + const additionalEnvironment: IEnvironment | undefined = + (environment ?? process.env)[TYPESCRIPT_WATCH_FILE_VARIABLE] === undefined + ? { [TYPESCRIPT_WATCH_FILE_VARIABLE]: TYPESCRIPT_WATCH_FILE } + : undefined; + const childProcess: ChildProcess = Utilities.executeLifecycleCommandAsync(command, { + rushConfiguration, + workingDirectory: projectFolder, + initCwd: rushConfiguration.commonTempFolder, + handleOutput: true, + environmentPathOptions: { + includeProjectBin: true + }, + ipc: true, + // Isolate the process tree, so that it can be terminated. + connectSubprocessTerminator: true, + initialEnvironment: environment, + additionalEnvironment + }); + const worker: WarmWorker = new WarmWorker(childProcess, command); + this.#worker = worker; + return worker; + } + + async #runWorkerAsync( + context: IOperationRunnerContext, + worker: WarmWorker, + terminal: ITerminal, + terminalProvider: ITerminalProvider + ): Promise { + const { process: subProcess } = worker; + const { abortSignal } = context; + let hasWarningOrError: boolean = false; + const stdoutDecoder: StringDecoder = new StringDecoder('utf8'); + const stderrDecoder: StringDecoder = new StringDecoder('utf8'); + const onStdout = (data: Buffer): void => { + terminalProvider.write(stdoutDecoder.write(data), TerminalProviderSeverity.log); + }; + const onStderr = (data: Buffer): void => { + terminalProvider.write(stderrDecoder.write(data), TerminalProviderSeverity.error); + hasWarningOrError = true; + }; + subProcess.stdout?.on('data', onStdout); + subProcess.stderr?.on('data', onStderr); + + let sentRun: boolean = false; + let sendError: Error | undefined; + const onAbort = (): void => { + worker.terminate(); + }; + const status: OperationStatus = await new Promise( + (resolve: (status: OperationStatus) => void) => { + let detach: () => void = () => undefined; + const finish = (result: OperationStatus): void => { + detach(); + resolve(result); + }; + const onMessage = (message: unknown): void => { + if (sentRun && isAfterExecuteEventMessage(message)) { + const memory: number | undefined = message.residentMemoryBytes; + worker.residentMemoryBytes = + typeof memory === 'number' && Number.isSafeInteger(memory) && memory > 0 ? memory : undefined; + if (worker.runCount === 1) { + worker.firstResidentMemoryBytes = worker.residentMemoryBytes; + } + // These types are distinct, but have the same values. + finish(message.status as unknown as OperationStatus); + } + }; + const onClose = (exitCode: number | null, signal: NodeJS.Signals | null): void => { + if (abortSignal?.aborted) { + finish(OperationStatus.Aborted); + } else if (sendError) { + context.error = new OperationError('error', sendError.message); + finish(OperationStatus.Failure); + } else if (sentRun) { + context.error = new OperationError( + 'error', + `The warm worker exited before it reported the result of its run (${ + signal ? `signal ${signal}` : `exit code ${exitCode}` + }).` + ); + finish(OperationStatus.Failure); + } else if (signal) { + // It exited without using the IPC protocol, so it ran once, like a command in a shell. + context.error = new OperationError('error', `Terminated by signal: ${signal}`); + finish(OperationStatus.Failure); + } else if (exitCode !== 0) { + context.error = new OperationError('error', `Returned error code: ${exitCode}`); + finish(OperationStatus.Failure); + } else { + finish(OperationStatus.Success); + } + }; + + detach = () => { + subProcess.off('message', onMessage); + subProcess.off('close', onClose); + }; + subProcess.on('message', onMessage); + subProcess.once('close', onClose); + if (abortSignal?.aborted) { + onAbort(); + } else { + abortSignal?.addEventListener('abort', onAbort, { once: true }); + } + worker.readyPromise.then( + () => { + if (!worker.isAlive || abortSignal?.aborted) { + return; + } + const runCommand: IRunCommandMessage = { command: 'run' }; + worker.isRunRequested = false; + worker.runCount++; + sentRun = true; + try { + subProcess.send(runCommand); + } catch (error) { + sendError = new Error(`Could not send "run" to the warm worker: ${error}`); + worker.terminate(); + } + }, + () => undefined + ); + } + ); + abortSignal?.removeEventListener('abort', onAbort); + subProcess.stdout?.off('data', onStdout); + subProcess.stderr?.off('data', onStderr); + terminalProvider.write(stdoutDecoder.end(), TerminalProviderSeverity.log); + terminalProvider.write(stderrDecoder.end(), TerminalProviderSeverity.error); + + if (status === OperationStatus.Aborted) { + // The worker was terminated, so the next run starts a new one. + await this.#closeWorkerAsync(undefined, ''); + terminal.writeLine('Terminated because the operation was aborted.'); + } else if (!worker.isAlive) { + await this.#closeWorkerAsync(undefined, ''); + } + return { status, hasWarningOrError }; + } + + async #recycleWorkerIfNeededAsync(terminal: ITerminal): Promise { + const worker: WarmWorker | undefined = this.#worker; + if (!worker?.isAlive) { + return; + } + const { runCount, firstResidentMemoryBytes, residentMemoryBytes } = worker; + if (runCount >= this.#maxRunsPerWorker) { + await this.#closeWorkerAsync(terminal, `it has run ${runCount} times`); + } else if ( + firstResidentMemoryBytes !== undefined && + residentMemoryBytes !== undefined && + residentMemoryBytes > firstResidentMemoryBytes * this.#maxMemoryGrowth + ) { + await this.#closeWorkerAsync( + terminal, + `its memory grew from ${formatMegabytes(firstResidentMemoryBytes)} after its first run to ${formatMegabytes( + residentMemoryBytes + )}` + ); + } + } + + /** + * Closes the worker, if there is one. Returns the line that says so if it was running, and writes it to the + * terminal if one is given. + */ + async #closeWorkerAsync(terminal: ITerminal | undefined, reason: string): Promise { + const worker: WarmWorker | undefined = this.#worker; + if (!worker) { + // It may still be closing for another caller. + await this.#lastClosePromise; + return undefined; + } + this.#worker = undefined; + const note: string | undefined = + worker.isAlive && reason + ? `Closing the warm worker (pid ${worker.pid}), because ${reason}.` + : undefined; + if (note) { + terminal?.writeLine(note); + } + const closePromise: Promise = worker.closeAsync(); + this.#lastClosePromise = closePromise; + await closePromise; + return note; + } +} diff --git a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts index ef643be028..2cf5e5b9be 100644 --- a/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/CacheableOperationPluginRetainedResults.test.ts @@ -60,7 +60,7 @@ import { OperationStatus } from '../OperationStatus'; import type { IOperationRunner, IOperationRunnerContext } from '../IOperationRunner'; import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; import type { OperationExecutionRecord } from '../OperationExecutionRecord'; -import { setCommandExecution } from '../IncrementalExecutionState'; +import { setCommandExecution, skipBuildCacheRead } from '../IncrementalExecutionState'; import { NullOperationRunner } from '../NullOperationRunner'; const mockPhase: IPhase = { @@ -603,6 +603,36 @@ describe(`${CacheableOperationPlugin.name} retained results`, () => { expect(testGraph.cacheWrites).toEqual(['c']); }); + it('does not restore an operation whose runner skipped the build cache read, and writes neither it nor its consumers', async () => { + // "a" <- "b" + const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); + // Like a warm worker, whose outputs are what it keeps in memory: a restore would not update them. + testGraph.graph.hooks.beforeExecuteOperationAsync.tapPromise( + { name: 'skip-read', stage: -1 }, + async (record: IOperationRunnerContext & IOperationExecutionResult): Promise => { + if (testGraph.incrementalNames.has(record.operation.associatedProject.packageName)) { + skipBuildCacheRead(record); + } + return undefined; + } + ); + await testGraph.executeAsync(); + expect(testGraph.cacheWrites).toEqual(['a', 'b']); + + testGraph.localHashes.set('a', 'a-v2'); + testGraph.incrementalNames.add('a'); + await testGraph.executeAsync(); + expect(testGraph.executions).toEqual(['a:incremental', 'b']); + expect(testGraph.cacheWrites).toEqual([]); + + // Revert "a": both operations have an entry from the first iteration, but only "b" may be restored. + testGraph.localHashes.set('a', 'a-v1'); + await testGraph.executeAsync(); + expect(testGraph.cacheRestores).toEqual(['b']); + expect(testGraph.executions).toEqual(['a:incremental']); + expect(testGraph.cacheWrites).toEqual([]); + }); + it('does not re-enable operations that another plugin disabled', async () => { const testGraph: ITestGraph = await createTestGraphAsync(['a', 'b']); // Like a plugin that performs the work itself. This tap runs after PhasedOperationPlugin's. diff --git a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts index a843903840..242f8ca749 100644 --- a/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/IncrementalExecutionGuardPlugin.test.ts @@ -65,9 +65,15 @@ import { Utilities } from '../../../utilities/Utilities'; import { InputsSnapshot, type IInputsSnapshotProjectMetadata } from '../../incremental/InputsSnapshot'; import { IncrementalExecutionGuardPlugin } from '../IncrementalExecutionGuardPlugin'; import { + getCommandExecution, + getIncrementalExecutionGuard, INPUTS_CHANGED_INVALIDATION_REASON, NATIVE_COMMAND_INVALIDATION_REASON, - wasExecutedIncrementally + setCommandExecution, + setIncrementalExecutionGuard, + wasExecutedIncrementally, + type ICommandExecution, + type IIncrementalExecutionGuardOptions } from '../IncrementalExecutionState'; import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; import { LegacySkipPlugin } from '../LegacySkipPlugin'; @@ -131,10 +137,19 @@ interface IWorkspaceOptions { * operation of its own project, which has no script and depends on the builds of the project's dependencies. */ readonly hasPassThroughPhase?: boolean; + /** + * If set, the runners pass these options to the guard, like `WarmWorkerOperationRunner`. + */ + readonly guardOptions?: IIncrementalExecutionGuardOptions; /** * If set, applies the skip detection that Rush uses when the build cache is not enabled. */ readonly hasLegacySkipDetection?: boolean; + /** + * If set, the runners report that each command ran in a process that keeps watching the input files, like + * `WarmWorkerOperationRunner`. + */ + readonly watchesInputs?: boolean; } interface ITestIteration { @@ -153,6 +168,10 @@ interface ITestWorkspace { readonly operations: ReadonlyMap; writeFile(relativePath: string, content: string): void; deleteFile(relativePath: string): void; + /** + * Deletes a folder and writes its files again, like a branch switch that removed the folder and restored it. + */ + recreateFolder(relativePath: string): void; executeAsync(environment?: Readonly>): Promise; /** * Resolves when the next command that hangs has written its outputs. It runs until it is terminated. @@ -188,8 +207,20 @@ function listFiles(folder: string, exclude: ReadonlySet = new Set()): st return files.sort(); } +function recreateFolder(folder: string): void { + const contentByFile: Map = new Map( + listFiles(folder).map((file: string) => [file, fs.readFileSync(`${folder}/${file}`)]) + ); + fs.rmSync(folder, { recursive: true }); + for (const [file, content] of contentByFile) { + fs.mkdirSync(path.dirname(`${folder}/${file}`), { recursive: true }); + fs.writeFileSync(`${folder}/${file}`, content); + } +} + // Like a compiler: writes a file per source file, and its incremental mode neither cleans the output folder nor // deletes the outputs of deleted source files. A source containing "emit:" also emits ".js", a +// source containing "recreate:" deletes and recreates that folder of the project while the build runs, a // source containing "error" fails the build, and after the output of a source containing "hang", the build runs // until it is terminated (it returns undefined). function build(projectFolder: string, isBundle: boolean, isIncremental: boolean): number | undefined { @@ -214,6 +245,10 @@ function build(projectFolder: string, isBundle: boolean, isIncremental: boolean) if (emitted) { fs.writeFileSync(`${outputFolder}/${emitted[1]}.js`, ''); } + const recreated: RegExpExecArray | null = /recreate:([\w/]+)/.exec(source); + if (recreated) { + recreateFolder(`${projectFolder}/${recreated[1]}`); + } if (source.includes('hang')) { return undefined; } @@ -293,7 +328,7 @@ class PluginOperationRunner implements IOperationRunner { async function createWorkspaceAsync( projectSpecs: ReadonlyArray, - { hasPassThroughPhase, hasLegacySkipDetection }: IWorkspaceOptions = {} + { hasPassThroughPhase, guardOptions, hasLegacySkipDetection, watchesInputs }: IWorkspaceOptions = {} ): Promise { const rootFolder: string = fs.mkdtempSync(path.join(os.tmpdir(), 'rush-incremental-guard-')); workspaceFolders.push(rootFolder); @@ -475,6 +510,35 @@ async function createWorkspaceAsync( isWatch: false, projectConfigurations } as unknown as IOperationGraphContext); + if (guardOptions) { + // After the guard plugin registered the guards of the iteration. + graph.hooks.beforeExecuteIterationAsync.tap( + { name: 'guardOptions', stage: 1 }, + (records: ReadonlyMap): void => { + for (const record of records.values()) { + const guard: IIncrementalExecutionGuard | undefined = getIncrementalExecutionGuard(record); + if (guard) { + setIncrementalExecutionGuard(record, { + getBlockReasonAsync: () => guard.getBlockReasonAsync(guardOptions), + verifyIncrementalResultAsync: () => guard.verifyIncrementalResultAsync(guardOptions) + }); + } + } + } + ); + } + if (watchesInputs) { + // Before the guard's taps + graph.hooks.afterExecuteOperationAsync.tap( + { name: 'watchesInputs', stage: -2 }, + (record: IOperationExecutionResult): void => { + const execution: ICommandExecution | undefined = getCommandExecution(record); + if (execution) { + setCommandExecution(record, { ...execution, watchesInputs: true }); + } + } + ); + } // Like `git hash-object` for each file, except the outputs, which are ignored by git. const createInputsSnapshot = (environment: Readonly>): InputsSnapshot => { @@ -507,6 +571,7 @@ async function createWorkspaceAsync( operations, writeFile, deleteFile: (relativePath: string) => fs.rmSync(`${rootFolder}/${relativePath}`), + recreateFolder: (relativePath: string) => recreateFolder(`${rootFolder}/${relativePath}`), executeAsync: async (environment: Readonly> = {}): Promise => { commands.length = 0; destination.reset(); @@ -793,6 +858,148 @@ describe(IncrementalExecutionGuardPlugin.name, () => { ); }); + it('runs the incremental command of an operation that builds a bundle for a runner that allows bundles', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a', isBundle: true }], { + guardOptions: { outputsMayBeBundles: true } + }); + await workspace.executeAsync(); + + for (const content of ['one 2', 'one 3']) { + workspace.writeFile('a/src/one.ts', content); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental']); + expect(changed.getStatus('a')).toBe(OperationStatus.Success); + expect(changed.output).not.toContain('Not using the incremental command'); + } + }); + + it('runs only the initial command after an incremental command renamed a content-hashed file, for a runner that allows bundles', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }], { + guardOptions: { outputsMayBeBundles: true } + }); + await workspace.executeAsync(); + + // Like a chunk with webpack's default hash length, which a bundler renames whenever its content changes. + workspace.writeFile('a/src/one.ts', 'one emit:chunk_0123456789abcdef0123'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental', 'a:initial']); + expect(changed.output).toContain( + 'Running the initial command, because the incremental command changed which output files it has: 1 added ("lib/chunk_0123456789abcdef0123.js").' + ); + + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + const next: ITestIteration = await workspace.executeAsync(); + expect(next.commands).toEqual(['a:initial']); + expect(next.output).toContain( + 'Not using the incremental command because its incremental command changed which output files it has in an earlier run: 1 added ("lib/chunk_0123456789abcdef0123.js").' + ); + }); + + it('runs the initial command only once after an incremental command that changed which output files exist, for a runner that allows bundles', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }], { + guardOptions: { outputsMayBeBundles: true } + }); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'one emit:chunk'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:incremental', 'a:initial']); + expect(changed.output).toContain( + 'Running the initial command, because the incremental command changed which output files it has: 1 added ("lib/chunk.js").' + ); + + // The initial command removed any stale outputs, and the runner keeps its build state in memory. + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + }); + + it('runs the initial command after a folder of its input files was recreated, for a runner that watches its inputs', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }], { watchesInputs: true }); + await workspace.executeAsync(); + + // Unlike an editor that saves a file by renaming a new file over it + workspace.writeFile('a/src/sub/two.ts.tmp', 'two 1'); + fs.renameSync(`${workspace.rootFolder}/a/src/sub/two.ts.tmp`, `${workspace.rootFolder}/a/src/sub/two.ts`); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + + // The input files did not change, but a watcher of the old folder misses later edits of its files. + workspace.recreateFolder('a/src/sub'); + expect((await workspace.executeAsync()).commands).toEqual([]); + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain( + 'Not using the incremental command because folders that held its input files were deleted or recreated since its last run ("a/src/sub").' + ); + + // The process of the initial command watches the new folder. + workspace.writeFile('a/src/sub/two.ts', 'two 3'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + }); + + it('names only the top folder of recreated folders of its input files, for a runner that watches its inputs', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }], { watchesInputs: true }); + await workspace.executeAsync(); + + // Recreates "a/src/sub" as well + workspace.recreateFolder('a/src'); + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + const changed: ITestIteration = await workspace.executeAsync(); + expect(changed.commands).toEqual(['a:initial']); + expect(changed.output).toContain( + 'Not using the incremental command because folders that held its input files were deleted or recreated since its last run ("a/src").' + ); + }); + + it('runs the initial command after a folder of its input files was recreated while it ran, for a runner that watches its inputs', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }], { watchesInputs: true }); + await workspace.executeAsync(); + + workspace.writeFile('a/src/one.ts', 'one recreate:src/sub'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + + workspace.writeFile('a/src/one.ts', 'one 3'); + const next: ITestIteration = await workspace.executeAsync(); + expect(next.commands).toEqual(['a:initial']); + expect(next.output).toContain( + 'Not using the incremental command because folders that held its input files were deleted or recreated since its last run ("a/src/sub").' + ); + }); + + it('runs the incremental command after a folder of its input files was recreated, for a runner that does not watch its inputs', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + await workspace.executeAsync(); + + workspace.recreateFolder('a/src/sub'); + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + }); + + it('runs the incremental command after a folder of its input files was recreated, if its last run was not in a process that watches its inputs', async () => { + const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); + let watchesInputs: boolean = true; + workspace.graph.hooks.afterExecuteOperationAsync.tap( + { name: 'watchesInputs', stage: -2 }, + (record: IOperationExecutionResult): void => { + const execution: ICommandExecution | undefined = getCommandExecution(record); + if (execution && watchesInputs) { + setCommandExecution(record, { ...execution, watchesInputs: true }); + } + } + ); + await workspace.executeAsync(); + workspace.writeFile('a/src/sub/two.ts', 'two 2'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + + // E.g. a runner that closed its worker ran the command in a shell. The next process watches the new folder. + watchesInputs = false; + workspace.writeFile('a/src/sub/two.ts', 'two 3'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + workspace.recreateFolder('a/src/sub'); + workspace.writeFile('a/src/sub/two.ts', 'two 4'); + expect((await workspace.executeAsync()).commands).toEqual(['a:incremental']); + }); + it('runs the initial command after an incremental command that emitted a content-hashed file', async () => { const workspace: ITestWorkspace = await createWorkspaceAsync([{ name: 'a' }]); await workspace.executeAsync(); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts index 25d66cf0c7..b2146360cf 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationExecutionRecord.test.ts @@ -162,5 +162,21 @@ describe(OperationExecutionRecord.name, () => { record.reportCommandExecution({ kind: 'initial', hasIncrementalCommand: true }); expect(wasExecutedIncrementally(record)).toBe(false); }); + + it('records whether the runner reports that the command ran in a process that watches the input files', () => { + const record: OperationExecutionRecord = createRecord( + createOperation({ project: createProject('project-watching') }) + ); + + record.reportCommandExecution({ + kind: 'incremental', + hasIncrementalCommand: true, + watchesInputs: true + }); + expect(getCommandExecution(record)?.watchesInputs).toBe(true); + + record.reportCommandExecution({ kind: 'initial', hasIncrementalCommand: true }); + expect(getCommandExecution(record)?.watchesInputs).toBeUndefined(); + }); }); }); diff --git a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts index ae29e6be3c..2e06310ac9 100644 --- a/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts +++ b/libraries/rush-lib/src/logic/operations/test/OperationOutputManifest.test.ts @@ -8,6 +8,7 @@ import * as path from 'node:path'; import { describeOutputFileChanges, getCleanOnlyReason, + hasContentHashedOutputChange, readOperationOutputManifestAsync, type IOperationOutputManifest } from '../OperationOutputManifest'; @@ -55,6 +56,33 @@ describe(describeOutputFileChanges.name, () => { ) ).toBeUndefined(); }); + + it.each([ + ['lib/0dd8cf755e5195a5.js', 'lib/7c1e0b4f9a2d3e6f.js'], + ['dist/app.0123456789abcdef0123.js', 'dist/app.3210fedcba9876543210.js'], + ['dist/2d0bb124b0ce1c65888f.png', 'dist/5c7a3e9f1b2d4068ace1.png'] + ])( + 'describes a content-hashed output that a bundler renamed from %s to %s', + (before: string, after: string) => { + expect(describeOutputFileChanges(new Set([before]), new Set([after]))).toBe( + `1 added (${JSON.stringify(after)}), 1 removed (${JSON.stringify(before)})` + ); + } + ); +}); + +describe(hasContentHashedOutputChange.name, () => { + it.each<[string[], string[], boolean]>([ + [['lib/a.js'], ['lib/a.js', 'lib/chunk_0123456789abcdef0123.js'], true], + [['lib/a.js', 'lib/chunk.main_1a2b3c4d.js'], ['lib/a.js'], true], + [['dist/2d0bb124b0ce1c65888f.png'], ['dist/5c7a3e9f1b2d4068ace1.png'], true], + [['lib/a.js'], ['lib/a.js', 'lib/b.js'], false], + [['dist/a.js'], ['dist/a.js', 'dist/b.js'], false], + [['lib/chunk.main_1a2b3c4d.js'], ['lib/chunk.main_1a2b3c4d.js', 'lib/b.js'], false], + [['lib/a.js'], ['lib/a.js', 'temp/jest-transform-cache-0123456789abcdef-0123456789abcdef/7f/a_1'], false] + ])('for %j before and %j after, returns %s', (before: string[], after: string[], expected: boolean) => { + expect(hasContentHashedOutputChange(new Set(before), new Set(after))).toBe(expected); + }); }); describe(readOperationOutputManifestAsync.name, () => { diff --git a/libraries/rush-lib/src/logic/operations/test/WarmWorkerOperationRunner.test.ts b/libraries/rush-lib/src/logic/operations/test/WarmWorkerOperationRunner.test.ts new file mode 100644 index 0000000000..3f415bd6b1 --- /dev/null +++ b/libraries/rush-lib/src/logic/operations/test/WarmWorkerOperationRunner.test.ts @@ -0,0 +1,936 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license. +// See LICENSE in the project root for license information. + +jest.mock('../../../utilities/Utilities'); +jest.mock('../OperationStateFile'); +jest.mock('../ProjectLogWritable', () => { + const actual = jest.requireActual('../ProjectLogWritable'); + const { MockWritable } = jest.requireActual('@rushstack/terminal'); + return { ...actual, initializeProjectLogFilesAsync: jest.fn(async () => new MockWritable()) }; +}); + +import { spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import { performance } from 'node:perf_hooks'; + +import { SubprocessTerminator } from '@rushstack/node-core-library'; +import { MockWritable } from '@rushstack/terminal'; + +import type { IPhase } from '../../../api/CommandLineConfiguration'; +import type { RushConfigurationProject } from '../../../api/RushConfigurationProject'; +import type { IOperationSettings } from '../../../api/RushProjectConfiguration'; +import { + PhasedCommandHooks, + type ICreateOperationsContext, + type IOperationGraphContext +} from '../../../pluginFramework/PhasedCommandHooks'; +import { Utilities, type IEnvironment, type ILifecycleCommandOptions } from '../../../utilities/Utilities'; +import { DaemonWarmWorkerPlugin } from '../DaemonWarmWorkerPlugin'; +import { + getCommandExecution, + isBuildCacheReadSkipped, + setIncrementalExecutionGuard, + type IIncrementalExecutionGuard, + type IIncrementalExecutionGuardOptions +} from '../IncrementalExecutionState'; +import type { IExecutionResult, IOperationExecutionResult } from '../IOperationExecutionResult'; +import { NullOperationRunner } from '../NullOperationRunner'; +import { Operation } from '../Operation'; +import { OperationExecutionRecord } from '../OperationExecutionRecord'; +import { OperationGraph } from '../OperationGraph'; +import { OperationStatus } from '../OperationStatus'; +import { ShellOperationRunner } from '../ShellOperationRunner'; +import { ShellOperationRunnerPlugin } from '../ShellOperationRunnerPlugin'; +import { + WarmWorkerOperationRunner, + type IWarmWorkerOperationRunnerOptions +} from '../WarmWorkerOperationRunner'; + +const INITIAL_COMMAND: string = 'node build.js'; +const INITIAL_IPC_COMMAND: string = 'node build.js --watch --clean'; +const INCREMENTAL_IPC_COMMAND: string = 'node build.js --watch'; + +type WorkerBehavior = 'crash' | 'fail' | 'fail-reused' | 'hang' | 'grow' | 'request'; + +// Like `heft run-watch` with IPC: it sends "sync" when it starts, and answers each "run" with "after-execute". It +// prints the number of each run, and answers 20 ms later, so that the line arrives first. Its behavior: +// - crash: it exits when it receives a run; +// - fail: it prints an error and reports that each run failed, like a build with a compiler error; +// - fail-reused: like fail, but only from its second run on, like a worker that kept an error from an earlier run; +// - hang: it never answers a run; +// - grow: it reports 100 MB more resident memory after each run, instead of 100 MB; +// - request: it reports a change 50 ms after each run, like its file watcher would. +const WORKER_SCRIPT: string = ` + const behavior = process.env.WARM_WORKER_TEST_BEHAVIOR; + let runs = 0; + process.on('message', (message) => { + if (message.command === 'exit') { + process.exit(0); + } else if (message.command === 'run') { + runs++; + if (behavior === 'crash') { + process.exit(3); + } + console.log('worker run ' + runs); + if (behavior === 'hang') { + return; + } + const fails = behavior === 'fail' || (behavior === 'fail-reused' && runs > 1); + if (fails) { + console.error('worker error ' + runs); + } + setTimeout(() => { + const residentMemoryBytes = 100000000 * (behavior === 'grow' ? runs : 1); + const status = fails ? 'FAILURE' : 'SUCCESS'; + process.send({ event: 'after-execute', status, residentMemoryBytes }); + if (behavior === 'request') { + setTimeout(() => process.send({ event: 'requestRun', requestor: 'watcher' }), 50); + } + }, 20); + } + }); + process.send({ event: 'sync' }); +`; + +const phase: IPhase = { + name: '_phase:build', + allowWarningsOnSuccess: false, + associatedParameters: new Set(), + dependencies: { self: new Set(), upstream: new Set() }, + isSynthetic: false, + logFilenameIdentifier: '_phase_build', + missingScriptBehavior: 'silent' +}; + +function createProject( + packageName: string, + scripts: Record | undefined +): RushConfigurationProject { + return { + packageName, + projectFolder: __dirname, + packageJson: { name: packageName, version: '1.0.0', scripts }, + rushConfiguration: { commonTempFolder: __dirname } + } as unknown as RushConfigurationProject; +} + +interface ITestIteration { + readonly result: IExecutionResult; + readonly record: IOperationExecutionResult; + readonly output: string; + /** + * The commands that started a process in this iteration, in order. + */ + readonly commands: ReadonlyArray; + /** + * The environment of each process in `commands`, merged from the options as Utilities merges it. + */ + readonly environments: ReadonlyArray; +} + +interface ITestHarness { + readonly runner: WarmWorkerOperationRunner; + readonly graph: OperationGraph; + /** + * The guard of the next iterations, or undefined for none. + */ + guard: IIncrementalExecutionGuard | undefined; + /** + * The options that the runner passed to the guard, in order. + */ + readonly guardOptions: IIncrementalExecutionGuardOptions[]; + blockReason: string | undefined; + rerunReason: string | undefined; + /** + * The behavior of the workers that start from now on. + */ + workerBehavior: WorkerBehavior | undefined; + /** + * The exit code of the initial commands that run in a shell from now on. + */ + initialExitCode: number; + shouldRunnerPersist: boolean; + /** + * The status that a tap at the default stage of `beforeExecuteOperationAsync` returns, e.g. `FromCache` for a + * build cache hit, so that the runner does not execute. + */ + earlyReturnStatus: OperationStatus | undefined; + /** + * For each iteration, whether the build cache read was skipped when a tap at the default stage of + * `beforeExecuteOperationAsync`, like `CacheableOperationPlugin`'s, ran. + */ + readonly isReadSkippedAtCacheStage: boolean[]; + executeAsync(): Promise; + /** + * Resolves when a worker printed the number of its next run. + */ + waitForWorkerRunAsync(): Promise; +} + +const children: ChildProcess[] = []; +const graphs: OperationGraph[] = []; + +beforeEach(() => { + // Each test process is its own process tree. + jest.spyOn(SubprocessTerminator, 'killProcessTree').mockImplementation((child: ChildProcess) => { + child.kill('SIGTERM'); + }); +}); + +afterEach(async () => { + for (const graph of graphs.splice(0)) { + await graph.closeRunnersAsync(); + graph.abortController.abort(); + } + for (const child of children.splice(0)) { + if (child.exitCode === null && child.signalCode === null) { + const closed: Promise = once(child, 'close'); + child.kill(); + await closed; + } + } + jest.mocked(Utilities.executeLifecycleCommandAsync).mockReset(); + jest.restoreAllMocks(); +}); + +async function createHarnessAsync( + options: Partial = {} +): Promise { + const project: RushConfigurationProject = createProject('a', undefined); + const runner: WarmWorkerOperationRunner = new WarmWorkerOperationRunner({ + phase, + rushProject: project, + displayName: 'a', + initialCommand: INITIAL_COMMAND, + initialIpcCommand: undefined, + incrementalIpcCommand: INCREMENTAL_IPC_COMMAND, + commandForHash: INITIAL_COMMAND, + ignoredParameterValues: [], + // Only the test of this wait waits. + changeReportTimeoutMs: 0, + ...options + }); + const operation: Operation = new Operation({ phase, project, runner, logFilenameIdentifier: 'a' }); + const destination: MockWritable = new MockWritable(); + const graph: OperationGraph = new OperationGraph(new Set([operation]), { + quietMode: false, + debugMode: false, + parallelism: 1, + allowOversubscription: true, + destinations: [destination], + abortController: new AbortController(), + // Like the graphs of the Rush daemon + supportsTerminateRunning: true + }); + graphs.push(graph); + + const commands: string[] = []; + const environments: IEnvironment[] = []; + const runWaiters: (() => void)[] = []; + const harness: ITestHarness = { + runner, + graph, + guard: undefined, + guardOptions: [], + blockReason: undefined, + rerunReason: undefined, + workerBehavior: undefined, + initialExitCode: 0, + shouldRunnerPersist: true, + earlyReturnStatus: undefined, + isReadSkippedAtCacheStage: [], + executeAsync: async (): Promise => { + commands.length = 0; + environments.length = 0; + destination.reset(); + const result: IExecutionResult = await graph.executeAsync({}); + return { + result, + record: result.operationResults.get(operation)!, + output: destination.getAllOutput(), + commands: [...commands], + environments: [...environments] + }; + }, + waitForWorkerRunAsync: () => new Promise((resolve: () => void) => runWaiters.push(resolve)) + }; + harness.guard = { + getBlockReasonAsync: async (guardOptions?: IIncrementalExecutionGuardOptions) => { + harness.guardOptions.push({ ...guardOptions }); + return harness.blockReason; + }, + verifyIncrementalResultAsync: async (guardOptions?: IIncrementalExecutionGuardOptions) => { + harness.guardOptions.push({ ...guardOptions }); + return harness.rerunReason; + } + }; + + jest + .mocked(Utilities.executeLifecycleCommandAsync) + .mockImplementation((command: string, lifecycleOptions: ILifecycleCommandOptions) => { + const { ipc, initialEnvironment, additionalEnvironment } = lifecycleOptions; + commands.push(command); + environments.push({ ...(initialEnvironment ?? process.env), ...additionalEnvironment }); + let child: ChildProcess; + if (ipc) { + child = spawn(process.execPath, ['-e', WORKER_SCRIPT], { + stdio: ['ignore', 'pipe', 'pipe', 'ipc'], + env: { ...process.env, WARM_WORKER_TEST_BEHAVIOR: harness.workerBehavior ?? '' } + }); + child.stdout!.on('data', (data: Buffer) => { + if (data.toString().includes('worker run')) { + for (const resolve of runWaiters.splice(0)) { + resolve(); + } + } + }); + } else { + child = spawn( + process.execPath, + ['-e', `console.log("one-shot"); process.exitCode = ${harness.initialExitCode};`], + { + stdio: ['ignore', 'pipe', 'pipe'] + } + ); + } + children.push(child); + return child; + }); + + // Like IncrementalExecutionGuardPlugin, which registers a guard for each record of an iteration. + graph.hooks.beforeExecuteIterationAsync.tap( + 'test', + (records: ReadonlyMap): void => { + for (const record of records.values()) { + if (harness.guard) { + setIncrementalExecutionGuard(record, harness.guard); + } + (record as OperationExecutionRecord).shouldRunnerPersist = harness.shouldRunnerPersist; + } + } + ); + // Registered before the plugin's tap, so that only the stages order them. + graph.hooks.beforeExecuteOperationAsync.tapPromise( + 'test', + async (record: IOperationExecutionResult): Promise => { + harness.isReadSkippedAtCacheStage.push(isBuildCacheReadSkipped(record)); + return harness.earlyReturnStatus; + } + ); + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new DaemonWarmWorkerPlugin().apply(hooks); + await hooks.onGraphCreatedAsync.promise(graph, { + isIncrementalBuildAllowed: true, + isWatch: false + } as unknown as IOperationGraphContext); + return harness; +} + +/** + * The `TSC_WATCHFILE` variable of each process that started in the iteration, in order. + */ +function getTypeScriptWatchFiles(iteration: ITestIteration): (string | undefined)[] { + return iteration.environments.map((environment: IEnvironment) => environment.TSC_WATCHFILE); +} + +function expectLinesInOrder(output: string, lines: ReadonlyArray): void { + let index: number = -1; + for (const line of lines) { + const next: number = output.indexOf(line, index + 1); + if (next < 0) { + throw new Error(`Expected ${JSON.stringify(line)} after offset ${index} in:\n${output}`); + } + index = next; + } +} + +describe(WarmWorkerOperationRunner.name, () => { + it('runs the initial command first, then starts a worker for an allowed incremental run, and sends it the next one', async () => { + const harness: ITestHarness = await createHarnessAsync(); + + // As for ShellOperationRunner, the first run is the initial command in a shell, and the guard is not asked. + const first: ITestIteration = await harness.executeAsync(); + expect(first.result.status).toBe(OperationStatus.Success); + expect(first.commands).toEqual([INITIAL_COMMAND]); + expect(first.output).toContain(`Invoking (initial): ${INITIAL_COMMAND}`); + expect(first.output).not.toContain('warm worker'); + expect(getCommandExecution(first.record)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + expect(harness.guardOptions).toEqual([]); + expect(harness.runner.isActive).toBe(false); + + const second: ITestIteration = await harness.executeAsync(); + expect(second.result.status).toBe(OperationStatus.Success); + expect(second.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expectLinesInOrder(second.output, [ + `Invoking (incremental): ${INCREMENTAL_IPC_COMMAND}`, + 'Starting a warm worker for it.', + 'worker run 1' + ]); + expect(getCommandExecution(second.record)).toEqual({ + kind: 'incremental', + hasIncrementalCommand: true, + watchesInputs: true + }); + expect(harness.runner.isActive).toBe(true); + const pid: number = harness.runner.workerPid!; + + const third: ITestIteration = await harness.executeAsync(); + expect(third.result.status).toBe(OperationStatus.Success); + expect(third.commands).toEqual([]); + expectLinesInOrder(third.output, [ + `Invoking (incremental): ${INCREMENTAL_IPC_COMMAND}`, + `Sending run 2 to the warm worker (pid ${pid}).`, + 'worker run 2' + ]); + expect(third.output).not.toContain('Starting a warm worker'); + expect(getCommandExecution(third.record)).toEqual({ + kind: 'incremental', + hasIncrementalCommand: true, + watchesInputs: true + }); + expect(harness.runner.workerPid).toBe(pid); + expect(harness.runner.residentMemoryBytes).toBe(100000000); + + // The build cache may restore an operation without a worker, but a restore would not update what a running + // worker keeps in memory. + expect(harness.isReadSkippedAtCacheStage).toEqual([false, false, true]); + // Each allowed run asks the guard twice: before it runs and after it succeeded. + expect(harness.guardOptions).toEqual(new Array(4).fill({ outputsMayBeBundles: true })); + + const worker: ChildProcess = children[1]; + await harness.graph.closeRunnersAsync(); + expect(harness.runner.isActive).toBe(false); + expect(worker.exitCode).toBe(0); + }); + + it('starts each worker with a TypeScript file watcher that sees replaced files', async () => { + const harness: ITestHarness = await createHarnessAsync(); + harness.graph.hooks.createEnvironmentForOperation.tap('test', (environment: IEnvironment) => { + delete environment.TSC_WATCHFILE; + return environment; + }); + + // The initial command in a shell does not watch files, and keeps the environment of the operation. + const first: ITestIteration = await harness.executeAsync(); + expect(first.commands).toEqual([INITIAL_COMMAND]); + expect(getTypeScriptWatchFiles(first)).toEqual([undefined]); + + const second: ITestIteration = await harness.executeAsync(); + expect(second.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(getTypeScriptWatchFiles(second)).toEqual(['UseFsEventsOnParentDirectory']); + }); + + it('keeps the TypeScript file watcher that the environment of the operation chooses', async () => { + const harness: ITestHarness = await createHarnessAsync({ initialIpcCommand: INITIAL_IPC_COMMAND }); + harness.graph.hooks.createEnvironmentForOperation.tap('test', (environment: IEnvironment) => { + environment.TSC_WATCHFILE = 'PriorityPollingInterval'; + return environment; + }); + + const first: ITestIteration = await harness.executeAsync(); + expect(first.commands).toEqual([INITIAL_IPC_COMMAND]); + expect(getTypeScriptWatchFiles(first)).toEqual(['PriorityPollingInterval']); + }); + + it('closes the worker before the initial command if the guard does not allow an incremental run', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + + harness.blockReason = 'its command line changed'; + const blocked: ITestIteration = await harness.executeAsync(); + expect(blocked.result.status).toBe(OperationStatus.Success); + expect(blocked.commands).toEqual([INITIAL_COMMAND]); + expectLinesInOrder(blocked.output, [ + 'Not using the incremental command because its command line changed.', + `Closing the warm worker (pid ${worker.pid}), because the initial command must run.`, + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + // It exited before the build cache could restore the operation. + expect(worker.exitCode).toBe(0); + expect(harness.isReadSkippedAtCacheStage[2]).toBe(false); + expect(getCommandExecution(blocked.record)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + expect(harness.runner.isActive).toBe(false); + + harness.blockReason = undefined; + const next: ITestIteration = await harness.executeAsync(); + expect(next.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(next.output).toContain('Starting a warm worker for it.'); + }); + + it('writes why it closed the worker if the build cache restores the operation', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + + harness.blockReason = 'its command line changed'; + harness.earlyReturnStatus = OperationStatus.FromCache; + const restored: ITestIteration = await harness.executeAsync(); + expect(restored.record.status).toBe(OperationStatus.FromCache); + expect(restored.commands).toEqual([]); + expectLinesInOrder(restored.output, [ + 'Not using the incremental command because its command line changed.', + `Closing the warm worker (pid ${worker.pid}), because the initial command must run.` + ]); + expect(worker.exitCode).toBe(0); + expect(harness.runner.isActive).toBe(false); + + // If the runner executes, its notes are written once. + harness.earlyReturnStatus = undefined; + const blocked: ITestIteration = await harness.executeAsync(); + expect(blocked.commands).toEqual([INITIAL_COMMAND]); + expect(blocked.output.split('Not using the incremental command because').length).toBe(2); + }); + + it('closes the worker before the initial command if the operation has no guard, e.g. in a rebuild', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + + harness.guard = undefined; + const rebuilt: ITestIteration = await harness.executeAsync(); + expect(rebuilt.commands).toEqual([INITIAL_COMMAND]); + expectLinesInOrder(rebuilt.output, [ + `Closing the warm worker (pid ${worker.pid}), because the initial command must run.`, + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + expect(rebuilt.output).not.toContain('Not using the incremental command'); + expect(worker.exitCode).toBe(0); + }); + + it('closes the worker before the initial command if the operation has no last state, even if it has a guard', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + + // Like a graph whose iteration allows no incremental build: it keeps the results, but passes no last state. + const { executeAsync } = OperationExecutionRecord.prototype; + jest.spyOn(OperationExecutionRecord.prototype, 'executeAsync').mockImplementation(function ( + this: OperationExecutionRecord, + lastState: OperationExecutionRecord | undefined, + executeContext: Parameters[1] + ): Promise { + return executeAsync.call(this, undefined, executeContext); + }); + const rebuilt: ITestIteration = await harness.executeAsync(); + expect(rebuilt.commands).toEqual([INITIAL_COMMAND]); + expectLinesInOrder(rebuilt.output, [ + `Closing the warm worker (pid ${worker.pid}), because the initial command must run.`, + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + expect(rebuilt.output).not.toContain('Not using the incremental command'); + expect(getCommandExecution(rebuilt.record)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + expect(worker.exitCode).toBe(0); + }); + + it('writes that a worker was closed between builds, e.g. by the warm set of the Rush daemon, once', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + + // Like the warm set of the Rush daemon, which closes the runners of a project and drops its results. + const operations: Operation[] = [...harness.graph.operations]; + await harness.graph.closeRunnersAsync(operations); + harness.graph.deleteResults(operations); + expect(worker.exitCode).toBe(0); + + const next: ITestIteration = await harness.executeAsync(); + expect(next.commands).toEqual([INITIAL_COMMAND]); + expectLinesInOrder(next.output, [ + `The warm worker (pid ${worker.pid}) was closed after the operation last ran.`, + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + + // Closing a runner without a worker writes nothing. + await harness.runner.closeAsync(); + const again: ITestIteration = await harness.executeAsync(); + expect(again.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(again.output).not.toContain('was closed after the operation last ran'); + }); + + it('writes that a worker was closed between builds if the build cache restores the operation, once', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + + await harness.runner.closeAsync(); + harness.earlyReturnStatus = OperationStatus.FromCache; + const restored: ITestIteration = await harness.executeAsync(); + expect(restored.record.status).toBe(OperationStatus.FromCache); + expect(restored.output).toContain( + `The warm worker (pid ${worker.pid}) was closed after the operation last ran.` + ); + + harness.earlyReturnStatus = undefined; + const next: ITestIteration = await harness.executeAsync(); + expect(next.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(next.output).not.toContain('was closed after the operation last ran'); + }); + + it('runs the initial command after an incremental run whose outputs the guard does not accept', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + + harness.rerunReason = + 'the incremental command changed which output files it has: 1 added ("lib/chunk.js")'; + const rerun: ITestIteration = await harness.executeAsync(); + expect(rerun.result.status).toBe(OperationStatus.Success); + expect(rerun.commands).toEqual([INCREMENTAL_IPC_COMMAND, INITIAL_COMMAND]); + expectLinesInOrder(rerun.output, [ + `Invoking (incremental): ${INCREMENTAL_IPC_COMMAND}`, + 'Starting a warm worker for it.', + 'worker run 1', + 'Running the initial command, because the incremental command changed which output files it has: 1 added ("lib/chunk.js").', + `Closing the warm worker (pid ${children[1].pid}), because the initial command must run.`, + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + expect(getCommandExecution(rerun.record)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + expect(harness.runner.isActive).toBe(false); + }); + + it('runs the initial command after a failed run on a reused worker, and reports the status of the initial command', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + // E.g. a worker whose module resolution kept an error from an earlier run. + harness.workerBehavior = 'fail-reused'; + expect((await harness.executeAsync()).result.status).toBe(OperationStatus.Success); + const worker: ChildProcess = children[1]; + const pid: number = harness.runner.workerPid!; + + const recovered: ITestIteration = await harness.executeAsync(); + expect(recovered.result.status).toBe(OperationStatus.Success); + expect(recovered.commands).toEqual([INITIAL_COMMAND]); + expectLinesInOrder(recovered.output, [ + `Sending run 2 to the warm worker (pid ${pid}).`, + 'worker error 2', + 'Running the initial command, because the run on the warm worker failed.', + `Closing the warm worker (pid ${pid}), because the initial command must run.`, + `Invoking (initial): ${INITIAL_COMMAND}`, + 'one-shot' + ]); + expect(recovered.record.error).toBeUndefined(); + expect(getCommandExecution(recovered.record)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + expect(worker.exitCode).toBe(0); + expect(harness.runner.isActive).toBe(false); + + // A genuine error fails the initial command too. + expect((await harness.executeAsync()).output).toContain('Starting a warm worker for it.'); + harness.initialExitCode = 2; + const failed: ITestIteration = await harness.executeAsync(); + expect(failed.result.status).toBe(OperationStatus.Failure); + expect(failed.commands).toEqual([INITIAL_COMMAND]); + expectLinesInOrder(failed.output, [ + 'worker error 2', + 'Running the initial command, because the run on the warm worker failed.', + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + expect(failed.record.error?.message).toContain('Returned error code: 2'); + expect(harness.runner.isActive).toBe(false); + }); + + it('reports a failed first run of a new worker as it stands, and closes the worker', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + harness.workerBehavior = 'fail'; + const failed: ITestIteration = await harness.executeAsync(); + expect(failed.result.status).toBe(OperationStatus.Failure); + expect(failed.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + const worker: ChildProcess = children[1]; + expectLinesInOrder(failed.output, [ + 'Starting a warm worker for it.', + 'worker error 1', + `Closing the warm worker (pid ${worker.pid}), because the operation failed, so its next build runs the initial command.` + ]); + expect(failed.output).not.toContain('Running the initial command'); + expect(worker.exitCode).toBe(0); + expect(harness.runner.isActive).toBe(false); + }); + + it('closes a worker whose initial run failed', async () => { + const harness: ITestHarness = await createHarnessAsync({ initialIpcCommand: INITIAL_IPC_COMMAND }); + harness.workerBehavior = 'fail'; + const failed: ITestIteration = await harness.executeAsync(); + expect(failed.result.status).toBe(OperationStatus.Failure); + expect(failed.commands).toEqual([INITIAL_IPC_COMMAND]); + const worker: ChildProcess = children[0]; + expectLinesInOrder(failed.output, [ + `Invoking (initial): ${INITIAL_IPC_COMMAND}`, + 'Starting a warm worker for it.', + 'worker error 1', + `Closing the warm worker (pid ${worker.pid}), because the operation failed, so its next build runs the initial command.` + ]); + expect(worker.exitCode).toBe(0); + expect(harness.runner.isActive).toBe(false); + }); + + it('runs the initial command in a new worker if the project has an ipc script, and reuses that worker', async () => { + const harness: ITestHarness = await createHarnessAsync({ initialIpcCommand: INITIAL_IPC_COMMAND }); + + const first: ITestIteration = await harness.executeAsync(); + expect(first.result.status).toBe(OperationStatus.Success); + expect(first.commands).toEqual([INITIAL_IPC_COMMAND]); + expectLinesInOrder(first.output, [ + `Invoking (initial): ${INITIAL_IPC_COMMAND}`, + 'Starting a warm worker for it.', + 'worker run 1' + ]); + expect(getCommandExecution(first.record)).toEqual({ + kind: 'initial', + hasIncrementalCommand: true, + watchesInputs: true + }); + const firstPid: number = harness.runner.workerPid!; + + const second: ITestIteration = await harness.executeAsync(); + expect(second.commands).toEqual([]); + expectLinesInOrder(second.output, [ + `Invoking (incremental): ${INCREMENTAL_IPC_COMMAND}`, + `Sending run 2 to the warm worker (pid ${firstPid}).`, + 'worker run 2' + ]); + expect(harness.isReadSkippedAtCacheStage).toEqual([false, true]); + + // A build from scratch must not run on top of what the worker keeps in memory. + harness.blockReason = 'its dependencies changed'; + const blocked: ITestIteration = await harness.executeAsync(); + expect(blocked.commands).toEqual([INITIAL_IPC_COMMAND]); + expectLinesInOrder(blocked.output, [ + 'Not using the incremental command because its dependencies changed.', + `Closing the warm worker (pid ${firstPid}), because the initial command must run.`, + `Invoking (initial): ${INITIAL_IPC_COMMAND}`, + 'Starting a warm worker for it.', + 'worker run 1' + ]); + expect(harness.runner.workerPid).not.toBe(firstPid); + }); + + it('closes a worker after its maximum number of runs', async () => { + const harness: ITestHarness = await createHarnessAsync({ maxRunsPerWorker: 2 }); + await harness.executeAsync(); + await harness.executeAsync(); + const pid: number = harness.runner.workerPid!; + + const last: ITestIteration = await harness.executeAsync(); + expectLinesInOrder(last.output, [ + `Sending run 2 to the warm worker (pid ${pid}).`, + 'worker run 2', + `Closing the warm worker (pid ${pid}), because it has run 2 times.` + ]); + expect(last.result.status).toBe(OperationStatus.Success); + expect(harness.runner.isActive).toBe(false); + expect((await harness.executeAsync()).output).toContain('Starting a warm worker for it.'); + }); + + it('closes a worker once its memory grew too much', async () => { + const harness: ITestHarness = await createHarnessAsync({ maxMemoryGrowth: 1.5 }); + harness.workerBehavior = 'grow'; + await harness.executeAsync(); + await harness.executeAsync(); + const pid: number = harness.runner.workerPid!; + expect(harness.runner.residentMemoryBytes).toBe(100000000); + + const grown: ITestIteration = await harness.executeAsync(); + expect(grown.output).toContain( + `Closing the warm worker (pid ${pid}), because its memory grew from 95 MB after its first run to 191 MB.` + ); + expect(harness.runner.isActive).toBe(false); + }); + + it('waits for a reused worker to report a change before it sends the next run, for at most the timeout', async () => { + const harness: ITestHarness = await createHarnessAsync({ changeReportTimeoutMs: 500 }); + await harness.executeAsync(); + await harness.executeAsync(); + const unreportedStart: number = performance.now(); + const unreported: ITestIteration = await harness.executeAsync(); + expect(performance.now() - unreportedStart).toBeGreaterThanOrEqual(450); + expect(unreported.output).toContain('Sending run 2 to the warm worker'); + + const reporting: ITestHarness = await createHarnessAsync({ changeReportTimeoutMs: 10000 }); + reporting.workerBehavior = 'request'; + await reporting.executeAsync(); + await reporting.executeAsync(); + const reportedStart: number = performance.now(); + const reported: ITestIteration = await reporting.executeAsync(); + expect(performance.now() - reportedStart).toBeLessThan(2000); + expect(reported.output).toContain('Sending run 2 to the warm worker'); + }); + + it('runs the initial command if the worker exits during a run, and starts a new worker for the next one', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + harness.workerBehavior = 'crash'; + const crashed: ITestIteration = await harness.executeAsync(); + expect(crashed.result.status).toBe(OperationStatus.Success); + expect(crashed.commands).toEqual([INCREMENTAL_IPC_COMMAND, INITIAL_COMMAND]); + expectLinesInOrder(crashed.output, [ + 'Starting a warm worker for it.', + 'The warm worker exited before it reported the result of its run (exit code 3).', + 'Running the initial command, because the run on the warm worker failed.', + `Invoking (initial): ${INITIAL_COMMAND}` + ]); + expect(crashed.record.error).toBeUndefined(); + expect(getCommandExecution(crashed.record)).toEqual({ kind: 'initial', hasIncrementalCommand: true }); + expect(harness.runner.isActive).toBe(false); + + harness.workerBehavior = undefined; + const next: ITestIteration = await harness.executeAsync(); + expect(next.result.status).toBe(OperationStatus.Success); + expect(next.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(next.output).toContain('Starting a warm worker for it.'); + }); + + it('terminates the worker if the operation is aborted, and starts a new worker for the next run', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + harness.workerBehavior = 'hang'; + const running: Promise = harness.waitForWorkerRunAsync(); + const execution: Promise = harness.executeAsync(); + await running; + const worker: ChildProcess = children[1]; + await harness.graph.abortCurrentIterationAsync({ terminateRunning: true }); + const aborted: ITestIteration = await execution; + expect(aborted.result.status).toBe(OperationStatus.Aborted); + // An aborted run is not a failure, so the initial command does not run. + expect(aborted.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(aborted.output).toContain('Terminated because the operation was aborted.'); + expect(worker.signalCode).toBe('SIGTERM'); + expect(harness.runner.isActive).toBe(false); + + harness.workerBehavior = undefined; + const next: ITestIteration = await harness.executeAsync(); + expect(next.result.status).toBe(OperationStatus.Success); + expect(next.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(next.output).toContain('Starting a warm worker for it.'); + }); + + it('closes the worker after its run if the runner should not persist', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + harness.shouldRunnerPersist = false; + const closed: ITestIteration = await harness.executeAsync(); + expect(closed.result.status).toBe(OperationStatus.Success); + expect(closed.commands).toEqual([INCREMENTAL_IPC_COMMAND]); + expect(harness.runner.isActive).toBe(false); + expect(children[1].exitCode).toBe(0); + }); + + it('waits for the worker to exit in each concurrent close', async () => { + const harness: ITestHarness = await createHarnessAsync(); + await harness.executeAsync(); + await harness.executeAsync(); + const worker: ChildProcess = children[1]; + const exitCodes: (number | null)[] = []; + await Promise.all( + [harness.runner.closeAsync(), harness.runner.closeAsync()].map(async (promise: Promise) => { + await promise; + exitCodes.push(worker.exitCode); + }) + ); + expect(exitCodes).toEqual([0, 0]); + await harness.runner.closeAsync(); + }); +}); + +describe(DaemonWarmWorkerPlugin.name, () => { + it('creates warm worker runners only for opted-in operations with an incremental ipc script and no other runner', async () => { + const scripts: Record = { + '_phase:build': 'heft run --only build -- --clean', + '_phase:build:incremental:ipc': 'heft run-watch --only build --' + }; + const optedIn: IOperationSettings = { operationName: phase.name, allowDaemonWarmWorker: true }; + const createOperation = ( + name: string, + projectScripts: Record | undefined, + runner?: NullOperationRunner, + settings: IOperationSettings | false = optedIn + ): Operation => + new Operation({ + phase, + project: createProject(name, projectScripts), + runner, + settings: settings || undefined, + logFilenameIdentifier: name + }); + const warm: Operation = createOperation('warm', scripts); + const warmWithIpc: Operation = createOperation('warm-with-ipc', { + ...scripts, + '_phase:build:ipc': 'heft run-watch --only build -- --clean' + }); + // `rush start` runs the same script in watch mode, so the script alone does not opt in. + const withoutSettings: Operation = createOperation('without-settings', scripts, undefined, false); + const withoutField: Operation = createOperation('without-field', scripts, undefined, { + operationName: phase.name + }); + const optedOut: Operation = createOperation('opted-out', scripts, undefined, { + operationName: phase.name, + allowDaemonWarmWorker: false + }); + const withoutIpcScript: Operation = createOperation('without-ipc-script', { + '_phase:build': scripts['_phase:build'] + }); + const withoutScript: Operation = createOperation('without-script', { + '_phase:build:incremental:ipc': scripts['_phase:build:incremental:ipc'] + }); + const existingRunner: NullOperationRunner = new NullOperationRunner({ + name: 'existing', + result: OperationStatus.NoOp, + silent: true + }); + const withRunner: Operation = createOperation('with-runner', scripts, existingRunner); + + // Like PhasedScriptAction, which applies the plugin after ShellOperationRunnerPlugin. + const hooks: PhasedCommandHooks = new PhasedCommandHooks(); + new ShellOperationRunnerPlugin().apply(hooks); + new DaemonWarmWorkerPlugin().apply(hooks); + const context: ICreateOperationsContext = { + isIncrementalBuildAllowed: true, + isWatch: false + } as unknown as ICreateOperationsContext; + await hooks.createOperationsAsync.promise( + new Set([ + warm, + warmWithIpc, + withoutSettings, + withoutField, + optedOut, + withoutIpcScript, + withoutScript, + withRunner + ]), + context + ); + + expect(warm.runner).toBeInstanceOf(WarmWorkerOperationRunner); + expect(warmWithIpc.runner).toBeInstanceOf(WarmWorkerOperationRunner); + expect(withoutSettings.runner).toBeInstanceOf(ShellOperationRunner); + expect(withoutField.runner).toBeInstanceOf(ShellOperationRunner); + expect(optedOut.runner).toBeInstanceOf(ShellOperationRunner); + expect(withoutIpcScript.runner).toBeInstanceOf(ShellOperationRunner); + expect(withoutScript.runner).toBeInstanceOf(NullOperationRunner); + expect(withRunner.runner).toBe(existingRunner); + + // Its build cache entries are those of ShellOperationRunner. + const shellHooks: PhasedCommandHooks = new PhasedCommandHooks(); + new ShellOperationRunnerPlugin().apply(shellHooks); + const shellTwin: Operation = createOperation('shell-twin', scripts); + await shellHooks.createOperationsAsync.promise(new Set([shellTwin]), context); + expect(shellTwin.runner).toBeInstanceOf(ShellOperationRunner); + expect(warm.runner!.getConfigHash()).toBe(shellTwin.runner!.getConfigHash()); + + // Watch mode has its own IPC runners, and a command that allows no incremental run does not need workers. + for (const otherContext of [ + { ...context, isWatch: true }, + { ...context, isIncrementalBuildAllowed: false } + ]) { + const operation: Operation = createOperation('other', scripts); + await hooks.createOperationsAsync.promise(new Set([operation]), otherContext); + expect(operation.runner).toBeInstanceOf(ShellOperationRunner); + } + }); +}); diff --git a/libraries/rush-lib/src/schemas/rush-project.schema.json b/libraries/rush-lib/src/schemas/rush-project.schema.json index e45130007c..67173fd2c9 100644 --- a/libraries/rush-lib/src/schemas/rush-project.schema.json +++ b/libraries/rush-lib/src/schemas/rush-project.schema.json @@ -60,6 +60,11 @@ } }, + "allowDaemonWarmWorker": { + "type": "boolean", + "description": "If true, and daemon.warmWorkers is enabled, the Rush daemon keeps the operation's \":incremental:ipc\" script running as a warm worker between builds, and sends it the incremental runs that the incremental execution guard allows. Set this only if that script, run in watch mode, runs every task and check that the \"\" script runs, because the daemon reports a run on the worker as if the \"\" script had run. For Heft, that means the \"lintInWatchMode\" option of heft-lint-plugin and the \"runInWatchMode\" setting of API Extractor, which both default to skipping their task in watch mode." + }, + "outputFolderNames": { "type": "array", "description": "Specify the folders where this operation writes its output files. If enabled, the Rush build cache will restore these folders from the cache. The strings are folder names under the project root folder. These folders should not be tracked by Git. They must not contain symlinks.", diff --git a/libraries/rush-lib/src/schemas/rush.schema.json b/libraries/rush-lib/src/schemas/rush.schema.json index ed1b151020..87939878e1 100644 --- a/libraries/rush-lib/src/schemas/rush.schema.json +++ b/libraries/rush-lib/src/schemas/rush.schema.json @@ -282,6 +282,11 @@ "default": true, "description": "Let daemon builds run an operation's `:incremental` script on top of the outputs of its last successful run in the daemon, when only files that it builds were edited since then and its output folders are unchanged. Otherwise the initial script runs, as it does for native Rush. Results of an incremental script are not written to the build cache. RUSH_DAEMON_INCREMENTAL_BUILDS overrides." }, + "warmWorkers": { + "type": "boolean", + "default": false, + "description": "Keep a watch-mode worker (the `:incremental:ipc` script) alive between daemon builds for each operation whose rush-project.json operation settings set `allowDaemonWarmWorker`, and send it the next incremental run when incrementalBuilds allows one. Otherwise the worker is closed and the initial script runs. Requires incrementalBuilds. RUSH_DAEMON_WARM_WORKERS overrides." + }, "queueTimeoutSeconds": { "type": "number", "minimum": 0, From 5a1996bb748a7f1fd28b8c93af0a74a5068bb173 Mon Sep 17 00:00:00 2001 From: Sean Larkin <3408176+TheLarkInn@users.noreply.github.com> Date: Tue, 29 Sep 2026 06:15:46 +0000 Subject: [PATCH 085/265] [rush-daemon] The daemon logs each shutdown with its reason, and exits when its socket file is lost Swarm integration step 70; original commit 689084765b (merge of swarm/o04-t151-x156 at 5503c45528). Scope: tasks 151 and 156. Brings o04's task 151 (board 3120) and r01's task 156 (board 3253), merged together by o04 (board 3190) because they conflict in one doc comment. 151: rushd logs each shutdown with its reason, time and PID, and its ready line gets the time and PID. 156: a daemon whose socket file was deleted or replaced notices within 5 s and exits once no request runs (initiator socketLost), so the next build starts a new daemon without the 15 s wait and the in-process fallback. Second agents: t02 CONFIRMED 151 board 3249 and 156 board 3302. s17 batch D3, item 2 of 8. Gate: ch01 GATE OK board 3747 (tree adbb8b5631) Commits folded into this step (3): - b5e7243165 rush-daemon: a daemon whose socket file was deleted or replaced exits once idle (task 156) - ad32ffa563 [rush-daemon] Log each shutdown with its reason and PID, and add the time and PID to the ready line - 5503c45528 Name a lost socket in the daemon's shutdown line Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c --- apps/rush-cli-client/README.md | 8 + .../src/test/launchClient.test.ts | 13 +- .../daemon-shutdown-log_2026-09-29-01-40.json | 10 + .../r01-socket-check_2026-09-29-02-10.json | 11 ++ .../daemon-shutdown-log_2026-09-29-01-40.json | 10 + .../r01-socket-check_2026-09-29-02-10.json | 11 ++ .../reviews/api/rush-daemon-transport.api.md | 4 + common/reviews/api/rush-daemon.api.md | 2 +- .../src/DaemonFileChange.ts | 57 ++++++ .../src/DaemonListener.ts | 6 + .../src/DaemonListenerLifetime.ts | 8 + libraries/rush-daemon-transport/src/index.ts | 1 + .../src/test/FileChange.test.ts | 66 +++++++ .../src/test/ListenerSocketCheck.test.ts | 75 ++++++++ libraries/rush-daemon/README.md | 6 + libraries/rush-daemon/src/DaemonIdleTimer.ts | 15 +- .../rush-daemon/src/DaemonShutdownError.ts | 10 +- .../rush-daemon/src/DaemonSocketWatch.ts | 36 ++++ .../rush-daemon/src/RushDaemonCommandLine.ts | 4 +- libraries/rush-daemon/src/RushDaemonHost.ts | 71 +++++++- .../src/SelectedDaemonBootstrap.ts | 5 +- .../src/test/DaemonIdleShutdown.test.ts | 3 + .../src/test/DaemonIdleTimer.test.ts | 33 ++++ .../src/test/DaemonInstallationChange.test.ts | 12 +- .../src/test/DaemonProcessExit.test.ts | 6 +- .../src/test/DaemonShutdown.test.ts | 55 ++++++ .../src/test/DaemonSocketLoss.test.ts | 171 ++++++++++++++++++ .../src/test/DaemonSocketWatch.test.ts | 54 ++++++ .../src/test/RushDaemonCommandLine.test.ts | 23 ++- .../src/test/SuccessfulNativeMutation.test.ts | 8 +- .../test/WorkspaceRestartArbitration.test.ts | 7 +- .../test/fixtures/SuccessfulMutationDaemon.ts | 3 +- 32 files changed, 784 insertions(+), 20 deletions(-) create mode 100644 common/changes/@rushstack/rush-cli-client/daemon-shutdown-log_2026-09-29-01-40.json create mode 100644 common/changes/@rushstack/rush-daemon-transport/r01-socket-check_2026-09-29-02-10.json create mode 100644 common/changes/@rushstack/rush-daemon/daemon-shutdown-log_2026-09-29-01-40.json create mode 100644 common/changes/@rushstack/rush-daemon/r01-socket-check_2026-09-29-02-10.json create mode 100644 libraries/rush-daemon-transport/src/DaemonFileChange.ts create mode 100644 libraries/rush-daemon-transport/src/test/FileChange.test.ts create mode 100644 libraries/rush-daemon-transport/src/test/ListenerSocketCheck.test.ts create mode 100644 libraries/rush-daemon/src/DaemonSocketWatch.ts create mode 100644 libraries/rush-daemon/src/test/DaemonSocketLoss.test.ts create mode 100644 libraries/rush-daemon/src/test/DaemonSocketWatch.test.ts diff --git a/apps/rush-cli-client/README.md b/apps/rush-cli-client/README.md index aad50dc10e..5255400fc7 100644 --- a/apps/rush-cli-client/README.md +++ b/apps/rush-cli-client/README.md @@ -540,6 +540,14 @@ parent closes its descriptor after spawning. On POSIX the launcher enforces mode `0600` and rejects linked destinations; Windows uses the existing per-user transport directory permissions. +Each daemon writes `