[AI Improvement] [Task] Add process-tree teardown and shutdown timeout to integration CLIProcess - #11104
[AI Improvement] [Task] Add process-tree teardown and shutdown timeout to integration CLIProcess#11104joehan wants to merge 3 commits into
Conversation
…LIProcess (b/563411339)
There was a problem hiding this comment.
Code Review
This pull request improves process termination handling in 'scripts/integration-helpers/cli.ts' by introducing explicit shutdown timeouts, spawning processes as detached on non-Windows platforms, and implementing a robust termination sequence using SIGINT with a fallback SIGKILL timeout. A review comment correctly identifies a potential race condition where stopping a process could clear 'this.process' even if a new process has already been started, and suggests a fix to ensure the reference is only cleared if it matches the stopped process.
ajperel
left a comment
There was a problem hiding this comment.
My AI had feedback after asking it a bunch of questions, but I admit my process magic is not at this level to have much direct feedback.
| import { ChildProcess, execSync } from "child_process"; | ||
| import * as spawn from "cross-spawn"; | ||
|
|
||
| const SHUTDOWN_TIMEOUT_MS = 3000; |
There was a problem hiding this comment.
🔴 [Process Lifecycle & Timeout Mismatch] SHUTDOWN_TIMEOUT_MS (3000ms) is shorter than firebase's internal emulator shutdown timeout (4000ms) and process.kill(-pid) does not reach detached: true Java emulators
Rationale:
- In
src/emulator/downloadableEmulators.ts:451,firebase-toolsspawns every downloadable emulator (firestore,database,pubsub,storage,ui,dataconnect) withdetached: true. On Unix,detached: trueinvokessetsid(), placing each Java emulator into its own process group (PGID = java_pid), separate fromCLIProcess's process group (-pid). - Consequently,
process.kill(-pid, "SIGINT")andprocess.kill(-pid, "SIGKILL")only signalfirebaseand non-detached children (e.g. Functions workers)—they never signal the Java emulator processes. - Worse,
firebase's owndownloadableEmulators.stop()waits up toEMULATOR_INSTANCE_KILL_TIMEOUT = 4000ms (downloadableEmulators.ts:27) aftercontroller.onExit()(--export-on-exit). BecauseSHUTDOWN_TIMEOUT_MSis only3000ms, if--export-on-exit+ JVM teardown takes >3s,process.kill(-pid, "SIGKILL")kills the parentfirebaseCLI process mid-shutdown, preventing it from stopping the detached Java emulators and leaving them orphaned under PID 1 holding ports 8080/4400.
Suggested Fix:
- Increase
SHUTDOWN_TIMEOUT_MS(e.g., to10000ms) sofirebasehas sufficient time to completeexportOnExitandEMULATOR_INSTANCE_KILL_TIMEOUT(4000ms). - Before
SIGKILL-ing-pid, collect descendant PIDs viaPPID(whilepis still alive, beforepexits and reparents them to PID 1) so detached child emulators are also terminated on timeout.
There was a problem hiding this comment.
Addressed in commit 7058d19. Updated SHUTDOWN_TIMEOUT_MS to 10,000ms (10s) to give firebase ample time to complete --export-on-exit and respect the 4000ms JVM kill timeout (EMULATOR_INSTANCE_KILL_TIMEOUT). Added explanatory comments documenting the rationale behind both SHUTDOWN_TIMEOUT_MS and WINDOWS_KILL_TIMEOUT_MS.
| p.once("exit", (/* exitCode, signal */) => { | ||
| const pid = p.pid; | ||
| if (!pid || pid <= 0) { | ||
| if (this.process === p) { |
There was a problem hiding this comment.
🟡 [Simplification] Clear this.process = undefined once at the start of stop() instead of checking if (this.process === p) 4 times
Rationale:
- Lines 114 and 121 execute synchronously after
const p = this.process;(line 79) with noawaitin between, sothis.process === pis unconditionally true there. - Once
stop()capturesconst p = this.process,stop()only needs the local variablep. Settingthis.process = undefinedimmediately at line 83 (right afterif (!p) return Promise.resolve();) removes all 4if (this.process === p)blocks, guaranteesstop()'s.then()callbacks never clobber a subsequentstart(), and ensures a duplicatestop()call becomes a no-op instead of sending a secondSIGINT(which incommandUtils.ts:297triggers the double-Ctrl-Cbranch that skipscleanShutdown()).
There was a problem hiding this comment.
Addressed in commit 7058d19. Set this.process = undefined immediately upon entering stop(), removing the redundant if (this.process === p) checks across all exit branches, protecting against race conditions with subsequent start() invocations, and making duplicate stop() calls a no-op so double-Ctrl-C bypasses are avoided.
| }, SHUTDOWN_TIMEOUT_MS); | ||
|
|
||
| try { | ||
| process.kill(-pid, "SIGINT"); |
There was a problem hiding this comment.
🔴 [Signal Routing & Orphan Cleanup] Send SIGINT to p first and ensure surviving process tree members are reaped when p exits
Rationale:
- Sending
process.kill(-pid, "SIGINT")immediately signals non-detached child worker processes at the exact same instantfirebasestartscontroller.onExit()(--export-on-exit), which can tear down workers whilefirebaseis still exporting data. Sendingp.kill("SIGINT")allowsfirebase(commandUtils.ts:295-296) to orchestrateonExit()followed bycleanShutdown(). - In both the early-exit path (lines 120–125) and
exitPromise.then()(lines 155–160), once the leader processpemits"exit",clearTimeout(timeoutId)cancels the fallbackSIGKILL. Ifpcrashed or exited before a child process in-pid(or a detached child withPPID = pid), those surviving processes are never killed.
Suggested Fix:
const killProcessTree = (sig: NodeJS.Signals): void => {
try {
const children = execSync(`ps -o pid= --ppid ${pid}`, { stdio: ["pipe", "pipe", "ignore"] })
.toString()
.trim()
.split(/\s+/)
.filter(Boolean)
.map(Number);
for (const childPid of children) {
try {
process.kill(-childPid, sig);
} catch {
try {
process.kill(childPid, sig);
} catch {
// Child already exited.
}
}
}
} catch {
// No child processes found.
}
try {
process.kill(-pid, sig);
} catch {
try {
p.kill(sig);
} catch {
// Process already exited.
}
}
};There was a problem hiding this comment.
Addressed in commit 7058d19. We now send p.kill("SIGINT") directly to the leader process first to allow firebase to run onExit() and cleanShutdown(). If the shutdown timeout expires, killProcessTree("SIGKILL") queries child PIDs via ps -o pid= --ppid ${pid} and kills both child emulators and the root process group. In exitPromise.then(), any surviving processes in -pid are also killed with SIGKILL.
| import { ChildProcess, execSync } from "child_process"; | ||
| import * as spawn from "cross-spawn"; | ||
|
|
||
| const SHUTDOWN_TIMEOUT_MS = 3000; |
There was a problem hiding this comment.
If there's a good reason for these timeout #s might be nice to explain them in a comment.
Summary
Addresses Buganizer issue b/563411339.
This PR enhances
CLIProcessinscripts/integration-helpers/cli.tsto ensure clean process teardown on Unix platforms:detached: process.platform !== "win32"inCLIProcess.start()so that a dedicated process group is created on Unix.CLIProcess.stop(), sendsSIGINTto the process group (process.kill(-pid, "SIGINT")) to gracefully stop the top-level CLI and all child emulator processes (e.g. Java Firestore/PubSub emulators and worker runtimes).SHUTDOWN_TIMEOUT_MS) to force-kill the process group withSIGKILL(process.kill(-pid, "SIGKILL")) if processes fail to exit afterSIGINT, preventing integration tests from hanging indefinitely until outer CI timeouts.this.process = undefined.taskkill /pid ${p.pid} /T /F) intact.Verification
npm run build: Successful (exit code 0).npm run test:compile: Successful (exit code 0).eslint scripts/integration-helpers/cli.ts: 0 errors.mocha scripts/emulator-tests/unzipEmulators.spec.ts: Passed (2 passing in 6s).lsof -i :4400,lsof -i :8080): Verified clean with no lingering orphan processes.