Fix TypeScript coordinator hanging forever without a supervisor greeting - #72116
Fix TypeScript coordinator hanging forever without a supervisor greeting#72116ColtenOuO wants to merge 1 commit into
Conversation
CommChannel.connect() awaited the supervisor's first frame (the greeting) with no timeout of its own, unlike every later request on the same channel, which already times out after 30 seconds. A supervisor that connects the comm socket but never sends the greeting (a wedged or misbehaving supervisor, or a protocol bug) left the Node coordinator process waiting forever with no way to recover. connect() now applies the same 30 second default (overridable via ConnectOptions.timeoutMs), destroying the socket on timeout so the runtime fails fast instead of hanging.
jason810496
left a comment
There was a problem hiding this comment.
Thanks for the PR.
IIUC, the purpose of the timeout here is to teardown the subprocess itself. In another word, self-destroy to avoid the subprocess being hanging forever.
The supervisor side (the coordinator interface) will ensure the Lang SDKs subprocess will be cleaned up:
airflow/task-sdk/src/airflow/sdk/coordinators/_subprocess.py
Lines 288 to 321 in be4a1e6
airflow/task-sdk/src/airflow/sdk/coordinators/_subprocess.py
Lines 232 to 260 in be4a1e6
Additionally, the Execution API client on the supervisor side will retry the Execution API (which might takes over 30s) and the ti.run (or ti.patch) endpoint sometime take more than 30s. Forcing kill the subprocess itself terminates the chance of a success task execution.
Therefore I don't think this PR is necessary.
Summary
CommChannel.connect()(ts-sdk/src/coordinator/comm-channel.ts) opens the comm TCP socket and then awaitschannel.greeting.promise— the supervisor's first frame (StartupDetailsorDagFileParseRequest) — with no timeout of its own. Once the TCP connection succeeds, nothing bounds how long the Node coordinator process will wait for that first frame: if the socket stays open but the supervisor never writes the greeting (a wedged or misbehaving supervisor, a stuck process on the Python side, a protocol bug),connect()never resolves and never rejects, and the coordinator subprocess hangs indefinitely with no way to recover on its own.This is inconsistent with the rest of the same class: every subsequent
request()call on the channel already times out afterCOORDINATOR_REQUEST_TIMEOUT_MS(30 seconds,ts-sdk/src/coordinator/comm-channel.ts:54) viaDeferred.rejectAfter(), andsendResponse()supports an equivalent optional timeout that destroys the socket when a terminal write wedges. The greeting wait — which happens once, at startup, before any request/response traffic — was the one gap left uncovered.Change
ConnectOptions({ timeoutMs?: number }) and a third parameter onCommChannel.connect().connect()now armschannel.greeting.rejectAfter(timeoutMs, ...)right after opening the socket, defaultingtimeoutMsto the existingCOORDINATOR_REQUEST_TIMEOUT_MS(30s) for consistency with the rest of the channel. This reuses the same self-clearing-timerDeferredmechanism already used byrequest()andsendResponse()— no new timer bookkeeping.sock.destroy(err)) with a descriptive error, mirroring the existingsendResponsetimeout pattern, so a wedged supervisor connection is actually torn down instead of left dangling.connect()immediately and clears the timer, so there's no behavior change on the working path.Was generative AI tooling used to co-author this PR?