Repository navigation
fix(deploy): stop waiting for missing hashes when the revision stream ends - #149
Conversation
… ends The diff-sync upload waits on `revisions.watchUntilReady` until the revision's labels carry `deno.diffsync.missing_hashes`. The promise was only resolved from `onData`, so when the console ended the stream first (the build failed or was cancelled while preparing, a concurrent deploy superseded the revision, or the label was never written) the CLI hung on "Loading previously uploaded files..." or crashed with "Top-level await promise never resolved". * `waitForMissingHashes` settles on every exit: label -> resolve; terminal snapshot, stream end, subscription error, malformed label or timeout -> reject with a `MissingHashesError` naming the cause. * Failed/cancelled revisions report their status, the `build_failure` message and the build page URL (REVISION_FAILED/REVISION_CANCELLED, exit 1); other causes exit 6 with DIFFSYNC_* codes and a retry hint. * The wait is bounded by MISSING_HASHES_TIMEOUT_MS (10 minutes). * Tests drive a fake subscription through every path. Fixes denoland/deploy_feedback#949.
avocet-bot
left a comment
There was a problem hiding this comment.
Review — #149
PR: fix(deploy): stop waiting for missing hashes when the revision stream ends
Reviewed head SHA: 2d3e4fc7474eca66a5039f916d29f7a995a48c32
Verdict: Not blocking. The fix is correct and well-tested. I'm leaving this as a comment (not an approval) because there are two concrete, non-blocking robustness/consistency findings worth the author's attention before or shortly after merge.
What this PR does (background for a reader new to the code)
deploy-cli is the command-line tool that uploads a Deno application to Deno Deploy. For incremental ("diff-sync") uploads, the CLI must learn which files it still needs to send: it uploads a manifest of file hashes, the server-side builder compares them against its cache, and the console publishes the set of still-missing hashes back as a label on the revision object — deno.diffsync.missing_hashes, a JSON-encoded array of hash strings ("[]" when nothing is missing).
The CLI learns that value by opening a tRPC subscription (revisions.watchUntilReady) and watching the stream of revision snapshots until one carries that label. A tRPC subscription delivers callbacks: onData (a new snapshot), onError (the subscription failed), onStopped (the subscription was stopped), and onComplete (the stream ended normally).
The bug being fixed (deno_feedback#949): the old code resolved its "Loading previously uploaded files..." promise only from onData when the label appeared. onStopped merely unsubscribed, onComplete was not handled, and a thrown JSON.parse was unhandled. So whenever the console ended the stream before the label was written — a build that failed or was cancelled during preparation, a revision superseded by a concurrent deploy, or the label simply never being written — the promise never settled. The CLI hung forever on the spinner, or Deno aborted the process with Top-level await promise never resolved, in both cases with no diagnostic.
The fix: a new waitForMissingHashes() (in deploy/publish.ts) owns the subscription and is guaranteed to settle on every exit of the stream:
- label present and parseable → resolve with the parsed hashes;
- revision reached a terminal status (
failed/cancelled) → reject (revision_terminal); - stream ended (
onStopped/onComplete) without the label → reject (stream_ended); - subscription error → reject (
subscription_error); - label present but not a JSON array of strings → reject (
malformed_label); - nothing within
MISSING_HASHES_TIMEOUT_MS(10 minutes) → reject (timeout).
Each rejection carries a typed MissingHashesFailure, and describeMissingHashesFailure() turns it into a user-facing message + exit code + a console-URL hint. publish() catches MissingHashesError and routes it through the existing error() helper (which prints and Deno.exits). Terminal revisions map to exit 1 (REVISION_FAILED/REVISION_CANCELLED); the other causes map to exit 6 (NETWORK) with DIFFSYNC_* codes and a retry hint. A new test file (tests/missing_hashes.test.ts) drives a fake subscription through every path.
How I reviewed it
- Read the full diff and the surrounding
publish()integration at the reviewed head SHA. - Traced the control/data flow against the error-helper contract (
util.ts:error()is typed: neverand callsDeno.exit;ExitCode.NETWORK === 6,GENERIC === 1) and the tRPC client wiring (auth.ts). - Confirmed CI: the
denojob (typecheck + the new tests) is green at this head, which also validates thatonStopped/onCompleteare legal subscription-observer options in the pinned@trpc/client@^11.
Correctness / concurrency checks — all pass
- Callback firing before
subscription()returns. A tRPC callback can run synchronously during thesubscription(...)call, before the returned handle is assigned tosub. The code handles this:finish()'ssub?.unsubscribe()is a no-op whilesubis stillundefined, and the trailingif (settled) sub.unsubscribe();performs exactly one unsubscribe oncesubis assigned. Covered by the "settles before subscribe returns" test (assertsunsubscribed === 1). - No double-unsubscribe. Async-settle paths unsubscribe once inside
finish()(the trailing guard is then a no-op becausesettledwas false at subscribe-return). Sync-settle paths unsubscribe once via the trailing guard.onStopped+onCompleteboth routing tostreamEnded()is idempotent via thesettledguard infail(). No path unsubscribes twice. - No timer leak. The 10-minute
setTimeoutis armed before the subscription; every settle path funnels throughfinish(), whichclearTimeouts it (including the resolve path, which callsfinish()beforeresolve(...)). Deno's test timer/resource sanitizers would fail the resolve-path tests otherwise. settledguard.fail()andonDataboth early-return once settled; the timeout test confirms a post-deadline snapshot is ignored.- Label precedence over terminal status.
onDatachecks the label branch before theisTerminalbranch, so a terminal snapshot that still carries the label resolves rather than rejecting — covered by a dedicated test. Correct. - Definite assignment in
publish(). ReadingmissingHashes/revisionafter thetry/catchis sound only because thecatchnever completes normally: it eitherthrows the non-MissingHashesErroror callserror()which returnsnever. Verified againstutil.ts. - Spinner handling.
existingFilesSpinner.stop()runs inside thecatchbefore both the rethrow and theerror()exit, so the spinner is always stopped on failure. The secondstop()after thetryis only reached on the success path (thecatchalways diverges), so it is never actually a double-stop at runtime. revisionis never undefined where consumed. On successwaitForMissingHashesresolves a concreteRevision; every failure path exits viaerror()beforerevisionis passed towaitForRevision(). The| undefinedis only a lexical type.- Malformed-label value detection is robust.
parseMissingHashesrejects non-JSON, non-arrays, and arrays containing non-strings, while accepting"[]"and valid string arrays. Covered by the["not json","{}",'["a1", 2]',"null"]test.
Non-blocking findings
1. onData trusts the envelope shape; a snapshot with a missing/non-object labels degrades to a misleading 10-minute timeout (or an uncaught TypeError)
Background & intended behavior. The PR is carefully defensive about the label value (parseMissingHashes), but the snapshot envelope is a blind revision = data as Revision. The very next line evaluates MISSING_HASHES_LABEL in revision.labels.
Failure scenario. If the server ever delivers a snapshot whose labels is absent or null, <key> in null throws a TypeError inside the onData callback. That throw does not pass through fail(), so waitForMissingHashes is not settled by it — the promise then settles only when the 10-minute MISSING_HASHES_TIMEOUT_MS fires, and the user is told DIFFSYNC_TIMEOUT ("the build may be stuck or queued"), which misdescribes the cause. Depending on how the tRPC link surfaces a throw from onData, the TypeError may instead surface as an opaque unhandled error.
Impact / why it matters. This is the same class of gap the PR set out to close (a stream event that doesn't settle the promise cleanly), just for a malformed envelope rather than a malformed value. It is pre-existing — the old code also did "deno.diffsync.missing_hashes" in typedData.labels — so it is not a regression, and it only triggers if the server violates its own contract. That's why it's non-blocking. Still, since this PR is specifically hardening this wait, guarding the shape would close the gap completely: treat a non-object/null labels as a malformed_label failure (e.g. const labels = (data as Revision)?.labels; if (typeof labels !== "object" || labels === null) { fail({ reason: "malformed_label", revision, value: Deno.inspect(data) }); return; }).
Anchor: deploy/publish.ts, onData handler — revision = data as Revision; if (MISSING_HASHES_LABEL in revision.labels) { ... }.
2. subscription_error is categorized inconsistently with the other failures (and with the commit message)
Background & intended behavior. The commit message states "other causes exit 6 with DIFFSYNC_* codes and a retry hint." stream_ended, timeout, and malformed_label all follow that: ExitCode.NETWORK (6), a DIFFSYNC_* errorCode, and (for the first two) a retry hint.
Failure scenario. A subscription error is the most network-shaped of all these causes (dropped connection, reset, TLS failure — the test even uses new Error("connection reset")), yet describeMissingHashesFailure's subscription_error case returns code: ExitCode.GENERIC (1), no errorCode (so the --json envelope falls back to "GENERIC" via exitCodeToName), and no retry hint. The message is a raw Deno.inspect(unknown), which can be opaque or multi-line.
Impact / why it matters. An automated caller that keys on exit code 6 to decide "transient network error, retry" will instead see 1 for a dropped connection and treat it as a hard failure. Two mitigating facts keep this non-blocking: (a) it contradicts only the commit message, not any documented API; and (b) this branch is in practice largely unreachable in production — the outermost errorLink in auth.ts intercepts subscription errors and itself calls error()/Deno.exit without forwarding observer.error, so the consumer's onError rarely fires there. (That same fact means the DIFFSYNC_STREAM_ENDED messaging only applies if the production tRPC/SSE link surfaces a premature server close as complete/stopped rather than as an error — consistent with the original hang in #949, where an error-shaped end would already have exited. The tests mock this with onStopped()+onComplete() rather than exercising the real link; worth confirming against the live console.) Suggested fix if you want the branch to be correct when it does fire: code: ExitCode.NETWORK, errorCode: "DIFFSYNC_SUBSCRIPTION_ERROR", plus a retry hint matching stream_ended.
Anchor: deploy/publish.ts, describeMissingHashesFailure — case "subscription_error":.
3. (Informational, out of scope) The sibling waitForRevision still has the exact bug this PR fixes
waitForRevision() (unchanged by this PR) resolves its completionPromise only in onComplete; its onStopped just unsubscribes. If the --wait path's stream is stopped with onStopped and no following onComplete, await completionPromise.promise hangs forever — the identical "stream ended without settling" hazard this PR eliminates for the missing-hashes wait. It's outside this diff, so not a finding against this PR, but it's a natural follow-up (and the new test helper's assumption that onStopped() is always followed by onComplete() is the same assumption that would need to hold for waitForRevision not to hang).
Tests
tests/missing_hashes.test.ts is thorough: label arrival (including "[]" and a report delivered synchronously before subscription() returns), stream end with and without a prior snapshot, failed/cancelled snapshots, terminal-with-label, subscription error, the four malformed-label values, the timeout (including a post-deadline snapshot being ignored), and the message/exit-code mapping for each cause. The only runtime-reachable path not exercised end-to-end is finding #1's malformed envelope (there's no test where labels is absent/null).
Review skill evidence
I invoked two Claude Code review subagents from the pr-review-toolkit plugin via the Task tool:
pr-review-toolkit:code-reviewer— full correctness/concurrency/security pass over the diff; confirmed no ≥80-confidence bugs and validated the callback-ordering / timer / unsubscribe / settled-guard logic.pr-review-toolkit:silent-failure-hunter— error-handling and silent-failure pass; surfaced findings #1 and #2 above and confirmed the catch-block/spinner and definite-assignment reasoning.
Their findings are reconciled with my own line-by-line trace above; nothing either agent reported rises to a merge blocker.
Conclusion: The change does what it claims and genuinely removes the hang/crash on a prematurely-ended revision stream, with strong test coverage and green CI (deno job passing at this head). No blockers. Please consider the two non-blocking items — hardening the onData envelope shape (#1) and aligning the subscription_error exit code/errorCode with the other NETWORK causes (#2) — as small robustness/consistency follow-ups.
…m error as retryable
avocet-bot
left a comment
There was a problem hiding this comment.
Review — #149
PR: fix(deploy): stop waiting for missing hashes when the revision stream ends
Reviewed head SHA: 3024f90ff5f323214c63deb09750e2edfb440a2e
Verdict: ✅ Approve. The change fixes a real hang/crash, is comprehensively tested, CI is green at this head, and the two follow-up commits on this re-queued head resolved both non-blocking findings from my earlier review of head 2d3e4fc. No blocking issues; no new issues introduced.
What this PR does (background for a reader new to the code)
deploy-cli is the command-line tool that uploads a Deno application to Deno Deploy. For incremental ("diff-sync") uploads, the CLI must learn which files it still needs to send: it uploads a manifest of file hashes, the server-side builder compares them against its cache, and the console publishes the set of still-missing hashes back as a label on the revision object — deno.diffsync.missing_hashes, a JSON-encoded array of hash strings ("[]" when nothing is missing).
The CLI learns that value by opening a tRPC subscription (revisions.watchUntilReady) and watching a stream of revision snapshots until one carries that label. A tRPC subscription delivers callbacks: onData (a new snapshot), onError (the subscription failed), onStopped (the subscription was stopped), and onComplete (the stream ended normally).
The bug being fixed (deno_feedback#949): the old code resolved its "Loading previously uploaded files..." promise only from onData when the label appeared. onStopped merely unsubscribed, onComplete was unhandled, and a thrown JSON.parse was unhandled. So whenever the console ended the stream before the label was written — a build that failed or was cancelled during preparation, a revision superseded by a concurrent deploy, or the label simply never being written — the promise never settled. The CLI hung forever on the spinner, or Deno aborted with Top-level await promise never resolved, in both cases with no diagnostic.
The fix: a new waitForMissingHashes() (in deploy/publish.ts) owns the subscription and is guaranteed to settle on every exit of the stream:
- label present and parseable → resolve with the parsed hashes;
- revision reached a terminal status (
failed/cancelled) → reject (revision_terminal); - stream ended (
onStopped/onComplete) without the label → reject (stream_ended); - subscription error → reject (
subscription_error); - a snapshot whose
labelsis missing/null/non-object/array, or a label value that is not a JSON array of strings → reject (malformed_label); - nothing within
MISSING_HASHES_TIMEOUT_MS(10 minutes) → reject (timeout).
Each rejection carries a typed MissingHashesFailure, and describeMissingHashesFailure() turns it into a user-facing message + exit code + a console-URL hint. publish() catches MissingHashesError and routes it through the existing error() helper (which prints and Deno.exits). Terminal revisions map to exit 1 (REVISION_FAILED/REVISION_CANCELLED); the other causes map to exit 6 (NETWORK) with DIFFSYNC_* codes and a retry hint. A new test file (tests/missing_hashes.test.ts) drives a fake subscription through every path.
Review history (this is a re-queued head)
I reviewed an earlier head (2d3e4fc) and left a non-blocking comment with two findings. The author then pushed two follow-up commits (297ea50, 3024f90, now the head) that address both. This review re-verifies the full function at the current head and focuses on the incremental delta:
- Prior finding #1 —
onDatatrusted the snapshot envelope shape, so a snapshot with a missing/nulllabelsthrewTypeErrorinside the callback (bypassing the settle machinery) and degraded to a misleading 10-minute timeout. Fixed.onDatanow guards the envelope before using it:const labels = (data as Partial<Revision> | null)?.labels; if (typeof labels !== "object" || labels === null || Array.isArray(labels)) { fail({ reason: "malformed_label", revision, value: Deno.inspect(data) }); return; }. Themalformed_labelfailure'srevisionfield was correctly widened toRevision | undefined(the guard can fire on the first snapshot, before anyrevisionis recorded), and no consumer dereferencesfailure.revisionfor that case. A new test exercises{status only},{labels:null},{labels:[]}, andnull. - Prior finding #2 —
subscription_errorexitedGENERIC(1) with noerrorCode/retry hint, inconsistent with the other NETWORK causes and the commit message. Fixed. It now returnsExitCode.NETWORK(6),errorCode: "DIFFSYNC_SUBSCRIPTION_ERROR", a descriptive message, and a retry hint. A comment also documents honestly that the production tRPC client'serrorLinkinauth.tsowns subscription-error exit, so this consumer-level handler primarily serves a link-less client — which is accurate.
How I reviewed it
- Read the full diff vs base plus the incremental diff
2d3e4fc..3024f90, and traced the new guard against every snapshot shape by hand. - Verified the error-helper contract (
util.ts:error()is typed: neverand callsDeno.exit;ExitCode.NETWORK === 6,GENERIC === 1), and the tRPC client wiring (auth.tserrorLink). - Confirmed CI: the
denojob (typecheck + the new tests, 15 cases) is green at this head. (A localdeno checkfailure onlib/rs_lib.jsis a pre-existing environmental artifact of the Rust/wasm build step, present at2d3e4fctoo, unrelated to this diff; CI builds that artifact and passes.)
Correctness / concurrency / throw-safety checks — all pass
- No remaining throw escapes the settle machinery. Every statement in
onDatawas checked: optional chaining makes thelabelsread null/primitive-safe; the guard is puretypeof/===/Array.isArray;KEY in revision.labelsis safe becauselabelsis already proven a non-null non-array object;parseMissingHasheswrapsJSON.parsein try/catch andToString-coerces any non-string value, returningundefined(→malformed_label) rather than throwing;Deno.inspectdoes not throw on tRPC payloads. The latentTypeError-on-KEY in nullthat previously bypassedfail()is gone. - Callback firing before
subscription()returns, no timer leak, no double-unsubscribe, idempotent settle. Unchanged from the previously-verified head and still correct:finish()/fail()are guarded bysettled, clear the timeout, and unsubscribe exactly once (the trailingif (settled) sub.unsubscribe()covers the synchronous-settle-before-sub-assigned case). The new guard branchreturns immediately afterfail(), so there is no fall-through intorevision = data. The new test assertsunsubscribed === 1for every malformed-envelope vector. - Label precedence over terminal status preserved (a terminal snapshot still carrying the label resolves).
- Definite assignment in
publish()is sound because thecatchnever completes normally (error()isnever, otherwise it rethrows). Spinner is stopped on every path. revisionis never dereferenced whileundefinedon any reachable path; the hint URL is built fromrevisionId, not fromfailure.revision.- Exit-code / cause mapping is now internally consistent: all five
MissingHashesFailurereasons produce distinct, actionable messages;revision_terminal→ exit 1, the rest → exit 6.
Non-blocking notes (not conditions of approval)
- (Informational) The improved
subscription_errormessage is largely test-only in production:auth.ts'serrorLinkintercepts a real stream error and callserror()/Deno.exitbefore the consumer'sonErrorruns, so on a genuine dropped connection the user sees the error-link'smapTrpcErrormessage rather than the newDIFFSYNC_SUBSCRIPTION_ERRORtext. No hang results either way, and the author's new comment documents this accurately. Nothing to change. - (Low, worth a one-line confirmation) The envelope guard now treats a snapshot whose
labelsobject is entirely absent asmalformed_labeland aborts the deploy. This matches the declaredRevisioncontract (labels: Record<string,string>is required) and is strictly better than the old behavior (which threw and hung for 10 minutes on the same input). The only way it could misfire is ifrevisions.watchUntilReadyever legitimately emits an in-progress snapshot that omits thelabelsobject rather than sending an empty{}(an empty object passes the guard and correctly keeps waiting). If you have the server contract handy, confirminglabelsis always serialized (empty, never omitted) on in-progress snapshots would fully retire this. - (Cosmetic)
malformed_label.valueis typedstring, but on the label-value path the value is onlystringby theRecord<string,string>type, not at runtime; it's consumed via template-literal coercion so there is no functional risk.String(value)at the call site would make the type honest. Optional. - (Out of scope, pre-existing) The sibling
waitForRevision()— unchanged by this PR — still resolves its completion promise only inonCompleteand merely unsubscribes inonStopped, so it carries the same "stream ended without settling" hazard this PR eliminates for the missing-hashes wait. Worth a follow-up, but not a finding against this diff.
Tests
tests/missing_hashes.test.ts is thorough and now covers the malformed-envelope guard (missing/null/array labels and null data) and the updated subscription_error mapping (exit 6, DIFFSYNC_SUBSCRIPTION_ERROR, retry hint), in addition to the previously-covered label arrival (incl. "[]" and sync-before-subscribe), stream end with/without a snapshot, failed/cancelled snapshots, terminal-with-label, subscription error, malformed label values, and the timeout. 15/15 pass on CI at this head.
Review skill evidence
I invoked two Claude Code review subagents from the pr-review-toolkit plugin via the Task tool, scoped to the current head 3024f90 and the incremental delta:
pr-review-toolkit:code-reviewer— confirmed the guard logic catches every malformed-envelope shape without rejecting legitimate snapshots, the type-widening is consistent with every consumer, no new race/double-settle, and thesubscription_errorremap type-checks; 15/15 tests pass.pr-review-toolkit:silent-failure-hunter— tracedonDatastatement-by-statement for any throw escapingfail()and found none; confirmed the consumer path never swallows errors and the exit-code mapping is correct; surfaced the three non-blocking notes above.
(Earlier heads in this session were also reviewed with the same two agents.)
Conclusion: This revision does what it claims and removes the hang/crash on a prematurely-ended or malformed revision stream, with strong test coverage and green CI. Both findings from the prior review round are resolved with accompanying tests, and the follow-up commits introduce no new issues. Approving.
The diff-sync upload learns which files to send by watching the
revision through
revisions.watchUntilReadyuntil its labels carrydeno.diffsync.missing_hashes. The promise behind "Loading previouslyuploaded files..." was only ever resolved from
onData;onStoppedmerely unsubscribed and nothing handled
onComplete. When the consoleended the stream before the builder reported (the build failed or was
cancelled during preparing, the revision was superseded by a concurrent
deploy, or the server never wrote the label at all) the CLI either hung
on the spinner or died with
Top-level await promise never resolvedand no indication of what went wrong (denoland/deploy_feedback#949).
waitForMissingHashesowns the watch and settles on every exit ofthe stream: the label resolves it; a failed or cancelled snapshot,
the stream ending (
onStopped/onComplete), a subscription error,a label that is not a JSON string array, or the timeout reject with
a
MissingHashesErrornaming the cause. The subscription is alwaysunsubscribed and the timer cleared, and callbacks that fire after
settling are ignored.
console's
build_failuremessage when it has one, the revision'sconsole URL as the hint, and
REVISION_FAILED/REVISION_CANCELLEDin the--jsonenvelope (exit 1), matchingwaitForRevision. A stream that ends without the label, thetimeout, and a malformed label exit 6 (NETWORK) with
DIFFSYNC_STREAM_ENDED,DIFFSYNC_TIMEOUTandDIFFSYNC_MALFORMED_RESPONSE, each pointing at the build page andsuggesting a retry.
MISSING_HASHES_TIMEOUT_MS(10 minutes). Thereport arrives while the build is still preparing, so a wait that
long means the build never started, and the deploy fails with a
message instead of running until the CI job's own timeout.
unsubscribes, and the upload proceeds with the parsed hashes.
tests/missing_hashes.test.tsdrives a fake subscription throughevery path: the label arriving (including
"[]"and one deliveredbefore
subscription()returns), the stream ending with and withouta snapshot, a failed snapshot, a cancelled stream end, a subscription
error, malformed labels, the timeout, and the message/exit-code
mapping.
Fixes denoland/deploy_feedback#949.