Skip to content

fix(deploy): stop waiting for missing hashes when the revision stream ends - #149

Merged
piscisaureus merged 3 commits into
mainfrom
fix/diffsync-stream-end
Nov 2, 2026
Merged

piscisaureus merged 3 commits into
mainfrom
fix/diffsync-stream-end

Conversation

@piscisaureus

Copy link
Copy Markdown
Member

The diff-sync upload learns which files to send by watching the
revision through revisions.watchUntilReady until its labels carry
deno.diffsync.missing_hashes. The promise behind "Loading previously
uploaded files..." was only ever resolved from onData; onStopped
merely unsubscribed and nothing handled onComplete. When the console
ended the stream before the builder reported (the build failed or was
cancelled during preparing, the revision was superseded by a concurrent
deploy, or the server never wrote the label at all) the CLI either hung
on the spinner or died with Top-level await promise never resolved
and no indication of what went wrong (denoland/deploy_feedback#949).

  • waitForMissingHashes owns the watch and settles on every exit of
    the stream: the label resolves it; a failed or cancelled snapshot,
    the stream ending (onStopped/onComplete), a subscription error,
    a label that is not a JSON string array, or the timeout reject with
    a MissingHashesError naming the cause. The subscription is always
    unsubscribed and the timer cleared, and callbacks that fire after
    settling are ignored.
  • A failed or cancelled revision is reported as such, with the
    console's build_failure message when it has one, the revision's
    console URL as the hint, and REVISION_FAILED /
    REVISION_CANCELLED in the --json envelope (exit 1), matching
    waitForRevision. A stream that ends without the label, the
    timeout, and a malformed label exit 6 (NETWORK) with
    DIFFSYNC_STREAM_ENDED, DIFFSYNC_TIMEOUT and
    DIFFSYNC_MALFORMED_RESPONSE, each pointing at the build page and
    suggesting a retry.
  • The wait is bounded by MISSING_HASHES_TIMEOUT_MS (10 minutes). The
    report arrives while the build is still preparing, so a wait that
    long means the build never started, and the deploy fails with a
    message instead of running until the CI job's own timeout.
  • The happy path is unchanged: the label arrives, the watch
    unsubscribes, and the upload proceeds with the parsed hashes.
  • tests/missing_hashes.test.ts drives a fake subscription through
    every path: the label arriving (including "[]" and one delivered
    before subscription() returns), the stream ending with and without
    a snapshot, a failed snapshot, a cancelled stream end, a subscription
    error, malformed labels, the timeout, and the message/exit-code
    mapping.

Fixes denoland/deploy_feedback#949.

… ends

The diff-sync upload waits on `revisions.watchUntilReady` until the
revision's labels carry `deno.diffsync.missing_hashes`. The promise was
only resolved from `onData`, so when the console ended the stream first
(the build failed or was cancelled while preparing, a concurrent deploy
superseded the revision, or the label was never written) the CLI hung
on "Loading previously uploaded files..." or crashed with "Top-level
await promise never resolved".

* `waitForMissingHashes` settles on every exit: label -> resolve;
  terminal snapshot, stream end, subscription error, malformed label
  or timeout -> reject with a `MissingHashesError` naming the cause.
* Failed/cancelled revisions report their status, the `build_failure`
  message and the build page URL (REVISION_FAILED/REVISION_CANCELLED,
  exit 1); other causes exit 6 with DIFFSYNC_* codes and a retry hint.
* The wait is bounded by MISSING_HASHES_TIMEOUT_MS (10 minutes).
* Tests drive a fake subscription through every path.

Fixes denoland/deploy_feedback#949.

@avocet-bot avocet-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — #149

PR: fix(deploy): stop waiting for missing hashes when the revision stream ends
Reviewed head SHA: 2d3e4fc7474eca66a5039f916d29f7a995a48c32
Verdict: Not blocking. The fix is correct and well-tested. I'm leaving this as a comment (not an approval) because there are two concrete, non-blocking robustness/consistency findings worth the author's attention before or shortly after merge.


What this PR does (background for a reader new to the code)

deploy-cli is the command-line tool that uploads a Deno application to Deno Deploy. For incremental ("diff-sync") uploads, the CLI must learn which files it still needs to send: it uploads a manifest of file hashes, the server-side builder compares them against its cache, and the console publishes the set of still-missing hashes back as a label on the revision object — deno.diffsync.missing_hashes, a JSON-encoded array of hash strings ("[]" when nothing is missing).

The CLI learns that value by opening a tRPC subscription (revisions.watchUntilReady) and watching the stream of revision snapshots until one carries that label. A tRPC subscription delivers callbacks: onData (a new snapshot), onError (the subscription failed), onStopped (the subscription was stopped), and onComplete (the stream ended normally).

The bug being fixed (deno_feedback#949): the old code resolved its "Loading previously uploaded files..." promise only from onData when the label appeared. onStopped merely unsubscribed, onComplete was not handled, and a thrown JSON.parse was unhandled. So whenever the console ended the stream before the label was written — a build that failed or was cancelled during preparation, a revision superseded by a concurrent deploy, or the label simply never being written — the promise never settled. The CLI hung forever on the spinner, or Deno aborted the process with Top-level await promise never resolved, in both cases with no diagnostic.

The fix: a new waitForMissingHashes() (in deploy/publish.ts) owns the subscription and is guaranteed to settle on every exit of the stream:

  • label present and parseable → resolve with the parsed hashes;
  • revision reached a terminal status (failed/cancelled) → reject (revision_terminal);
  • stream ended (onStopped/onComplete) without the label → reject (stream_ended);
  • subscription error → reject (subscription_error);
  • label present but not a JSON array of strings → reject (malformed_label);
  • nothing within MISSING_HASHES_TIMEOUT_MS (10 minutes) → reject (timeout).

Each rejection carries a typed MissingHashesFailure, and describeMissingHashesFailure() turns it into a user-facing message + exit code + a console-URL hint. publish() catches MissingHashesError and routes it through the existing error() helper (which prints and Deno.exits). Terminal revisions map to exit 1 (REVISION_FAILED/REVISION_CANCELLED); the other causes map to exit 6 (NETWORK) with DIFFSYNC_* codes and a retry hint. A new test file (tests/missing_hashes.test.ts) drives a fake subscription through every path.

How I reviewed it

  • Read the full diff and the surrounding publish() integration at the reviewed head SHA.
  • Traced the control/data flow against the error-helper contract (util.ts: error() is typed : never and calls Deno.exit; ExitCode.NETWORK === 6, GENERIC === 1) and the tRPC client wiring (auth.ts).
  • Confirmed CI: the deno job (typecheck + the new tests) is green at this head, which also validates that onStopped/onComplete are legal subscription-observer options in the pinned @trpc/client@^11.

Correctness / concurrency checks — all pass

  • Callback firing before subscription() returns. A tRPC callback can run synchronously during the subscription(...) call, before the returned handle is assigned to sub. The code handles this: finish()'s sub?.unsubscribe() is a no-op while sub is still undefined, and the trailing if (settled) sub.unsubscribe(); performs exactly one unsubscribe once sub is assigned. Covered by the "settles before subscribe returns" test (asserts unsubscribed === 1).
  • No double-unsubscribe. Async-settle paths unsubscribe once inside finish() (the trailing guard is then a no-op because settled was false at subscribe-return). Sync-settle paths unsubscribe once via the trailing guard. onStopped + onComplete both routing to streamEnded() is idempotent via the settled guard in fail(). No path unsubscribes twice.
  • No timer leak. The 10-minute setTimeout is armed before the subscription; every settle path funnels through finish(), which clearTimeouts it (including the resolve path, which calls finish() before resolve(...)). Deno's test timer/resource sanitizers would fail the resolve-path tests otherwise.
  • settled guard. fail() and onData both early-return once settled; the timeout test confirms a post-deadline snapshot is ignored.
  • Label precedence over terminal status. onData checks the label branch before the isTerminal branch, so a terminal snapshot that still carries the label resolves rather than rejecting — covered by a dedicated test. Correct.
  • Definite assignment in publish(). Reading missingHashes/revision after the try/catch is sound only because the catch never completes normally: it either throws the non-MissingHashesError or calls error() which returns never. Verified against util.ts.
  • Spinner handling. existingFilesSpinner.stop() runs inside the catch before both the rethrow and the error() exit, so the spinner is always stopped on failure. The second stop() after the try is only reached on the success path (the catch always diverges), so it is never actually a double-stop at runtime.
  • revision is never undefined where consumed. On success waitForMissingHashes resolves a concrete Revision; every failure path exits via error() before revision is passed to waitForRevision(). The | undefined is only a lexical type.
  • Malformed-label value detection is robust. parseMissingHashes rejects non-JSON, non-arrays, and arrays containing non-strings, while accepting "[]" and valid string arrays. Covered by the ["not json","{}",'["a1", 2]',"null"] test.

Non-blocking findings

1. onData trusts the envelope shape; a snapshot with a missing/non-object labels degrades to a misleading 10-minute timeout (or an uncaught TypeError)

Background & intended behavior. The PR is carefully defensive about the label value (parseMissingHashes), but the snapshot envelope is a blind revision = data as Revision. The very next line evaluates MISSING_HASHES_LABEL in revision.labels.

Failure scenario. If the server ever delivers a snapshot whose labels is absent or null, <key> in null throws a TypeError inside the onData callback. That throw does not pass through fail(), so waitForMissingHashes is not settled by it — the promise then settles only when the 10-minute MISSING_HASHES_TIMEOUT_MS fires, and the user is told DIFFSYNC_TIMEOUT ("the build may be stuck or queued"), which misdescribes the cause. Depending on how the tRPC link surfaces a throw from onData, the TypeError may instead surface as an opaque unhandled error.

Impact / why it matters. This is the same class of gap the PR set out to close (a stream event that doesn't settle the promise cleanly), just for a malformed envelope rather than a malformed value. It is pre-existing — the old code also did "deno.diffsync.missing_hashes" in typedData.labels — so it is not a regression, and it only triggers if the server violates its own contract. That's why it's non-blocking. Still, since this PR is specifically hardening this wait, guarding the shape would close the gap completely: treat a non-object/null labels as a malformed_label failure (e.g. const labels = (data as Revision)?.labels; if (typeof labels !== "object" || labels === null) { fail({ reason: "malformed_label", revision, value: Deno.inspect(data) }); return; }).

Anchor: deploy/publish.ts, onData handler — revision = data as Revision; if (MISSING_HASHES_LABEL in revision.labels) { ... }.

2. subscription_error is categorized inconsistently with the other failures (and with the commit message)

Background & intended behavior. The commit message states "other causes exit 6 with DIFFSYNC_* codes and a retry hint." stream_ended, timeout, and malformed_label all follow that: ExitCode.NETWORK (6), a DIFFSYNC_* errorCode, and (for the first two) a retry hint.

Failure scenario. A subscription error is the most network-shaped of all these causes (dropped connection, reset, TLS failure — the test even uses new Error("connection reset")), yet describeMissingHashesFailure's subscription_error case returns code: ExitCode.GENERIC (1), no errorCode (so the --json envelope falls back to "GENERIC" via exitCodeToName), and no retry hint. The message is a raw Deno.inspect(unknown), which can be opaque or multi-line.

Impact / why it matters. An automated caller that keys on exit code 6 to decide "transient network error, retry" will instead see 1 for a dropped connection and treat it as a hard failure. Two mitigating facts keep this non-blocking: (a) it contradicts only the commit message, not any documented API; and (b) this branch is in practice largely unreachable in production — the outermost errorLink in auth.ts intercepts subscription errors and itself calls error()/Deno.exit without forwarding observer.error, so the consumer's onError rarely fires there. (That same fact means the DIFFSYNC_STREAM_ENDED messaging only applies if the production tRPC/SSE link surfaces a premature server close as complete/stopped rather than as an error — consistent with the original hang in #949, where an error-shaped end would already have exited. The tests mock this with onStopped()+onComplete() rather than exercising the real link; worth confirming against the live console.) Suggested fix if you want the branch to be correct when it does fire: code: ExitCode.NETWORK, errorCode: "DIFFSYNC_SUBSCRIPTION_ERROR", plus a retry hint matching stream_ended.

Anchor: deploy/publish.ts, describeMissingHashesFailure — case "subscription_error":.

3. (Informational, out of scope) The sibling waitForRevision still has the exact bug this PR fixes

waitForRevision() (unchanged by this PR) resolves its completionPromise only in onComplete; its onStopped just unsubscribes. If the --wait path's stream is stopped with onStopped and no following onComplete, await completionPromise.promise hangs forever — the identical "stream ended without settling" hazard this PR eliminates for the missing-hashes wait. It's outside this diff, so not a finding against this PR, but it's a natural follow-up (and the new test helper's assumption that onStopped() is always followed by onComplete() is the same assumption that would need to hold for waitForRevision not to hang).

Tests

tests/missing_hashes.test.ts is thorough: label arrival (including "[]" and a report delivered synchronously before subscription() returns), stream end with and without a prior snapshot, failed/cancelled snapshots, terminal-with-label, subscription error, the four malformed-label values, the timeout (including a post-deadline snapshot being ignored), and the message/exit-code mapping for each cause. The only runtime-reachable path not exercised end-to-end is finding #1's malformed envelope (there's no test where labels is absent/null).

Review skill evidence

I invoked two Claude Code review subagents from the pr-review-toolkit plugin via the Task tool:

  • pr-review-toolkit:code-reviewer — full correctness/concurrency/security pass over the diff; confirmed no ≥80-confidence bugs and validated the callback-ordering / timer / unsubscribe / settled-guard logic.
  • pr-review-toolkit:silent-failure-hunter — error-handling and silent-failure pass; surfaced findings #1 and #2 above and confirmed the catch-block/spinner and definite-assignment reasoning.

Their findings are reconciled with my own line-by-line trace above; nothing either agent reported rises to a merge blocker.


Conclusion: The change does what it claims and genuinely removes the hang/crash on a prematurely-ended revision stream, with strong test coverage and green CI (deno job passing at this head). No blockers. Please consider the two non-blocking items — hardening the onData envelope shape (#1) and aligning the subscription_error exit code/errorCode with the other NETWORK causes (#2) — as small robustness/consistency follow-ups.

@avocet-bot avocet-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — #149

PR: fix(deploy): stop waiting for missing hashes when the revision stream ends
Reviewed head SHA: 3024f90ff5f323214c63deb09750e2edfb440a2e
Verdict: ✅ Approve. The change fixes a real hang/crash, is comprehensively tested, CI is green at this head, and the two follow-up commits on this re-queued head resolved both non-blocking findings from my earlier review of head 2d3e4fc. No blocking issues; no new issues introduced.


What this PR does (background for a reader new to the code)

deploy-cli is the command-line tool that uploads a Deno application to Deno Deploy. For incremental ("diff-sync") uploads, the CLI must learn which files it still needs to send: it uploads a manifest of file hashes, the server-side builder compares them against its cache, and the console publishes the set of still-missing hashes back as a label on the revision object — deno.diffsync.missing_hashes, a JSON-encoded array of hash strings ("[]" when nothing is missing).

The CLI learns that value by opening a tRPC subscription (revisions.watchUntilReady) and watching a stream of revision snapshots until one carries that label. A tRPC subscription delivers callbacks: onData (a new snapshot), onError (the subscription failed), onStopped (the subscription was stopped), and onComplete (the stream ended normally).

The bug being fixed (deno_feedback#949): the old code resolved its "Loading previously uploaded files..." promise only from onData when the label appeared. onStopped merely unsubscribed, onComplete was unhandled, and a thrown JSON.parse was unhandled. So whenever the console ended the stream before the label was written — a build that failed or was cancelled during preparation, a revision superseded by a concurrent deploy, or the label simply never being written — the promise never settled. The CLI hung forever on the spinner, or Deno aborted with Top-level await promise never resolved, in both cases with no diagnostic.

The fix: a new waitForMissingHashes() (in deploy/publish.ts) owns the subscription and is guaranteed to settle on every exit of the stream:

  • label present and parseable → resolve with the parsed hashes;
  • revision reached a terminal status (failed/cancelled) → reject (revision_terminal);
  • stream ended (onStopped/onComplete) without the label → reject (stream_ended);
  • subscription error → reject (subscription_error);
  • a snapshot whose labels is missing/null/non-object/array, or a label value that is not a JSON array of strings → reject (malformed_label);
  • nothing within MISSING_HASHES_TIMEOUT_MS (10 minutes) → reject (timeout).

Each rejection carries a typed MissingHashesFailure, and describeMissingHashesFailure() turns it into a user-facing message + exit code + a console-URL hint. publish() catches MissingHashesError and routes it through the existing error() helper (which prints and Deno.exits). Terminal revisions map to exit 1 (REVISION_FAILED/REVISION_CANCELLED); the other causes map to exit 6 (NETWORK) with DIFFSYNC_* codes and a retry hint. A new test file (tests/missing_hashes.test.ts) drives a fake subscription through every path.

Review history (this is a re-queued head)

I reviewed an earlier head (2d3e4fc) and left a non-blocking comment with two findings. The author then pushed two follow-up commits (297ea50, 3024f90, now the head) that address both. This review re-verifies the full function at the current head and focuses on the incremental delta:

  • Prior finding #1 — onData trusted the snapshot envelope shape, so a snapshot with a missing/null labels threw TypeError inside the callback (bypassing the settle machinery) and degraded to a misleading 10-minute timeout. Fixed. onData now guards the envelope before using it: const labels = (data as Partial<Revision> | null)?.labels; if (typeof labels !== "object" || labels === null || Array.isArray(labels)) { fail({ reason: "malformed_label", revision, value: Deno.inspect(data) }); return; }. The malformed_label failure's revision field was correctly widened to Revision | undefined (the guard can fire on the first snapshot, before any revision is recorded), and no consumer dereferences failure.revision for that case. A new test exercises {status only}, {labels:null}, {labels:[]}, and null.
  • Prior finding #2 — subscription_error exited GENERIC (1) with no errorCode/retry hint, inconsistent with the other NETWORK causes and the commit message. Fixed. It now returns ExitCode.NETWORK (6), errorCode: "DIFFSYNC_SUBSCRIPTION_ERROR", a descriptive message, and a retry hint. A comment also documents honestly that the production tRPC client's errorLink in auth.ts owns subscription-error exit, so this consumer-level handler primarily serves a link-less client — which is accurate.

How I reviewed it

  • Read the full diff vs base plus the incremental diff 2d3e4fc..3024f90, and traced the new guard against every snapshot shape by hand.
  • Verified the error-helper contract (util.ts: error() is typed : never and calls Deno.exit; ExitCode.NETWORK === 6, GENERIC === 1), and the tRPC client wiring (auth.ts errorLink).
  • Confirmed CI: the deno job (typecheck + the new tests, 15 cases) is green at this head. (A local deno check failure on lib/rs_lib.js is a pre-existing environmental artifact of the Rust/wasm build step, present at 2d3e4fc too, unrelated to this diff; CI builds that artifact and passes.)

Correctness / concurrency / throw-safety checks — all pass

  • No remaining throw escapes the settle machinery. Every statement in onData was checked: optional chaining makes the labels read null/primitive-safe; the guard is pure typeof/===/Array.isArray; KEY in revision.labels is safe because labels is already proven a non-null non-array object; parseMissingHashes wraps JSON.parse in try/catch and ToString-coerces any non-string value, returning undefined (→ malformed_label) rather than throwing; Deno.inspect does not throw on tRPC payloads. The latent TypeError-on-KEY in null that previously bypassed fail() is gone.
  • Callback firing before subscription() returns, no timer leak, no double-unsubscribe, idempotent settle. Unchanged from the previously-verified head and still correct: finish()/fail() are guarded by settled, clear the timeout, and unsubscribe exactly once (the trailing if (settled) sub.unsubscribe() covers the synchronous-settle-before-sub-assigned case). The new guard branch returns immediately after fail(), so there is no fall-through into revision = data. The new test asserts unsubscribed === 1 for every malformed-envelope vector.
  • Label precedence over terminal status preserved (a terminal snapshot still carrying the label resolves).
  • Definite assignment in publish() is sound because the catch never completes normally (error() is never, otherwise it rethrows). Spinner is stopped on every path.
  • revision is never dereferenced while undefined on any reachable path; the hint URL is built from revisionId, not from failure.revision.
  • Exit-code / cause mapping is now internally consistent: all five MissingHashesFailure reasons produce distinct, actionable messages; revision_terminal → exit 1, the rest → exit 6.

Non-blocking notes (not conditions of approval)

  1. (Informational) The improved subscription_error message is largely test-only in production: auth.ts's errorLink intercepts a real stream error and calls error()/Deno.exit before the consumer's onError runs, so on a genuine dropped connection the user sees the error-link's mapTrpcError message rather than the new DIFFSYNC_SUBSCRIPTION_ERROR text. No hang results either way, and the author's new comment documents this accurately. Nothing to change.
  2. (Low, worth a one-line confirmation) The envelope guard now treats a snapshot whose labels object is entirely absent as malformed_label and aborts the deploy. This matches the declared Revision contract (labels: Record<string,string> is required) and is strictly better than the old behavior (which threw and hung for 10 minutes on the same input). The only way it could misfire is if revisions.watchUntilReady ever legitimately emits an in-progress snapshot that omits the labels object rather than sending an empty {} (an empty object passes the guard and correctly keeps waiting). If you have the server contract handy, confirming labels is always serialized (empty, never omitted) on in-progress snapshots would fully retire this.
  3. (Cosmetic) malformed_label.value is typed string, but on the label-value path the value is only string by the Record<string,string> type, not at runtime; it's consumed via template-literal coercion so there is no functional risk. String(value) at the call site would make the type honest. Optional.
  4. (Out of scope, pre-existing) The sibling waitForRevision() — unchanged by this PR — still resolves its completion promise only in onComplete and merely unsubscribes in onStopped, so it carries the same "stream ended without settling" hazard this PR eliminates for the missing-hashes wait. Worth a follow-up, but not a finding against this diff.

Tests

tests/missing_hashes.test.ts is thorough and now covers the malformed-envelope guard (missing/null/array labels and null data) and the updated subscription_error mapping (exit 6, DIFFSYNC_SUBSCRIPTION_ERROR, retry hint), in addition to the previously-covered label arrival (incl. "[]" and sync-before-subscribe), stream end with/without a snapshot, failed/cancelled snapshots, terminal-with-label, subscription error, malformed label values, and the timeout. 15/15 pass on CI at this head.

Review skill evidence

I invoked two Claude Code review subagents from the pr-review-toolkit plugin via the Task tool, scoped to the current head 3024f90 and the incremental delta:

  • pr-review-toolkit:code-reviewer — confirmed the guard logic catches every malformed-envelope shape without rejecting legitimate snapshots, the type-widening is consistent with every consumer, no new race/double-settle, and the subscription_error remap type-checks; 15/15 tests pass.
  • pr-review-toolkit:silent-failure-hunter — traced onData statement-by-statement for any throw escaping fail() and found none; confirmed the consumer path never swallows errors and the exit-code mapping is correct; surfaced the three non-blocking notes above.

(Earlier heads in this session were also reviewed with the same two agents.)


Conclusion: This revision does what it claims and removes the hang/crash on a prematurely-ended or malformed revision stream, with strong test coverage and green CI. Both findings from the prior review round are resolved with accompanying tests, and the follow-up commits introduce no new issues. Approving.

@piscisaureus
piscisaureus merged commit 7bf3358 into main Nov 2, 2026
4 checks passed
@piscisaureus
piscisaureus deleted the fix/diffsync-stream-end branch November 2, 2026 04:43
@piscisaureus piscisaureus mentioned this pull request Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI: publish crashes with 'Top-level await promise never resolved' when watchUntilReady closes early

2 participants