Skip to content

Match CLI downloads to Assembly results - #611

Merged
kvz merged 37 commits into
mainfrom
cli-downloads
Oct 24, 2026
Merged

kvz merged 37 commits into
mainfrom
cli-downloads

Conversation

@kvz

@kvz kvz commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Why

CLI downloads flatten HLS directories that playlists reference and save converted or bundled results under upload filenames. Public HTTP(S) inputs also fail before upload. Fixing those bugs must preserve the directory layout and freshness checks that existing scripts use.

Closes #593.
Closes #587.

Changes

  • Use Assembly result names and extensions for directory downloads, including bundled files and video-to-JPEG/JSON conversions. Preserve ordinary input-relative parent directories and the existing <input>/<step>/… layout for multiple results.
  • Retain time-based skipping for directory reruns: inputs with newer mapped outputs do not start another Assembly. --reprocess-stale still forces processing. Explicit file-output freshness behavior remains intact.
  • Keep HLS paths from meta.relative_path, separate independent source bundles, reserve linked paths together, and stage linked watch downloads before publishing them. Preserve source ownership across result reordering using the original path and filename. Reject unsafe paths and descendant symlinks.
  • Download public HTTP(S) inputs into isolated temporary uploads and clean them afterward. URL download deadlines default to five minutes and can be configured with --url-download-timeout; watch mode requires local inputs.
  • Reserve result paths with case and Unicode normalization rules detected independently at the destination root; normalize collision keys after case folding.
  • Ship minor changesets for @transloadit/node and transloadit, describing the layout repairs as bug fixes, per Kevin’s 2026-10-23 decision. Synchronize the generated legacy README.

Removing input parent folders only relocates working files, so that change has been dropped from this PR. #621 tracks an opt-in layout instead. HLS directories and incorrect result filenames remain repaired.

#595 already fixed inputless directory downloads and non-TTY stdin. Both checkboxes are ticked on #593; live imports on main succeeded with /dev/null and socket-backed fd 0.

Validation

  • The restored directory freshness test first failed because the CLI started an extra Assembly. It now passes for standalone and directory inputs, verifies that existing bytes stay intact, and checks the explicit reprocessing override.
  • Red-first CLI behavior tests cover video-to-JPEG/JSON filenames and the input/Step layout. Reinstating the old single-result download behavior reproduces both Save CLI results with their own file extension, not the input's #587 extension failures. A live /video/thumbs CLI run saved testsrc_0.jpg with verified JPEG bytes and no .mp4 output.
  • Existing HLS, URL, collision, watch-ordering, and Unicode behavior regressions pass. The ordering test uses explicit concurrency 1 and associates results with their actual uploads.
  • corepack yarn check passes, including all 1,119 Node tests (one test skipped), including 60 download-layout cases. Two live freshness E2E tests pass. Generated legacy-package sync and type checks pass.
  • Earlier live reproductions demonstrated the original failures; all 13 HLS result checksums matched, independent and combined playlists decoded with ffmpeg, and repeated URL inputs retained both uploads/results.

Final CI passes on 1d5bb2aa: all 14 checks, including Node 20/22/24/26, E2E, full verification, and fixtures. The single blocking gh-run-watch completed with exit 0.

Review and follow-ups

Completed the two authorized review rounds on the rework diff only (DSP-473, option B, 2026-10-23), then stopped. All four retained findings are fixed:

  • Round 1: preserve per-input folders for directory-output intents, propagate URL/base64 staging metadata, and retain watch changes when an older download finishes during their freshness check. Four behavior cases failed before the fixes and now pass.
  • Round 2: refresh an input’s mapped single-file output in place after changes, so the next unchanged rerun skips processing again. Two behavior cases failed before the fix and now pass for standalone and directory inputs, including the explicit reprocessing override. Unrelated result paths keep collision protection.

No retained review finding remains unaddressed. The final freshness fix was tested after round 2; no third round was run.

Reviewer tim-kos remains requested. The PR is assigned to kvz on project 12.

Overlap

Rebased onto current main after #610 and #612 merged. #610’s CLI reporting and error formatting are preserved; shared assemblies.ts and intentRuntime.ts hunks cover download/path logic, transient-input metadata, and the URL download option. #612’s upload stream cleanup is preserved; this PR does not change Transloadit.ts or tus.ts.

🤖 Claimed by kvz on studio1/node-sdk since 2026-10-23 · resume: codex resume 01a16a73-d73f-7e11-83fe-db48d62fa526

@kvz kvz self-assigned this Oct 23, 2026
@kvz
kvz marked this pull request as ready for review October 23, 2026 16:56
kvz and others added 10 commits October 23, 2026 19:02
* Report Assembly references and concise CLI failures

* Preserve reporting output modes and sync legacy CLI docs

* Skip empty ignored-error warning details

* Report command failures once at the boundary

* Preserve transport and batch failure diagnostics

* Explain missing Template and steps inputs

* Keep accepted Assembly references and sanitize parse failures

* Retain creation attempt references and sync input errors

* Limit ignored-error JSON to diagnostic fields

* Validate Assembly diagnostic payloads before reporting

* Include structured ignored-error details in CLI warnings
* Close upload file streams when Assembly creation fails

* Track and close tus chunk streams on upload failure

* Track tus chunks from caller-supplied file streams

* Wait for pending file slices before rejecting Assembly creation

* Stop sends after abort and clean up caller cancellation

* Guard queued cancellation at the HTTP send boundary

* Stop tus retry timers after startup cancellation

* Allow upload listeners on the private cancellation signal

* Synchronize retry cancellation tests with send attempts
@kvz
kvz merged commit 3675b8a into main Oct 24, 2026
17 checks passed
@kvz
kvz deleted the cli-downloads branch October 24, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make CLI downloads match the Assembly's results Save CLI results with their own file extension, not the input's

1 participant