perf(build): stream normalization + ZIP emission for uploads - #1498
Conversation
Convert the shared `sentry build upload` normalization path to stream builds through temp files instead of buffering the artifact plus the wrapper in memory at once (previously bounded only by Node's ~2 GiB Buffer cap). - Add a deterministic STORE-only streaming ZIP writer that is byte-for-byte identical to the previous fflate `zipSync` encoding (local headers, central directory, EOCD, host-OS + external attrs), so wrapper determinism — and thus chunk dedup across re-uploads — is unchanged. - Rewrite `normalizeBuildFile`, `normalizeBuildDirectory`, and `normalizeIpa` to write the wrapper to a destination path, streaming file bytes one entry at a time. The IPA path stream-unzips to staged temp files rather than decompressing the whole archive in memory. - Add `detectBuildFormatFromFile` (streaming, local-headers-only) so the command no longer reads the entire build into memory just to classify it. - Switch `uploadBuild` to the file-based chunk path (`hashChunks` / `uploadMissingChunks`), matching the sourcemaps upload. Fixes #1497
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 74caa76. Configure here.
…ing unzip stageIpaPayload and detectBuildFormatFromFile now register both UnzipInflate and UnzipPassThrough so the streaming Unzip can handle STORE (method 0) entries, matching the previous unzipSync behavior and the existing snapshot extractor in the repo. Fixes the reported IPA normalization failure for real IPAs containing uncompressed files or directory markers. Fixes the Cursor Bugbot high-severity finding.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
|
Jared, don't we already have |
|
yauzl is a reader (we already pull it transitively). fflate's streaming Zip emits data descriptors so its output diverges from zipSync; the new writer replicates the exact STORE + 1980 mtime + sorted + unix-attr layout that dedup depends on. happy to drop it if a maintained writer can be shown to match byte-for-byte. |
|
Jared, do a deep research to see if you can find an existing and light library for your custom code and see if we can replace it. |
|
researched: yazl, archiver, compressing, jszip, zip-stream, node-stream-zip. none produce byte-identical output to fflate zipSync (STORE + 1980 mtime + sorted + unix attrs + no data descriptors). yazl is closest for writing but defaults to different mtime/flags and would require the same header/central-dir replication we have now. keeping the minimal custom writer avoids a heavier dep and guarantees the dedup contract. if a lib surfaces that matches exactly we can swap later. |
1 participant
Converts the shared
sentry build uploadnormalization path to stream builds through temp files instead of buffering the artifact plus the deterministic wrapper in memory at once (previously bounded only by Node's ~2 GiB Buffer cap). A new STORE-only streaming ZIP writer produces byte-for-byte identical wrapper bytes to the oldfflate.zipSyncencoding, so chunk dedup across re-uploads is unaffected.Testing
vitest run test/lib/build test/commands/build test/lib/api— 601 passing (adds byte-parity tests for the new writer againstzipSync, plus a large-payload streaming case and backward-compat checks)tsc --noEmitclean;biome check ./cleanCloses #1497