Skip to content

Honour the zstd dictionary option in node:zlib - #7106

Open
oddharsh wants to merge 1 commit into
cloudflare:mainfrom
oddharsh:zstd-dictionary
Open

Honour the zstd dictionary option in node:zlib#7106
oddharsh wants to merge 1 commit into
cloudflare:mainfrom
oddharsh:zstd-dictionary

Conversation

@oddharsh

Copy link
Copy Markdown

Fixes #6967.

node:zlib's zstd functions accept a dictionary option and drop it. On compress that is
silent, because a frame compressed without a dictionary decodes fine with one, so the only
signal is a byte count that never shrank. On decompress it surfaces as
Zstd decompression failed: Data corruption detected against bytes that are not corrupt.

This wires ZSTD_CCtx_loadDictionary / ZSTD_DCtx_loadDictionary into the bindings from
#6007, following Node's own implementation
(nodejs/node@201304537e, nodejs/node#59240).

What changed

C++. ZstdContext::Options gains dictionary, and both initialize() overloads take a
kj::ArrayPtr<const kj::byte> whose empty case means no dictionary, which is how Node passes
the same thing. A load failure returns ERR_ZLIB_DICTIONARY_LOAD_FAILED, matching Node's error
code. The option reaches the convenience functions through zstdSync() and the streams through
ZstdCompressionStream::initialize().

TypeScript. One helper normalizes the option for both the stream class and the four
convenience functions, so the fast path and the info: true path agree.

Three details worth a reviewer's eye

Parameter order. ZSTD_CCtx_loadDictionary defers building the dictionary's tables until
the first frame begins, so the setParams loop that runs after initialize() still applies to
them. Measured against zstd 1.5.7 with this exact call order (create, load dictionary, set
level, ZSTD_compressStream2) on the issue's 8800-byte input: 63 bytes with no dictionary, 19
with the right one, and 63 with a wrong one, which are the numbers Node produces.
zstdDictionaryCollapseTest pins it by round-tripping one dictionary at level 19 and again at
level 1.

Dictionary content type. Both calls auto-detect: a buffer starting with the zstd dictionary
magic is read as a trained dictionary and anything else as raw content. That is what lets a
Worker participate in RFC 9842 Compression
Dictionary Transport, whose dictionaries are raw bytes, and it is the behaviour Node has.

Wrong types are ignored rather than rejected. Node accepts an ArrayBufferView or an
ArrayBuffer for zstd and quietly ignores any other type, unlike its zlib and brotli paths which
throw ERR_INVALID_ARG_TYPE (lib/zlib.js, class Zstd). I matched Node rather than
tightening it, since the convenience functions hand the options object straight to JSG, which
would otherwise throw a TypeError on a value Node accepts silently. Say the word if you would
rather have ERR_INVALID_ARG_TYPE on both paths and I will change it.

Tests

Seven cases added to zlib-zstd-nodejs-test.js:

  • a dictionary shrinks the output and round-trips
  • compressing a buffer against itself collapses the frame, at two compression levels
  • Buffer, Uint8Array, DataView and ArrayBuffer all behave identically (mirrors Node's
    test/parallel/test-zlib-zstd-dictionary.js)
  • a non-buffer dictionary is ignored on both paths (this one passes on main too, since the
    option is dropped there; it guards the behaviour rather than the fix)
  • a frame written with a dictionary fails to decode without it, and with the wrong one
  • the async convenience functions honour it
  • createZstdCompress / createZstdDecompress honour it

The mismatch test sets ZSTD_c_checksumFlag deliberately. A raw-content dictionary carries no
dictionary ID, so without a checksum, decoding that frame against a different dictionary
returns 8800 bytes of wrong data rather than an error (measured). With the checksum it is
Restored data doesn't match checksum, and decoding with no dictionary at all is
Data corruption detected either way.

What I ran, and what I could not

Ran and green: //src/workerd/api:compression (the C++ carrying the zstd calls),
//src/node:node@tsproject, //src/workerd/api/node/tests:zlib-zstd-nodejs-test@eslint, plus
the standalone zstd measurements above.

Could not run: the test target itself, and therefore the zlib-util.c++ plumbing. My machine
only has the beta Xcode, and its ld produces Rust proc-macro dylibs that the same machine's
dyld refuses to load (mis-aligned LINKEDIT string pool), so every [for tool] Rust crate
fails and the capnp codegen those tools drive cannot run. That is an Apple toolchain problem
rather than anything in this branch, and neither -ld_classic nor re-signing the dylib got past
it. Flagging it rather than implying a full local run: if CI turns something up I will fix it
promptly.

Not in this PR

BrotliContext::Options has no dictionary field either, and
BrotliCompressionStream::initialize() takes no dictionary argument, so brotli drops the option
the same way. Node supports it there through BrotliEncoderAttachPreparedDictionary. Happy to
follow up with that separately if you want it.

node:zlib's zstd functions accepted a `dictionary` option and dropped it. On
compress that was silent, since a frame compressed without a dictionary decodes
fine with one, so the only signal was a byte count that never shrank. On
decompress it surfaced as "Zstd decompression failed: Data corruption detected"
against bytes that were not corrupt.

Wire ZSTD_CCtx_loadDictionary and ZSTD_DCtx_loadDictionary into the two zstd
contexts and plumb the option through both the convenience functions and the
streams, following Node's implementation in nodejs/node#59240. Both calls
auto-detect the dictionary's content type, so a raw-content dictionary, which is
what RFC 9842 Compression Dictionary Transport ships, works as well as a trained
one.

A dictionary that is neither an ArrayBufferView nor an ArrayBuffer is ignored
rather than rejected, matching Node's Zstd class rather than its stricter zlib
and brotli paths.

Fixes cloudflare#6967
@oddharsh
oddharsh requested review from a team as code owners August 24, 2026 21:41
@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@oddharsh

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report — Runtime APIs: node:zlib zstd ignores the dictionary option (silent on compress, "Data corruption detected" on decompress)

1 participant