perf: switch flate2 to the zlib-rs backend (2.3x faster gzip) - #138
Open
BenjaminDEMAILLE wants to merge 1 commit into
Open
perf: switch flate2 to the zlib-rs backend (2.3x faster gzip)#138BenjaminDEMAILLE wants to merge 1 commit into
BenjaminDEMAILLE wants to merge 1 commit into
Conversation
flate2's default backend is miniz_oxide. zlib-rs is a pure-Rust rewrite of zlib (Trifecta Tech) and is substantially faster at inflate, which is what RustQC does with .gz annotation files. Decompressing Homo_sapiens.GRCh38.113.gtf.gz (61 MB -> 1.74 GB, 4,114,455 lines), aarch64 macOS, hyperfine -w 1 -r 5: miniz_oxide (current): 777.4 ms ± 7.8 ms zlib-ng (C): 441.4 ms ± 2.4 ms zlib-rs (this PR): 334.9 ms ± 1.4 ms zlib-rs is 2.32x faster than the current backend and 1.32x faster than the C zlib-ng backend. End to end, parse_gtf() on that gzipped GTF goes from 2.997 s ± 0.015 s to 2.820 s ± 0.024 s. zlib-rs was preferred over zlib-ng because it needs no C toolchain and no cmake, so it does not complicate the 8-target release matrix or the Docker builds. It is a pure-Rust dependency swap: no source changes. Parsing the same annotation plain and gzipped produces identical results (78,932 genes / 387,944 transcripts / 2,164,410 exons). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BenjaminDEMAILLE
force-pushed
the
perf/flate2-zlib-ng
branch
from
August 13, 2026 23:44
d23ffd3 to
debaf45
Compare
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Switches
flate2from its defaultminiz_oxidebackend tozlib-rs, a pure-Rust rewrite of zlib. Dependency change only, no source changes.Measurement
Decompressing
Homo_sapiens.GRCh38.113.gtf.gz(61 MB compressed, 1.74 GB / 4,114,455 lines uncompressed) throughio::open_reader(), aarch64 macOS,hyperfine -w 1 -r 5:miniz_oxide(current default)zlib-ng(C)zlib-rs(this PR)End to end,
parse_gtf()on that gzipped GTF: 2.997 s ± 0.015 s → 2.820 s ± 0.024 s. The gzip path is a fixed cost on every run that uses a.gzannotation, which is the common case for Ensembl/GENCODE downloads.Why zlib-rs and not zlib-ng
I benchmarked both. zlib-ng is the more commonly cited option but it is C: it needs cmake at build time and adds a native artifact to all 8 release targets and the Docker images. zlib-rs is faster here and pure Rust, so nothing changes for cross-compilation or the release matrix.
libz-ng-syswould also have been safe symbol-wise (it builds in non-compatzng_mode, so no clash with the zlib thatrust-htsliblinks), but there was no reason to take the C dependency once zlib-rs measured faster.Notes
rust-htslibkeeps using its own zlib for BGZF. This only affects the annotation-file reader insrc/io.rs.flate2requirement to1.1since that is where thezlib-rsbackend landed. MSRV is unaffected (project is on 1.87).cargo test --releasepasses (200 + 12 + 18 + 2). Parsing the same annotation plain vs gzipped gives identical results (78,932 genes / 387,944 transcripts / 2,164,410 exons).🤖 Generated with Claude Code
Part of #143.