perf: buffer all output file writers (2.4x wall clock) - #139
Open
BenjaminDEMAILLE wants to merge 1 commit into
Open
perf: buffer all output file writers (2.4x wall clock)#139BenjaminDEMAILLE wants to merge 1 commit into
BenjaminDEMAILLE wants to merge 1 commit into
Conversation
Most of the writer functions wrote line by line straight into a `std::fs::File`, which means one `write(2)` syscall per line. Nine call sites already wrapped the file in a `BufWriter`; the other 28 did not, so the behaviour was inconsistent rather than deliberate. It matters because several outputs are large and scale with input size: inner_distance's detail file emits one line per read pair (1,000,000 lines on the benchmark input below), the TIN table one line per transcript (150,001), and each of the three junction_annotation files one line per junction (73,663). Measured on a synthetic 4M-read coordinate-sorted BAM (24 contigs, random sequences) with a 1.2M-line GTF (60k genes), aarch64 macOS, --threads 4, 5 interleaved runs, medians: before: real 23.16 s user 24.03 s sys 13.45 s after: real 9.72 s user 23.20 s sys 0.70 s 2.4x wall clock. System time drops by 95%, which is the syscall traffic disappearing; user time is unchanged, as expected for a change that does no less work, only fewer syscalls. Each converted writer gets an explicit `flush()?` before returning so a failed final flush is reported rather than swallowed by `BufWriter`'s drop. All deterministic outputs are byte-identical before and after. `cargo test --release` passes (200 + 12 + 18 + 2). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BenjaminDEMAILLE
force-pushed
the
perf/buffer-output-writers
branch
from
August 14, 2026 00:32
7200ebd to
41afdcf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wraps the remaining 28 output-file writers in
BufWriter. Nine call sites already did this; the rest wrote line by line straight into astd::fs::File, i.e. onewrite(2)syscall per line.Why it matters
Several outputs scale with input size:
rseqc/inner_distance/*.inner_distance.txtrseqc/tin/*.tin.xlsrseqc/junction_annotation/*.junction.{xls,bed,Interact.bed}featurecounts/*.tsv,dupradar/*_dupMatrix.txtMeasurement
Synthetic 4M-read coordinate-sorted BAM (24 contigs, random sequences), 1.2M-line GTF / 60k genes, aarch64 macOS,
--threads 4. Five interleaved runs per binary, medians:main2.4x wall clock. System time falls 95% — that is the syscall traffic disappearing. User time is unchanged, which is the expected signature: the same work, far fewer syscalls.
Correctness
Each converted writer gets an explicit
flush()?before returning, so a failing final flush is reported instead of being swallowed byBufWriter'sDrop.All deterministic outputs are byte-identical before and after.
cargo test --releasepasses (200 + 12 + 18 + 2).Pre-existing issue found while verifying
Three outputs are not reproducible run-to-run on
mainalone — two runs of the same binary on the same input differ:dupradar/*_duprateExpDens.svg(and the matching.png)qualimap/qualimapReport.htmlqualimap/rnaseq_qc_results.txtThe differences are pure line/element reordering of equal-valued entries, so no number changes, but the files do not hash-match. It looks like
HashMapiteration order reaching output (compute_biasinqualimap/output.rshas a comment acknowledging a related tie-break). Unrelated to this PR — happy to open a separate issue.🤖 Generated with Claude Code
Part of #143.