perf: parse GTF without per-line allocations (1.88x) - #140
Open
BenjaminDEMAILLE wants to merge 1 commit into
Open
perf: parse GTF without per-line allocations (1.88x)#140BenjaminDEMAILLE wants to merge 1 commit into
BenjaminDEMAILLE wants to merge 1 commit into
Conversation
The GTF parser allocated a `String` per line (`reader.lines()`), a `Vec` per line for the split fields, and rescanned the attribute column once per attribute it wanted. On a human annotation that is millions of allocations before any real work happens. Changes: - Read into a reused byte buffer with `read_until` instead of `lines()`. - Split the 9 tab-separated fields into a fixed `[&str; 9]` using `memchr`, stopping after the 9th field instead of collecting a `Vec`. - Extract `gene_id` and `transcript_id` in a single pass over the attribute column rather than scanning it twice; `get_attribute` now returns a borrowed `&str` so callers allocate only when they keep the value. - Look up genes and transcript builders with borrowed keys and allocate the owned `String` keys only when inserting a new entry. `TxKeyRef` implements `Hash` + `indexmap::Equivalent` so the `(String, String)` transcript map can be probed with `&str` slices. Measured with `parse_gtf()` on Homo_sapiens.GRCh38.113.gtf (4,114,455 lines, 1.66 GB), aarch64 macOS, hyperfine -w 1 -r 5: plain: 2.373 s ± 0.017 s -> 1.264 s ± 0.010 s (1.88x) gzipped: 2.997 s ± 0.015 s -> 1.924 s ± 0.034 s (1.56x) Output is unchanged: 78,932 genes / 387,944 transcripts / 2,164,410 exons / 195,851,333 total effective length, identical before and after. Adds unit tests for the three new helpers, covering the short-line skip, the 10th-field truncation, CRLF handling, and first-occurrence-wins attribute semantics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes the per-line allocations from
parse_gtf().Before, each line cost a
String(reader.lines()) plus aVec<&str>for the split fields, the attribute column was scanned once per attribute wanted, andgene_id/transcript_idwere cloned 3-4 times per line to build map keys. On a human annotation that is tens of millions of allocations.read_untilinstead oflines().[&str; 9]withmemchr, stopping after the 9th field.gene_idandtranscript_idin one pass over the attribute column instead of two;get_attributereturns a borrowed&strso callers allocate only when keeping the value.Stringkeys only on insert.TxKeyRefimplementsHash+indexmap::Equivalentso anIndexMap<(String, String), _>can be looked up with&strslices.Measurement
parse_gtf()onHomo_sapiens.GRCh38.113.gtf(4,114,455 lines, 1.66 GB uncompressed), aarch64 macOS,hyperfine -w 1 -r 5:.gtf.gtf.gz(The gzipped row is on top of the current
miniz_oxidebackend; it compounds with #138.)Parity
Identical output on the same annotation before and after: 78,932 genes / 387,944 transcripts / 2,164,410 exons / 195,851,333 total effective length.
Behaviour deliberately preserved in the edge cases:
split('\t').collect()+len() < 9did;split('\t')did;lines()did.New unit tests cover each of those.
cargo test --releasepasses (203 + 12 + 18 + 2).🤖 Generated with Claude Code
Part of #143.