Skip to content

perf: parse GTF without per-line allocations (1.88x) - #140

Open
BenjaminDEMAILLE wants to merge 1 commit into
seqeralabs:mainfrom
BenjaminDEMAILLE:perf/gtf-parse-memchr
Open

perf: parse GTF without per-line allocations (1.88x)#140
BenjaminDEMAILLE wants to merge 1 commit into
seqeralabs:mainfrom
BenjaminDEMAILLE:perf/gtf-parse-memchr

Conversation

@BenjaminDEMAILLE

@BenjaminDEMAILLE BenjaminDEMAILLE commented Aug 14, 2026

Copy link
Copy Markdown

What

Removes the per-line allocations from parse_gtf().

Before, each line cost a String (reader.lines()) plus a Vec<&str> for the split fields, the attribute column was scanned once per attribute wanted, and gene_id / transcript_id were cloned 3-4 times per line to build map keys. On a human annotation that is tens of millions of allocations.

  • Read into a reused byte buffer with read_until instead of lines().
  • Split the 9 tab-separated fields into a fixed [&str; 9] with memchr, stopping after the 9th field.
  • Extract gene_id and transcript_id in one pass over the attribute column instead of two; get_attribute returns a borrowed &str so callers allocate only when keeping the value.
  • Probe the gene map and the transcript-builder map with borrowed keys, allocating owned String keys only on insert. TxKeyRef implements Hash + indexmap::Equivalent so an IndexMap<(String, String), _> can be looked up with &str slices.

Measurement

parse_gtf() on Homo_sapiens.GRCh38.113.gtf (4,114,455 lines, 1.66 GB uncompressed), aarch64 macOS, hyperfine -w 1 -r 5:

input before after
plain .gtf 2.373 s ± 0.017 s 1.264 s ± 0.010 s 1.88x
gzipped .gtf.gz 2.997 s ± 0.015 s 1.924 s ± 0.034 s 1.56x

(The gzipped row is on top of the current miniz_oxide backend; it compounds with #138.)

Parity

Identical output on the same annotation before and after: 78,932 genes / 387,944 transcripts / 2,164,410 exons / 195,851,333 total effective length.

Behaviour deliberately preserved in the edge cases:

  • lines with fewer than 9 fields are skipped, as split('\t').collect() + len() < 9 did;
  • a 10th field is truncated at the tab, as split('\t') did;
  • first occurrence wins when an attribute key repeats;
  • CRLF line endings are handled like lines() did.

New unit tests cover each of those. cargo test --release passes (203 + 12 + 18 + 2).


🤖 Generated with Claude Code

Part of #143.

The GTF parser allocated a `String` per line (`reader.lines()`), a `Vec`
per line for the split fields, and rescanned the attribute column once per
attribute it wanted. On a human annotation that is millions of allocations
before any real work happens.

Changes:

- Read into a reused byte buffer with `read_until` instead of `lines()`.
- Split the 9 tab-separated fields into a fixed `[&str; 9]` using `memchr`,
  stopping after the 9th field instead of collecting a `Vec`.
- Extract `gene_id` and `transcript_id` in a single pass over the attribute
  column rather than scanning it twice; `get_attribute` now returns a
  borrowed `&str` so callers allocate only when they keep the value.
- Look up genes and transcript builders with borrowed keys and allocate the
  owned `String` keys only when inserting a new entry. `TxKeyRef` implements
  `Hash` + `indexmap::Equivalent` so the `(String, String)` transcript map
  can be probed with `&str` slices.

Measured with `parse_gtf()` on Homo_sapiens.GRCh38.113.gtf (4,114,455 lines,
1.66 GB), aarch64 macOS, hyperfine -w 1 -r 5:

  plain:   2.373 s ± 0.017 s  ->  1.264 s ± 0.010 s   (1.88x)
  gzipped: 2.997 s ± 0.015 s  ->  1.924 s ± 0.034 s   (1.56x)

Output is unchanged: 78,932 genes / 387,944 transcripts / 2,164,410 exons /
195,851,333 total effective length, identical before and after.

Adds unit tests for the three new helpers, covering the short-line skip, the
10th-field truncation, CRLF handling, and first-occurrence-wins attribute
semantics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant