Repository navigation
perf: parse computeMatrix matrices in Rust in read_matrix_file - #1492
Open
MisterSheikh wants to merge 1 commit into
Open
MisterSheikh wants to merge 1 commit into
MisterSheikh wants to merge 1 commit into
Conversation
heatmapper.read_matrix_file, used by plotHeatmap, plotProfile and computeMatrixOperations, splits every line in Python and converts each value with float(). For an 82k x 3,600 matrix this takes about 20 s, half of plotHeatmap's runtime. hp.r_read_matrix (matrixreader.rs) decompresses the file on one thread and parses chunks of lines on a rayon pool with the GIL released. Values are parsed with str::parse::<f64>, which is correctly rounded like Python's float() and accepts nan/inf. It returns the parameter lines, the six region fields per row and the values as a float64 bytearray. read_matrix_file wraps the buffer with np.frombuffer as a MaskedArray without a mask (the same type the per-line reader produced) and builds regions and groups as before. The reader uses up to 16 threads from the CPU affinity mask. The per-line reader is kept as _read_matrix_file_legacy and is used for anything the fast path does not handle: file objects, rows of unequal length, tokens str::parse rejects (e.g. "1_0"), parameter lines after data rows, or an hp module without r_read_matrix. test_readMatrixFile.py reads every matrix in the test data with both readers and checks that matrix type, dtype, shape, mask, value bits (NaN and -0.0 included), regions, parameters, labels, boundaries and sorting settings are identical, and that the fallback is taken for a file object and for an unusual token.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1490.
read_matrix_fileparses the matrix line by line in Python, which takes about 20 s on an 81,918 x 3,600 matrix. This addshp.r_read_matrix, which does the decompression and parsing in Rust in parallel.read_matrix_filereturns exactly the same matrix, regions and groups as before. The old parser stays as a fallback for anything the fast path doesn't handle, such as file objects or ragged rows.test_readMatrixFile.pyruns both parsers on every matrix in the test data and checks that they match exactly. They also match on the public matrix from #1490, andplotHeatmapproduces the same PNG.Median of 2 runs on that matrix:
read_matrix_fileplotHeatmapplotProfilecomputeMatrixOperations infoPeak memory is 8–20% lower.
Two notes:
plotHeatmapandplotProfilehave no-p, so the reader uses up to 16 threads. Happy to change that.src/lib.rs, where both add amodline; keep both. If chore: Split hp into core rlib + optional python layer #1485 lands first,matrixreaderneeds to go behind itspythonfeature.