Cut the peak memory of binsparse_to_ssmc_problem - #34
Merged
Conversation
Reading a very large problem back needs several times the memory of writing it, because every intermediate in the conversion is an nnz-length array and the biggest matrices in the collection have nnz in the billions. For Sybrandt/AGATHA_2015 (n=183,964,077, nnz=11,588,725,964) the export runs but the read back fails with MATLAB:pmaxsize, which on Linux is an out-of-memory error in disguise. None of this changes what the converter produces; it changes how much scratch space it takes to produce it. - checked_indices validates on the stored integer type before widening to double. For an unsigned index array the finiteness and integrality tests are vacuous and the range tests collapse to min and max reductions, so the check no longer builds a second nnz-length temporary to hold fix(indices). - matrix_values keeps an ISO matrix in ISO form and returns a flag saying so. sparse() expands a scalar over the index vectors, so the single stored value never has to be repmat'ed to nnz length. expand_iso does the expansion in the three dense branches, which genuinely need an array. - expand_structure keeps the ISO form across mirroring whenever mirroring leaves the stored value alone, which covers every symmetric matrix along with the real hermitian and all-zero skew-symmetric ones. - convert_matrix takes a separate branch for ISO matrices, whose entries are either all zero or all nonzero, so it needs no mask and no masked copies of the index vectors. For the general case it tests values == 0 directly, as == already compares both parts of a complex value; the old form called imag() on real values and got back an nnz-length array of zeros. When nothing is an explicit zero it hands the index vectors to sparse() untouched rather than copying them through a mask. - require_ordered_pairs and require_strictly_increasing scan in blocks with a one-entry overlap. require_ordered_pairs also evaluated diff(first) twice. Tests cover the new ISO paths: a nonzero ISO COOR matrix, ISO with symmetric_lower staying ISO, ISO with skew_symmetric_lower and with hermitian_lower dropping it, and a dense ISO DVEC. The 14-matrix BSP test collection still round-trips byte-exact through ssread; five of those matrices, Pajek/IMDB among them, store ISO values and so exercise the new path on real data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Memory optimizations to reduce the amount of memory used by when converting from BSP to a SSMC Matlab struct.