wwinp files: Fix MemoryError in WeightWindowsList.export_to_hdf5 and speed up from_wwinp#3942
Open
yrrepy wants to merge 2 commits into
Open
wwinp files: Fix MemoryError in WeightWindowsList.export_to_hdf5 and speed up from_wwinp#3942yrrepy wants to merge 2 commits into
yrrepy wants to merge 2 commits into
Conversation
Float/complex ndarrays are dtype-validated, so the per-element isinstance() scan is redundant. Also construct upper_ww_bounds in WeightWindows.__init__ as an ndarray multiplication (not a list comprehension) so the upper-bounds setter benefits too. ~11x speedup on 172M-element wwinp inputs (397 s -> 35 s).
The XML serialization raised MemoryError on bound arrays >~200M elements -- lxml's intermediate ASCII allocation fails before the text node can be built. Write HDF5 directly via h5py, mirroring the C++ WeightWindows::to_hdf5 writer. Critical details for C++ compatibility: - Bounds are 2D (ne, n_voxels) on disk (4D would segfault the C++ tensor::Tensor<double> reader). - max_lower_bound_ratio is written unconditionally (default 1.0). - Root attrs filetype and version are required by openmc_weight_windows_import. Mesh writing is handled by a private _write_mesh_group helper in weight_windows.py that dispatches by mesh type, matching the reference implementation. UnstructuredMesh raises NotImplementedError (wwinp cannot produce one).
yrrepy
force-pushed
the
wwinp_2GB_faster_B
branch
from
09ed5dc to
d3124ee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR:
WeightWindowsList.export_to_hdf5raisedMemoryErroron multi-GB wwinp inputs because the XML path built multi-GB ASCII strings inside lxml. Now writes HDF5 directly via h5py, matching the C++ writer.WeightWindowsList.from_wwinpwas dominated by per-elementisinstancechecks incheck_iterable_type(~90% of total time on multi-million- element bound arrays). Added a fast path for numpy float/complex ndarrays; ~11× speedup on a 172M-element wwinp (397 s → 35 s).This enables support for many-GB wwinp files and faster processing of them.
An alternative re-factoring style is available here:
https://github.com/yrrepy/openmc/tree/wwinp_2GB_faster
Checklist