Export large weight window files through openmc.lib without XML serialization - #4057
Export large weight window files through openmc.lib without XML serialization#4057paulromano wants to merge 8 commits into
Conversation
| uid = self.id | ||
| base_dir = Path.cwd() if base_dir is None else Path(base_dir) | ||
|
|
||
| if isinstance(self, RegularMesh): |
There was a problem hiding this comment.
IMO, each mesh subclass should implement how to convert itself to its lib counterpart.
That way we don't have to edit this function when implementing a new mesh type.
There was a problem hiding this comment.
I tend to agree.
In the vein of my other comment, this also feels appropriate for a classmethod approach on the openmc.lib.Mesh object where each subclass handles its own property settings for the resulting openmc.lib object.
e.g.
lib_mesh = openmc.lib.Mesh.from_python_object(spherical_mesh).This makes it more natural (to me) to ensure that the openmc.lib module is initialized before performing these operations.
There was a problem hiding this comment.
Agreed. I moved the conversion to openmc.lib as openmc.lib.Mesh.from_python(mesh). Each concrete openmc.lib mesh subclass now implements its own _from_python() conversion hook, while the base method only selects the matching subclass and handles shared behavior such as the initialization check, ID, name, and base directory.
pshriwise
left a comment
pshriwise
left a comment
There was a problem hiding this comment.
Looks nice @paulromano! Good idea to address the duplicate implementation issue. There's more than enough to keep track of as it is.
Some design conversation to he bad about which side (Python API or openmc.lib) should initiate the object transfers.
| lib_meshes[mesh.id] = mesh.to_lib_object( | ||
| base_dir=original_dir) | ||
|
|
||
| lib_ww = openmc.lib.WeightWindows(ww.id) |
There was a problem hiding this comment.
Perhaps the openmc.lib.WeightWindows class could have a classmethod that takes in an openmc.WeightWindows object to handle some of the setup going on here.
There was a problem hiding this comment.
Good suggestion! I've added openmc.lib.WeightWindows.from_python(), which limits the logic here to managing the temporary session, deduplicating shared meshes, invoking the conversion, and calling the existing C++ exporter.
| int openmc_mesh_filter_set_translation(int32_t index, double translation[3]); | ||
| int openmc_mesh_get_id(int32_t index, int32_t* id); | ||
| int openmc_mesh_set_id(int32_t index, int32_t id); | ||
| int openmc_mesh_get_name(int32_t index, const char** name); |
There was a problem hiding this comment.
Let's add some tests for these new functions in test_lib.py
There was a problem hiding this comment.
Implemented new tests that cover these
|
|
||
| @pytest.mark.parametrize('dtype', (np.float16, np.float32, np.float64)) | ||
| @pytest.mark.parametrize('expected_type', (Real, float)) | ||
| def test_check_iterable_type_float_array(dtype, expected_type): |
There was a problem hiding this comment.
Not going to argue against more testing of course, but I'm curious as to what was the motivation for the addition of these wrt this PR?
There was a problem hiding this comment.
The motivation is the other major performance issue encountered with multi-GB wwinp files. WeightWindowsList.from_wwinp creates large NumPy floating-point arrays, and the previous check_iterable_type implementation performed an isinstance() check for every value. That accounted for most of the processing time in the 172 million element example (see #3942). The fast path uses the NumPy dtype when the dimensionality is valid, reducing that example from approximately 397 seconds to 35 seconds.
| uid = self.id | ||
| base_dir = Path.cwd() if base_dir is None else Path(base_dir) | ||
|
|
||
| if isinstance(self, RegularMesh): |
There was a problem hiding this comment.
I tend to agree.
In the vein of my other comment, this also feels appropriate for a classmethod approach on the openmc.lib.Mesh object where each subclass handles its own property settings for the resulting openmc.lib object.
e.g.
lib_mesh = openmc.lib.Mesh.from_python_object(spherical_mesh).This makes it more natural (to me) to ensure that the openmc.lib module is initialized before performing these operations.
|
Thanks @GuySten and @pshriwise for the review! All your comments have been addressed. |
3 participants
Description
Background
WeightWindowsList.export_to_hdf5()currently creates a temporary model containing the weight windows, writes that model to XML, initializes the OpenMC shared library from the XML, and then calls the existing C++ HDF5 exporter. For weight window files containing hundreds of millions of values, constructing the ASCII representation of the bounds can require multiple GBs of additional memory and eventually raiseMemoryError.#3942 addressed this by implementing a direct HDF5 writer in Python with h5py. #3951 refined that design by moving mesh serialization into methods on each mesh subclass. Both approaches avoid the large XML document, but they introduce a second implementation of the weight window HDF5 format alongside the existing C++ writer. This duplicates format logic across Python and C++, requires the Python implementation to remain synchronized with future format changes, and makes the C API responsible for passing HDF5-specific
hid_tvalues across the language boundary. PR #3951 also introduces a separate cleanup operation to manage objects created for this export path.Approach
This PR avoids XML serialization while continuing to use the existing C++ HDF5 writer.
WeightWindowsList.export_to_hdf5()initializes a minimal temporary OpenMC library session and creates the required meshes and weight windows directly throughopenmc.lib. Once the C++ objects have been populated, it calls the existingopenmc.lib.export_weight_windows()function. The weight window bounds therefore never need to be represented in XML.A new public
MeshBase.to_lib_object()method creates the corresponding runtime mesh in an initialized OpenMC library session. It supports regular, rectilinear, cylindrical, spherical, and unstructured meshes. The necessary mesh names, origins, unstructured-mesh options, length multipliers, and IDs are transferred through APIs using ordinary C-compatible data types. The existingopenmc_add_unstructured_mesh()API is extended to accept all properties needed to reproduce a Python unstructured mesh directly in the library.This design retains the primary benefit of PRs #3942 and #3951: multi-GB weight window data no longer passes through XML. Unlike those approaches, it preserves a single implementation of the weight window HDF5 format. Changes to the format only need to be made in the existing C++ writer, and Python does not need to reproduce C++ serialization behavior with h5py.
Checklist