replace openmatrix with h5py - #87
Open
jpn-- wants to merge 5 commits into
Open
Conversation
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.
There was a problem hiding this comment.
Pull request overview
Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.
Changes:
- Refactors OMX reading, writing, splitting, and Zarr conversion.
- Adds parallel loading and configurable shared-memory array ordering.
- Removes OpenMatrix dependencies and updates tests and documentation.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Removes OpenMatrix and PyTables dependencies. |
sharrow/translate.py |
Uses the shared OMX reader for Zarr conversion. |
sharrow/tests/test_omx.py |
Adds OMX splitting and conversion tests. |
sharrow/tests/test_datasets.py |
Migrates tests to h5py and covers loading modes. |
sharrow/shared_memory.py |
Adds array ordering and memmap cleanup. |
sharrow/omx.py |
Reimplements OMX splitting with h5py. |
sharrow/omx_reader.py |
Narrows filename handling to h5py inputs. |
sharrow/example_data.py |
Reads example OMX data with h5py. |
sharrow/dataset.py |
Adds h5py-based loading and parallel storage modes. |
pyproject.toml |
Removes the OpenMatrix development dependency. |
envs/testing.yml |
Removes OpenMatrix from testing dependencies. |
envs/development.yml |
Removes OpenMatrix from development dependencies. |
docs/walkthrough/loading-skims.ipynb |
Updates the OMX walkthrough to h5py. |
.github/workflows/run-tests.yml |
Removes OpenMatrix from CI installation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if load == "shared": | ||
| result.shm.release_shared_memory() | ||
| else: | ||
| result.shm.delete_shared_memory_files(key) |
Comment on lines
+60
to
+67
| output_paths = [] | ||
| for number, matrix_name in enumerate(matrix_names): | ||
| output_path = destination / chunk_names[number % len(chunk_names)] | ||
| output_paths.append(output_path) | ||
| logger.info(f"writing {matrix_name} to {output_path}") | ||
| with h5py.File(output_path, "a") as target: | ||
| _initialize_omx_file(source, target) | ||
| _copy_dataset(source["data"], target["data"], matrix_name) |
| " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n", | ||
| " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])" | ||
| " omx_file.create_dataset(\"lookup/taz\", data=zones)\n", | ||
| " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request removes the use of the
openmatrixpackage throughout the codebase and replaces it with direct usage of theh5pylibrary for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to useh5pyAPIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.Dependency and Documentation Updates:
openmatrixfrom all dependency files (envs/development.yml,envs/testing.yml,pyproject.toml,.github/workflows/run-tests.yml) and updated code, documentation, and examples to useh5pydirectly for OMX file operations. [1] [2] [3] [4] [5] [6] [7] [8] [9]Code Refactoring:
openmatrixto useh5py, including insharrow/example_data.py,sharrow/omx.py, andsharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1] [2] [3] [4] [5]OMX File Handling Improvements:
split_omxfunction insharrow/omx.pyto useh5pyfor splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.Shared Memory Enhancements:
sharrow/shared_memory.py. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]API and Type Annotations:
openmatrixand clarify the use ofh5pyobjects, especially insharrow/omx_reader.py. [1] [2]