Open
Conversation
ax3l
changed the title
ax3l
force-pushed
the
topic-filter-particles
branch
from
81e7482 to
8af4bbe
Compare
Notebook that filters & copies particles and their attributes in chunk-wise (fixed slice) manner.
ax3l
force-pushed
the
topic-filter-particles
branch
from
8af4bbe to
930dc01
Compare
... and do only one time step.
ax3l
force-pushed
the
topic-filter-particles
branch
from
36d52ee to
c68fd99
Compare
for more information, see https://pre-commit.ci
Contributor
The Notebook does reimplement a lot of the functionality that openPMD-pipe has already, yes. |
Member
Author
|
I realized the attribute copying changes a few attributes, some of which are then unexpected types and need to be fixed/preserved in type: |
Member
Author
|
Added a work-around to conserve particle attribute dtypes more carefully. |
ax3l
commented
May 8, 2023
| dataset.options = json.dumps(dataset_config_resizable) | ||
| out_p_rc.reset_dataset(dataset) | ||
| out_p_rc[out_slice] = data[accepted] | ||
| output_series.flush() |
Member
Author
There was a problem hiding this comment.
We could avoid OOM for the output of the while step by using:
- BP5 and adding
- flush_target="disk"
Contributor
There was a problem hiding this comment.
Looking into the documentation on this, I notice a documentation bug.
https://openpmd-api.readthedocs.io/en/0.15.1/backends/adios2.html mentions adios2.preferred_flush_target = "disk" while https://openpmd-api.readthedocs.io/en/0.15.1/details/backendconfig.html talks of adios2.engine.preferred_flush_target = "disk".
The correct one is with engine, so: series.flush("""adios2.engine.preferred_flush_target = "disk" """)
ax3l
commented
May 8, 2023
| } | ||
| } | ||
| } | ||
| dataset_config['adios2']['dataset'] = { |
Member
Author
There was a problem hiding this comment.
This should work now with ADIOS 2.9.0+
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Notebook that filters & copies particles and their attributes in chunk-wise (fixed slice) manner. This is a prototype of functionality.
The filter can be a criteria on any point-wise combination of records of the current slice/chunk.
This is a common data map-reduce operation and we need to provide a API/tool for this functionality to filter very large data sets for interesting particles. Example: laser-plasma beam from a laser-plasma source, filtered by momentum and position.
We could maybe implement this in
openpmd-pipe.Alternatively, we could parallelize this as a tool and simplify it. Our pandas converter could simplify filters and chunking for the actual data. We could implement a lazily evaluated
from_dffor writing.Or we could extent our Dask reader and even could think about parallelism.