New epochs class #14
New epochs class #14
Conversation
Refactored the Epochs class to use a dictionary of epochs indexed by epoch index, added properties for empty epochs, and improved overlap checking and warnings. Updated the to_numpy method to support flexible sampling rates and interpolation, and improved baseline correction error handling. Modified plot_epochs and its helpers to work with the new Epochs API, and updated the pupil_size_and_epoching tutorial to use the new sample data and API.
Expanded and clarified docstrings for epoching functions and the Epochs class, including detailed parameter and return value descriptions. Refactored Dataset to automatically handle native data without requiring a 'custom' flag, and improved section construction for native recordings. Updated tutorials to use consistent event naming and native data examples. Minor bugfixes and doc improvements in export and utility modules.
|
@JGHartel Currently the baseline correction method is not updated yet. What do you think would be the best API/code for it? For operability with other methods such as |
JGHartel
left a comment
JGHartel
left a comment
There was a problem hiding this comment.
Overall, I welcome the suggested changes. In particular, the rename towards epoch_info and the unified access to epochs data as epoch.epochs, rather than the previous mixed access, is appreciated. Still, I would like to invite some discussions on the assumed datatypes, e.g. epochs.epochs and epochs.annotate being a dictionary. This would especially futureproof the development, if dataset level operations should be introduced
| ``t_after``: Time after the reference time to end the epoch.\n | ||
| ``description``: Description or label associated with the epoch. | ||
| """ | ||
|
|
There was a problem hiding this comment.
I will defer to you on this, but I personally think it a bit over the top to use decorators for the docstring. Besides generating the API doc, sometimes one just wants to look on the functions/methods themselves. I would not want to have to dig through a dictionary of docstring defaults for that.
That being said, the current ones seem to be simple enough to be redundant in code
pyneon/epochs.py
Outdated
| # Create epochs | ||
| self.epochs, self.data = _create_epochs(source, times_df) | ||
| @cached_property | ||
| def epochs(self) -> dict[int, Stream | Events | None]: |
There was a problem hiding this comment.
flagging this in case we ever want to support multi-recording functionality. It would be good to have a Unique ID for each event, so that one can easily concat these dicts between recordings. Alternatively, we could then construct an implicit multiindex (recording, epoch_number), which would require a custom concat function.
There was a problem hiding this comment.
This seems a bit of an advanced user case and also in principle should apply to other classes (e.g. streams and events). My evaluation would be it would require another PR if we see value in supporting multi-recording concatenation
There was a problem hiding this comment.
Pull request overview
This PR introduces a major refactoring of the Epochs class to make it more intuitive and easier to use. The key changes include moving away from nested DataFrames to a dictionary-based structure, renaming methods for clarity, and introducing lazy computation of epochs using cached properties.
Changes:
- Refactored
Epochsclass to useepochs_dict(cached property returning a dictionary) instead of nested DataFrames - Renamed
event_timestoepochs_infoandevents_to_times_dftoevents_to_epochs_infofor better comprehension - Changed
Eventsclass to use event IDs as DataFrame index instead of separate columns - Added new
filter_by_namemethod to Events class and updatedfilter_by_durationbehavior - Added PLR sample dataset support with Figshare hosting
- Consolidated CI workflows (removed separate tests.yml, integrated into main.yml)
Reviewed changes
Copilot reviewed 19 out of 22 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| pyneon/epochs.py | Major refactoring of Epochs class with new dictionary-based structure, cached properties, and helper functions renamed |
| pyneon/events.py | Modified to use event IDs as index, added filter_by_name method, updated filter_by_duration |
| pyneon/stream.py | Added stub for detect_events_from_derivative method, updated docstrings with reusable doc snippets |
| pyneon/vis/vis.py | Updated plot_epochs to work with new epochs_dict structure |
| pyneon/dataset.py | Made sections.csv optional, improved error handling for missing recordings |
| pyneon/utils/doc_decorators.py | Added reusable documentation snippets for common return types and parameters |
| pyneon/utils/sample_data.py | Added PLR dataset URL and test |
| tests/conftest.py | Added simple_events fixture for testing Events functionality |
| tests/test_events.py | Added (empty) test stub for crop functionality |
| tests/test_streams.py | Added tests for interpolation and concatenation methods |
| .github/workflows/main.yml | Consolidated CI workflows, added test job dependencies |
| source/tutorials/*.ipynb | Updated tutorial notebooks to use new API names |
| README.md | Added information about sample datasets on Figshare |
Comments suppressed due to low confidence (1)
pyneon/epochs.py:223
- The properties
columnsanddtypes(lines 216-223) referenceself.datawhich no longer exists in the newEpochsclass structure. These properties need to be updated or removed.
@property
def columns(self) -> pd.Index:
return self.data.columns[:-3]
@property
def dtypes(self) -> pd.Series:
"""The data types of the epoched data."""
return self.data.dtypes[:-3]
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @fill_doc | ||
| def filter_by_name( | ||
| self, | ||
| names: str | list[str], | ||
| col_name: str = "name", | ||
| reset_id: bool = False, | ||
| inplace: bool = False, | ||
| ) -> Optional["Events"]: | ||
| """ | ||
| Filter events by matching values in a specified column. | ||
| This method selects only the events whose value in ``col_name`` matches | ||
| one or more of the provided ``names``. If no events match, a | ||
| ``ValueError`` is raised. | ||
| Parameters | ||
| ---------- | ||
| names : str or list of str | ||
| Event name or list of event names to keep. Matching is exact | ||
| and case-sensitive. | ||
| col_name : str, optional | ||
| Name of the column in ``self.data`` to use for filtering. | ||
| Must exist in the ``Events`` instance's DataFrame. | ||
| Defaults to ``"name"``. | ||
| reset_id: bool = False, optional | ||
| Whether to reset event IDs after filtering. | ||
| Defaults to ``False``. | ||
| %(inplace)s | ||
| Returns | ||
| ------- | ||
| %(events_or_none)s | ||
| """ | ||
| if col_name not in self.data.columns: | ||
| raise KeyError(f"No `{col_name}` column found in the instance.") | ||
|
|
||
| names = [names] if isinstance(names, str) else names | ||
| mask = self.data[col_name].isin(names) | ||
| if not mask.any(): | ||
| raise ValueError( | ||
| f"No data found matching the specified event names {names}" | ||
| ) | ||
|
|
||
| inst = self if inplace else self.copy() | ||
| inst.data = self.data[mask].copy() | ||
| if reset_id: | ||
| inst.data.index = pd.RangeIndex( | ||
| start=0, stop=len(inst.data), name=inst.data.index.name | ||
| ) | ||
| return None if inplace else inst |
There was a problem hiding this comment.
The new method filter_by_name in the Events class lacks test coverage. Since other similar files in the tests directory have test coverage (e.g., test_streams.py tests interpolation and concatenation), this new filtering functionality should also be tested.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Developers and users have found the old
Epochsclass quite unintuitive and operationally hard to use (e.g., nested DataFrame). This PR aims to reorganize the class.Current features:
event_timesis renamed toepochs_infofor easier comprehensionEpochsinstance will not compute the epochs or annotated data automatically.epochsis now a cached property in the form of a dictionary. Keys are indices ofepochs_infoand values areStream/Eventsinstances.annotate_epochs.