Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 5, 2025

Change Description

  • My PR includes a link to the issue that I am addressing

Closes #9

Implements time-series heatmap visualization showing when I/O operations of different sizes occur during execution. Requested feature to complement existing --histogram flag.

Solution Description

Timestamp Capture

  • Modified strace command to include -ttt flag (Unix epoch timestamps with microsecond precision)
  • Enhanced parse_strace_line() and parse_fs_usage_line() to extract and store timestamps
  • Timestamp detection handles both Unix epoch (strace) and HH:MM:SS (fs_usage) formats

Heatmap Visualization

Added generate_heatmap() function producing two heatmaps:

  • X-axis: Runtime with automatic time unit selection (s, ms, μs, ns) and limited decimal places for readability
  • Y-axis: Operation size (bytes, log scale)
  • Color: Operation count or total bytes transferred

The time axis automatically selects the most appropriate unit based on the time scale, making it easy to understand temporal patterns at any magnitude.

Supports inline display (notebooks) and file export (terminal).

Heatmap Example

CLI Integration

%%iops --heatmap
import tempfile, os, time

test_dir = tempfile.mkdtemp()
try:
    # Small operations
    for i in range(5):
        with open(os.path.join(test_dir, f'small_{i}.txt'), 'w') as f:
            f.write('x' * 1024)
        time.sleep(0.05)
    
    # Large operations
    for i in range(5):
        with open(os.path.join(test_dir, f'large_{i}.txt'), 'w') as f:
            f.write('z' * (100 * 1024))
        time.sleep(0.05)
finally:
    import shutil
    shutil.rmtree(test_dir)

Documentation

  • Merged with main branch to integrate new documentation structure
  • Added heatmap_visualization.ipynb following the established documentation pattern
  • Integrated into Read The Docs documentation structure in docs/notebooks.rst
  • Updated README with usage examples

Code Quality

  • I have read the Contribution Guide
  • My code follows the code style of this project
  • My code builds (or compiles) cleanly without any errors or warnings
  • My code contains relevant comments and necessary documentation

Project-Specific Pull Request Checklists

New Feature Checklist

  • I have added or updated the docstrings associated with my feature using the NumPy docstring format
  • I have updated the tutorial to highlight my new feature (if appropriate)
  • I have added unit/End-to-End (E2E) test cases to cover my new feature
  • My change includes a breaking change
    • My change includes backwards compatibility and deprecation warnings (if possible)
Original prompt

This section details on the original issue you should resolve

<issue_title>Output a spectrogram like time-series view</issue_title>
<issue_description>Similar to —histogram, make a command line flag which outputs byte and operations focused heatmaps.

The x axis of the heatmap should be runtime
The y axis of the heatmap should be a log scale of the size of the operation
The color is either the number of operations or the number of bytes.</issue_description>

Comments on the Issue (you are @copilot in this section)

@mtauraso Once implemented add an example notebook to the documentation showcasing the use of this, with a simple sample i/o workload that has io of different sizes across time.

The sample notebook should be run during docs generation such that the readthedocs docs have a visual reference on the spectrogram-like view of the captured i/o data.</comment_new>


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

@codecov
Copy link

codecov bot commented Nov 5, 2025

Codecov Report

❌ Patch coverage is 81.16883% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.28%. Comparing base (cbe715b) to head (2aa6284).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
src/iops_profiler/magic.py 4.76% 20 Missing ⚠️
src/iops_profiler/display.py 90.62% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
+ Coverage   45.49%   54.28%   +8.79%     
==========================================
  Files           4        4              
  Lines         455      595     +140     
==========================================
+ Hits          207      323     +116     
- Misses        248      272      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI and others added 3 commits November 5, 2025 23:31
- Enhanced collector to capture timestamps from strace and fs_usage
- Added --spectrogram flag to magic.py for time-series visualization
- Created generate_spectrogram function in display.py with heatmap plots
- Added comprehensive tests for timestamp parsing and spectrogram generation
- All existing tests pass with new timestamp fields

Co-authored-by: mtauraso <31012+mtauraso@users.noreply.github.com>
- Created spectrogram_example.ipynb showcasing time-series visualization
- Configured docs to execute notebooks during build with nbsphinx
- Updated README with spectrogram usage examples and documentation
- Added nbsphinx and ipykernel to docs requirements

Co-authored-by: mtauraso <31012+mtauraso@users.noreply.github.com>
- Clarified strace timestamp comment (Unix epoch with microsecond precision)
- Added named constants for spectrogram magic numbers
- Compiled regex pattern for timestamp detection to improve performance
- Reduced test operation count from 1000 to 200 for faster tests
- Fixed line length issues

Co-authored-by: mtauraso <31012+mtauraso@users.noreply.github.com>
Copilot AI changed the title [WIP] Add spectrogram-like output for time-series view Add --spectrogram flag for time-series heatmap visualization of I/O patterns Nov 5, 2025
Copilot AI requested a review from mtauraso November 5, 2025 23:42
@mtauraso
Copy link
Collaborator

mtauraso commented Nov 6, 2025

Notes for me:

  • Come up with a different name for the flag
  • Test on OSX to make sure fs_util still works properly
  • Consider the layout of the plot and x axis. Probably needs human-understandable time unit.
  • Look at the measurement code, some of the functions are getting long.

@mtauraso
Copy link
Collaborator

mtauraso commented Nov 7, 2025

@copilot I'd like you to:

  • 1st update your view of this PR to include things on main right now, resolving merge conflicts as appropriate.
  • Make the x axis of the plot in a human readable unit (s, ms, us, ns) with a limited number of decimal places, such that the axis is basically readable.
  • Come up with a better name for the flag. "spectrogram" while evocative of this view, has a technical meaning which doesn't align with what this visualization actually is.
  • Following the documentation already on main, make an example notebook and add readthedocs documentation for this feature.

Copilot AI and others added 2 commits November 7, 2025 21:25
- Renamed generate_spectrogram to generate_heatmap
- Renamed --spectrogram flag to --heatmap throughout codebase
- Added format_time_axis function to auto-select appropriate time unit (s, ms, μs, ns)
- Updated x-axis labels to show time in human-readable units with limited decimal places
- Renamed test file from test_spectrogram.py to test_heatmap.py
- Renamed notebook from spectrogram_example to heatmap_visualization
- Updated all documentation and constants to use 'heatmap' terminology
- Added heatmap notebook to docs/notebooks.rst with description
- All 136 tests pass

Co-authored-by: mtauraso <31012+mtauraso@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Nov 7, 2025

@copilot I'd like you to:

  • 1st update your view of this PR to include things on main right now, resolving merge conflicts as appropriate.
  • Make the x axis of the plot in a human readable unit (s, ...

I've completed all requested changes in commit 2aa6284:

1. Updated and merged with main - Resolved merge conflicts and integrated the new documentation structure from main branch.

2. Made x-axis human readable - Added format_time_axis() function that automatically selects appropriate time units:

  • Seconds (s) for times ≥ 1.0s
  • Milliseconds (ms) for times ≥ 1ms
  • Microseconds (μs) for times ≥ 1μs
  • Nanoseconds (ns) for smaller times

The axis labels now display in the selected unit with limited decimal places (e.g., "Time (ms)" with 1 decimal place).

3. Renamed the flag - Changed from --spectrogram to --heatmap:

  • More accurate description (it's a 2D heatmap, not a spectrogram)
  • Avoids technical term confusion
  • Updated throughout: function names, constants, tests, notebook, documentation

4. Added documentation - Created comprehensive documentation following the structure on main:

  • Added heatmap_visualization.ipynb notebook with examples
  • Integrated into docs/notebooks.rst with description section
  • Follows same style as histogram_visualization and basic_usage notebooks

Heatmap Example

The heatmap now shows time in appropriate units with readable formatting, making it easy to understand temporal I/O patterns at any time scale.

Copilot AI changed the title Add --spectrogram flag for time-series heatmap visualization of I/O patterns Add --heatmap flag for time-series heatmap visualization of I/O patterns Nov 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Output a spectrogram like time-series view

2 participants