Performance improvement: Avoid unnecessary GPU detail queries in _monitor_power()#1264
Open
vishali-mp wants to merge 3 commits into
Open
Performance improvement: Avoid unnecessary GPU detail queries in _monitor_power()#1264vishali-mp wants to merge 3 commits into
vishali-mp wants to merge 3 commits into
Conversation
_get_gpu_details makes 8 NVML calls per GPU (memory, temperature, power, energy, utilization, compute mode, compute processes, graphics processes) but _monitor_power only consumes gpu_index and gpu_utilization. Add get_gpu_utilization_lightweight() which makes 1 NVML call per GPU (nvmlDeviceGetUtilizationRates), and switch _monitor_power to use it. Reduces NVML calls per monitoring cycle from 8 to 1 per GPU — on an 8-GPU system that's 56 fewer NVML calls per second (64 → 8). get_gpu_details() is preserved unchanged for measure_power_and_energy where power_usage is consumed.
Covers all guard conditions in the monitoring loop: - gpu_index is None → entry skipped - gpu_index not in monitored set → entry skipped - gpu_utilization key missing → entry skipped - empty list from get_gpu_utilization_list() → nothing collected
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
_monitor_power()hot path invokesget_gpu_details()every second, but onlygpu_indexandgpu_utilizationare consumed by the caller.get_gpu_details()retrieves memory info, temperature, compute processes, graphics processes, compute mode, and other metadata — all discarded in the monitoring loop. On multi-GPU systems this triggers dozens of unnecessary NVML calls per second, with process enumeration APIs being the most expensive.Changes:
codecarbon/core/gpu_device.py: Newget_gpu_utilization_lightweight()that returns onlygpu_indexandgpu_utilization, skipping memory, temperature, process lists, compute mode, and other unused attributes.codecarbon/core/gpu.py: Newget_gpu_utilization_list()that iterates devices using the lightweight method.codecarbon/emissions_tracker.py: Updated_monitor_power()to callget_gpu_utilization_list()instead ofget_gpu_details().tests/test_gpu_nvidia.py: Added tests forget_gpu_utilization_lightweight(),get_gpu_utilization_list(), and exception handling (returns[]on NVML error).tests/test_emissions_tracker.py: Added 4 edge case tests covering all guard conditions in_monitor_power():gpu_indexisNone, GPU not in monitored IDs, missinggpu_utilizationkey, and empty result list.scripts/benchmark_gpu_monitoring.py: Benchmark harness to measure performance delta between old and new paths.Related Issue
Closes #1237
Motivation and Context
_monitor_power()runs every second and only needs GPU utilization to compute a rolling average. The fullget_gpu_details()call retrieves memory (nvmlDeviceGetMemoryInfo), temperature (nvmlDeviceGetTemperature), compute processes (nvmlDeviceGetComputeRunningProcesses), graphics processes (nvmlDeviceGetGraphicsRunningProcesses), and compute mode — none of which are used in this hot path. The process enumeration APIs are especially expensive as they iterate active GPU processes and collect PID-level information.The lightweight path reduces NVML calls per GPU per second from ~7 to just 1 (only
nvmlDeviceGetUtilizationRates), preserving the existingget_gpu_details()for code paths that need full metadata (e.g.__repr__, static info queries).How Has This Been Tested?
tests/test_gpu_nvidia.py:test_gpu_utilization_list,test_gpu_utilization_lightweight,test_gpu_utilization_list_empty_on_exception.tests/test_emissions_tracker.py:test_monitor_power_collects_gpu_utilization_lightweight(happy path),test_monitor_power_skips_gpu_when_index_is_none,test_monitor_power_skips_gpu_not_in_monitored_ids,test_monitor_power_skips_gpu_when_utilization_key_missing,test_monitor_power_handles_empty_gpu_utilization_list.scripts/benchmark_gpu_monitoring.py: compares calls/sec of lightweight vs full path.Types of changes
AI Usage Disclosure
Checklist