Improve Monthly/Yearly Log - #3829
Draft
AlexanderHa98 wants to merge 9 commits into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the measurement logging pipeline to improve monthly/yearly log generation by deriving aggregates from daily logs and persisting computed daily/monthly “totals” artifacts.
Changes:
- Simplifies
write_log.save_log()to only write daily logs (removesLogTypeand monthly-log creation). - Reworks
process_log.get_monthly_log()/get_yearly_log()to build aggregates from stored daily/monthly totals, with fallback generation when totals files are missing. - Adds midnight tasks in
main.pyto persist daily totals (and monthly totals on month rollover).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| packages/main.py | Switches 5-min logging to daily-only and adds midnight persistence of daily/monthly totals. |
| packages/helpermodules/measurement_logging/write_log.py | Removes LogType and makes save_log() daily-only. |
| packages/helpermodules/measurement_logging/process_log.py | Replaces monthly/yearly aggregation logic and adds persistence/loading of daily/monthly source totals. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+823
to
+831
| daily_log_files = list(daily_log_dir.glob("*.json")) | ||
| if not daily_log_files: | ||
| return None | ||
|
|
||
| # st_ctime = creation time of the file | ||
| # st_mtime = last modification time of the file | ||
| oldest_file = min(daily_log_files, key=lambda f: f.stat().st_ctime) | ||
| oldest_date = oldest_file.stem # Get the filename without extension | ||
| return oldest_date |
Comment on lines
+234
to
+241
| today = timecheck.create_timestamp_YYYYMMDD() | ||
| previous_day = timecheck.get_relative_date_string(today, day_offset=-1) | ||
| save_daily_source_totals(previous_day) | ||
|
|
||
| prev_month = timecheck.get_relative_date_string(today, month_offset=-1)[:6] | ||
| # Neuer Monat hat angefangen, daher Monats Totals speichern | ||
| if today[6:8] == "01": | ||
| save_monthly_source_totals(prev_month ,None, saveing=True) |
Comment on lines
+685
to
+686
| def save_daily_source_totals(date: str, saveing: bool = True): | ||
| try: |
Comment on lines
+686
to
+704
| try: | ||
| data = _collect_daily_log_data(date) | ||
| processed_entries = _process_entries(data.get("entries", []), calculation=CalculationType.ENERGY) | ||
| totals = get_totals(processed_entries, process_entries=False) | ||
| analysed_data = _analyse_energy_source({ | ||
| "entries": processed_entries, | ||
| "totals": totals, | ||
| "names": data.get("names", {}) | ||
| }) | ||
| totals = analysed_data["totals"] | ||
|
|
||
| source_entries = data.get("entries", []) | ||
| daily_entry = {} | ||
| if len(source_entries) > 0: | ||
| # Nur den letzten Eintrag des Tages nehmen | ||
| daily_entry = deepcopy(source_entries[-1]) | ||
| daily_entry["date"] = date | ||
| _apply_source_totals(daily_entry, totals) | ||
|
|
| log.exception(f"Fehler beim Laden der Tages-Summen für {date}") | ||
|
|
||
|
|
||
| def save_monthly_source_totals(date: str, data: Dict, saveing: bool = True): |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
Suppressed comments (9)
packages/helpermodules/measurement_logging/process_log.py:827
- This skips totals for modules that disappeared before the final snapshot. The daily/monthly source totals can still contain such a module because it was present earlier in the period, but the last raw entry does not; dropping it here makes
get_monthly_log/get_yearly_logomit its accumulated energy even though the persisted totals contain it. Initialize a missing module entry and merge its totals instead of continuing.
module_data = section_data.get(module)
if not isinstance(module_data, dict) or not isinstance(module_totals, dict):
continue
packages/helpermodules/measurement_logging/process_log.py:315
- The entries here are copied from raw daily snapshots;
_apply_source_totalsonly adds module-level energy fields. Unlike the previous implementation, this path never runs_analyse_energy_source, so monthly entries have noenergy_sourceand the response has no top-levelmessage, changing the established monthly-log response shape. Run the normal analysis ondataafter calculating the totals.
data["totals"] = analyse_percentage_totals(data["entries"], data["totals"])
packages/helpermodules/measurement_logging/process_log.py:332
- The comment misspells
unnötigeasunötige.
# Sonst werden unötige totals Werte gespeichert
packages/helpermodules/measurement_logging/update_yields.py:125
monthly_totalsis a new storage format, so existing installations will have historicalmonthly_logfiles but no files in this directory. This loop only sums already-createdmonthly_totals, causingyearly_exportedto reset to the current month's production after upgrade and never recover prior months unless another path happens to rebuild them. Add a migration/fallback for missing months.
content = load_monthly_source_totals_content(month)
if content is not None:
totals = content.get("totals", {})
packages/helpermodules/measurement_logging/process_log.py:379
- The yearly path has the same response-shape regression: its entries are assembled without
energy_source, and no top-levelmessageis produced because only percentage totals are calculated. Apply_analyse_energy_sourceto the assembled yearly data before returning it, as the old yearly implementation did.
data["totals"] = analyse_percentage_totals(data["entries"], data["totals"])
packages/helpermodules/measurement_logging/process_log.py:283
- Unlike the yearly loop, this monthly loop has no future-date bound. A request for any month after the current month therefore walks every day, calls
save_daily_source_totalswithsaving=True, and creates a file of empty daily-total records even though no daily log exists. Bound the loop to the current month before attempting the fallback save.
This issue also appears in the following locations of the same file:
- line 315
- line 825
while day.startswith(date):
packages/helpermodules/measurement_logging/process_log.py:269
- The comment misspells
unnötigeasunötige.
This issue also appears on line 332 of the same file.
# Sonst werden unötige totals Werte gespeichert
packages/helpermodules/measurement_logging/update_yields.py:94
- On an upgrade, existing
daily_logfiles do not have correspondingdaily_totalsfiles. When that happens this branch leaves the day out instead of deriving/migrating its totals, so the published current-month PV yield silently loses all pre-upgrade days (and any day whose midnight save failed). Please calculate and persist missing daily totals rather than treating them as zero.
This issue also appears on line 123 of the same file.
content = load_daily_source_totals_content(day)
if content is not None:
totals = content.get("totals", {})
packages/helpermodules/measurement_logging/update_yields.py:116
- The comment misspells
MonatsasMontas.
# Wenn es noch keinen Montas Totals gibt
| update_pv_monthly_yearly_yields() | ||
| entries = save_log() | ||
| daily_totals = update_daily_yields(entries) | ||
| update_pv_monthly_yearly_yields(daily_totals) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.