[weather] refactor: migrate to server-side providers with centralized HTTPFetcher#4032
Open
KristjanESPERANTO wants to merge 100 commits intoMagicMirrorOrg:developfrom
Open
[weather] refactor: migrate to server-side providers with centralized HTTPFetcher#4032KristjanESPERANTO wants to merge 100 commits intoMagicMirrorOrg:developfrom
KristjanESPERANTO wants to merge 100 commits intoMagicMirrorOrg:developfrom
Conversation
…PFetcher Migrate the OpenMeteo weather provider from client-side (performWebRequest + CORS proxy) to server-side architecture using HTTPFetcher for consistency with Calendar and Newsfeed modules. Changes: - Add node_helper.js for server-side weather provider management - Add openmeteo_server.js using HTTPFetcher with periodic auto-fetch - Modify weather.js with hybrid client/server-side provider support - Remove client-side openmeteo.js (now obsolete) Architecture: - weather.js sends INIT_WEATHER → node_helper loads provider - Provider uses HTTPFetcher for automatic polling (reloadInterval) - Data flows via callbacks → socket notifications → weather.js Benefits: - No CORS proxy needed - API keys stay server-side - Unified architecture with Calendar/Newsfeed - HTTPFetcher retry strategies (429/5xx) built-in
Add server-side OpenWeatherMap provider using HTTPFetcher, following the same pattern as OpenMeteo migration. Changes: - Add openweathermap_server.js with OnceCall API 3.0 support - Update weather.js to recognize openweathermap as server-side provider - Remove client-side openweathermap.js (now obsolete) Features: - Supports current, forecast/daily, and hourly weather types - Timezone offset handling for accurate timestamps - UV index, precipitation, and feels-like temperature - HTTPFetcher automatic polling and retry strategies
Add server-side WeatherGov (US National Weather Service) provider using HTTPFetcher. Changes: - Add weathergov_server.js with 2-step initialization (grid-point → station) - Update weather.js to recognize weathergov as server-side provider - Remove client-side weathergov.js (now obsolete) Implementation notes: - SunCalc integration for sunrise/sunset (API doesn't provide this) - User-Agent header required by API - US locations only, no API key required
Add server-side Yr.no (Norwegian Meteorological Institute) provider with proper HTTP caching support. Changes: - Add yr_server.js using HTTPFetcher for periodic weather fetching - Implement If-Modified-Since header support per API recommendations - Handle 304 Not Modified responses to reduce API calls - Cache Last-Modified and Expires headers for efficient updates - Fetch stellar data (sunrise/sunset) separately with daily caching - Update weather.js to recognize yr as server-side provider - Remove client-side yr.js (now obsolete) Implementation notes: - Enforce 10-minute minimum update interval per API terms - Coordinate precision limited to 4 decimals per API requirements - Weather data cached in-memory with Last-Modified tracking - Stellar data refetched daily or when using cached weather data - Sunrise API v3.0 with timezone offset parameter
Add server-side SMHI (Swedish Meteorological and Hydrological Institute) provider with precipitation category handling. Changes: - Add smhi_server.js using HTTPFetcher for periodic weather fetching - Implement gap filling for hourly data interpolation - Calculate apparent temperature using heat index formula - Handle precipitation categories (snow, rain, mixed, drizzle, freezing) - Support configurable precipitation values (pmin, pmean, pmedian, pmax) - Update weather.js to recognize smhi as server-side provider - Remove client-side smhi.js (now obsolete) Implementation notes: - Sweden only, metric system required - Coordinate precision limited to 6 decimals per API requirements - Data gaps filled by duplicating previous hour's data - Weather type selected from median of daytime hours for forecasts - Uses SunCalc for sunrise/sunset times
- Updated to new Environment Canada MSC Datamart API structure
* Changed base URL from /citypage_weather/ to /today/citypage_weather/
* Updated filename pattern to timestamped format: {timestamp}_MSC_CitypageWeather_{siteCode}_en.xml
- Implemented regex-based XML parsing (no external dependencies)
- Two-step data fetch: index page → city XML file
- Supports current conditions, forecast (12 periods), and hourly (24 hours)
- Features:
* Wind chill and humidex for feels-like temperature
* Today/Tonight forecast logic
* Sunrise/sunset times from XML
* Weather alerts/warnings support
- All three types tested and working (Toronto)
- Migrated from WeatherProvider.register() to HTTPFetcher-based provider - Supports current conditions, forecast (daily), and hourly forecasts - Features: * API key validation with helpful error messages * Precipitation type detection (rain/snow) * Apparent temperature (feels-like) support * Sunrise/sunset from daily data - Tested and working with valid API key
- Migrated from WeatherProvider.register() to HTTPFetcher-based provider - Supports current conditions, forecast (daily), and hourly (3-hourly) forecasts - Features: * API key validation with helpful error messages * SunCalc integration for sunrise/sunset times * All 31 Met Office significant weather codes mapped * Handles different field names for hourly vs 3-hourly data * Temperature averaging for 3-hourly data (max/min avg) * Precipitation probabilities for rain, snow, hail * Feels-like temperature support - Tested and working with valid API key (London)
- Migrated from WeatherProvider.register() to HTTPFetcher-based provider - Supports current conditions, forecast (daily), and hourly forecasts - Deleted client-side weatherbit.js
- Delete client-side weatherflow.js - Add weatherflow_server.js (303 lines) * HTTPFetcher with logContext for clear error identification * API key and station ID validation * Support for current/forecast/hourly types * 18 weather icon mappings (Dark Sky compatible icons) * UV index aggregation from hourly data * Wind speed conversion from kph to m/s * Precipitation aggregation from hourly data - Update weather.js serverSideProviders array Note: WeatherFlow requires personal weather station hardware (Tempest)
- Delete overrideWrapper.js (no longer needed) - Move CURRENT_WEATHER_OVERRIDE logic directly into weather.js * Override functionality now works with server-side providers * Merges override data with currentWeatherObject * Triggers DOM update after override - Remove client-side provider script loading from getScripts() * All providers now run server-side * Only load rendering dependencies (moment, weatherutils, etc.) Tested: Override changes temperature and humidity in real-time No user-facing changes: allowOverrideNotification config works as before
Provider Files: - Rename all *_server.js to *.js (migration complete) * envcanada_server.js → envcanada.js * openmeteo_server.js → openmeteo.js * openweathermap_server.js → openweathermap.js * pirateweather_server.js → pirateweather.js * smhi_server.js → smhi.js * ukmetofficedatahub_server.js → ukmetofficedatahub.js * weatherbit_server.js → weatherbit.js * weatherflow_server.js → weatherflow.js * weathergov_server.js → weathergov.js * yr_server.js → yr.js - Update node_helper.js to load providers without _server suffix - Delete weatherprovider.js (client-side loader no longer needed) Code Simplification (weather.js): - Remove all usesServerSideProvider() conditionals - Remove weatherProvider property (unused) - Remove scheduleUpdate() method (server-driven fetching only) - Simplify getTemplateData() - use server-side properties directly - Simplify getHeader() - use fetchedLocationName directly - Simplify updateAvailable() - no client-side scheduling - Simplify override notification handler - Remove getForecastArray/getHourlyArray conditionals CORS Proxy Removal: - Remove performWebRequest and all CORS functions from utils.js - Remove /cors endpoint from server.js - Remove cors() function from server_functions.js - Update utils_spec.js to only test formatTime All weather providers now run exclusively server-side via node_helper. No more client-side provider code, no more CORS proxy needed. Weather module is now 100% server-side architecture.
Environment Canada changed their datamart file naming format on
2026-01-31 from simple suffix pattern to timestamped format:
- Old: *_MSC_CitypageWeather_{siteCode}_en.xml
- New: {timestamp}_MSC_CitypageWeather_{siteCode}_en.xml
(e.g., 20260131T120112.758Z_MSC_CitypageWeather_s0000458_en.xml)
Changes:
- Updated regex pattern in #extractCityPageURL() to match new format
- Added hour-change detection to reinitialize fetcher on UTC hour change
- Prevents 404 errors when hourly directories are cleaned up
The provider now dynamically updates its URL when the hour changes,
ensuring continuous operation across hour and day boundaries.
Tested: current, forecast, and hourly types all working
Add 17 smoke tests for OpenMeteo and OpenWeatherMap providers covering constructor, configuration, callbacks, API key validation, and public methods. Tests validate the public API surface to prevent breaking changes. Parser tests not implemented due to Vitest ESM mocking complexity with private methods and module aliases.
…r provider lifecycle
These providers had timezone bugs both before and after server-side migration - only working correctly when browser/server timezone matched weather location timezone. OpenMeteo: - Removed #checkDST() and #isDST() server timezone-based logic - API with timezone: "auto" already returns location-local timestamps Weatherbit: - Removed getTimezoneOffset() server timezone offset from sunrise/sunset - API returns "HH:mm" already in location time Fixes weather times being displayed in browser/server timezone instead of actual location timezone.
Replace mockData injection in provider with direct socket injection following the weather module's migration to server-side architecture. The previous mockData approach required test-specific code in the production provider, which is no longer possible with the server-side implementation. The new approach tests only client-side rendering by injecting data directly via socketNotificationReceived. Changes: - Remove mockData config from all weather test configs - Add new OneCall format mock JSON files (current, forecast, hourly) - Rewrite weather-functions.js to inject data via socketNotificationReceived - Update test specs to pass mock data filename as parameter - Apply timezone offset correctly matching provider's #applyOffset method Benefits: - Zero test-specific code in production provider - E2E tests validate client rendering only - Explicit mock data control via JSON files - Clean separation between test and production code
- Updated weather-setup.js to use direct socket injection instead of mockData replacement in config files - Removed cleanupMockData call from weather_spec.js - Fixed timezone_offset in mock data (set to 0 for UTC-based timestamps) - Deleted obsolete weather_mocker.js utility This completes the weather module client-to-server migration for tests.
…weather providers
…wait times in E2E tests
- Created defaultmodules/weather/utils.js with convertWeatherType and applyTimezoneOffset - Updated openweathermap.js provider to use shared utilities directly - Updated test helpers (weather-setup.js, weather-functions.js) to use shared utilities - Eliminates code duplication across providers and tests - Removed unnecessary wrapper methods for cleaner code
- Change 'Could not find city page URL' from warn to debug level - This happens normally during hour transitions when old responses arrive - After reinitialization, old fetcher may still send cached responses - Add explanatory comment about stale responses from previous hour
parseFloat(null) returns NaN, causing invalid data in weather objects. Added null checks before parseFloat() calls in forecast and hourly generation methods. - Missing API fields (e.g., rain_sum) now return null instead of NaN - Affects: rain, snow, precipitation, humidity, uvIndex properties - Updated test assertions to expect null for missing data Fixes #generateWeatherObjectsFromForecast and #generateWeatherObjectsFromHourly
Icon code 40 represents "Blowing Snow" in Environment Canada's API but was not mapped, causing weatherType to return null. Added mapping: 40 -> "snow" Updated test assertion to expect "snow" instead of null for icon code 40.
All 10 weather providers now use the same import pattern.
- OpenMeteo: Fix hourly logic that skipped first valid future hour - OpenMeteo: Add bounds check to prevent array out-of-bounds crash - Pirateweather: Fix null-safe checks to preserve 0 values (humidity, feelsLike, windSpeed, tempMin/Max, precipProb)
- http_fetcher: Wrap URL parsing in try-catch to prevent crash during error logging
- E2E weather-functions: Fix precipitationProbability to use null-safe check - Electron weather-setup: Fix precipitationProbability to use null-safe check - Ensures 0% precipitation is preserved instead of becoming undefined
- openmeteo_spec: Remove debug console.log statements - envcanada_spec: Fix comment (windChill is -31, not -12) - envcanada_spec: Fix inverted callbacks in error handling test - smhi_spec: Remove unused SMHI_URL constant
- Split precipitationProbability/uvIndex assignments - Split precip initialization and if-statement - Improves readability and follows coding standards
- Fixes inconsistent field names across providers - Templates expect windFromDirection (current.njk uses it) - Affected providers: Pirateweather, UkMetOffice, Weatherbit, WeatherFlow - Ensures wind direction displays correctly in UI
- provider-utils.js: Remove unused Log import - openmeteo_spec.js: Remove unused beforeEach import and OPEN_METEO_URL constant
- Remove unused errorPromise variable - Simplify callback setup without unused resolve/reject - Makes test intent clearer
- Adds back getVersion() function and /version route - Was incorrectly removed - /version endpoint is unrelated to weather module - Restores public API endpoint
- Increase timeout from 5s to 10s for geocoding API - Change error log level from ERROR to DEBUG - Geocoding is optional (only for location name display)
Restore original day/night-specific icons and missing codes 41-48 (tornado, windy, smoke, sandstorm, thunderstorm variants) that were lost during migration.
Rename windDirection → windFromDirection in generateHourly() and precipitation → precipitationAmount in generateForecast() and generateHourly() to match WeatherObject properties.
Compare year, month, and day when matching hourly data to daily forecasts to prevent incorrect matches across month boundaries (e.g., Jan 31 vs Feb 31).
Use waitForSelector and waitForFunction instead of fixed 1000ms/500ms delays in weather-setup.js for more robust and faster test execution.
Move describe() and it() opening to separate lines for better readability.
Ensure current.temperature is always set (either to a value, cached value, or null) to prevent undefined temperature in weather display.
Environment Canada's XML feed now returns <currentConditions/> as an empty element. Adapt by extracting current weather from the first forecast period instead. Additional fixes: - Add missing return when city page URL not found - Restore sunrise/sunset data (from riseSet element) - Accept both "high" and "low" temperature classes
…urrentConditions Wind speed, bearing, temperature, and humidity now correctly read from currentConditions element instead of forecast, with fallback logic.
More robust property existence check in PirateWeather provider instead of value-based undefined check.
khassel
approved these changes
Feb 8, 2026
Collaborator
|
I'm fine with this but I think @rejas should review this because "weather" was always his baby ;) 2 things:
|
Collaborator
Author
Oh, good point. It's currently used by 3 modules: MMM-WeatherAlerts, MMM-Metar, MMM-HK-Transport-ETA Should I restore it (with deprecation warning) or remove it and create PRs for those 3 modules? The migration is simple - they can use
Yes, should still work. The notification system is unchanged - 3rd-party modules can continue using it. |
Collaborator
I vote for removal ... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This migrates the Weather module from client-side fetching to use the server-side centralized HTTPFetcher (introduced in #4016), following the same pattern as the Calendar and Newsfeed modules.
Motivation
This brings consistent error handling and better maintainability and completes the refactoring effort to centralize HTTP error handling across all default modules.
Migrating to server-side providers with HTTPFetcher brings:
Changes
Breaking Changes
None. Existing configurations continue to work.
Testing
To ensure proper functionality, I obtained API keys and credentials for all providers that require them. I configured all 10 providers in a carousel setup and tested each one individually. Screenshots for each provider are attached below demonstrating their working state.
I even requested developer access from the Tempest/WeatherFlow team to properly test this provider.
Comprehensive test coverage: A major advantage of the server-side architecture is the ability to thoroughly test providers with unit tests using real API response snapshots. Don't be alarmed by the many lines added in this PR - they are primarily test files and real-data mocks that ensure provider reliability.
Review Notes
I know this is an enormous change - I've been working on this for quite some time. Unfortunately, breaking it into smaller incremental PRs wasn't feasible due to the interdependencies between providers and the shared architecture.
Given the scope, it's nearly impossible to manually review every change. To ensure quality, I've used both CodeRabbit and GitHub Copilot to review the code multiple times in my fork, and both provided extensive and valuable feedback. Most importantly, my test setup with all 10 providers working successfully is very encouraging.
Related
Part of the HTTPFetcher migration #4016.
Screenshots