fix(config): disable default Prometheus telemetry sink (CON-335) - #4032
fix(config): disable default Prometheus telemetry sink (CON-335)#4032amir-deris wants to merge 1 commit into
Conversation
Default prometheus-retention-time to 0 so freshly initialized nodes keep the bounded in-memory sink without starting the Prometheus sink until an operator sets a positive retention value. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
PR SummaryLow Risk Overview Telemetry stays
Reviewed by Cursor Bugbot for commit 559e4b3. Bugbot is set up for automated code reviews on this repo. Configure here. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4032 +/- ##
==========================================
- Coverage 61.21% 60.21% -1.01%
==========================================
Files 2153 2054 -99
Lines 188381 176696 -11685
==========================================
- Hits 115324 106401 -8923
+ Misses 62315 60520 -1795
+ Partials 10742 9775 -967
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
The default prometheus-retention-time drops to 0 and the characterization records (agreement map, divergence table, fuzz row rationale, golden) are updated consistently with that. However the change only reaches the seid init pipeline: initAppConfig still hard-sets 60, so the second app.toml-generation path keeps enabling the Prometheus sink, and the PR's stated behavior does not hold there.
Findings: 1 blocking | 2 non-blocking | 1 posted inline
Blockers
- None at the file/PR level.
- 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- [suggestion]
config/cosmosbase/cosmosbase.go:54-56is now factually wrong: theforModegodoc says the self-written file pipeline writes "a metric retention of sixty where this writes seven thousand two hundred". After this changeseid initwrites 0, not 7200. That paragraph is the package's record of how the two generation pipelines differ, so it should be corrected in the same PR (and, given the divergence is now qualitative — one path enables the Prometheus sink and the other does not — it is worth pinning with a test rather than prose). - [suggestion] Operator-visible consequence not covered anywhere in the PR: with retention 0 and
telemetry.enabled = true,Metrics.gatherPrometheusreturnsprometheus metrics are not enabled, so the app API server'sGET /metrics?format=prometheus(sei-cosmos/server/api/server.go:155) starts failing for any freshlyseid init-ed node that previously scraped it. Worth a release note / upgrade-guide line so operators know to set a positive retention explicitly.
| Telemetry: telemetry.Config{ | ||
| Enabled: true, | ||
| PrometheusRetentionTime: 7200, | ||
| PrometheusRetentionTime: 0, |
There was a problem hiding this comment.
[blocker] This only changes one of the two app.toml-generation pipelines. cmd/seid/cmd/root.go:421 still sets srvCfg.Telemetry.PrometheusRetentionTime = 60 on the config returned by initAppConfig, and that config is what PersistentPreRunE → mgr.Apply → InterceptConfigsPreRunHandler (sei-cosmos/server/util.go:296-313) writes whenever app.toml is missing for any command other than init. So a node whose app.toml is auto-created by seid start (or any other non-init subcommand) still gets prometheus-retention-time = 60 and still starts the Prometheus sink with no operator opt-in — the PR description's claim that "the Prometheus sink is only created when an operator explicitly sets a positive retention value" does not hold for that path.
Before this change both pipelines enabled the sink (7200 vs 60) and only the retention window differed; now they disagree about whether the sink exists at all, which is a new and silent inconsistency. Either drop the = 60 override in root.go too, or state explicitly why the self-written file should keep Prometheus on.
There was a problem hiding this comment.
Worth taking a look at this one Amir. This is one of the challenges of existing config generation. It's done in multiple places. Ideally we won't need to deal with this in the near future
Describe your changes and provide context
CON-335
Default
app.tomlhadtelemetry.enabled = trueandprometheus-retention-time = 7200. That activates the Prometheus metrics sink intelemetry.New()even when Tendermint's[instrumentation].prometheusis disabled and nothing is scraping application metrics. Metrics accumulate in the sink for the full retention window without being collected.This PR changes the default
prometheus-retention-timefrom7200to0. Freshly initialized nodes still get the bounded in-memory telemetry sink (useful for SIGUSR1 dumps), but the Prometheus sink is only created when an operator explicitly sets a positive retention value.Existing nodes with
prometheus-retention-timealready written in theirapp.tomlare unchanged. Explicit configs (e.g.docker/rpcnode/config/app.toml) are left as-is.Testing performed to validate your change
go test ./sei-cosmos/server/config/go test ./sei-cosmos/telemetry/go test ./config/cosmosbase/TestMetrics_PromDisabledByDefaultRetentionto assert retention0does not enable the Prometheus sink