fix(sei-tendermint): bound P2P and consensus Prometheus label cardinality (PLT-1070, PLT-1071)

[amir-deris]

Summary

• Remove peer_id from tendermint_consensus_block_parts and tendermint_p2p_peer_receive_bytes_total
• Gate tendermint_consensus_late_votes to the current validator set: label active-set senders by validator_address, roll all others into other
• Keep remaining labels bounded (chID, message_type on receive-bytes)

Context

peer_id labels on P2P-fed counters grow without bound as peers churn over a node's lifetime. Those metrics are aggregated at the node level (or by channel/message type where labels are fixed).

late_votes previously labeled every distinct sender address forever, which also inflated cardinality as the validator set churned. Full label removal would have broken the Top-10 validators by late-votes rate panel on the Sei L1 Block Time SLO dashboard (sei-l1-block-time-slo). Active-set gating caps series at ~|validator set| + 1 per node while preserving per-validator attribution for live validators.

Platform impact

• No alert or recording-rule changes required
• SLO dashboard panel 21 (tendermint_consensus_late_votes) continues to work for current-set validators; an other overflow series may appear during catch-up or after set churn
• block_parts and peer_receive_bytes_total are not referenced in Platform dashboards or alerts

Closes PLT-1070 and PLT-1071.

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Aug 27, 2026, 11:11 AM

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.25%. Comparing base (4820c92) to head (8755d2c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4029      +/-   ##
==========================================
- Coverage   61.21%   60.25%   -0.97%     
==========================================
  Files        2153     2054      -99     
  Lines      188381   176693   -11688     
==========================================
- Hits       115324   106469    -8855     
+ Misses      62315    60459    -1856     
+ Partials    10742     9765     -977     

Flag	Coverage Δ
sei-chain-pr	77.61% <100.00%> (?)
sei-db	69.80% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-tendermint/internal/consensus/metrics.gen.go	98.48% <100.00%> (ø)
sei-tendermint/internal/consensus/metrics.go	97.22% <100.00%> (+0.07%)	⬆️
sei-tendermint/internal/consensus/reactor.go	76.36% <100.00%> (+5.99%)	⬆️
sei-tendermint/internal/consensus/state.go	74.88% <100.00%> (+0.81%)	⬆️
sei-tendermint/internal/p2p/metrics.gen.go	97.43% <100.00%> (ø)
sei-tendermint/internal/p2p/metrics.go	100.00% <ø> (ø)
sei-tendermint/internal/p2p/transport.go	83.72% <ø> (-0.19%)	⬇️

... and 133 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

Low Risk
Metrics-only changes with no consensus or P2P protocol impact; breaking change for PromQL that relied on peer_id on the affected counters.

Overview
Caps unbounded Prometheus series on Tendermint consensus and P2P metrics by changing how labels are applied, without altering consensus or networking behavior.

tendermint_consensus_block_parts drops the peer_id label and becomes a single node-level counter; block-part handling in the reactor increments BlockPartsAt() with no per-peer breakdown.

tendermint_p2p_peer_receive_bytes_total drops peer_id and aggregates receive bytes by chID and message_type only.

tendermint_consensus_late_votes still uses validator_address, but MarkLateVote now takes the vote address and the current validator set: active validators keep their address label; everyone else rolls into other, so historical or out-of-set addresses no longer create new time series forever.

Generated metrics.gen.go and help strings are updated to match. Dashboards that grouped by peer_id on the two P2P/consensus counters need query updates; the late-votes panel can keep per-validator views for the live set plus an other bucket.

^{Reviewed by Cursor Bugbot for commit 8755d2c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[seidroid]

Focused, correct cardinality fix: peer_id is dropped from two P2P-fed counters and late_votes is gated to the current validator set with an other overflow bucket, which also closes a path where any peer could mint arbitrary validator_address series. No blockers; the residual note is that Prometheus never evicts series, so the cap is the union of validator sets over the process lifetime rather than the set size claimed in the PR description.

Findings: 0 blocking | 2 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] No test covers the new MarkLateVote branching. A small unit test in sei-tendermint/internal/consensus asserting that an in-set address yields its own series while an out-of-set address (and a nil validator set) folds into other would pin the behavior the SLO dashboard panel depends on, and would catch a future refactor silently reverting to per-sender labeling.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] Active-set gating bounds the rate of new series but not the total: a Prometheus CounterVec never evicts a child, so every address that was in the set at the moment it sent a late vote keeps its series for the life of the process. The cap is therefore |union of validator sets over process lifetime| + 1, not the ~|validator set| + 1 the PR description states.

That is still a large improvement — the unbounded axis (peer churn, and pre-fix any peer minting a series with an arbitrary ValidatorAddress, since this marking happens before the vote's validator is checked against the set) is gone, and validator churn is slow and governance-paced. So this is worth adjusting the claim in the description/help text for rather than redesigning the label. If a hard cap is actually required, it needs an explicit DeleteLabelValues sweep on validator-set change, which is a meaningfully larger change and probably not warranted here.