Skip to content

fix(sei-tendermint): bound P2P and consensus Prometheus label cardinality (PLT-1070, PLT-1071) #4029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
masih merged 3 commits into from
Aug 27, 2026
Merged

fix(sei-tendermint): bound P2P and consensus Prometheus label cardinality (PLT-1070, PLT-1071) #4029

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e8dc19c
fix(sei-tendermint): bound P2P and consensus Prometheus label cardina…
amir-deris Aug 27, 2026
4c9cf93
fix(sei-tendermint): gate late_votes labels to the active validator set
amir-deris Aug 27, 2026
8755d2c
Merge branch 'main' into amir/plt-1070-plt-1071-metric-cardinality-issue
masih Aug 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions sei-tendermint/internal/consensus/metrics.gen.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 12 additions & 7 deletions sei-tendermint/internal/consensus/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ type Metrics struct {
// Whether or not a node is state syncing. 1 if yes, 0 if no.
StateSyncing tmprometheus.GaugeIntVec

// Number of block parts transmitted by each peer.
BlockParts tmprometheus.CounterIntVec `metrics_labels:"peer_id"`
// Number of block parts received from peers.
BlockParts tmprometheus.CounterIntVec

// Histogram of durations for each step in the consensus protocol.
StepDuration tmprometheus.HistogramVec `metrics_labels:"step" metrics_buckets:"exprange(0.1, 100, 8)"`
Expand Down Expand Up @@ -145,8 +145,8 @@ type Metrics struct {

// LateVotes stores the number of votes that were received by this node that
// correspond to earlier heights and rounds than this node is currently
// in.
//metrics:Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in.
// in, labeled by validator address for the current validator set.
//metrics:Number of late votes received by the node, labeled by validator address for the current validator set or other.
LateVotes tmprometheus.CounterIntVec `metrics_labels:"validator_address"`

// FinalRound stores the final round id the proposal block reach consensus in.
Expand Down Expand Up @@ -227,9 +227,14 @@ func (m *Metrics) MarkRound(r int32, st time.Time) {
m.RoundVotingPowerPercentAt(pcn).Set(0)
}

func (m *Metrics) MarkLateVote(vote *types.Vote) {
validator := vote.ValidatorAddress.String()
m.LateVotesAt(validator).Add(1)
const lateVoteOtherLabel = "other"

func (m *Metrics) MarkLateVote(addr types.Address, validators *types.ValidatorSet) {
label := lateVoteOtherLabel
if validators != nil && validators.HasAddress(addr) {

@seidroid seidroid Bot Aug 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Active-set gating bounds the rate of new series but not the total: a Prometheus CounterVec never evicts a child, so every address that was in the set at the moment it sent a late vote keeps its series for the life of the process. The cap is therefore |union of validator sets over process lifetime| + 1, not the ~|validator set| + 1 the PR description states.

That is still a large improvement — the unbounded axis (peer churn, and pre-fix any peer minting a series with an arbitrary ValidatorAddress, since this marking happens before the vote's validator is checked against the set) is gone, and validator churn is slow and governance-paced. So this is worth adjusting the claim in the description/help text for rather than redesigning the label. If a hard cap is actually required, it needs an explicit DeleteLabelValues sweep on validator-set change, which is a meaningfully larger change and probably not warranted here.

label = addr.String()
}
m.LateVotesAt(label).Add(1)
}

func (m *Metrics) MarkFinalRound(round int32, proposer string) {
Expand Down
2 changes: 1 addition & 1 deletion sei-tendermint/internal/consensus/reactor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -761,7 +761,7 @@ func (r *Reactor) handleDataMessage(ctx context.Context, m p2p.RecvMsg[*tmcons.M
return nil
case *BlockPartMessage:
ps.SetHasProposalBlockPart(msg.Height, msg.Round, int(msg.Part.Index))
Global.BlockPartsAt(string(m.From)).Add(1)
Global.BlockPartsAt().Add(1)
return utils.Send(ctx, r.state.peerMsgQueue, msgInfo{msg, m.From, tmtime.Now()})
default:
return fmt.Errorf("received unknown message on DataChannel: %T", msg)
Expand Down
2 changes: 1 addition & 1 deletion sei-tendermint/internal/consensus/state.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2456,7 +2456,7 @@ func (cs *State) addVote(
"cs_height", cs.roundState.Height(),
)
if vote.Height < cs.roundState.Height() || (vote.Height == cs.roundState.Height() && vote.Round < cs.roundState.Round()) {
Global.MarkLateVote(vote)
Global.MarkLateVote(vote.ValidatorAddress, cs.roundState.Validators())
}

// A precommit for the previous height?
Expand Down
8 changes: 4 additions & 4 deletions sei-tendermint/internal/p2p/metrics.gen.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions sei-tendermint/internal/p2p/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ var (
type Metrics struct {
// Number of peers.
peers prometheus.GaugeIntVec
// Number of bytes per channel received from a given peer.
peerReceiveBytesTotal prometheus.CounterIntVec `metrics_labels:"peer_id, chID, message_type"`
// Number of bytes per channel received.
peerReceiveBytesTotal prometheus.CounterIntVec `metrics_labels:"chID, message_type"`
// Number of newly established connections.
newConnections prometheus.CounterIntVec `metrics_labels:"direction, success"`

Expand Down
1 change: 0 additions & 1 deletion sei-tendermint/internal/p2p/transport.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ func (r *Router) connRecvRoutine(ctx context.Context, conn *ConnV2) error {
Global.queueDroppedMsgsAt(fmt.Sprint(chID), "in").Add(1)
}
Global.peerReceiveBytesTotalAt(
string(conn.ID),
fmt.Sprint(chID),
r.lc.ValueToMetricLabel(msg),
).Add(int64(gogoproto.Size(msg)))
Expand Down
Loading